Lesson 6: Hands-On: Wiring Checkpointing and Resume onto the Harness
Learning goals:
- Actually weld the "save the dangling call at Point A, clear it at Point B" checkpoint scheme into the
runAgentloop from Course 7 in this series, instead of leaving it as a concept diagram- Attach a side-effects ledger to
runToolUses: the moment a tool succeeds, write one record to disk so resume can tell whether "this tool actually ran or not"- Write the three-way split in
reconcile, and use a controlled "simulated kill +--resume" run to watch, with your own eyes, that recovery behaves the way it shouldPrerequisites: You've read Lessons 1-5 and can run the harness loop from Course 7 in this series, "Agent Harness Fundamentals: Loops and Control" | Prev: Lesson 5 <<
See It Run First
The first five lessons pulled checkpoints, resume, idempotency, and rewind/fork apart and explained each one. This lesson welds them into a harness that actually runs: the same familiar loop — call the model with messages, and when stop_reason === "tool_use" execute the tool and call again — except this time every turn writes two checkpoints to disk, plus a ledger recording tool-execution results. The task is "turn sales notes into a report," calling three tools in sequence: read_notes, count_words, write_report. Here's what it looks like running normally up to turn three and then getting killed outright:
Turns 1 and 2 both walked the full save A → execute → save B three-step, all normal. Turn 3 saved save A (recording that the dangling call is write_report), the tool did in fact finish executing, and its result was already written into the ledger — but the next step's save B never got saved before the process was killed. This is exactly the window this lesson is out to nail: at this moment checkpoint.json still holds a dangling pendingToolUse. Carrying that scene, pick it back up with --resume:
The resume flow reads turn=3 pending=write_report, checks the ledger — and finds this call had actually finished and been recorded before the kill, so it reuses that record directly and does not re-execute write_report, fills in the save B this turn was missing, and carries on to the model's wrap-up as usual. The whole task never started over from scratch, and the report never got written twice.
These two terminal outputs aren't hand-written examples. They're the real output of the Node script driven by a fixed response queue in the "Verification Harness" section below, copied here line for line.
Building It Block by Block
Reading and Writing Checkpoints: saveCheckpoint / loadCheckpoint
A checkpoint is just this scene — {version, task, turns, tokensUsed, messages, pendingToolUse} — serialized to disk. The only thing to be careful about is not corrupting the file: write to a temp file first, then swap it in atomically with fs.renameSync — rename is an indivisible operation within the same filesystem, so there's never a "half-written" intermediate state:
Reading has to hold up against two things: the file not existing (never run before, or meant to start fresh), and the file failing to parse. The second case deserves extra care — a JSON.parse failure usually means the previous write itself was interrupted (saveCheckpoint is atomic in theory, but if the process is killed before even the .tmp file is fully written, or the disk itself has a problem, a half-finished file from before the rename might get misread). At that point you must never quietly reset the state to empty and pretend nothing happened — that's the real place tasks get lost. The right move is to throw the error out plainly, telling the user this checkpoint is no longer trustworthy and should be deleted so they can start over, rather than letting the program guess its way back to a whole state:
Check the version field while you're at it: if the checkpoint structure changes later, an old file shouldn't be force-parsed as the new format — better to refuse to load than to read out a state that's half right and half wrong. Both functions were tested with real truncated JSON: feed in a half-written {"version":1,"turns":3,"pendingT and loadCheckpoint throws exactly the "delete it and start over" error above, never returning any plausible-looking default.
Point A and Point B: Wiring Them into the runAgent Loop
The loop skeleton from Course 7 in this series hasn't changed — while (response.stop_reason === "tool_use"), push assistant → execute tool → push tool_result → request the model again. This lesson inserts two checkpoints into the loop body, and where they go is the whole point of the lesson:
Point A goes in after response arrives and before messages.push({ role: "assistant", ... }) — the moment the model has "named a tool but not actually executed it," and pendingToolUse records that naming verbatim. Point B goes in after runToolUses finishes and the tool_result has been pushed into messages — at that point this turn is fully closed out, and pendingToolUse is cleared to null. Sandwiched between the two saves is exactly the stretch of code where the tool really executes; if the process happens to die during that stretch or right after it, what's left on disk is the "Point A saved, Point B not saved" scene — pendingToolUse non-empty, which is precisely the signal the recovery logic is built to handle.
To make this single-dangling-call protocol (pendingToolUse is one object, not an array) hold together, this lesson designs the task so the model names exactly one tool per turn — a deliberate simplification whose boundary the "Proportion" section spells out.
The Effects Ledger: Wiring It into runToolUses
The problem the ledger solves is: if a crash lands right between "the tool actually finished executing" and "the result landed in messages," how does resume know whether this call already ran and must not run again. The approach is, the moment a tool succeeds, write its result separately into a ledger keyed by tool_use_id (again with the temp-file-plus-rename atomic write):
The order can't be swapped: you have to get the real result of toolImpls[block.name](block.input) first, and only then can saveEffect write it down — execute first, record after. The ledger records "this thing really happened, and this was its result." If you flipped it and recorded before executing, all that could land in the ledger would be a placeholder, and the ledger would lose the whole meaning of its "already done" promise (the Level 2 exercise has you reproduce this anti-pattern with your own hands).
In a normal single execution, runToolUses walks the "execute → record" two-step, because each tool_use_id shows up for the first time with nothing to look up. The one dangling call that resume has to handle walks the fuller "check the ledger → execute (if needed) → record (if executed)" three-step — reconcile below is that three-step's implementation, and both follow the same discipline: never write "already done" into the ledger before you have a real result.
reconcile: The Three-Way Split for a Dangling Call After a Crash
What resume has to handle is that one (if any) pendingToolUse in the checkpoint. It corresponds to three possibilities:
Three branches, for three scenarios that were all really tested:
- Ledger hit — this is the crash demo from the top of the lesson:
write_reporthad in fact finished executing and been recorded, onlysave Bdidn't make it. On resume, reuse the result in the ledger directly, don't re-execute, avoid writing the report twice. - Ledger miss + read-only tool — something like
read_notes, a tool with no side effects; crashing before the record lands doesn't matter, so just re-run it once to get the result and record this run into the ledger while you're at it: - Ledger miss + side effects — something like
write_report, a tool that changes outside state, crashing before the record lands: you don't know whether it actually ran (on a real filesystem,write_report's side effect could well have already happened, just without getting recorded to the ledger). Here, rather than guess, use anis_error: truetool_resultto honestly tell the model "this call's state is unknown," handing the judgment back to it:
All three log lines are real output, not invented — reconcile itself doesn't need to know what the task is; give it a pendingToolUse and the matching ledger state and each of the three branches is independently testable.
The Entry Point: --resume in main()
Last is the entry point. main() makes exactly one decision: does the command line have --resume. If it does, go through loadCheckpoint() to recover; if it doesn't, clear out the checkpoint and ledger files left over from last time and start fresh — this cleanup guarantees that "start over without --resume" is always a clean opening, never polluted by a half-finished scene from a previous run:
Inside runAgent there are two matching paths: when opts.resume is true it calls loadCheckpoint(), runs reconcile, pushes the reconciled result (if any) into messages and saves one Point B checkpoint, then sends the request to the model as usual; when it's false it fs.rmSyncs the old checkpoint and ledger and starts from an empty messages. In the real agent.js, the model client is swapped for @anthropic-ai/sdk's client.messages.create({ model, max_tokens, tools, messages }), and nothing else about the structure changes.
Citing the Protocol
Neither of this lesson's two design decisions was set arbitrarily.
When the ledger is missing and reconcile can't be sure of the state, it chooses to append an is_error: true tool_result rather than silently skip — resting on the protocol's hard requirement about pairing content blocks: every tool_use must come back with a matching tool_result, all returned together, each claimed by its tool_use_id1. Skipping the Point A save would leave the resume flow unaware that the call ever happened, so it couldn't satisfy that pairing rule at all; the whole point of reconcile existing is to guarantee that, ledger hit or miss, the dangling call ends up with a paired tool_result.
Choosing "resume and keep going" over "error out and start over" echoes what Anthropic's engineering team described in the retrospective on its research system: when errors occur you can't just restart, because "restarts are expensive and frustrating for users," so instead they "built systems that can resume from where the agent was when the errors occurred"2. The same retrospective notes that an agent's adaptability can be paired with — rather than pitted against — deterministic safeguards, combining "the adaptability of AI agents built on Claude with deterministic safeguards like retry logic and regular checkpoints"2. Checkpoints catch the deterministic failure — "the process died" — while the model's adaptability handles the kind of case code can't hard-decide, like "the ledger is unresolvable." reconcile's is_error branch is where the two meet: it tells the model the truth about the unknown state and lets it decide whether to verify or retry, and "letting the agent know when a tool is failing and letting it adapt works surprisingly well"2.
The Verification Harness
The two terminal demos in this lesson don't rely on actually killing a process to see what happens — that way the crash timing would be different every run, and you couldn't make a targeted assertion like "the crash happened after the Nth tool call, and the recovery behavior is correct." The approach is to swap the model client for a stub that plays cards in a fixed order: a response queue that, on each call to messages.create, hands out the next pre-written response in sequence, and throws outright if you keep calling after the queue is drained — so which tool the task calls on which turn, and when the model wraps up, are all hard-coded constants that don't shift because of one real call.
"Killing the process" is a crashPoint(label) controlled by an environment variable: every time runToolUses finishes a ledger write, it stitches "which write this is" into a string label, compares it against the CRASH_AFTER environment variable, and on a match throws a dedicated SimulatedCrash exception. This turns "crash after the Nth tool call" into an integer you can specify precisely, rather than a chance event at the mercy of timing. main() catches only this one exception at the outermost layer, prints a single [kill] log line, and exits with 137 (the conventional "killed by SIGKILL" exit code), so the demo reads like a real process kill instead of an ugly stack trace.
This "pin the content with a response queue, pin the crash count with a label" method is the same idea Lesson 6 of Course 8 in this series, "Context Engineering: Spending Finite Attention Where It Counts," used to verify context engineering: pin down the things that were otherwise nondeterministic (what the model says this time, where the process dies this time) into fixed quantities first, and only then can recovery behavior be asserted line by line instead of coming out different every run. That's how this lesson verified all three branches — "ledger hit, don't re-run," "ledger miss on a read-only tool, re-run directly," "ledger miss on a side-effecting tool, append is_error" — plus loadCheckpoint's tolerance of a truncated file, each checked one by one with a real node run rather than reasoned about only on paper.
Proportion: Not Every Task Needs This
The machinery welded on in this lesson — two checkpoints, one ledger, a three-way reconcile — is meant for long tasks that run many turns in a row and have side effects in between. A small task that finishes in a few seconds and can just be re-run on failure may not be worth carrying this whole apparatus of disk I/O and state machine; here you can borrow the same proportion Course 7 in this series, "Agent Harness Fundamentals: Loops and Control," cited: what's worth considering is that "you should consider adding complexity only when it demonstrably improves outcomes"3. This isn't a hard "you must do it this way" rule, more a question to ask yourself before you start: is this task really long enough, really important enough, to be worth maintaining a checkpoint for?
This lesson's implementation also draws two explicit boundaries, worth saying out loud so you don't treat it as "learn it and drop it straight into production":
- Each turn handles exactly one dangling
pendingToolUse, matching the demo task where the model names one tool per turn. In a real setting a single model response could well carry several concurrenttool_useblocks (therunToolUsesfrom Course 7 in this series runs them concurrently withPromise.all); extending this lesson's single-dangling-call protocol into a set of dangling calls means turningpendingToolUsefrom an object into an array and runningreconcileover each one. This lesson deliberately left that layer of complexity out, to get the reconciliation logic for a single dangling call across clearly first. - The checkpoint and ledger in this lesson govern one thing: "one process, running one task." How multiple sessions share state, whether multiple processes touching the same checkpoint at once conflict, how cross-machine consistency is guaranteed — these belong to multi-session concurrency and distributed consistency, and they're not in this lesson, nor in the scope of this course.
Recap
- The checkpoint saves twice per turn: Point A records the dangling
pendingToolUseafter the model response arrives, Point B clears it to null after the tool result lands inmessages; saving only Point B makes the window between "the model names a tool" and "the result is recorded" completely invisible in the checkpoint, and since everytool_usemust come back paired with atool_result1, Point A is exactly what makes the dangling call inside that window traceable - The side-effects ledger records by
tool_use_id, and its discipline is "execute first, record after" — recording is premised on already having a real result; reverse it and you mis-record "not yet run" as "already done" reconcile's three-way split handles the dangling call on resume: ledger hit, reuse and don't re-run; ledger miss but read-only, re-run directly; ledger miss with side effects, don't guess, append anis_errortool_resulthanding the state honestly back to the model — this echoes the two engineering lessons of "you can't restart from scratch on error, you have to resume from where it hit" and "let the model know a tool failed, leave it to adapt, and it works surprisingly well"2, and it lines up with the "deterministic safeguards paired with model adaptability" idea2- The checkpoint-and-ledger machinery isn't free; add it only when the complexity demonstrably improves outcomes3; this lesson's implementation only governs "one process running one task," and multi-session concurrency and distributed consistency are not among its concerns, nor in the scope of this course
You've now finished this course. Starting from the judgment that "agents are stateful and errors compound," you worked all the way through what a checkpoint should save, when to write it to disk, how to handle a dangling call on resume, how idempotency backstops recovery, and how a checkpoint can further serve rewind and fork — up to this lesson, where you welded them by hand into a harness that really runs, really gets killed, and really picks back up and finishes. What you hold now isn't just a set of concepts but a stretch of code verified by real node execution. Wire it onto your own harness, and the next time it really does get killed, it'll pick right back up from where it left off.
Footnotes
-
Handle tool calls — Claude API — https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls ↩ ↩2
-
How we built our multi-agent research system — Anthropic Engineering — https://www.anthropic.com/engineering/multi-agent-research-system ↩ ↩2 ↩3 ↩4 ↩5
-
Building Effective AI Agents — Anthropic Engineering — https://www.anthropic.com/engineering/building-effective-agents ↩ ↩2
Exercises
The incident report reads: "The user interrupted a task, and after --resume a tool got skipped — the log showed it as 'already done,' but this tool was never actually executed, and the file it was supposed to write simply doesn't exist." You dig up the runToolUses that was running in production at the time and find one difference from this lesson's version:
Level 2: Find the Order Error Where the Ledger Was Written BackwardsFind this order error, explain clearly why it causes "a tool that clearly never ran gets treated as done," and fix the order. Then, following the method in this lesson's "Verification Harness" section, write a small script to reproduce it: insert an environment-variable-controlled simulated crash point between saveEffect and toolImpls[block.name](...), and run it for real with node — under the wrong order, the ledger already holds a result: null record before the crash; with the order fixed, the same crash point leaves no entry for this tool_use_id in the ledger at all.