Lesson 5: Probes at the Gates: Hooks and a Debugging Workflow

Learning goals:

• Explain what hooks are: user-defined handlers that execute automatically at fixed points in the agent lifecycle, organized into three cadences (once per session / once per turn / per tool call in the loop), and be able to pick the right event for an observability need, write the correct matcher, and identify which payload fields to use
• Avoid two traps that actually bite: hook subprocesses don't inherit the harness's OTel export config, and transcript files are written asynchronously so they may not contain recent messages when the hook fires
• Apply a five-step workflow to locate problems under non-determinism: narrow by prompt id, find the first divergence, replay with identical inputs, repeatedly stress the same component, and recover from the failure point after fixing

Prerequisites: Completed Lessons 1–4 (non-determinism and "can't tell why," raw records as primary evidence, structured logs and metrics, trace trees and telemetry pipeline traps) | Previous: << Lesson 4 | Next: Lesson 6 >>

You want to record before and after each tool execution

Lesson 3 taught you to write a structured log entry for each tool call in your own harness. Lesson 4 taught you to thread those records into a trace tree. That was your loop, you had the source code, you could drop a log() call wherever you wanted. But now you're running Claude Code in production — you don't have its runToolUses. What you want isn't complicated: record once before tool execution (what parameters it's about to use) and once after (what it got back). But the insertion points are in someone else's process.

The product anticipated this. The answer is called hooks.

Hooks are user-defined shell commands, HTTP endpoints, or LLM prompts that execute automatically at specific points in Claude Code's lifecycle¹. In plain English: you declare in config "at such-and-such moment, run this script for me," and when Claude Code reaches that moment it runs it. And it doesn't come empty-handed — when an event fires and a matcher matches, Claude Code passes JSON context about the event to your handler¹.

In course 7 of this series, you wrote an approval gate in your harness loop — when it encountered a HIGH_IMPACT tool, it stopped and waited for human confirmation before executing. That was a hand-rolled interception point. Hooks do the same thing, but they've organized "which positions in the loop are worth pausing at" into a named list where each position has a fixed, dependable payload. For observability, the value of this list isn't that you "can change behavior" — it's that you can see without changing behavior.

Three cadences: session, turn, tool call

Events fall into three cadences¹:

• Once per session: SessionStart and SessionEnd
• Once per turn: UserPromptSubmit, Stop, and StopFailure (from the names, StopFailure corresponds to the exit where the turn didn't finish normally — the official citation only gave the cadence classification; rely on your reference page for precise semantics)
• On every tool call inside the agentic loop: PreToolUse and PostToolUse

Line this up against the loop anatomy from course 7, and the three layers match immediately:

SessionStart                       ← session begins (entire runAgent starts)  UserPromptSubmit                 ← turn begins (user submits prompt)    while (stop_reason === "tool_use") {       ... model request ...       PreToolUse                  ← inside loop body, parameters generated, not yet executed       ... tool execution ...       PostToolUse                 ← inside loop body, tool finished executing    }  Stop / StopFailure               ← turn endsSessionEnd                         ← session ends

Picking the wrong cadence gives you numbers that are hard to explain: if you want to count "how many tool calls did one task use" but hook into the per-turn gate, you'll only get zero. Before picking an event, ask: how many times per session does the thing I'm counting happen?

One detail worth calling out: SessionStart fires when you open a new session and also when you resume an existing one¹. Course 9 covered --resume — from the loop's perspective that's not "starting fresh," but it still rings the SessionStart bell. If you write "initialize a new log when session starts," the first time you hit resume you'll overwrite the previous segment.

Gate details useful for observability

PreToolUse runs after Claude creates tool parameters and before processing the tool call¹. This gap matters: parameters are finalized (you can see exactly what the model intends to use) but the tool hasn't run yet. Lesson 2 covered a real case — the team discovered Claude would needlessly append 2025 to the search tool's query parameter, biasing results². Evidence for this kind of bug lives in the parameters that PreToolUse can see.

PostToolUse hooks fire after a tool has already executed successfully. The input includes both tool_input, the arguments sent to the tool, and tool_response, the result it returned¹. One firing gives you a complete call record; you don't have to piece together "which request goes with which response" yourself. The principle Lesson 2 emphasized — the complete round-trip is the primary evidence² — is handed to you here field by field. Note its trigger condition is "already executed successfully"¹; if you want to cover cases where parameters were generated but execution didn't succeed, you need to pair it with PreToolUse and diff the two sides.

How to write matchers: To run a hook after any tool completes successfully, omit the matcher or set it to "*"¹. Observability scenarios want exactly this one-hook-records-all behavior — you don't know which tool will break, so you record them all.

How handlers send and receive data: Command hooks receive JSON data via stdin and communicate results through exit codes, stdout, and stderr¹. So a minimal observability handler is just "read JSON from stdin, pick a few fields to append to a file, exit with code 0" — no magic involved.

This is the JSON Lines log from Lesson 3, except the one writing the log has changed from "your harness" to "a small script you hung on someone else's loop." Consult the reference page for your version for exact config structure and field names.

The payload includes duration, but check the definition: The payload carries an optional field for tool execution time in milliseconds, which excludes time spent in permission prompts and PreToolUse hooks¹. The second half is key: user-perceived "how long did I wait for this step" includes the time they spent clicking confirm, but this number strips it out. Using it to answer "is the tool slow" is correct; using it to answer "how long did the user wait" will be systematically low. Lesson 4 mentioned the same thing another way — the tool span has two child spans underneath, one for waiting on the permission decision and one for execution itself³. They're recorded separately precisely because these two segments shouldn't be mixed.

Hooks themselves are in the trace too: Each user prompt starts a claude_code.interaction root span. API calls, tool calls, and hook executions are recorded as its children³. The observability mechanism is itself an observed object.

Two warnings that bite

First: hook subprocesses don't inherit OTEL_* exporter variables. One set of variables is not inherited: Claude Code removes OTEL_* exporter variables from every subprocess it spawns, including hooks¹.

This sentence directly kills a very natural idea: "The harness already has the endpoint, protocol, and auth headers configured. I'll just import an OTel SDK in my hook and the environment variables will work out of the box." — They won't. When the handler process starts, those variables have already been stripped. The data either won't export or will hit the default endpoint and vanish. Don't expect anyone to yell and warn you: Lesson 4 covered that the CLI's own export fails silently⁴, and by default the exporter you build yourself in the hook won't be any louder — when no data reaches the backend, both sides are quiet.

You have two paths forward: either the hook carries its own complete export config (explicitly write the endpoint and auth in the script, don't rely on inheritance), or don't emit telemetry from the hook at all — let it write a structured log and align it with telemetry in the backend using IDs. The second path has official support: the UUID identifying the user prompt currently being processed in the hook payload matches the prompt.id attribute on OpenTelemetry events, so you can correlate hook output with telemetry for a single prompt¹. Each side writes its own, then you join on the same prompt id at the end — same trick as Lesson 4's "thread into a tree using correlation IDs," except this time across two data sources.

Second: the transcript file is written asynchronously. The payload gives you a path to the conversation JSON, but this file is written asynchronously and may lag the in-memory conversation, so it may not yet include the current turn's most recent messages when the hook fires¹.

The trap is it doesn't error, it just gives you stale data. Someone might think "the payload fields aren't enough, I'll read the transcript directly, it has everything," and end up with records that intermittently miss half their content. If you want tool_input and tool_response, use the payload fields — those are what PostToolUse explicitly guarantees to provide¹. Transcripts are good for looking back afterward, not for using as a real-time data source at the instant the hook fires.

A reminder tied to your machine: command hooks execute shell commands with your full user permissions. They can modify, delete, or access any files your user account can access. Review and test all hook commands before adding them to your configuration¹.

When hooks don't do their job

You hung a hook, the log file is empty. Did it not fire? Matcher didn't match? Or did the script itself crash? At this point the object you need to observe is the observability mechanism itself. Lesson 4's principle applies again here: verify the probe first.

Hook execution details — which hooks matched, their exit codes, and full stdout and stderr — are written to the debug log file¹. Get it one of two ways: use claude --debug-file <path> to write to a location you specify, or run claude --debug and then read ~/.claude/debug/<session-id>.txt¹. For more granular hook matching details, set CLAUDE_CODE_DEBUG_LOG_LEVEL=verbose to see additional log lines such as hook matcher counts¹.

Troubleshooting sequence: log file is empty └─ Does the debug log have a match record for this hook?      No   → matcher is wrong; confirm it fires first with omitted matcher      Yes  → Is the exit code 0?               No  → read stderr, likely script path, permissions, or JSON parse               Yes → ran but didn't write to the right place; check write path and directory existence

A debugging workflow: how to track down a problem under non-determinism

First half of the lesson covered where to place probes. Second half covers how to actually use that data to track down a real problem.

Start by stating the difficulty clearly. Lesson 1 said: agents make dynamic decisions and are non-deterministic between runs, even with identical prompts. This makes debugging harder⁵. Step one in traditional debugging is "reproduce," and that step doesn't hold here — you re-run it, and it might take a completely different but equally valid path.

The five steps below are this course's own arrangement, not anyone's official methodology; but each step stands on a source.

Step one: narrow

In the records you've accumulated across lessons 1–4, first narrow the scope to "this one problematic prompt." Lesson 4 gave you the method: to trace all activity triggered by a single prompt, filter your events by a specific prompt.id value³.

This step's significance isn't technical, it's psychological. "The agent broke" is a proposition you can't debug; "one of these 11 events under this prompt id is off" is a proposition you can debug.

Step two: find the first divergence

Read from the beginning forward and find the first step where behavior starts to deviate from expectations. Why insist on "first"? Because one step failing can cause agents to explore entirely different trajectories, leading to unpredictable outcomes⁵. The absurdities you see at the end (referencing non-existent files, erroring repeatedly, taking the long way around) are mostly downstream noise. Fix event #8 and you're probably just cleaning up after event #4's mistake.

Judging "deviation" has a few useful shapes²: called a tool it shouldn't have called, called the right tool with wrong parameters, called the right tool too few times, or processed the tool response incorrectly. The last kind is hardest to spot because the tool itself returned success, everything's green in the logs — what's wrong is the agent's interpretation of that successful result.

Step three: replay and observe

The official approach: to understand the effects of prompts, built simulations using the exact prompts and tools from the system, then watched agents work step-by-step; this immediately revealed failure modes like agents continuing when they already had sufficient results, using overly verbose search queries, or selecting incorrect tools⁵.

"Immediately revealed" is worth thinking about. These same bugs are invisible in aggregate metrics (success rate is pretty high), take line-by-line reading to notice in after-the-fact logs, but when you watch it run step by step, the human eye spots "it already has enough but keeps searching" in seconds. Two things to control during replay: input must be identical (prompts, tool definitions unchanged), and observation must be step-by-step.

Step four: repeatedly stress the same component

If suspicion falls on a specific tool, running it a few times likely won't show anything — non-determinism makes bugs appear and vanish.

The team built a tool-testing agent: give it a flawed MCP tool, it attempts to use it, then rewrites the tool description to avoid failures. By testing the tool dozens of times, this agent found key nuances and bugs⁵.

"Dozens of times" is the point. A bug that one run can hide, dozens of runs will force out: a return shape that's odd under boundary inputs, an ambiguous description, an error message that makes the model misjudge it as "just retry." The diagnostic readings from Lesson 3 connect here — lots of redundant tool calls might suggest pagination or token limit parameters need rightsizing; lots of tool errors for invalid parameters might suggest tools could use clearer descriptions or better examples². One thing that saves many round-trips: when a tool call raises an error, you can prompt-engineer your error responses to clearly communicate specific and actionable improvements, rather than opaque error codes or tracebacks².

Step five: recover from the failure point after fixing

After you fix it, don't reflexively hit "start over." The official line is direct: when errors occur, we can't just restart from the beginning: restarts are expensive and frustrating for users. Instead, we built systems that can resume from where the agent was when the errors occurred⁵.

Course 9 covered how to build recovery mechanisms. That lesson's scenario was "how to pick up after task interruption." In debugging it's a different use: the dozens of tool calls before the failure point were valid, cost money, gave correct results. Re-running them gives you nothing but burned tokens, and introduces a bunch of new non-determinism so you can't tell whether "it worked this time" is because you fixed it right or because you got lucky.

About "reproduction," an honest word

You might have seen elsewhere a set of techniques to make agents reproducible: fix the random seed, set temperature to 0, record real tool returns and use them as replay stubs.

These practices do exist in engineering. Courses 8–10's hands-on labs use that same stub client approach — save the tool_result from one real run, return the same data every time afterward, and the tool side becomes deterministic. Good for verifying "did the line of code I changed break the parsing logic."

But two things need saying. First, these techniques have no first-party source backing. I gave citations for every step of the five-step flow above. For this paragraph I'm not giving any, because there genuinely aren't any. If you see someone claim "official recommendation is temperature 0 to reproduce agent issues," ask them for the link.

Second, they lock down less than they sound like they do. Stubbing tool returns locks down the environment; the model side is still non-deterministic⁵. So it turns "two variables moving" into "one variable moving" — that's valuable, but it's not the "same input must give same output" kind of reproduction. Don't treat it as a guarantee, treat it as noise reduction.

What problems are worth this workflow

Walking all five steps has a cost: narrowing requires reading logs, replay requires building a simulation, stressing a component means dozens of runs. A rough but workable split:

• Low-frequency, harmless jitter — like one time it called search once extra, but the result was still right. Log it, accumulate. Each individual instance isn't worth investigating; after you've collected a dozen you can often see a common pattern, and investigating once at that point is much more efficient.
• High-impact — gave a factually wrong answer, modified a file it shouldn't have touched, hung without returning. Doesn't matter how low the frequency, file a case. One occurrence is already expensive enough.
• Recurring — same failure shape shows up a third time, which means it's not luck, it's structural. File a case.

Lesson 1's line is the decision criterion here: for instance, users would report agents "not finding obvious information," but we couldn't see why. Were the agents using bad search queries? Choosing poor sources? Hitting tool failures?⁵ You decide not to investigate a problem, you're accepting "I don't know which cause it is" — for harmless jitter that's fine, for high-impact problems you're gambling.

This lesson's been on paper so far. Lesson 6 puts both halves together hands-on: wire up a full observability layer onto the course 7 harness, then take an "I can't tell why" symptom and chase it all the way down.

Recap

• Hooks are user-defined commands, HTTP endpoints, or LLM prompts that execute automatically at specific points in Claude Code's lifecycle; when an event fires and a matcher matches, Claude Code passes JSON context about the event to your handler¹.
• Events fall into three cadences — once per session (SessionStart/SessionEnd), once per turn (UserPromptSubmit/Stop/StopFailure), on every tool call inside the loop (PreToolUse/PostToolUse)¹. Fix the cadence before picking the event. SessionStart also fires when resuming a session¹.
• PreToolUse runs after Claude creates tool parameters and before processing the tool call; PostToolUse fires after successful tool execution, with input carrying both tool_input and tool_response, giving you a complete call record in one firing¹. Omit the matcher or set it to "*" for one-hook-records-all¹.
• Command hooks receive JSON via stdin, respond through exit codes, stdout, and stderr¹; that execution milliseconds field excludes time spent in permission prompts and PreToolUse¹, so don't use it as user wait time.
• Two negative conclusions: Claude Code removes OTEL_* exporter variables from every subprocess it spawns (including hooks)¹, so if you want to emit telemetry from a hook you must carry your own export config; the transcript file provided in the payload is written asynchronously and may not yet include the current turn's most recent messages when the hook fires¹. Command hooks execute with your full user permissions, so review before installing¹.
• When hooks don't work, check the debug log for which hooks matched, exit codes, and full stdout/stderr using --debug-file¹; for matcher counts set log level to verbose¹. The prompt id in the hook payload and the prompt.id on telemetry events are the same value, so the two data sources can align¹.
• Five-step debugging workflow (this course's arrangement): filter events by prompt.id to narrow to this prompt³ → find the first divergence, because one step failing changes the entire trajectory⁵ → replay using identical prompts and tools, watching step-by-step⁵ → stress the suspected component dozens of times to force out the bug⁵ → after fixing, recover from the failure point instead of restarting from the beginning⁵.
• Techniques like fixed seed, temperature 0, and recorded tool return replay have no first-party source backing; this course presents them as engineering practice: they lock down the environment side, but the model remains non-deterministic⁵, so they're noise reduction, not reproduction guarantees.
• Not every problem warrants the full workflow: log low-frequency harmless jitter, accumulate for patterns, investigate later; file cases for high-impact or recurring issues immediately.

>> Lesson 6: Hands-On: Wiring an Observability Layer onto the Harness

💻 Exercises