Lesson 3: Deterministic Verifiers: Only Checks That Output Pass/Fail Count
Learning goals:
- Rank candidate verifiers by "fastest, most reliable, most scalable" and pick the right one for a specific output
- Write a deterministic verification script that returns pass/fail in a form the agent can read and iterate on
- Recognize false negatives where over-strict verifiers reject correct outputs, fix them with normalization, and understand where deterministic checks hit their ceiling
Prerequisites: Lessons 1 and 2 (without a runnable check, "looks done" is the only signal; verify end state first, process as backstop; success criteria must be measurable) | Previous: << Lesson 2 | Next: Lesson 4 >>
From "what to verify" to "how to verify it"
By the end of Lesson 2, you should have a concrete success criteria written down. Take the example this lesson uses: the agent reads a batch of sales CSVs, aggregates them into a report.json with a title, per-channel items, and a total. Lesson 2 taught you to frame the criteria as end state—file exists, fields present, total equals sum of item counts—rather than turn-by-turn process checks like "read file, then compute sum, then write file."
You have the criteria. The next question is practical: what do you use to check against that criteria?
You could eyeball it. You could have another model read it and give feedback. Or you could write a dozen lines of Node that loads the JSON, sums the counts, and exits non-zero if they don't match. All three reach a conclusion, but the cost and reliability differ wildly.
The official docs offer a ranking principle: choose the fastest, most reliable, most scalable grading method1. By that yardstick, the three categories land in clear order:
- Code-based grading—fastest and most reliable, extremely scalable; the weakness is that it lacks nuance for complex judgments that need less rule-based rigidity1.
- LLM-based grading—fast and flexible, scalable and suitable for complex judgment, but test to ensure reliability first then scale1. (That's Lesson 4.)
- Human grading—most flexible and high quality, but slow and expensive; avoid if possible1.
"Reliable" here means same input, same verdict every time. Code-based grading ranks first not because it's smart, but because it's predictably dumb—it won't pass you today on good vibes and fail you tomorrow over phrasing. In computing, deterministic systems produce the same output every time given identical inputs, while non-deterministic systems—like agents—can generate varied responses even with the same starting conditions2. The "deterministic verifiers" this lesson covers are that kind of predictably dumb check: a deterministic thing evaluating a non-deterministic thing.
One related principle for designing eval tasks: structure questions to allow for automated grading (for example, multiple-choice, string match, code-graded, LLM-graded)1. Where you have a shot at automation, take it.
"Verifier" sounds like a specialized framework, but the bar is much lower. The official docs definition is almost blunt: the check is anything that returns a signal Claude can read in the conversation: a test suite, a build exit code, a linter, a script that diffs output against a fixture, or a browser screenshot compared against a design3.
Unpack that list. Running npm test that goes all green is a pass, perfect for code deliverables—code solutions are verifiable through automated tests4. Build exit codes are the easiest: the toolchain already wrote the assertions for you. A linter alone isn't enough, but it's great as the floor for "mistakes you shouldn't make." A diff script compares this run's output to a fixture file you prepared (a known-good sample), suitable when output is stable and format is fixed. Screenshot comparison is for when you're tweaking frontend styles.
One engineering-practice note: compilers and static type checkers (like tsc --noEmit) are often used this way in real projects because they also emit exit codes and readable error locations. That's my addition, not on the official list—don't treat it as officially endorsed.
These methods aren't mutually exclusive. Verifiers form a spectrum—one end is "exact string match against a fixture," the other end is "ask Claude to judge"2. This lesson covers the left half, Lesson 4 goes right.
Minimal form: output == golden_answer
The left edge of the spectrum looks like this1:
Just an equality check. This is called exact match. It measures whether the model's output matches a predefined correct answer, typically after normalizing whitespace and case; it's a simple, unambiguous metric that's perfect for tasks with clear-cut, categorical answers like sentiment analysis (positive, negative, neutral)1.
"Normalization" here is plain: before comparing, erase differences that don't carry meaning. In code:
The second call's result is worth staring at. trim() and toLowerCase() rescue newlines and case, but not that period. How far normalization should go depends on which differences are irrelevant to your task—a judgment no library can make for you. The pitfall section later in this lesson is about what happens when you get that judgment wrong.
Make the verifier's output readable
That definition had a clause often skipped: the signal must be something Claude can read in the conversation3. That clause determines how your verification script should write its output. Compare two failure messages:
The first tells the agent "you're wrong," then leaves it guessing. The second tells it which field failed, what was expected, what was actual—next turn it can go fix that number directly. Same pass/fail, order-of-magnitude difference in information. Another piece of official guidance says the same thing: have Claude show evidence rather than asserting success—the test output, the command it ran and what it returned, or a screenshot of the result; reviewing evidence is faster than re-running the verification yourself, and it works for sessions you weren't watching3. Your verification script is the producer of that evidence. If it's vague, the evidence is vague.
Two practical rules: use exit code 0 for pass, non-zero for fail (CI and shell && can use it directly). In stdout, one failure per line, stating "which field, expected what, got what."
With pass/fail, the agent's behavior changes
During execution, agents need "ground truth" from the environment at each step (such as tool call results or code execution) to assess progress4. Without a verifier, the only progress signal it can get is the paragraph it just wrote—if it thinks it's done, it's done. With a verifier, the environment contains a source of fact independent of its own judgment.
So the behavior chain changes: give Claude something that produces a pass or fail, and the loop closes on its own; Claude does the work, runs the check, reads the result, and iterates until the check passes3. Put another way: agents can iterate on solutions using test results as feedback4.
In the kind of harness loop you hand-wrote in Course 7 of this series, the implementation is to expose the check as a tool:
The verifier's output flows back into the conversation via tool_result. The model reads total mismatch: declared 48, item counts sum to 50, and next turn it goes to fix it. You participated in zero steps.
Two convenient extensions. First, checks don't have to sit only at the end. Lesson 2 covered how complex workflows can be broken into discrete checkpoints where specific state changes should have occurred, rather than validating every intermediate step5. Those verification checkpoints are natural landing spots for deterministic checks—like "after all CSVs are read, row count should equal sum of individual file row counts." The same retrospective also mentions combining the adaptability of agents with deterministic safeguards like retry logic and regular checkpoints5 (note that "checkpoints" there refers to the kind of state-saving recovery checkpoints from Course 9 of this series).
Second, one class of verification can move upstream to the API layer. Add strict: true to your tool definitions to ensure Claude's tool calls always match your schema exactly6. The schema is your declaration of parameter structure.
With that one line, structural issues like "typo in field name" or "count passed as string" go from "something you write code to check" to a platform-level guarantee. Tools are a contract between deterministic systems and non-deterministic agents2, and strict is how you write that contract into the interface.
But it polices structure, not semantics. Whether total actually equals the sum of all count values, the schema has nothing to say—that part you still verify yourself.
The pitfall: over-strict verifiers reject correct outputs
This is the most common way deterministic checks fail, and you often won't notice when it happens.
The official guidance for tool evals is sharp: avoid overly strict verifiers that reject correct responses due to spurious differences like formatting, punctuation, or valid alternative phrasings2.
"Spurious differences" is the key phrase. The same correct answer might carry a trailing space, might write positive as Positive, might have two spaces between words instead of one. These differences mean nothing to the task, but to a byte-by-byte comparison verifier they're catastrophic. The failure direction is also insidious: it doesn't let errors through, it punishes correct outputs—that's a false negative.
Here's a concrete crash and fix. The fixture title is 2026 年 Q1 渠道汇总. The agent's report.json has all data correct, just whitespace before and after the title and two spaces between words. The naive strict verifier looks like this:
Run it, actual output:
A report with entirely correct content, failed by a trailing newline. If this result gets fed back to the agent, it'll go tinker with the title's whitespace—a direction unrelated to the task.
The fix is one function:
replace(/\s+/g, " ") folds consecutive whitespace into one, trim() removes leading and trailing, toLowerCase() unifies case. After the change, the same file passes—the full script and real run results are in Level 2 of the exercises below.
Flip side reminder: normalization isn't a "the more the better" thing. If you also strip punctuation, you might smooth over real errors like total: 48 vs. total: 4.8. The judgment standard is always the same—does this difference carry meaning? If yes, be strict. If no, normalize it.
Where deterministic checks hit their ceiling
Deterministic verifiers have a well-defined boundary of applicability.
First boundary is free text. Research outputs are difficult to evaluate programmatically, since they are free-form text and rarely have a single correct answer5. You can't write output == golden_answer for a summary—given the same material, two well-written summaries can use completely different wording. That kind of judgment goes to Lesson 4's domain.
Second boundary is "fits broader system requirements." Code solutions are verifiable through automated tests, but to ensure solutions align with broader system requirements, human review remains crucial4. A patch can pass all tests while being a bad design that makes the whole module unmaintainable. The multi-agent retrospective echoes this: even in a world of automated evaluations, manual testing remains essential5.
Deterministic checks own the floor of "things that shouldn't be wrong aren't wrong." Above that floor, you need different tools.
Proportion: not everything deserves a verifier
The opposite mistake is also common: building a full verification suite for a one-off script.
The agent writes a data migration script that runs once then gets deleted, and you set up structure validation, fixture comparison, regression samples—the time spent writing the verifier exceeds the time to just eyeball the output.
The judgment yardstick is still that old line: you should consider adding complexity only when it demonstrably improves outcomes4. For verifiers specifically, ask yourself one question:
- How many times will this check run? If it's once and you're standing right there watching, your eyes might be faster.
- Without it, how long until errors get discovered? "Immediately, I'm looking right at it" versus "when downstream breaks, two days later" yield totally different conclusions.
- How many prompt iterations do you plan for this task? As soon as it's more than one, you need a stable yardstick to compare before and after, or "did this get better" is always just a guess.
As for when you must write one, the official guidance is hard—always provide verification (tests, scripts, screenshots); if you can't verify it, don't ship it3.
💻 Exercises
Recap
- The ranking principle for grading methods is "fastest, most reliable, most scalable"; code-based grading ranks first on all three, the tradeoff is it lacks nuance for complex judgments that need less rule-based rigidity1.
- A check can be anything that returns a signal Claude can read in the conversation: a test suite, build exit code, linter, a script that diffs output against a fixture, or a screenshot compared to a design3.
- Verifiers form a spectrum; the left edge is exact string match against a fixture, the right edge is asking Claude to judge2; minimal form is just
output == golden_answer, typically after normalizing whitespace and case, perfect for tasks with clear-cut, categorical answers1.
- With pass/fail, the loop closes on its own: do work, run check, read result, iterate until it passes3; that's because agents need ground truth from the environment at each step to assess progress4, and why they can iterate using test results as feedback4.
- Complex workflows can be broken into discrete verification checkpoints where specific state changes should have occurred, rather than validating every intermediate step5; one class of structure validation can even move upstream to the API layer with
strict: true to make tool calls strictly conform to schema6.
- The biggest pitfall is over-strict verifiers: they reject correct responses due to spurious differences like formatting, punctuation, or valid alternative phrasings2. The fix is to normalize first, then compare, saving strictness for the parts that truly carry meaning.
- Deterministic checks have a ceiling: research outputs are difficult to evaluate programmatically5; automated testing verifies functionality, but to ensure solutions align with broader system requirements, human review remains crucial4, and even with mature automated evals, manual testing remains essential5.
- Don't build a full verifier for a one-off script—you should consider adding complexity only when it demonstrably improves outcomes4; but in the other direction, if you can't verify it, don't ship it3.
>> Lesson 4: LLM as Judge: Rubrics, Formats, and What Not to Let It Judge