Lesson 2: What to Verify: End State First, Process as Backstop

Learning goals:

• Understand why "record the correct steps then check them one by one" is guaranteed to misjudge agents, and switch to end-state evaluation instead
• For complex workflows, identify a few discrete verification checkpoints that confirm "the expected state changes happened" rather than validating every step
• Turn a vague requirement into measurable, achievable, multi-dimensional success criteria, and know how far to take trajectory assertions

Prerequisites: Finished Lesson 1, know that when there's no runnable check you become the verification loop | Previous: Lesson 1 << | Next: Lesson 3 >>

Three Runs, Three "Failed" Verdicts

You decide to add verification for your agent. The first impulse is nearly universal: record the "standard approach." Run through the task manually once, write down every step—step 1 should call search, step 2 should call fetch_page, step 3 should call write_note—and save that as the answer key. From then on, every time the agent runs, compare its call sequence against this answer step by step. One mismatch and it fails.

You run it three times. Three failures.

You check the outputs: three summaries, facts correct, sources reliable, all requested angles covered. The only difference was the path—first run searched three sources and that was enough, second run searched ten, third run looked up terminology definitions first before searching. This is exactly what Anthropic observed in their own multi-agent research system: even with identical starting points, agents might take completely different but valid paths to reach their goal, one searching three sources while another searches ten, or using different tools to find the same answer¹.

What failed wasn't the agent. It was your verification method.

You Don't Actually Know What the "Correct Steps" Are

Traditional evaluation carries a deeply buried default assumption: given input X, the system should follow path Y and produce output Z—the same steps every time¹. This assumption holds so naturally for deterministic systems that most people never realize it's an assumption at all. Agents upend it immediately.

What's truly uncomfortable isn't just "the path will vary"—it's this sentence:

Because we don't always know what the right steps are, we usually can't just check if agents followed the "correct" steps we prescribed in advance. Instead, we need flexible evaluation methods that judge whether agents achieved the right outcomes while also following a reasonable process¹.

"We don't always know what the right steps are"—that's the key. The path you recorded isn't the only correct path. It's just the path you happened to take that time. You elevated it to the answer key, so every other approach became an error.

Note the second half: "while also following a reasonable process." This doesn't mean ignore process entirely. It means don't use one fixed path as the yardstick.

End-State Evaluation: Judge Results, Not Flow

Anthropic's approach is direct: focus on end-state evaluation rather than turn-by-turn analysis—don't judge whether the agent followed a specific process, judge whether it achieved the correct final state¹. This approach acknowledges that agents may find alternative paths to the same goal while still ensuring they deliver the intended outcome¹.

First, a plain-language definition of "end state": after the task completes, the state you can observe in the environment and verify after the fact. Which files appeared in the filesystem, what values that database record's fields have now, which label the ticket carries, whether the returned JSON has status: "resolved".

A useful litmus test: end state is a noun, not a verb. "Called rename_file" is a verb; "all filenames match a certain format" is a noun. Verification accepts only nouns.

Task: Rename invoice PDFs in downloads/ to "date_vendor.pdf"
Step-based verification (brittle)     End-state verification (stable)1. Call list_files                    downloads/ has no original filenames2. Call read_pdf for each             Each filename matches ^\d{8}_[a-z0-9-]+\.pdf$3. Call extract_date                  File count matches pre-run count, nothing lost or added4. Call rename_file for each          Filename dates match invoice dates in PDF content

The left column fails the moment the agent uses one batch read instead of four individual reads, even if results are identical. The right column doesn't care how it reads—because it describes "what downloads/ looks like now," unrelated to how it got that way.

There's an easy-to-miss spot when writing end-state assertions: write down what shouldn't change too. That "file count matches pre-run" line above is one. If the agent renames two files to the same name and the second overwrites the first, the "all filenames are valid" check passes perfectly. End-state assertions must guard both "expected changes happened" and "unexpected changes didn't."

Complex Workflows: Split Evaluation Across Checkpoints

End-state evaluation isn't "ignore process entirely." The very next sentence in the source gives an out: for complex workflows, break evaluation into discrete checkpoints where specific state changes should have occurred, rather than attempting to validate every intermediate step¹.

Notice the phrasing—"specific state changes should have occurred," still state, still nouns. Just moving the observation point from "the finish line" to "a few spots along the way."

Term collision warning: The "checkpoint" in Course 9 of this series means saving context—writing the agent's running state to disk so it can resume from there after a crash, for recovery. The "checkpoint" in this lesson means verifying state—confirming expected state changes happened at a point in the flow, for validation. The positions often overlap (checkpointing where you also verify is natural), but they solve different problems. Mixing them in discussion leads to confusion, so below we'll call these "verification checkpoints" and those in Course 9 "recovery checkpoints."

When a verification checkpoint is worth adding, look at three things:

• Long workflow, end state too far from start. When it fails you only know "didn't reach the end," not where the deviation began.
• Irreversible operations. Emails sent, inventory deducted, files overwritten—by the time end state reveals the error it's too late.
• Intermediate output is the foundation for subsequent steps. Migration script creates table then loads data; wrong table structure means all loaded data is garbage, rework cost multiplies.

If none of these apply, don't add one.

Where to add them: at positions where state undergoes substantive change, not after every tool call.

Task: Migrate user table from old schema to new schema
Verification checkpoint 1 (after table creation): new_users table exists, columns exactly match target schemaVerification checkpoint 2 (after data load): new_users row count == old_users row count, no duplicate primary keysEnd state: App reads new table and passes smoke test (most basic "does it run" check), old_users renamed to old_users_backup
What we don't check: whether it wrote CREATE TABLE or copied from a template,whether it loaded all at once or in batches, how many batches, how many rows per batch.

Two checkpoints, one end state. Three assertions govern the entire migration. If you went with "validate every intermediate step," this workflow could generate dozens of assertions, most of them penalizing legitimate implementation differences.

Pause Here: What's Wrong with This Proposal

How to Define Success Criteria: Measurable, Achievable, Multi-Dimensional

Those four words "correct end state" need to land on concrete numbers or clear judgments, otherwise you've just circled back to "looks right." The official docs give two hard requirements for success criteria:

• Measurable: Use quantitative metrics or well-defined qualitative scales (a scale is a scoring rubric checklist; specifics of how to write one are Lesson 4 content). Numbers provide clarity and scalability, but qualitative measures can be valuable if consistently applied along with quantitative measures².
• Achievable: Base your targets on industry benchmarks, prior experiments, AI research, or expert knowledge. Your success metrics should not be unrealistic to current frontier model capabilities².

And one more: most use cases need multidimensional evaluation along several success criteria².

The official docs' complete example is one sentence (annotations in parentheses are from the original):

The sentiment analysis model should achieve an F1 score of at least 0.85 (Measurable, Specific) on a held-out test set* of 10,000 diverse Twitter posts (Relevant), which is a 5% improvement over the current baseline (Achievable)².

This sentence is worth unpacking because each component blocks a specific failure mode:

That last row is the concrete method for "achievable": the threshold isn't reverse-engineered from wishes; it's a small step forward from current state. If you have no baseline, run one version of the most naive implementation and use its score as baseline. When you can't even run a baseline, don't rush to set numbers.

There's an even earlier question. When Anthropic describes agent-appropriate scenarios, they say: agents add the most value for tasks that require both conversation and action, have clear success criteria, enable feedback loops, and integrate meaningful human oversight³. Read it backwards—if you can't write success criteria for this task no matter what, the problem isn't the verification step; this task shouldn't have been handed entirely to the agent to run alone. Can't write criteria, you have to watch the whole thing—as Lesson 1 said, you become the verification loop at that point.

Trajectory Assertions: You Can Add Them, But Don't Hard-Code

End-state evaluation catches "is the result correct," but it misses one class of problem: does the agent actually recognize that new tool you gave it.

Official docs give an optional addition: for each prompt-response pair, you can optionally also specify the tools you expect an agent to call in solving the task, to measure whether or not agents are successful in grasping each tool's purpose during evaluation⁴. This is trajectory assertion—not judging order, not judging count, only judging whether certain tools appeared in the trajectory.

When it's useful: you just added a search_internal_docs tool and want the agent to use it for internal-process questions. But it goes searching the public web, finds a similar enough answer, and end-state validation still passes. Only trajectory can see that difference.

The limit is written in the immediately following sentence: because there might be multiple valid paths to solving tasks correctly, try to avoid overspecifying or overfitting to strategies⁴.

Concrete boundaries:

• Assert only "set contains," not order, not count
• List only the one or two tools you truly care about; don't copy the whole sequence in—that's record-replay again
• Trajectory assertion fails but end state passes: log an observation, don't fail the whole evaluation
• It's an optional addition, not the default. Default is still end state¹

Beyond Pass Rate: What Else to Record

After an eval run, if you only get a pass rate, you'll find yourself with nothing to say—what does 78% mean? Should you adjust the prompt next or the tools?

Official docs recommend collecting these metrics beyond top-level accuracy: the total runtime of individual tool calls and tasks, the total number of tool calls, the total token consumption, and tool errors⁴. These metrics don't participate in judgment; they participate in diagnosis.

Official docs give two readings:

• Lots of redundant tool calls might suggest some rightsizing of pagination or token limit parameters is warranted⁴
• Lots of tool errors for invalid parameters might suggest tools could use clearer descriptions or better examples⁴

What these share: they point the finger at tool design, not the model. Lots of redundant calls usually means it can only fetch 20 items at a time so it has to paginate through ten pages; lots of parameter errors usually means the tool description didn't explain what format that field needs. These problems root on the tool side—just adding "call less often" or "write parameters carefully" to the system prompt usually doesn't work; you have to adjust the tool's parameter design and description.

Following this thread yields a few more (below aren't officially endorsed, they're engineering judgments extrapolated from the above two—verify on your own data): pass rate unchanged but token consumption doubled means this change isn't free; one category of tasks has especially large duration variance, likely hiding retries or circling; errors concentrated on one tool, look at that tool first, don't suspect the prompt.

An eval run should drop at least these columns; Lesson 6 when building the eval harness will use them directly (to save column width, tokens in/out will be merged into one):

case_id | passed | duration_ms | tool_calls | tokens_in | tokens_out | tool_errors

Boundaries: Three Things Not to Do

One: Don't try to validate every intermediate step¹. This is the easiest boundary in this lesson to break because the "verify more is safer" intuition is so strong. The actual result is opposite: the finer the assertions, the more legitimate differences get penalized, the noisier the eval, until you start ignoring red—at which point it's completely useless.

Two: Don't make trajectory assertions the default. Adding a trajectory assertion is so cheap you can write another expectedTools entry with no effort. By the tenth entry you're already substantively prescribing strategy, only formally still calling it "assertion." Each time you add one, ask yourself: will output truly break if this tool isn't called? If the answer is "not necessarily," don't add it.

Three: Don't set thresholds after the run. Looking at a score of 0.82 and saying "0.8 should be enough," and looking at 0.86 and saying "needs to be 0.85," are the same self-deception. Set thresholds before the run, and write down the justification.

💻 Exercises

Recap

• Traditional evaluation assumes "given input X follow path Y get output Z"; agents don't satisfy this assumption: identical starting points can still take completely different but valid paths, one searching three sources another searching ten¹
• You don't always know what the right steps are, so you usually can't check if agents followed your prescribed steps; use flexible evaluation methods that judge whether they achieved the right outcomes while following a reasonable process¹
• Default approach is end-state evaluation not turn-by-turn analysis: don't judge whether it followed a specific process, judge whether it achieved the correct final state¹. End state is nouns not verbs, and must guard both "expected changes happened" and "unexpected changes didn't"
• For complex workflows break evaluation into discrete checkpoints confirming "specific state changes should have occurred," don't try to validate every intermediate step¹. The "checkpoint" here means verifying state, different from the context-saving checkpoint in Course 9
• Success criteria must be measurable (quantitative metrics or well-defined qualitative scales), achievable (base targets on industry benchmarks, prior experiments, or expert knowledge), and most use cases need multidimensional evaluation²
• Official docs list "have clear success criteria, enable feedback loops" as conditions where agents add the most value³; read backwards—tasks for which you can't write success criteria shouldn't be handed entirely to agents to run alone; you'll have to watch the whole time
• Trajectory assertions are an optional addition: you can specify which tools you expect it to call, to measure whether it grasps tool purpose, but because valid paths aren't singular, avoid overspecifying or overfitting to strategies⁴
• Beyond pass rate also record runtime, call count, token consumption, tool errors; lots of redundant calls consider adjusting pagination and token limit parameters, lots of invalid parameter errors consider making tool descriptions and examples clearer⁴

>> Lesson 3: Deterministic Verifiers: Only Checks That Output Pass/Fail Count