Glossary
66 terms from “Verification and Quality Assurance: Don't Let 'Looks Right' Slip Through.” Hover the first occurrence in the lesson for its definition.
| Term | Definition | Source |
|---|---|---|
| looks done | Claude stops when the work looks done; without a check it can run, 'looks done' is the only signal available in the entire system. | Best practices for Claude Code — Claude Code official documentation |
| verification loop | The role that falls to you when there's no runnable check in the pipeline: every mistake waits for you to notice it. | Best practices for Claude Code — Claude Code official documentation |
| trust-then-verify gap | The official name for this phenomenon: Claude produces a plausible-looking implementation that doesn't handle edge cases; you trust it first, and verification either doesn't happen or happens too late. | Best practices for Claude Code — Claude Code official documentation |
| assertion | Statements you can only choose to believe or not: 'logic is correct,' 'should be fine,' 'already optimized,' 'no errors during execution.' | Best practices for Claude Code — Claude Code official documentation |
| evidence | Something a second person can re-run exactly the same way: a command plus its raw output, an exit code, a list of failed test names, a screenshot, a before/after numeric comparison. | Best practices for Claude Code — Claude Code official documentation |
| deterministic system | In computing, systems that produce the same output every time given identical inputs; this course's verifiers are this kind of 'predictably dumb' thing, using a deterministic thing to gate a non-deterministic thing. | Writing effective tools for agents — with agents — Anthropic Engineering |
| non-deterministic | Agent systems can generate varied responses even with identical starting conditions; even if the prompt hasn't changed a single character, decisions across runs aren't guaranteed to match. | Writing effective tools for agents — with agents — Anthropic Engineering |
| compounding errors | In agent systems errors snowball: one step failing causes agents to explore entirely different trajectories, and they make new decisions based on bad results, outcomes become unpredictable. | How we built our multi-agent research system — Anthropic Engineering |
| sandbox environment | Where Anthropic recommends agents be thoroughly tested: autonomy means higher costs and potential for compounding errors, so test in sandboxes with appropriate guardrails. | Building Effective AI Agents — Anthropic Engineering |
| ground truth | Real feedback agents get from the environment during execution, like tool call results or code execution results, used to assess their own progress. | Building Effective AI Agents — Anthropic Engineering |
| demonstrably improves outcomes | The admission condition for adding complexity: you should consider adding complexity only when it demonstrably improves outcomes. | Building Effective AI Agents — Anthropic Engineering |
| stop_reason | The field in model responses that drives the harness loop: when the value is tool_use, continue executing tools and feeding back; when it becomes end_turn, the loop exits. | Tool use with Claude — Claude API documentation |
| end_turn | One value of stop_reason, meaning the model doesn't plan to call more tools this turn, nothing more. | Tool use with Claude — Claude API documentation |
| tool_result | The message block that feeds tool execution results back to the model, paired with the corresponding tool_use via tool_use_id; verifier output flows back into the conversation through this. | Tool use with Claude — Claude API documentation |
| run_check | Wrapping a verification script as a tool the model can call, letting it run the check itself within the loop and read results, with exit code and stdout passed through as-is. | Best practices for Claude Code — Claude Code official documentation |
| exit code | How verification scripts express conclusions: 0 means pass, non-zero means fail; CI and shell && can use it directly. | Best practices for Claude Code — Claude Code official documentation |
| pass/fail | The objective binary result a check can produce; with it the loop closes on its own—Claude does work, runs check, reads result, iterates until check passes. | Best practices for Claude Code — Claude Code official documentation |
| fixture | A golden-answer file saved beforehand; after running, compare output to it; the script in the official check menu that 'diffs output against it' uses this. | Best practices for Claude Code — Claude Code official documentation |
| end state | The default judgment method: don't judge whether the agent followed a specific process, judge whether it achieved the correct final state. | How we built our multi-agent research system — Anthropic Engineering |
| turn-by-turn analysis | The approach end-state evaluation replaces: checking agent actions turn by turn, attempting to validate every intermediate step. | How we built our multi-agent research system — Anthropic Engineering |
| verification checkpoint | A few discrete observation positions in complex workflows, where you verify 'specific state changes should have occurred' rather than validating every intermediate step. | How we built our multi-agent research system — Anthropic Engineering |
| recovery checkpoint | The Course 9 meaning: writing the agent's running state to disk so it can resume from there after a crash; purpose is recovery, not judgment. | How we built our multi-agent research system — Anthropic Engineering |
| success criteria | The set of requirements that turn 'correct end state' into concrete numbers or clear judgments; if you can't write it no matter what, this task shouldn't have been handed entirely to the agent to run alone. | Define success criteria and build evaluations — Claude API documentation |
| measurable | The first hard requirement for success criteria: use quantitative metrics or well-defined qualitative scales; numbers provide clarity and scalability. | Define success criteria and build evaluations — Claude API documentation |
| achievable | The second hard requirement for success criteria: base targets on industry benchmarks, prior experiments, AI research, or expert knowledge; shouldn't be unrealistic to current frontier model capabilities. | Define success criteria and build evaluations — Claude API documentation |
| multidimensional evaluation | Most use cases need evaluation along several success criteria—dimensions undermine each other, leaving less room for shortcuts. | Define success criteria and build evaluations — Claude API documentation |
| trajectory assertion | An optional addition: for a set of prompts and responses, additionally specify which tools you expect the agent to call, to measure whether it grasps each tool's purpose. | Writing effective tools for agents — with agents — Anthropic Engineering |
| expectedTools | The optional field in eval cases that carries trajectory assertion: expects it touched at least these tools, regardless of order or call count. | Writing effective tools for agents — with agents — Anthropic Engineering |
| redundant tool calls | One diagnostic metric: call count is noticeably high, usually suggests pagination or token limit parameters need rightsizing. | Writing effective tools for agents — with agents — Anthropic Engineering |
| tool errors | One diagnostic metric: lots of errors due to invalid parameters usually suggests these tools need clearer descriptions or better examples. | Writing effective tools for agents — with agents — Anthropic Engineering |
| code-based grading | First place in the grading method ranking: fastest, most reliable, extremely scalable; the weakness is it lacks nuance for complex judgments requiring less rule-based rigidity. | Define success criteria and build evaluations — Claude API documentation |
| LLM-based grading | Second place in the ranking: fast, flexible, scalable, suitable for complex judgment; prerequisite is test to ensure reliability first, then scale. | Define success criteria and build evaluations — Claude API documentation |
| human grading | Third place in the ranking: most flexible, highest quality, but slow and expensive; avoid if possible. | Define success criteria and build evaluations — Claude API documentation |
| exact match | The leftmost form on the verifier spectrum: measures whether model output matches a predefined correct answer, typically after normalizing whitespace and case. | Define success criteria and build evaluations — Claude API documentation |
| output == golden_answer | The minimal form of exact match, just an equality check; simple, unambiguous, suitable for tasks with clear-cut categorical answers. | Define success criteria and build evaluations — Claude API documentation |
| normalization | Before comparing, erase differences that don't carry meaning: fold consecutive whitespace, trim leading/trailing, unify case; for amounts, also wash away currency symbols, thousand separators, units. | Define success criteria and build evaluations — Claude API documentation |
| deterministic verifier | This course's core technique: using code that gives the same conclusion every time to gate a non-deterministic system's output, returning pass/fail and readable failure reasons. | Writing effective tools for agents — with agents — Anthropic Engineering |
| spectrum | Verifiers aren't a few mutually exclusive options but a continuous band: one end is exact string comparison against baseline, the other end is asking Claude to judge. | Writing effective tools for agents — with agents — Anthropic Engineering |
| strict: true | A line added to tool definitions, ensuring Claude's tool calls strictly conform to your declared schema, moving one class of structure validation forward as a platform-level guarantee. | Tool use with Claude — Claude API documentation |
| false negative | The verifier judges correct output as failure: it doesn't let errors through, it wrongly accuses correct ones. | Writing effective tools for agents — with agents — Anthropic Engineering |
| overly strict verifier | The most common way deterministic checks fail: due to spurious differences like formatting, punctuation, or valid alternative phrasings, they judge correct responses as failures. | Writing effective tools for agents — with agents — Anthropic Engineering |
| free text | Outputs like research and summaries: free-form, rarely have a single correct answer, difficult to evaluate programmatically; LLMs are a natural fit for scoring this output. | How we built our multi-agent research system — Anthropic Engineering |
| human review | A step still indispensable beyond automated testing: automated tests verify functionality is correct, but ensuring solutions align with broader system requirements still needs human eyes. | Building Effective AI Agents — Anthropic Engineering |
| LLM judge | Using a model call to score free-form text output, positioned where deterministic checks can't reach, not as their replacement. | How we built our multi-agent research system — Anthropic Engineering |
| rubric | Breaking down vague 'is it good' into several concrete questions, answering and scoring each separately; for research tasks the ready-made breakdown is five dimensions. | How we built our multi-agent research system — Anthropic Engineering |
| reason first then score | A key technique for judge prompts: have it write reasoning first then produce a score, discard reasoning afterward; improves grading performance, especially noticeable for complex judgments. | Define success criteria and build evaluations — Claude API documentation |
| fresh context | Where the reviewer should sit: only sees output and the criteria you gave, can't see the reasoning process that produced this change, so judges the result on its own terms. | Best practices for Claude Code — Claude Code official documentation |
| self-report | The agent's description of its own process ('I searched three authoritative sources, cross-checked and confirmed the percentage'); doesn't count as evidence by itself. | Writing effective tools for agents — with agents — Anthropic Engineering |
| raw transcript | The execution log including tool calls and tool responses, used to catch behaviors not explicitly stated in the agent's chain-of-thought. | Writing effective tools for agents — with agents — Anthropic Engineering |
| find problems | The judge's first failure mode: a reviewer prompted to find gaps will usually report some, even when the work is sound—because that's what you asked it to do. | Best practices for Claude Code — Claude Code official documentation |
| over-engineering | The consequence of chasing every review finding: extra abstraction layers, defensive code, and tests for cases that can't possibly happen. | Best practices for Claude Code — Claude Code official documentation |
| effect size | How big the gap is from one change—is it from 71.2% to 72.4%, or from 30% to 80%; larger gaps need fewer samples. | How we built our multi-agent research system — Anthropic Engineering |
| eval set | A batch of tasks paired with verifiable results; official starting scale is about 20 queries representing real usage; don't wait to hoard hundreds. | How we built our multi-agent research system — Anthropic Engineering |
| verifiable result | What every eval prompt should be paired with: a judgeable end state, string, state change, or rubric; without it the prompt isn't an eval task, it's a demo. | Writing effective tools for agents — with agents — Anthropic Engineering |
| edge case | Low-frequency but actually-occurring situations (missing info, mixed demands, tool errors); the insurance in eval sets, not the main body. | Define success criteria and build evaluations — Claude API documentation |
| volume over quality | An eval set design principle: more questions with slightly lower signal automated grading beats fewer questions with high-quality human hand-grading. | Define success criteria and build evaluations — Claude API documentation |
| real distribution | What evals should stick to: your eval tasks should be grounded in real-world usage, able to point to any prompt and say 'three users asked this way last week.' | Define success criteria and build evaluations — Claude API documentation |
| held-out set | A batch of cases separated from the start, not looked at or run or touched during daily tuning, relies on it to ensure no overfitting to the 'training' evals. | Writing effective tools for agents — with agents — Anthropic Engineering |
| dev set | The batch of cases run repeatedly when changing prompts daily, the more frequently the better; its scores are navigation, not conclusion. | Writing effective tools for agents — with agents — Anthropic Engineering |
| overfitting | The entire set of prompt plus tool configuration learned features of the eval materials rather than patterns of the task itself—like hardcoding a sentence in system prompt for one persistently failing case. | Writing effective tools for agents — with agents — Anthropic Engineering |
| not guaranteed | Official docs' wording for certain ideal behaviors (like the model will proactively ask for missing required parameters); especially so for more ambiguous prompts and less capable models. | Tool use with Claude — Claude API documentation |
| eval track | The runnable program that welds eval sets, layered grading, and harness loops together: type one command, scores tell you which got better, which didn't move, whether anything that was right got broken. | Writing effective tools for agents — with agents — Anthropic Engineering |
| stub client | A fake messages.create that spits responses in a fixed queue order, turning model behavior into a controlled variable, making the whole track reproducible. | Writing effective tools for agents — with agents — Anthropic Engineering |
| response queue | The pre-written response sequence behind the stub client; the difference between two prompt versions is pinned by two sets of queues, 'assuming new prompt takes effect and model answers this way' is encoded as data. | Writing effective tools for agents — with agents — Anthropic Engineering |
| task isolation | One eval task one independent loop, one independent messages; finished then discarded; implementation is just don't hoist messages outside the function. | Writing effective tools for agents — with agents — Anthropic Engineering |
| is_error | The marker attached to tool_result when feeding back tool exceptions: catch the exception, wrap it into a result with this marker to return to the model, simultaneously incrementing the error counter. | Tool use with Claude — Claude API documentation |