Lesson 5: Eval Sets: Start with 20 Real Tasks

Learning goals:

• Explain why "wait until we have hundreds of cases" is wrong, and why early-stage changes have effect sizes large enough that a few cases can distinguish between versions
• Design a batch of eval cases following these principles: grounded in real usage, supplemented with edge cases, automated grading when possible, volume over per-case quality, and deliberate inclusion of ambiguous cases — each paired with a verifiable outcome
• Use held-out sets to prevent overfitting prompts into "models that only pass these specific questions," and identify which issues automated evals can't see and must be caught by manual testing

Prerequisites: Lessons 1–4 ("looks done" ≠ done, end-state-first verification targets, deterministic verifiers, LLM judges) | Previous: << Lesson 4 | Next: Lesson 6 >>

A particularly common reason for delay

You've probably seen this scenario. Someone raises the idea: "We should build an eval set for our agent." Another person responds: "An eval set needs hundreds of cases to have statistical significance, right? Running ten or twenty cases produces meaningless scores that mislead more than inform. Let's collect real user cases first and build the set once we have enough."

Sounds professional. Disciplined. Then six months pass, cases sit in a shared doc, the prompt has been revised thirty times, and no one can say whether any revision was an improvement or a regression.

Anthropic's engineering postmortem on their multi-agent research system directly calls out this excuse: they often hear that AI developer teams delay creating evals because they believe only large evals with hundreds of test cases are useful, when it's actually best to start with small-scale testing right away with a few examples rather than delaying¹.

This lesson clarifies exactly that: why small sets genuinely work early on, what one case should look like, how to select these cases, and how to prevent "tuning the agent into an expert on these specific questions and nothing else."

Effect size: why a few cases suffice to spot differences early

Start with a term. Effect size refers to how large the gap from a change is — is it a shift from 71.2% to 72.4%, or a jump from 30% to 80%? Larger gaps require fewer samples; smaller gaps require more. This isn't mystical, it's the same common sense you use judging which of two cups of water is hotter: a 30-degree difference is obvious at a touch, a half-degree difference demands a thermometer.

Early agent development belongs to the former category. The official postmortem states it plainly: in early agent development, changes tend to have dramatic impacts because there is abundant low-hanging fruit; a prompt tweak might boost success rates from 30% to 80%, and with effect sizes this large, you can spot changes with just a few test cases¹.

Imagine this scenario: you have 6 cases, 2 pass before the change, 5 pass after. Do you need a p-value? No. What you need is to lock in this version of the prompt and go find the next 30%-to-80% improvement.

The scale they started with isn't mysterious either — a set of about 20 queries representing real usage patterns¹. Twenty isn't a magic threshold, it's just a quantity you can finish in an afternoon and start using that same day.

The reverse also holds: when your agent is already at 80%+ and remaining changes move the needle by only a point or two, a few cases genuinely can't distinguish them. At that point you need more cases — but by then you already have a functioning eval set, and expanding it is far easier than building it from scratch. Get the ruler first, then talk precision; don't wait for the ruler to be precise before you start measuring.

Minimum composition of one eval case

An eval set isn't "a pile of prompts." A pile of prompts only lets you eyeball outputs, which gets tedious after two reads and self-deceptive after three.

The tool engineering article is direct: each evaluation prompt should be paired with a verifiable response or outcome; your verifier can be as simple as an exact string comparison between ground truth and sampled responses, or as advanced as enlisting Claude to judge the response². This connects directly to the success criteria from Lesson 2, the deterministic verifiers from Lesson 3, and the LLM judges from Lesson 4 — those lessons taught you how to verify, this lesson teaches you what to verify.

So one usable case needs at least three components written clearly:

prompt   : Input to the agent (write it the way a user would say it)expected : A verifiable outcome (end state, string, state change, or rubric)verifier : Who judges — deterministic verifier or LLM judge

Structured as data, it looks like this:

The expected field here describes end states and observable evidence, not "the sequence the agent should think through." Lesson 2's conclusion continues to apply: for the same goal, the agent might take several valid paths, so don't hard-code the path. The tool engineering article also reminds you that you can optionally specify the tools you expect an agent to call to measure whether agents grasp each tool's purpose, but because there might be multiple valid paths, avoid overspecifying or overfitting to strategies².

The mustCallTools field might remind you of expectedTools from Lesson 2. The relationship between them is worth nailing down. Negative assertions (mustNotContain, mustNotCallTools — don't say "changed for you," don't guess an order number and query it) are essentially end-state guardrails describing "things that shouldn't happen didn't happen," and can be enforced strictly. Positive tool assertions (must have called a certain tool) are the trajectory assertions from Lesson 2, and those three disciplines apply unchanged: only assert set membership, only list the one or two tools you genuinely care about, and if the assertion fails but the end state passes, log an observation rather than immediately failing the case. It only hardens in one scenario: the key information in the response can only come from that tool's return value. Case cs-003 is exactly this type — the judgment "already shipped" can only come from getOrder, so if the tool wasn't called, that statement is fabricated, making this positive assertion enforceable. When in doubt, treat it as soft.

One more note: this case has mustNotContain listing "changed for you" — it guards against the agent verbally agreeing to change the address while actually doing nothing. This "verbal completion" is precisely Lesson 1's theme.

Five rules for designing eval sets

The five rules below combine the design principles from the Claude platform's "Test and Evaluate" docs and the practices from the tool engineering article into a single checklist — citations after each rule indicate its source.

One: Ground in real usage. Generate lots of evaluation tasks, grounded in real-world uses²; design evals that mirror your real-world task distribution³. The criterion is straightforward: if this prompt wasn't copied from real logs, can you point at it and say "three users asked exactly this last week"? If not, it's probably something you imagined while sitting at your desk.

Two: Don't miss edge cases. The official docs follow "mirror your real-world task distribution" immediately with a reminder: don't forget to factor in edge cases³. The real distribution is the body, edge cases are insurance. An eval set composed entirely of edge cases will mislead you — you'll spend a month fixing an issue that appears twice a month.

Three: Automate grading when possible. Structure questions to allow for automated grading, for example multiple-choice, string match, code-graded, LLM-graded³. This determines whether your eval set can be run repeatedly. Cases requiring you to read for five minutes to judge pass/fail — after writing ten of those, you'll never want to run them a second time.

Four: Prioritize volume over quality. The official wording: more questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals³. This is the most counterintuitive and also the biggest time-saver — spend your time writing ten more cases, not polishing one case's grading criteria to perfection.

Five: Deliberately collect ambiguous cases. The docs explicitly list a type of test case worth including: ambiguous test cases where even humans would find it hard to reach an assessment consensus³. These aren't for boosting scores, they're for surfacing disagreements. When your agent waffles on these cases, it signals that the product-level rules themselves aren't settled, which is something product needs to resolve, not something prompt engineering can fix.

One more point that's not a "design principle" but equally critical: don't make the eval environment too simple. The tool engineering article recommends avoiding overly simplistic or superficial "sandbox" environments that don't stress-test your tools with sufficient complexity; strong evaluation tasks might require multiple tool calls, potentially dozens². A question answerable with a single database query won't reveal whether your agent loses context by the seventh tool call.

Held-out sets: don't tune your prompt into "only good at these specific questions"

Suppose you dutifully built 20 cases and started tuning prompts. First version passes 8 cases, one revision passes 12, another passes 16, another passes 19. Great.

The problem: how much of that 19-case success comes from the agent genuinely getting stronger, and how much from you quietly embedding the characteristics of these 20 questions into the prompt? For instance, you notice case 7 keeps failing, so you add a line to the system prompt: "for return issues prioritize citing the 7-day no-questions-asked policy" — case 7 passes, but what you actually did was write the answer to that question.

This phenomenon is called overfitting: the model (or in this context, the entire prompt-plus-tools configuration) learns the features of the training material rather than the regularities of the task itself.

The tool engineering article's remedy is one sentence but critical: they relied on held-out test sets to ensure they didn't overfit to their "training" evaluations².

A held-out set is a batch of cases set aside from the start, that you don't look at, don't run, don't touch during day-to-day tuning. Its entire value comes from "not being contaminated." A few disciplines worth codifying in team agreements:

• Run only at milestones. Daily prompt tuning runs only the dev set (the "training" evaluations from the earlier quote — this lesson calls it the dev set, recorded as dev in the JSON field); run the held-out set only before a release or after structural changes (switching models, rewriting tool descriptions, changing the harness loop).
• Look only at aggregates. Check overall pass rate and the IDs of failed cases, don't open the full transcript of each failed held-out case to debug them one by one. Once you modify a prompt to fix a specific held-out case, that case has already become part of the dev set.
• Retire if contaminated. If you genuinely did tear open a few held-out cases while debugging an issue, merge them into the dev set and replenish with a fresh batch of held-out cases. The held-out set is consumable, not an heirloom.
• Document who can access it. Small teams often skip this. Agreeing "held-out set runs are executed by one person before releases, results are posted in the channel as a single score line" is far more effective than verbal promises of "everyone just be disciplined."

One common engineering practice worth mentioning: hooking evals into CI so every commit auto-runs the dev set, compares scores to the previous version, and blocks if the score drops. This is standard engineering practice, the orchestration depends on your pipeline, this course won't expand on it — Lesson 6 will build "the eval track that actually runs," and whether you connect it to CI is your choice.

What automated evals miss, humans catch

Once the eval set is built, running, and scoring well, can you remove human testing?

No. The official postmortem is explicit: even in a world of automated evaluations, manual testing remains essential¹. The reason is that people testing agents find edge cases that evals miss — including hallucinated answers on unusual queries, system failures, or subtle source selection biases¹.

The third category deserves special mention because it's so typical. Their human testers noticed: early agents consistently chose SEO-optimized content farms over authoritative but less highly-ranked sources like academic PDFs or personal blogs¹.

Pause and consider what this bias looks like. Each individual instance appears fine — the agent gives a cited, sourced, plausible-sounding answer. Fact-checking passes, citation format passes, completeness passes, and none of your scoring dimensions catch any issue (unless your rubric happens to include Lesson 4's "source quality" dimension and its criteria are sharp enough). But looking at a hundred outputs together reveals the pattern: it's systematically picking the easiest-to-find type of content.

Your eval set can't see this class of pattern in advance, because the eval set is written based on failure modes you already know — modes you haven't thought of naturally have no cases guarding against them. Manual testing doesn't replace evals, it supplies new entries for the eval set: every time you discover such a pattern, codify it into a case so it gets checked automatically next time.

Another class naturally belongs in the eval set: behaviors the docs explicitly say are "not guaranteed." The tool use documentation offers a good example — if the user's prompt doesn't include enough information to fill all required parameters for a tool, Claude Opus is much more likely to recognize that a parameter is missing and ask for it; but the docs immediately clarify that this behavior is not guaranteed, especially for more ambiguous prompts and for less capable models⁴.

"Much more likely" and "not guaranteed" are signals: whether this holds in your scenario, you have to test yourself. Any behavior your product logic depends on but the docs only describe probabilistically is worth monitoring with a few cases on an ongoing basis.

The eval set is a ruler: get the ruler first, then changes matter

Bringing this lesson back together.

The tool engineering article's sequence is: start by standing up a quick prototype of your tools and testing them locally, then run a comprehensive evaluation to measure subsequent changes². Note "subsequent changes" — the eval isn't for giving the current state a score and calling it done, its value is making all subsequent changes measurable. With it, "this version of the prompt is better" shifts from a feeling to a conclusion.

The agent-building article goes further: as with any LLM features, the key to success is measuring performance and iterating on implementations; to repeat — you should consider adding complexity only when it demonstrably improves outcomes⁵.

This statement carries weight when read with context. The techniques you learned in the earlier nine courses — multi-agent division of labor, memory systems, context compaction, checkpoint recovery — every single one adds complexity. Without an eval set, you can't answer "did adding this actually make things better," so you can only add by instinct, and adding by instinct means you keep adding. The eval set is the evidence that lets you remove a fancy design.

How large is large enough: there's no numeric answer

Finally, a word on proportions.

I searched every primary source this course cites, and not one gives a threshold for "how many cases make an eval set sufficient." What exists is a starting scale (about 20 real queries) and a directional guideline (prioritize volume over per-case quality). So don't go looking for that number, and don't trust anyone who casually quotes "at least 50."

The real criterion is what this lesson opened with: given the effect size of your current changes, can your existing cases still distinguish between them?

• Change a version, passing count jumps from 6 to 15 — sufficient, keep going.
• Change a version, passing count wobbles between 17 and 18, and running twice gives different results — insufficient now, time to add cases or reduce noise in the grading method (revisit Lesson 4's judge consistency).
• The two approaches you want to compare differ by only one case — that's not a "which is better" question, it's a "your ruler can't resolve this difference" question.

Case count follows the need for resolution, not some psychological number.

As for how to actually run this batch of cases — one loop per task, how to layer grading, what to track besides pass rate — that's Lesson 6. What you need to take away from this lesson is the batch of cases itself.

💻 Exercises

Recap

• "Wait until we have hundreds of cases before building an eval set" is the most common delay excuse, which the official postmortem directly refutes: teams often delay creating evals because they believe only large evals with hundreds of cases are useful, when the right approach is to start small-scale testing right away with a few examples¹.
• Small sets genuinely work early on, the basis is effect size — a single prompt tweak might lift success rates from 30% to 80%, and a gap that large is distinguishable with just a few cases; they themselves started with a set of about 20 queries representing real usage patterns¹.
• Evaluation tasks should be grounded in real-world uses², mirror your real-world task distribution and factor in edge cases³; simultaneously avoid overly simplistic sandbox environments, as strong evaluation tasks might require multiple tool calls, potentially dozens².
• Structure questions to allow for automated grading (multiple-choice, string match, code-graded, LLM-graded)³, and prioritize volume over quality — more questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals³.
• Deliberately collect a category of ambiguous test cases where even humans would find it hard to reach an assessment consensus³: they aren't for boosting scores, they're for surfacing disagreements in the product rules themselves.
• Each evaluation prompt should be paired with a verifiable outcome, with verifiers ranging from exact string comparison to enlisting Claude as judge² — this connects directly to Lessons 2, 3, and 4.
• Held-out test sets prevent overfitting, relying on them to ensure you didn't overfit to your "training" evaluations²; the core disciplines are don't look at them daily, only look at aggregates, and retire them if contaminated.
• Automated evals have inherent blind spots: people testing agents find edge cases that evals miss, including hallucinated answers on unusual queries, system failures, and subtle source selection biases¹; the real example is early agents consistently chose SEO-optimized content farms over authoritative but less highly-ranked sources like academic PDFs or personal blogs¹. Even with automated evals, manual testing remains essential¹.
• Behaviors the docs explicitly say are "not guaranteed" naturally belong in the eval set, like whether the model will proactively ask for missing required parameters — the docs state this behavior is not guaranteed, especially for more ambiguous prompts and for less capable models⁴.
• The eval set's value is making all subsequent changes measurable: run a comprehensive evaluation to measure subsequent changes²; success depends on measuring performance and iterating, complexity is only worth adding when it demonstrably improves outcomes⁵.
• No primary source gives a numeric threshold for "how large is large enough." The criterion is: given the effect size of your current changes, can your existing cases still distinguish between them? When you can't distinguish, that's when to expand.

>> Lesson 6: Hands-On: Build an Eval Track for Your Agent