Lesson 2: Anatomy of Context: System Prompt, Tools, and Examples

Learning goals:

• Take apart the context of a single LLM request (system prompt, tool definitions, examples, message history) and explain why all four are spending the same attention budget
• Diagnose the "altitude" problem in a system prompt — hardcoded and brittle at one end, vague and signal-free at the other — and rewrite it to the right altitude
• Audit tool definitions and examples through the lens of context cost: merge overlapping functionality, trim what comes back, and replace a list of edge cases with a few canonical examples

Prerequisites: You've finished Lesson 1 and accept the premise that context is a finite resource | Prev: Lesson 1 << | Next: Lesson 3 >>

Open the Hood: What's Actually Loaded in a Request

Lesson 1 borrowed Anthropic's definition: context engineering is "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference"¹. That's still abstract — which tokens does "the optimal set" actually consist of? This lesson does one thing: opens the hood so you can see what's loaded into the window every time a request goes out.

You hand-wrote a harness loop back in "Agent Harness Fundamentals: Loops and Control," so the shape of a request should be familiar. Sorted by what it holds, the context of a single request is roughly these four blocks:

Context of one request├── System prompt (role, rules, background knowledge)├── Tool definitions (each tool's name, description, parameter schema)├── Examples (few-shot: input/output samples that portray the expected behavior)└── Message history (user messages, model replies, each turn's tool calls and results)

Here's what matters: these four aren't four sealed-off pockets, they're neighbors in one window. "LLMs have an "attention budget" that they draw on when parsing large volumes of context," and "Every new token introduced depletes this budget by some amount"¹. An extra paragraph of padding in the system prompt is attention the message history doesn't get; ten tools sitting in the manifest that nobody ever calls thin out the share the examples receive. And as Lesson 1 covered, "as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases"¹ — a degradation that arrives as a slope, not a drop, since "These factors create a performance gradient rather than a hard cliff"¹. Which is why waste in any one block never announces itself with an error. It just quietly makes the whole thing a little dumber, and by the time you notice, you usually can't point at the line that did it.

The four blocks also grow at different rates. System prompt, tool definitions, and examples are basically static — however long you wrote them, that's how long they stay. Message history swells inside the loop instead, because "An agent running in a loop generates more and more data that could be relevant for the next turn of inference"¹. In the harness you built in "Agent Harness Fundamentals: Loops and Control," the line that pushes each turn's tool result onto the messages array is that swelling, live. What to do about history isn't this lesson's job — Lesson 4, "Compaction and Notes: Context Management for Long Tasks," is dedicated to it. This lesson gets the three static blocks under control, since those are the fixed cost you pay on every single turn.

Turning "context, therefore, must be treated as a finite resource with diminishing marginal returns"¹ into something you can act on means asking each block the same question: how much behavioral improvement did these tokens buy? Let's go block by block.

The Altitude of a System Prompt: Two Ways to Fail

Anthropic's engineering write-up uses "altitude" for the level of abstraction a system prompt sits at, and names two failure extremes. At one end, "engineers hardcoding complex, brittle logic in their prompts to elicit exact agentic behavior"¹. At the other, "vague, high-level guidance that fails to give the LLM concrete signals for desired outputs"¹. Let's write three versions of a system prompt for the same order-support agent, so both traps and the good answer sit side by side.

Flying too low — the logic gets hardcoded:

You are an order support agent. Follow these rules exactly:1. If the user says "it never arrived," reply with template A.2. If the user says "it arrived damaged," reply with template B and issue a $10 coupon.3. Order numbers starting with TB: check shipping first. Starting with JD: check inventory first.4. If the user's message contains the word "complaint," hand off to a human.… (14 more rules omitted)19. When a user matches both rule 2 and rule 4, rule 4 wins.20. If nothing above matches, reply "Sorry, I'm not sure I understand."

Each rule covers exactly the case it spells out. What happens when the user writes "the box got crushed" instead of "damaged"? No rule catches it, so it falls through to rule 20 and plays dumb. Worse, the rules start fighting each other, so you add rule 19 to referee — and at that point you're writing an if-else interpreter in natural language. Every addition makes the prompt longer, more brittle, and more expensive in attention, while the cases you didn't write down will always outnumber the ones you did.

Floating too high — nothing but slogans:

You are an order support agent. Be professional and friendly, handle user problems flexibly, and keep users happy.

You saved the tokens, but the model gets no concrete signal at all. Where's the boundary on refunds? How much compensation can it authorize? What has to go to a human? It's all guesswork. "Keep users happy," pushed to its limit, can mean refunding money that should never have been refunded — and that's not disobedience, that's you telling it nothing.

The right altitude — Anthropic's standard is a prompt that's "specific enough to guide behavior effectively, yet flexible enough to provide the model with strong heuristics"¹:

You are an order support agent. Your goal is to resolve the user's post-purchase problem within a single conversation.
Judgment principles:- If an order hasn't shipped, you can refund on request; if it has shipped, guide the user to refuse delivery or start a return.- Damage, wrong item, and missing item are our fault: offer compensation up front, don't wait for the user to ask.- When fault is unclear, confirm the key facts first (order number, photos), then decide.
Hard limits (never cross):- No single compensation above $50 — hand off to a human above that.- Any conversation involving physical injury or a legal dispute: hand off to a human immediately and flag it.

Look at the structure of that version: principles do the generalizing, red lines do the compliance. "The box got crushed" isn't written verbatim into any rule, yet it lands naturally under "damage is our fault"; the things that genuinely aren't negotiable (the compensation ceiling, legal disputes) are pinned down as a handful of hard limits. It's less than half the length of the 20-rule version and covers strictly more ground.

There's a shortcut question for judging altitude: faced with a case you didn't write down, does this prompt give the model a direction to reason from? The low version can't (it only knows how to fall through), the high version gives an empty direction ("happy"), and the right-altitude version gives principles that transfer.

The Always-Loaded Layer: CLAUDE.md Discipline

The system prompt isn't only the string you typed. Plenty of harnesses park project-level configuration permanently in the system layer, and Claude Code's CLAUDE.md is the canonical case: the docs say "CLAUDE.md is a special file that Claude reads at the start of every conversation."² "Read every time" means every line of it spends attention budget in every session, which is why the discipline the docs impose on it is strict.

Rule one: only put in what applies broadly — "CLAUDE.md is loaded every session, so only include things that apply broadly."² A build command only one subdirectory needs, a convention only one kind of task needs: neither has earned a seat in the always-loaded layer.

Rule two: run a deletion test line by line. "Keep it concise. For each line, ask: "Would removing this cause Claude to make mistakes?" If not, cut it."² This isn't fastidiousness. The docs warn outright that "Bloated CLAUDE.md files cause Claude to ignore your actual instructions!"² That's the attention budget made concrete — every optional line you stuff in dilutes the few lines that genuinely matter. The same page goes so far as to call the context window "the most important resource to manage," noting that "Claude's context window fills up fast, and performance degrades as it fills."²

So where does the rarely-needed-but-occasionally-essential material live? Claude Code's answer is skills: loaded on demand, since "Claude loads them on demand without bloating every conversation."² That pairing — keep the resident layer minimal, fetch the rest when needed — is exactly Lesson 3's subject, so let's just book it here.

One aside: if your own project has an AGENTS.md, a system_prompt.txt, or anything similar sitting permanently in context, run the same deletion test on it. Bloat in the always-loaded layer is the sneakiest kind — it never shows up in any one turn of the conversation, and it taxes every one of them.

Tools Are Context: The Definition Costs, and So Does the Return

A tool shows up in context twice: its definition rides along with every request, and its return value enters the message history. Both ends spend budget.

The earlier course "Agent Tool Calling: Getting Agents to Actually Do Things" covered the functional side of tool design — how to define parameters, how to handle errors. This lesson takes a different angle: every tool definition is a stretch of tokens the model has to read and understand. Anthropic's bar is that "tools should be self-contained, robust to error, and extremely clear with respect to their intended use"¹, and that you're "building tools that are well understood by LLMs and have minimal overlap in functionality"¹. Here's a pair that fails both (schemas abbreviated for readability):

Both tools can look up an order, both descriptions are vague, and the boundary between them isn't something even their author pinned down. That's the whole point of the "minimal overlap in functionality"¹ requirement: pile up tools with fuzzy boundaries and you've handed the model the "which one do I use?" question fresh on every turn. Merge them into one and state the purpose and behavior clearly, and the question disappears:

That description says more than "what it does" — it spells out the shape of what comes back and what happens when there's nothing to return, which is "robust to error"¹ made literal: the model doesn't have to guess what a miss looks like, so it's far less likely to invent some odd recovery move when a lookup comes up empty.

Now the return side. Anthropic asks for tools "returning information that is token efficient and by encouraging efficient agent behaviors"¹. A tool that dumps back an order's forty-plus internal fields plus its complete audit log pours a bucket of low-value tokens into the message history on every call — and those tokens stay in the history, taxed again on every turn that follows. "Trim by default, detail=true on demand," as in the example above, is the standard shape of the fix.

Finally, the tool manifest itself needs subtraction. Ten tools in context that never get called are still billed in full every turn for their definitions. Auditing a tool list and auditing a CLAUDE.md are the same motion — would removing it cause mistakes? If not, cut it.

Examples: Pick Canonical Ones, Don't Pile Up a List

Examples (few-shot) are the third static cost. Their usual rot goes like this: every production badcase gets a matching example added to the prompt, and six months later you own a 30-entry catalog of edge cases. Anthropic is blunt about it — don't "stuff a laundry list of edge cases into a prompt"¹; instead "curate a set of diverse, canonical examples that effectively portray the expected behavior of the agent"¹.

"Canonical" means one example stands for a class of behavior, not one specific situation. Back to the support agent: three examples are enough to frame the whole behavior space.

1. The standard flow: the full walk of looking up the order, assigning fault, offering a resolution.
2. Taking responsibility up front: we shipped the wrong item, and compensation is offered before the user asks.
3. Escalation: it's beyond the agent's authority, so it explains politely and hands off to a human.

"Diverse" means those three cover different branches of the judgment, rather than being three variations on the same behavior.

So where do the edge cases go? Most of them should be promoted back up to the principle layer of the system prompt. "What if the user is abusive" doesn't need a 300-token full conversation example; one principle — "when a user is upset, stay measured and keep the focus on solving the problem" — does the job. Examples teach what the expected behavior looks like; principles teach which direction to reason in when something new shows up. You may already have spotted it: adding an edge case to the example list and adding a branch to the system prompt are two faces of the same coin — both are patching at low altitude, and only rising to the principle layer actually seals the gap.

A Block-by-Block Checklist: Putting the Anatomy to Work

Collapsing this lesson into something executable. Before you add anything to context, run the matching question:

There's one more principle worth carrying out of this lesson, beyond the table. Writing about building agent systems, Anthropic offers a sense of proportion: "you should consider adding complexity only when it demonstrably improves outcomes."³ The original context is system architecture, but it holds just as well for every block of context — one more tool, one more rule, one more example is all added complexity. Make it prove it buys behavioral improvement before you let it on board.

That's a pass over all three static blocks. But there's a further question waiting: some information shouldn't be loaded into the window ahead of time at all — rather than guessing what the model will need, let the agent go find it at runtime. That's what the next lesson is about.

Recap

• The context of one request is four blocks — system prompt, tool definitions, examples, message history. They share a single attention budget, and every new token depletes it by some amount¹.
• Context is a finite resource with diminishing marginal returns¹; asking block by block "what did these tokens buy" is far more actionable than vaguely "tweaking the prompt."
• A system prompt has two failure extremes — hardcoded brittle branches, and vague slogans that carry no signal. The right altitude is specific enough to guide behavior yet flexible enough to leave the model strong heuristics¹, and "principles plus hard limits" is the practical structure for landing there.
• Always-loaded content like CLAUDE.md gets read at the start of every conversation: include only what applies broadly, and run the deletion test on each line, because bloated files make the model ignore your actual instructions².
• Tools spend budget at both ends, definition and return: keep the intended use extremely clear, overlap minimal, and returns token-efficient¹; tools that never get called are still billed in full every turn.
• Curate a few diverse, canonical examples instead of stuffing in a laundry list of edge cases¹; most edge cases belong back at the principle layer.
• Before adding any complexity to context, remember Anthropic's sense of proportion: add it only when it demonstrably improves outcomes³.

>> Lesson 3: Just-in-Time Retrieval: Letting the Agent Fetch Its Own Context