Context Windows & Prompting

A language model has no hidden memory between calls. For one inference, the only thing it can reason about is the text you place in its context window — and nothing outside that window exists to it. The window is the model's entire working memory.

What lives in there is broader than the question you typed. On a single agent turn the context typically holds:

• the system prompt (policy, persona, rules)
• the conversation history (every prior user and assistant turn)
• tool schemas (the definitions of functions the model can call)
• tool results returned from those calls
• retrieved documents (RAG passages, files, search hits)
• the model's own prior output in this turn

Everything is concatenated into one sequence of tokens, and the model attends over all of it at once. This single fact explains most of an agent's behavior. If a detail isn't in the window, the model cannot use it, no matter how obvious it seems to you. If contradictory instructions are both in the window, the model has to reconcile them. Mastering agents starts with controlling exactly what enters this window — and what you keep out.

Think of the context window like a desk: you can spread out only so much paper before things fall off the edge. That edge is measured in tokens — chunks of text, roughly 3–4 characters of English each — and the window is a hard cap, because input plus output together must fit on the desk. As of mid-2026, those desks have grown enormous:

But a big window is a budget, not a free lunch. Tokens are priced, so a bloated context costs real money on every call. More importantly, models often degrade in quality before the advertised limit — effective, reliable capacity is frequently only 60–70% of the marketed number for demanding tasks. In a long-running agent the window fills relentlessly with tool results and prior reasoning, so you must treat it as a scarce, contested resource. Practical levers — compaction (summarizing history can cut 60–80% of tokens), just-in-time retrieval (pull a document only when needed), and sub-agents that return distilled summaries — exist precisely because a larger window does not eliminate the need to manage what's in it.

Imagine a transcript where every line is labeled with who said it — the boss, a customer, the assistant, or a machine reporting back. That labeling is what roles do: APIs don't hand the model one undifferentiated blob of text, they tag each message with a role so the model knows its source and how much authority it carries. The common roles:

• system — high-priority instructions: policy, persona, rules. Set by the developer, not the end user.
• user — input from the end user.
• assistant — the model's own prior turns.
• tool — results returned from a function/tool call.

Providers differ in the details. Anthropic keeps system as a top-level field separate from the messages array. OpenAI puts roles in the array and adds a distinct higher-trust developer role; their model follows a chain of command — developer > user — analogous to function definitions outranking arguments. Google uses system_instruction plus user/model roles.

A crucial subtlety: roles set priority, not isolation. The model still processes every role's tokens together in one window. The system prompt is weighted more heavily, but it is not a sealed channel the model obeys unconditionally — which is exactly what prompt injection exploits by smuggling adversarial instructions into user or tool content.

Prompting is how you shape behavior without retraining. Five durable levers do most of the work:

1. Clear instructions — be specific, prefer positive phrasing ("do X") over negative ("don't do Y"), and explain the why so the model generalizes.
2. Role assignment — give the model a role in the system prompt ("You are a meticulous financial analyst") to focus tone and behavior.
3. Delimiters — separate sections unambiguously. Anthropic recommends XML tags; OpenAI recommends XML tags and Markdown headers. Both help any model parse multi-part prompts.
4. Examples (few-shot) — show 3–5 worked examples to lock in format and style.
5. Output format — state the exact shape you want, ideally with a schema.

Here's the toolkit assembled:

<role>You are a precise data-extraction assistant.</role>

<instructions>
Extract the company, amount, and date from each invoice line.
Return only valid JSON. If a field is missing, use null.
</instructions>

<example>
Input: "Acme Corp billed $4,200 on 2026-03-01"
Output: {"company": "Acme Corp", "amount": 4200, "date": "2026-03-01"}
</example>

<document>
{{ the text to process }}
</document>

Notice the document sits last, right before the model answers — that placement is deliberate, as the next section explains.

These three strategies are really one question: how much do you show the model before asking it to work? They sit on a spectrum of scaffolding, from "just tell it what to do" to "walk it through worked examples and reasoning."

• Zero-shot — instructions only, no examples. Fastest, and often enough for strong modern models.
• Few-shot (multishot) — 3–5 in-context examples. It dramatically improves format consistency on structured tasks. Wrap examples in <example> tags (Anthropic) or as user/assistant turn pairs (OpenAI).
• Chain-of-thought (CoT) — ask the model to reason step by step before answering. The original work was few-shot (Wei et al., 2022); the zero-shot trigger "Let's think step by step" (Kojima et al., 2022) is now widely preferred.

Here is the counterintuitive recent finding: for highly capable current models, zero-shot CoT often matches or beats few-shot CoT (arXiv:2506.14641, EMNLP 2025). When the model already reasons well, hand-picked exemplars can constrain it rather than help; few-shot's main remaining value is aligning output format, not teaching reasoning.

One more distinction that trips people up: CoT prompting is not the same as a reasoning model. CoT instructs a standard model to externalize reasoning in its visible output. Reasoning models (the o-series, Claude extended thinking) have a separate internal thinking budget they spend before the final answer — a different mechanism, covered later in the course.

Two failures account for a surprising share of bad outputs.

Ambiguity. Vague prompts force the model to guess your intent, and it often guesses wrong or inconsistently. The fix is unglamorous but reliable: state the audience, the constraints, the format, and what to do in edge cases. Spelling out "if the field is missing, return null" removes a whole class of silent errors.

Lost in the middle. Models have a U-shaped attention bias: they reliably use information at the start and end of a long context but show 10–25% accuracy degradation for material buried in the middle (Liu et al., TACL 2024). This is structural, not a bug you can prompt away.

The practical fix is positional:

1. Put long documents at the top of the prompt.
2. Put the question or task instruction at the very end, right before the model generates.

Anthropic reports this ordering can improve response quality by up to 30%. It's also why the toolkit example earlier placed <document> last. When you can't avoid a long middle, lean on retrieval to surface only the relevant passages rather than dumping everything and hoping the model finds it.

A chatbot prompt mostly sets tone, scope, and response format — it shapes one good reply. An agent system prompt is categorically different because it governs an autonomous loop, not a single turn. It must encode policy and procedure, not just personality.

An effective agent system prompt specifies:

• Role and goal — what the agent is for.
• Available tools and when to use each — the heuristics for choosing actions.
• Multi-step strategy — how to plan, sequence, and recover from errors.
• Stopping conditions — when the task is done and it should hand back a final answer.
• Output format for downstream consumers — other code or agents may parse the result.

You are a research agent. Goal: answer the user's question with cited sources.

Tools:
- web_search(query): use FIRST when facts may be recent or you're unsure.
- fetch_page(url): use to read a promising result in full.

Strategy: search, read the 2 most relevant pages, cross-check claims across
sources, then answer. If a page fails to load, try the next result.

Stop when: you can answer with at least two agreeing sources, OR after 5 tool
calls — then answer with the caveat that evidence was limited.

Output: a 3-sentence answer followed by a `Sources:` list of URLs.

This is the on-ramp to context engineering — deliberately curating the whole window across a long run — which the course returns to in depth.