Gemini 1.5 Pro (1M), GPT-4.1 (1M), Llama 4 (10M). The context window arms race suggests more tokens = smarter agent. Chroma (2025) proves the opposite: performance degrades as input tokens grow, even for simple tasks. They call it context rot. Needle-in-a-Haystack (lexical match) is solved; semantic synthesis over millions of tokens is not. Irrelevant context distracts. Relevant context buried in the middle is ignored. Liu et al. (2023) showed this years ago: “lost in the middle” is a measurement, not a metaphor.

The winning strategy is context engineering: the careful construction and management of what enters the model’s context window. Where and how information is presented matters more than how much fits. This post is a practical guide to building that discipline.

The Context Budget: Tokens Are Money

Every host has a hard limit (model max) and a soft limit (cost/latency). Treat both as a budget. Allocate explicitly:

Budget Category	Typical %	Contents
System prompt	5–10%	Instructions, tool schemas, guardrails, few-shot examples
Working memory	15–25%	User profile, preferences, session state, active task spec
Retrieved context	40–60%	Top-k from memory/vector/graph, tool results, web extracts
Conversation history	15–25%	Recent turns (full), older turns (summarized)
Headroom	5–10%	Model’s own reasoning + next tool calls

Rule: If retrieved context + history > 70%, you are starving the model’s reasoning space. Trim.

Retrieval: Precision > Recall

Dump-the-database-into-context fails. Zhou et al. (2026) show hybrid retrieval (vector + graph + keyword + agentic routing) wins because each query type needs a different index. Build a retrieval pipeline:

query → classify intent (fact / temporal / procedural / semantic)
       → route to index (SQL / graph / vector / keyword)
       → rerank (cross-encoder / LLM judge)
       → top-k with diversity (MMR / cluster)
       → format with citations [doc_id]
       → inject into budget slot

Critical: Format retrieved chunks with provenance (source, timestamp, validity window). The model cannot weigh evidence it cannot attribute.

Truncation Policy: Lose the Right Things

When budget overflows, what goes first? Not the system prompt. Not the task spec. Not the most recent turns.

Priority order (keep → drop):

1. System prompt + tool schemas (immutable)
2. Current task specification + active constraints
3. Most recent 3–5 turns (full fidelity)
4. Retrieved context (highest rerank score first)
5. User profile / long-term preferences
6. Older conversation history (summarized → dropped)
7. Few-shot examples (if any)

Implement as a sliding window with summarization: when history exceeds N tokens, summarize oldest M turns into a single “prior context” block. Preserve: decisions made, facts established, open questions. Drop: pleasantries, repeated context, tool output verbose logs.

Placement Matters: Primacy and Recency

Liu et al. (2023): model attends best to start and end of context. Structure accordingly:

[System Prompt]
[Task Spec + Active Constraints]
[Working Memory: User Profile / Session State]
[Retrieved Context: most relevant FIRST]
[Conversation History: recent turns LAST]

Put the answer-critical evidence at the very top (after system) or very bottom (before generation). Never bury it in the middle of a 50k token dump.

Tool Results: Compress, Don’t Pass Through

A web_extract call returns 5k tokens of HTML-converted markdown. The model needs 200. Compress at the host layer:

def compress_tool_result(result: str, max_tokens: int, query: str) -> str:
    # 1. Extract sections relevant to query (embedding similarity)
    # 2. Summarize each section to 1–2 sentences
    # 3. Keep tables/lists intact (high info density)
    # 4. Add citation markers [source_id]
    # 5. Enforce token budget
    return compressed

This is not “dumbing down” — it is preventing context rot. The model reasons better over 500 tokens of distilled evidence than 5000 tokens of noise.

Parallel Calls: Budget for Concurrency

Hosts run independent tool calls in parallel. Good. But each returns tokens. If you fire 5 searches in parallel, you get 5× context. Budget for the sum of parallel results. Either:

• Limit parallelism (max 2–3 retrieval calls per turn)
• Compress each result before injecting (see above)
• Use a “synthesis” turn: fire tools → compress → next turn reasons over compressed

The Context Engineering Checklist

Before deploying an agent, answer:

• Budget defined — hard max, soft target, per-category allocation
• Retrieval pipeline — intent classification → multi-index → rerank → format with provenance
• Truncation policy — priority order coded, summarization tested
• Placement strategy — critical evidence at primacy/recency positions
• Tool result compression — every tool has a compressor; none pass raw
• Parallelism cap — max concurrent calls × max compressed tokens ≤ budget
• Eval includes context ablation — drop each category; measure US/HR delta
• Observability — log: tokens per category, retrieval latency, compression ratio, final context size

The Host Owns This

The model does not manage its context. The host does. The model benefits from discipline it cannot enforce. Context engineering is host engineering.

If you are building the host: this is your core product. Not the prompt. Not the model choice. The pipeline that turns a messy world into the right 8k tokens, every turn, predictably.

If you are using an agent: ask the builder “show me your context budget.” If they cannot, the agent is a toy.