Every agent presentation shows the prompt. The prompt is 5% of the system. The other 95% is the host: the runtime that loads the model, manages the context window, executes tools, persists memory, enforces budgets, and decides when to stop. Xu et al. (2026) survey 200+ papers and conclude the unsolved problems are scheduling, recovery, state consistency, and cost control — all host problems. This post is a checklist for building a host that doesn’t embarrass you in production.

The Host Responsibilities (Exhaustive)

Subsystem	What It Does	Failure If Missing
Model loader	Loads weights, applies chat template, manages KV cache	OOM, wrong format, silent degradation
Context manager	Assembles prompt: system + working memory + retrieved + history + headroom	Context rot, lost facts, budget overflow
Tool executor	Validates schemas, runs code, times out, retries, compresses results	Hanging calls, schema errors, token explosion
Memory system	Extraction (S), Storage (R), Retrieval (Q), Maintenance (U) Zhou et al. (2026)	Amnesia, stale facts, contradictions
Scheduler	Parallelizes independent calls, sequences dependent, enforces concurrency limits	Deadlocks, latency spikes, budget blowout
Budget enforcer	Token counter per turn, per session, per user; hard stops	Cost surprises, degraded quality
Checkpointing	Snapshots state (context + memory + tool results) every N turns	Unrecoverable failures, no replay
Eval harness	Runs snapshot tests Sun et al. (2025), regression suite, canary deployments	Silent regressions, hallucination drift
Observability	Logs: tokens/category, tool latency, retrieval precision, US/HR, cost	Blind debugging, no capacity planning
Deployment	Blue/green, rollback, feature flags, config versioning	Bad deploy = downtime

Context Manager: The Heart

class ContextManager:
    def __init__(self, budget: TokenBudget):
        self.budget = budget
    
    def assemble(self, turn: Turn) -> Prompt:
        sections = [
            ("system", turn.system_prompt, Priority.CRITICAL),
            ("task_spec", turn.task_spec, Priority.CRITICAL),
            ("working_memory", turn.working_memory, Priority.HIGH),
            ("retrieved", self.retrieve(turn.query), Priority.HIGH),
            ("history", self.compress_history(turn.history), Priority.MEDIUM),
            ("few_shot", turn.few_shot, Priority.LOW),
        ]
        return self.pack(sections, self.budget)
    
    def pack(self, sections, budget):
        # 1. Sort by priority
        # 2. Fill from highest priority, truncate lowest
        # 3. Preserve primacy/recency for retrieved + history
        # 4. Return tokens_used for observability

Key invariants: System prompt never truncated. Task spec never truncated. Retrieved context sorted by rerank score. History compressed via summarization (not truncation).

Tool Executor: The Muscle

class ToolExecutor:
    def __init__(self, registry: ToolRegistry, compressor: Compressor):
        self.registry = registry
        self.compressor = compressor
    
    async def execute(self, calls: list[ToolCall]) -> list[ToolResult]:
        # 1. Validate all schemas in parallel (fail fast)
        # 2. Group: independent → parallel; dependent → sequence
        # 3. Run with timeout + retry (exponential backoff)
        # 4. Compress each result BEFORE returning to context
        # 5. Log: call_id, tool, latency, tokens_in, tokens_out, status

Non-negotiable: Every tool has a compress(result, query, max_tokens) method. Raw tool output never enters context.

Memory System: The Four Modules

Following Zhou et al. (2026):

class MemorySystem:
    def __init__(self):
        self.R = HybridStorage(vector=pgvector, graph=kuzu, keyword=sqlite_fts)
        self.S = SchemaExtractor(schema=FactSchema)
        self.Q = HybridRouter(indices=["vector", "graph", "keyword"])
        self.U = MaintenanceScheduler(jobs=[consolidate, dedupe, version])
    
    def extract(self, dialogue: Dialogue) -> list[MemoryUnit]:
        return self.S.extract(dialogue)
    
    def store(self, units: list[MemoryUnit]):
        self.R.write(units)
    
    def retrieve(self, query: Query) -> list[MemoryUnit]:
        return self.Q.route_and_search(query)
    
    def maintain(self):
        self.U.run_due_jobs()

Deployment reality: Start with vector + keyword. Add graph when temporal/entity queries exceed 20% of workload. Run U nightly.

Scheduler: Concurrency Without Chaos

class Scheduler:
    def __init__(self, max_parallel: int = 3):
        self.semaphore = asyncio.Semaphore(max_parallel)
        self.dependency_graph = DependencyGraph()
    
    async def run(self, plan: Plan) -> list[Result]:
        # 1. Build DAG from plan (explicit deps or infer from tool signatures)
        # 2. Execute ready nodes in parallel (semaphore-limited)
        # 3. On failure: mark subgraph failed, continue independent branches
        # 4. Support checkpoint: save DAG state + completed results
        # 5. On retry: resume from checkpoint, re-run failed subgraph only

Key metric: Parallel efficiency = (sequential time) / (parallel time). Target > 2.5×.

Budget Enforcer: Tokens Are Money

class BudgetEnforcer:
    def __init__(self, limits: Limits):
        self.limits = limits  # per_turn, per_session, per_user, per_day
        self.usage = UsageTracker()
    
    def check(self, scope: Scope, estimated: int) -> bool:
        return self.usage.get(scope) + estimated <= self.limits.get(scope)
    
    def record(self, scope: Scope, actual: int):
        self.usage.add(scope, actual)
        if self.usage.get(scope) > self.limits.get(scope) * 0.9:
            alert("budget_near_limit", scope=scope, usage=self.usage.get(scope))

Hard stops: Per-turn (model max), per-session (user expectation), per-day (cost control). Soft alerts at 80%.

Checkpointing: Replayability

Every N turns (or on tool error), persist:

{
  "turn": 12,
  "context_hash": "sha256(...)",
  "memory_snapshot_id": "mem_abc123",
  "tool_results": [...],
  "model_output": "...",
  "budget_used": 45231
}

Enables: replay from any turn, A/B prompt testing on real trajectories, forensic debugging.

Eval Harness: Gate Every Deploy

# eval/gate.yml
stages:
  - name: snapshot_regression
    dataset: eval/snapshots/v3.jsonl
    metric: utility_score
    threshold: 0.70
  - name: hallucination_rate
    dataset: eval/snapshots/v3.jsonl
    metric: hallucination_rate
    threshold: 0.15
  - name: memory_update
    dataset: eval/memory/v1.jsonl
    metric: update_accuracy
    threshold: 0.85
  - name: cost_per_task
    dataset: eval/production_traces/last_week.jsonl
    metric: p95_tokens
    threshold: 8000

CI blocks on failure. Canary: 5% traffic → full metrics → promote/rollback.

Observability: The Dashboard You Need

Panel	Query
Tokens per turn (p50/p95)	sum(tokens) by turn
Tool latency (p99)	histogram(tool_latency_ms) by tool
Retrieval precision@k	judge_score(retrieved, relevant)
Utility Score / HR	eval.mirage.us, eval.mirage.hr
Cost per session	sum(tokens * $/token) by session_id
Error rate by type	count(error_type) / total_turns
Checkpoint lag	now - last_checkpoint_ts

The Host Checklist Before Launch

• Context budget defined + enforced + tested (ablation per category)
• Tool schemas versioned; compressor per tool; timeout/retries tuned
• Memory: extraction schema, hybrid storage, nightly maintenance job
• Scheduler: DAG support, checkpoint/resume, parallel efficiency > 2.5×
• Budget: per-turn/session/user/day limits + alerts at 80%
• Checkpoint: every 5 turns + on error; replay tested weekly
• Eval: snapshot gate (US ≥ 0.7, HR ≤ 0.15), memory gate, cost gate
• Observability: all 7 panels live; alerts on regression
• Deploy: blue/green, rollback < 2 min, config versioned
• Runbook: “agent goes rogue” → kill switch, memory quarantine, manual review

The Model Is Interchangeable

Swap GPT-4o → Claude 3.5 → Llama-3.1-405B. The host stays. The prompt templates adapt. The tool schemas stay. The memory system stays. The eval harness stays. The host is your product. The model is your supplier. Negotiate accordingly.