“ My agent passes the benchmark” usually means it got the right answer on a static dataset. That metric is insufficient for agents. Agents act in multi-turn, dynamic environments. Their outputs are actions (click, call API, write file), not text. An action can be syntactically valid but contextually unfaithful — clicking a button that doesn’t exist, assuming a search succeeded when it returned empty, fabricating a user reply. Sun et al. (2025) call this agentic hallucination and introduce MIRAGE-Bench to measure it. Hu et al. (2025) introduce MemoryAgentBench for memory-specific evaluation. Artificial Analysis (2026) runs AA-Omniscience for factual reliability. Together they form a new evaluation stack.

The Three-Taxonomy of Agentic Failure

Sun et al. (2025) categorize hallucination by what the action is unfaithful to:

Category	Definition	Example
Unfaithful to task instructions	Violates goal, exceeds constraints, assumes unstated intent	User: “summarize this PDF” → Agent: “translates it to French”
Unfaithful to interaction history	Repeats completed steps, ignores prior outcomes, contradicts own trajectory	Agent clicks “Submit” → gets error → clicks “Submit” again with same data
Unfaithful to environment observations	Hallucinates elements/properties absent from env	Clicks button#save when DOM shows only button#submit

These are not text hallucinations. They are action hallucinations — and they compound. A single unfaithful action poisons the subsequent history, cascading into further errors.

MIRAGE-Bench: The Evaluation Framework

Core innovation: Contextual snapshots. Instead of running agents in live, stochastic environments (where reproduction is impossible), they:

1. Roll out agents on risk-inducing tasks across 6 environments (WebArena, TheAgentCompany, SWE-Bench, OSWorld, WorkArena, τ-bench)
2. Filter trajectories to isolate decision points where a risk condition is present and context contains sufficient evidence for a correct decision
3. Freeze the full state (instructions + history + observation) → snapshot
4. Re-evaluate any model on the same frozen snapshot → deterministic, reproducible

LLM-as-Judge with risk-aware prompts scores each action:

Utility(c) = 1   if Faithful Action
           0.5  if Incomplete Action (uncertain, explores without progress)
           0    if Hallucinated Action

Metrics:

• Utility Score (US) = mean Utility(c) over all snapshots
• Hallucination Rate (HR) = proportion with Utility = 0

The Sobering Results (June 2026)

Model	Utility Score	Hallucination Rate
GPT-4o-2024-11-20	0.569	0.339
Gemini-2.5-flash	0.586	0.308
Qwen2.5-32B-Instruct	0.581	0.324
Claude-3.5-Sonnet	~0.58	~0.31
DeepSeek-reasoner	incomplete (context window)	—

Three findings that should change how you build:

1. Hallucination is persistent. Even top models hallucinate ~30% of actions in adversarial snapshots. This is not a “long tail” — it is the base rate under realistic perturbations.
2. Proprietary lead is marginal. Qwen2.5-32B matches GPT-4o. Scaling + instruction tuning alone do not solve faithfulness. Targeted alignment for interactive contexts is needed.
3. Patterns are recurrent. Fabricating buttons, assuming prior success, inventing details — these are inductive biases from dialogue training (speculative completion) misaligned with agentic environments.

MemoryAgentBench: Memory-Specific Evaluation

Hu et al. (2025) argue existing benchmarks (LongBench, Needle-in-Haystack) test static long-context QA, not incremental multi-turn memory. They build MemoryAgentBench by transforming long-context datasets into multi-turn format: information arrives incrementally; the agent must memorize, update, and retrieve across turns.

Key insight: Memory agents fail differently than chat models. They forget updates (user changed preference), conflate episodes (which session had which fact), and hallucinate procedures (how to do X). MemoryAgentBench isolates these with:

• Memorization tasks — retain facts across turns
• Update tasks — revise stale facts
• Retrieval tasks — fetch relevant slice under distraction
• Reasoning tasks — synthesize across episodes

AA-Omniscience: Factual Reliability Without Guessing

Artificial Analysis (2026) tackles the “guessing incentive”: standard benchmarks reward correct answers (1 pt) and treat abstention same as wrong (0 pt). Rational model → always guess. AA-Omniscience uses 6,000 questions across 42 topics where abstention is scored higher than a wrong answer. The Reliability Index measures: does the model know when to stop?

June 2026 leaderboard:

• Claude Opus 4.8 (reasoning): Index +40, Accuracy ~52%
• Gemini 3.1 Pro: Index +33, Accuracy ~48%
• GPT-5.5: Accuracy ~57% (highest), Index lower — answers more of what it doesn’t know

Lesson for agents: A model that knows more but abstains less is less reliable as an agent. You want the model that says “I don’t know” → triggers retrieval / asks user / defers. Reliability ≠ Accuracy.

Practical Evaluation Stack for Builders

Layer	Tool	What It Catches
Unit	Snapshot tests (frozen context → expected action)	Regressions on known risk patterns
Integration	MIRAGE-Bench snapshots (subset)	Instruction/history/observation faithfulness
Memory	MemoryAgentBench workloads	Update robustness, cross-episode retrieval
Factual	AA-Omniscience style eval on your domain	Guessing vs abstaining on domain facts
Production	LLM-as-judge on live traces (sampled)	Drift, novel failure modes, cost/utility tradeoff

Minimal viable eval for a new agent:

1. Curate 50–100 snapshots from your own trajectories (successes + failures)
2. Label: faithful / incomplete / hallucinated (per Sun et al. taxonomy)
3. Run your agent on snapshots → compute US / HR
4. Gate deployment: US ≥ 0.7, HR ≤ 0.15 (adjust for risk tolerance)
5. Nightly: re-run on growing snapshot corpus; alert on regression

The Trap: Task Success Rate

An agent can have 90% task success on a benchmark and 40% hallucination rate on MIRAGE snapshots. How? The benchmark tasks are forgiving — multiple paths to success, no adversarial perturbations. The snapshots are adversarial — isolated decision points where one wrong action = failure.

Task success measures “can it sometimes work?” Faithfulness measures “does it work for the right reasons?” You need both. The second is harder and more predictive of production reliability.

What to Do Tomorrow

1. Stop reporting only task success. Add US/HR from a snapshot eval.
2. Build your snapshot corpus. Every production failure → extract snapshot → add to eval set.
3. Penalize guessing in your prompts. Explicit “if uncertain, call retrieve / ask_user / abstain” with higher reward than wrong action.
4. Evaluate the host, not just the model. Context truncation, tool timeouts, parallel call ordering — these are host decisions that cause agent failures. Log them. Judge them.
5. Run MemoryAgentBench workloads if your agent has persistent memory. Update robustness is the silent killer.

The model is a commodity. The evaluation harness is your moat.