---
title: Is Your MCP Mock Lying? Raising Fidelity From 48% to 95%
description: Our first mock of a production MCP server matched 48.4% of 612 live checks; its made-up values caused 47 false failures. Rebuilt from live shapes: 94.8%. How.
canonical: https://spiritroom.app/blog/mcp-mock-fidelity/
published: 2026-10-21
updated: 2026-10-21
---

> Markdown version of https://spiritroom.app/blog/mcp-mock-fidelity/ for agents. Index: /llms.txt

[Blog](/blog/) / [Test MCP servers with agents](/guides/test-mcp-servers-with-agents/)

# Is your MCP mock lying to you? Raising mock fidelity from 48% to 95%

Mocks are how you test write paths, spend paths and dozens of runs without touching production. But agents read every value. A mock that's wrong in small ways makes agents fail in ways your users never would, and you'll fix the wrong thing.

Leonid Bugaev · Published 21 Oct 2026

**TL;DR**

- **Before:** a mock generated from types matched 48.4% of 612 checks against 30 live read calls.
- **It caused false failures:** of 151 failed checks in earlier agent runs, 47 were mock artifacts, 63 scorer or harness bugs, 41 agent variance, and 0 real server issues.
- **One bad value did most damage:** a status flag the mock set to false made Codex correctly refuse to act: 27 failed checks.
- **After:** rebuilt from live shapes, sizes and enums: 94.8% on reads, 71.4% → 89.3% on writes.
- **Then re-run:** the corrected experiment gave 72% → 91% (baseline → redesign), replacing a headline we had to withdraw.
[Why mock fidelity matters](#why)[How we measured fidelity](#measure)[What the first mock got wrong](#before)[What fixed it](#after)[What a mock still can't tell you](#limits)[How we measured](#how-we-measured)[FAQ](#faq)

## Why mock fidelity matters

Agents act on values. If the mock says a resource isn't admitted, a careful agent refuses to use it, and your test records a failure your users would never see. You then “fix” your server for a bug that only exists in your mock.

That happened to us. An early agent experiment on a mock of a 99-tool production server produced a striking before/after headline. When we traced every failure, most came from the mock and the scorer. We withdrew it.

## How we measured fidelity

Replay every distinct live read call (30 calls, from the read-only proxy logs) against a fresh mock, and score 612 checks:

- error class, error text, error guidance;
- framing: text plus structured content, number of items;
- keys (Jaccard ≥ 0.9) and value types;
- text and structured tokens within ±10% (o200k\_base);
- array lengths, enum coverage (≥ 80%), per-item tokens;
- tools/list equality.

The test fails below 90%. Live data is never stored: only shapes and sizes, with synthetic look-alike fixtures.

**Want a mock agents can't tell from your server?**

We build a live-shaped mock of your MCP server, measure its fidelity against read-only live calls, and run agent tests on it, including write paths.

[Talk to us →](/#contact)

## What the first mock got wrong

Reads, before

*296/612 checks (48.4%)*

Reads, after

*580/612 (94.8%)*

Writes, before → after

*71.4% → 89.3% (28 checks)*

| Check | Before | After |
| --- | --- | --- |
| Shape keys | 14/27 | 26/27 |
| Text tokens ±10% | 7/27 | 23/27 |
| Enum values | 34/176 | 168/176 |
| Array lengths | 47/120 | 117/120 |
| Per-item tokens | 9/51 | 39/51 |
| Error guidance | 0/3 | 3/3 |

Values generated from types only produced things the real server never returns:

- A status flag set to false made Codex correctly refuse to search: **27 failed checks**.
- Contradictory counts: a total of 9 next to 2 items.
- Run-to-record links that covered only one group, so agents acted on 2 of 5 records.
- A preview tool pointed at a group that didn't exist.

Across the earlier runs' 151 failed checks: 47 mock artifacts, 63 scorer or harness bugs, 41 agent variance, 0 real server issues. Scorer bugs are their own trap: [validate your eval scorer](/blog/validate-your-agent-eval-scorer/).

## What fixed it

- **Byte-identical tools/list and initialize** from the live capture.
- **Live envelopes:** default and max page sizes, offset guidance, end state.
- **Live mixes:** a synthetic pool with the real stage distribution.
- **Token-matched synthetic text** (±5%), so results are as heavy as real ones. A mock list call came out at 104k–111k tokens against 110,721 live.
- **Real error texts and guidance,** including the unhelpful ones, because agents react to them.
- **A side-effect ledger** so graders read what changed, not what the agent said.

With the fixed mock and scorer, the re-run (150 runs) gave 54/75 (61–81%) on the raw tool surface and 68/75 (82–95%) on the redesign.

**Go deeper:** [Test against production read-only](/blog/read-only-agent-testing-production/), [Pass rates with ranges](/blog/agent-pass-rates-confidence-intervals/), [The two-step confirm pattern](/blog/mcp-two-step-confirm-spend/).

## What a mock still can't tell you

- Fidelity is measured on the same probes the mock was built from: a regression check, not independent validation.
- Write, spend and send responses are partly inferred, because live testing stayed read-only.
- Exact token counts depend on fixtures; re-measure live before quoting a size.
- Error sizes especially: our mock's error for one bad update call was 296 tokens; the live server's was 12,643.

Back to [testing MCP servers with agents](/guides/test-mcp-servers-with-agents/).

## How we measured

- **Fidelity:** 30 distinct live read calls and 7 observed non-mutating write-shape calls, 16 Oct 2026; 612 and 28 checks.
- **Failure audit:** every failed check from two earlier mock experiments, classified by reading transcripts.
- **Re-run:** 5 jobs × 3 apps × 5 runs × 2 surfaces; 95% Wilson ranges.

## FAQ

### How do I build a realistic mock of an MCP server for agent tests?

Start from a read-only capture of the live server: byte-identical tools/list and initialize, real response shapes, sizes (in tokens), enum mixes, array lengths, error texts and guidance. Fill it with synthetic look-alike data, then score the mock against live calls and fix until it passes a threshold.

### What should mock fidelity checks cover?

Error class and text, response framing, keys and types, token size within about ±10%, array lengths, enum coverage, per-item size and the tool list itself. We used 612 checks over 30 live calls and a 90% pass bar.

### Can a mock replace live testing?

No. It's a regression check built from the same probes, and write responses are partly inferred. Re-measure sizes and behaviours live, read-only, before quoting them.

**Want a mock agents can't tell from your server?**

We build a live-shaped mock of your MCP server, measure its fidelity against read-only live calls, and run agent tests on it, including write paths.

[Talk to us →](/#contact)

## Keep reading

- Up[How to test an MCP server with real AI agents (guide)](/guides/test-mcp-servers-with-agents/)
- Related[Validate your agent eval scorer: 0 of our first 151 failures were the server's fault](/blog/validate-your-agent-eval-scorer/)
- Related[How to test agents against a production MCP server without spending, writing or sending anything](/blog/read-only-agent-testing-production/)