Blog / Test MCP servers with agents
Is your MCP mock lying to you? Raising mock fidelity from 48% to 95%
Mocks are how you test write paths, spend paths and dozens of runs without touching production. But agents read every value. A mock that's wrong in small ways makes agents fail in ways your users never would, and you'll fix the wrong thing.
Why mock fidelity matters
Agents act on values. If the mock says a resource isn't admitted, a careful agent refuses to use it, and your test records a failure your users would never see. You then “fix” your server for a bug that only exists in your mock.
That happened to us. An early agent experiment on a mock of a 99-tool production server produced a striking before/after headline. When we traced every failure, most came from the mock and the scorer. We withdrew it.
How we measured fidelity
Replay every distinct live read call (30 calls, from the read-only proxy logs) against a fresh mock, and score 612 checks:
- error class, error text, error guidance;
- framing: text plus structured content, number of items;
- keys (Jaccard ≥ 0.9) and value types;
- text and structured tokens within ±10% (o200k_base);
- array lengths, enum coverage (≥ 80%), per-item tokens;
- tools/list equality.
The test fails below 90%. Live data is never stored: only shapes and sizes, with synthetic look-alike fixtures.
Want a mock agents can't tell from your server?
We build a live-shaped mock of your MCP server, measure its fidelity against read-only live calls, and run agent tests on it, including write paths.
What the first mock got wrong
| Check | Before | After |
|---|---|---|
| Shape keys | 14/27 | 26/27 |
| Text tokens ±10% | 7/27 | 23/27 |
| Enum values | 34/176 | 168/176 |
| Array lengths | 47/120 | 117/120 |
| Per-item tokens | 9/51 | 39/51 |
| Error guidance | 0/3 | 3/3 |
Values generated from types only produced things the real server never returns:
- A status flag set to false made Codex correctly refuse to search: 27 failed checks.
- Contradictory counts: a total of 9 next to 2 items.
- Run-to-record links that covered only one group, so agents acted on 2 of 5 records.
- A preview tool pointed at a group that didn't exist.
Across the earlier runs' 151 failed checks: 47 mock artifacts, 63 scorer or harness bugs, 41 agent variance, 0 real server issues. Scorer bugs are their own trap: validate your eval scorer.
What fixed it
- Byte-identical tools/list and initialize from the live capture.
- Live envelopes: default and max page sizes, offset guidance, end state.
- Live mixes: a synthetic pool with the real stage distribution.
- Token-matched synthetic text (±5%), so results are as heavy as real ones. A mock list call came out at 104k–111k tokens against 110,721 live.
- Real error texts and guidance, including the unhelpful ones, because agents react to them.
- A side-effect ledger so graders read what changed, not what the agent said.
With the fixed mock and scorer, the re-run (150 runs) gave 54/75 (61–81%) on the raw tool surface and 68/75 (82–95%) on the redesign.
Go deeper: Test against production read-only, Pass rates with ranges, The two-step confirm pattern.
What a mock still can't tell you
- Fidelity is measured on the same probes the mock was built from: a regression check, not independent validation.
- Write, spend and send responses are partly inferred, because live testing stayed read-only.
- Exact token counts depend on fixtures; re-measure live before quoting a size.
- Error sizes especially: our mock's error for one bad update call was 296 tokens; the live server's was 12,643.
Back to testing MCP servers with agents.
How we measured
- Fidelity: 30 distinct live read calls and 7 observed non-mutating write-shape calls, 16 Oct 2026; 612 and 28 checks.
- Failure audit: every failed check from two earlier mock experiments, classified by reading transcripts.
- Re-run: 5 jobs × 3 apps × 5 runs × 2 surfaces; 95% Wilson ranges.
FAQ
How do I build a realistic mock of an MCP server for agent tests?
Start from a read-only capture of the live server: byte-identical tools/list and initialize, real response shapes, sizes (in tokens), enum mixes, array lengths, error texts and guidance. Fill it with synthetic look-alike data, then score the mock against live calls and fix until it passes a threshold.
What should mock fidelity checks cover?
Error class and text, response framing, keys and types, token size within about ±10%, array lengths, enum coverage, per-item size and the tool list itself. We used 612 checks over 30 live calls and a 90% pass bar.
Can a mock replace live testing?
No. It's a regression check built from the same probes, and write responses are partly inferred. Re-measure sizes and behaviours live, read-only, before quoting them.
Want a mock agents can't tell from your server?
We build a live-shaped mock of your MCP server, measure its fidelity against read-only live calls, and run agent tests on it, including write paths.