Spirit Room / Guides

How to test an MCP server with real AI agents (and why passing MCP Inspector isn't enough)

Your server passes MCP Inspector. Every tool returns the right data. Then a user asks Codex a simple question and it says there's no connector, and opens a browser instead. That happened in 6 of 40 live runs on a production server that listed all its tools correctly. Protocol tests can't see it. Here's how to test for it.

Leonid Bugaev · Published 21 Oct 2026

Why protocol tests aren't enough

Most MCP testing advice starts with MCP Inspector, then unit tests, then maybe some evals. That's the right order, and Inspector is worth running. But it answers one question: does the server speak MCP correctly? Your users ask a different one: when I ask my agent to do this job, does it get it done?

Between those two questions sits the agent app. Claude Code, Codex, Grok Build and OpenCode each decide which tools the model sees, how much of your server instructions survive, and what happens when a result is too big. We measured a 99-tool production server that returned correct data on every probe. In live, read-only runs on 8 everyday questions:

• Codex never reached the server's tools in 6 of 40 runs (7–29%). It said no connector was available and tried a browser instead. When it did find them, it answered correctly in 34 of 34.
• Claude Code found the tools every time but refused 40 results in 23 of 40 runs for being over its 25,000-token limit, then read them back from a file in pieces: 138 extra reads.
• Codex repeated identical calls in 13 of 40 runs; Claude Code in 0 of 40.

None of that is visible to a protocol check. All of it is visible to an agent run.

The five layers of MCP testing

A good public guide to the first four layers is Scalar's MCP testing guide. This guide is about the fifth.

Agent run testing, step by step

1. Write down the real jobs, in users' words

Not “call list_items with limit 10”, but “which of my projects is doing best?”. Include jobs that need several calls, jobs that shouldn't touch your server at all, and vague requests. Our live set was 8 read-only questions, from “summarise my plan and usage” to “find this person in my data”.

2. Get the expected answers from fixed calls

Run fixed tool calls yourself to get the ground truth, before and after the agent runs, and check it didn't change. Don't derive it from what the agents answered. Our first ground truth for one job was wrong: the answer lived in records hidden by a default filter. The agents found them; our own probe didn't, until we fixed it.

3. Run in the apps your users run, in their default settings

Check each app's configuration against what a normal user has, before every run. We learned this the hard way: a lab flag meant to strip Claude Code's built-in tools also removed its tool search, so all 99 tool definitions were loaded up front. The first request went from 22,791 tokens to 110,998, and a whole set of token savings measured a setup no user has. See how agent apps load MCP tools.

4. Isolate the run and guard production

Turn off the user's other MCP servers, so you measure yours. Put a read-only proxy in front of the live server: an explicit allowlist of read tools, default deny, bodies recorded, headers never. Block the shell and browser for MCP-only jobs. Without that last step, 13 of 36 runs in an early pass drove a signed-in browser through a browser skill instead of using MCP. Nothing was changed, but those runs weren't testing the server at all, and we discarded them.

5. Repeat, and report pass/N with a range

Agent runs are random, and most apps don't let you fix a seed or temperature. Run each job several times and report passes over runs with a 95% confidence range. On our 8 live jobs × 5 runs, Codex scored 34/40 (71–93%) and Claude Code 32/40 (65–90%). Those ranges overlap almost entirely, so the honest conclusion is “no evidence either is better”, with different failure modes. More in pass rates and confidence intervals for agent runs.

6. Score in code, then read the transcripts

Automated checks first, then read failing and passing transcripts by hand. Scorers have bugs too: in one experiment a question detector missed some phrasings and ended conversations early, producing false failures. After the fix and a full re-run, one server version passed those jobs in 30 of 30 runs. Fix the scorer, re-run everything, and disclose it. See validate the scorer before you trust the eval.

7. Measure cost as well as success

Record tokens per run from the app's own usage report, and separate cumulative input (what's billed) from the largest single request (how full the context window got). On the same jobs, Codex averaged about 218k cumulative input tokens per question and Claude Code about 210k; the largest request filled 18.1% of Codex's window and 4.1% of Claude's. The full metric set is in how to measure MCP server quality.

8. Test fixes with a control

Change one thing, keep a baseline, and rerun. On a mock of the same server, adding a get_instructions tool alone got it called first in 1 of 24 Claude Code runs; with a short hint it was 24 of 24 (86–100%). Without the baseline, the tool would have looked like the fix. See make the agent read your playbook first.

What agent run testing finds

Each of these points to a server-side fix: words users actually use in tool names and first lines, smaller default results, clearer status fields, and next-step hints. The fixes are in how to optimise your MCP tools.

Pitfalls that fake results

• Lab config that differs from the user's. Proxying the model endpoint turned Claude Code's tool search off: 143,357 first-request tokens instead of 22,446 with it forced back on.
• Agents escaping the guard through a shell, a browser or an installed skill. Fail those runs automatically.
• Broken mock framing. A pretty-printed JSON response over stdio makes clients cancel tools/list, and every app then looks like it loads no tools.
• Only explicit arguments tested. Agents usually call with {}; measure the default call.
• Single runs quoted as results. One Codex control run read 18,138 tokens once and 17,098 four times. Take at least two runs per size claim and report outliers.

Guides in this section

• How to measure MCP server quality: the agent scorecard

  Fifteen metrics, from discovery rate to error recovery, with targets and a worked example.

Go deeper: MCP Inspector says it works; agents still fail, Evaluate AI agents on your API, MCP testing tools compared, Pass rates and confidence intervals, Validate the scorer first, MCP server self-check.

Related: how many MCP tools is too many and which MCP channels agents actually see.

How we measured

• Live runs: a 99-tool production MCP server, 8 read-only jobs × 5 runs each in Codex CLI 0.161.0 and Claude Code 2.1.301 with tool search on, 16 Oct 2026. Every server call went through a read-only allowlist proxy; 0 write attempts in either app.
• Scoring: deterministic checks against ground truth from fixed calls made before and after the runs, then every answer read by hand. Three scorer or ground-truth fixes were applied to both apps, and three false negatives were overridden after hand review, before final scoring.
• Tokens: from each app's own usage report; result sizes in o200k_base. Claude's and Codex's tokenizers differ, so the totals compare roughly, not exactly.
• Mock runs: the get_instructions numbers come from a local mock serving the same recorded tool list, 232 runs on 18 Oct 2026.
• Limits: one server, one workspace with small data, one model per app, one day. Read-only jobs only, so nothing here tests writes or spending.