Blog / Test MCP servers with agents
MCP Inspector says your server works. Agents still fail. Here's what it can't catch
MCP Inspector is the tool everyone recommends for testing an MCP server, and it's good at its job. But you choose the tool and the arguments, so it can't tell you what an agent will do. We ran a production server that answered every direct call correctly through Claude Code and Codex. Here is what broke anyway.
What MCP Inspector checks well
MCP Inspector is the official tool for testing and debugging MCP servers. It comes as a web app, a CLI and a terminal UI. You connect, list tools, resources and prompts, call a tool with arguments you type, and read the raw request and response. Its CLI can run smoke tests in CI.
That is exactly what you want when the server doesn't connect, a schema is broken, auth fails, or a tool returns the wrong thing. Keep using it.
What it can't see
In Inspector, you are the agent. You already know which tool to call and what to pass. A real agent doesn't. It has to find your tools among others, pick one from a name and a description, build the arguments, and fit the result into its context. None of that happens in Inspector.
The highlighted steps are where our agent runs failed. The server step never did.
Five failures from a server that passed
We tested a 99-tool production MCP server, read-only, with 8 real jobs × 5 runs in Codex and in Claude Code. Before and after the runs we called the tools directly to get the right answers. Those direct calls matched every time, and the server returned no errors in the Codex round. Results:
1. The agent never found the tools
The server listed all 99 tools. In 6 of 40 Codex runs, its tool search never surfaced them. Codex said no connector was available and tried the browser. When Codex did find the tools, it answered correctly in 34 of 34 runs. Inspector lists your tools perfectly, because it doesn't search. See how Codex finds MCP tools.
2. The client refused a correct result
The default list call returned more than 25,000 tokens. Claude Code refused 40 results in 23 of 40 runs as “exceeds maximum allowed tokens”, saved them to files, and read them back in pieces with 138 extra local reads. In Inspector the same result just shows up. See how to reduce MCP tool response size.
3. The user's word wasn't the API's word
The app's screen showed a tab label that isn't an API value. Claude Code used the screen label in 5 answers (strict score 32/40; 37/40 if you accept it). In 3 of 5 runs of one job, it said the server didn't expose the data, which was wrong: the list tool returns it per item. Codex hit the same ambiguity in an earlier pass. You'd never see this in Inspector, because you'd type the right value. See why agents call the wrong tool.
Your server passes Inspector. Find out what agents do with it.
We run your users' real jobs through Claude Code, Codex and the other apps they use, record every call, and show where they fail and why.
4. The same call, twice
Codex repeated a call with the same or near-same arguments in 13 of 40 runs, mostly the big list call. Each repeat cost tens of thousands of tokens. One direct call in Inspector can't show a pattern that only appears across a multi-step job.
5. Correct, but expensive
A simple read question cost Codex 114k–301k input tokens on average per job, summed over its requests. A list call returned up to about 57k tokens. Inspector shows you the result; it doesn't show what that result costs inside a job.
Go deeper: How to measure MCP server quality, MCP output limits by agent, MCP testing tools compared.
Inspector vs agent runs
| Question | MCP Inspector | Agent runs |
|---|---|---|
| Does the server connect and speak MCP? | Yes | Indirectly |
| Are schemas and auth right? | Yes | Indirectly |
| Does a tool return the right data? | Yes, for the args you type | Yes, for the args the agent builds |
| Does the agent find the tools? | No | Yes (6/40 missed) |
| Does it pick the right tool and value? | No | Yes |
| Does the client accept the result size? | No | Yes (40 refusals) |
| How often does the job pass? | No | pass/N with a range |
| What does a job cost in tokens? | No | Yes |
How to add agent runs on top of Inspector
- Write 5–10 real jobs in your users' words, each with a checkable answer.
- Get the right answers by calling tools directly (Inspector's CLI works for this), right before and after the runs.
- Run each job several times in the apps your users use, such as Claude Code and Codex, with writes blocked.
- Record every call and score each answer; read the failures by hand.
- Report pass/N with a range, plus tokens per job, refusals and repeats.
The full method: how to test an MCP server with real AI agents.
How we measured
- Server: a 99-tool production MCP server, read-only through an allowlist proxy that blocks writes (0 write attempts), 16 Oct 2026.
- Apps: Codex CLI 0.161.0 and Claude Code 2.1.301 with tool search on, each 8 jobs × 5 runs. Right answers from direct calls before and after; identical both times.
- Scoring: one fixed scorer for both apps; failed answers read by hand. Ranges are 95% Wilson intervals.
- Limits: one server, one small workspace, one model per app. The 3 of 5 result comes from one 5-run job. Tool and label names are simplified.
FAQ
Is MCP Inspector enough to test an MCP server?
For the protocol, mostly yes: connection, transports, auth, schemas, and manual tool calls. For agent use, no. In Inspector you choose the tool and the arguments, so it can't show whether a model finds the tool, picks the right one, sends valid arguments, or copes with a large result.
What can MCP Inspector not test?
Tool discovery inside an agent app, tool choice, argument building, result size limits in the client, multi-step jobs, and how often each of those fails. On a 99-tool server whose direct calls were all correct, Codex still never found the tools in 6 of 40 runs.
Should I stop using MCP Inspector?
No. It is the right tool for debugging the protocol and for smoke tests in CI with its CLI. Treat agent runs as the next layer, not a replacement.
How many agent runs do I need?
Enough to report pass/N with a range. Five runs per job show what happens; 40 runs per app narrow the range. 6 of 40 is 7–29%, still wide, which is why we always print the range.
Does a correct direct call mean the agent will get the right answer?
No. In our runs the data was right every time, but agents missed tools, hit size limits, called the same thing twice, and misread a screen label as an API value.
Your server passes Inspector. Find out what agents do with it.
We run your users' real jobs through Claude Code, Codex and the other apps they use, record every call, and show where they fail and why.