---
title: How to Test an MCP Server With Real AI Agents (2026)
description: Your MCP server passes Inspector, yet agents fail. How to test it in Claude Code and Codex on real jobs, with ground truth, pass/N and read-only safety.
canonical: https://spiritroom.app/guides/test-mcp-servers-with-agents/
published: 2026-10-21
updated: 2026-10-21
---

> Markdown version of https://spiritroom.app/guides/test-mcp-servers-with-agents/ for agents. Index: /llms.txt

[Spirit Room](/) / [Guides](/guides/agent-ready-tools/)

# How to test an MCP server with real AI agents (and why passing MCP Inspector isn't enough)

Your server passes MCP Inspector. Every tool returns the right data. Then a user asks Codex a simple question and it says there's no connector, and opens a browser instead. That happened in 6 of 40 live runs on a production server that listed all its tools correctly. Protocol tests can't see it. Here's how to test for it.

Leonid Bugaev · Published 21 Oct 2026

**TL;DR**

- **Protocol tests prove your server responds. They don't prove agents can use it.** On a 99-tool production server that answered every probe correctly, Codex never reached the tools in 6 of 40 live runs (7–29%).
- **Test in the agent apps your users run, in their default settings.** Apps load tools, instructions and results differently; one lab flag silently turned Claude Code's tool search off and multiplied the first request about five times.
- **Score against ground truth from fixed calls**, not against what looks plausible, and read transcripts by hand: our own scorer produced false failures until we fixed it.
- **Report pass/N with a confidence range.** Codex 34/40 (71–93%) and Claude Code 32/40 (65–90%) on the same 8 jobs overlap, so neither is shown to be better.
- **Test production read-only.** An allowlist proxy kept 80 live runs at 0 writes and 0 spend, and caught agents leaving MCP for the browser.
[Why protocol tests aren't enough](#why)[The five layers of MCP testing](#layers)[Agent run testing, step by step](#method)[What it finds](#found)[Pitfalls that fake results](#pitfalls)[Guides in this section](#guides)[How we measured](#how-we-measured)[FAQ](#faq)

## Why protocol tests aren't enough

Most MCP testing advice starts with MCP Inspector, then unit tests, then maybe some evals. That's the right order, and Inspector is worth running. But it answers one question: does the server speak MCP correctly? Your users ask a different one: when I ask my agent to do this job, does it get it done?

Between those two questions sits the agent app. Claude Code, Codex, Grok Build and OpenCode each decide which tools the model sees, how much of your server instructions survive, and what happens when a result is too big. We measured a 99-tool production server that returned correct data on every probe. In live, read-only runs on 8 everyday questions:

- Codex never reached the server's tools in **6 of 40 runs** (7–29%). It said no connector was available and tried a browser instead. When it did find them, it answered correctly in 34 of 34.
- Claude Code found the tools every time but **refused 40 results in 23 of 40 runs** for being over its 25,000-token limit, then read them back from a file in pieces: 138 extra reads.
- Codex repeated identical calls in 13 of 40 runs; Claude Code in 0 of 40.

None of that is visible to a protocol check. All of it is visible to an agent run.

## The five layers of MCP testing

What each kind of test catches

| Layer | Question it answers | Typical tool | Misses |
| --- | --- | --- | --- |
| Protocol | Does the server start, list tools and return valid messages? | MCP Inspector | Everything about agent behaviour |
| Tool logic | Does each tool do the right thing for valid and invalid input? | Unit tests with an in-memory client | Whether anyone calls it |
| Definitions | Did names, descriptions or schemas change, and how big are they? | Snapshot of `tools/list`, token counts | How each app loads them |
| Model evals | Does a model pick the right tool and arguments? | An SDK plus a model, scored in code | App behaviour: tool search, instruction cutoffs, result limits |
| **Agent run testing** | **Does the job get done in the apps users run?** | Real apps, real jobs, ground truth, repeats | Cost and time, so run it before releases and after description changes |

A good public guide to the first four layers is [Scalar's MCP testing guide](https://scalar.com/learn/mcp/test-mcp-servers). This guide is about the fifth.

**Get your MCP server tested by real agents.**

We run your users' real jobs through Claude Code, Codex and the other apps they use, read-only on production, and report pass rates with confidence ranges, failures and the fixes that move them.

[Test your MCP →](/#contact)

## Agent run testing, step by step

Real jobs**→**Ground truth**→**Agent apps, default settings**→**Read-only guard**→**Repeat runs**→**Score + read transcripts**→**pass/N with range

### 1. Write down the real jobs, in users' words

Not “call list\_items with limit 10”, but “which of my projects is doing best?”. Include jobs that need several calls, jobs that shouldn't touch your server at all, and vague requests. Our live set was 8 read-only questions, from “summarise my plan and usage” to “find this person in my data”.

### 2. Get the expected answers from fixed calls

Run fixed tool calls yourself to get the ground truth, before and after the agent runs, and check it didn't change. Don't derive it from what the agents answered. Our first ground truth for one job was wrong: the answer lived in records hidden by a default filter. The agents found them; our own probe didn't, until we fixed it.

### 3. Run in the apps your users run, in their default settings

Check each app's configuration against what a normal user has, before every run. We learned this the hard way: a lab flag meant to strip Claude Code's built-in tools also removed its tool search, so all 99 tool definitions were loaded up front. The first request went from 22,791 tokens to 110,998, and a whole set of token savings measured a setup no user has. See [how agent apps load MCP tools](/guides/how-agents-load-mcp-tools/).

### 4. Isolate the run and guard production

Turn off the user's other MCP servers, so you measure yours. Put a read-only proxy in front of the live server: an explicit allowlist of read tools, default deny, bodies recorded, headers never. Block the shell and browser for MCP-only jobs. Without that last step, 13 of 36 runs in an early pass drove a signed-in browser through a browser skill instead of using MCP. Nothing was changed, but those runs weren't testing the server at all, and we discarded them.

### 5. Repeat, and report pass/N with a range

Agent runs are random, and most apps don't let you fix a seed or temperature. Run each job several times and report passes over runs with a 95% confidence range. On our 8 live jobs × 5 runs, Codex scored 34/40 (71–93%) and Claude Code 32/40 (65–90%). Those ranges overlap almost entirely, so the honest conclusion is “no evidence either is better”, with different failure modes. More in [pass rates and confidence intervals for agent runs](/blog/agent-pass-rates-confidence-intervals/).

### 6. Score in code, then read the transcripts

Automated checks first, then read failing and passing transcripts by hand. Scorers have bugs too: in one experiment a question detector missed some phrasings and ended conversations early, producing false failures. After the fix and a full re-run, one server version passed those jobs in 30 of 30 runs. Fix the scorer, re-run everything, and disclose it. See [validate the scorer before you trust the eval](/blog/validate-your-agent-eval-scorer/).

### 7. Measure cost as well as success

Record tokens per run from the app's own usage report, and separate cumulative input (what's billed) from the largest single request (how full the context window got). On the same jobs, Codex averaged about 218k cumulative input tokens per question and Claude Code about 210k; the largest request filled 18.1% of Codex's window and 4.1% of Claude's. The full metric set is in [how to measure MCP server quality](/blog/measure-mcp-server-quality/).

### 8. Test fixes with a control

Change one thing, keep a baseline, and rerun. On a mock of the same server, adding a `get_instructions` tool alone got it called first in 1 of 24 Claude Code runs; with a short hint it was 24 of 24 (86–100%). Without the baseline, the tool would have looked like the fix. See [make the agent read your playbook first](/blog/get-instructions-first-24-of-24/).

## What agent run testing finds

Failures seen in live read-only runs on one production server, 16 Oct 2026

| Failure | Where | Would a protocol test catch it? |
| --- | --- | --- |
| Agent never finds the tools and leaves for the browser | Codex, 6 of 40 runs | No: the server listed all 99 tools |
| Results refused as too large, re-read from a file in pieces | Claude Code, 40 results in 23 of 40 runs | No: the results were valid |
| The same call repeated | Codex, 13 of 40 runs | No |
| A screen label used instead of the API's status value | Claude Code, 5 answers | No: both values were correct data |

Each of these points to a server-side fix: words users actually use in tool names and first lines, smaller default results, clearer status fields, and next-step hints. The fixes are in [how to optimise your MCP tools](/blog/optimise-mcp-tools-for-agents/).

## Pitfalls that fake results

- **Lab config that differs from the user's.** Proxying the model endpoint turned Claude Code's tool search off: 143,357 first-request tokens instead of 22,446 with it forced back on.
- **Agents escaping the guard** through a shell, a browser or an installed skill. Fail those runs automatically.
- **Broken mock framing.** A pretty-printed JSON response over stdio makes clients cancel `tools/list`, and every app then looks like it loads no tools.
- **Only explicit arguments tested.** Agents usually call with `{}`; measure the default call.
- **Single runs quoted as results.** One Codex control run read 18,138 tokens once and 17,098 four times. Take at least two runs per size claim and report outliers.

## Guides in this section

- [How to measure MCP server quality: the agent scorecard](/blog/measure-mcp-server-quality/)

  Fifteen metrics, from discovery rate to error recovery, with targets and a worked example.

**Go deeper:** [MCP Inspector says it works; agents still fail](/blog/mcp-inspector-vs-agent-testing/), [Evaluate AI agents on your API](/blog/evaluate-ai-agents-on-your-api/), [MCP testing tools compared](/blog/mcp-testing-tools-compared/), [Pass rates and confidence intervals](/blog/agent-pass-rates-confidence-intervals/), [Test agents against production read-only](/blog/read-only-agent-testing-production/), [Agents escape to the browser](/blog/agents-escape-to-the-browser/), [Validate the scorer first](/blog/validate-your-agent-eval-scorer/), [See what your agent sends to the model](/blog/capture-agent-model-traffic/), [MCP server self-check](/tools/mcp-scorecard/).

Related: [how many MCP tools is too many](/blog/how-many-mcp-tools-is-too-many/) and [which MCP channels agents actually see](/blog/mcp-prompts-resources-tools-which-agents-use/).

## How we measured

- **Live runs:** a 99-tool production MCP server, 8 read-only jobs × 5 runs each in Codex CLI 0.161.0 and Claude Code 2.1.301 with tool search on, 16 Oct 2026. Every server call went through a read-only allowlist proxy; 0 write attempts in either app.
- **Scoring:** deterministic checks against ground truth from fixed calls made before and after the runs, then every answer read by hand. Three scorer or ground-truth fixes were applied to both apps, and three false negatives were overridden after hand review, before final scoring.
- **Tokens:** from each app's own usage report; result sizes in `o200k_base`. Claude's and Codex's tokenizers differ, so the totals compare roughly, not exactly.
- **Mock runs:** the `get_instructions` numbers come from a local mock serving the same recorded tool list, 232 runs on 18 Oct 2026.
- **Limits:** one server, one workspace with small data, one model per app, one day. Read-only jobs only, so nothing here tests writes or spending.

## FAQ

### How do I test my MCP server with AI agents?

Write the real jobs your users need done, in their words, with expected answers taken from fixed tool calls. Run each job several times in the agent apps your users use, such as Claude Code and Codex, in default settings. Record every tool call and result, score answers against the expected ones, read the transcripts, and report pass/N with a confidence range.

### Is MCP Inspector enough to test an MCP server?

No. MCP Inspector checks the protocol: the server starts, lists tools and returns valid responses. It can't show whether an agent finds the right tool, reads the result, or recovers from an error. A production server that passed those checks still saw Codex miss its tools in 6 of 40 live runs.

### How many runs do I need to trust an agent test?

More than one, and report the range. 24 of 24 passes gives a 95% range of 86–100%; 12 of 12 gives 76–100%. Two runs per setup only show direction. Treat a difference as real only when the ranges don't overlap.

### How do I test an MCP server against production safely?

Put a read-only proxy between the agent and the server that forwards only an explicit allowlist of read tools and blocks everything else, record bodies but never headers, and block the agent's shell and browser for MCP-only jobs. In our live runs this kept writes and spend at zero.

### What is agent run testing?

Agent run testing means running real user jobs through the actual agent apps your users run, with your server connected, and scoring the outcome against known answers, repeated enough times to give a pass rate with a confidence range. It sits above unit tests and model-only evals.

### Should I test with an SDK and a model, or with Claude Code and Codex?

Both have a place. A model-plus-SDK eval is cheap and good for comparing descriptions. But the apps decide what the model sees: which tools are loaded, how instructions are cut, how big results are handled. Failures like missed tools and refused results showed up only in the apps.

**Get your MCP server tested by real agents.**

We run your users' real jobs through Claude Code, Codex and the other apps they use, read-only on production, and report pass rates with confidence ranges, failures and the fixes that move them.

[Test your MCP →](/#contact)

## Keep reading

- Up[How to make your MCP server or API agent-ready, with proof (guide hub)](/guides/agent-ready-tools/)
- Related[How to measure MCP server quality: the agent scorecard (2026)](/blog/measure-mcp-server-quality/)
- Related[MCP Inspector says your server works. Agents still fail. Here's what it can't catch](/blog/mcp-inspector-vs-agent-testing/)
- Related[How to evaluate AI agents on your API: real jobs, ground truth and pass/N](/blog/evaluate-ai-agents-on-your-api/)
- Related[MCP benchmark and testing tools compared: load testing vs agent-quality testing (2026)](/blog/mcp-testing-tools-compared/)
- Related[How to test agents against a production MCP server without spending, writing or sending anything](/blog/read-only-agent-testing-production/)
- In this guide[“10 of 10 passed” isn't 100%: reporting agent test results with confidence ranges](/blog/agent-pass-rates-confidence-intervals/)
- In this guide[Your agent test may not be testing your MCP: 13 of 36 runs drove the browser instead](/blog/agents-escape-to-the-browser/)
- In this guide[Is your MCP mock lying to you? Raising mock fidelity from 48% to 95%](/blog/mcp-mock-fidelity/)
- In this guide[Validate your agent eval scorer: 0 of our first 151 failures were the server's fault](/blog/validate-your-agent-eval-scorer/)
- In this guide[See exactly what your agent sends to the model: wire-capturing Claude Code, Codex and Grok](/blog/capture-agent-model-traffic/)
- In this guide[Free MCP server scorecard: paste your tools/list, get size, annotation and description checks](/tools/mcp-scorecard/)