Blog / Test MCP servers with agents
Before you trust an agent eval, check the scorer: 0 of our first 151 failures were the server's fault
Our first agent experiments on a mock of a 99-tool server produced a long list of failures. It looked like a report. Then we read every failure and asked why. Not one was a real problem with the server. This is what we found, and the checks we now run before any number leaves the lab.
Where 151 failures came from
We ran early experiments on a mock of a 99-tool production MCP server: five jobs, including ones that spend credits and need the agent to ask first, across Claude Code, Codex and OpenCode. Then we classified every failed check by cause.
Plus 84 runs that failed before scoring, on infrastructure. If we had reported the raw failures, more than two thirds of them would have been our own bugs presented as findings about the server.
The real server problems came from elsewhere: live read-only runs and direct calls, such as a 12,689-token error and every refusal telling agents to ask a human. See tool results and errors for agents.
Scorer and harness bugs (63)
- One right answer, several right paths. Process checks accepted only one tool when several gave the same information. Agents that checked concurrency through a preview tool instead of the run list failed 11 checks.
- A check the prompt never asked for. One check required a specific action the prompt didn't request.
- Unrealistic timing. A check required a long-running job to finish inside the test episode. Real jobs take minutes to hours.
- A harness wrapper bug. Confirmation tokens weren't carried across calls in one wrapper, so every confirm returned a fresh token.
- A question detector that missed phrasings. In a separate experiment it didn't recognise “Confirm and I'll add all 5…?” as a question, ended conversations early and produced false failures.
Even after fixes, one over-strict check moved a later baseline from 54/75 (61–81%) to 59/75 (68–86%) when excluded. We report both.
Get agent test results that survive a second look.
We run your users' jobs through Claude Code, Codex and the other apps they use, validate every check by hand, and report only what holds up.
Mock artefacts (47)
Our first mock generated values from the schema's types only. That produced data no real account would have:
- A flag saying new searches were disabled for one group. Agents correctly refused to search: 27 failed checks for doing the right thing.
- Counts that contradicted the items: a total of 9 with 2 items returned.
- A preview that pointed to a group that didn't exist.
The agents were behaving sensibly on bad data. The scorer couldn't tell.
Agent variance (41)
These were real agent behaviour, not bugs in the server: mostly a small open model on OpenCode re-sending a confirmation itself, plus extra follow-up questions beyond what a job allowed. Agent variance is worth reporting, as pass/N per app, but it isn't a server finding unless a change to the server fixes it.
Infrastructure (84 runs)
80 runs failed because a container daemon wasn't running; 4 on network errors. None of them tested anything. Exclude them, keep them for audit, re-run them, and say so. Our later corrected re-run completed 150 of 150 runs with 0 infrastructure failures.
Check the mock against live before you trust it
We compared the mock's responses with recorded live read calls: shape, types, sizes within 10%, arrays, enums, errors and framing.
| Mock fidelity | Before | After |
|---|---|---|
| Read paths: 612 checks over 30 live calls | 48.4% | 94.8% |
| Write paths: 28 checks over 7 observed live calls (none changed anything) | 71.4% | 89.3% |
Even a good mock isn't live for everything: our mock's validation error was 296 tokens where the live one was over 12,000. Size claims about errors need live calls.
Go deeper: Report pass rates with ranges, Evaluate AI agents on your API, Mock vs live MCP testing.
Scorer checklist
- Read transcripts: at least 3 failing and 2 passing runs per job.
- Classify every failure: server, mock, scorer or harness, agent, infrastructure.
- Accept equivalent paths: if two tools give the same fact, both pass.
- Match the prompt: no check for something the user didn't ask.
- Use real timing and real data shapes; check the mock against live calls.
- Re-score everything after a fix, not just what you read.
- Exclude infrastructure failures, keep them, re-run them, disclose them.
- Disclose every scorer bug that touched a published number.
Full method: how to test an MCP server with real AI agents.
How we measured
- Experiments: early mock-lab experiments on a stand-in for a 99-tool production MCP server, Claude Code, Codex and OpenCode, 5 jobs, 16 Oct 2026. Every failed check was classified by hand into one category.
- Fidelity: mock responses compared with 30 recorded live read calls and 7 live write-path calls that changed nothing.
- Re-run: 5 jobs × 3 apps × 5 runs per variant, 150 runs, on the corrected mock with a fixed scorer.
- Limits: one server; categories are our judgement, with an example run kept for each row.
FAQ
How do I validate an agent eval scorer?
Read real transcripts: at least 3 failing and 2 passing runs per job. For each failure, decide whether it's the server, the mock, the scorer or harness, the agent, or infrastructure. Fix scorer bugs and re-score every run, not just the ones you read. Disclose any bug that affected a published number.
How often are agent eval failures false?
Often, early on. In our first mock experiments, 63 of 151 failed checks were scorer or harness bugs and 47 were mock artefacts. None of the 151 was a real problem with the server.
What is a mock artefact?
A failure caused by the test double, not the product. Our first mock generated values from types only, so one item said new searches were disabled. Agents correctly refused to search, and the scorer failed them in 27 checks.
Should failed infrastructure runs count?
No. Exclude runs that failed on infrastructure, keep them for audit, and re-run them. We lost 84 runs: 80 to a container daemon that was down, 4 to network errors.
How do I check that a mock matches the real API?
Compare the mock's responses with recorded live read calls: shape, types, sizes, enums and errors. Our read-path match went from 48.4% to 94.8% over 612 checks on 30 live calls.
Get agent test results that survive a second look.
We run your users' jobs through Claude Code, Codex and the other apps they use, validate every check by hand, and report only what holds up.