Blog / Test MCP servers with agents
“10 of 10 passed” isn't 100%: how to report agent test results with confidence ranges
You ran your agent test ten times and it passed ten times. Is it 100% reliable? No. Ten runs can't tell 100% from 75%. If you're testing an MCP server or API with real agents, the number you report should be pass/N with a range, and you should know when two results are actually different.
Why one run proves nothing
Agent apps don't let you fix the seed in headless mode, and the same prompt can take a different path each time. One run tells you what can happen. A pass rate needs repeats, and a pass rate from a few repeats needs a range, because it would move if you ran more.
We report every agent result as pass/N with a 95% Wilson range, and we never quote a single run.
What “N of N passed” really means
Bars show the range a true pass rate could be in. Ten clean runs are still consistent with a job that fails one time in four.
When two results are actually different
Overlap: no winner
On a 99-tool production MCP server, 8 read-only jobs × 5 runs each: Codex passed 34 of 40 (71–93%) and Claude Code 32 of 40 (65–90%). Codex “won” by two runs, but the ranges overlap almost entirely. That is no evidence either app is better. The useful finding was that they failed differently: Codex never found the tools in 6 runs; Claude Code found them every time but used a screen label in 5 answers.
No overlap: a real change
On a mock of the same server we added a “start here” tool. With the tool alone, Claude Code called it first in 1 of 24 runs (1–20%). With the full recipe (a hint in the server instructions, an always-loaded flag, and a first-line nudge in other tools) it did so in 24 of 24 (86–100%). The ranges don't touch, so the change is real. Codex went from 9 of 24 (21–57%) to 24 of 24. See make agents read your playbook first.
| Comparison | A | B | Verdict |
|---|---|---|---|
| Codex vs Claude Code, live | 34/40 (71–93%) | 32/40 (65–90%) | Overlap: no difference shown |
| Claude: tool only vs full recipe | 1/24 (1–20%) | 24/24 (86–100%) | No overlap: real |
| Codex: tool only vs full recipe | 9/24 (21–57%) | 24/24 (86–100%) | No overlap: real |
Get agent test results you can defend.
We run your users' jobs through Claude Code, Codex and the other apps they use, enough times to give real ranges, and read every failure by hand.
Repeats aren't fully independent
The range assumes each run is an independent try. Agents don't quite behave that way. In the 24-of-24 test, the two runs of the same prompt made identical calls in 12 of 15 pairs for Claude Code and 10 of 15 for Codex. So the ranges above are somewhat optimistic.
A stricter check counts distinct prompts instead of runs: all 12 job prompts passed for both apps, 12 of 12, 76–100%. Report both when repeats look alike, and vary the wording of your jobs rather than just re-running one prompt.
2-run cells show direction, not a rate
Early on we tried a live comparison with and without agent skills: 3 jobs × 2 conditions × 2 runs. Without skills, agents failed in 2 of 12 runs (5–45%), both on Codex. A 2-run cell like “0 of 2” is 0–66%, which says almost nothing about the rate. It did say where to look: the follow-up that tested the fix properly used 232 runs.
Use small cells to explore. Don't put them in a headline.
Go deeper: How to measure MCP server quality, Evaluate AI agents on your API, Validate your eval scorer.
How to compute the range
The Wilson score interval works with small N and with 0% or 100% results, where the textbook ± formula gives nonsense like 100% ± 0.
import math
def wilson(k, n, z=1.96):
p = k / n
d = 1 + z*z/n
c = p + z*z/(2*n)
r = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n))
return (c - r)/d, (c + r)/d
wilson(10, 10) # (0.72, 1.0)- Fix the jobs and the scorer first, and check the scorer by hand; a wrong scorer makes any range meaningless.
- Pick N for the claim you want. 2–5 runs to explore, 20–40 per cell to compare.
- Report pass/N, the range, and N for every cell, plus how many distinct prompts.
- Call a difference real only if ranges don't overlap or a stated test passes.
The full method: how to test an MCP server with real AI agents.
Where these numbers come from
- Live: a 99-tool production MCP server, read-only, Codex CLI 0.161.0 and Claude Code 2.1.301, 8 jobs × 5 runs each, 16 Oct 2026.
- Mock: a stand-in with the same recorded tool list, 232 runs, Claude Code and Codex, 18 Oct 2026.
- Small live trial: 24 runs, with and without skills, Claude Code and Codex, 16 Oct 2026.
- All ranges are 95% Wilson intervals.
FAQ
How many runs do I need to test an AI agent?
It depends on how precise you need to be. All-pass results give these 95% ranges: 2 of 2 is 34–100%, 5 of 5 is 57–100%, 10 of 10 is 72–100%, 24 of 24 is 86–100%, 40 of 40 is 91–100%. Use 2–5 runs to explore and 20–40 per cell to make a claim.
What is a Wilson confidence interval?
A range for a pass rate that behaves well with small samples and with 0% or 100% results, unlike the simple ± formula. For 10 of 10 passes it gives 72–100%.
How do I know if one agent or change is really better?
If the two ranges don't overlap, the difference is real at roughly the 95% level. If they overlap, run more or use a test such as Fisher's exact test. Codex 34/40 and Claude Code 32/40 overlap almost entirely.
Why not just report the pass percentage?
Because 100% from 10 runs and 100% from 40 runs mean different things. The range shows how much the result could move with more runs.
Are repeated agent runs independent?
Not fully. Agents often repeat themselves exactly: in one test, two runs of the same prompt made identical calls in 12 of 15 pairs. Count distinct prompts as a stricter check.
Get agent test results you can defend.
We run your users' jobs through Claude Code, Codex and the other apps they use, enough times to give real ranges, and read every failure by hand.