--- title: Agent Pass Rates: Why 10/10 Isn't 100% (Use Ranges) description: 10 of 10 passing agent runs means 72–100%, not 100%. How to report MCP and agent test results as pass/N with Wilson ranges, with real examples from our runs. canonical: https://spiritroom.app/blog/agent-pass-rates-confidence-intervals/ published: 2026-10-21 updated: 2026-10-21 --- > Markdown version of https://spiritroom.app/blog/agent-pass-rates-confidence-intervals/ for agents. Index: /llms.txt [Blog](/blog/) / [Test MCP servers with agents](/guides/test-mcp-servers-with-agents/) # “10 of 10 passed” isn't 100%: how to report agent test results with confidence ranges You ran your agent test ten times and it passed ten times. Is it 100% reliable? No. Ten runs can't tell 100% from 75%. If you're testing an MCP server or API with real agents, the number you report should be pass/N with a range, and you should know when two results are actually different. Leonid Bugaev · Published 21 Oct 2026 **TL;DR** - **Report pass/N with a range.** 10 of 10 is 72–100%; 24 of 24 is 86–100%; 2 of 2 is 34–100% (95% Wilson). - **Overlap means no winner.** Codex 34/40 (71–93%) vs Claude Code 32/40 (65–90%) on the same server: no evidence either is better. - **No overlap means a real change.** A start-here tool alone: 1 of 24 (1–20%). With the full recipe: 24 of 24 (86–100%). - **Repeats aren't fully independent.** Same-prompt runs made identical calls in 12 of 15 pairs, so ranges are a bit optimistic. - **2 runs show direction, not a rate.** Use them to choose what to test properly. [Why one run proves nothing](#why)[What N of N really means](#table)[When two results are different](#overlap)[Repeats aren't independent](#independent)[2-run cells: direction only](#small)[How to compute it](#how)[Where these numbers come from](#how-we-measured)[FAQ](#faq) ## Why one run proves nothing Agent apps don't let you fix the seed in headless mode, and the same prompt can take a different path each time. One run tells you what *can* happen. A pass rate needs repeats, and a pass rate from a few repeats needs a range, because it would move if you ran more. We report every agent result as **pass/N with a 95% Wilson range**, and we never quote a single run. ## What “N of N passed” really means 2 of 2 *34–100%* 5 of 5 *57–100%* 10 of 10 *72–100%* 24 of 24 *86–100%* 40 of 40 *91–100%* Bars show the range a true pass rate could be in. Ten clean runs are still consistent with a job that fails one time in four. ## When two results are actually different ### Overlap: no winner On a 99-tool production MCP server, 8 read-only jobs × 5 runs each: Codex passed **34 of 40** (71–93%) and Claude Code **32 of 40** (65–90%). Codex “won” by two runs, but the ranges overlap almost entirely. That is no evidence either app is better. The useful finding was that they failed differently: Codex never found the tools in 6 runs; Claude Code found them every time but used a screen label in 5 answers. ### No overlap: a real change On a mock of the same server we added a “start here” tool. With the tool alone, Claude Code called it first in **1 of 24** runs (1–20%). With the full recipe (a hint in the server instructions, an always-loaded flag, and a first-line nudge in other tools) it did so in **24 of 24** (86–100%). The ranges don't touch, so the change is real. Codex went from 9 of 24 (21–57%) to 24 of 24. See [make agents read your playbook first](/blog/get-instructions-first-24-of-24/). | Comparison | A | B | Verdict | | --- | --- | --- | --- | | Codex vs Claude Code, live | 34/40 (71–93%) | 32/40 (65–90%) | Overlap: no difference shown | | Claude: tool only vs full recipe | 1/24 (1–20%) | 24/24 (86–100%) | No overlap: real | | Codex: tool only vs full recipe | 9/24 (21–57%) | 24/24 (86–100%) | No overlap: real | **Get agent test results you can defend.** We run your users' jobs through Claude Code, Codex and the other apps they use, enough times to give real ranges, and read every failure by hand. [Test your MCP →](/#contact) ## Repeats aren't fully independent The range assumes each run is an independent try. Agents don't quite behave that way. In the 24-of-24 test, the two runs of the same prompt made **identical calls in 12 of 15 pairs** for Claude Code and 10 of 15 for Codex. So the ranges above are somewhat optimistic. A stricter check counts distinct prompts instead of runs: all 12 job prompts passed for both apps, **12 of 12, 76–100%**. Report both when repeats look alike, and vary the wording of your jobs rather than just re-running one prompt. ## 2-run cells show direction, not a rate Early on we tried a live comparison with and without agent skills: 3 jobs × 2 conditions × 2 runs. Without skills, agents failed in 2 of 12 runs (5–45%), both on Codex. A 2-run cell like “0 of 2” is 0–66%, which says almost nothing about the rate. It did say *where to look*: the follow-up that tested the fix properly used 232 runs. Use small cells to explore. Don't put them in a headline. **Go deeper:** [How to measure MCP server quality](/blog/measure-mcp-server-quality/), [Evaluate AI agents on your API](/blog/evaluate-ai-agents-on-your-api/), [Validate your eval scorer](/blog/validate-your-agent-eval-scorer/). ## How to compute the range The Wilson score interval works with small N and with 0% or 100% results, where the textbook ± formula gives nonsense like 100% ± 0. ``` import math def wilson(k, n, z=1.96): p = k / n d = 1 + z*z/n c = p + z*z/(2*n) r = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) return (c - r)/d, (c + r)/d wilson(10, 10) # (0.72, 1.0) ``` 1. **Fix the jobs and the scorer first**, and check the scorer by hand; a wrong scorer makes any range meaningless. 2. **Pick N for the claim you want.** 2–5 runs to explore, 20–40 per cell to compare. 3. **Report pass/N, the range, and N** for every cell, plus how many distinct prompts. 4. **Call a difference real only if ranges don't overlap** or a stated test passes. The full method: [how to test an MCP server with real AI agents](/guides/test-mcp-servers-with-agents/). ## Where these numbers come from - **Live:** a 99-tool production MCP server, read-only, Codex CLI 0.161.0 and Claude Code 2.1.301, 8 jobs × 5 runs each, 16 Oct 2026. - **Mock:** a stand-in with the same recorded tool list, 232 runs, Claude Code and Codex, 18 Oct 2026. - **Small live trial:** 24 runs, with and without skills, Claude Code and Codex, 16 Oct 2026. - All ranges are 95% Wilson intervals. ## FAQ ### How many runs do I need to test an AI agent? It depends on how precise you need to be. All-pass results give these 95% ranges: 2 of 2 is 34–100%, 5 of 5 is 57–100%, 10 of 10 is 72–100%, 24 of 24 is 86–100%, 40 of 40 is 91–100%. Use 2–5 runs to explore and 20–40 per cell to make a claim. ### What is a Wilson confidence interval? A range for a pass rate that behaves well with small samples and with 0% or 100% results, unlike the simple ± formula. For 10 of 10 passes it gives 72–100%. ### How do I know if one agent or change is really better? If the two ranges don't overlap, the difference is real at roughly the 95% level. If they overlap, run more or use a test such as Fisher's exact test. Codex 34/40 and Claude Code 32/40 overlap almost entirely. ### Why not just report the pass percentage? Because 100% from 10 runs and 100% from 40 runs mean different things. The range shows how much the result could move with more runs. ### Are repeated agent runs independent? Not fully. Agents often repeat themselves exactly: in one test, two runs of the same prompt made identical calls in 12 of 15 pairs. Count distinct prompts as a stricter check. **Get agent test results you can defend.** We run your users' jobs through Claude Code, Codex and the other apps they use, enough times to give real ranges, and read every failure by hand. [Test your MCP →](/#contact) ## Keep reading - Up[How to test an MCP server with real AI agents (guide)](/guides/test-mcp-servers-with-agents/) - Related[How to evaluate AI agents on your API: real jobs, ground truth and pass/N](/blog/evaluate-ai-agents-on-your-api/) - Related[Validate your agent eval scorer: 0 of our first 151 failures were the server's fault](/blog/validate-your-agent-eval-scorer/) - Related[Make the agent read your playbook first: 24 of 24 runs in Claude Code and Codex](/blog/get-instructions-first-24-of-24/)