Blog / Test MCP servers with agents
How to measure MCP server quality: the agent scorecard (2026)
A 99-tool production MCP server passed every direct probe. Then real agents used it: Codex missed its tools in 8 of 80 runs, Claude Code refused its biggest results 40 times, and one typo cost 12,689 tokens. A quality score that only checks the spec would have rated it fine.
Most “MCP quality” checks score the server on its own: are the descriptions filled in, do the schemas validate, does the protocol behave. That's useful, and it's where we start too. But users don't meet your server on its own. They meet it through Claude Code, Codex, Cursor, OpenCode or Grok Build, and each of those apps decides what the model sees, how much of a result gets through and how tools are found. Quality is what survives that trip.
Why spec scores aren't enough
Here is what happened on a 99-tool production MCP server we tested read-only on 16 October 2026. Every direct probe returned correct data. A spec-level score would have looked healthy. Then we ran eight everyday read questions through real agents, 40 to 80 runs per app:
- Discovery: Codex never reached the server's tools in 8 of 80 runs. It said no connection was available and tried a browser instead.
- Result size: the default list call returned 46,253 tokens. Claude Code refused 40 of those results, in 23 of 40 runs, because they exceeded its 25,000-token limit, then made 138 extra file reads to recover.
- Errors: one malformed ID returned 12,689 tokens of schema, and the error told the agent to stop and ask a human.
- Vocabulary: a count labelled “To review” mixed two stages. On a per-stage question, agents in all three apps reported that mixed number in some runs.
Most answers were still right: 74 of 80 in Claude Code, 69 of 80 in Codex, 39 of 40 in OpenCode. That's exactly why you need a scorecard with more than one number. A pass rate alone says “fine”. The other metrics say what it costs and where it will break.
The agent scorecard: 15 metrics in three layers
Each metric has a definition, how to measure it, our starting target and the value we measured on the example server. The targets are design budgets from our runs, not standards. Tune them to your jobs.
Layer 1: server checks (no model needed)
These are deterministic. Run them on every release, in CI if you can. They predict cost and failure, but they don't measure it.
| # | Metric | How to measure | Starting target | Example server |
|---|---|---|---|---|
| 1 | Tool-list size | Tokens of tools/list as compact JSON | Know it; watch the heaviest tools | 68,286 tokens for 99 tools; two tools are about a third of it |
| 2 | Instruction length | Characters of the server instructions | Under ~2,000, key rule first | 5,092 characters |
| 3 | Default response size | Tokens returned by each read tool called with {} | ~5,000 per call as a starting budget | 46,253 for the main list |
| 4 | Duplicate payload | Read tools whose text content repeats the structured content | Know which apps read which half | 27 of 34 read tools |
| 5 | Error size | Tokens returned for a fixed set of invalid calls | A few hundred tokens | 12,689 on one tool; 110–150 on others |
| 6 | Error guidance | Does each error say retry, fix the arguments, or ask the user? | “Ask the user” only for real decisions | All 3 validation errors said a human must decide |
| 7 | Annotation accuracy | Compare readOnlyHint, destructiveHint, idempotentHint with what each tool really does | Every spend or start marked destructive, not idempotent | 2 credit-spending tools marked non-destructive; 1 launch tool marked idempotent |
| 8 | Self-noise | Share of a feed or log made of the agent's own calls | Excluded by default | 152 of 200 activity events |
Layer 2: agent behaviour, per agent app
These need real runs. Report each one per app and version, as pass/N with a 95% range.
| # | Metric | How to measure | Starting target | Example server |
|---|---|---|---|---|
| 9 | Up-front cost | First-request tokens with and without your server, “say hi” prompt | Know it per app | Codex +0; Claude Code +1,715; OpenCode 1 +73,415 (each app's own count) |
| 10 | Discovery rate | Runs with at least one call to your server ÷ runs | Every run, every app | Codex 72/80 (81–95%); Claude Code 40/40 (91–100%) |
| 11 | Right first call | Runs whose first call to your server is the intended one | Every run | Mock: 24/24 (86–100%) with a start-here hint, 1/24 and 9/24 without |
| 12 | Result usability | Runs where the app refused, truncated or spilled a result to a file | None on the main jobs | Claude Code 23/40 runs refused; OpenCode 24/40 cut off |
| 13 | Repeat calls | Runs with an identical call made twice | Rare, and explained | Codex 13/40 (20–48%); Claude Code 0/40 (0–9%) |
Get your MCP server scored on all 15 metrics.
We run the jobs your users need done in Claude Code, Codex and the other apps they use, fill in this scorecard with your numbers, and show which fixes move them.
Layer 3: outcome and cost
| # | Metric | How to measure | Starting target | Example server |
|---|---|---|---|---|
| 14 | Task pass rate | Answers that match ground truth from fixed probes, checked by script and by hand | Set per job; ranges that don't overlap your last release's | Claude Code 74/80 (85–97%); Codex 69/80 (77–92%); OpenCode 39/40 (87–100%) |
| 15 | Tokens per task | Input tokens summed over the run's model requests (the app's own usage) | Falling release over release | About 222k (Claude Code), 221k (Codex), 342k (OpenCode) per read question |
Token totals come from each app's own usage figures, counted with different tokenizers and mostly served from cache, so compare them within an app across releases, not across apps. If you need one number for the business case, divide tokens (or cost) by the number of passing runs: that's cost per successful task.
How to read the scorecard
Read it from the bottom up. Layer 3 tells you whether users get answers and what that costs. Layer 2 tells you why, per app. Layer 1 points at the fix. On the example server:
- Pass rates were high, but tokens per task were large and two apps had to work around oversized results (metrics 12, 15). Cause: the default response size (3) and duplicate payload (4).
- The Codex misses (10) traced to discovery through search over a large tool list (1), not to wrong data.
- Wrong per-stage counts traced to a label that didn't match the API's vocabulary, which no static metric catches. That's why the hand-read step matters.
How to run the scorecard on your server
- Freeze the server checks. Script metrics 1–8 with fixed inputs, including the default call (
{}) for every read tool and a fixed set of invalid calls. Count tokens with a named tokenizer (we useo200k_base). - Write 8–20 real jobs. Requests your users actually make, some that don't name your product, each with an expected answer taken from fixed probes of the same data.
- Pick the apps and pin versions. Use default settings, as a normal user would have them. A proxy or a lab flag can silently change how tools load, so check the first request.
- Run each job at least 5 times per app. Record every call, result and the final answer. Keep the runs read-only if you test against production.
- Score, then read by hand. Automate the comparison against ground truth, then read every failure and a few passes. We have overturned scorer results this way more than once.
- Report pass/N with a 95% range, per app. Change one thing, rerun, and keep regressions as test cases.
The whole method is in how to test an MCP server with real AI agents. For why the range matters, see pass rates and confidence intervals for agent runs.
How we measured
- Server: a 99-tool production MCP server, one workspace, tested read-only on 16 October 2026. Layer 1 numbers come from direct tool calls with fixed inputs, repeated 3 times with identical results.
- Agents: Codex CLI 0.161.0 (gpt-6-astra) and Claude Code 2.1.301 (claude-opus-5-5), 8 read-only jobs × 5 runs in two rounds (80 runs each); OpenCode 1.18.34 (GLM-5.3-Flash), one round of 40. Every call went through a proxy that only forwarded read tools; there were 0 write attempts.
- Scoring: answers compared with ground truth from fixed probes taken immediately before and after the runs; every failed answer read by hand. Discovery and repeat-call figures for Claude Code are from the first round.
- Up-front cost: first request captured before any model ran, with and without the 99 tools; single runs, each app's own token count.
- Right first call (metric 11): a mock serving the same 99 tool definitions, 18 October 2026, Claude Code 2.1.302 and Codex 0.161.0. Mock results are labelled as such.
- Ranges are 95% Wilson intervals. Each app used a different model, so these numbers don't rank the apps.
FAQ
How do you measure the quality of an MCP server?
Measure it in three layers. First, server checks that need no model: tool-list size, instruction length, default response size, error size and annotations. Second, agent behaviour in each agent app your users run: does the agent find the tools, can it use the results, does it repeat calls. Third, the outcome: the share of real jobs answered correctly, as pass/N with a confidence range, and the tokens each job costs.
What metrics matter most for MCP tools?
Start with task pass rate per agent app, discovery rate (did the agent reach your tools at all), default response size, error size and clarity, and tokens per task. In our worked example, the most damaging problems were a 46,253-token default result and discovery misses, neither of which a schema check reports.
Is a static MCP quality score enough?
No. Static scanners check descriptions, schemas, protocol and security, which is useful. But they can't see what each agent app shows the model, how it caps results, or whether the agent finds the tool through search. Those only show up when you run agents on real jobs.
How many runs do I need to measure an MCP server?
Enough to report a range. With 40 runs, 34 correct answers is 71–93% at 95% confidence; with 2 runs you only learn the direction. Repeat each job at least 5 times per agent app, and treat differences as real only when the ranges don't overlap.
What is a good tool-selection accuracy for an MCP server?
There's no universal threshold, because it depends on the jobs and the agent app. Track it per job and per app, and compare against your own previous release. In agent apps that search for tools, also track how often the agent reached your server at all, because a missed search never shows up as a wrong tool call.
Get your MCP server scored on all 15 metrics.
We run the jobs your users need done in Claude Code, Codex and the other apps they use, fill in this scorecard with your numbers, and show which fixes move them.