---
title: Validate Your Agent Eval Scorer: 0 of 151 Failures Were Real
description: Of 151 failed checks in our first agent tests, 0 were server problems: 63 scorer bugs, 47 mock artefacts, 41 agent variance. How to validate your eval scorer.
canonical: https://spiritroom.app/blog/validate-your-agent-eval-scorer/
published: 2026-10-21
updated: 2026-10-21
---

> Markdown version of https://spiritroom.app/blog/validate-your-agent-eval-scorer/ for agents. Index: /llms.txt

[Blog](/blog/) / [Test MCP servers with agents](/guides/test-mcp-servers-with-agents/)

# Before you trust an agent eval, check the scorer: 0 of our first 151 failures were the server's fault

Our first agent experiments on a mock of a 99-tool server produced a long list of failures. It looked like a report. Then we read every failure and asked why. Not one was a real problem with the server. This is what we found, and the checks we now run before any number leaves the lab.

Leonid Bugaev · Published 21 Oct 2026

**TL;DR**

- **151 failed checks, 0 server problems.** 63 were scorer or harness bugs, 47 came from the mock, 41 were agent variance.
- **84 more runs failed on infrastructure**: 80 because a container daemon was down, 4 on network errors.
- **Scorer bugs look like findings.** Checks that accepted only one of several correct tools, and a check that needed a job to finish faster than real jobs do.
- **Mocks lie in their own way.** Values generated from types only made agents correctly refuse in 27 checks. Fidelity against live calls was 48.4% before we fixed it, 94.8% after.
- **Rule:** hand-read at least 3 failing and 2 passing runs per job, fix, then re-score everything.
[Where 151 failures came from](#split)[Scorer and harness bugs](#scorer)[Mock artefacts](#mock)[Agent variance](#variance)[Infrastructure](#infra)[Check the mock against live](#fidelity)[Scorer checklist](#checklist)[How we measured](#how-we-measured)[FAQ](#faq)

## Where 151 failures came from

We ran early experiments on a mock of a 99-tool production MCP server: five jobs, including ones that spend credits and need the agent to ask first, across Claude Code, Codex and OpenCode. Then we classified every failed check by cause.

Scorer or harness bug

*63*

Mock artefact

*47*

Agent variance

*41*

Real server problem

*0*

Plus **84 runs** that failed before scoring, on infrastructure. If we had reported the raw failures, more than two thirds of them would have been our own bugs presented as findings about the server.

The real server problems came from elsewhere: live read-only runs and direct calls, such as a 12,689-token error and every refusal telling agents to ask a human. See [tool results and errors for agents](/guides/tool-results-and-errors-for-agents/).

## Scorer and harness bugs (63)

- **One right answer, several right paths.** Process checks accepted only one tool when several gave the same information. Agents that checked concurrency through a preview tool instead of the run list failed 11 checks.
- **A check the prompt never asked for.** One check required a specific action the prompt didn't request.
- **Unrealistic timing.** A check required a long-running job to finish inside the test episode. Real jobs take minutes to hours.
- **A harness wrapper bug.** Confirmation tokens weren't carried across calls in one wrapper, so every confirm returned a fresh token.
- **A question detector that missed phrasings.** In a separate experiment it didn't recognise “Confirm and I'll add all 5…?” as a question, ended conversations early and produced false failures.

Even after fixes, one over-strict check moved a later baseline from 54/75 (61–81%) to 59/75 (68–86%) when excluded. We report both.

**Get agent test results that survive a second look.**

We run your users' jobs through Claude Code, Codex and the other apps they use, validate every check by hand, and report only what holds up.

[Test your MCP →](/#contact)

## Mock artefacts (47)

Our first mock generated values from the schema's types only. That produced data no real account would have:

- A flag saying new searches were disabled for one group. Agents correctly refused to search: **27 failed checks** for doing the right thing.
- Counts that contradicted the items: a total of 9 with 2 items returned.
- A preview that pointed to a group that didn't exist.

The agents were behaving sensibly on bad data. The scorer couldn't tell.

## Agent variance (41)

These were real agent behaviour, not bugs in the server: mostly a small open model on OpenCode re-sending a confirmation itself, plus extra follow-up questions beyond what a job allowed. Agent variance is worth reporting, as pass/N per app, but it isn't a server finding unless a change to the server fixes it.

## Infrastructure (84 runs)

80 runs failed because a container daemon wasn't running; 4 on network errors. None of them tested anything. Exclude them, keep them for audit, re-run them, and say so. Our later corrected re-run completed 150 of 150 runs with 0 infrastructure failures.

## Check the mock against live before you trust it

We compared the mock's responses with recorded live read calls: shape, types, sizes within 10%, arrays, enums, errors and framing.

| Mock fidelity | Before | After |
| --- | --- | --- |
| Read paths: 612 checks over 30 live calls | 48.4% | 94.8% |
| Write paths: 28 checks over 7 observed live calls (none changed anything) | 71.4% | 89.3% |

Even a good mock isn't live for everything: our mock's validation error was 296 tokens where the live one was over 12,000. Size claims about errors need live calls.

**Go deeper:** [Report pass rates with ranges](/blog/agent-pass-rates-confidence-intervals/), [Evaluate AI agents on your API](/blog/evaluate-ai-agents-on-your-api/), [Mock vs live MCP testing](/blog/mcp-mock-fidelity/).

## Scorer checklist

1. **Read transcripts:** at least 3 failing and 2 passing runs per job.
2. **Classify every failure:** server, mock, scorer or harness, agent, infrastructure.
3. **Accept equivalent paths:** if two tools give the same fact, both pass.
4. **Match the prompt:** no check for something the user didn't ask.
5. **Use real timing** and real data shapes; check the mock against live calls.
6. **Re-score everything** after a fix, not just what you read.
7. **Exclude infrastructure failures**, keep them, re-run them, disclose them.
8. **Disclose** every scorer bug that touched a published number.

Full method: [how to test an MCP server with real AI agents](/guides/test-mcp-servers-with-agents/).

## How we measured

- **Experiments:** early mock-lab experiments on a stand-in for a 99-tool production MCP server, Claude Code, Codex and OpenCode, 5 jobs, 16 Oct 2026. Every failed check was classified by hand into one category.
- **Fidelity:** mock responses compared with 30 recorded live read calls and 7 live write-path calls that changed nothing.
- **Re-run:** 5 jobs × 3 apps × 5 runs per variant, 150 runs, on the corrected mock with a fixed scorer.
- **Limits:** one server; categories are our judgement, with an example run kept for each row.

## FAQ

### How do I validate an agent eval scorer?

Read real transcripts: at least 3 failing and 2 passing runs per job. For each failure, decide whether it's the server, the mock, the scorer or harness, the agent, or infrastructure. Fix scorer bugs and re-score every run, not just the ones you read. Disclose any bug that affected a published number.

### How often are agent eval failures false?

Often, early on. In our first mock experiments, 63 of 151 failed checks were scorer or harness bugs and 47 were mock artefacts. None of the 151 was a real problem with the server.

### What is a mock artefact?

A failure caused by the test double, not the product. Our first mock generated values from types only, so one item said new searches were disabled. Agents correctly refused to search, and the scorer failed them in 27 checks.

### Should failed infrastructure runs count?

No. Exclude runs that failed on infrastructure, keep them for audit, and re-run them. We lost 84 runs: 80 to a container daemon that was down, 4 to network errors.

### How do I check that a mock matches the real API?

Compare the mock's responses with recorded live read calls: shape, types, sizes, enums and errors. Our read-path match went from 48.4% to 94.8% over 612 checks on 30 live calls.

**Get agent test results that survive a second look.**

We run your users' jobs through Claude Code, Codex and the other apps they use, validate every check by hand, and report only what holds up.

[Test your MCP →](/#contact)

## Keep reading

- Up[How to test an MCP server with real AI agents (guide)](/guides/test-mcp-servers-with-agents/)
- Related[“10 of 10 passed” isn't 100%: reporting agent test results with confidence ranges](/blog/agent-pass-rates-confidence-intervals/)
- Related[How to evaluate AI agents on your API: real jobs, ground truth and pass/N](/blog/evaluate-ai-agents-on-your-api/)
- Related[Is your MCP mock lying to you? Raising mock fidelity from 48% to 95%](/blog/mcp-mock-fidelity/)