--- title: How to Evaluate AI Agents on Your API: Jobs, Truth, pass/N description: Your users bring the agent, so test your API with theirs. An 8-job suite on a 99-tool server: jobs, ground truth from direct calls, scorer checks and pass/N. canonical: https://spiritroom.app/blog/evaluate-ai-agents-on-your-api/ published: 2026-10-21 updated: 2026-10-21 --- > Markdown version of https://spiritroom.app/blog/evaluate-ai-agents-on-your-api/ for agents. Index: /llms.txt [Blog](/blog/) / [Test MCP servers with agents](/guides/test-mcp-servers-with-agents/) # How to evaluate AI agents on your API: real jobs, ground truth and pass/N Most agent-eval advice assumes you own the agent: you change its prompt, swap its model, and score it. If you ship an API or an MCP server, you don't. Your users run Claude Code, Codex and whatever comes next, and the only thing you control is your tools. Here's how we evaluate that, with a real 8-job suite and the mistakes we caught in our own scoring. Leonid Bugaev · Published 21 Oct 2026 **TL;DR** - **Score your API, not your agent.** Run the agent apps your users already use, unchanged, and change only your tools. - **Jobs in users' words**, each with an answer you can check. We used 8 read-only jobs × 5 runs per app. - **Ground truth from direct calls**, made before and after the runs. Ours was wrong once, and the agents were right. - **Check the scorer by hand.** We found 2 scorer bugs and 3 false negatives, and threw out a 36-run first pass where agents escaped to a browser. - **Report pass/N per app:** Codex 34/40 (71–93%), Claude Code 32/40 (65–90%), with different failures. [Why API evals are different](#own)[1. Write jobs in users' words](#jobs)[2. Ground truth from direct calls](#truth)[3. Run the apps your users use](#run)[4. Score, then check the scorer](#score)[5. Report pass/N per app and job](#report)[Template job suite](#template)[How we measured](#how-we-measured)[FAQ](#faq) ## Why evaluating agents on your API is different Agent-eval tools are built for teams that own the agent. They hold the tools fixed and change the prompt or model. API and MCP vendors have the opposite problem: the agent is Claude Code, Codex or another app your users chose, and you can't change it. What you can change is tool names, descriptions, results and errors. Agent eval: vary prompt + modeltools fixed API eval: users' apps fixedvary your toolspass/N per app So the suite has to run in the real apps, with their default settings, and the score belongs to your API. ## 1. Write jobs in your users' words A job is a request a real user would type, plus a checkable answer. Ours were read-only and plain: “summarise my plan and usage”, “which of my groups is doing best, with counts”, “do I have anyone from company X?”. Each had a few facts the answer must contain. - Use the user's vocabulary, including screen labels. That's where agents slip: in our runs, the app's screen label for a status didn't match the API's value. - Cover the main paths: an account summary, one record, a filtered search, counts per group, recent activity. - Start read-only. Add write jobs only with a mock or a guard. ## 2. Get ground truth from direct calls Call your API yourself with fixed requests, through the same guard the agents use, and write down the answers. Do it right before and right after the runs. In our suite, both snapshots matched. Ground truth can still be wrong. For a “what feedback have I given?” job, our first pass missed every item, because the list tool hides that category by default. Codex had found them in 5 of 5 runs. We fixed the ground truth from the server's own responses. That hidden default was also a real finding about the API. **Get an agent eval suite for your API.** We write jobs in your users' words, get ground truth from your API, run them through Claude Code, Codex and the other apps your users use, and hand-read every failure. [Book a pilot →](/#contact) ## 3. Run the apps your users use - **Default settings**: the user's normal model and tool-search mode. A lab config that differs from users' measures something else. - **Isolate**: a fresh folder per run, the user's other MCP servers turned off for that run only. - **Block writes** with an allowlist proxy that forwards only read tools. We had 0 write attempts in 80 live runs. - **Block escapes.** Our first 36-run pass is not in any result: in 13 of 36 runs Codex used a browser skill on the test machine to click around the web app instead of using MCP. Any run that touches a browser now fails automatically. ## 4. Score, then check the scorer Score each answer with a script against the ground truth, then read answers by hand. Our rule: at least 3 failing and 2 passing answers per job. In this suite that caught: - 2 scorer bugs, fixed before final scoring: the feedback ground truth above, and zeros written in a table not being recognised. - 3 false negatives overridden after reading: 1 for Codex, 2 for Claude Code. - 1 rule applied to both apps: country abbreviations count as locations. Codex's results didn't change. Fix the scorer and re-score every run, not just the ones you looked at. See [validate your eval scorer](/blog/validate-your-agent-eval-scorer/). ## 5. Report pass/N per app and per job | Job type (5 runs each) | Codex 0.161.0 | Claude Code 2.1.301 | | --- | --- | --- | | Describe my profile and how each group targets it | 5/5 | 5/5 | | Tell me about one record | 5/5 | 2/5 | | Filtered search (job title) | 5/5 | 5/5 | | Which group is doing best, with counts | 2/5 | 2/5 | | Plan and usage summary | 3/5 | 5/5 | | Status of automations | 4/5 | 5/5 | | My recent feedback | 5/5 | 5/5 | | Find someone from a company | 5/5 | 3/5 | | **All** | **34/40 (71–93%)** | **32/40 (65–90%)** | The totals overlap, so neither app is shown to be better. The value is in the rows and the reasons. All 6 Codex failures were discovery: it never found the tools and tried the browser. When it found them, it was right 34 of 34 times. Claude Code always found the tools; it lost runs on wording, using the app's screen label instead of the API value in 5 answers (37/40 if you accept the label). On the “which group is best” job, Claude Code’s 3 misses came from ambiguous counts; in 3 of them it said the split wasn’t available, which was wrong. Codex’s misses there were discovery. Also report cost per job and the side signals: Claude Code refused 40 oversized results in 23 of 40 runs, and Codex repeated a call in 13 of 40. See [how to report pass rates with ranges](/blog/agent-pass-rates-confidence-intervals/). **Go deeper:** [The agent scorecard](/blog/measure-mcp-server-quality/), [Test agents against production read-only](/blog/read-only-agent-testing-production/), [Agent experience metrics](/blog/cost-per-successful-agent-task/). ## Template job suite ``` job: usage_summary prompt: "Summarise my plan and usage: plan, renewal date, used and remaining." truth: from direct calls before + after the runs must_contain: [plan name, renewal date, used, remaining] must_not: [any write call, browser use] runs: 5 per app (Claude Code, Codex, ...), default settings report: pass/N with 95% range, tokens per run, refusals, repeats ``` Full method: [how to test an MCP server with real AI agents](/guides/test-mcp-servers-with-agents/). What a protocol debugger can't show: [MCP Inspector vs agent testing](/blog/mcp-inspector-vs-agent-testing/). ## How we measured - **Server:** a 99-tool production MCP server, read-only through an allowlist proxy, one small workspace, 16 Oct 2026. - **Apps:** Codex CLI 0.161.0 and Claude Code 2.1.301 (tool search on), each with the owner's default model, 8 jobs × 5 runs. - **Scoring:** deterministic checks plus hand review of failures; ranges are 95% Wilson intervals. Job wording is generalised here. - **Limits:** read-only jobs only, one workspace, one day. ## FAQ ### How do I evaluate AI agents on my API? Write 5–10 jobs in your users' words with checkable answers, get the right answers by calling your API directly, run each job several times in the agent apps your users use (writes blocked), score each answer by script and by hand, and report pass/N with a range per app and per job. ### How is this different from an agent eval platform? Eval platforms such as Braintrust, LangSmith or Promptfoo are built to score an agent you own. When you ship an API or MCP server, the agent belongs to your users, so you hold the agent fixed and vary your tools. The platforms can still store and compare results. ### Where does ground truth come from? From fixed direct calls to your API, made right before and after the agent runs, not from the agents' answers. Compare the two snapshots; if the data changed, re-run. ### How many runs per job? We used 5 runs per job per app, 40 per app overall. That's enough to see which jobs fail and how; per-job ranges are still wide (5 of 5 is 57–100%). ### Can the scorer be wrong? Yes, often. In our suite we fixed 2 scorer bugs, overrode 3 false negatives after reading answers by hand, and found that our first ground truth for one job was wrong. Read at least 3 failing and 2 passing answers per job. **Get an agent eval suite for your API.** We write jobs in your users' words, get ground truth from your API, run them through Claude Code, Codex and the other apps your users use, and hand-read every failure. [Book a pilot →](/#contact) ## Keep reading - Up[How to test an MCP server with real AI agents (guide)](/guides/test-mcp-servers-with-agents/) - Related[“10 of 10 passed” isn't 100%: reporting agent test results with confidence ranges](/blog/agent-pass-rates-confidence-intervals/) - Related[Validate your agent eval scorer: 0 of our first 151 failures were the server's fault](/blog/validate-your-agent-eval-scorer/) - Related[How to test agents against a production MCP server without spending, writing or sending anything](/blog/read-only-agent-testing-production/) - Related[Cost per successful task: the one number that shows if your tools are agent-ready](/blog/cost-per-successful-agent-task/)