---
title: Sample MCP Agent-Readiness Report (Demo Server)
description: A sample agent-readiness report on a made-up demo MCP server: 90 runs in Claude Code, Codex and a small model, findings, fixes and before/after pass/N.
canonical: https://spiritroom.app/sample-report/
published: 2026-10-21
updated: 2026-10-21
---

> Markdown version of https://spiritroom.app/sample-report/ for agents. Index: /llms.txt

[Blog](/blog/) / [Agent experience (AX)](/guides/agent-experience-ax/)

# Sample agent-readiness report: what a Spirit Room test finds, on a demo server

This is what a report looks like. It's a sample: the server is Acme Tasks, a made-up task tracker with 36 synthetic tasks that we built for experiments. No customer, no real data. The method, the format and the numbers are real: 90 agent runs, scored and read by hand.

Leonid Bugaev · Published 21 Oct 2026

**TL;DR**

- **Sample only:** a made-up demo server (Acme Tasks, 21 tools, synthetic data), not a customer.
- **Every job passed in every setup:** 10/10 (72–100%) in all 9 cells, 90/90 overall. The findings are about cost and errors, not wrong answers.
- **Errors:** runs with an error went from 4/30 to 0/30 after the fixes. One error had dumped 3,109 tokens of schema.
- **Size:** the default list went from 11,776 to 546 tokens; tool result text fell 87–93% per app.
- **Cost:** median input per run fell 25% in Claude Code, 12% in Codex and 20% with the small model.
[Summary](#summary)[What we tested](#scope)[Findings and fixes](#findings)[Before and after](#results)[What we'd do next](#next)[How we measured](#how-we-measured)[FAQ](#faq)

**Sample report.** The server below is **Acme Tasks**, a made-up task tracker with 4 projects, 36 synthetic tasks and 21 tools, built by us for experiments. It is not a customer, and no real data is involved.

## Summary

We ran 5 everyday jobs through three agent setups, 10 times each, on three versions of the server: as shipped, with better descriptions, and with descriptions plus behaviour fixes. Agents answered correctly in every run. The problems were elsewhere: tools picked by guesswork, an error that dumped every tool's schema, and default results far larger than the question needed.

| Measure | As shipped | After fixes |
| --- | --- | --- |
| Jobs passed (all apps) | 30/30 (89–100%) | 30/30 (89–100%) |
| Runs with an error | 4/30 (5–30%) | 0/30 (0–11%) |
| Default list call | 11,776 tokens | 546 tokens |
| Workspace overview | 12,060 tokens | 198 tokens |
| Tool-list size | 1,369 tokens | 1,730 tokens |

The tool list got slightly bigger: the better descriptions cost 361 tokens per request in apps that load every definition, against thousands saved per result.

## What we tested

| Job (user's words) | Checked for |
| --- | --- |
| How many tasks in our Mobile app project are waiting on someone right now? | The right count |
| Which of our projects has the most tasks ready for review, and how many? | Project and count |
| Who on the team has the most open tasks, and how many? | Person and count |
| Which Billing migration tasks were closed since October 14? | Both closed tasks, not the older one |
| Who is working on the payment webhook retry task, and what's its status? | Person and status |

Apps: Claude Code 2.1.304 and Codex CLI 0.161.0 with their default models, and OpenCode 1.18.34 with a small free model. 2 runs per job per cell, 10 runs per cell, 90 runs in all.

## Findings and fixes

### 1. Descriptions too short to choose from

**Evidence:** the stats tool's description was one word: “Stats.” On the three “how many” jobs, Claude Code used it in 1 of 6 runs and the small model in 0 of 6. They paged through the task list instead: 12 and 17 list calls.

**Fix:** say what it returns and when to use it, including the app's own words:

```
Task counts per status for one project (or all projects if project_id is omitted),
plus open tasks per person. Status values: open; blocked (shown in the app as
"Waiting on someone"); in_review (shown as "Ready for review"); done.
Use this for "how many" questions.
```

**Result:** Claude Code used it in 6 of 6 runs, the small model in 5 of 6 (the other used the overview). Codex already chose well: 5 of 6 before, 6 of 6 after.

### 2. One date format, one huge error

**Evidence:** the activity tool rejected a plain date like “2026-10-14” and returned every tool's schema: 13,837 characters, 3,109 tokens. All 10 errors across the first two versions were this one. Every run recovered, at a cost.

**Fix:** accept plain dates, and make errors one line that names the allowed values:

```
list_items: status "waiting" is not valid. Use one of: open, blocked ("Waiting on someone"),
in_review ("Ready for review"), done.        (51 tokens, was 3,118)
```

**Result:** 0 errors in 30 runs.

**Want this report for your MCP server or API?**

We run your users' real jobs through Claude Code, Codex and the other apps they use, find what slows or breaks them, and measure each fix the same way.

[Book a pilot →](/#contact)

### 3. Default results far larger than the question

**Evidence:** the list tool returned all 36 tasks with full details by default (11,776 tokens), and the overview returned 12,060 tokens. In one setup two overview calls added up to 80,216 characters.

**Fix:** a summary line, a few fields per item, and a next-page offset. The overview became counts per project.

```
{"summary": "36 matching tasks, showing 10; next page: offset=10",
 "items": [{"id": "it001", "title": "Update hero copy", "status": "in_review", ...}]}
```

**Result:** 546 and 198 tokens; the overview's two calls in that setup fell to 944 characters.

## Before and after, per app

| App | Version | Passed | Median input tokens | Tool result text (10 runs) | Runs with errors |
| --- | --- | --- | --- | --- | --- |
| Claude Code | As shipped | 10/10 | 26,960 | 84,740 chars | 2 |
| Claude Code | Fixed | 10/10 | 20,249 (−25%) | 10,068 (−88%) | 0 |
| Codex | As shipped | 10/10 | 89,129 | 32,929 chars | 1 |
| Codex | Fixed | 10/10 | 78,126 (−12%) | 4,440 (−87%) | 0 |
| Small model | As shipped | 10/10 | 21,344 | 68,521 chars | 1 |
| Small model | Fixed | 10/10 | 17,168 (−20%) | 4,911 (−93%) | 0 |

Each 10/10 is 72–100%. The middle version (descriptions only) also passed 10/10 everywhere and fixed tool choice, but kept the big error and results; for Codex and the small model its result text was even higher than as shipped. Token counts are each app's own figures; compare within an app.

## What we'd do next

- Add harder, multi-step jobs: these were 1–4 call lookups, so they show cost and errors more than failure.
- Test write jobs on a mock with confirmation steps.
- Re-run on each release and compare ranges.

Related: [how to reduce MCP tool response size](/blog/reduce-mcp-tool-response-size/), [per-model MCP tool surfaces](/blog/per-model-mcp-tool-surfaces/), [do cheaper models make MCP less reliable?](/blog/cheaper-models-mcp-reliability/) and [the agent scorecard](/blog/measure-mcp-server-quality/).

## How we measured

- **Server:** Acme Tasks, a made-up demo with synthetic data, served over MCP; writes return “nothing changed”. Three versions: as shipped, descriptions, descriptions plus fixes.
- **Runs:** 21 Oct 2026, 90 runs, each app isolated from other MCP servers. Answers checked against fixed answers by script; 5 automatic fails were overridden after reading (answers that named an excluded task while explaining why it was excluded).
- **Units:** sizes in o200k\_base tokens from the server itself; input tokens from each app's usage. Ranges are 95% Wilson intervals.
- **Limits:** simple lookup jobs, a small dataset, one model per app.

## FAQ

### What does an agent-readiness report include?

A summary, the jobs and apps tested, each finding with evidence and a concrete fix, before-and-after numbers as pass/N with 95% ranges plus tokens and errors, what to do next, and the method and limits.

### Is this a real customer report?

No. It's a sample built on Acme Tasks, a task tracker we made up for experiments, with synthetic data. The runs and numbers are real runs on that demo server.

### Which agent apps are tested?

Here, Claude Code, Codex CLI and OpenCode with a small free model. In a pilot we test the apps your users use.

### Why report pass/N with a range?

Agent runs vary. 10 of 10 passing means a true pass rate of about 72–100%, not 100%, so we always show the range.

**Want this report for your MCP server or API?**

We run your users' real jobs through Claude Code, Codex and the other apps they use, find what slows or breaks them, and measure each fix the same way.

[Book a pilot →](/#contact)

## Keep reading

- Up[Agent experience (AX): how to measure it (guide)](/guides/agent-experience-ax/)
- Related[How to measure MCP server quality: the agent scorecard (2026)](/blog/measure-mcp-server-quality/)
- Related[How to test an MCP server with real AI agents (guide)](/guides/test-mcp-servers-with-agents/)
- Related[How to make your MCP server or API agent-ready, with proof (guide hub)](/guides/agent-ready-tools/)