Blog / Agent experience (AX)

Sample agent-readiness report: what a Spirit Room test finds, on a demo server

This is what a report looks like. It's a sample: the server is Acme Tasks, a made-up task tracker with 36 synthetic tasks that we built for experiments. No customer, no real data. The method, the format and the numbers are real: 90 agent runs, scored and read by hand.

Leonid Bugaev · Published 21 Oct 2026

Sample report. The server below is Acme Tasks, a made-up task tracker with 4 projects, 36 synthetic tasks and 21 tools, built by us for experiments. It is not a customer, and no real data is involved.

Summary

We ran 5 everyday jobs through three agent setups, 10 times each, on three versions of the server: as shipped, with better descriptions, and with descriptions plus behaviour fixes. Agents answered correctly in every run. The problems were elsewhere: tools picked by guesswork, an error that dumped every tool's schema, and default results far larger than the question needed.

The tool list got slightly bigger: the better descriptions cost 361 tokens per request in apps that load every definition, against thousands saved per result.

What we tested

Apps: Claude Code 2.1.304 and Codex CLI 0.161.0 with their default models, and OpenCode 1.18.34 with a small free model. 2 runs per job per cell, 10 runs per cell, 90 runs in all.

Findings and fixes

1. Descriptions too short to choose from

Evidence: the stats tool's description was one word: “Stats.” On the three “how many” jobs, Claude Code used it in 1 of 6 runs and the small model in 0 of 6. They paged through the task list instead: 12 and 17 list calls.

Fix: say what it returns and when to use it, including the app's own words:

Task counts per status for one project (or all projects if project_id is omitted),
plus open tasks per person. Status values: open; blocked (shown in the app as
"Waiting on someone"); in_review (shown as "Ready for review"); done.
Use this for "how many" questions.

Result: Claude Code used it in 6 of 6 runs, the small model in 5 of 6 (the other used the overview). Codex already chose well: 5 of 6 before, 6 of 6 after.

2. One date format, one huge error

Evidence: the activity tool rejected a plain date like “2026-10-14” and returned every tool's schema: 13,837 characters, 3,109 tokens. All 10 errors across the first two versions were this one. Every run recovered, at a cost.

Fix: accept plain dates, and make errors one line that names the allowed values:

list_items: status "waiting" is not valid. Use one of: open, blocked ("Waiting on someone"),
in_review ("Ready for review"), done.        (51 tokens, was 3,118)

Result: 0 errors in 30 runs.

3. Default results far larger than the question

Evidence: the list tool returned all 36 tasks with full details by default (11,776 tokens), and the overview returned 12,060 tokens. In one setup two overview calls added up to 80,216 characters.

Fix: a summary line, a few fields per item, and a next-page offset. The overview became counts per project.

{"summary": "36 matching tasks, showing 10; next page: offset=10",
 "items": [{"id": "it001", "title": "Update hero copy", "status": "in_review", ...}]}

Result: 546 and 198 tokens; the overview's two calls in that setup fell to 944 characters.

Before and after, per app

Each 10/10 is 72–100%. The middle version (descriptions only) also passed 10/10 everywhere and fixed tool choice, but kept the big error and results; for Codex and the small model its result text was even higher than as shipped. Token counts are each app's own figures; compare within an app.

What we'd do next

• Add harder, multi-step jobs: these were 1–4 call lookups, so they show cost and errors more than failure.
• Test write jobs on a mock with confirmation steps.
• Re-run on each release and compare ranges.

Related: how to reduce MCP tool response size, per-model MCP tool surfaces, do cheaper models make MCP less reliable? and the agent scorecard.

How we measured

• Server: Acme Tasks, a made-up demo with synthetic data, served over MCP; writes return “nothing changed”. Three versions: as shipped, descriptions, descriptions plus fixes.
• Runs: 21 Oct 2026, 90 runs, each app isolated from other MCP servers. Answers checked against fixed answers by script; 5 automatic fails were overridden after reading (answers that named an excluded task while explaining why it was excluded).
• Units: sizes in o200k_base tokens from the server itself; input tokens from each app's usage. Ranges are 95% Wilson intervals.
• Limits: simple lookup jobs, a small dataset, one model per app.