--- title: Spirit Room · Optimise your MCP, CLI or API for AI agents description: Give us the jobs AI agents must get right on your MCP, CLI or API. We run them through hundreds of simulations and keep the wording that helps agents finish in fewer steps, with fewer mistakes, on cheaper models. canonical: https://spiritroom.app/ source: https://spiritroom.app/ status: early-access (not generally available) generated: 2026-10-21 --- > Markdown version of https://spiritroom.app/ for agents. Product screens are illustrative (Example data). Agent instructions: /for-agents.md · Index: /llms.txt · Contact API: POST /api/contact # Optimise your MCP, CLI or API for AI agents. You give us the jobs agents need to get right. We run every major agent through hundreds of simulations, trying new tool descriptions, help text, errors and combined tools. The result: agents finish your jobs in fewer steps, with fewer mistakes, even on cheaper models. [Test your MCP](/#contact)[Get updates](/#contact) [New research**How to measure MCP server quality: the agent scorecard**15 metrics in three layers, with the numbers from a 99-tool production server that passed every probe and still tripped up agents.Read →](/blog/measure-mcp-server-quality/) 01 / The problem ## Your next million users are agents. Most are failing silently. Every failed agent run is a developer who decides your product “doesn't work with AI.” ### Agents fail quietly They skim `--help`, guess parameters, fall back to curl and declare victory. You see churn, never the reason. ### Most tools aren't agent-ready MCP tool names, schemas, descriptions and error messages decide whether a model picks the right tool. Most were written in an afternoon. ### Every model and agent app behaves differently What works for one model in Cursor breaks for another in Codex. Nobody tests the matrix. 02b / The time-lapse ## Round 1 wanders. Round 365 knows the way. **The job:** an agent must rotate an API key on an example API, from START to GOAL. **Success** means the key is rotated with no dead ends. In round 1 it guesses its way through help pages, wrong endpoints and errors. Each round Spirit Room rewrites one confusing part of the tools; by round 365 the agent takes the short path. Day outside**00h 00m***/24h* Round inside**1***/365* Steps**41** Tokens**0%** Illustrative example 02 / Who it's for ## Built for teams whose users are now agents. i. ### MCP servers Names, descriptions and schemas tuned so models pick the right tool first time. ii. ### Public APIs Docs, OpenAPI descriptions and errors agents can recover from. iii. ### Dev-tool companies CLIs and SDKs that work in Claude Code, Codex and Cursor, out of the box. iv. ### Internal agents System prompts and internal tools verified against your requirements. 03 / Dream outcome ## Agents that get it right on the first try. Every agent, in every popular agent app, reads the right line, picks the right tool, recovers from errors and finishes the task your way, measured against your requirements. Your agent, before Skims the help. Guesses a flag. Reaches for curl. Declares victory. 9.4 steps · 2/5 reqs → After one year inside Reads the right line. Picks the right tool. Recovers from the error. Done. 3 steps · 5/5 reqs 03b / Cost optimization ## Same pass rate. Cheaper model. Frontier models can muddle through vague tool names, thin schemas and cryptic errors. Smaller, cheaper models can't: they need every description, parameter and error message to be unmistakable. Spirit Room tunes your MCP server, CLI and API until the cheaper tier succeeds too, so you can downshift models and cut token cost without losing quality. Model downshift · task pass rate · | Model tier | Before tuning | After tuning | Cost / successful task | | --- | --- | --- | --- | | Frontier | 86% | 94% | $0.42 → $0.36 | | Mid-tier | 58% | 91% | $0.19 → $0.09 | | Small / cheap | 31% | 88% | $0.11 → $0.03 | Cost per successful task = token cost ÷ passing runs. The value equation - **Dream outcome:** frontier-level pass rate on your tasks - **Likelihood:** measured per model × agent app against your requirements, not guessed - **Time:** one pilot, about two weeks - **Effort:** zero code changes, winners come back as a patch or via Wrap Tell us your target model in the pilot form and we will test whether it can carry your tasks. [Test your MCP on a cheaper model →](/#contact) 04 / Mechanism ## The mechanism: a training loop for your tools. Zero code. Spirit Room wraps your existing MCP server, CLI or API and swaps in new wording to run experiments. Your source and MCP definitions are never touched. [Test your MCP →](/#contact) 1. ### Wrap Proxy your MCP server, CLI or API and override names, descriptions, schemas, help, errors, docs and prompts. 2. ### Run A task suite, QA for agents, runs real tasks with evaluation criteria across models and agent apps. 3. ### Score Every run is checked against written requirements (scoring engine: Proof). 4. ### Listen Agents self-report friction and confusion during each run. 5. ### Rewrite Variants of descriptions, docs and errors are generated, re-run and kept only if they win. 6. ### Re-score. Repeat. Hundreds of rounds in a day. Winners ship as a patch you review, or are served live by Wrap. 05 / Agent journey ## See exactly where your agent cut the corner. Every run becomes a journey: what it read, what it guessed, where it gave up and improvised. Each step is annotated with what to change in your tools. Two phases, as in regulated engineering: **design-time verification** through the training loop, and **runtime monitoring** in production to confirm agents still succeed as models, agent apps and your API change. **[Illustrative mock · Example data, not a real benchmark]** journey · rotate\_api\_keyclaude-sonnet × claude-code · round 0 00:00.401 $ acme --help | head -10 Read 10 of 42 lines. CUT CORNERThe `keys rotate` subcommand is documented on line 11. The agent never saw it. 00:03.102 $ acme keys create --replace error: unknown flag (no hint) INSIGHTThe error offers no recovery. Variant B: “did you mean `keys rotate`?” 00:07.903 $ curl -X DELETE …/keys/old && curl -X POST …/keys Deleted the old key before creating the new one. 00:09.204 proof verify --task rotate\_api\_key 2 / 5 requirements · REQ-3 “zero-downtime rotation” violated agent self-report › “I wasn't sure whether rotate existed, so I used the API.” ROUND 6 Help reordered, error hint added **3 steps · 5/5 requirements · 94% pass across 12 cells** Your docs got written for humans. Your users aren't. Claude CodeCodexCursorChatGPTcustom agentGPTClaudeGeminiopen-weightMCPRESTCLI--helperror messagestool schemasClaude CodeCodexCursorChatGPTcustom agentGPTClaudeGeminiopen-weightMCPRESTCLI--helperror messagestool schemas 06 / Inside the room ## What you get to see. Every model, every agent app. **[Illustrative mock · Example data, not a real benchmark]** matrix · pass rate by model × agent app | | Claude Code | Codex | Cursor | ChatGPT | | --- | --- | --- | --- | --- | | Claude | | | | | | GPT | | | | | | Gemini | | | | | | Open-weight | | | | | **[Illustrative mock · Example data, not a real benchmark]** requirement score · by round 0% from 31% at round 0 · 25 tasks **[Illustrative mock · Example data, not a real benchmark]** variant B · tools/search\_issueswinner · +32pp ``` - "description": "Search issues." + "description": "Full-text search over issues in ONE project. Requires project_key + (e.g. 'WEB'); call list_projects first if unknown. Max 50 results; use cursor." … - error: "400 Bad Request" + error: "project_key missing. Call list_projects to find it, then retry." ``` wrong-tool calls **−62%**avg steps **9.4 → 4.1**pass **58% → 90%** **[Illustrative mock · Example data, not a real benchmark]** runs · live - create\_invoicegpt · codex**5/5** - refund\_partialgemini · cursor**3/5** - rotate\_api\_keyclaude · cc**5/5** - paginate\_allopen · custom**2/5** - search\_issuesclaude · cursor**4/4** Want this matrix for your own MCP server, CLI or API?[Test your MCP →](/#contact) 06b / What a pilot can show ## The numbers you'll get for your own product. **0%**pass rate across 12 model × agent app cells after round 6 **0**scored rounds in one day **0%**wrong-tool calls **6→1**calls after a merged tool **0**model × agent app cells tested per task **0%**cost per successful task on a cheaper model tier 07 / In use > “I built Spirit Room out of what we learned shipping Probe and Proof: agents kept tripping over the same tool descriptions, help text and errors. Now we use it on both products to find what agents struggle with and fix it before anyone files a ticket.” **Leonid Bugaev**Founder, Probe Labs Used by[Probe](https://probelabs.com)[Proof](https://reqproof.com) 08 / The offer ## Train once, or wrap and stay ahead. A · Train pilot · “Agent Readiness Sprint” ### Fixes, exported into your code. A pilot finds the best descriptions, help text and errors per model and agent app, then exports the winners into your code and MCP definitions as a reviewed patch. - **Agent Readiness Audit**: where agents fail today, 3 models × 2 agent apps, up to 25 tasks - **Training rounds**: up to 10 rewrite / re-score rounds - **Agent Journey Report**: annotated journeys for every failing task - **The Winning Patch**: best variants as a reviewed PR - **Evidence pack**: before/after per requirement, per model × agent app - Pilot bonuses Error-message rewrite pack · Agent-ready docs checklist · One re-score after your next release (within 60 days) [Book a Train pilot →](/#contact) B · Wrap · “Always-on Agent Layer” ### Always-on, optimised tools. A hosted layer in front of your MCP server, CLI or API that serves the optimised version to each model and agent app. Zero code changes. - Runtime monitoring: are agents still succeeding? - Agent-journey analytics on real traffic - Engineered for reliability, with pass-through fallback to your original setup - Optional, opt-in docs interception on domains you allowlist (off by default) [Talk to us about Wrap →](/#contact) 08b / Wrap · Per-model tuning ## A tuned version for every model. Wrap detects which model is behind each session and serves the version that model does best with: tool names, descriptions, schemas, help, errors and docs, tuned per model when it matters. For specific tasks it can serve a super-optimised toolset, merge several tool calls into one, and add scripted virtual tools composed from your existing ones. Everything is opt-in and customer-controlled, with pass-through fallback to your original setup. Per-model tuning · live sessions · | Model detected | Variant served | Changes | | --- | --- | --- | | frontier-large | base | original setup (pass-through) | | mid-tier | v-mid-3 | explicit enums, shorter descriptions | | small-fast | v-small-7 | renamed tools, examples in schema, error hints | Merged call · opt-in per task ``` before: list_projects → get_project → list_issues → get_issue ×3 (6 calls) after: project_issue_digest(project="api", state="open") (1 call) ``` Virtual tool · scripted from existing endpoints ``` tool release_readiness(repo) = GET /checks?ref=main + GET /issues?label=blocker + GET /deploys/latest → { ready: bool, blockers: [...], last_deploy: "…" } ``` Model names generic; merges and virtual tools run only where you allow them. Customer-controlled - **Per-model variants:** served only when they score better for that model - **Merged calls:** opt-in per task, only for calls you mark safe to combine - **Virtual tools:** scripted on top of your endpoints, reviewed by you - **Fallback:** any error or unknown model gets your original setup - **Audit:** every variant, merge and script is versioned and logged [Talk to us about Wrap →](/#contact) **[Illustrative mock · Example data, not a real benchmark]** job-scoped toolset · task: triage\_bug36 tools list\_projectsget\_projectcreate\_projectarchive\_projectlist\_issuesget\_issuecreate\_issueupdate\_issueclose\_issuereopen\_issuesearch\_issuesadd\_labelremove\_labellist\_labelsassign\_userlist\_usersget\_useradd\_commentlist\_commentsdelete\_commentlist\_milestonesset\_milestonelist\_webhookscreate\_webhookexport\_csvimport\_csvbulk\_closebulk\_labellist\_releasesget\_releaselist\_checkslist\_deploysget\_deployrollbackrotate\_api\_keylist\_keys For this task the model sees 5 of 36 tools · opt-in 09 / Wrap · real traffic ## Wrap keeps learning from real agent traffic. Local-first and opt-in. Nothing leaves a machine without explicit consent. ### Gateway mode Run inside LLM apps and gateways (e.g. Tyk AI Studio or OpenRouter-style gateways) to observe how real agents use your tools, and turn failures into training tasks. ### Local agent An opt-in agent on developer machines reads Codex, Claude Code and Cursor session history to spot friction with your tools, MCP servers and APIs. ### Docs interception Optional and off by default. When enabled, Spirit Room intercepts the agent's HTTP requests to your docs sites and serves tuned, experimental versions. A/B per model and agent app shows which wording leads to success. Strictly opt-in, scoped to domains you own and allowlist. 10 / Pricing ## Two weeks inside. Priced as a pilot. Platform fee plus compute. Choose the models and agent apps to test, or bring your own API keys and pay only the platform fee. What this takes by hand · rough estimates | | | | --- | --- | | Build a cross-model agent test setup | 3–6 eng-weeks | | Run and grade hundreds of agent sessions | 1–2 eng-weeks / round | | Write and A/B test descriptions, errors, docs | ongoing | | Requirement-level evidence for every run | rarely done | What the pilot delivers Evidence and an experience: measured before/after results for every model × agent app you pick, annotated journeys, and the winning variants. If your tools are already agent-ready, you get proof of that, which is useful too. Limited pilot slots · we run pilots hands-on 1 · Pilot · start here $1,999one-time · 2 weeks - 1 product: MCP, CLI or API - Up to 25 tasks - 3 models × 2 agent apps - Up to 10 training rounds - Compute included (capped) - Journey report + winning patch - Cheaper-model downshift check - Evidence pack + 3 bonuses [Test your MCP →](/#contact) 2 · Enterprise · ongoing optimisation Customafter the pilot · monthly Models, agent apps and your product keep changing. We keep the agents winning. - Simulations re-run as models, agent apps and your product change - Descriptions, help, errors and tools re-tuned continuously - Real agent usage monitored (opt-in) - Monthly report: what changed, what improved - Private deployment, SSO, your own keys - Priority support [Talk to us →](/#contact) Pricing may change. Compute above the included cap billed at provider cost plus a small margin. 11 / Questions ## Before you step inside. [All questions →](/faq/) ### Can Wrap serve a different version to each model? Yes, if you turn it on. Wrap detects the model behind a session and serves the variant that scored best for it: tool names, descriptions, schemas, help, errors and docs. For specific tasks it can also merge several tool calls into one (only where you allow it) and expose scripted virtual tools composed from your existing endpoints. It is all opt-in, customer-controlled and versioned, with pass-through fallback to your original setup. ### Can this let us use cheaper models? Often, yes, and that is a key use case. Cheaper and smaller models need tool descriptions to be very clear. Spirit Room tunes your descriptions, schemas, errors, docs and agent prompts, then measures whether a cheaper tier reaches your target pass rate. Results vary per product; we report what we measure, with no guarantees. ### Is Spirit Room available today? We're running a limited number of hands-on pilots. Request a pilot and we'll reach out, or get updates by email. ### Will changes break my existing users or agents? No. Before anything goes live, we run every agent against your current version and the new one, and replay your real traffic against the new one. You only ship changes that are proven not to break anything and to make agents better. Old tool names keep working, you roll out gradually, and you can roll back instantly. ### Do we have to route traffic through you? No. Pick whichever fits: 1. A Cloudflare Worker we give you, running on your own Cloudflare account, so all traffic stays on your infrastructure. 2. Our SDK, dropped into your own MCP server. 3. Apply the fixes to your MCP server directly. We hand them over as a ready-to-paste list your coding agent can apply. Whichever you pick, nothing calls us while your users work. ### Do I need to change my code or MCP server? No. Spirit Room sits in front of your tools and swaps in new descriptions, help, errors and docs during experiments. Winners return as a patch you choose to merge, or are served live through Wrap. ### Which models and agent apps? GPT, Claude, Gemini and open-weight models, across Claude Code, Codex, Cursor, ChatGPT and your own internal agents. ### What does “scored by Proof” mean? Each task has written requirements. Proof checks every run against them, so improvements are measurable and traceable. ### What about my data, keys and docs? Use sandbox credentials or bring your own keys. The local agent is opt-in and local-first. Docs interception is off by default and limited to domains you allowlist. ### What if my tools are already good? Then you get proof of that: measured results per model × agent app and the journeys to back it up. ### Are the numbers on this page real? No. Every product screen is an illustrative mock labeled “Example data”. [Test your MCP →](/#contact)[Get updates →](/#contact) 12 / Close ## Your agents get one day outside. Give them a year inside. Tell us what your agents should master. We reply within two business days. Limited pilot slots. Test your MCPGet updates ## Contact Pilot request: `POST /api/contact` JSON with `name`*, `email`*, `train`* (product + jobs), optional `company`, `tier` (pilot|enterprise). Updates signup: `POST /api/contact` with `{"type":"updates","email":"..."}`. Agents: ask your human before submitting. Details: /for-agents.md