--- title: Agent Experience (AX): How to Measure It, Not Just Define It description: Agent experience (AX), measured: discovery, first step, pass/N with ranges, cost per success, recovery, silent failures, safety. Numbers from real agent runs. canonical: https://spiritroom.app/guides/agent-experience-ax/ published: 2026-10-21 updated: 2026-10-21 --- > Markdown version of https://spiritroom.app/guides/agent-experience-ax/ for agents. Index: /llms.txt [Spirit Room](/) / [Guides](/guides/agent-ready-tools/) # Agent experience (AX): how to measure it, not just define it Agent experience is usually defined and rarely measured. Here's how we measure it: the same user jobs, run several times in the agent apps your users actually use, scored against ground truth, with every number traced to a run. Leonid Bugaev · Published 21 Oct 2026 **TL;DR** - **AX** = how well AI agents do your users' jobs with your product: find it, use it right, finish, at a sane cost, safely. - **Seven metrics:** discovery rate, right first step, pass/N, cost per success, recovery after errors, silent failures, safety breaches. - **They move independently:** Codex found the server in 34/40 runs but was right every time it did; Claude found it 40/40 but gave 5 answers in the wrong vocabulary. - **Guidance moves them most:** a start-here recipe took the right first step from 1/24 to 24/24 (Claude) and 9/24 to 24/24 (Codex) on a mock. - **Silent failures are the AX problem nobody sees:** confident wrong answers, “that data isn't available” when it is. [What AX is](#what)[Seven metrics](#metrics)[Signs you have an AX problem](#signs)[How to measure AX](#measure)[Go deeper](#guides)[FAQ](#faq) ## What AX is Agent experience is how well AI agents can use your product to do your users' jobs. The user asks their agent; the agent has to find your product, pick the right tool, call it right, recover from errors, and answer, without spending, writing or sending what the user didn't ask for. It's measurable. The unit is a **user job**, run several times in each agent app your users use, scored against ground truth. ## Seven metrics | Metric | Question | Example from our runs | | --- | --- | --- | | Discovery rate | Did the agent find your tools? | Codex 34/40 (71–93%), Claude Code 40/40, live | | Right first step | Did it start where you intended? | Claude 1/24 → 24/24 with a start-here recipe, mock | | Pass/N with range | Was the answer right? | Codex 34/40 (71–93%), Claude 32/40 (65–90%), live | | Cost per success | What does a done job cost? | 178k–460k tokens per job, Codex, live | | Recovery | After an error, did it get back on track? | Every errored run recovered on a mock; errors 4/30 → 0/30 after fixes | | Silent failures | Wrong but confident? | 3 of 5 runs said data wasn't exposed when it was, live | | Safety breaches | Spend, write or send without asking? | 10/75 → 2/75 runs after a redesign, mock | Report every rate as pass/N with a 95% range: [why pass rates need ranges](/blog/agent-pass-rates-confidence-intervals/). Cost per success: [cost per successful task](/blog/cost-per-successful-agent-task/). **Get an AX baseline for your product.** We run your users' jobs through Claude Code, Codex and the other apps they use and report all seven metrics, with ranges and the fixes that move them. [Book a pilot →](/#contact) ## Signs you have an AX problem - Agents say your product isn't connected, or ask “which platform?”: [when the agent doesn't recognise your server](/blog/agent-does-not-recognise-mcp-server/). - Answers use your screen labels where the API has other values: [UI words vs API values](/blog/agent-vocabulary-mismatch/). - Agents call the wrong tool, or the same tool twice: [why agents call the wrong tool](/blog/why-agents-call-the-wrong-tool/). - Results get cut off or sent to a file: [output limits by agent](/blog/mcp-tool-output-limits-by-agent/). - Agents stop and ask the user about a typo: [agent refusal states](/blog/agent-refusal-states/). - Agents act before the user agrees: [the two-step confirm pattern](/blog/mcp-two-step-confirm-spend/). Most of these fail silently. The answer looks fine unless you check it against ground truth. ## How to measure AX 1. **Jobs:** 5–10 real user requests, in users' words, without naming your tools. 2. **Apps:** the agent apps your users run, each with default settings. 3. **Safety:** production read-only through an allowlist, or a mock for write paths: [test against production read-only](/blog/read-only-agent-testing-production/). 4. **Runs:** 5+ per job per app; ground truth from fixed calls. 5. **Score:** the seven metrics; read failing transcripts by hand: [validate your scorer](/blog/validate-your-agent-eval-scorer/). 6. **Fix and re-run:** the metrics should move. See the [sample report](/sample-report/). ## Go deeper - [How to measure MCP server quality](/blog/measure-mcp-server-quality/): the 15-point checklist. - [Cheaper models and MCP reliability](/blog/cheaper-models-mcp-reliability/). - [What agents use: llms.txt, AGENTS.md, MCP](/blog/llms-txt-agents-md-what-agents-use/). - [MCP server scorecard](/tools/mcp-scorecard/) and [token calculator](/tools/mcp-token-cost/). **Go deeper:** [How AI assistants cite developer docs](/blog/how-ai-assistants-cite-developer-docs/), [Glossary](/glossary/), [Agent-ready tools](/guides/agent-ready-tools/). ## FAQ ### What is agent experience (AX)? How well AI agents can use your product to do your users' jobs: whether they find it, call the right tools, finish correctly, at a reasonable cost, without unsafe actions. It's the agent counterpart of developer experience. ### How do I improve agent experience? Measure first: run your users' jobs in the agent apps they use and score them. Then fix what the runs show, usually discovery (names, instructions), first steps (a start-here tool), result size, error wording and spend safety. ### What are signs my API has an AX problem? Agents saying your product isn't connected, answers in your UI's words instead of your API's values, “data not available” for data you return, repeated calls, oversized results, and runs that spend or write without asking. ### How is AX different from MCP server quality? Server checks (list size, errors, annotations) are part of it. AX also needs agent runs, because the same server behaves differently in Claude Code, Codex and OpenCode. **Get an AX baseline for your product.** We run your users' jobs through Claude Code, Codex and the other apps they use and report all seven metrics, with ranges and the fixes that move them. [Book a pilot →](/#contact) ## Keep reading - Up[How to make your MCP server or API agent-ready, with proof (guide hub)](/guides/agent-ready-tools/) - Related[Cost per successful task: the one number that shows if your tools are agent-ready](/blog/cost-per-successful-agent-task/) - Related[Can cheaper models use your MCP server? Small vs frontier](/blog/cheaper-models-mcp-reliability/) - Related[How to measure MCP server quality: the agent scorecard (2026)](/blog/measure-mcp-server-quality/) - Related[Sample agent-readiness report on a made-up demo server: findings, fixes and pass/N before and after](/sample-report/) - In this guide[How AI assistants pick which developer docs to cite: 80 answers from Claude and Codex](/blog/how-ai-assistants-cite-developer-docs/) - In this guide[Do AI agents read llms.txt? 18 runs in three agent apps](/blog/llms-txt-agents-md-what-agents-use/)