Agent experience (AX): how to measure it, not just define it
Agent experience is usually defined and rarely measured. Here's how we measure it: the same user jobs, run several times in the agent apps your users actually use, scored against ground truth, with every number traced to a run.
What AX is
Agent experience is how well AI agents can use your product to do your users' jobs. The user asks their agent; the agent has to find your product, pick the right tool, call it right, recover from errors, and answer, without spending, writing or sending what the user didn't ask for.
It's measurable. The unit is a user job, run several times in each agent app your users use, scored against ground truth.
Seven metrics
| Metric | Question | Example from our runs |
|---|---|---|
| Discovery rate | Did the agent find your tools? | Codex 34/40 (71–93%), Claude Code 40/40, live |
| Right first step | Did it start where you intended? | Claude 1/24 → 24/24 with a start-here recipe, mock |
| Pass/N with range | Was the answer right? | Codex 34/40 (71–93%), Claude 32/40 (65–90%), live |
| Cost per success | What does a done job cost? | 178k–460k tokens per job, Codex, live |
| Recovery | After an error, did it get back on track? | Every errored run recovered on a mock; errors 4/30 → 0/30 after fixes |
| Silent failures | Wrong but confident? | 3 of 5 runs said data wasn't exposed when it was, live |
| Safety breaches | Spend, write or send without asking? | 10/75 → 2/75 runs after a redesign, mock |
Report every rate as pass/N with a 95% range: why pass rates need ranges. Cost per success: cost per successful task.
Get an AX baseline for your product.
We run your users' jobs through Claude Code, Codex and the other apps they use and report all seven metrics, with ranges and the fixes that move them.
Signs you have an AX problem
- Agents say your product isn't connected, or ask “which platform?”: when the agent doesn't recognise your server.
- Answers use your screen labels where the API has other values: UI words vs API values.
- Agents call the wrong tool, or the same tool twice: why agents call the wrong tool.
- Results get cut off or sent to a file: output limits by agent.
- Agents stop and ask the user about a typo: agent refusal states.
- Agents act before the user agrees: the two-step confirm pattern.
Most of these fail silently. The answer looks fine unless you check it against ground truth.
How to measure AX
- Jobs: 5–10 real user requests, in users' words, without naming your tools.
- Apps: the agent apps your users run, each with default settings.
- Safety: production read-only through an allowlist, or a mock for write paths: test against production read-only.
- Runs: 5+ per job per app; ground truth from fixed calls.
- Score: the seven metrics; read failing transcripts by hand: validate your scorer.
- Fix and re-run: the metrics should move. See the sample report.
Go deeper
- How to measure MCP server quality: the 15-point checklist.
- Cheaper models and MCP reliability.
- What agents use: llms.txt, AGENTS.md, MCP.
- MCP server scorecard and token calculator.
Go deeper: How AI assistants cite developer docs, Glossary, Agent-ready tools.
FAQ
What is agent experience (AX)?
How well AI agents can use your product to do your users' jobs: whether they find it, call the right tools, finish correctly, at a reasonable cost, without unsafe actions. It's the agent counterpart of developer experience.
How do I improve agent experience?
Measure first: run your users' jobs in the agent apps they use and score them. Then fix what the runs show, usually discovery (names, instructions), first steps (a start-here tool), result size, error wording and spend safety.
What are signs my API has an AX problem?
Agents saying your product isn't connected, answers in your UI's words instead of your API's values, “data not available” for data you return, repeated calls, oversized results, and runs that spend or write without asking.
How is AX different from MCP server quality?
Server checks (list size, errors, annotations) are part of it. AX also needs agent runs, because the same server behaves differently in Claude Code, Codex and OpenCode.
Get an AX baseline for your product.
We run your users' jobs through Claude Code, Codex and the other apps they use and report all seven metrics, with ranges and the fixes that move them.