Cost per successful task: the one number that shows if your tools are agent-ready
“How much do AI agents cost on our API?” Most teams answer with tokens per call, or a pass rate. Neither tells you what a done job costs. One division does.
The metric
cost per success = (input tokens summed over all runs of the job) ÷ (runs that passed)It combines pass rate and tokens per task into one number that goes down only if your server gets cheaper or more reliable.
Live numbers
Codex CLI on a 99-tool production MCP server, 8 read-only jobs × 5 runs.
Input tokens per successful run. The cheapest job per run (plan and credits, 113,803) isn't the cheapest per success, because it failed 2 of 5 times: the agent didn't find the tools.
| All 8 jobs | Passed | Tokens per run | Per success |
|---|---|---|---|
| Codex | 34/40 (71–93%) | 218,032 | ~256,508 |
| Claude Code, strict | 32/40 (65–90%) | 209,722 | ~262,152 |
| Claude Code, label accepted | 37/40 | 209,722 | ~226,726 |
Get cost per successful task for your server.
We run your users' jobs in the agent apps they use and report tokens, passes with ranges, and cost per success per job.
Before and after on a mock
On a made-up task-tracker server every run passed (10/10 per cell), so the cost per success is just tokens per run. Median input tokens per run, before → after fixing descriptions, errors and result sizes:
Details: sample report and cheaper models and MCP reliability.
How to measure it
- Pick your users' real jobs, in their words.
- Run each 5+ times per agent app, with ground truth from fixed calls.
- Record each app's own input-token usage per run (including cached), and score pass/fail.
- Per job: sum tokens ÷ passes. Show the pass rate with its range next to it.
- Re-run after each server change; the number should fall.
Caveats
- With few passes the number is unstable: 2 of 5 could be 1 or 3 next time.
- Apps report tokens with their own tokenizers; compare within an app, not across.
- If no run passes, the cost is undefined: report it as a failure, not a number.
Go deeper: Agent experience (AX), How to measure MCP server quality, MCP context window usage.
Back to agent experience (AX).
How we measured
- Live: Codex CLI 0.161.0 and Claude Code 2.1.301, 8 jobs × 5 runs, 16 Oct 2026, read-only proxy; tokens from each app's usage, summed per run.
- Mock: 90 runs, 3 apps × 3 surfaces × 10, 21 Oct 2026; medians.
- Limits: one server; per-job figures from 5 runs. 95% Wilson ranges.
FAQ
How do I measure the cost of AI agents using my API?
Run each user job several times in the agent apps your users use, sum the input tokens for all runs (passing and failing), and divide by the number that passed. Report it per job and per app, alongside the pass rate with a range.
Why not just tokens per run?
Because failed runs cost tokens too and deliver nothing. In our live data, one job averaged 184k tokens per run but passed only 2 of 5, so each success cost about 460k.
Should I convert tokens to money?
You can, with each vendor's price and cache discount, but much agent input is cached and priced differently. Tokens compare more fairly across time; we report tokens.
Get cost per successful task for your server.
We run your users' jobs in the agent apps they use and report tokens, passes with ranges, and cost per success per job.