Blog / Test MCP servers with agents
MCP benchmark and testing tools compared: load testing vs agent-quality testing (2026)
Search for MCP testing tools and you'll find roundups written by the vendors in them. The useful question isn't which tool is best. It's which question each tool answers: does the server speak MCP, does it hold up under load, does a model pick the right tool, or do your users' agents get the job done? Those need different tools.
Pick by question, not by brand
An MCP server can fail in five different places, and each needs a different kind of test. Before choosing a tool, choose the question:
- Protocol: does the server start, list its tools and return valid messages?
- Logic: does each tool do the right thing with good and bad input?
- Load: does it hold up with many sessions at once?
- Model behaviour: does a model pick the right tool with the right arguments?
- Agent outcome: do your users' actual agent apps get the job done?
Disclosure: Spirit Room wrote this, and we sell the fifth kind as a service. The other tools below are worth using, and most are free.
The comparison
| Tool | Answers | Model in the loop? | Runs real agent apps? | Notes |
|---|---|---|---|---|
| MCP Inspector | Protocol | No | No | Official, free, web UI + CLI for CI checks |
| MCPJam | Protocol, model evals | Yes | No; emulates client profiles | Open core; some AI features use its hosted service |
| SDK in-memory client, FastMCP + pytest | Logic | No | No | Ordinary unit tests |
| k6 + xk6-mcp | Load | No | No | Experimental extension, not officially supported |
| mcpbench, mcp-load-lab | Load | No | No | Per-tool latency percentiles, CI gates |
| mcp-eval | Model evals | Yes | No | Open-source scenarios and CI |
| Promptfoo | Model evals, security | Yes | No | Strong on red-teaming |
| Braintrust, LangSmith | Model and agent evals | Yes | Only agents you build | General eval platforms, not MCP-specific |
| MCPMark, MCP-Universe, MCP-Bench | Model ranking | Yes | Mostly no | Fixed servers and tasks; they test models, not your server |
| Spirit Room (service, the author) | Agent outcome | Yes | Yes: Claude Code, Codex, Grok Build, OpenCode | Your real jobs, pass/N with ranges, read-only on production |
Protocol inspectors
MCP Inspector is the official reference client: connect over stdio, SSE or Streamable HTTP, list and call tools, debug OAuth, and script checks from its CLI. It's free and tracks the spec first. It doesn't run a model, so it can't tell you whether an agent will use your tools well.
MCPJam started from the same idea and grew into a workbench: inspection, an LLM playground, eval suites, generated test cases and CI reports. Its client “compatibility” runs are emulations; MCPJam's own guide says emulated client profiles are not production sessions.
Load testing
MCP sessions have state: an initialize handshake, a tools/list, then calls. A plain HTTP benchmark skips most of that. xk6-mcp adds MCP clients to k6 (experimental). mcpbench and mcp-load-lab are small, free load testers that report latency percentiles and fail CI on a budget. None of them involve a model, and none should: load is about your servers, not agent behaviour.
Find where agents fail on your tools. Measure what fixes it.
Bring your MCP server, CLI or API and the jobs your users need done. We run real agents against them, read every call, and hand back measured gaps and fixes.
Model evals
A model eval connects a model to your server through an SDK, runs a task list and scores the tool calls and answers. MCPJam evals, mcp-eval and Promptfoo do this for MCP; Braintrust and LangSmith are broader platforms for evaluating agents you build. This is the right place to compare tool descriptions quickly and cheaply.
What it can't see is what the agent app does before and after the model: whether it loads all tools or searches them, how much of your instructions it keeps, and what happens to a 46,000-token result. That's where the next category comes in.
Benchmarks rank models, not your server
MCPMark (127 tasks with programmatic checks), MCP-Universe (231 tasks across 11 real servers) and MCP-Bench (28 servers, 250 tools) are research benchmarks. They're useful for choosing a model and for task-design ideas, such as checking final state instead of using an LLM judge. They don't test your server unless you rebuild them around it.
Testing in real agent apps
Your users don't call your server through an SDK. They use Claude Code, Codex, Cursor and others, and those apps change what the model sees. On a 99-tool production server that passed every protocol check, live read-only runs showed:
- Codex never reached the tools in 6 of 40 runs (7–29%) and tried a browser instead.
- Claude Code refused 40 results in 23 of 40 runs for exceeding its 25,000-token limit.
- A test setting that stripped Claude Code's built-in tools also turned off its tool search, multiplying the first request about five times. A setup no user has.
None of these show up in a protocol check or a model-only eval. The method for this kind of testing is in how to test an MCP server with real AI agents, and the metrics in how to measure MCP server quality.
A sensible stack
| When | Run |
|---|---|
| Every commit | Unit tests with an in-memory client; a tools/list snapshot with token counts |
| Every pull request | MCP Inspector CLI checks; a model eval on your top tasks |
| Before launch and capacity changes | A load test with the full session flow |
| Before releases and after description changes | Agent run testing in the apps your users run, with repeats |
Go deeper: MCP Inspector says it works; agents still fail, Evaluate AI agents on your API, Pass rates and confidence intervals.
Sources
- Tool pages linked in the table, checked 21 Oct 2026.
- MCP server testing tools compared, 2026, an independent roundup.
- MCPJam: best MCP testing tools and Braintrust: best MCP testing tools, vendor roundups.
- Spirit Room live read-only runs on a production MCP server, 16 Oct 2026.
FAQ
What tools can test an MCP server?
Protocol: MCP Inspector and MCPJam. Unit tests: the official SDKs' in-memory clients or FastMCP with pytest. Load: k6 with xk6-mcp, mcpbench or mcp-load-lab. Model evals: MCPJam evals, mcp-eval, Promptfoo, Braintrust or LangSmith. Agent behaviour in real apps: run Claude Code, Codex and the other apps your users have, on real jobs, with repeats.
Is there a load testing tool for MCP servers?
Yes. Grafana's xk6-mcp adds MCP clients (stdio, SSE, Streamable HTTP) to k6, though it is marked experimental and not officially supported. mcpbench and mcp-load-lab are free, MCP-specific load testers with latency percentiles and CI gates.
What is the difference between MCP benchmarks and MCP testing tools?
Benchmarks such as MCPMark, MCP-Universe and MCP-Bench measure how well models use a fixed set of servers, so they rank models. Testing tools check your own server: whether it speaks the protocol, holds up under load, and gets used correctly.
Is MCPJam better than MCP Inspector?
They overlap on protocol inspection. MCPJam adds model evals, test generation and CI features, some of which rely on its hosted service; MCP Inspector is the free reference client and doesn't run models. Neither runs the actual Claude Code or Codex apps your users use.
Do I need to test my MCP server in Claude Code and Codex specifically?
If your users run them, yes. Each app decides which tools the model sees, how instructions are cut and how big results are handled. In our runs Codex missed a server's tools in 6 of 40 runs and Claude Code refused 40 oversized results; a model-only eval wouldn't have shown either.
Find where agents fail on your tools. Measure what fixes it.
Bring your MCP server, CLI or API and the jobs your users need done. We run real agents against them, read every call, and hand back measured gaps and fixes.