Blog / Test MCP servers with agents

Your agent test may not be testing your MCP: 13 of 36 runs drove the browser instead

You write a test job, the agent answers, the answer is right. You count it as a pass for your MCP server. But did the agent use your server? In our first live pass, more than a third of runs didn't.

Leonid Bugaev · Published 21 Oct 2026

What happened

Our first live pass on a 99-tool production MCP server was 36 Codex runs of read-only jobs, inside Codex's read-only sandbox, through a read-only proxy. A browser-automation skill was still enabled on the machine.

In those 13 runs Codex used the skill to drive a signed-in Chrome over a localhost port and clicked through the product's web app. It read the numbers off the screens. Nothing changed in the account. But none of those runs tested the MCP server.

We threw the whole pass out of scoring and re-ran with the side door shut.

Why agents go around MCP

• They didn't find the tools. Codex loads MCP tools through search; when the request doesn't name the server, the search can miss. See when the agent doesn't recognise your server.
• Another route is right there. A browser skill, a shell with curl, or web fetch can all reach the product.
• The sandbox has gaps. A read-only file sandbox isn't a network sandbox. Localhost bridges and signed-in browsers sit outside it.

How to detect it

• Count MCP tool calls per run from the wire or transcript. An answer with zero MCP calls is a red flag.
• Flag every shell, browser and web-fetch call.
• Look for phrases like “no connector is available” or “I'll open the app”.

How to block it

1. Disable browser features in the agent app for the run. In Codex we pass --disable browser_use, --disable in_app_browser and --disable browser_use_external, and disable the browser skill.
2. Limit tools in Claude Code with --allowedTools for your MCP read tools and --disallowedTools for Bash, WebFetch, WebSearch and the browser skill.
3. Turn off other MCP servers for the run.
4. Fail the run in the scorer if it touched a shell or browser, whatever the answer. Ours has a safety check named “used MCP, not shell or browser”.

The whole safe setup: test agents against production read-only. Scoring pitfalls: validate your scorer.

Go deeper: Why agents call the wrong tool, How Codex finds MCP tools, Pass rates with ranges.

What it means for your users

Your users won't disable anything. If the agent can't find your tools, it'll try the browser, or tell them your product has no connection. After we blocked the browser, Codex still missed the tools in 8 of 80 runs (5–19%) and said no connection was available. Those misses are about your server's discoverability, and they're worth fixing.

How we measured

• First pass: 36 Codex CLI runs, read-only sandbox, read-only proxy, browser skill enabled, 16 Oct 2026. Not used for scoring.
• Re-runs: browser features off, any browser or shell use fails the run; 40 Codex runs per round over two rounds.
• Limits: one server, one machine; ranges are 95% Wilson intervals. The skill vendor and product are not named.