--- title: Your Agent Test May Not Test Your MCP: 13/36 Used a Browser description: In 13 of 36 live runs Codex drove a signed-in browser instead of calling MCP, and the sandbox let it. How to spot agents bypassing your MCP server and block it. canonical: https://spiritroom.app/blog/agents-escape-to-the-browser/ published: 2026-10-21 updated: 2026-10-21 --- > Markdown version of https://spiritroom.app/blog/agents-escape-to-the-browser/ for agents. Index: /llms.txt [Blog](/blog/) / [Test MCP servers with agents](/guides/test-mcp-servers-with-agents/) # Your agent test may not be testing your MCP: 13 of 36 runs drove the browser instead You write a test job, the agent answers, the answer is right. You count it as a pass for your MCP server. But did the agent use your server? In our first live pass, more than a third of runs didn't. Leonid Bugaev · Published 21 Oct 2026 **TL;DR** - **13 of 36 runs** (22–52%) in our first live pass used a browser-automation skill to click through the web app instead of MCP. - **The sandbox didn't stop it:** the app's read-only sandbox doesn't block localhost, where the browser bridge listened. - **Answers can still look right,** so a pass rate that counts them measures the browser, not your server. - **Later, with the browser blocked,** Codex still missed the tools in 8 of 80 runs and said “no connection is available”. Those are real MCP failures. - **Fix:** disable browser and shell tools for test runs, and fail any run that uses them. [What happened](#what)[Why agents go around MCP](#why)[How to detect it](#detect)[How to block it](#block)[What it means for your users](#users)[How we measured](#how-we-measured)[FAQ](#faq) ## What happened Our first live pass on a 99-tool production MCP server was 36 Codex runs of read-only jobs, inside Codex's read-only sandbox, through a read-only proxy. A browser-automation skill was still enabled on the machine. Runs that drove the browser *13/36 (22–52%)* In those 13 runs Codex used the skill to drive a signed-in Chrome over a localhost port and clicked through the product's web app. It read the numbers off the screens. Nothing changed in the account. But none of those runs tested the MCP server. We threw the whole pass out of scoring and re-ran with the side door shut. ## Why agents go around MCP - **They didn't find the tools.** Codex loads MCP tools through search; when the request doesn't name the server, the search can miss. See [when the agent doesn't recognise your server](/blog/agent-does-not-recognise-mcp-server/). - **Another route is right there.** A browser skill, a shell with `curl`, or web fetch can all reach the product. - **The sandbox has gaps.** A read-only file sandbox isn't a network sandbox. Localhost bridges and signed-in browsers sit outside it. **Make sure your agent tests test your MCP.** We run your users' jobs through the agent apps with the side doors closed, and fail any run that goes around your server. [Test your MCP →](/#contact) ## How to detect it - Count MCP tool calls per run from the wire or transcript. An answer with zero MCP calls is a red flag. - Flag every shell, browser and web-fetch call. - Look for phrases like “no connector is available” or “I'll open the app”. ## How to block it 1. **Disable browser features** in the agent app for the run. In Codex we pass `--disable browser_use`, `--disable in_app_browser` and `--disable browser_use_external`, and disable the browser skill. 2. **Limit tools in Claude Code** with `--allowedTools` for your MCP read tools and `--disallowedTools` for Bash, WebFetch, WebSearch and the browser skill. 3. **Turn off other MCP servers** for the run. 4. **Fail the run** in the scorer if it touched a shell or browser, whatever the answer. Ours has a safety check named “used MCP, not shell or browser”. The whole safe setup: [test agents against production read-only](/blog/read-only-agent-testing-production/). Scoring pitfalls: [validate your scorer](/blog/validate-your-agent-eval-scorer/). **Go deeper:** [Why agents call the wrong tool](/blog/why-agents-call-the-wrong-tool/), [How Codex finds MCP tools](/blog/how-codex-finds-mcp-tools/), [Pass rates with ranges](/blog/agent-pass-rates-confidence-intervals/). ## What it means for your users Your users won't disable anything. If the agent can't find your tools, it'll try the browser, or tell them your product has no connection. After we blocked the browser, Codex still missed the tools in **8 of 80** runs (5–19%) and said no connection was available. Those misses are about your server's discoverability, and they're worth fixing. ## How we measured - **First pass:** 36 Codex CLI runs, read-only sandbox, read-only proxy, browser skill enabled, 16 Oct 2026. Not used for scoring. - **Re-runs:** browser features off, any browser or shell use fails the run; 40 Codex runs per round over two rounds. - **Limits:** one server, one machine; ranges are 95% Wilson intervals. The skill vendor and product are not named. ## FAQ ### Why does my agent bypass my MCP server? Usually because it didn't find or recognise the tools, or another route looked easier. If a browser skill or shell is available, the agent may use the web app or the API directly instead. ### Does a read-only sandbox stop browser use? Not necessarily. In our runs Codex's read-only sandbox didn't block a browser bridge on localhost, so the agent drove a signed-in browser. ### How do I know whether a run used MCP? Check the transcript or wire log for MCP tool calls, and flag any shell, web-fetch or browser tool calls. A run with an answer but no MCP calls didn't test your server. ### Should I count browser runs as failures? In a test of your MCP server, yes. Score them separately from wrong answers so you can see both problems. **Make sure your agent tests test your MCP.** We run your users' jobs through the agent apps with the side doors closed, and fail any run that goes around your server. [Test your MCP →](/#contact) ## Keep reading - Up[How to test an MCP server with real AI agents (guide)](/guides/test-mcp-servers-with-agents/) - Related[How to test agents against a production MCP server without spending, writing or sending anything](/blog/read-only-agent-testing-production/) - Related[Validate your agent eval scorer: 0 of our first 151 failures were the server's fault](/blog/validate-your-agent-eval-scorer/) - Related[Why AI agents call the wrong tool: 7 causes we saw in recorded runs](/blog/why-agents-call-the-wrong-tool/)