There's a new failure mode in the AI-agent era, and it's not in your code: the security layer in front of your API is quietly deciding that your customers' AI assistants are bots — and blocking them. We hit this in production while verifying a customer MCP deployment against Claude, and the diagnosis pattern is worth writing down.
What it looks like
- Your customer adds your MCP URL to Claude. Sign-in works. Then: "Couldn't connect."
- Your server logs show the OAuth handshake… and nothing after it.
- curl from your laptop works perfectly. The AI client fails every time.
That mismatch — residential traffic passes, AI-client traffic vanishes — is the signature. AI clients identify themselves honestly: Claude's user-initiated calls carry the Claude-User user agent from published Anthropic IP ranges; ChatGPT uses ChatGPT-User. Honest identification is exactly what AI-crawler blocking matches.
Why this got worse in 2026
CDNs shipped anti-AI-crawler features to protect content sites from training scrapers — a reasonable goal. But the same switch that blocks a training crawler also blocks a user-initiated agent doing work your customer asked for. If you sell an API, an MCP server, or any AI-facing surface, "block AI bots" now means "block my customers".
The fix, step by step (Cloudflare)
- AI Crawl Control: set user-initiated agents —
Claude-User,ChatGPT-User— to Allow. You can keep training crawlers (ClaudeBot,GPTBot) blocked; they're different signatures. - Security → Settings → "Block AI bots": if the legacy managed rule is on, scope it away from your API/MCP paths.
- Security → Events: confirm nothing on your MCP paths shows "Block" in the last hour, then have a real client retry.
- If customers use custom domains for their MCPs: document this — they will hit the same trap on their own Cloudflare zone.
Diagnosing it in minutes instead of hours
The reason this class of bug burns a day is that the evidence is an absence. Two habits fix that:
- Log every request, including rejections. If auth failures and rate-limit rejections are visible, "requests arrive and fail" vs "requests never arrive" becomes a one-glance distinction.
- Log client IP and user agent. Seeing
Claude-Userfrom a datacenter IP on the working path, and nothing on the broken one, points straight at the edge.
Hosted gateways carry this scar tissue for you
FAQ
- Why is Cloudflare blocking Claude from reaching my API or MCP server?
- Cloudflare's AI-bot features (AI Crawl Control, the legacy 'Block AI bots' managed rule, Bot Fight Mode) match AI-client signatures such as the Claude-User user agent and datacenter egress IPs. They were designed to stop training crawlers, but they also block legitimate, user-initiated AI traffic — like your customer's Claude session calling your MCP server.
- How do I allow Claude and ChatGPT through Cloudflare without disabling bot protection?
- In Cloudflare's AI Crawl Control, set user-initiated agents (Claude-User, ChatGPT-User) to Allow while keeping training crawlers blocked if you wish. Review Security → Settings → 'Block AI bots' and scope it away from your API/MCP paths, and check Security → Events to confirm nothing on those paths is being blocked.
- How can I tell edge blocking apart from an auth bug?
- Edge blocks leave zero server logs: OAuth may succeed (different egress/signature) while tool calls vanish. If your server logs every request including auth failures and you still see nothing while the client errors, look at the CDN's security events — not your code.