- A block can sit in five places: robots.txt, your CDN or bot management settings, a web application firewall, your host, and a security plugin. A clean robots.txt does not prove a bot can reach your pages.
- A help article dated 21 October 2026 from a publisher platform says Cloudflare enabled AI bot blocking by default for free plan users and leaves it off for paid plans. Confirm in your own dashboard, because settings and names change.
- The proof is in your server logs. Look for OAI-SearchBot, GPTBot and other bot names and read the status codes. A 200 means served. A 403 or 429 means blocked or throttled.
How can I tell if Cloudflare or a firewall is blocking ChatGPT?
Read your logs for the bot names you care about and look at the response codes. If OAI-SearchBot requests return 200, the page was served. If they return 403, 429 or a challenge page, something is blocking or throttling the bot, and it does not matter what your robots.txt says. If the bot never appears in the logs at all, either it has not tried yet or a layer in front of your server is dropping it before your server sees it.
This matters for one specific reason. OpenAI's crawler overview, which we read in October 2026, says OAI-SearchBot is used to surface sites in ChatGPT's search features, and that sites that opt out will not be shown in ChatGPT search answers. A block you did not intend is an opt out you did not choose. Most small businesses we speak to have never looked. We have not measured how many sites block these bots by accident, and we will not quote a figure.
Where can a block hide?
| Layer | What it does | How it blocks AI bots | Where to look |
|---|---|---|---|
| Robots.txt | Public rules for crawlers | A Disallow rule for a bot name or for all bots | yourdomain/robots.txt |
| CDN and bot settings | Sits in front of your server | Bot fight or AI bot blocking features, rate limits, managed rules | Your CDN dashboard, security section |
| Web application firewall | Filters requests by pattern | Rules that challenge unknown user agents or data centre addresses | Firewall rules and event logs |
| Hosting security | Server level protections | Automatic blocks on bursts of requests or unknown bots | Host control panel, security logs |
| Security plugin or CMS setting | Runs inside your site | Rules that return 403 for listed user agents | Plugin settings, .htaccess file |
What do we know about Cloudflare specifically?
We know less than we would like, and we say so. A help article from the publisher platform Raptive, dated 21 October 2026, says Cloudflare announced that free plan users will have AI bot blocking enabled by default, that paid plans do not get it automatically, and that free users can change the behaviour in their dashboard. It lists bots such as GPTBot, ClaudeBot and CCBot among those blocked. The article does not name the dashboard setting or give a start date. We could not open Cloudflare's own documentation from our research environment. So treat this as a lead to confirm, not a fact to rely on.
The practical lesson holds either way. If your site sits behind a CDN, open its security or bot settings and read what is switched on. Look for anything described as blocking AI crawlers, AI bots or scrapers, and for managed rules that challenge unknown automated traffic. Then decide for each bot whether you want it blocked.
Step 1: Read your robots.txt
Open the file and look for the bot names. Note whether a blanket Disallow applies to all agents and whether a specific group overrides it. Use the free AI Crawler Access Checker for a quick read, and see our test of how parsers treat rule order and wildcards in testing robots.txt for AI bots. Also remember that a standard tool can mislead you about duplicate groups.
Step 2: Check your CDN or bot settings
Log in to your CDN and open the security and bots sections. Write down every feature that is on. If you find a setting that blocks AI crawlers, read its description, then decide whether to switch it off or to allow specific bots. If you do not manage the CDN yourself, ask whoever does, and send them the list of bot names and your decision for each.
Step 3: Check firewall rules and events
Open the firewall event log and filter by user agent or by action taken. Look for blocked or challenged requests whose user agent contains OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot and similar names. If you find them, open the matching rule and see why it fired. Some rules block all requests from data centre addresses. That is a common reason a legitimate crawler gets caught.
Step 4: Read your server logs
If you have access logs, search for the bot names and tabulate the status codes. A simple count by bot and status code is enough. Many shared hosting panels offer raw logs for download. If you use a managed WordPress host, ask support for the logs of the last 30 days.
| Status code | What it usually means for a bot |
|---|---|
| 200 | Served normally |
| 301 or 302 | Redirected. Check that it lands on the right page |
| 403 | Forbidden. A rule or a plugin is blocking |
| 429 | Too many requests. A rate limit is throttling the bot |
| 503 | Server unavailable or a protection layer is challenging |
| No requests at all | Not visited yet, or blocked before reaching your server |
A caution on testing by hand. You can send a request with a bot name in the user agent from your own computer, but many firewalls verify bots by their published address ranges, so a hand made request may be treated differently from the real crawler. Use the logs for the real behaviour.
Step 5: Check security plugins and platform settings
On WordPress, look at security and firewall plugins and at the .htaccess file for rules that match user agents. On Wix, the robots.txt editor lives under the SEO section of the dashboard, and Wix documents examples of blocking GPTBot, ChatGPT-User and others, so check nobody added them. For Shopify, one article we read says the default file allows most major AI bots and that blocks usually come from rules someone added to the robots.txt.liquid template, but that article cited no Shopify documentation, so check your own template. See the platform by platform settings guide.
After you fix it: how do you confirm?
- Save a dated copy of the old settings and of the new ones.
- Wait at least a day. OpenAI says a robots.txt update can take about 24 hours to take effect for search.
- Check the logs again for the bot names and the status codes.
- Re run your buyer prompts with a fixed method. See why AI answers change between runs.
- Repeat the log check monthly, because CDN and host defaults change.
What should you write down for your developer or host?
Send one short message: the list of bots you want to allow, the list you want to block, the reason for each, and the date. Ask them to confirm in writing which layers they changed. This avoids the common situation where one person edits robots.txt, another changes the CDN and a third installs a security plugin, and nobody knows which rule wins.
What should you do next?
Spend 30 minutes today on the five layers above, in order. If you find a block, fix it and note the date. Then see why you may still be missing in the eight causes to check first. If you would rather have this found and fixed for you, crawler access checks and technical fixes are in weeks one and two of the GEO Pilot, a fixed $999 over 30 days.