AI crawler access checker

The checker reads the robots.txt file at the root of the domain you enter. It then tests the crawlers that matter for AI search against the rules in that file, using the same matching approach crawlers use: the most specific user agent group wins, and for a path the longest matching rule wins.

We split the crawlers into two groups. Search and answer crawlers, such as OAI SearchBot, ChatGPT User, PerplexityBot and Claude SearchBot, decide whether you can appear in AI answers. Training crawlers, such as GPTBot, ClaudeBot and Google Extended, decide whether your pages are collected to train models.

Most businesses want to be allowed in the first group. Whether to allow the second is a business decision, and the answer differs for a publisher, a clinic and a software company.

• Allowed, blocked or partly limited for each crawler
• The exact rule that decided the result
• Sitemaps listed in robots.txt
• Whether an llms.txt file is published

Run the check whenever you need to know, in plain terms, whether the bots behind AI answers are allowed to read your site. The most common triggers are simple. Your brand never appears when you ask ChatGPT or Perplexity about your own category. A developer, a security plugin or a host changed something and nobody wrote down what. You are about to relaunch or migrate. Or you are choosing a robots.txt policy and want to see what is in force today before you change it.

It is also the right first step before any larger GEO project, for a reason that is easy to forget. Content work only pays off if the crawlers can fetch the pages. A single blanket rule can make every later improvement invisible to the engines that need to see it. A check takes seconds, and it removes the cheapest, most common cause of absence before anyone spends money on anything else.

Run it again after every change to your CDN, firewall, security plugin, SEO plugin, hosting plan or site platform. These layers write or override robots.txt without announcing it. Some hosts add bot rules by default. A check after the change costs nothing and catches the surprise early.

You can also point it at other sites. Checking a competitor or a well known publisher shows how others handle the same decision. Remember that it reads only public rules, so use it for learning, not for guessing at anyone's strategy.

• Before a relaunch or migration
• After a security, CDN or SEO plugin change
• When your brand is missing from AI answers
• Before choosing a training policy

Understanding the matching logic stops you misreading a result. The tool follows the approach set out in the robots exclusion standard and in Google's documentation. It splits the file into groups. A group is one or more User-agent lines followed by Allow and Disallow rules. A bot uses the group that names it most specifically. If no group names it, the bot uses the group marked with an asterisk. If there is no matching group at all, the bot is allowed.

Inside the chosen group, the tool tests the path of your home page, which is the slash. It finds every rule whose pattern matches that path. The longest matching pattern wins. If an Allow and a Disallow match with equal length, Allow wins. An empty Disallow line means nothing is blocked. Wildcards with an asterisk and the dollar sign end marker are understood.

One consequence surprises people. A bot that has its own named group ignores the asterisk group completely. So if you add a group for OAI-SearchBot that says Allow everything, any rule you wrote under the asterisk group no longer applies to that bot. This is why the robots.txt generator repeats your private path rules inside each allowed bot group.

The status labels follow from this. Allowed means the home page path is not blocked. Blocked means a Disallow rule covers it. Partly limited means the bot has its own group and some folders are closed while the home page is open.

A corrected robots.txt is not felt the moment you publish it. OpenAI's documentation says changes to robots.txt can take about 24 hours to take effect for search. Perplexity's documentation gives a similar figure of up to 24 hours. Treat that as the order of magnitude and not a promise. After a fix, the checker will show the new rule at once, because it reads your live file, but the engines will take longer to recrawl and re index.

Our checker caches each result for ten minutes, so if you edit the file and check again within that window you may see the old answer. Wait ten minutes, then re check. If the file is served by a cache or CDN with a long lifetime, the delay can be longer. Add a cache purge to your change checklist.

Appearing in an answer then depends on more than access. A crawler must fetch the page, the engine must judge it relevant and trustworthy for the question, and the answer must have room to name you. Do not expect a citation the day after a fix. Test the same buyer questions in two to four weeks and compare.

Four limits matter in practice. First, the tool tests only the home page path. A site could allow the home page and block a folder that holds all your service pages. The Partly limited label catches many such cases for bots with their own group, but it does not test every URL. For one page, open Google Search Console's robots.txt report and URL inspection, or test the path by hand.

Second, it does not read crawl delay lines, IP allow lists or the headers your server sends. Anthropic notes that it supports a non standard crawl delay rule, and OpenAI recommends allowing its published IP ranges if you filter by address. Neither shows in this tool.

Third, if the request for robots.txt is itself refused by a firewall, the result can look like a missing file. A 403 or a challenge page seen by our checker may be reported as a 4xx note, which reads as open. If you can load the file in your browser but the tool says it is missing, suspect bot protection.

Fourth, it cannot confirm that a vendor obeys the file. OpenAI and Perplexity both state that user triggered fetchers may not follow robots.txt in the usual way. The only evidence of actual behaviour is your server log. Our guide on AI crawlers and robots.txt shows how to read it.

If the result shows a block you did not intend, build a corrected file with the AI crawler robots.txt generator, then re run this checker. If an llms.txt file is missing, read llms.txt: do you need one before you spend time on it. For deeper crawl, rendering and indexing problems, see our technical GEO page.

In the GEO Pilot, crawler access is part of Week 1, and fixing crawler access and robots rules is part of Week 2. We test with the access checker, with real requests and with your logs where you can share them. The Pilot is $999 fixed, one time, and delivered inside 30 days, on any platform. We make no promise of rankings, citations or traffic. See the GEO Pilot page for the full scope.