AI crawler robots.txt generator

The biggest mistake we see is treating every AI bot the same. They fall into three groups, and your decision differs for each.

Search crawlers build the index that an AI product uses to find and link to pages. If you block them, your pages may not appear as sources in that product. OpenAI uses OAI-SearchBot for ChatGPT search results. Perplexity uses PerplexityBot. Anthropic uses Claude-SearchBot.

User fetchers visit a page because a person asked an assistant to look at it. OpenAI calls this ChatGPT-User, Perplexity calls it Perplexity-User, and Anthropic calls it Claude-User. Vendors note that these requests are triggered by a user, so some may not follow robots.txt the way scheduled crawlers do. Read each vendor page for the current position.

Training crawlers collect content that may be used to train models. OpenAI uses GPTBot. Anthropic uses ClaudeBot. Common Crawl uses CCBot, and its dataset is used by many model builders. Apple offers Applebot-Extended and Google offers Google-Extended as opt out controls for AI training and related uses.

Robots.txt controls crawling, not indexing. A blocked page can still appear as a bare URL if other sites link to it. To keep a page out of search results, use a noindex directive and leave the page crawlable so the directive can be seen.

It also does not control how Google uses your content in AI Overviews and AI Mode. Google states there are no extra requirements for those features beyond normal search eligibility: pages must be indexed and eligible to show a snippet. The Google-Extended control covers Gemini training and grounding, not Search.

Finally, it is voluntary. Reputable crawlers follow it. A scraper that ignores it will not be stopped. If you need hard enforcement, use authentication or server level rules.

Use the generator when you need to set a clear policy for AI crawlers and do not want to write the file by hand. The usual cases are a site with no robots.txt, a site whose file was written years ago and knows nothing about AI bots, a site where the access checker showed an accidental block, and a business that has made a decision, for example allow search, block training, and needs it written correctly.

It is a good fit when your needs are ordinary. You want the answer crawlers in, you have a short list of private folders to keep out, and you have one sitemap. It is a weaker fit for complex setups: a site with many subdomains and different policies, a file that serves other crawlers with detailed rules, or a platform that generates robots.txt dynamically and will overwrite your upload. In those cases, use the output as a reference block and merge it by hand.

The generator does one job. It writes the AI crawler section of a file and a catch all group for everyone else. It does not read your current file, so it will not merge with what is already live. Before you replace a file, copy the old one somewhere safe and compare the two line by line.

The output begins with two comment lines. Then it lists every allowed crawler as a User-agent line, one per bot, all stacked above a single Allow: / rule. In robots.txt, several User-agent lines in a row share the rules beneath them, so this is one group, not many. Any private paths you entered follow as Disallow lines.

Next comes the blocked group. Every blocked crawler gets a User-agent line, followed by one Disallow: / rule. After that comes the group for everyone else, which uses User-agent: * and either your private paths or a bare Disallow: line, which means nothing is disallowed. Last is the Sitemap line, which stands outside any group and applies to the whole file.

The repeated private paths are deliberate. A bot with its own named group ignores the asterisk group. If the generator only put your private folders under the asterisk, an allowed AI bot would be free to crawl them. By repeating the rules in each allowed group, the file keeps the bots out of the folders you named.

Comments start with a hash and are ignored by crawlers. You can change the wording of the two header comments. Keep the rest as generated, unless you know the format well.

The most common reason a generated file does not take effect is that the platform writes its own. Check where yours comes from before you upload anything.

On WordPress, the file is often virtual. WordPress generates it on the fly when no physical file exists, and SEO plugins can edit it from a settings screen. A physical file in the site root will override the virtual one. Security and performance plugins may also add lines. If you paste into a plugin, remember the plugin may keep adding its own rules above yours. See GEO for WordPress for the full picture.

On Shopify, robots.txt is produced from a template and can be customised through the robots.txt.liquid file, which keeps the default rules and lets you add groups. On a custom or headless build, the file is usually a static asset or a route in the application, and a deployment can replace it. Add a test for it to your release checklist.

If you use a CDN or a managed security service, check its bot settings too. Some of these services can serve their own robots.txt or block unknown bots at the edge. The generated file will not help if the edge refuses the request before it reaches your server.

The generator cannot tell you which choice is right for your business. It only writes the file. The decision about training use involves your content, your contracts and your risk appetite, and it should be made by someone who owns that decision.

It cannot make a crawler obey. Anthropic says ClaudeBot, Claude-User and Claude-SearchBot honor robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User because users trigger those visits. Perplexity says Perplexity-User generally ignores robots.txt. These are the vendors' own statements, read in October 2026, and they can change. If you need a firm block for private content, use authentication.

It does not verify that a vendor has a current bot with that name. We reviewed the names against vendor documentation, but vendors add, rename and retire bots. Review the list at least once a quarter. When a new bot shows in your logs, read its vendor page before you allow or block it.

It also does not cover Google's AI features in Search. Google's documentation says there are no additional requirements to appear in AI Overviews or AI Mode, beyond being indexed and eligible to show a snippet. Google-Extended does not affect Search inclusion or ranking, according to Google. So blocking or allowing it will not change your standing in those features.

Last, it does not handle crawl delay. Anthropic says it supports a non standard crawl delay extension. If a bot is overloading your server, add that line by hand, and consider rate limiting at the server.

The long form of this decision is in AI crawlers and robots.txt: what to allow. If you also want to publish an overview file for AI tools, the llms.txt generator is the companion tool, though that file is optional. The wider crawl and rendering picture is on our technical GEO page.

In the GEO Pilot, we set crawler access and robots rules in Week 2, after the Week 1 access check, and we confirm the effect with a re test in Week 4. It is a fixed $999, one time, inside 30 days, and we do the implementation with your approval on WordPress, WooCommerce, Shopify or custom code. We cannot promise any ranking, citation or traffic result. Read more on the GEO Pilot page.