Technical GEO that lets AI crawlers reach and read your site

Not all AI bots do the same job. The big providers now separate training, search indexing and live user requests into different user agents. That lets you make different choices for each. The details below come from each provider's own documentation, read in October 2026. They change, so confirm before you edit production rules.

OpenAI documents three. OAI-SearchBot is used to surface sites in ChatGPT's search features. GPTBot crawls content that may be used to train OpenAI's generative models. ChatGPT-User handles certain user actions in ChatGPT and Custom GPTs, for example when a user asks a question that makes it visit a page. OpenAI says each setting is independent, so you can allow OAI-SearchBot and disallow GPTBot. It also says robots.txt changes can take about 24 hours to take effect for search, and that for ChatGPT-User, because actions are user initiated, robots.txt rules may not apply.

Anthropic documents three. ClaudeBot collects content that may be used for model training. Claude-User visits pages when a Claude user asks a question that needs web access. Claude-SearchBot indexes content to improve search results. Anthropic says its bots honor standard robots.txt directives and that it supports the non standard crawl delay extension. It says it does not publish IP ranges at present and advises against blocking by IP.

Google is different. Google AI Overviews and AI Mode use Google Search. Google's documentation says a page needs to be indexed and eligible to show a snippet to be a supporting link, and that there are no additional technical requirements. Googlebot rules therefore control access. Google-Extended is a separate robots.txt token that controls whether content is used for Gemini training and grounding in some products. Google states it does not affect inclusion in Google Search and is not a ranking signal.

Perplexity documents PerplexityBot, which crawls and indexes content for its search engine and which you can allow or disallow in robots.txt. Microsoft Copilot relies on Bing's index, so Bingbot access and Bing indexing matter. For Bing's answer features, Microsoft documented in September 2023 that the NOCACHE and NOARCHIVE robots directives control how content is used in its chat answers. Check Microsoft's current guidance for any updates.

Structured data is machine readable labelling that sits alongside your visible content, usually as JSON LD using the schema.org vocabulary. It tells a system that a page describes an Organization, a Service, a Product, a Person or a set of questions and answers. That helps with entity clarity, which means making it unmistakable who you are and what you offer.

Be careful with claims. Google's documentation for AI features says no special schema is required to appear in AI Overviews or AI Mode. We do not tell clients that schema forces a citation. We use it because it removes ambiguity, supports rich results where Google offers them, and gives every engine one consistent set of facts.

The rule we follow is simple: markup must match what a visitor can see. Organization details, service descriptions, product data, authors and FAQs are all fair game when they are on the page. Markup for content that is not visible is a risk and a waste. See schema markup for AI search for the longer explanation, and the FAQ Schema Generator for a quick start.

• Organization with name, URL, logo and same as profile links
• Service or Product markup on the matching pages
• FAQ markup only where the FAQ is visible
• Validated, then rechecked after template changes

llms.txt is a proposed convention: a plain text or Markdown file at the root of your site that lists your key pages with short descriptions, written for language models. It is a community proposal, not an official standard of any search engine.

Google's AI features documentation says you do not need new machine readable files or AI text files to appear in AI Overviews and AI Mode. We have not seen the other providers above document that they rely on llms.txt for citations. So we do not sell it as a ranking lever.

We add one when it is cheap and helpful. It can be a useful, tidy map of your best pages for tools and agents that do read it, and it forces you to decide which pages matter. If a client prefers not to publish one, nothing important is lost. The llms.txt Generator creates a draft, and llms.txt: do you need one? gives our full view.

Some crawlers run scripts and some do not. Even those that do may run them later or less completely than a browser. If your service descriptions, prices, reviews or FAQs appear only after client side scripts run, a crawler that fetches raw HTML may see an empty shell. This is common on single page apps and on some headless builds.

The fix is not to abandon your framework. It is to make sure the content that matters is in the HTML the server returns. Server side rendering, static generation, or prerendering of key pages usually does this. For a Shopify or WordPress site, the common cause is a page builder or a plugin that loads blocks late, or tabs and accordions that fetch their content on click. Our post on GEO for JavaScript and headless sites covers the options.

The test is quick. Compare the raw source with the rendered page for your top ten pages. Whatever text is in one and not the other is at risk.

• Main answer text present in view source
• Prices, specs and FAQs not injected after load
• Internal links are real anchor links, not click handlers
• Lazy loaded content not required to understand the page

If the same page exists at several URLs, engines have to guess which to trust, and your signals split. Common causes are tracking parameters, trailing slash variants, http and https versions, www and non www, and filtered product listings. A canonical tag tells crawlers the preferred URL. Redirects send old addresses to current ones.

We check that each important page has one preferred URL, that canonicals point to live, indexable pages, and that redirect chains are short. We also check that the sitemap lists canonical URLs only. These fixes are unglamorous and they matter, because every cited link should resolve cleanly to the page you want people to land on.

One more check is language and region. If you publish in Arabic and English, each version needs its own URL and clear hreflang relationships. Our post on GEO for multilingual sites explains the setup.

A change is not done when it is published. It is done when it is verified. For crawler rules we retest with the checker and a direct request. For schema we run validators and compare to the rendered page. For sitemaps we confirm submission in Google Search Console and Bing Webmaster Tools. For rendering we compare source and rendered text again.

We then keep a change log: what changed, where, when, who approved it, and how to roll it back. If something breaks, you can undo it. That log is part of the Pilot handover, so your developer or any future provider sees exactly what was done and why.

Recrawling takes time. OpenAI says robots.txt changes can take about 24 hours for search. Other engines vary, and we do not claim a fixed timeline. The week 4 retest records what has changed by then and what is still pending.