- llms.txt is a proposal, not a standard that search engines have confirmed they use. It is not a ranking factor.
- It is cheap to make and harmless if done honestly, so it is a reasonable extra for documentation heavy sites. It is never a substitute for good pages.
- Spend your first hours on crawlable pages, clear answers and robots.txt access. Add llms.txt last.
The short answer
No, you do not need an llms.txt file. It is an optional proposal, and no major search company has confirmed that it uses the file to rank or cite pages. You can publish one in about ten minutes if you like, but fix your content and crawler access first.
That answer disappoints people who read that it is the new robots.txt. It is not. robots.txt is an established convention that crawlers follow. llms.txt is a suggestion from one author, adopted by some sites, with unproven effect on visibility. This post separates what is known from what is hoped.
What llms.txt is
The idea was proposed in September 2024 by Jeremy Howard of Answer.AI. The file is written in Markdown and sits at the root of your site, at /llms.txt. It gives a language model a short, curated map of your site: who you are, what matters, and links to the best pages, each with a one line note.
The motivation is practical. A model working with limited context cannot read a whole website, and HTML pages carry a lot of navigation and script noise. A clean index of your key pages, ideally with plain text versions, could save it effort. That is the theory, and for developer documentation it is a sensible one.
# Example Co
> Example Co is a Dubai based accounting firm for small businesses. We file VAT and corporate tax returns and keep monthly books.
## Services
- [VAT filing](https://example.com/vat-filing/): who needs it, what we file, fees as ranges
- [Corporate tax](https://example.com/corporate-tax/): registration and annual returns
## Company
- [About](https://example.com/about/): team and licences
- [Contact](https://example.com/contact/): email and phoneThe format is simple. A single H1 with the site name, a short summary in a block quote, then sections of links with brief descriptions. An optional section can mark secondary links. Keep it truthful and short.
What the evidence says
Here is the honest state of play. Some AI developer tools and documentation platforms read or generate llms.txt, and a number of technology companies publish one. That makes it useful in coding assistants that fetch library docs.
For general web search and citations, the picture is different. Google representatives have said publicly that Google does not use llms.txt for Search, and that its AI features rely on normal SEO signals. OpenAI, Perplexity and Anthropic have not published confirmation that their search products rank or cite pages based on the file. Several independent tests have not found a clear lift in citations from adding one.
Treat any claim that llms.txt boosts AI visibility as unproven. If a vendor shows you a case study, ask how they separated the file from every other change made on the same site in the same period.
| Question | Honest answer | Confidence |
|---|---|---|
 Is it a Google ranking factor? | No confirmed use for Search or AI Overviews | High |
| Do ChatGPT, Perplexity or Claude require it? | No published requirement | High |
| Can it help developer tools and coding assistants? | Yes, in tools built to read it | Medium |
| Can it hurt? | Only if it is inaccurate or exposes private URLs | High |
| Does it replace robots.txt or sitemaps? | No, it does a different job | High |
| Will it make AI cite me? | No evidence it does | High |
Why people hope it works
The appeal is easy to understand. AI answers feel like a black box, and a small text file that seems to speak to the machine directly looks like a lever. Vendors then sell it as one. The honest position is that a lever needs a connection to the thing it moves, and nobody has shown that connection for general AI search.
There is also a quieter benefit that has nothing to do with rankings. Writing the file forces you to decide which pages best explain your business. Many teams discover that their best page is hard to find, or that two pages say conflicting things. That discovery is worth more than the file itself.
A last practical point: if you do publish one, keep a plain text or Markdown version of key pages if you can. Clean text is easier for any tool to use than a page wrapped in scripts, pop ups and cookie banners.
Who should publish one
Good candidates
- Software companies with developer documentation, an API or an SDK.
- Sites with many long guides where a curated index genuinely helps a reader or a tool.
- Businesses that already have strong pages and crawler access sorted, and want a low cost extra.
Poor candidates
- Sites whose pages are thin, duplicated or blocked from crawling. The file will not rescue them.
- Anyone hoping it will fix an absence from ChatGPT answers. The causes are almost always elsewhere.
- Teams without time to keep it accurate. A stale file listing dead URLs does more harm than none.
How much a file is worth
Step by step: create and publish llms.txt
Step 1. Choose the pages worth including
Pick 10 to 30 pages that best explain what you do: core services, pricing guidance, key guides, about and contact. Leave out login pages, thin tag archives and anything private. Quality of selection is the whole value of the file.
Step 2. Write one honest line per link
Describe what the reader will find, in plain words. Avoid marketing claims you could not defend on the page itself. The summary line at the top should say who you are and what you do in one or two sentences.
Step 3. Format it in Markdown
Use the structure shown earlier. If you would rather not write it by hand, our free llms.txt generator builds the file from a simple form.
Step 4. Upload it to your root
Place it at yourdomain.com/llms.txt and confirm it loads as plain text. Do not block it in robots.txt. Add a reminder to review it whenever you launch or retire a major page.
Step 5. Measure honestly
Record your AI visibility before you publish, using a fixed set of prompts. Recheck monthly. Because many things change at once on a real site, do not credit the file with any movement unless you changed nothing else, which is rare.
| Prompt | Engine | Visibility | Status |
|---|---|---|---|
| best geo agency for small business |  ChatGPT | 82% | Cited |
| how much does geo cost per month | Google AI Mode | 64% | Mentioned |
| generative engine optimization services |  Perplexity | 47% | Mentioned |
| llms.txt do i need it | ChatGPT | 31% | Not shown |
| ai visibility audit checklist |  Gemini | 58% | Cited |
| geo vs aeo vs seo | Google AI Overviews | 22% | Not shown |
The prompt table above is sample data showing how a baseline looks. It is not a client result and it does not show an llms.txt effect.
What to do before llms.txt
If you have limited time, these actions have a far better evidence base than a summary file.
- Make sure your key pages are crawlable and indexed. Our technical SEO service checks this.
- Allow the right AI crawlers in robots.txt. See the post on AI crawlers and robots.txt.
- Put a direct answer under each main heading so a passage can be quoted cleanly.
- Keep your business facts consistent across your site and profiles.
- Add truthful structured data. See schema markup for AI search.
Once those are done, llms.txt is a fine finishing touch. It costs almost nothing, and if tools do start relying on it, you are ready.
Do these before llms.txt
- ✓Key pages crawlable and indexed
- ✓Right AI crawlers allowed in robots.txt
- ✓A direct answer under each main heading
- ✓Business facts consistent across profiles
- ✓Truthful structured data
- Treating llms.txt as a fix for blocked crawlers
- Listing every page in the file
- Claiming it raised citations
Honest limits
- The file is a proposal, so its behaviour can change or vanish.
- No tool can promise a citation from it, and we will not either.
- Fetch logs may show some AI agents requesting it. That shows interest, not influence on answers.
Not sure where you stand? Request a free LLM visibility audit and we will show which fixes matter most for your site.
What the llms.txt format actually requires
Most online examples add rules that the proposal does not contain. The proposal itself, version 2 published by Jeremy Howard on llmstxt.org and dated 3 September 2024, is short. Reading it takes five minutes and removes a lot of confusion.
| Part | Required? | What it is |
|---|---|---|
| H1 heading | Yes, the only required part | The name of the project or site |
| Blockquote | Optional | A short summary with the key context a reader needs |
| Detail text | Optional | Paragraphs or lists, but not headings, explaining the site and how to read the linked files |
| H2 sections of links | Optional | Each section holds a Markdown list. Each item is a link in the form name and URL, optionally followed by a colon and a note |
| A section named Optional | Optional | By convention, secondary links an agent can skip when it needs less context |
The order matters: H1 first, then the summary, then detail text, then the link sections. The proposal also suggests that pages with information an agent may need should offer a clean Markdown version at the same address as the original page, by adding .md to the address. A page at /pricing/ would offer /pricing/index.html.md or a similar clean text copy. That second idea is where most of the practical value sits, because it is an answer to a real problem: HTML is noisy.
What Google has said, with the source
Earlier we said Google representatives have spoken on this. Here is the most quotable instance we could verify. In April 2025, Search Engine Journal reported on a Reddit thread titled with the question of where llms.txt stands. John Mueller of Google replied that, as far as he knew, none of the AI services had said they use it, and that server logs show they do not even check for the file. He compared it to the keywords meta tag, a signal that is a site owner's own claim about what the site is about. The article is dated 17 April 2025.
Two cautions. First, this is a comment from one Google employee on a forum, made in 2025, so it is not formal documentation and it speaks about Google. It says nothing for or against any other engine. Second, products change. If an engine announces support later, the facts change with it. As of October 2026 we have not found official documentation from OpenAI, Anthropic, Perplexity or Google that says their search products rank or cite pages because of this file.
A realistic llms.txt for three kinds of site
The right file depends on what you publish. These examples are made up and use the example.com domain. They show shape, not a template to copy blindly.
A software product with documentation
# Acme Billing
> Acme Billing is invoicing software for small agencies. It sends invoices, takes card payments and syncs with common accounting tools.
The API reference is versioned. Use the latest version unless a page says otherwise.
## Docs
- [Quick start](https://example.com/docs/quick/): create your first invoice in ten minutes
- [API reference](https://example.com/docs/api/): endpoints, authentication and errors
- [Webhooks](https://example.com/docs/webhooks/): events and retry rules
## Optional
- [Changelog](https://example.com/changelog/): dated release notesDeveloper documentation is where the idea is strongest. A coding assistant that fetches docs on demand benefits from a short map and from clean Markdown pages.
A local service business
# Palm Dental Clinic
> Palm Dental Clinic is a family dental practice in Dubai. We offer check ups, orthodontics and emergency care, in English and Arabic.
## Services
- [Treatments](https://example.com/treatments/): what we treat and how a first visit works
- [Fees](https://example.com/fees/): price ranges and what changes them
## Practice
- [Our dentists](https://example.com/team/): qualifications and licence details
- [Contact and location](https://example.com/contact/): address, hours, phoneFor a clinic, the file adds little that a good home page and contact page do not already say. Its value is the discipline of deciding which five pages carry the facts.
An online store
Stores change daily. A hand written file listing products goes stale fast. A better use is a short file that points to category pages, delivery and returns policy, and sizing or care guides, while product data stays in feeds and product markup. If your platform can generate the file from live data, check the output. If it cannot, keep the file limited to stable pages.
Common llms.txt mistakes and how to fix each
- Listing every page. A file with 400 links is a sitemap in disguise. Cut it to the pages that explain the business.
- Marketing language in the summary. Replace claims with plain facts that your pages support.
- Dead or redirected links. Check each URL returns a normal 200 response. Review the file whenever you retire a page.
- Private or staging URLs. Never list anything you would not want a stranger to open.
- Blocking the file. If robots.txt disallows /llms.txt, a tool cannot read it. Keep it reachable.
- Treating it as a fix for access problems. A file cannot let a blocked crawler through. Fix robots.txt and the firewall first.
- Claiming results. Do not tell your team it raised citations unless you changed nothing else, which is almost never true.
How to decide in five minutes
Answer these questions in order. Stop at the first one that gives you a clear answer.
- Do you have developer documentation, an API or a large library of reference guides? If yes, publish a file and clean Markdown copies of the key pages. This is the case with the best fit.
- Are your key pages blocked, thin or hard to render? If yes, fix that first. The file will not help.
- Is your business a small service firm with a dozen pages? A file is optional. Your time is better spent on direct answers and consistent business facts.
- Can you commit to reviewing the file every quarter? If not, skip it. A stale file is worse than none.
- Do you want a cheap extra after everything else is done? Then it is fine. Spend ten minutes and move on.
Should you publish one
Example, hypothetical: how to run a fair test
Suppose a software company wants to know whether publishing the file changes anything. A fair test needs discipline. Freeze a prompt set of 20 buyer questions. Record how five engines answer them, with dates and sources. Publish the file and change nothing else on the site for four weeks. Run the same prompts again. Compare.
Even then, the result is weak evidence. AI answers vary from run to run, and engines update their models and indexes without notice. A difference could be noise. The test is still worth running because it stops you from paying for a claim nobody has proven. It also shows how little a single file moves compared with fixing access or adding a clear answer.
If you cannot freeze everything else on the site for a month, you cannot attribute a change to the file. Say so honestly in any report.
What the file can still do for you
- It forces a decision about which pages best represent the business.
- It exposes conflicts, such as two pages that give different prices or opening hours.
- It creates a place to point tools that do read it, mainly in developer workflows.
- It gives a content team a short, current map to review each quarter.
Those are internal benefits. They are real, and they cost little. They are different from a promise of visibility.
How llms.txt fits the GEO Pilot
In the GEO Pilot, llms.txt appears in the week 2 technical list as a fix we add if it is useful for your site, after crawler access, sitemap, canonicals and structured data. For a documentation heavy site we will usually add it. For a small service business we may skip it and say why. You can build a draft yourself with the free llms.txt generator, and the wider GEO checklist for your website shows where it sits among the other checks. We do not promise citations or traffic from any single change.
