Do you need an llms.txt file?
You don’t need an llms.txt file for Google. Google’s own AI features guide says Search ignores the file entirely, so publishing one will neither help nor hurt your rankings. The file is real and some AI agents do read it, but it was never designed as a search optimisation tool.
The pitch you keep seeing calls llms.txt “the new sitemap for AI” and warns that sites without one will be invisible to assistants within a year. Nobody selling that has produced evidence for it, and the person who wrote the specification never claimed it. If you have been putting the job off while every newsletter tells you that you are already behind, you are not. The confusion is worth untangling, because one Google product now audits the file while another says it is ignored.
What is an llms.txt file supposed to do?
An llms.txt file is a Markdown file at the root of a site that lists the site’s most important pages, so a language model can find them without reading every page of HTML. Jeremy Howard published the llms.txt proposal in September 2024 and revised it in August 2026. The format is deliberately plain: a heading with the site name, an optional summary, then sections of links with a line of description each.
The proposal is explicit about how this differs from robots.txt. Robots.txt tells automated tools what they may access, and crawlers fetch it before they fetch anything else. An llms.txt file is meant to be read on demand, when an agent already working on your site needs to find its way around. Nothing goes out looking for these files.
Does Google read your llms.txt file?
No. Google’s guide to generative AI features says you don’t need to create machine-readable files, AI text files, markup or Markdown to appear in Google Search, because Search itself doesn’t use them. The same page adds that an llms.txt file will neither harm nor help your visibility, since Search ignores it.
Google added that wording in June 2026 after enough people asked, and the page carries a July 2026 update date. That is about as direct as Google’s documentation ever gets. So if you added the file hoping for a lift in AI Overviews, you can stop watching for it.
Why did Lighthouse flag llms.txt on your site?
Lighthouse flags llms.txt only when the URL returns a server error, not when the file is missing. The Agentic Browsing audit in Lighthouse tries to retrieve the file from your root, and when nothing is there and the URL answers 404, the audit is marked Not Applicable, because the file is optional. A real failure means your server broke while answering the request.
This is where two Google teams look like they contradict each other. Chrome is measuring how ready a site is for browser agents that visit and act on it. Search is answering a different question entirely: does this file change what appears in search results? Both answers are right for the question being asked, and only one of them is about your traffic.
Will an llms.txt file get you into ChatGPT or AI Overviews?
There is no published evidence that any major AI search product uses llms.txt to find or rank content. Google states plainly that it ignores the file. No other large AI company documents reading it during search retrieval, and none of them publish their crawl logs, so anyone telling you it works is describing a hope rather than a measurement.
What assistants demonstrably do is fetch your actual pages. That puts the decision back on the file that controls access, which is still robots.txt. To see whether a named bot such as OAI-SearchBot can reach a given URL, the Robots.txt Tester takes your rules, the URL and a user-agent name, and reports which line decided the outcome.
Who actually gets value from publishing one?
Documentation sites do. Anthropic’s documentation and Cloudflare’s developer docs both serve an llms.txt, and the reason is the same in each case: their readers point coding assistants at those pages while working, and a compact index of the docs saves the assistant a dozen fetches. The file earns its place when an agent is already inside your site with a job to finish.
A blog with forty posts about garden furniture is not that. Nobody is going to aim an agent at it and ask it to work through the archive. When your content gets found through search instead, an index written for agents does nothing your navigation and your sitemap aren’t already doing.
What does Google say to do instead?
Google’s advice for AI features is its advice for Search, almost word for word: write unique, genuinely useful content, and give the site a clear technical structure that can be crawled and indexed. The guide goes further and says structured data isn’t required for generative results either, though it still earns rich results in ordinary search.
There is already a machine-readable index that search engines do read, and it is your XML sitemap. It has a documented protocol behind it, Search Console reports on what happened to it, and Google fetches it without being asked twice. If yours is missing or years out of date, the XML Sitemap Generator builds one from your URLs. That is an afternoon better spent than hand-writing a file nothing has requested.
Does publishing an llms.txt file cost you anything?
The cost is not writing the file. It is keeping it true. A hand-written list of links drifts the moment you delete a page or change a slug, and a stale llms.txt hands an agent your worst answer with total confidence. The specification was itself revised in August 2026, two years after it first appeared, so a file written against the original version may already be out of step with it.
Nothing in Google’s documentation suggests a penalty for having one. Publish an llms.txt because a specific tool you use reads it and you have checked that it does. Publishing one because an article announced that 2026 is the year of AI search is how sites end up maintaining files nothing has ever asked for.
How do you check what your /llms.txt URL returns?
Request the URL and read the status code, because three very different outcomes look the same in a browser window. A 404 means no file at all, which is fine, and is the result Lighthouse treats as not applicable. A 500 is the single case the audit flags. The third is the sly one: a 200 that hands back an HTML page, because your CMS answered an unknown path with a catch-all template rather than a text file.
The Bulk HTTP Status Code Checker accepts a list of URLs, so you can paste the addresses of your llms.txt, your robots.txt and your sitemap together and read all three responses side by side. That check is worth running after any server or hosting change, whether or not llms.txt interests you.
Final Thoughts
Publish an llms.txt if you can name the tool that will read it and you intend to keep it current. Otherwise, leave the root of your site as it is. Google has told you in writing that the file makes no difference to search, and the author of the specification never pitched it as SEO. That part was added by people with something to sell.
Frequently Asked Questions
No. Robots.txt controls access, and every major crawler fetches it before anything else, so the rules inside it are enforced. An llms.txt file grants nothing and blocks nothing. It is a content index a model may read if something points it there, and no search engine requires it.
No. An XML sitemap follows a protocol search engines have supported for years, it reports back through Search Console, and Google fetches it on its own. An llms.txt file has none of that behind it. Keep the sitemap, and treat llms.txt as a separate thing for agents that ask.
Markdown, served as plain text from the root of your site. The specification asks for a heading carrying the site name, an optional short summary, then sections of links with a brief description on each line. A wall of prose or a bare list of URLs does not match the format.
The specification allows a subpath too. A file at /docs/llms.txt covers the URLs beneath that path, which suits a documentation section inside a larger site. The copy at the root is the one a tool looks for first, so put it there if you are only publishing one.