llms.txt is a simple idea borrowed from robots.txt. You place a Markdown file at your domain root that points language models to your most important pages in clean, readable form. The pitch is that instead of crawling cluttered HTML, a model reads a tidy summary you wrote yourself.
What the file actually is
It is a plain Markdown file at /llms.txt. At the top, a short description of what your site is. Below it, grouped links to the pages you most want understood: docs, key product pages, pricing, comparisons. Optionally a fuller version that inlines the content itself, so a model does not even need to follow the links.
# YourBrand > One honest sentence about what the product does and for whom. ## Core pages - [Pricing](https://example.com/pricing): Plans from $X/mo, what each includes. - [Product](https://example.com/product): What it does, in plain language. ## Guides - [Guide title](https://example.com/guides/one): One-line summary a model can trust.
What it does, honestly
At the time of writing, no major answer engine has committed to reading llms.txt as a ranking input. It is a proposal with growing adoption, not a standard the big models obey. Treat it as low-cost hygiene: it cannot hurt, it makes your site easier to parse, and it positions you well if support arrives.
- Do add it if you have real docs or a content-heavy site. It is an afternoon of work.
- Do keep it honest and current. A stale map is worse than no map.
- Do not expect it to move answers on its own today. It is not a growth lever yet.
Know your crawlers
The AI traffic hitting your site is not one thing, and the robots.txt decisions are different for each kind. Three groups matter. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect data for future models. Retrieval bots (OAI-SearchBot, PerplexityBot) fetch pages live to answer a user's question right now. And classic search crawlers (Googlebot, Bingbot) still matter because answer engines lean on search indexes to decide what to retrieve.
Blocking a training crawler keeps you out of the next model version's memory. Blocking a retrieval bot keeps you out of today's cited answers, which is usually the opposite of what a brand wants. The most common self-inflicted wound we see is a WAF or bot-protection preset that silently 403s every non-Google crawler — the site looks fine in a browser while every AI engine sees a wall. If you want to be recommended, verify each of these bots gets a 200 from your key pages.
What actually moves answers now
The files models genuinely read are the boring ones: a robots.txt that does not block them, a complete sitemap, fast server-rendered HTML, and structured data on the pages that matter. Those are doing the heavy lifting while llms.txt is still optional.
Ship llms.txt because it is cheap and tidy, not because it is the lever. The lever is still clear, quotable, crawlable pages.
A sane order of operations
Make sure you are crawlable and rendered, add structured data where it earns its keep, then add llms.txt as the finishing touch. Doing it in that order means the work that actually counts is done before you reach for the file that might count later.
