What Is llms.txt — And Should You Actually Add It in 2026?

llms.txt is a plain-text Markdown file you place at the root of your site (/llms.txt) that hands AI models a curated map of your most important pages, the same way robots.txt hands crawlers a set of rules. The five-second version: add the file, list your best URLs, and in theory tools like ChatGPT, Claude, and Gemini understand your site faster. The honest version, and the reason this guide exists, is that as of 2026 almost no major AI crawler actually reads it — so whether you should bother depends entirely on what kind of site you run.

This is the full picture: what the file is, how to write one correctly, and the server-log evidence that decides whether it's worth ten minutes of your time or a distraction from the work that actually wins AI visibility.

What llms.txt actually is

The standard was proposed by Jeremy Howard on September 3, 2024. The idea is simple: web pages are cluttered with navigation, ads, and scripts that waste an LLM's limited context window, so give the model a clean, human-curated index instead of making it parse your HTML.

The spec is short. A valid llms.txt file contains, in order:

• An H1 with the site or project name — the only required element.
• A blockquote summary giving the key context a model needs to understand the rest of the file.
• Optional Markdown prose (paragraphs, lists — no headings).
• H2 sections containing file lists: each entry is a Markdown link [name](url), optionally followed by : and a short note.
• An optional section literally named Optional, marking links a model can skip when context is tight.

A minimal example:

# TechRiseUps

> Independent tech reviews and hosting guides, written by engineers who ship.

## Guides
- [Best VPS hosting 2026](https://techriseups.com/articles/best-vps-hosting-in-2026-ranked-by-real-price-performance): price/performance ranking
- [Generative Engine Optimization](https://techriseups.com/articles/generative-engine-optimization-geo-2026-get-cited-ai-overviews-chatgpt-perplexity): how to get cited by AI

## Optional
- [About](https://techriseups.com/about)

The proposal also suggests publishing clean .md versions of your pages (append .md to the URL) and an llms-full.txt that inlines the full content for models that want everything in one fetch.

llms.txt vs robots.txt — they are not the same thing

This trips people up constantly. robots.txt is a set of directives: it tells crawlers what they may and may not fetch, and well-behaved bots obey it. llms.txt is a suggestion: a curated reading list with no enforcement and no requirement that anyone honor it. One controls access; the other politely offers a map. Nothing breaks if a model ignores llms.txt, and — as the data below shows — ignoring it is exactly what most of them do.

How to create one (the easy part)

If you decide it's worth doing, shipping a file takes minutes:

1. Write the file following the structure above. Lead with a tight summary; link only your genuinely important pages, not your whole sitemap.
2. Save it as llms.txt and upload it to your web root so it resolves at https://yoursite.com/llms.txt.
3. Skip the manual route if you're on WordPress — Yoast SEO now generates llms.txt automatically, and Webflow and other platforms have added native support.
4. Keep it curated. A bloated file that dumps every URL defeats the purpose; the whole value proposition is editorial selection.

That's the entire how-to. The setup has never been the hard part. Whether it does anything is.

The uncomfortable part: do AI crawlers actually use it?

Here is the section most "add llms.txt today!" tutorials leave out. The server logs are brutal.

Ahrefs analyzed 137,000 domains in May 2026 and found that 97% of llms.txt files received zero requests. Of roughly 38,000 sites with valid files, nearly none saw any bot fetch them. Google's John Mueller, reviewing server logs, noted that the major AI services "don't even check for it." Ahrefs' verdict was blunt: it's "a solution in search of a problem," with "no evidence that llms.txt improves AI retrieval, boosts traffic, or enhances model accuracy."

The citation data is no kinder. A March 2026 SE Ranking study, covered by Search Engine Journal, examined 300,000 domains and found no relationship between having an llms.txt file and how often a domain gets cited in major LLM answers. Adoption sat at about 10% and was essentially flat across traffic tiers. Most telling: among the 50 most-cited domains in AI answers, only one had an llms.txt file. The sites winning AI citations are winning despite not having the file, not because of it.

The Google split: Search says no, agentic browsing says "maybe later"

Google's position is genuinely two-faced, and it's worth understanding the split. On the Search side, Gary Illyes stated in July 2025 that Google has no plans to support llms.txt and won't crawl it for ranking — normal SEO is what feeds AI Overviews. (The irony wasn't lost on anyone when an llms.txt briefly surfaced on Google's own developer docs in December 2025, then vanished the same day.)

But on the agentic side, the story shifts. Chrome's Lighthouse documentation, updated in May 2026, describes llms.txt as "an emerging convention used to provide a machine-readable summary of a website's content," and notes that without it "agents may spend more time crawling the site to understand its high-level structure." The framing matters: this is about AI agents browsing in real time — the kind of automated assistants that fetch your docs to complete a task — not about search ranking. Chrome is careful to call the file "optional at the moment," and its Lighthouse audit only flags an actual server error, treating a missing file as "Not Applicable."

So the use case that has any traction isn't SEO at all. It's making your documentation cheaper for AI agents to consume when one is actively working through your site.

So should you add llms.txt in 2026?

A straight answer, by site type:

Your site	Worth adding llms.txt?
Dev tool / API / docs-heavy product	Yes — agents fetch your docs; a clean map genuinely helps them
SaaS with technical documentation	Probably — low cost, plausible upside as agentic browsing grows
Marketing site chasing AI Overviews	No measurable benefit — invest in content and SEO instead
Blog / publisher	Optional — harmless, but don't expect citation lift

The cost-benefit is lopsided in a specific way: the file is trivial to create and carries essentially no downside, but the demonstrated upside today is close to zero outside the documentation-for-agents case. If you run a developer tool whose users point AI assistants at your docs, ship one — it's cheap insurance for a future that may arrive. If you're adding llms.txt hoping to rank better in ChatGPT or Google's AI Overviews, the evidence says your time is better spent on the fundamentals that actually earn AI citations: clear, well-structured, genuinely useful content that search and LLMs already reward.

Add it if it's free to do and fits your stack. Don't add it expecting magic, and don't let it distract you from the work that moves the needle.

FAQ

Does Google use llms.txt? No. Google Search confirmed in July 2025 it has no plans to support or crawl llms.txt for ranking. Separately, Chrome's Lighthouse docs reference it for real-time AI agents, not search — two different things.

Will llms.txt help me rank in AI Overviews or ChatGPT? There's no evidence it does. A 300,000-domain study found no link between having the file and AI citation frequency, and only one of the 50 most-cited domains had one.

Is llms.txt the same as robots.txt? No. robots.txt gives crawlers binding access rules; llms.txt offers an optional curated content map with no enforcement. They solve different problems.

Is it worth adding anyway? For docs-heavy products and dev tools whose users run AI agents against your site, yes — it's cheap and plausibly useful. For most marketing sites, it's optional and shows no measurable SEO benefit today.

What's the difference between llms.txt and llms-full.txt? llms.txt is a curated index of links; llms-full.txt inlines the full text of those resources into a single file, so a model can ingest everything in one request without following links.