There are a lot of confident numbers floating around about llms.txt adoption. I wanted to see the real picture, so I measured it directly.
I crawled the top 10,000 sites in the Majestic Million, pulled every llms.txt that exists, and read what was inside. Here is what the data shows, including a few things I did not expect.
The short version of the method: I fetched /llms.txt for the top 10,000 domains in the Majestic Million in June 2026, followed redirects so apex and www both count, and kept only files that were genuinely Markdown rather than HTML. One caveat up front: the top 10,000 by backlink profile skews toward established, technical, SEO-aware sites, so read this as one well-defined slice, not the whole web.
New to llms.txt? Start with my plain-English guide to what it is and how to write one. This piece is about what the data says.
How I measured llms.txt adoption
A status code alone isn’t enough to trust. Plenty of sites return a clean 200 OK for /llms.txt and then serve their homepage or an error page as HTML, with no real file behind it. That’s a soft 404: the server reports success, but there’s nothing actually there. Count those as adopters and the number gets inflated.
Across the top 10,000, 1,050 sites returned a 200 for /llms.txt, but about a third of them (313) turned out to be soft 404s, serving an HTML page rather than a real Markdown file. The 737 that served an actual file are the real adopters.
So the real number, once you strip those out, is 737 sites, or 7.4%.
Everything below uses that cleaned figure. If a study reports a much higher number, it’s worth asking whether they filtered out the soft 404s.
How widespread is adoption?
Adoption numbers for llms.txt swing wildly with the sample. Ahrefs, looking at a traffic-weighted set of sites in their analytics (which skews technical and SEO-aware, and which they call an upper bound), reported 28%. This study looks at the top 10,000 sites by backlink profile and finds 7.4%. Same screening standard, but very different populations, which is why the numbers land so far apart. What is useful here is less a single universal percentage and more the shape of adoption inside a defined, high-authority slice of the web.
And the shape held a surprise: adoption barely tracks with rank.
The top 100 lead at 11%, but after that it flattens out and wobbles between 6% and 9% all the way down to rank 10,000, and the 2,000 to 5,000 tier actually adopts more than the 500 to 1,000 tier. So this is less a clean “bigger sites adopt more” story and more a thin, even layer of early adopters spread across the whole top tier, which reads to me like something still driven by individual awareness rather than any kind of industry default.
Most llms.txt files are custom, for now
This is the finding I keep coming back to. Of the 737 real files, only 12% are generated by an SEO plugin, and the other 88% have no plugin fingerprint at all. I am calling those custom rather than handcrafted on purpose, since some are genuinely written by hand while others are produced with AI or some other tool. The accurate way to put it is just that they did not come from one of the SEO plugins that auto-generate this file.
- Custom 648 (88%)
- Yoast 59 (8%)
- AIOSEO 22 (3%)
- Rank Math 8 (1%)
I detected generators by their own attribution line, the one plugins write into the file, like Generated by All in One SEO or Generated by Yoast SEO. That is far more reliable than guessing from a site’s tech stack, and it caught a few files that the lazier method would have mislabeled.
Among the plugins, Yoast leads by a wide margin: 59 files, versus 22 for AIOSEO and 8 for Rank Math. All three are general-purpose SEO plugins, which suggests plugin-driven adoption is riding on tooling sites already run rather than anything llms.txt-specific.
That 12% is the number to watch, because custom files appear one decision at a time while plugin adoption jumps every time a plugin ships the feature on by default. The day a major plugin flips that switch, this chart changes overnight.
Only just over half use the full structure
A file existing is not the same as a file being useful, so it is worth being precise about what “good” means. The spec is short. The only strictly required element is an H1 with the site or project name. On top of that it recommends a blockquote summary, then one or more sections of curated [name](url) links, with an optional section literally labeled Optional for things that can be skipped.
I checked every real file for that full recommended shape: a title, a summary, and at least one section of real links. Just 55% have all three. The other 45% are missing pieces: no summary, no real sections, or a flat wall of links with no structure. Auto-generated files are a big part of that gap. A plugin dumping every page, post, and category into one list technically produces a file, but it is not the curated map the format was designed to be.
That gap is also the opportunity. If you are going to publish one, a good one still puts you ahead of nearly half the sites that already have a file.
Blocking the crawlers they invited
This is the one I found genuinely funny.
llms.txt exists to help AI crawlers understand your site. So I cross-referenced every adopter’s robots.txt to see whether they also block the major AI bots. 57 of the 737, about 8%, publish an llms.txt while actively blocking AI crawlers in robots.txt.
A lot of them are publishers: weather.com, coindesk, msnbc, hackernoon, linktr.ee. They have rolled out a welcome mat for AI and bolted the front door at the same time. In most cases I would bet the two decisions were made by different people who never talked to each other, which is its own lesson about how this stuff actually gets deployed.
If you take one operational thing from this post: whatever you decide about AI crawlers, decide it once and make robots.txt and llms.txt agree.
What’s inside a typical llms.txt file
A couple of quick texture notes for anyone building one.
Size is wildly bimodal. The median file is about 7 KB, a tidy curated guide, but the range runs from a literal empty file to one that is 38 MB, which is what happens when a CMS dumps an entire site into a single text file. The value is in the curation rather than the sheer volume.
The section headings reveal the emerging conventions. The most common is Optional, which is straight from the spec. After that you see the CMS fingerprints (Pages, Posts, Categories) and then the genuinely useful curated sections (Resources, Products, Features).
And who is publishing? Technology and SaaS companies lead by a wide margin, followed by media and publishing, e-commerce, and education.
Should you publish an llms.txt?
Before anything else, be clear about what it is for. In June 2026 Google confirmed that Search does not use llms.txt, and that having one will neither help nor hurt your Google rankings. The audience is the AI tools and agents that opt to read it, plus checks like Chrome’s Lighthouse agentic-browsing audit. This is not a ranking hack, just a way to give the systems that do read it a clean map of your site.
With that in mind, yes, it is worth doing, on two conditions.
First, make it a real one. Curate it. Title, summary, a handful of well-chosen sections pointing at the pages you actually want represented. You will already be in better shape than the 45% who shipped something malformed.
Second, make your signals agree. If you are publishing an llms.txt, do not turn around and block the crawlers in robots.txt. Pick a posture and make it consistent.
I built two free tools for exactly this. The llms.txt generator helps you write a clean, spec-correct file in a few minutes. The AI crawler access checker shows you which AI bots your robots.txt currently allows or blocks, so you can catch the contradiction before you ship it.
Frequently asked questions
What percentage of websites have an llms.txt file?
In the top 10,000 sites by backlink profile, 7.4% publish a real llms.txt, which is 737 sites. That figure only counts genuine Markdown files. Of the 1,050 sites that returned a 200 for /llms.txt, 313 were soft 404s serving an HTML page with no real file behind them, so they were left out.
Why is this adoption number lower than others I have seen?
Two reasons: the sample and the screening. Ahrefs reported 28% from a traffic-weighted set they describe as an upper bound, which skews technical and SEO-aware. This study measures the top 10,000 by backlinks and removes soft 404s. Different populations and stricter filtering give a lower number. There is no single universal adoption rate, only the rate inside a defined slice.
Do bigger sites adopt llms.txt more often?
Barely. The top 100 lead at 11%, but after that adoption flattens to between 6% and 9% all the way down to rank 10,000, and the 2,000 to 5,000 tier adopts slightly more than the 500 to 1,000 tier. It reads as a thin, even layer of early adopters rather than a clear size advantage.
Are llms.txt files usually written by hand or generated by a plugin?
Most are custom. Of the 737 real files, 88% have no SEO-plugin fingerprint, and only 12% were generated by a plugin. Among plugins, Yoast leads with 59 files, ahead of AIOSEO at 22 and Rank Math at 8. Plugin adoption is the number to watch, because it can jump the day a major plugin turns the feature on by default.
How many llms.txt files are actually well-structured?
About 55%. That share has the full recommended shape: a title, a summary, and at least one section of real curated links. The other 45% are missing pieces or dump every page into a flat list. Having a file is common, but having a good one still puts you ahead of nearly half the sites that publish one.
Do some sites block the AI crawlers their llms.txt is meant for?
Yes. 57 of the 737 adopters, about 8%, publish an llms.txt while blocking major AI crawlers in robots.txt, including publishers like weather.com and CoinDesk. Usually that is two teams making opposite calls without talking. Whatever you decide about AI crawlers, set it once and make robots.txt and llms.txt agree.
Does Google use llms.txt?
No. In June 2026 Google confirmed that Search does not use llms.txt and that publishing one will neither help nor hurt your rankings. The audience is the AI tools and agents that choose to read it, plus checks like Chrome's Lighthouse agentic-browsing audit, not Google ranking.
A note on the data
This is a snapshot of the top 10,000 Majestic Million domains, crawled in June 2026. I plan to re-run it and track how these numbers move, because the interesting part of an emerging standard is the slope rather than any single reading. If you want the underlying adopter list or have a question about the method, get in touch.
Working through the same shift?
I write about SEO, GEO, and getting found by AI search.
If this was useful, I'd love to compare notes.