Research
Original studies on SEO and AI search
First-party data on how the shift to AI search is actually playing out. I crawl, measure, and publish the numbers, with the method in the open so you can check the work.
Who Blocks AI Crawlers? robots.txt in the Top 10,000 Sites
Original research on AI crawler blocking: how often the top 10,000 sites allow or block GPTBot, ClaudeBot, CCBot, Google-Extended and other AI bots in robots.txt, by bot and by site size.
llms.txt Adoption: Data From the Top 10,000 Sites
Original research on llms.txt adoption across the top 10,000 websites: real adoption rates, who publishes one, custom vs plugin-generated files, and the sites blocking the AI crawlers they invited.
