Guides
How search engines and AI services read your website: robots.txt, AI crawlers, llms.txt and what actually works.
Besides Google, AI services such as ChatGPT, Claude and Perplexity now read websites: for their search, for fetches on behalf of users and for training their models. Which crawlers these are and what you allow them is controlled in a few places.
The articles in this topic explain the crawlers per provider, what the robots.txt looks like for them, and what an llms.txt can and cannot do. Every statement about a provider rests on its own documentation with an access date.
How your website does on this, the free website check from Helferlain shows in about a minute.
SEO and AI visibility
How agencies use an SEO report in a first meeting: a conversation guide, what the report should contain, the proposal and the email marketing rules.
· 9 min read
SEO and AI visibility
From indexability to AI visibility: what a technical SEO audit checks in ten areas, with a typical finding and the matching action for each one.
· 10 min read
SEO and AI visibility
GPTBot, ClaudeBot, Google-Extended and others: which AI crawlers come for search, user fetches or training, and how to control them in your robots.txt.
· 10 min read
SEO and AI visibility
llms.txt is a proposal, not a standard. What goes in the file, what Google, OpenAI and Anthropic say about it, and when it is still worth having one.
· 8 min read
