Named crawlers
Assistants read the web through their own user agents, each of which can be allowed or blocked independently of Googlebot.
- GPTBot and OAI-SearchBot
- ClaudeBot and PerplexityBot
- Google-Extended
Free check
Can AI assistants
read your site?
A site can rank perfectly in search and still never appear in an assistant's answer, because one line in robots.txt excludes the crawlers that feed them.
One request to your robots.txt. Nothing is crawled and nothing is stored.
Why this matters
Being unreadable is
a silent failure.
No error appears anywhere. Rankings hold, traffic looks normal, and the site is simply absent from a channel that keeps growing.
Usually accidental
Most blocks are inherited from a template or added years ago for a different reason, and nobody revisits robots.txt.
- A wildcard rule catches everything
- Copied boilerplate
- No warning anywhere
Access is only the start
Being readable is necessary, not sufficient. What matters is whether assistants name you when someone asks.
- Mention rate over time
- Position against competitors
- Which sources got cited
Beyond the check