AI Crawler Access Scanner
Checks whether your robots.txt blocks AI bots like GPTBot or ClaudeBot, limiting your AI visibility.
What it checks
This scanner checks whether your robots.txt file blocks AI bots, such as OpenAI's GPTBot or Anthropic's ClaudeBot. It detects rules that prevent those crawlers from accessing your content.
Why it matters
If you block AI crawlers, those assistants can't read your site or cite you in their answers, reducing your visibility right where search is growing. Sometimes this block is enabled by default or by copying templates unintentionally.
How to fix it
Review your robots.txt and remove the rules blocking AI bots if you want your content to appear in their answers. If you prefer to allow only some, adjust the directives to grant access to the crawlers you care about.
Check it on your site
Scan your site free and see if this and 44 other checks pass, with a fix prompt for every issue.
Scan my site free