How this check works
Scanaris downloads your /robots.txt (if it does not exist or is HTML, it does nothing) and parses it by groups, the way crawlers do: each group is one or more User-agent lines followed by their rules. For six specific AI crawlers, GPTBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended and CCBot, it applies the real rule of the standard: if the bot has its own group, only that group counts and the * wildcard is ignored for it; if it does not, it inherits the * group. A bot is blocked if the group that applies to it contains Disallow: /. If any is, it opens an informational finding listing which ones and explaining whether it is through a dedicated block or through a User-agent: * with Disallow: / that sweeps them all up. It is informational because blocking them can be a legitimate decision.
What each bot does
They are not all the same, and it is worth knowing before deciding. GPTBot is OpenAI's crawler for training models; OpenAI also has OAI-SearchBot, which feeds ChatGPT search, and ChatGPT-User, which visits pages when a user asks. ClaudeBot and anthropic-ai are Anthropic's. PerplexityBot crawls for Perplexity's search engine, which cites its sources. Google-Extended is not a crawler: it is a switch controlling whether Google may use what it already crawled with Googlebot to train Gemini; blocking it does not affect your search ranking. CCBot is Common Crawl, a public archive of the web that many models use as training data.
What to decide
The question is what you want. If you want to appear when someone asks an assistant about your category, you need them to read you: assistants recommend what they know and cite what they can visit. If what worries you is your content training models without compensation, you can block the training bots and leave the search ones, which is the distinction OpenAI lets you make between GPTBot and OAI-SearchBot. For most product sites, blocking assistants means giving up a growing channel; for a content site whose business is the content itself, the answer may be different.
To allow assistants
Remove the Disallow: / lines from their groups, or remove the groups. If the block comes from a User-agent: * with Disallow: /, that block is also shutting out Google: review it with robots and sitemap.
To block training only
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Bear in mind that robots.txt is a request, not a barrier: serious crawlers honour it; the others do not. If you need to block for real, it has to be done at the server or the CDN by user agent.
Check it is fixed
curl -s https://yourdomain.com/robots.txt | grep -iE -A3 "user-agent: *(\*|gptbot|claudebot|anthropic-ai|perplexitybot|google-extended|ccbot|oai-searchbot)"Check which Disallow follows each group. A group without Disallow: / allows crawling.
Frequently asked questions
My host or CDN blocks AI bots by default.
Some do (Cloudflare offers it as an option), and that does not show in robots.txt: the bot gets a 403 directly. If you want visibility in assistants, turn it off in the provider's panel. This check only sees the file.
Does blocking them save bandwidth?
Some, but less than it seems: these crawlers are usually moderate. If cost is the reason, rate-limit instead of blocking.
I allow them and assistants still do not know me.
Letting them read you is the necessary condition, not the sufficient one. The rest, being in the sources models learn from, is measured by AI visibility, and why so many AI-built sites are invisible is explained in this article.