You can have a beautiful site that AI simply cannot see. When someone asks ChatGPT, Claude or Perplexity for a tool like yours, the model does not "browse" like a person: it requests pages, reads what it receives and decides whom to cite. If your content does not arrive in that read, you do not exist to AI. This guide explains how AI crawls, how to check in ten minutes whether it sees you, the mistakes that most often get in the way, and how to fix them depending on how your site is built.
How AI crawls your site (and why there are three different bots)
Each provider runs several crawlers for different purposes, and it pays to tell them apart because you can allow some and block others:
- Training.
GPTBot(OpenAI),ClaudeBot(Anthropic),Google-Extended(controls use for Gemini, not search),Bytespider(ByteDance),CCBot(Common Crawl, which many models draw from). They collect pages to train future models. - AI search.
OAI-SearchBot(ChatGPT search),Claude-SearchBot,PerplexityBot. They build the index the assistant consults when it answers with sources. - Live visits.
ChatGPT-User,Claude-User,Perplexity-User: the assistant opens your page on the spot because a user asked it to.
Blocking training while allowing search is a perfectly reasonable choice. Blocking everything by accident, through an inherited template, is what happens to many apps.
Seeing is not the same as rendering
Many vibe coders ship apps with a front end that assembles entirely in the browser with JavaScript (an SPA built with Vite, Lovable, Bolt or v0). To a human it looks perfect. But most AI bots request the HTML and read what arrives before running anything; some never run JavaScript at all. If your content only appears after the JS, those bots see an almost empty page: a <div id="root"> and little else.
Step 1: look at your raw HTML
Open your site, right-click and choose view source (not "inspect", which shows the rendered DOM). Look for your main headline and a couple of paragraphs. If they are not in that HTML, AI probably will not see them either. Another quick test: disable JavaScript in your browser and reload. What remains is roughly what a basic bot sees.
If you prefer the terminal, request the page the way ChatGPT's bot would and count how much text arrives:
curl -s -A "GPTBot" https://yourdomain.com/ | sed 's/<[^>]*>//g' | wc -wA few dozen words means the content lives in the JavaScript. Several hundred means it arrives in the HTML.
Step 2: check your robots.txt
Visit yourdomain.com/robots.txt. Make sure you are not blocking AI bots by accident. A generic Disallow: /, a copied template or your CDN's "bot fight mode" may be shutting the door on them. If you want to allow search and live visits but not training, the file would look like this:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlBlocking is a legitimate choice, but make it on purpose, not by accident. And remember that robots.txt is a request, not a barrier: well-behaved bots honour it; the others do not.
Step 3: offer an llms.txt
The llms.txt file at the root of your domain is a plain-text (Markdown) summary of what you are and which pages matter. It is neither mandatory nor magic, and not every model reads it yet, but it gives a clean entry point to your site and forces you to write the one sentence that defines your product. A minimal example:
# Your product
> Your product is a [what it is] for [whom] that [what it achieves]. One sentence, no adjectives.
## Main pages
- [What it does](https://yourdomain.com/product): one-line description
- [Pricing](https://yourdomain.com/pricing): plans and what each includes
- [Docs](https://yourdomain.com/docs): how to get startedThat first sentence, the one starting with "Your product is…", is worth repeating verbatim on the home page, in the meta description and in the structured data: it is the one models end up quoting.
Step 4: structured data that describes you
A JSON-LD block with Organization (name, logo, URL, description) and, if you sell something, SoftwareApplication or Product with prices, gives the model unambiguous facts: what you are, what you cost, where you are. If you have frequently asked questions, mark them up as FAQPage: it is the format closest to how users ask AI.
Step 5: ask the AI directly
Open ChatGPT, Claude and Perplexity and ask the same three questions in each: what is X, what does X do and alternatives to X, plus a fourth, more useful one: recommend me a tool for [the problem you solve]. Watch three things: whether it mentions you, whether what it says is correct, and whether it confuses you with someone else. Write it down with the date. Repeat monthly: this is your real baseline and the only way to know whether what you change works.
Step 6: feed it citable content
Models cite paragraphs that answer a question without depending on the rest of the page. That translates into concrete habits: a heading that is the question, the direct answer in the first two sentences, concrete facts (figures, names, dates) instead of adjectives, and one page per topic instead of one page that tells everything. If your best information lives in a video, an image or a PDF, for AI it does not exist.
How to fix it depending on how your site is built
- Next.js or Nuxt. They already render on the server unless you turned it off. Check that marketing pages do not use
'use client'for everything or depend on data that only loads in the browser. - Vite SPA (Lovable, Bolt, v0). The HTML arrives empty. The options, from least to most work: prerender the public pages at build time (prerender plugins), move marketing to a separate static site, or migrate the public part to a framework with SSR.
- Webflow, Framer, WordPress. They serve full HTML; here the problem is usually
robots.txt, a forgottennoindexor the CDN's anti-bot protection. - Behind a login. Nothing that requires a session is visible to anyone. Put the description of what you do on a public page.
The mistakes that most often get in the way
- A
<meta name="robots" content="noindex">left over from the testing phase. - The CDN's "bot fight mode" or a JavaScript challenge that blocks every crawler alike.
- A site that loads content through infinite scroll or after clicking a button.
- Text inside images: the product name and value proposition in a PNG.
- One long page for everything, without headings that say what each part is about.
How to measure it over time
Look at your server or CDN logs and filter by the user agents above: it will tell you which bots visit you, how often and which pages they request. If none shows up, check robots.txt and the CDN protection first. Combine it with the monthly round of questions from step 5 and you have both halves: whether they read you and whether they cite you.
Scanaris automates this check
The Scanaris AI-visibility pillar reviews whether your content is readable without JS, whether you block AI bots in robots.txt, whether you have llms.txt and structured data, and even how a model describes you when asked about your brand, with a fix prompt for each issue. All read-only, on public information. And the visibility tracker checks every day whether you show up when someone asks for a recommendation in your category.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
It removes you from the training of future models, not from answers with search, which use OAI-SearchBot and ChatGPT-User. If you want to show up when someone asks, those two must be allowed.
How long until a change shows?
Live visits, immediately: as soon as the HTML carries the content, the assistant reads it. Search indexes, days to weeks. The model's "memorised" knowledge, until the next training run, which you do not control.
Do I need llms.txt if I already have a sitemap?
It is not mandatory. The sitemap says which pages exist; llms.txt says what you are and which pages matter, in plain language. It takes ten minutes and does no harm.