How this check works
Scanaris does three things. First, it requests /robots.txt and treats it as valid only if it responds 200 and is not HTML (single-page apps return their page for any path); otherwise it opens a low-severity finding. Second, it requests /sitemap.xml and treats it as valid if it responds 200 and contains a <urlset> or <sitemapindex> tag; if it does not find it there but robots.txt has a Sitemap: line, it accepts that too; otherwise, a low-severity finding. Third, it checks whether the scanned page carries noindex, either in the <meta name="robots"> tag or in the X-Robots-Tag HTTP header; if it does, it opens a medium-severity finding, because that page is not going to appear on Google.
Why it matters
robots.txt is the first URL any crawler visits and where you tell it what it may crawl and where your sitemap is. The sitemap is the list of URLs you want Google to index, with their modification dates: without it, Google discovers your pages by following links, which is slower and misses the poorly linked ones. Both are low severity because their absence does not prevent indexing, it just makes it worse. noindex is another matter: it is a direct order to Google not to show that page, and on the home page it means your site does not exist in the search engine. It is the most devastating and most silent SEO mistake there is, and it is almost always a leftover from the staging environment.
How to fix it
robots.txt
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlA plain text file at the root. Nuxt and Next.js have modules and conventions (app/robots.ts) that generate it; on a static site, it goes in the public folder. Do not list private routes in it: it is public and you would be pointing them out to anyone, as sensitive paths in robots explains.
Sitemap
- Nuxt: the sitemap module generates it from the routes and the URLs you pass it (posts, products).
- Next.js: an
app/sitemap.tsfile that returns the list of URLs. - WordPress: generates one on its own since version 5.5 at
/wp-sitemap.xml; add that line to robots or let Yoast or Rank Math put it at/sitemap_index.xml. - Astro, Hugo, Jekyll: an official plugin or integration.
Only indexable, canonical URLs that respond 200. Submit it in Google Search Console and Bing Webmaster Tools: that way you will know how many of those URLs are actually indexed.
noindex on the page
Find where it comes from. In Nuxt, the robots module can set noindex based on an environment variable that sometimes gets copied from staging to production. In Next.js, metadata.robots. In WordPress, the "Discourage search engines" setting under Reading settings, which someone enabled while the site was under construction. On Vercel, preview deployments carry X-Robots-Tag: noindex automatically, but production does not; if you see it in production, you added it yourself in the headers configuration.
Check it is fixed
curl -s https://yourdomain.com/robots.txt
curl -s https://yourdomain.com/sitemap.xml | head -5
curl -sI https://yourdomain.com | grep -i x-robots-tag
curl -s https://yourdomain.com | grep -oiE '<meta name="robots"[^>]*>'The first two should show text and XML, not HTML. The last two must not contain noindex (a robots meta with index, follow or max-image-preview is normal).
Frequently asked questions
My sitemap is at another path, such as /sitemap_index.xml.
Any path works as long as robots.txt references it with the Sitemap: line. The check looks there before raising the finding.
I do not want my site on Google for now.
Then noindex is correct and you can accept the finding. But set yourself a reminder: most sites with noindex in production do not have it on purpose.
Does robots.txt affect AI bots?
Yes, it is the same file where GPTBot, ClaudeBot or PerplexityBot are allowed or blocked. What you are allowing and what makes sense is covered by AI crawlers.