robots.txt Exposure Scanner

How this check works

Scanaris downloads your /robots.txt and first confirms it is a real robots file: it responds 200, it is not HTML, and it contains at least one directive (User-agent, Disallow, Allow or Sitemap). It then walks through every Disallow: line and checks whether the path contains one of these indicators: wp-admin, phpmyadmin, /admin, backup, .git, .env, /private, /secret, dump, .sql, password or credential. If any match, it opens a low-severity finding with the paths found. The list is deliberately short: words like test, old, tmp or logs appear in perfectly legitimate robots files and produced false positives.

Why it matters

robots.txt is public by definition: anyone can read it, and in fact it is one of the first files a reconnaissance tool looks at. When you write Disallow: /backups/ or Disallow: /old-admin/ you are not hiding those paths; you are pointing them out to whoever is looking, with the added guarantee that you care about them. Google will not crawl them, but an attacker does not read robots to obey it, they read it to know where to go. It is a low-severity finding because listing a path does not make it accessible; the problem appears when the listed path is also unprotected.

How to fix it

• Remove from robots.txt the paths you do not want to announce. If you do not want Google to index a private area, the right approach is for that area to require login; what sits behind authentication is not indexed anyway.
• If you need a reachable page to stay out of search engines without announcing it, use the X-Robots-Tag: noindex header or the robots meta tag on that page. Search engines honour it and it is not written in a public file.
• Protect the path for real. A panel at /admin needs authentication, not a line in robots. A /backups/ folder should not be in the public directory at all; the sensitive files and directory listing checks cover those cases.
• WordPress: the virtual robots it generates includes Disallow: /wp-admin/ by default. It is not serious, because wp-admin already requires login and sends noindex; you can leave it or replace robots with your own file without that line. What matters is that the login has strong passwords, attempt limits and, if you can, two-factor authentication.

A healthy robots.txt is usually short: a User-agent: *, maybe a couple of Disallow lines for internal search or parameters, and the Sitemap: line. If yours has twenty lines of "private" paths, it is doing the job of a permission system, and badly.

Check it is fixed

curl -s https://yourdomain.com/robots.txt | grep -iE "^disallow:.*(admin|phpmyadmin|backup|\.git|\.env|private|secret|dump|\.sql|password|credential)"

Nothing should come out. And for each path you removed, confirm it really asks for authentication or returns 404:

curl -s -o /dev/null -w "%{http_code}\n" https://yourdomain.com/admin/

Frequently asked questions

If I remove /admin from robots.txt, will Google index it?

If /admin requires login, Google only sees the login screen, and that screen normally carries noindex. If it does not require login, the problem is not indexing, it is that your panel is open.

Can I block everything and list only what is public?

For a site that wants organic traffic, no: blocking everything with Disallow: / removes it from search engines. If what you want is to control crawling, the robots and sitemap check makes sure you are not blocking yourself.

What about AI bots?

Blocking or allowing GPTBot, ClaudeBot and company is a separate business decision and goes in the same file. It has nothing to do with this check, which only looks at sensitive paths.