A robots.txt audit should verify what crawlers are asked to fetch, but it must not treat the file as access control or a guaranteed way to remove a URL from search. Review the public file, test representative URLs, protect private content with real authentication, and keep indexing directives separate from crawl directives.
Fetch the public file first
Request `/robots.txt` at the root of every important scheme and hostname. Record the status, final URL, content type, encoding, and response body. RFC 9309 defines the file at the lowercase top-level `/robots.txt` path and requires UTF-8 plain text (IETF).
Check production directly rather than reviewing only a repository copy. A CDN, CMS, host, or deployment rule can serve different content from the file an editor expects. Preserve the previous version before changing it.
Read rules as groups
Map each `User-agent` group and its `Allow` and `Disallow` rules. RFC 9309 says crawlers match paths case-sensitively, use the most specific match, and prefer `Allow` when equally specific allow and disallow rules conflict. Test the actual URL paths, including capitalization, encoded characters, trailing slashes, query patterns, and nested folders.
Do not rely on a comment or visual spacing to define behavior. Confirm that every user-agent line belongs to the intended group and that a later group does not accidentally change the result.
Separate crawling, indexing, and security
Google explains that robots.txt is primarily a crawl-traffic control and is not a mechanism for keeping a webpage out of search results. A disallowed URL can still appear if other pages link to it. For content that must not be public, use authentication or remove public access; for eligible pages that should not appear in Google, use an appropriate indexing control on a crawlable response (Google Search Central).
Never list secret paths, credentials, private customer identifiers, or internal notes in robots.txt. The file is public and can reveal names even when a cooperative crawler does not fetch the listed pages.
Test valuable pages and resources
Build a test list for the homepage, main categories, articles, product or service pages, images, CSS, JavaScript, APIs used for rendering, staging paths, search results, and duplicate parameter URLs. Check each against the intended crawler group. Important rendering resources should not be blocked accidentally.
Test both allowed and intentionally disallowed examples. If a sitemap line is present, verify the sitemap URL separately and confirm it belongs to the correct site. A sitemap entry does not override a disallow rule.
Review failure behavior
Monitor the status of robots.txt itself. RFC 9309 distinguishes successful access, redirects, unavailable responses, and server errors; crawlers can treat these states differently. Keep the route simple, fast, and independent of fragile application logic where possible.
After deployment, fetch the file from outside the hosting account and use the search engine's current testing or reporting tools. Recheck after domain, CDN, CMS, or routing changes.
Quick completion test
The audit is complete when every public hostname serves the intended root file, syntax and grouping are understood, important pages and resources are tested, private content uses real access controls, indexing goals use the right mechanism, and a rollback copy and review owner are recorded.
Sources
- RFC 9309: Robots Exclusion Protocol — Internet Engineering Task Force, Sept. 1, 2022
- Introduction to robots.txt — Google Search Central