robots.txt checklist before deploy

A robots.txt checklist before deploy: ten checks covering file location, server response, size, user-agent groups, rule length, wildcards and test URLs, so a release never blocks a live section from Google. Full walkthrough: https://uskotoivo81-netizen.github.io/test-robots-txt-rules-without-search-console-tester/

Transcript

1. robots.txt checklist before deploy One line in robots.txt can stop Googlebot from crawling a whole section, and nobody notices until traffic drops. The file is easy to change and easy to ship by accident with an unrelated release. This robots.txt checklist before deploy covers ten checks, in the order that catches the costly mistakes first. robots.txt Checklist Before Deploy 1/13
2. Check 1 — Right file, right place Google reads robots.txt only from the top-level directory of a host. A file in a subfolder is not checked. Rules apply only to the host, protocol and port where the file is served. shop.example.com needs its own file. Confirm the file you are about to deploy is the production version, not the staging one with Disallow: / . robots.txt Checklist Before Deploy 2/13
3. Check 2 — The server answers correctly 200 with plain text is the goal. A 4xx response (except 429) makes Google assume there are no restrictions at all. A 5xx response makes Google stop crawling the site for the first 12 hours, then fall back to the last cached copy for up to 30 days. Google follows at least five redirects for robots.txt, then treats the result as 404. robots.txt Checklist Before Deploy 3/13
4. Check 3 — Size and syntax Google ignores everything after the first 500 KiB. Supported fields: user-agent , allow , disallow , sitemap . crawl-delay is ignored by Google. Paths are case-sensitive: /Shop/ and /shop/ are different rules. robots.txt Checklist Before Deploy 4/13
5. Check 4 — Which group Googlebot follows Google uses the single most specific user-agent group that matches. Groups with the same specific name are combined. A User-agent: Googlebot group and the * group are never combined. Add a Googlebot group and every * rule stops applying to Google. Group order in the file does not matter. robots.txt Checklist Before Deploy 5/13
6. Check 5 — Rule length decides Google applies the matching rule with the longest path, not the first one in the file. On a tie between Allow and Disallow, the least restrictive rule (Allow) wins. For every new Disallow, ask: is there a longer Allow that overrides it? For every new Allow: what does it reopen? robots.txt Checklist Before Deploy 6/13
7. Check 6 — Wildcards do what you think * matches any sequence of characters, including none. A trailing * is ignored. $ marks the end of the URL: /*.pdf$ does not match /file.pdf?v=2 . Parameter rules should anchor on ? : /*?*sort= rather than /*sort= . robots.txt Checklist Before Deploy 7/13
8. Check 7 — Test a list of real URLs Search Console's robots.txt report shows fetch status and parsing issues, but it does not test individual URLs. Before deploy, run 15–30 real addresses against the draft: home, one URL per template, filtered URLs, pagination, service pages. Every page that should rank must come out "allowed"; every service URL "blocked". Note the deciding line for each. robots.txt Checklist Before Deploy 8/13
9. Check 8 — Sitemap lines and conflicts Sitemap: takes a full absolute URL and is not tied to any group. Make sure the sitemap path itself is not disallowed; Google respects robots.txt when fetching sitemaps. Make sure no URL listed in the sitemap is disallowed, or Search Console reports "Sitemap contains urls which are blocked by robots.txt". robots.txt Checklist Before Deploy 9/13
10. Check 9 — robots.txt is not a removal tool A disallowed URL can still be indexed from links, shown without a description. To keep a page out of search, leave it crawlable and use noindex , or put it behind a password. If you plan to add noindex , do not disallow the same URL: Google would never see the tag. robots.txt Checklist Before Deploy 10/13
11. Check 10 — After the deploy Fetch /robots.txt on production and compare it with the expected file. In the robots.txt report, confirm the new version is "Fetched" with no errors; use "Request a recrawl" after an urgent fix. Google generally caches robots.txt for up to 24 hours, so a correction is not instant. Watch the Page indexing report for growth in "URL blocked by robots.txt" over the following days. robots.txt Checklist Before Deploy 11/13
12. The checklist ☐ Production file, top-level directory, one per host ☐ 200 response, no 5xx, no redirect chain ☐ Under 500 KiB, only supported fields ☐ Googlebot group decision made on purpose ☐ Every new rule checked against longer rules ☐ Wildcards anchored, $ used deliberately ☐ 15–30 real URLs tested against the draft ☐ Sitemap path and sitemap URLs not blocked ☐ Pages to remove use noindex, not Disallow ☐ Post-deploy fetch and report check done robots.txt Checklist Before Deploy 12/13
13. Resources Full walkthrough with a worked example and a Python checker: robots.txt testing without the old Search Console tester — https://uskotoivo81-netizen.github.io/test-robots-txt-ruleswithout-search-console-tester/ Free browser tools for crawling and indexing checks: IndexFix Tools — https://indexfixtools.com/ robots.txt Checklist Before Deploy 13/13