How to Test Your robots.txt Before It Costs You Traffic

A single Disallow: / in the wrong place can drop a site out of search results. Because robots.txt is rarely looked at after setup, these mistakes can sit unnoticed for months. Testing takes a minute and is worth doing after any change.

How the rules are read

robots.txt is grouped by user-agent. For each crawler, the rules are matched like this:

The longest matching path wins.
If an Allow and a Disallow match with the same length, Allow wins.
* matches any sequence, and $ anchors the end of the URL.

So Disallow: /wp-admin/ plus Allow: /wp-admin/admin-ajax.php correctly blocks the admin area while keeping the one file WordPress needs.

Test a URL in seconds

Open the robots.txt Tester.
Paste your robots.txt, pick a crawler (Googlebot, Bingbot, and others), and enter the URLs to check.
See Allowed or Blocked for each, plus the exact rule that decided it.

Everything runs in your browser.

Crawling is not indexing

This trips people up constantly. robots.txt controls crawling, not indexing. A blocked page can still appear in search results (without a description) if other sites link to it. To keep a page out of the index, let it be crawled and add a noindex meta tag instead.

Common mistakes to check for

Blocking /wp-content/ or your CSS and JS, which stops Google from rendering the page.
A leftover Disallow: / from a staging site.
Blocking a page you also want indexed (use noindex, not Disallow).

Building the file from scratch? Use the robots.txt Builder.
Checking if a specific page can be indexed? Try the Indexability Checker.
Controlling AI crawlers too? See Block AI Crawlers.

After any change to robots.txt, paste it into the tester and check your key URLs. It is the cheapest insurance in SEO.