How to Write a robots.txt File

3 min read by Mohammad Shoeb

How to Write a robots.txt File

A robots.txt file tells search engine crawlers which parts of your site they may request. It is a plain text file that lives at your domain root, so it is always reachable at /robots.txt. Here is the syntax, how rule matching works, and the one thing people get wrong about it.

robots.txt syntax in four directives

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines, then the rules for those crawlers.

  • User-agent: which crawler the rules apply to. Use * to match all crawlers, or a specific name like Googlebot.
  • Disallow: a path the crawler should not request. Disallow: /admin/ blocks that folder. An empty Disallow: blocks nothing.
  • Allow: an exception that re-opens a path inside a disallowed area. Allow: /admin/public/ lets that subfolder through.
  • Sitemap: a full URL pointing to your XML sitemap. It is independent of any group and can sit anywhere in the file.

A minimal file looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

How rule matching works

Paths are matched from the start of the URL path. Two pattern characters help:

  • * matches any sequence of characters. Disallow: /*.pdf blocks every PDF.
  • $ anchors the end of the URL. Disallow: /*.php$ blocks paths that end in .php.

When more than one rule matches a URL, the most specific rule wins, meaning the one with the longest matching path. If an Allow and a Disallow are the same length, the Allow wins. This is why a broad Disallow: / plus a narrow Allow: can still let specific pages through.

Crawling is not indexing

This is the part that trips people up. robots.txt controls crawling, not indexing. Blocking a URL with Disallow tells crawlers not to fetch it, but if other sites link to that URL, Google can still list it in search results, often with no description because it never read the page.

To keep a page out of the index, use a noindex meta tag (or X-Robots-Tag header) on the page itself:

<meta name="robots" content="noindex">

The catch: Google has to crawl the page to see the noindex. So do not Disallow a page you are trying to deindex, or the crawler will never read the tag. Let it be crawlable, add noindex, and remove the noindex (or the Disallow) only after it drops out.

One more warning: never ship Disallow: / on a live site by accident. That single line blocks your entire domain from crawling. Also avoid blocking the CSS and JavaScript that Google needs to render your pages, or your rankings can suffer.

Build it without typos

A stray slash or a missing colon can silently block the wrong thing, so it helps to generate the file from a form.

  1. Open the robots.txt Builder and pick which crawlers your rules target.
  2. Add Disallow and Allow paths for the folders you want to control.
  3. Paste in your sitemap URL so crawlers can find it.
  4. Copy the generated file and upload it to your domain root as /robots.txt.

The robots.txt Builder runs entirely in your browser, so nothing you type is sent anywhere.

Write the rules carefully, remember that blocking a URL does not deindex it, and test before you ship.

SEOHow-to

← All posts