How to Block AI Crawlers from Your Site

To block AI crawlers from scraping your content, add User-agent rules to your robots.txt file that target the known AI bot tokens and disallow them. That stops the well-behaved crawlers in minutes. Here is how it works, which bots to name, and where robots.txt falls short.

Block AI crawlers with robots.txt

Every crawler announces itself with a user-agent string. robots.txt lets you write a rule per agent that says “do not crawl this site.” To stop the major AI bots, you list each one and disallow the whole site:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Writing these by hand is error-prone, since one typo in a token means a bot sails right through. The fastest path is to generate them with Block AI Crawlers, pick the bots you want to stop, and paste the output into your file.

Which AI bots to name

The list changes often as new crawlers appear, so treat this as a starting point, not a fixed set:

• GPTBot: OpenAI’s crawler for training models.
• Google-Extended: controls whether Google uses your content for AI training. It does not affect normal Google Search indexing, so blocking it keeps you in Search.
• CCBot: Common Crawl, whose dataset feeds many AI models.
• ClaudeBot: Anthropic’s crawler.
• PerplexityBot: powers Perplexity’s answer retrieval.
• Bytespider: ByteDance’s crawler, known for aggressive scraping.

Roughly speaking, some bots crawl to train models on your text, while others crawl to retrieve live content for answers. You can block both, or allow retrieval while blocking training, depending on whether you want your pages to show up in AI answers at all.

Add the rules to your site

1. Open Block AI Crawlers and select the AI bots you want to block.
2. Generate the robots.txt rules.
3. Add them to the robots.txt file at your site root, so it lives at https://yoursite.com/robots.txt.
4. Visit the file in a browser to confirm the rules are live.

If you already have a robots.txt, append the new User-agent blocks rather than replacing the file, so your existing rules stay intact.

The honest limitation

robots.txt is voluntary. It is a request, not a wall. Reputable crawlers like GPTBot and Google-Extended read it and obey, but a bad actor can ignore it completely or simply spoof a browser user-agent and scrape anyway.

To stop the crawlers that do not play fair, you need server-level blocks: deny their user-agents or IP ranges in your web server config (Nginx or Apache), or use a firewall or CDN rule, for example a WAF rule at Cloudflare. That actually rejects the request instead of politely asking it to leave. Use robots.txt for the honest bots and server blocks for everyone else.

Related tools

• robots.txt Builder: build your full robots.txt with crawl and sitemap rules.
• llms.txt Generator: tell AI tools how to use the content you do want shared.
• Security Headers Generator: harden your responses with the right HTTP headers.

robots.txt handles the polite bots in two minutes, so generate your rules, ship them, and back them with a server block for the rest.