robots.txt Generator
Build a valid robots.txt file with common disallow rules, AI bot blocking, and sitemap declaration.
Adds Disallow: / for all user-agents. This removes your site from all search engines.
Common paths to disallow
Select paths you want to prevent bots from crawling.
One path per line. e.g. /private
Overrides disallow rules for specific paths. One per line.
Adds Disallow: / for GPTBot, Google-Extended, CCBot, anthropic-ai, Claude-Web, PerplexityBot, Bytespider, and Applebot-Extended.
robots.txt
User-agent: *
Save this as robots.txt in the root of your domain (e.g. https://example.com/robots.txt).