robots.txt for AI Crawlers: GPTBot, ClaudeBot, PerplexityBot Explained

For years, robots.txt only had to think about Googlebot and Bingbot. Now there is a whole new category of crawler to consider: AI bots that gather content for tools like ChatGPT, Claude, and Perplexity.

The bots you should know about

• GPTBot — used by OpenAI to gather training and browsing data
• ClaudeBot — used by Anthropic
• PerplexityBot — used by Perplexity for search and citations
• Google-Extended — controls whether Google can use your content for AI features, separate from regular Googlebot search indexing
• CCBot — Common Crawl, a dataset many AI companies train on

Training use vs search/citation use

This is the part most people get wrong. Blocking an AI bot from training on your content is a different decision than blocking it from citing or linking to your content in AI search answers. Some bots separate these two behaviors, some don't yet. Read each bot's current documentation before deciding, since this changes as the tools evolve.

A simple starting point

If your goal is visibility in AI search results, allowing the major bots is usually the right call. If you run a subscription content site where your text is your product, you may want to block training bots while still allowing citation-focused crawlers. There is no single correct answer, it depends on your business model.

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Test it

After updating robots.txt, test it with Google Search Console's robots.txt tester to make sure you haven't accidentally blocked something important, like your CSS or JS files needed for rendering.

Section 4 of the AI SEO Mastery Notion template includes a full checklist and a ready-to-use prompt to generate your robots.txt file with your exact preferences filled in.