robots.txt and AI crawlers: GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended

Quick summary

• Every AI company runs several bots with different jobs: some crawl to train models, some index for their search engine, and some fetch a page when a user asks.
• You can block training and still appear in answers. With OpenAI, block GPTBot and allow OAI-SearchBot; with Anthropic, block ClaudeBot and allow Claude-SearchBot and Claude-User.
• Google-Extended doesn't remove you from Google Search or AI Overviews. According to Google, it governs Gemini training and grounding in the Gemini app and Vertex AI.
• User-initiated agents (ChatGPT-User, Perplexity-User) may not apply robots.txt, according to their own companies.
• robots.txt isn't access authorization. And even when it's right, your CDN or firewall may be blocking the bots anyway.

The three kinds of bot

Before changing anything, separate three functions, because blocking one doesn't have the same effect as blocking another:

1. Training. Collect content that may be used to train models. Blocking them doesn't remove you from search-backed answers.
2. Search. Index pages so the AI's search engine can find and cite them. These are the ones that matter for being cited.
3. User. Fetch a specific page because a user asked for it right then (for example, "summarize this URL" or a live search).

Bot by bot, from the official documentation

OpenAI

Source: OpenAI, "Overview of OpenAI Crawlers".

• OAI-SearchBot (search): used to surface websites in ChatGPT's search features. Sites that opt out won't be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI recommends allowing it and its published IP ranges. Changes take about 24 hours to apply.
• GPTBot (training): crawls content that may be used to train its foundation models. Disallowing it signals your content shouldn't be used for that.
• ChatGPT-User (user): visits pages when a user asks ChatGPT or a custom GPT something. OpenAI clarifies it isn't used to determine what appears in search and that, because these actions are user-initiated, robots.txt rules may not apply.

Each setting is independent: you can allow one and block another.

Perplexity

Source: Perplexity, "Perplexity Crawlers".

• PerplexityBot (search): surfaces and links websites in Perplexity's results. Perplexity says it isn't used to train foundation models.
• Perplexity-User (user): fetches pages when a user asks. Because a person requested the fetch, it generally ignores robots.txt.

Anthropic

Source: Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?".

• ClaudeBot (training): collects content that could contribute to training. Blocking it excludes your future content.
• Claude-SearchBot (search): improves search result quality. Blocking it may reduce your visibility.
• Claude-User (user): accesses sites when someone asks Claude a question. Blocking it prevents Claude from retrieving your content for that answer.

Anthropic says all its bots honor robots.txt, supports Crawl-delay, and warns that IP blocking isn't reliable because it stops them reading your robots.txt.

Google

Sources: Google's common crawlers and AI features and your website.

• Googlebot governs search, including AI Overviews and AI Mode. Google says that because AI is built into Search, robots.txt directives for Googlebot are the control. To limit what's shown, use nosnippet, data-nosnippet, max-snippet or noindex.
• Google-Extended isn't a crawler with its own user-agent: it's a control token in robots.txt. It manages whether content Google crawls may be used to train future Gemini models and for grounding in the Gemini app and Vertex AI. Google states it doesn't affect inclusion in Google Search and isn't a ranking signal.

That last point has an under-discussed consequence: blocking Google-Extended doesn't take you out of AI Overviews, but it may limit the Gemini app from using your content to ground its answers.

Summary table

Bot	Company	Type	If you block it
OAI-SearchBot	OpenAI	Search	You don't appear in ChatGPT search answers
GPTBot	OpenAI	Training	Your content isn't used for training
ChatGPT-User	OpenAI	User	May not apply robots.txt
PerplexityBot	Perplexity	Search	You drop out of Perplexity's index
Perplexity-User	Perplexity	User	Generally ignores robots.txt
Claude-SearchBot	Anthropic	Search	Less visibility in Claude
Claude-User	Anthropic	User	Claude can't read you at a user's request
ClaudeBot	Anthropic	Training	Your content isn't used for training
Google-Extended	Google	Control (Gemini training and grounding)	No effect on Google Search

A sample robots.txt

For a site that wants to appear in every engine's answers without giving content for training:

# Search and user agents: allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# Training: blocked
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Everyone else, as usual
User-agent: *
Allow: /
Disallow: /admin

Sitemap: https://yourdomain.com/sitemap.xml

Two warnings before you copy it:

• Google-Extended is left out on purpose. Blocking it is a legitimate training decision, but it may also affect grounding in the Gemini app. Decide knowing that.
• Under the robots.txt standard (RFC 9309), a crawler obeys the group whose User-agent matches its name and only falls back to the * group if none does. If you define a group for GPTBot, the User-agent: * rules no longer apply to it. Repeat any exclusions you need in each group.

If you'd rather do the opposite, open everything to every AI bot (as we do on our own site, since we sell visibility in their answers), just don't have groups that block them.

What robots.txt doesn't do

RFC 9309 says it bluntly: robots.txt rules are not a form of access authorization. They're a request that well-behaved crawlers respect. If content must not be read by anyone, protect it with authentication.

And the reverse: even if your robots.txt allows a bot, it may never get through. The usual culprits:

• CDN bot protection ("anti-bot" or "block AI crawlers" modes) switched on by default.
• Firewall rules blocking unknown user-agents or data-center IP ranges.
• 403 or 503 responses served to the bot while the site works fine for people.

Perplexity devotes a whole section of its docs to configuring Cloudflare and AWS firewalls for this reason, and OpenAI recommends allowing its published IP ranges.

How to check

1. Your robots.txt: run your domain through the AI crawler checker. It tells you, bot by bot, whether it's allowed and which rule decides.
2. Your CDN and firewall: check their logs for blocks on the user-agents in the table.
3. Your server logs: look for visits from OAI-SearchBot, PerplexityBot or Claude-SearchBot. If they never show up, something is stopping them (or they haven't found you yet).

Citable's weekly technical audit includes AI crawler access in robots.txt and flags any search bot that's blocked. It's on every plan; see pricing.

FAQ

Will blocking GPTBot remove me from ChatGPT?

Not from search. According to OpenAI, GPTBot is for training and OAI-SearchBot for ChatGPT search, and they're controlled separately. To appear in search answers, allow OAI-SearchBot.

Will blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended doesn't affect inclusion in Google Search. AI Overviews and AI Mode are governed by Googlebot directives and snippet controls. Google-Extended does affect Gemini training and grounding in the Gemini app and Vertex AI.

Do AI bots respect robots.txt?

The training and search bots from OpenAI, Perplexity, Anthropic and Google say they do. User-initiated agents vary: OpenAI says ChatGPT-User may not apply it and Perplexity says Perplexity-User generally ignores it; Anthropic says all its bots honor it.

What if I don't have a robots.txt?

If /robots.txt returns a 4xx error such as a 404, RFC 9309 says crawlers may access everything. That's not a problem for AI visibility, though it's worth having one to declare your sitemap and exclude private areas.

Conclusion

Most sites that don't show up in ChatGPT or Perplexity don't have a content problem, they have an access problem: a blanket block, a CDN anti-bot mode, or a rule copied from a template. Separate training from search, allow search and user agents, take a calm decision on training, and make sure your firewall agrees with your robots.txt.