A robots.txt for AI search engines.
Start from your current file, decide bot by bot, and get a new robots.txt with only the changes you asked for, shown line by line.
1. Your current robots.txt
2. Choose what each AI bot can do
Search and answer engines
They index pages to answer questions. Blocking them removes you from their answers.
OAI-SearchBot· OpenAI allowed nowChatGPT search results
PerplexityBot· Perplexity allowed nowPerplexity search index and citations
Claude-SearchBot· Anthropic allowed nowClaude search results
Googlebot· Google allowed nowGoogle Search, including AI Overviews and AI Mode
Bingbot· Microsoft allowed nowBing and Copilot; Bing's index is also one of the sources other assistants use
Applebot· Apple allowed nowSiri, Spotlight and Safari suggestions
DuckAssistBot· DuckDuckGo allowed nowDuckDuckGo's AI-assisted answers
User-triggered fetchers
They open a page when someone asks an assistant about it.
ChatGPT-User· OpenAI allowed nowpages ChatGPT opens when a user asks
Perplexity-User· Perplexity allowed nowpages Perplexity opens when a user asks
Claude-User· Anthropic allowed nowpages Claude opens when a user asks
Meta-ExternalFetcher· Meta allowed nowpages Meta AI opens when a user shares or asks about a link
MistralAI-User· Mistral allowed nowpages Le Chat opens when a user asks
Training crawlers
They collect data to train models. Blocking them is a legitimate choice and doesn’t remove you from search answers.
GPTBot· OpenAI allowed nowtraining OpenAI models
ClaudeBot· Anthropic allowed nowtraining Anthropic models
Google-Extended· Google allowed nowGemini training and grounding in Gemini apps; it does NOT control AI Overviews (those follow Googlebot and nosnippet)
Applebot-Extended· Apple allowed nowtraining Apple's foundation models
CCBot· Common Crawl allowed nowCommon Crawl, a public dataset used to train many models
Meta-ExternalAgent· Meta allowed nowtraining Meta AI models
Bytespider· ByteDance allowed nowtraining ByteDance models
3. Your new robots.txt
3 lines change
+ added− removed
User-agent: *Allow: /
Upload it as robots.txt at the root of your domain, replacing the current one. Then run the AI crawler checker to confirm. Open the AI crawler checker
Readable is step one. Are you cited?
The free report asks up to five of your buyers’ questions on three AI engines and shows who gets cited instead of you.
Get the free reportPaid plans check robots.txt, firewalls and about 30 more things every week. See plans
How this generator works
- Allow takes out only the rules that block the bot from the whole site. Section rules it inherited from the * group (like /admin/) stay.
- Block adds a group of its own with Disallow: / at the end and removes the bot from the groups it shared, without touching the other agents in them. If it was already fully blocked, nothing changes.
- Keep leaves the bot exactly as it was. Comments, sitemaps, the * group and rules for agents not on the list aren’t touched.
- We then re-evaluate the new file with the same RFC 9309 parser the AI crawler checker uses, path by path.
- Everything runs in your browser. To read your domain’s robots.txt, our server fetches it once (5 redirects, 8 seconds, 500 KiB at most).
robots.txt and AI crawler questions.
What people ask before they start. If something’s missing, write to us.
Write to usWill blocking GPTBot remove me from ChatGPT?
According to OpenAI’s documentation, no. GPTBot collects training data; ChatGPT search uses OAI-SearchBot, and ChatGPT-User opens pages when someone asks. Block GPTBot if you don’t want to be in training data, and keep the other two allowed.
Which bots should I keep allowed to be cited by AI?
The search and user-triggered ones: OAI-SearchBot and ChatGPT-User (ChatGPT), PerplexityBot and Perplexity-User, Claude-SearchBot and Claude-User, Googlebot (Google Search, AI Overviews and AI Mode), Bingbot (Bing and Copilot) and Applebot.
Does Google-Extended control AI Overviews?
No. Google-Extended controls Gemini training and grounding in Gemini apps. AI Overviews and AI Mode are part of Google Search and follow Googlebot and snippet controls such as nosnippet.
Does robots.txt actually stop AI crawlers?
It’s a request, not a wall. OpenAI, Anthropic, Google, Perplexity and Apple say their crawlers respect it, but not every bot does. Firewalls and bot protection (Cloudflare, Vercel) can also block crawlers you allowed here.
Where do I upload the new robots.txt?
It has to be served at https://yourdomain/robots.txt. In Webflow it’s in Site settings, SEO tab. Most site builders have a similar field in their SEO settings; on WordPress, an SEO plugin or a file at the root of the site.