Can AI crawlers read your site?
Enter a domain. We fetch its robots.txt and tell you, bot by bot, whether AI search engines, assistants and training crawlers can reach your homepage.
Crawlable is step one. Are you cited?
The free report checks up to five buyer questions on three AI engines and shows who gets cited instead of you.
Get the free reportPaid plans run this check, and about 30 more, every week. See plans
How this checker works
- We fetch https://yourdomain/robots.txt from our server, following up to five redirects, with an 8-second timeout and a 500 KiB limit.
- Rules are evaluated as in RFC 9309 and Google’s parser: the longest matching rule wins, a bot’s own group replaces the * group.
- We check the homepage path (/). A site can still allow the homepage and block sections.
- robots.txt is a request, not a wall. Firewalls and bot protection (Cloudflare, Vercel) can block crawlers too; the full audit tests that.
AI crawler questions.
What people ask before they start. If something’s missing, write to us.
Write to usShould I block GPTBot?
GPTBot collects data to train OpenAI models. Blocking it is a legitimate choice and, according to OpenAI’s documentation, doesn’t remove you from ChatGPT search, which uses OAI-SearchBot. If you want to appear in ChatGPT answers, don’t block OAI-SearchBot or ChatGPT-User.
Does Google-Extended control AI Overviews?
No. Google-Extended controls Gemini training and grounding in Gemini apps. AI Overviews and AI Mode are part of Google Search and follow Googlebot and snippet controls such as nosnippet.
My robots.txt allows everything but I’m still not cited. Why?
Being crawlable is necessary, not sufficient. Engines cite pages that answer the question well and that other sites reference. The free report shows which pages get cited for your buyers’ questions.
Do you store the domains I check?
Only for 10 minutes in memory, to avoid hitting the same site repeatedly. We don’t log them to a database.