AI Crawlers and Access Control
Each AI company runs one or more crawlers with separate jobs: training, search indexing, or fetching a page a user asked about. Your robots.txt decides which of them can read your site, and the choice has consequences for visibility.
The main user agents
- OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (fetches a page a user requested).
- Anthropic: ClaudeBot (training), Claude-SearchBot (search), Claude-User (user requests).
- Perplexity: PerplexityBot (search index) and Perplexity-User.
- Google: Googlebot crawls for Search, including AI Overviews. Google-Extended is a control token that decides whether your content may be used for Gemini model training and grounding; it does not affect Search.
- Apple: Applebot for search, Applebot-Extended for training use.
Names change; confirm each company’s current list before editing your rules.
Deciding what to allow
Blocking training crawlers keeps your content out of future model training. Blocking search crawlers keeps you out of that assistant’s cited answers. Most businesses that want visibility allow search and user-fetch agents, and decide training access as a separate policy question.
Check the whole path
A robots.txt allow means nothing if a firewall, CDN bot protection or rate limiter blocks the crawler. Check server logs for the user agents above and confirm they receive status 200, not 403.
# Allow AI search and user-requested fetches
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: / This is one possible policy, not a recommendation for every site. Decide training access deliberately.
Pro tips
- Review robots.txt and CDN bot settings together; they often disagree.
- Search server logs monthly for AI user agents and their response codes.
- Document your AI access policy so future changes are deliberate.
Common mistakes
- Blocking Googlebot to "stop AI" and losing all search visibility.
- Assuming Google-Extended controls AI Overviews.
- Leaving a CDN setting that silently blocks every bot you allowed.
AI companies run separate crawlers for training, search and user fetches. Allow the ones you want answers from, decide training separately, and check firewalls too.