AI Crawler Access Checker
Check whether a website allows or blocks AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended in its robots.txt.
How do I know if my site blocks AI crawlers?
Check your robots.txt for rules naming AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. An AI crawler checker reads the file and shows whether each one is allowed or blocked.
About the AI Crawler Access Checker
AI companies run separate crawlers for different jobs: some collect training data, some build an index for AI search answers, and some fetch a page when a user asks about it. Each obeys its own name in robots.txt, so a site can allow one and block another.
Many sites block AI crawlers by accident, through a firewall preset or a plugin, and then wonder why AI assistants never mention them. Check here, then decide deliberately: blocking training crawlers protects content, while blocking search crawlers removes you from AI answers.
How to use the AI Crawler Access Checker
- Enter a website URL.
- Read the allow or block verdict for each AI crawler.
- Update robots.txt if the result is not what you want.
Questions
Does blocking GPTBot remove me from ChatGPT search?
GPTBot is for training. ChatGPT search uses OAI-SearchBot, so check that one separately.
Why does this tool use your server?
Browsers are not allowed to read other websites directly, so our server fetches the public page you enter and returns what it found. Private and internal addresses are blocked, and nothing is stored.