Skip to content
Free SEO tool

Robots.txt Tester

Fetch a site’s robots.txt and test whether a specific path is allowed or blocked for Googlebot, Bingbot and other crawlers.

Free, no signup Public pages only, nothing stored Instant results
Or paste a robots.txt to test instead of fetching it
Quick answer

How do I test robots.txt?

A robots.txt tester reads the site’s robots.txt, picks the rule group for the crawler you choose and tells you whether a URL is allowed. Google applies the most specific matching rule, and Allow wins when rules are equally specific.

About the Robots.txt Tester

This fetches a site’s robots.txt, finds the group of rules that applies to the crawler you choose and tells you whether a given URL is allowed. It follows Google’s rules: the most specific matching path wins, and Allow wins a tie.

Use it before launching a site, after editing robots.txt, or when a page is missing from Google and you want to rule out a blocked crawl. You can also paste a draft robots.txt to test it before uploading.

How to use the Robots.txt Tester

  1. Enter a URL to test.
  2. Choose a crawler.
  3. See the matching rule and the verdict.

Questions

How are rules matched?

The most specific matching rule wins, and Allow beats Disallow when they are equally specific, the same way Google reads robots.txt.

Why does this tool use your server?

Browsers are not allowed to read other websites directly, so our server fetches the public page you enter and returns what it found. Private and internal addresses are blocked, and nothing is stored.