Skip to content
Free SEO tool

Robots.txt Generator

Build a correct robots.txt file: allow or block folders, add your sitemap, and choose how AI crawlers such as GPTBot and ClaudeBot may use your site.

Free, no signup Runs in your browser Instant results

Details

Output

    Quick answer

    What should a robots.txt file contain?

    A robots.txt file lists which crawlers may visit which parts of a site, using User-agent, Allow and Disallow lines, plus a Sitemap line with your sitemap’s full URL. Most business sites only need to block admin, cart and internal search pages.

    About the Robots.txt Generator

    robots.txt is the first file a crawler reads. It says which parts of a site may be crawled, and where the sitemap is. A single wrong line, such as "Disallow: /" left over from a staging site, can remove a whole website from Google within weeks.

    Block only what has no search value: admin areas, carts, internal search results and filter combinations. Never block CSS or JavaScript files, because Google needs them to render the page. Decide separately whether AI companies may use your content for training and whether AI search tools may cite you.

    How to use the Robots.txt Generator

    1. Choose the default rule for all crawlers.
    2. List folders to block and add your sitemap URL.
    3. Pick AI crawler rules and copy the file to your site root.

    Questions

    Does robots.txt hide a page from Google?

    No. It stops crawling, not indexing. To keep a page out of search results, use a noindex tag and let Google crawl it.