Robots.txt Generator
Build a correct robots.txt file: allow or block folders, add your sitemap, and choose how AI crawlers such as GPTBot and ClaudeBot may use your site.
Output
What should a robots.txt file contain?
A robots.txt file lists which crawlers may visit which parts of a site, using User-agent, Allow and Disallow lines, plus a Sitemap line with your sitemap’s full URL. Most business sites only need to block admin, cart and internal search pages.
About the Robots.txt Generator
robots.txt is the first file a crawler reads. It says which parts of a site may be crawled, and where the sitemap is. A single wrong line, such as "Disallow: /" left over from a staging site, can remove a whole website from Google within weeks.
Block only what has no search value: admin areas, carts, internal search results and filter combinations. Never block CSS or JavaScript files, because Google needs them to render the page. Decide separately whether AI companies may use your content for training and whether AI search tools may cite you.
How to use the Robots.txt Generator
- Choose the default rule for all crawlers.
- List folders to block and add your sitemap URL.
- Pick AI crawler rules and copy the file to your site root.
Questions
Does robots.txt hide a page from Google?
No. It stops crawling, not indexing. To keep a page out of search results, use a noindex tag and let Google crawl it.