Controlling Crawling & Indexing
Technical SEO starts with controlling what Google can crawl and index. Get this wrong and even great content stays invisible.
robots.txt
robots.txt is a file at your domain root that tells crawlers which paths they may or may not crawl. It controls crawling, not indexing — a blocked page can still be indexed if linked elsewhere.
The noindex tag
To keep a page out of the index, use a meta robots noindex tag (and let Google crawl it so it sees the tag). Use this for thank-you pages, thin filters and admin pages.
XML sitemaps
An XML sitemap lists your important URLs to help Google discover them. Submit it in Search Console and keep it updated automatically.
Pro tips
- Never block a page in robots.txt if you want it noindexed — Google must crawl it to see the noindex.
- Keep your sitemap free of noindex, redirected and 404 URLs.
- Audit indexing with the Pages report in Search Console.
Common mistakes
- Blocking CSS or JS in robots.txt so Google cannot render the page.
- Accidentally noindexing important pages during a redesign.
- A sitemap full of broken or non-canonical URLs.
robots.txt controls crawling; the noindex tag controls indexing. XML sitemaps aid discovery. Confusing these is a classic, costly mistake.