Google’s Martin Splitt recently cleared up confusion around the use of robots.txt directives and noindex tags in a YouTube video, emphasizing the distinct purposes of each and why they shouldn’t be mixed.

Don’t Combine Noindex and Disallow

Splitt warns against using the “noindex” tag and the “disallow” directive on the same page. The main reason? When a page is disallowed in robots.txt, search engines can’t access the page’s meta tags, including “noindex.” This means the page could still be indexed—just without much content.

When to Use Noindex

The “noindex” directive is used to keep a page from appearing in search results, while still allowing search engines to crawl and read the page’s content. It’s perfect for thank-you pages, internal search result pages, or other content you don’twant in search results but still want indexed for internal use.

When to Use Disallow

The “disallow” directive in robots.txt blocks search engines from crawling specific URLs or patterns entirely. Use this when you want to prevent search engines from accessing or processing sensitive content, such as private user data, or when a page has no value for search engines.

Common Mistakes to Avoid

A frequent error is using both “noindex” and “disallow” for the same page. This can lead to issues since disallowing thepage blocks crawlers from seeing the noindex tag. Instead, Splitt advises using “noindex” on pages you want crawlers to read but not index, without adding them to the robots.txt disallow list.

Why This Matters

Properly using “noindex” and “disallow” is crucial for SEO success. By following Google’s guidelines and utilizing tools like Google Search Console’s robots.txt testing feature, you can control that search engines interact with your site and ensure your content appears as intended.

If all of this feels overwhelming, don’t worry—our monthly SEO packages are here to make it easy. Let the experts handle it for you!