You can opt-out of having a page or other resource appear in Google Search. Google explains how to hide a website from search results in three different ways, and which one you should use depending on your situation and what’s best for your website.

Why Would You Want To Prevent Your Site From Being Indexed By Search Engines?

There are some situations where people wish search engines to avoid indexing their sites:

Unfinished websites – it’s preferable not to make your website public while you’re still working out the flaws.

Restricted websites — you don’t want your invite-only website to appear in the SERPs.

Test accounts – for testing and trial purposes, publishers create a copy of their site. Because some sites aren’t intended for the general audience, search engines shouldn’t index them.

Google’s John Mueller responds to a question on how to prevent content from being indexed in search and whether or not this is something that websites can do

He explained how password protection, blocking crawling, and blocking indexing are three ways to prevent a website from showing up in search engines, while keeping them accessible. 

• Websites can either opt out of indexing entirely or get indexed and use a password to hide information from Googlebot.

• It is not against webmaster standards to restrict content from Googlebot if it is also blocked from users.

• If the site is password protected when Googlebot crawls it, it must also be password protected for users.

• Alternatively, directives must be in place on the site to prevent Googlebot from crawling or indexing it.

• If your website offers different material to Googlebot than it does to users, you could get into trouble.

So let’s dig deeper and see how you can prevent your content from indexing in such ways.

1. Secure your website with a Password

If you want to keep your website confidential due to any reasons, locking it down using a high security  password is usually the best option.

Search engines and random web users will not be able to see your content if you use a password and your content will be prevented from indexing.

This is a normal protocol for developing web pages . It’s a simple approach to share in-progress work with clients while also preventing Google from visiting a website that isn’t yet ready to be seen.

Here are the steps, using which you can set a password.

Password protection using CPanel

1. Navigate to Directory Privacy on your cPanel account.
2. Choose your root directory. 
3. Select the Password-protect this directory option and give the directory a name. Save the file.
4. Simply create a new user to login to the secured website, and you’re done!

If you wish to know more about password protection click here.

Blocking Crawling 

Blocking crawling is another approach to prevent Googlebot from accessing your site. The robots.txt file is used for this.

The robots.txt file notifies search engine crawlers which URLs on your site they can access. This is mostly intended to prevent your site from becoming overburdened with requests; it is not a strategy for keeping a web page out of Google.

A robots.txt file is used primarily to manage crawler traffic to your site, and usually to keep a file off Google, depending on the file type.

According to Mueller, this isn’t the ideal choice because search engines may still index the website’s address without reading the content.

Although this is an unusual thing, BUT,  it is something you should be aware of.

Block indexing 

The third and last method is to prevent search engines from crawling your website.

A noindex meta tag or header in the HTTP response can be used to prevent a page or other resource from showing in Google Search. When Googlebot examines that page again and discovers the tag or header, it will remove it from Google Search results entirely, regardless of whether other sites link to it.

Using noindex is useful if you don’t have root access to your server, as it allows you to control access to your site on a page-by-page basis.

Mueller concludes the video by stating that Google’s top suggestion is to use a password:

“Overall, for private content, our recommendation is to use password protection. It’s easy to check that it’s working, and it prevents anyone from accessing your content.

Blocking crawling or indexing are good options when the content isn’t private. Or if there’s just parts of a website which you’d like to prevent from appearing in search.”