Site owners will do anything to index the content of their site. Various popular SEO practices like Keyword research, meta descriptions, meta tags, etc are done with the purpose of indexing a site. However there can be reasons you don’t want search engines to crawl through your content and index your site.

Just like there are ways to encourage indexing on your site, you can also block indexing on your WordPress site. There are a number of ways to stop search engines from indexing your posts and pages on your WordPress site. In this article, I am going to explain 4 different methods to block indexing on your WordPress site. But before the methods, let’s understand what exactly is Google Indexing and why do you need to block it.

What is Google Indexing?

There are different search engines available that use different indexing methods. Most popular and commonly used search engine is Google. Google indexing is basically a process of adding new web pages, including text, files, pictures, videos, etc, and storing them in its database. Basically, Google needs to store your site’s content in Google index in order to display your content or site in its search results. Google indexes these websites by using its spiders, bots, or crawlers. They crawl all over the internet to index and store. Moreover, they also follow website owners’ instructions on what to crawl and what to ignore. So in some specific cases you can prevent crawlers from indexing your site.

Why are websites Indexed?

There are millions or even billions of sites available on the internet with data, content, and information. It is next to impossible to go through billions of websites and locate the relevant information on a particular topic. To make it easier, Google uses some tools and techniques to categorize trustworthy and relevant websites. This is the reason Google ranking system exists, to return the relevant search results to users. Indexing is one of the most useful techniques used by search engines for their general purpose. It is useful in identifying words and expressions that best describe a page and overall decides the ranking of a webpage or website.

Having said that, indexing is undoubtedly the indispensable step for websites to rank on search engines. Even websites owners are aware that they need to index their websites including digital content like images, videos, and documents to appear on the first page of Google search results.

Why do you need to block content from search engines from indexing ?

Indexing is basically a way of making your site visible to your targeted audience. Preventing your site from indexing will make it more difficult for your audience to find your site. Here are some common reasons to block indexing of your WordPress site. If you have one of the following reasons to block indexing on your WordPress site, then go ahead and read.

Creating a development or test site

A professional developer knows that before creating a new WordPress theme, it’s better to create a duplicate of the current website to work for development purposes. In that case, you don’t want Google to index your test site to compete with you for ranking or completely destroying your chance of ranking in future.

Dynamic content

If you update the content on your website very frequently, it’s better if you just block indexing on your site. It takes some time for search engines to crawl your site and index the content on your site and there is no use of indexing outdated content. Therefore, if your changes are too fast for the search engines to keep up with, it’s better to block search engines from indexing your outdated content and hurt your ranking.

Intentional duplicate content

We can’t state enough time how important is to have unique and original content on your site. Having duplicate content on your site can result in heavy Google penalties like getting your page de-indexed from the search engine. However, there can be scenarios like an advertising landing page that requires you to add duplicate page on your site. In that case, consider adding a canonical link to the page in question. However, if you need to publish an entire website of duplicate content, its best to determine how to discourage search engines from indexing specific pages.

Private content

If your website has some exclusive content for a handful of users only. For instance, if you run a membership program on your website and want to make some of the content available to the paid members only. Therefore, it is probably not a good idea to let search engine crawlers to index that paid content.

How to Prevent Search Engines from Indexing WordPress Site?

Discourage Search Engines from Indexing this Site

There is a misconception that not linking other websites to your domain name will prevent search engines from finding your site. However, that’s not the case, there are pretty other ways search engines can still find your website. For instance, domain search site results could be indexed with your link on them or someone previously owning your domain and still having links pointed to your site. Therefore, you have to do more than just avoid linking other websites to your domain. There are different ways to prevent your site from linking, these are-

  • Checking the WordPress search engines visibility checkbox
  • Editing the Robots.txt file
  • Applying password protection on your WordPress site through your web hosting cPanel
  • Applying password protection on your WordPress site through a plugin

Now that you are aware of the different ways to prevent Google from indexing your WordPress site, let me explain the step by step process to do it. In the next section, I am going to explain all 4 aforementioned ways to prevent indexing on your site. Let’s get started.

Steps to Prevent Search Engines from Indexing WordPress Site

There can be various reasons you don’t want search engines to get their hands on your site’s content. Therefore, here is the step by step process to help you block it.

Method 1: Block search engines from indexing WordPress site From WordPress Settings

This one is by far the simplest way to stop search engines from indexing your website is by preventing them from crawling it. If you are a WordPress site owner, you must be familiar with the working of your WordPress dashboard. Therefore, using this method should be like a piece of cake to you. To do this, login to your WordPress Dashboard

Step 1: Navigate to your WordPress dashboard, locate Settings> Reading.

Step 2: Scroll down to the Search Engine Visibility option and you will see a WordPress search engine visibility checkbox. Check the box that says Discourage search engines from indexing this site.

Step 3: Save your changes by clicking Save Changes.

When you check the checkbox, WordPress makes some edits in the robots.txt file and adds a line to your website header that discourages search engines from indexing your website. Most search engines respect your request and skip indexing your website. However, some search engines can also ignore your request and may continue to index some of your pages or even image files.

Method 2: Prevent Search engines from indexing WordPress site by editing robots.txt file using cPanel of your WordPress hosting.

If you want to do it manually, there is a way to stop search engines from crawling your site manually. You can use File Manager or an FTP client to edit the robots.txt file. This method instructs all search engines to skip crawling and prevent them from indexing your site. All you need to do is add a simple syntax to the robots/txt file to your website’s root directory.

Step 1: Login to the cPanel of your web hosting and navigate to your File Manager.

Step 2: Navigate to your WordPress root directory folder, which is often public_html, and find the robots.txt file under it. In case you can’t find it, create a new blank file.

Step 3: Right-click on the file and select Edit.

Step 4: Enter the following syntax-

User-agent: *

Disallow: /

This syntax will prevent search engines from indexing your whole site. In case you want to block indexing of a specific page or post, write the page’s subdirectory and slug. For instance: Disallow/blog/prestashop-review. Moreover, be careful when editing as the syntaxes in robots.txt files are case sensitive.

Method 3: Prevent Search engines from indexing WordPress site by Password Protecting using web hosting cPanel

If you are concerned that some search engines may not honor your request to skip indexing your WordPress site, you can try this method. Search engines and web crawlers don’t have access to password-protected files. You can password protect your web files to stop search engines to get access to your content and index it. Follow the given instruction to password-protect your files-

Step 1: Login to your web hosting cPanel and navigate to Password Protect Directories

Step 2: Choose the directory you want to protect with a password. For WordPress, it is usually located under the public_html folder or the www directory.

Step 3: After selecting the directory, enter your username and password and click on the Password Protect This Directory or Save option.

Step 4: Now, create a new user to login to the protected website.

That’s it, you just password protected your site!

Method 4: Block search engines from indexing WordPress site with Password Protecting using Plugin

In case above given method feels overwhelming to you, you can also automate this method. Or in case you are using managed WordPress hosting solutions, you don’t have access to cPanel. In such cases, you can use a plugin to password protect your WordPress site. There are tons of plugins that can help to password protect your site. You can use a Password Protected plugin . Password Protected plugin allows you to protect your WordPress site with a single password. Keep in mind that this plugin can only protect your WordPress site’s content, not your images or other uploaded files. Therefore, if someone enters the URL of your images, they will be able to gain access to your image file, even when you have this plugin activated. You can read about how to Remove Malware from a WordPress Website and how to install WordPress. Check out the WordPress theme builder and best WordPress website builders.

How to prevent search engines from indexing WordPress site: Conclusion

Here you have four different methods to stop search engines from indexing your WordPress websites. Whatever your reason may be, if you don’t want search engines crawling on your WordPress site, you can use any of the aforementioned methods to keep your site’s content away from them. You can use any of the given methods depending upon your requirements and comfort. Unfortunately, not all search engines play nice and some search engines may ignore your request to not index a page. I hope this tutorial is useful to you and you get what you are looking for. If you have ideas or any other technique to prevent indexing, do share with me. For more queries and information, please comment below.