For various reasons, we may be interested in Google not provisionally indexing our website that is under construction or not indexing certain pages of our WordPress.
In this post we are going to see what this is about the indexing of contents by search engines, in what circumstances we may be interested in not producing this indexing and depending on the type of NO indexing we want to do, how we can do it.
1.- What does it mean NOT to index a page of a web?
Google and the rest of the search engines crawl and visit the different pages of the web sites, register them and store them in their database in order to offer them as results in the different searches made by the users in the search engine.
Therefore, NOT indexing a page of a website, means that search engines will not be able to register and save that page in their database and therefore NOT be able to offer users in the search results.
When we prevent a page from being indexed, we do it for all search engines: Google, Bing, Yahoo, etc..
2.- Differentiate between not indexing a web page or not blocking an entire website so that it is not indexed by Google.
We have to differentiate between them:
a.- DO NOT index a web page: we can prevent Google and other search engines from indexing certain pages on our website. But there will be other pages of our website that will be indexed.
b.- DO NOT index an entire website: we can prevent Google and other search engines from indexing each and every page of our website. In any case, it is possible for search engines if they index the home or home page. But they will not index any page inside the website.
3.- For what reason can we need that certain pages of a web are not indexed?
There may be several reasons why we do not want certain pages of our website or the entire website to be indexed.
We must bear in mind that time is limited, also for Google bots, which are dedicated to crawling the Internet. The best thing is to tell Google which pages you don’t want to show in the search results, so that you don’t invest time in them and you can skip a page that is important (Crawl Budget).
a.- Pages that we do not want to show in results: there are pages such as the legal notice, privacy policy, cookie policy and others, which do not interest us that Google shows in the search results. In addition, these pages are usually copied from each other and therefore, it is not even good that they are indexed by duplicate content.
b.- Site under construction: while we are building our website, we may not be interested in Google indexing our site and offer our pages half-built in search results. When we finish building the website, we will allow search engines to index our website.
c.- Products copied store: it is possible that we work the SEO of the categories of our online store and certain products. But there are other products that we are not interested in Google indexing because they are copied from the manufacturer. We want customers to reach the products through the positioning of categories and not products.
If you can think of any other reason why we might want Google not to index the content of a website, put it in the comments.
4.- How to block a complete wordpress website from being indexed
We can block a complete wordpress website in two ways, so that it is not indexed:
4.1.- Do not index WordPress, through Settings
The most effective way to prevent Google from indexing our entire WordPress website is through the Settings. To do this we go to the administration of our WordPress, and click on: “Settings > Read”.
You’ll see a field that says the following:
Search Engine Visibility: Discourages search engines from indexing this site. If we mark the checkbox, we will prevent Google and other search engines from indexing our entire site. If we do NOT mark the checkbox, our site can be indexed by search engines.
Remember to click on “Save changes”.
4.2.- Activate a maintenance page
We can use a plugin like Maintenance or similar to show a single maintenance page.
In this way, Google will not be able to access the web pages (only the maintenance page) and therefore, it will not be able to index the content of the different web pages.
Unlike the previous method, if we use this form, users will not be able to access our website either. With the previous method, users if they could access our website, and only search engines would not be able to do so.
In this post you have how to install and configure the Maintenance plugin, to put a WordPress web in maintenance.
Note: Even if we put the web in maintenance mode, we could also add the option seen in section 4.1.
5.- How to prevent certain pages or wordpress entries from being indexed
To avoid indexing one or more specific WordPress pages, I use a feature that includes the Yoast SEO plugin.
Yoast SEO, is the plugin for SEO, most used with WordPress. In addition to the basic meta tag editing functionalities, it also includes page-specific indexing functionalities or WordPress entries.
To do this, we will proceed to edit the page or entry that we want to prevent Google from indexing. Then, in the Yoast SEO block, click on the tab with the icon of a gear.
In the next option: “Allow search engines to show this page in the results?, we will select: “NO”.
Then we’ll save the page
5.1.- It is recommended to put nofollow type links if we have the pages that we don’t want to index linked.
It could happen that if we have links from the home page or other pages of the website that point to the pages that we do not want Google to index, it could happen that the latter if they were indexed by Google.
This could happen for example with the legal notice, privacy policy, etc. that are usually linked from the footer.
There are different solutions to avoid this. In principle, it has worked very well for me to put those links as nofollow, but there may be other solutions, as recommended by Miguel, it is possible that we address in other posts.
Note: to make a nofollow link you simply have to add the text nofollow in the rel attribute of the link. To do this, you have to see the link in the HTML tab of the WordPress editor. If it already has the rel attribute, then you add the nofollow text inside it. If it doesn’t have the rel link yet, then you put it in the following way rel=”nofollow”.
An example of a nofollow link would be this:
<a href=”https://www.example.com/aviso-legal” rel=”nofollow”>Legal Notice</a>
In this post you have the most detailed process: How to put a nofollow link
6.- How to make a Woocommerce product not be indexed
As I commented before (section 3.c), it is possible that for different reasons, we are interested in not indexing some products of our Woocommerce store.
For this, we will use the Yoast SEO plugin.
We will proceed to edit the product that we want Google not to index, and then follow the same process followed in section 5
7.- Do not index WordPress image files
WordPress, by default, creates a page or card for each of the images we upload, in addition to the URL of the image itself. These cards or pages (also called attachments) should not be indexed by Google, as we can be penalized for what is known as Thin Content (poor quality content).
The pages created by WordPress associated with each image (attached) only contain that image and no text content, so they correspond to what Google calls Thin Content.
In order for Google not to index these pages or files associated with the images we must do the following in our Yoast SEO.
We should go to the part: “SEO > Search Engine Appearance > Media”.
The selector: “Reroute URLs from attachments to the attachments themselves” we will check the option: “YES” and save.
We are saying in this way that redirects the URLs of these files to the urls of the images themselves.