When it comes to search engine optimization, most website owners only pay attention to rel=nofollow attribute for links. They seem to forget how to use robots meta tag for SEO and to provide specific directives to search engines on which pages should be indexed and which pages should not be indexed.
If your website has lots of pages, it is not necessary that every page on the website has to be indexed by search engines. There are situations when you would want to prevent search engines from indexing a page (or a bunch of pages) on your website. There are situations when you would not want search engines to crawl any link on a specific webpage.
In all these situations, a robots meta tag comes in extremely handy as it can be used to tell search engines not to index a specific page or not to crawl any link on a given page on your website. Think of Robots meta tag as a common directive given to all search engine bots about a specific page on the website which should be treated in a special way.
How to Use the Robots Meta Tag?
Like any <meta> tag it should be placed in the <head> section of an HTML page on your website, before the </head> section. A sample example is given below
This is just a simple example of the usage of Robots meta tag in a webpage. Depending on your requirements, you can modify the content of this meta tag and achieve desired objectives. But before we discuss that, let’s see all the directives which are supported by the robots meta directive.
| Directive | What it does |
|---|---|
| noindex | Do not index this page and do not show this page in search results for user queries |
| nofollow | Do not follow any links placed on this page (both internal as well as external links) |
| noarchive | Do not show a “Cached” link in search results. |
| nosnippet | Do not show a text snippet or video preview in the search results for this page. A static thumbnail (if available) will still be visible. |
| notranslate | Do not offer the translation of this page in search results. |
| noimageindex | Do not index images on this page. |
| unavailable_after: [RFC-850 date/time] | Do not show this page in search results after the specified date/time. The date/time must be specified in the RFC 850 format. |
So if you are using the following code in the head section of a webpage, you are actually telling search engines not to index the content of the page and not to show the page in Google search results
<meta name="robots" content="noindex" />
Similarly, if you are using the following code in the head section of a webpage, you are actually telling search engines not to follow any link (both internal and external links) on that page and not to pass Google pagerank through these links.
<meta name="robots" content="nofollow" />
Note that you can combine multiple directives into a single code and tell multiple things at once. For example, the following code tells search engines not to index the content of the page and at the same time, not to follow any links on the page.
<meta name="robots" content="noindex,nofollow" />
Similarly, the following code tells search engines not to index the content of the page but to follow all the links on the page.
<meta name="robots" content="noindex,follow" />
Now depending on the requirements, you can use these directives in the meta robots tag of your website.
Frequently Asked Questions with Respect to Meta Robots Tag
Here are some frequently asked questions with related to Robots meta tag.
Is it Okay if I don’t have a Robots Meta Tag on my Website or on specific pages of my site?
It is perfectly all right not to have a robots meta tag in your website at all. It is completely optional. Search engines will not be offended not to find the robots meta tag on your website if you do not have them, this is just an additional requirement for those website owners who want to noindex or nofollow specific pages on their website. If your website does not have these requirements, there is no need for you to use the robots meta tag.
Does having the Robots meta tag add any additional SEO benefit?
No, not at all. Adding a robots meta tag does not give you an “edge” or additional benefit compared to not having a robots meta tag. It has no effect on search rankings and other metrics, so there is no need to obsess over the usage of robots meta tag for improving search rankings of a webpage.
Are search engines bound to follow the directive given in the Robots meta tag?
No.
Search engines are not bound to follow the directive given in the robots meta tag since the directive is given by you as a guideline, and it is upto the search engines to decide or honor that request. That said, in 99% of the cases, search engines do honor your request and carry out the instructions provided in the Robots meta tag.
Does adding Robots meta tag on a webpage prevent Crawling?
No, adding a robots meta tag does not prevent or prohibit search engine bots to crawl that specific page. Search engine bots will need to recrawl that URL occasionally to check if the meta tag has changed.
Is it a good idea to change the Robots meta tag too often on a section of your website?
No, it is generally not a good idea to frequently change the Robots meta tag on specific sections of your website. There must be a clear purpose of using the robots meta tag and you must not keep switching from one preference to another, without any clear objective in mind.
To wrap up, here is a short video by ex-Googler Matt Cutts where he explains when you should use The Robots meta tag to noindex a specific page but let Googlebot follow all the links in that page
Be Sure to read our SEO Guide which contains useful information about SEO and we have discussed in detail key SEO Concepts with examples