The ever-changing Google has been upgraded with some new rules. And this time, Google is going to stop supporting all the unpublished and unsupported rules in the robots exclusive protocol.
Google has already announced on the Google Webmaster Blog that this rule will be effective from September 1, 2019. This means GoogleBot will no longer support the robots.txt file that is related to indexing. The company has also announced to remove it and start using an alternative as soon as possible.
Why This Change?
Google has been trying to change it for years to standardize the protocol. According to the company, the robots.txt rules were never documented by Google, that means it is not an official directive. Hence, in comparison to GoogleBots, their usage is very low. It also explains that the mistakes of these unsupportive rules affect the websites’ presence in the search results of Google.
Google officially twitted that they are going to say goodbye to unofficial and unsupported rules in robots.txt. It also has requested to the publishers, who use these rules, to learn about the other options from their blog post.
The Alternative Options:
The alternative options, declared by Google on their Google Webmaster Blog, are –
- Noindex in robots meta tags: Since noindex is supported both on the HTML and HTTP response headers, it is the most effective way to remove the URLs.
- 404 and 410 HTTP Status codes: The status of these codes refer that the page does not exist. They can also drop the URL from Google’s index when crawling is allowed.
- Password protection: One can hide a page behind a login ID and password to remove it from Google’s index when required.
- Disallow in robots.txt: Search engines can index a page that is visible. So, by blocking the pages, one can stop them from being crawled.
- Search console removes the URL tool: This is a quick and easy process of removing URLs temporarily from Google’s search results.
Read the official Google announcement : Click Here
Read the official Google tweet : Click Here