This code is for Google Adsense robots which help them to serve better ads on your blog. Either you are using Google Adsense on your blog or not simply leave it as it is. This is for all robots marked with asterisk (*). In default settings our blog’s labels links are restricted to indexed by search crawlers that means the web crawlers will not index our labels page links because of below code.
Now suppose if we want to exclude a particular post from indexing then we can add below lines in the code.
Here yyyy and mm refers to the publishing year and month of the post respectively. For example if we have published a post in year 2013 in month of March then we have to use below format.
This code refers to the sitemap of our blog. By adding sitemap link here we are simply optimizing our blog’s crawling rate. Means whenever the web crawlers scan our robots.txt file they will find a path to our sitemap where all the links of our published posts present. Web crawlers will find it easy to crawl all of our posts. Hence, there are better chances that web crawlers crawl all of our blog posts without ignoring a single one.
This was the today’s complete tutorial on how to add custom robots.txt file in blogger. I really try with my heart to make this tutorial as simple and informative as possible. But still if you have and doubt or query then feel free to ask me. Don’t put any code in your custom robots.txt settings without knowing about it. Simply ask to me to resolve your queries. I’ll tell you everything in detail. Thanks guys to read this tutorial. If you like it then please supports me to spread my words by sharing this post on your social media profiles. How to Check Your Robots.txt File?