Customise Robot text for Blog/Website Optimization

Customise Robot text for Blog/Website Optimization

Sponsor


Optimization of robot.txt is technique used by content marketers to gain organic traffic. What are robots.txt? Simply put Robots.txt is a text file on the server in that you can write which directories, web pages, or links should not be included for search results. It means you can restrict search engine bots to crawl some directories and web pages or links of your website or blog.

Custom robots.txt is available for Blogger. In Blogger search option is related to Labels. If you are not using labels wisely per post, you should disallow the crawl of the search link.

In Blogger, by default, the search link is disallowed to crawl. In this robots.txt, you can also write the location of your sitemap file. A sitemap is a file located on a server that contains all posts’ permalinks of your website or blog. Mostly sitemap is found in XML format, i.e., sitemap.xml.

If your blog posts  is less than  500 posts it is recommendable to you use the robot.txt for maximum optimization:

User-agent: Mediapartners-Google
Disallow: User-agent: *
Allow: /
Disallow: /search Disallow: /b
Sitemap:https://www.simnify.blogspot.com/feeds/posts/default?orderby=updated

However, you have more than 500 posts then you need to update this robots.txt into this; 

User-agent: Mediapartners-Google
Disallow: User-agent: *
Allow: /
Disallow: /search Disallow: /b
Sitemap:https://www.simnify.com/atom.xml?redirect=false&start-index=1&max-results=500

Sitemap:https://www.simnify.com/atom.xml?redirect=false&start-index=501&max-results=500


You need to change site URL from www.simnify.blogspot.com or to your customised domain. 
                                                                
Sponsor