via fixrunner.com => original post link
A deep understanding of WordPress robots.txt will go a long way in helping you improve your website’s SEO. And in this guide, you will get to learn what robot.txt is all about, and most importantly learn how to use it.
Basically, robot.txt is made for robots – which, for instance, are software that crawls through web pages and index them for search results.
It allows website owners bar search bots from crawling certain pages or content on their website. The wrong use of robot.txt could ruin your site SEO.