via yoast.com => original post link
The robots.txt file is one of the main ways of telling a search engine where it can and can’t go on your website. All major search engines support its basic functionality, but some respond to additional rules, which can be helpful too. This guide covers all the ways to use robots.txt on your website.
Warning!
Any mistakes you make in your robots.txt can seriously harm your site, so read and understand this article before diving in.