via yoast.com => original post link
One of the most important aspects of SEO is optimizing the crawlability of your site. Search engines have near-endless resources, so they have the power to crawl everything they find — and they will. But, that is not the way it should be. Almost every CMS outputs URLs that don’t make sense and that crawlers could safely skip. With Yoast SEO Premium 18.6, we’re starting a series of additions to clean up those unnecessary URLs, feeds, and assets so that the more critical stuff stands a better chance of being crawled.
Making your site easier to crawl
Google and other search engines crawl almost everything they can find — as Yoast founder Joost de Valk proves in a post on his site. But it can be hard to get them to crawl what you want them to crawl. Moreover, crawlers can come by many times each day and still not pick up the important stuff. There’s a lot to gain for every party involved — from the crawlers, site owners, and environment — to make this process more sensible. Yoast SEO Premium will help search engines crawl your site more efficiently.