Faster Crawl

If you haven’t setup your sitemaps in Google’s Webmaster Central, then do that first, then come back and read this. Google released a new feature a month or two back that lets you see what the ole’ GoogleBot has been up to. It shows you how many pages per day, bytes per day, and average load time of each page GoogleBot crawls. It also lets you set the crawl rate. If you’ve got a fast enough server, or a small site, you can probably set the speed to fast. This will tell Google to crawl more pages, more quickly and not throttle GoogleBot. Since making this change I’ve noticed a 10-15% increase in the number of pages crawled each day and I’m seeing pages get indexed more quickly (within 24 hours where it used to take 72+)

Enjoy the faster indexing, and don’t complain to me if your web host bitches about GoogleBot killing their server!