Google developed it highly technical platform where algorithms fix how much each website crawl, Google crawler focus on without overwhelming your web server bandwidth, by default Google put it 0.03-requests per second, 33.333 seconds between requests, this is by default by Google for each site when you register website with Google webmasters tool.
Even you can increase those both number and increase, it’s all up to you, what happen sometimes when we register our whole website (ex. www.website.com) on Google webmasters tool, Google bot start crawling website on daily or weekly basis to take all the information to Google server for indexing all the necessary content to fixing it’s ranking on Google search result.
Sometimes website hosting server goes down due to high traffic simultaneously, at that particular time Google crawler crawl maximum pages per second if you will increase the speed of Google crawl rate, so before increasing your website crawl rate you should check your hosting server configuration and capability to handle that much amount of traffic.
Steps to Increase and Change Googlebot crawl rate
- First login to your website webmaster tool by following link fot Google webmasters www.google.com/webmasters.
- Now you will at homepage of Google webmasters tool, where you will see website those you add for crawling and indexing.
- Click on your website now it will give you many option related to your website, but you need to click on right hand side gear sign where you will get an option to change crawl rate of your website.
- After clicking on gear sign it will shows more option click on site setting
- Now your site setting will be on screen, where you can fix Preferred domain and Crawl rate of your website.
- Now you get under Crawl rate Let Google optimize for my site (recommended) selected, you need to choose an option Limit Google’s maximum crawl rate.
- After choosing this option it will shows more related to fix your website crawl rate, increasing Speed must first take care of your server.
- Then press save button after fixing your server crawl rate.
Most of the users choose this option those who have more then 50000 pages, because by default Google take long time to Crawl those pages, if you have high performance server you can easily increase default setting for crawl pages, this setting set Google to crawl your site faster.
