crawling rates RRS feed

  • Question

  • Is there any way to throttle up or throttle down the crawling speed in Webmaster tools?  I've recently updated millions of URLs and see that my new URLs (with 301 redirects from the old URLs) are not being picked up very quickly by the MSN bot.

    Wednesday, July 2, 2008 12:59 AM


  • Throttle down: yes. Use the crawl-delay REP directive.

    Throttle up: No


    Wednesday, July 9, 2008 2:53 AM