locked
How can I speed up the crawling-speed of msnbot RRS feed

  • Question

  • Hi,

    we have a big site. Currently we have 2,4 Million pages in Google index (+30k pages/day). Google spiders about 200k pages/day. MSNBot only spiders about 500 pages/day. We only have 1,4k pages in bing index. How can we speed up msnbot? Why is it so slow? We do not have crawl-delay set in robots.txt
    Wednesday, June 10, 2009 8:47 AM