locked
crawl rate limit RRS feed

  • Question

  • Is there a way to limit the amount of data MSNBOT can download from my website daily?

    I get around 1K hits a day and with my pages being 500KB this is somewhat of a problem.

    What is even more annoying is that pages hardly ever change yet they are being recrawled and recrawled...

    Please help!

    Thursday, June 19, 2008 11:52 PM

Answers

All replies