Sign in

Microsoft.com

Microsoft

Remove From My Forums

crawl rate limit

Question
0

Sign in to vote

Is there a way to limit the amount of data MSNBOT can download from my website daily?

I get around 1K hits a day and with my pages being 500KB this is somewhat of a problem.

What is even more annoying is that pages hardly ever change yet they are being recrawled and recrawled...

Please help!

Thursday, June 19, 2008 11:52 PM

Answers

0

Sign in to vote

Hi,

after posting the above comment, I ran into an article that might help you: Save bandwidth costs: Dynamic pages can support If-Modified-Since too

Hope that helps!

Brett

Wednesday, July 9, 2008 4:10 PM

All replies

0

Sign in to vote

four days and still no answer

Tuesday, June 24, 2008 4:12 AM
0

Sign in to vote

Hi,

yes, there is a way to set the crawl rate using robots.txt.

Please review the robots post:Crawling Performance and Crawl-Delay

Wednesday, July 9, 2008 1:59 AM
0

Sign in to vote

Hi,

after posting the above comment, I ran into an article that might help you: Save bandwidth costs: Dynamic pages can support If-Modified-Since too

Hope that helps!

Brett

Wednesday, July 9, 2008 4:10 PM