locked
msnbot is overly indexing my site RRS feed

  • Question

  • From the server logs, I found out that the msnbot is indexing my site too much.  Usually from 2 am to 4 am est, I see that the index page itself is indexed every minute, along with many other pages.  I'm getting 3000+ views at an hourly time-frame in the early morning hours.  Can someone explain to me why this is happening?  I'm trying to reduce the number of views by using a robots.txt file, so that msnbot doesn't index certain parts of the site:

    User-agent: msnbot
    Disallow: /page/

    User-agent: *
    Disallow:

    Do you think this will work?  Any other suggestions?
     
    Tuesday, February 17, 2009 6:18 PM

Answers

  • That will work.

    Other options to think about: have you tried setting a crawl delay? this crawl delay is in seconds. You could also use conditional get if you have not instituted that yet too.

    Brett
    Program Manager, Live Search Webmaster Tools
    • Marked as answer by Brett Yount Monday, February 23, 2009 8:10 PM
    Monday, February 23, 2009 8:10 PM

All replies

  • That will work.

    Other options to think about: have you tried setting a crawl delay? this crawl delay is in seconds. You could also use conditional get if you have not instituted that yet too.

    Brett
    Program Manager, Live Search Webmaster Tools
    • Marked as answer by Brett Yount Monday, February 23, 2009 8:10 PM
    Monday, February 23, 2009 8:10 PM
  • hi, i've the same problem, i test my robots.txt and dont founde any errors, MSNBot doesnt visit my website www.clubdownload.com.ar
    Thursday, February 26, 2009 6:33 PM
  • my problem still, i trie to find www.club-download.com and www.club-download.com/webmal  in msn search and dont found any results
    Admin of: HTTP://www.clubdownload.com.ar & http://www.leonardosasso.com.ar MCP & Developer
    Wednesday, April 1, 2009 8:02 PM