locked
MSN bot crashes my site (like a DoS attack) RRS feed

  • Question

  • Hi,

    I'm Klaus, the man behind GPSies.com. Since several hours 2 bots (65.55.51.6 and 65.55.51.10) of MSN sends to many requests (always the same requests, about 100 requests per minute, sometimes 5 per second). The problem is, the the special content is delivered by a database and my server hangs. I stopped the server and I added the both IPs to an application blacklist - it works now for me.

    The user-agent is: msnbot/1.1 (+http://search.msn.com/msnbot.htm)

    IP and name:
    65.55.51.6 -- msnbot-65-55-51-6.msn.com
    IP and name: 65.55.51.10 -- msnbot-65-55-51-10.msn.com

    How can I tell the MSB bot, that he should do not crawl the content? How can I stop it?

    I read in a forum, that a way is, to set the robots.txt to:

    User-agent: msnbot
    Crawl-delay: 30

    I did it. Is there also another way to stop the "DOS"?

    Thank you very much!

    Best regards from Berlin, Germany

    Klaus Bechtold

    Tuesday, January 6, 2009 1:27 PM

Answers

All replies

  • Hi,

     

    I suggest reading our FAQ located at the top of this forum as well as http://blogs.msdn.com/webmaster/archive/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation.aspx. Please note that if you block MSNbot you will not get indexed in live.com

     

    Brett

     

    Tuesday, January 6, 2009 11:54 PM
  • Hi Brett,

    thank you for your hint :-)

    My problem is, that the MSNbot is totally crazy. Here is an example of my LogFile (only MSNbot entries). The bot is visiting GPSies.com
    (always with exactly the same request) in the following example 37 times in just 3 seconds! I do only have one server! The bot tries to get an "expensive" dynamic (database) content.

    I'll accept the exclusion at live.com - it's better, to have a server online than a listed (offline) site at live.com ;-)

    Here is my log - brandnew:

    090107 11:25:07.916: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:07.917: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:07.955: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:07.971: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.108: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.379: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.536: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.543: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.595: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.629: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.662: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.751: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.792: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.835: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.867: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:08.912: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.030: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.221: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.323: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.326: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.368: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.405: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.485: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.488: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.526: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.529: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.566: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.714: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.718: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.880: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:09.884: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:10.042: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:10.046: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:10.055: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:10.066: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:10.078: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)
    090107 11:25:10.079: remoteAddr: 65.55.51.6, remoteAddrName: msnbot-65-55-51-6.msn.com, useragent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)

    Is there possibility to report live.com my problem?

    Thank you!

    Cheers, Klaus

    Wednesday, January 7, 2009 10:41 AM