locked
MSbot not obeying robots.txt RRS feed

  • Question

  • Dear Sir,

    Can you please explain why your Live Search robot is accessing parts of my website that are expressly forbidden to be accessed by my robots.txt file?
    Namely -
    User-agent: msnbot
    Disallow: /bbrd/

    The following IP addresses have tried to access files in this directory this morning and in so doing have been identified as a Security Threat by my systems.
    If they try to access these files again they will be banned and forbidden access to my website entirely.

    02-03-2009 09:09:54     65.55.165.102     
    02-03-2009 09:09:54     65.55.165.99    
    02-03-2009 09:09:54     65.55.165.65    
    02-03-2009 09:09:53     65.55.165.51    
    02-03-2009 09:09:53     65.55.165.68    
    02-03-2009 09:09:53     65.55.165.59    
    02-03-2009 09:09:53     65.55.165.115    
    02-03-2009 09:09:53     65.55.165.87    
    02-03-2009 09:09:53     65.55.165.20    
    02-03-2009 09:09:53     65.55.165.123    
    02-03-2009 04:53:47     65.55.110.204    

    If we cannot trust a corporation as big as Microsoft to obey the rules and protocols on the internet who can we trust.

    Regards,
    Adrian Hall

    Monday, March 2, 2009 10:22 AM

Answers

  •  My Apologies.  Are there any other directories it is crawling that it shouldn't? If so, please provide them and I will exclude them from future crawling.

    Thanks,

    Brett
    Program Manager, Live Search Webmaster Tools
    • Marked as answer by Brett Yount Friday, March 20, 2009 6:35 PM
    Thursday, March 5, 2009 5:52 PM

All replies

  •  My Apologies.  Are there any other directories it is crawling that it shouldn't? If so, please provide them and I will exclude them from future crawling.

    Thanks,

    Brett
    Program Manager, Live Search Webmaster Tools
    • Marked as answer by Brett Yount Friday, March 20, 2009 6:35 PM
    Thursday, March 5, 2009 5:52 PM
  • Why should ANYONE have to tell YOU where not to go, when there is an established method for doing exactly this (ITS CALLED ROBOTS.TXT, AND YOU ARE IGNORING IT)?

    Seriously Brett, you're annoying as ____. I don't envy your job (lying non-stop to people you are entrusted to help), but that's no excuse for all the dirty ____ microsoft is doing.

    MS is ignoring robots.txt. This is fact. You are aware of this. EXPLAIN
    MS is referer-spamming sites with bogus keywords. You are aware of this. EXPLAIN

    Why do you respond to every question with lies?
    Wednesday, April 22, 2009 12:00 AM