MSNBOT_Mobile Not honoring robots.txt RRS feed

  • Question

  • Hi Folks,

    I've seen the other thread about the msnbot not honoring but there is really no resolution there. I have several sites and one of those sites has a bot trap with a directory excluded in the robots.txt file. Other than a few wayward bots, all the major players honor the directives except the MSNBOT_Mobile.

    Over the last week or so the MSNBOT_Mobile bot has triggered my bot trap at least 6 times. I'm not particularly worried about the visits but the bot should not be ignoring the directives in the file.

    Anyone else experience this problem and if so, is anyone working on fixing the problem?

    Thursday, January 17, 2008 6:38 AM


  • Can you give more specifics about where it is not following the directive?  I will be glad to pass this on to the crawler team.   It is our goal to make sure we respect the directives.




    Monday, February 4, 2008 3:46 AM