Sign in
Microsoft.com
 
Microsoft
 
 
 

locked
MSNBot ignores nofollow/robots RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote

     

    Pretty sure this has been widely reported previously but our page count is through the roof because it's indexing images with nofollow tags attached on our site.  Long and short is there a solution coming soon to MSN not following nofollow/robots directives?   This doesn't appear to be something that should require a major modification.
    Thursday, March 6, 2008 1:30 AM

Answers

  • Question
    Sign in to vote
    0
    Sign in to vote

    In closing out this thread I just wanted to make note that the MSN team apparently has rectified the problem.  The visits have returned to a more reasonable number representative of the size of the site.  The visits are being spaced normally and are not taxing resources. 

     

    MS is actively addressing issues -although not always as fast as all of our businesses would like and need at times.   It's been a long road with many bumps but in the end the people that do help are very professional.

     

    Wednesday, March 12, 2008 3:43 AM

All replies

  • Question
    Sign in to vote
    0
    Sign in to vote

     

    Anyone home at MSN?   It's pulling 1.5k pages a day going on 3 days.  It's pulling 2-3 files per second at times which hardly would seem to be good bot behavior.

    We're considering blocking it for a short time.   Obviously something has gone wrong - again.  

    Thursday, March 6, 2008 3:08 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    Working with MS but no resolution yet.  Just as an update at times it's pulling close to 2k pages in a 24 hour period.  Reviewing the logs it's now up to 4-5 pages a second at times and is pulling the same pages over and over again in the same period.   It almost looks like it's just duplicating or triplicating it's normal activity on the site.  Will update when it is resolved - seems to be two extremes with MS, no indexing or uber indexing.
    Friday, March 7, 2008 8:54 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    This will be the last update until it's resolved, but information was forwarded to the MS teams.  Something is bonkers with the bot,

     

    2008-03-10 01:39:26 mysite GET /robots.txt - 80 - 65.55.104.29 HTTP/1.1 msnbot/1.1+(+http://search.msn.com/msnbot.htm) - -

    2008-03-10 01:39:29 mysite GET /robots.txt - 80 - 65.55.104.29 HTTP/1.1 msnbot/1.1+(+http://search.msn.com/msnbot.htm) - -mysite
    2008-03-10 01:39:29 mysite GET /robots.txt - 80 - 65.55.104.29 HTTP/1.1 msnbot/1.1+(+http://search.msn.com/msnbot.htm) - -mysite

     

     

    There were no breaks between requests...it went for the same file from the same IP 3 times in 3 seconds.  It will sometimes request the same url 3-5 times in a few hours.   Obviously it's "confused"

     

    If it starts to go hyper on your site take a peak at the robots file in the logs.   The other true live bots are behaving fine.

     

     

    Monday, March 10, 2008 1:46 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    In closing out this thread I just wanted to make note that the MSN team apparently has rectified the problem.  The visits have returned to a more reasonable number representative of the size of the site.  The visits are being spaced normally and are not taxing resources. 

     

    MS is actively addressing issues -although not always as fast as all of our businesses would like and need at times.   It's been a long road with many bumps but in the end the people that do help are very professional.

     

    Wednesday, March 12, 2008 3:43 AM