locked
MSN Bot ignoring robots.txt? RRS feed

  • Question

  • I have a number of pages in the index that should be being ignored if MSNBOT is reading my robots.txt.
    It seems these pages that should be ignored are being indexed whilst the rest of the site isn't, any ideas what's gone wrong?

    My site was fully indexed a month or so ago but now down to about 20 pages and probably half of them are pages that should be getting ignored.

    The robots.txt file isn't new, it's been there for at least 6 months and I'm sure the pages that should be getting ignored weren't listed lasted time I checked a couple of months ago.
    Friday, March 20, 2009 7:30 PM

Answers

  • Hi,

    Would you mind monitoring this? I know we made a few changes on the backend, so hopefully this is corrected. If the bot continues to crawl them, please email me at lswmc@microsoft.com and I will add your site to our list.

    Thanks
    Program Manager, Live Search Webmaster Tools
    • Marked as answer by Brett Yount Friday, April 3, 2009 2:55 PM
    Friday, April 3, 2009 2:49 PM

All replies

  • Hi,

    Could you give me your domain name and I will take a look.

    Thanks,

    Brett
    Program Manager, Live Search Webmaster Tools
    Tuesday, March 31, 2009 8:05 PM
  • Hi Brett
    Thanks for offering to take a look, the domain is www.thefeedstation.com

    You'll see that url's like www.thefeedstation.com/tellafriend/tell_74.html are being indexed and these should be being ignored. Many seem to have disappeared from the search results since I posted so I think maybe the msnbot made an error and now it's slowly being corrected?
    Friday, April 3, 2009 9:27 AM
  • Hi,

    Would you mind monitoring this? I know we made a few changes on the backend, so hopefully this is corrected. If the bot continues to crawl them, please email me at lswmc@microsoft.com and I will add your site to our list.

    Thanks
    Program Manager, Live Search Webmaster Tools
    • Marked as answer by Brett Yount Friday, April 3, 2009 2:55 PM
    Friday, April 3, 2009 2:49 PM
  • Brett,
    All the urls that shouldn't have been indexed have now gone from your index but also it seems so have all my other urls as well. My site isn't blocked so I'm not sure what we've done to upset the msnbot, first it decided to index pages it should have ignored now it's decided to index nothing at all.

    From my logs I can see MSNBot occassionally acessing my sitemap.xml but then it leaves.

    Appreciate any advice you can offer.
    Wednesday, May 6, 2009 12:15 PM
  • I see we have 2 pages in the index. Hopefully the other good pages will start reappearing over the next week or so. If not, please ping me again and I will dig deeper.
    Program Manager, Live Search Webmaster Tools
    Wednesday, May 6, 2009 3:58 PM