locked
msnbot ignoring my robots.txt RRS feed

  • Question

  • I just looked at my sites indexed pages and there are clearly pages there which are disallowed in my robots.txt. Quite a few even!

    I checked the .txt in the validator, all is well. Google and Yahoo dont seem to have this issue. I've also had dozens of pages showing in the index without any of the title or description information (random pages for no reason).

    I have several pages which need to be removed from the index immediately as I really dont want to ask for spam or attempted hacks on those pages!
    Monday, December 1, 2008 2:48 AM

Answers

  • Hi Aukuma,

     

    could you please send the URLs to lswmc@microsoft.com and I will look into why we disregarded your robots.txt.

     

    Thanks,

     

    Brett

    Wednesday, December 3, 2008 6:33 PM