locked
Help!!! MSNBot showing results for my Disallowed directories RRS feed

  • Question

  • Hello,

    I placed a robots.txt file in my website's root directory a couple of days ago.  I used the WebTools Validator to check my robots.txt file, and all is a-ok.

    But!  Whenever I try to search for files in my "disallowed" directories ... results DO show up.

    Please help. 

    The website is: http://www.primaryimmune.org

    Let me know if you need any other info.

    Many thanks,

    Frustrated WebMistress
    Friday, September 19, 2008 4:15 PM

Answers

  • Hi,

     

    1. I am sorry, but I didn't receive them. Could you resend with the same subject as this thread to a-bryoun at microsoft(you know the rest)?

     

    Thank you again,

     

    Brett
    Wednesday, October 8, 2008 4:57 PM

All replies

  • How long had the website been up before you added the robots.txt file, and had the MSN bot crawled the site before it was added?

     

    ... Duane

     

    Saturday, September 20, 2008 3:13 AM
  • Hello Duane,

     

    And thank you for replying. 

     

    The website itself has probably been up for about 10 years ... although a redesigned version was rolled out recently (first week of July '08).

     

    I just put the robots.txt file in the root directory about ... 3 or 4 days ago.  But I'm surprised that results for the 2 directories I'm disallowing are showing up in the MSN searches, since the other engines I tried have already started blocking them.  (Google, Yahoo, Ask, AltaVista, and AllTheWeb ... more than a little shocked that those last two are still out there ;o).

     

    I suspect that MSNBot is a little slower in checking for robots.txt than other engines ... just wondering when we can expect the file to be picked up.

     

    Thank you,

     

    Susan

     

    Saturday, September 20, 2008 12:42 PM
  • Hello again,

     

    I've checked again this morning, and my disallowed directories are still getting picked up.

     

    The reason I'm antsy about this is that one of our users put up some personal info that they shouldn't have; though they later removed that info, the CACHED version is still showing up in the MSN Live Search.

     

    I'd really like this info to be blocked ASAP. 

     

    As noted, all the other search engines I tested are already blocking my disallowed directories.

     

    If there's anything you can do to expedite the robots.txt file being picked up by your crawler, I'd very much appreciate it.

     

    Thank you,

     

    Susan

    Monday, September 22, 2008 1:33 PM
  • Hi again out there,

     

    I've checked the LiveSearch engine again today, and it is STILL showing directories that, according to my robots.txt file, should be blocked.

     

    How can we expedite the blocking of my disallowed directories?????

     

    Looking forward to your response,

     

    Susan

    Tuesday, September 23, 2008 7:49 PM
  • Hi,

     

    Could you please send the URLs to lswmc@microsoft.com. Please also fill out our support form so that we can get people looking at this issue. 

     

    Also, what is your domain name so that I can do some initial investigation.

     

    Thanks,

     

     

    Brett

    Wednesday, September 24, 2008 6:10 PM
  • Hi Brett,

    Thanks so much for replying to my post!

    The domain I'm concerned about is: http://www.primaryimmune.org

    I'll send the other info to the email address you provided, as you requested.

    Thanks again!

    Susan
    Wednesday, September 24, 2008 7:21 PM
  • Hi again Brett,

    FYI, I tried to send you an email, but ... according to MAILER-DAEMON, the account I'm trying to write to doesn't exist. 

    I think maybe you made a typo when you typed out your email address ... no biggie!  Please just send a response and try to type out your address again - hopefully this time it will work out alright Smile.

    Thank you,

    Susan
    Wednesday, September 24, 2008 7:54 PM
  • Hi Brett,

    I just wanted to give you the latest update on this issue.

    I filled out the support form, and received a response from Customer Service.  However, the response was just an explanation of how MSN crawls, with an added explanation about the directory being found on other sites, which would result in LiveSearch picking up the directory.

    I understand that if someone else posts a link to one of my "disallowed directories"  on their own website, that that link will show up in Search Results. I'm fine with that.  My problem is that LiveSearch is displaying results from MY website, which it should not be doing.

    I responded to Cust Svc with the above info, and I'll await their response.

    In the meantime, have you gained any  insight as to why MSN is ignoring my robots.txt file??

    Thank you,

    Susan
    Friday, September 26, 2008 3:36 PM
  •  

     

    Hi Susan,

     

    Looking at your robots.txt file, I am assuming we are crawling one or both excluded paths. Sorry for the email address. That is the correct one, but it isn't working for me either, so I'll get that checked out. In the mean time, could you send the URLs and some log files for our devs to review? They will probably need this information to figure out what is going on (and maybe how to fix it). I'll also place a request for permanent URL removal of said pages. Please send this information to (a-bryoun at Microsoft period com). Sorry for not giving it to you straight, but I don't want a ton of spam in my inbox.

     

    Thanks 

    Friday, September 26, 2008 11:09 PM
  • Hello Brett,

     

    I checked MSN's Search Engine today, and it's still showing results for my disallowed directories... yowza!

     

    Tech Support contacted me a week or so ago and said a specialist would be contacting me soon with an update, but I've yet to hear from anyone.  So, I've written back to Tech Support, and also sent you an email ... not sure who will respond first and where, but I just wanted to keep this topic alive, as I'd like to get those directories out of the search index.

     

    I was wondering if you were able to determine anything from the logs I sent you ... if so, do let me know!

     

    Thank you,

     

    Susan

    Tuesday, October 7, 2008 7:07 PM
  • Hi,

     

    1. I am sorry, but I didn't receive them. Could you resend with the same subject as this thread to a-bryoun at microsoft(you know the rest)?

     

    Thank you again,

     

    Brett
    Wednesday, October 8, 2008 4:57 PM
  • Hello Brett,

     

    Thank you for responding to my post.  I have just sent you a new email, at the address that you provided - hope you get this one!

     

    Also, I have news - I plugged in some of the search phrases that have been pulling up the disallowed directories, and today, MSN’s Search Engine didn’t pull up any results! I don’t know if it’s magic, or it’s temporary, or it’s luck, or a weird gremlin that went away, or someone at Microsoft did something, but this is the first time the search engine has behaved as expected.

     

    I’ll check again tomorrow and make sure this behavior is persisting – hope things are fixed!

     

    Best,

     

    Susan

    Thursday, October 9, 2008 9:17 PM
  • Whoops ... I have an update. 

     

    The email I sent to you just bounced (I sent the email to a-bryoun at microsoft dot com, I swear I did).  This time the undeliverable message says "Unknown User".  Not sure what's going on, but am glad you can still read these posts.

     

    On the plus side, MSN's search engine is still working properly -- that is, it's no longer showing search results for my disallowed directories --  and I really hope this behavior continues!

     

    I don't know how this got fixed, but I  hope it stays fixed ... I'll post again if things go on the fritz.

     

    Many thanks,

     

    Susan

    Thursday, October 9, 2008 9:26 PM