locked
MSNBot Not Hitting Robots.txt RRS feed

  • Question

  • Hi, I have several websites all experiencing the same issue. All have sitemaps loaded into Webmaster Central and all have a robots.txt. MSNBot seems to hit the / directory of each site but does not hit robots.txt (or any of the other pages) which after reading some of the posts on this forum seems to be a pre-requisite for Live search. All the robots.txt files validate in using the roots.txt validator. One of the URL's is www.mycreditstatus.co.uk - if anyone can shed some light on why this is happening, I'd be most grateful.
    Tuesday, December 9, 2008 9:10 AM

Answers

  • Hi Altnet,

    I have submitted your sites using an internal submission tool, so maybe that will help. Good news is, you aren't blocked on our end.



    Brett
    Program Manager, Live Search Webmaster Tools
    • Marked as answer by Brett Yount Tuesday, February 24, 2009 5:45 PM
    Tuesday, February 24, 2009 5:44 PM

All replies

  • Your current robots file is:

    User-agent: *
    Sitemap: http://www.mycreditstatus.co.uk/sitemap.xml


    try adding the "disallow" line in the robots file so it's

    User-agent: *
    Disallow:
    Sitemap: http://www.mycreditstatus.co.uk/sitemap.xml


    I don't know how much of a difference this will make but it all helps.
    -Nicky

    Tuesday, December 9, 2008 10:02 AM
  • Thanks Nicky, have added as suggested, although I think the issue is that MSNBot is not even trying to hit the robots.txt file, rather than mis-interpret whats in it.

     

    Tuesday, December 9, 2008 10:12 AM
  •  

    Hi Altnet,

     

    I'm assuming you are looking at your sever logs? Are there URLs that are disallowed or is it a concern that we aren't seeing your sitemap?

     

    Cheers!

     

    Brett

    Wednesday, December 10, 2008 10:05 PM
  • Hi Brett,

     

    Yes, looking at the server logs there are no hits beyond the root folder of each website listed. The sitemaps are indicated to all of the websites via webmaster central and I resubmit them approx once per week as the sites are updated. None of the sitemaps have been hit by MSNBot.

     

    None of the sites are listed in MSN although most are at least several months old now. None of the sites have disallowed urls.

     

    Thanks for looking into this for me. Let me know if there is any further information I can provide you to help resolve this.

     

     

    Thursday, December 11, 2008 12:23 AM
  • Hi,

     

    Could you please list the domains you are refering to as well as a keyword phrase for each and I will take a closer look?

     

    Thanks,

     

    Brett

     

    Thursday, December 11, 2008 9:54 PM
  • Hi,

     

    Sorry for the delay. I'm looking at this a bit closer and will get back to you as soon as I can with an update. Thank you for your patience.

    Monday, January 26, 2009 5:29 PM
  • Hi,

    interesting to see that one of my new sites is being crawled with no problems.
    Monday, February 2, 2009 7:37 PM
  • Hi Altnet,

    I have submitted your sites using an internal submission tool, so maybe that will help. Good news is, you aren't blocked on our end.



    Brett
    Program Manager, Live Search Webmaster Tools
    • Marked as answer by Brett Yount Tuesday, February 24, 2009 5:45 PM
    Tuesday, February 24, 2009 5:44 PM