Need MSNbot to follow the robots.txt RRS feed

  • Question

  • robots.txt contains something like

    # go away ? tell all others not in the list below to stay out!
    User-agent: *
    Disallow: /

    User-agent:  MSNbot
    Disallow: /landingpage
    Disallow: /register
    Crawl-delay: 5

    User-agent:  msnbot
    Disallow: /landingpage
    Disallow: /register
    Crawl-delay: 5

    The validation tool says msnbot not allowed to crawl the site which wasnt what this was being requested.
    See lots of checks for robots.txt from msnbots and lots of msnbot requests.

    1 ) Is your validation tool really working correctly?
    2 ) If your really honoring the robots.txt why are you still hitting the site.


    Thursday, May 7, 2009 2:35 PM


  • replied offline
    Program Manager, Live Search Webmaster Tools
    • Marked as answer by Brett Yount Thursday, May 7, 2009 5:07 PM
    Thursday, May 7, 2009 5:07 PM