locked
Why is MSNBOT ignoring our robots.txt RRS feed

  • Question

  • Hi,

    Our robots.txt file is currently:

    User-agent: msnbot
    Disallow: /Web2/JourneyPlanning/JourneyPlannerInput.aspx

    User-agent: *
    Disallow: /Web2/JourneyPlanning/JPLanding.aspx
    Disallow: /Web2/JourneyPlanning/jplandingpage.aspx
    Disallow: /Web2/JourneyPlanning/JPLandingPage.aspx
    Disallow: /web2/journeyplanning/jplandingpage.aspx
    Disallow: /NaptanViewer/
    Disallow: /TDPWebservices/
    Disallow: /Transportdirect/
    Disallow: /Web/

    Sitemap: http://www.transportdirect.info/sitemap.xml

     

    Why are we still being visited by MSNBOT on the URLS that are clearly disallowed?

     

    2008-07-30 23:38:41 65.55.105.15 - 10.93.108.200 80 GET /web2/journeyplanning/jplandingpage.aspx id=traintaxi.co.uk&do=n&oo=n&o=9100REDHILL&on=Redhill%20Rail%20Station&d=&dn=&CurrentLanguage=English&repeatingloop=Y 302 0 435 1157 www.transportdirect.info msnbot/1.1+(+http://search.msn.com/msnbot.htm) - -
    2008-07-30 23:50:39 65.55.105.15 - 10.93.108.200 80 GET /web2/journeyplanning/jplandingpage.aspx id=traintaxi.co.uk&do=n&oo=n&o=9100REDHILL&on=Redhill%20Rail%20Station&d=&dn=&CurrentLanguage=English&repeatingloop=Y 302 0 435 1109 www.transportdirect.info msnbot/1.1+(+http://search.msn.com/msnbot.htm) - -

    Any advice would greatly appreciated, as last time we checked we were being hit up to 41000 times a night on a disallowed URL.

     

    Many Thanks

    Steve

     

     

    Tuesday, August 5, 2008 12:14 PM

Answers

  • Hi,

     

    If you call out a bot, it will ignore the * section, so you have to specify all paths you want disallowed.

     

    add:

     

    Disallow: /Web2/JourneyPlanning/JPLanding.aspx
    Disallow: /Web2/JourneyPlanning/jplandingpage.aspx
    Disallow: /Web2/JourneyPlanning/JPLandingPage.aspx
    Disallow: /web2/journeyplanning/jplandingpage.aspx
    Disallow: /NaptanViewer/
    Disallow: /TDPWebservices/
    Disallow: /Transportdirect/
    Disallow: /Web/

     

    to the msnbot section

     

    Monday, August 25, 2008 8:13 PM