Does pattern matching (wildcards) work in robots.txt? RRS feed

  • Question

  • Does MSN accept wildcards in the robots.txt like Google and Yahoo?

    For example, if I have 2 URLs ---


    If I wanted to block the first URL from being crawled, I could use this disallow in robots.txt for yahoo and google...

    Disallow: /*cr

    This would keep my first URL from being crawled and makes my life much easier. 

    Does MSN support this too?
    Thursday, February 7, 2008 4:13 PM