locked
Robots.txt Validator, again RRS feed

  • General discussion

  • Hi,

    Your validator find now errors on a robots.txt as basic as this one :
    Code Block

    User-agent: Googlebot
    User-agent: Slurp
    User-agent: msnbot
    Disallow: /cgi-bin/

    Sitemap: http://www.example.com/sitemap.xml



    Errors obtained are:
    Code Block

    Line #2: User-agent: Slurp
    Error: 'user-agent' tag should be followed by a different tag.
    **************************************************
    Line #3: User-agent: msnbot
    Error: 'user-agent' tag should be followed by a different tag.



    In fact it does not understand block for several agents and want that:
    Code Block

    User-agent: Googlebot
    Disallow: /cgi-bin/

    User-agent: Slurp
    Disallow: /cgi-bin/

    User-agent: msnbot
    Disallow: /cgi-bin/

    Sitemap: http://www.example.com/sitemap.xml



    But if we refer to the link you give as a reference :
    "A Standard for Robot Exclusion" : http://www.robotstxt.org/orig.html
    we can read :
    "The record starts with one or more User-agent lines, followed by one or more Disallow lines..."

    The first exemple is accepted by Google Analyze robots.txt tool and other validators I used.

    Hope this small tool will soon be usefull...
    Oukiva
    Friday, December 14, 2007 4:22 AM