locked
robots.txt file RRS feed

  • Question

  • When I validate my robots.txt file I get (Warning: 'sitemap' - tag isn't specified.) I do not want to disallow my site map. I'm not understanding what the validator is stating about my site map. My robots.txt file reads
    (# robots.txt for http://www.truckerservicesus.com/

    user-agent: msnbot
    Disallow: /contact

    user-agent: googlebot
    Disallow: /contact

    user-agent: sulrp
    Disallow: /contact

    user-agent: *
    Disallow: /contact)

    without the () keeping the robots out of and avoiding having the contact us page indexed.
    As you can see that is all that is disallowed.
    Msnbot has full access to my sitemap.xml and it has been submitted.
    So I'm not understanding what the problem is, so I do not know what to fix.
    I'm tring to keep the site 100% W3 compliant.
    Can someone please let me know what the robot.txt needs in regards to the site map?
    Saturday, January 12, 2008 7:36 AM

Answers

  • I appologize, the message needs some work. There isn't a problem with Robots.txt file.  The message should tell you that you could benefit by pointing to the location of the Sitemap. You can read more about this at Sitemaps.org 

     

    Essentially this will inform search crawlers of the wereabouts of your Sitemap.

     

    Jeremiah

    Monday, January 14, 2008 5:52 PM

All replies

  • I appologize, the message needs some work. There isn't a problem with Robots.txt file.  The message should tell you that you could benefit by pointing to the location of the Sitemap. You can read more about this at Sitemaps.org 

     

    Essentially this will inform search crawlers of the wereabouts of your Sitemap.

     

    Jeremiah

    Monday, January 14, 2008 5:52 PM
  • That was the problem, the site map information doesn't go into placing it on the robots.txt file. Robots.txt information doesn't go into placing site maps on them, but rather keeping the search bots off of files that you don't want indexed. However by helping others here with their sitemap questions, I did come across the answer. I do not want my contact us or the about us indexed and there is no reason to have a bot on those 2 files so I wanted them disallowed, but I would like the rest of the site indexed so this is what my robots file should look like with the site map information

    # robots.txt for http://www.truckerservicesus.com/

    user-agent: msnbot
    Disallow: /contact
    Disallow: /about
    user-agent: googlebot
    Disallow: /contact
    Disallow: /about
    user-agent: sulrp
    Disallow: /contact
    Disallow: /about
    user-agent: *
    Disallow: /contact
    Disallow: /about
    Sitemap: http://www.truckerservicesus.com/sitemap.xml

    Running this through the validator I get no problems with robots.txt file. So I got it right.
    Now, the problem I have and it appears that a lot of the problems posted here could be better helped
    if you could give us examples. For example with this post on this problem I was having, it would
    have been easier for me if the question was answered as

    The message is telling you that you could benefit by pointing to the location of the Sitemap. You can do this by adding

    Sitemap: http://www.the location of sitemap/sitemap.xml
    
    the same as you would for a disallow:
    You can read more about this at Sitemaps.org

    Thank you for your help I do have it fixed now.
    P.S. It would be great if your valiudator came with some instructions. Everybody seems to think that you just put in the location of your robots.txt file and
    msn has a bot that goes and reads it as it is online not as it is in the box. The validator is a good tool if people just knew how to use it.

    Friday, January 18, 2008 3:06 AM