That was the problem, the site map information doesn't go into placing it on the robots.txt file. Robots.txt information doesn't go into placing site maps on them, but rather keeping the search bots off of files that you don't want indexed. However by helping others here with their sitemap questions, I did come across the answer. I do not want my contact us or the about us indexed and there is no reason to have a bot on those 2 files so I wanted them disallowed, but I would like the rest of the site indexed so this is what my robots file should look like with the site map information
# robots.txt for http://www.truckerservicesus.com/
user-agent: msnbot
Disallow: /contact
Disallow: /about
user-agent: googlebot
Disallow: /contact
Disallow: /about
user-agent: sulrp
Disallow: /contact
Disallow: /about
user-agent: *
Disallow: /contact
Disallow: /about
Sitemap: http://www.truckerservicesus.com/sitemap.xml
Running this through the validator I get no problems with robots.txt file. So I got it right.
Now, the problem I have and it appears that a lot of the problems posted here could be better helped
if you could give us examples. For example with this post on this problem I was having, it would
have been easier for me if the question was answered as
The message is telling you that you could benefit by pointing to the location of the Sitemap. You can do this by adding
Sitemap: http://www.the location of sitemap/sitemap.xml
the same as you would for a disallow:
You can read more about this at Sitemaps.org
Thank you for your help I do have it fixed now.
P.S. It would be great if your valiudator came with some instructions. Everybody seems to think that you just put in the location of your robots.txt file and
msn has a bot that goes and reads it as it is online not as it is in the box. The validator is a good tool if people just knew how to use it.
