Answered by:
robots.txt file

Question
-
When I validate my robots.txt file I get (Warning: 'sitemap' - tag isn't specified.) I do not want to disallow my site map. I'm not understanding what the validator is stating about my site map. My robots.txt file reads
(# robots.txt for http://www.truckerservicesus.com/
user-agent: msnbot
Disallow: /contact
user-agent: googlebot
Disallow: /contact
user-agent: sulrp
Disallow: /contact
user-agent: *
Disallow: /contact)
without the () keeping the robots out of and avoiding having the contact us page indexed.
As you can see that is all that is disallowed.
Msnbot has full access to my sitemap.xml and it has been submitted.
So I'm not understanding what the problem is, so I do not know what to fix.
I'm tring to keep the site 100% W3 compliant.
Can someone please let me know what the robot.txt needs in regards to the site map?Saturday, January 12, 2008 7:36 AM
Answers
-
I appologize, the message needs some work. There isn't a problem with Robots.txt file. The message should tell you that you could benefit by pointing to the location of the Sitemap. You can read more about this at Sitemaps.org
Essentially this will inform search crawlers of the wereabouts of your Sitemap.
Jeremiah
Monday, January 14, 2008 5:52 PM
All replies
-
I appologize, the message needs some work. There isn't a problem with Robots.txt file. The message should tell you that you could benefit by pointing to the location of the Sitemap. You can read more about this at Sitemaps.org
Essentially this will inform search crawlers of the wereabouts of your Sitemap.
Jeremiah
Monday, January 14, 2008 5:52 PM -
That was the problem, the site map information doesn't go into placing it on the robots.txt file. Robots.txt information doesn't go into placing site maps on them, but rather keeping the search bots off of files that you don't want indexed. However by helping others here with their sitemap questions, I did come across the answer. I do not want my contact us or the about us indexed and there is no reason to have a bot on those 2 files so I wanted them disallowed, but I would like the rest of the site indexed so this is what my robots file should look like with the site map information
# robots.txt for http://www.truckerservicesus.com/
user-agent: msnbot
Disallow: /contact
Disallow: /about
user-agent: googlebot
Disallow: /contact
Disallow: /about
user-agent: sulrp
Disallow: /contact
Disallow: /about
user-agent: *
Disallow: /contact
Disallow: /about
Sitemap: http://www.truckerservicesus.com/sitemap.xml
Running this through the validator I get no problems with robots.txt file. So I got it right.
Now, the problem I have and it appears that a lot of the problems posted here could be better helped
if you could give us examples. For example with this post on this problem I was having, it would
have been easier for me if the question was answered asThe message is telling you that you could benefit by pointing to the location of the Sitemap. You can do this by adding
Sitemap: http://www.the location of sitemap/sitemap.xml
the same as you would for a disallow:
You can read more about this at Sitemaps.org
Thank you for your help I do have it fixed now.
P.S. It would be great if your valiudator came with some instructions. Everybody seems to think that you just put in the location of your robots.txt file and
msn has a bot that goes and reads it as it is online not as it is in the box. The validator is a good tool if people just knew how to use it.Friday, January 18, 2008 3:06 AM