locked
robots.txt error RRS feed

Answers

  • You'll probably want to change it to: 

     

    Sitemap: http://www.credit-report-repair.us/sitemap.xml

     

    or

     

    Sitemap: http://www.credit-report-repair.us/sitemap.gz

     

    That should help.

     

    Andy

    Wednesday, January 16, 2008 4:56 PM
  • The problem is a simple one. The first time I used the validator I had the same problem. It wasn't that the validator wasn't working correctly it was a matter of me using it incorrectly. What you are doing wrong is you need to put the contents of your robots.txt file in not just the title. The validator doesn't go read and validate your robots.txt file online but reads what you put in the box. Here is the corrected robots.txt file for your site. Copy and paste all of it the way it is into the validator and check, then put it up as your robots.txt file. By the way I have a posted question on the robots.txt file on the sitemap issue your txt file helped me fig it out thank you.

    # robots.txt for http://www.credit-report-repair.us/

    User-agent: *
    Disallow: /cgi-bin/
    Sitemap: http://www.credit-report-repair.us/site_map.shtml


    Friday, January 18, 2008 2:10 AM
  • Your'e right each engine does it differently, but typically each of the engines will visit you with a frequency based on the rank. As you improve in rank the likelyhood of more frequent visits goes up.  While there are other factors this is the best way to increase the crawl frequency. 

     

    Jeremiah

    Monday, January 21, 2008 5:50 AM

All replies

  • You'll probably want to change it to: 

     

    Sitemap: http://www.credit-report-repair.us/sitemap.xml

     

    or

     

    Sitemap: http://www.credit-report-repair.us/sitemap.gz

     

    That should help.

     

    Andy

    Wednesday, January 16, 2008 4:56 PM
  • as andy suggested, i added the sitemap.xml file with proper syntaxt at

    http://www.credit-report-repair.us/sitemap.xml and tried the validator again and when i clicked the validate button once i received the

     

    Line #1: http://www.credit-report-repair.us/robots.txt
    Error: 'user-agent' -  should be the first tag.

     

    then when i clicked the validate button again this message also popped up again:

     

    **************************************************

    Warning: 'sitemap' - tag isn't specified.
    **************************************************

    so again the validator results read:

     

    Line #1: http://www.credit-report-repair.us/robots.txt
    Error: 'user-agent' -  should be the first tag.
    **************************************************
    Warning: 'sitemap' - tag isn't specified.
    **************************************************

     

    Wednesday, January 16, 2008 5:17 PM
  • The problem is a simple one. The first time I used the validator I had the same problem. It wasn't that the validator wasn't working correctly it was a matter of me using it incorrectly. What you are doing wrong is you need to put the contents of your robots.txt file in not just the title. The validator doesn't go read and validate your robots.txt file online but reads what you put in the box. Here is the corrected robots.txt file for your site. Copy and paste all of it the way it is into the validator and check, then put it up as your robots.txt file. By the way I have a posted question on the robots.txt file on the sitemap issue your txt file helped me fig it out thank you.

    # robots.txt for http://www.credit-report-repair.us/

    User-agent: *
    Disallow: /cgi-bin/
    Sitemap: http://www.credit-report-repair.us/site_map.shtml


    Friday, January 18, 2008 2:10 AM
  • excellent post lrmcspeedy , your code for the validator returned "no problems found" in the validator, now what do you do once the robots.txt file is validated? how do you get the msnbot to come over and look at the robots.txt file on your site to reevaluate and index the proper upto date pages and text?

    Friday, January 18, 2008 3:08 AM
  • Good question, let me know when you get the answer. The 3 major search engines do not work the same. With Google you will see results in 24hrs. Yahoo will update theirs within 24hrs but it could take a month to help move you up. Msn Live I don't have any luck with at all. Here is a site of mine http://www.truckerservicesus.com/ that is 1 month old and I get ranked in Google in the top 10 over content in many areas, but only Google. Yahoo still has a person doing the evaluation that is why it takes awhile with them. Msn Live I don't know. The site I gave as a reference gets about 20 hits a day from Google 3 hits this week from Yahoo and 0 from Msn Live. It has a SEO ranking of 87% but only gets 1 bar in Msn tools, the site is 100% W3 complient.
    Sunday, January 20, 2008 4:06 PM
  • Your'e right each engine does it differently, but typically each of the engines will visit you with a frequency based on the rank. As you improve in rank the likelyhood of more frequent visits goes up.  While there are other factors this is the best way to increase the crawl frequency. 

     

    Jeremiah

    Monday, January 21, 2008 5:50 AM
  • Also note that in terms of a properly formatted robots.txt file, there should be a blank line above the Sitemap: declaration.

    Although both Google & MSN (don't know about Yahoo) treat it correctly without the space I would recommend adding it because it's better to be correct on these things that are machine readable, just in case they do a strict interpretation.
    Wednesday, January 23, 2008 4:40 PM