locked
What do they mean "Warning: 'sitemap' - tag isn't specified." RRS feed

  • Question

  •  

    I would also like to know the answer to this question as exactly the same is happening to my robots.txt file when I try to get it verified?

     

    I am using my robots file as followed:

     

    User-agent: *
    Disallow:

     

    Simple this should find all the files to my web site?

     

    What do they mean "Warning: 'sitemap' - tag isn't specified."

    Wednesday, April 9, 2008 5:59 PM

Answers

  • Hi,

     

    Are you using a sitemap?

     

    a robot.txt sitemap tag is something like:          Sitemap: http://www.yoursite.com/sitemap.xml

     

     

    Brett

    Friday, April 11, 2008 5:53 PM
  • Hi Gary. The only folders you want to Disallow are those your DON'T want the search engines to crawl (to stay out of).

     

    Say for example you have a folder that contains sensitive user information, or a database configuration. These files you don't want the search engines to crawl or index.  Lets call this folder/file: www.MyDomain.com/privatefolder/file.private

     

    To keep the search engine crawlers out of that folder and files (/privatefolder/), you place the address of it in your Disallow tag in the robots.txt file.  Using the above example, your robots.txt file will look like this:

     

    Code Snippet

     

    User-agent: *

    Disallow: /privatefolder/


     

    Sitemap: http://www.MyDomain.com/sitemap.xml

     

     

    Always upload your robots.txt file in the root URL of your domain.

     

     

     

     

    Monday, April 14, 2008 5:27 PM

All replies

  • Hi,

     

    Are you using a sitemap?

     

    a robot.txt sitemap tag is something like:          Sitemap: http://www.yoursite.com/sitemap.xml

     

     

    Brett

    Friday, April 11, 2008 5:53 PM
  • Brett

     

    Yes I am using a site map and its placed in the root ftp folder as sitemap.xml where the robots.txt file is also placed.

     

    What is not clear is do I need to place some text in to my robots.txt file.

     

    An example of whats needed would be realy helpful.

     

    Thanks

    Gary

    Saturday, April 12, 2008 11:12 AM
  • Gary, you do need to place a "sitemap tag" in your robots.txt file.  This is correct.  Your robots.txt file should look something like this:

     

     

    Code Snippet

    User-agent: *
    Disallow: /software/
    Disallow: /js/

    Sitemap: http://www.mydomain.com/sitemap.xml

     

     

    The "Sitemap Tag" helps the crawler find your sitemap.  It is needed in the robots.txt file because this is the first file the crawler bot looks at PRIOR to crawling or indexing your site.

     

    The crawler first looks at the robots.txt file as a "map" sort to speak. The robot.txt file tells the crawler what folders to Disallow, and where exactly your Sitemap is.  If you want your entire site crawled, then your Sitemap will look like this:

     

    Code Snippet

     

    User-agent: *
    Disallow: /

    Sitemap: http://www.mydomain.com/sitemap.xml

     

     

     

    In the above example, change "mydomain.com" to your site's domain name.

    I hope this has been helpful.

     

    Sunday, April 13, 2008 4:53 PM
  • Thanks some much for you help and I am beginning to understand more about the robots file.

     

    In your example below how do I determine which files to disallow?

     

    Disallow: /software/
    Disallow: /js/

    I have around 160 html pages with a few js pages to run some flash animations inside the html page and css files for the text.

     

    What are the advantages or disadvantages of excluding certain file like the js and css files?

     

    Would you know of any easy to follow links as after searching for information on the robots file I have been unable to find anything that explains these things in a simple and easy to understand format.

     

    Many thanks

    Regards

    Gary

     

     

     

    Monday, April 14, 2008 4:28 PM
  • Hi Gary. The only folders you want to Disallow are those your DON'T want the search engines to crawl (to stay out of).

     

    Say for example you have a folder that contains sensitive user information, or a database configuration. These files you don't want the search engines to crawl or index.  Lets call this folder/file: www.MyDomain.com/privatefolder/file.private

     

    To keep the search engine crawlers out of that folder and files (/privatefolder/), you place the address of it in your Disallow tag in the robots.txt file.  Using the above example, your robots.txt file will look like this:

     

    Code Snippet

     

    User-agent: *

    Disallow: /privatefolder/


     

    Sitemap: http://www.MyDomain.com/sitemap.xml

     

     

    Always upload your robots.txt file in the root URL of your domain.

     

     

     

     

    Monday, April 14, 2008 5:27 PM