Sitemap url in robots.txt RRS feed

  • Question

  • Hi,

    i have read that its possible to put the sitemap url in robots.txt file. i have also found a post that mentioned to put the link and disallow bots. something like this:
    User-agent: *
    Sitemap: http://www.yourdomain.com/sitemap.xml
    Disallow: /

    1. what is your opinion about this?
    2. is this a good approach?
    3. Does it mean that robots only crawl links inside sitemap.xml?

    best regards,
    Friday, January 2, 2009 5:55 PM






    The sitemap link is not a REP command per se. It is only there to help robots locate your sitemap. Normal REP rules apply, so this will block the robots from crawling any portion of your site. However, this might not be a properly formed REP command and they could ignore the disallow command altogether.

    Friday, January 2, 2009 8:04 PM