Sign in
Microsoft.com
 
Microsoft
 
 
 

locked
Sitemap url in robots.txt RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote
    Hi,

    i have read that its possible to put the sitemap url in robots.txt file. i have also found a post that mentioned to put the link and disallow bots. something like this:
    User-agent: *
    Sitemap: http://www.yourdomain.com/sitemap.xml
    Disallow: /

    1. what is your opinion about this?
    2. is this a good approach?
    3. Does it mean that robots only crawl links inside sitemap.xml?

    best regards,
    Friday, January 2, 2009 5:55 PM

Answers

  • Question
    Sign in to vote
    0
    Sign in to vote

     

     

    Hi,

     

    The sitemap link is not a REP command per se. It is only there to help robots locate your sitemap. Normal REP rules apply, so this will block the robots from crawling any portion of your site. However, this might not be a properly formed REP command and they could ignore the disallow command altogether.

    Friday, January 2, 2009 8:04 PM