Sign in
Microsoft.com
 
Resources for IT Professionals
 
 
 
Resources for IT Professionals > Forums Home > Webmaster Forums > Sitemap Protocol > Sitemap url in robots.txt
Ask a questionAsk a question
Search Forums:
 

AnswerSitemap url in robots.txt

  • Friday, January 02, 2009 5:55 PMNelson Monteiro Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    Hi,

    i have read that its possible to put the sitemap url in robots.txt file. i have also found a post that mentioned to put the link and disallow bots. something like this:
    User-agent: *
    Sitemap: http://www.yourdomain.com/sitemap.xml
    Disallow: /

    1. what is your opinion about this?
    2. is this a good approach?
    3. Does it mean that robots only crawl links inside sitemap.xml?

    best regards,
       

Answers

  • Friday, January 02, 2009 8:04 PMBrett Yount Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     Answer
    Sign In to Vote
    0
    Sign In to Vote

     

     

    Hi,

     

    The sitemap link is not a REP command per se. It is only there to help robots locate your sitemap. Normal REP rules apply, so this will block the robots from crawling any portion of your site. However, this might not be a properly formed REP command and they could ignore the disallow command altogether.

       

All Replies

  • Friday, January 02, 2009 8:04 PMBrett Yount Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     Answer
    Sign In to Vote
    0
    Sign In to Vote

     

     

    Hi,

     

    The sitemap link is not a REP command per se. It is only there to help robots locate your sitemap. Normal REP rules apply, so this will block the robots from crawling any portion of your site. However, this might not be a properly formed REP command and they could ignore the disallow command altogether.