Crawling FAQs RRS feed

  • General discussion

  • Before you post... review these frequently asked questions:

    Q:  Why isn’t Live Search crawling or is only partially crawling my site?

    A: Depth of crawl is based on a lot of factors, including how your site ranks and the architecture of your site.  However, here are some steps you can take to help us crawl your site more effectively:

    • Verify that you are not blocked and there are no site crawling errors in Webmaster Center tools
    • Make sure you are not blocking access to the Live Search web crawler, MSNbot, in your Robots.txt file
    • Make sure you are not blocking MSNbot’s IP range
    • Ensure you have no site navigation issues that might prevent MSNbot from crawling your site:
      • Invalid page coding (HTML, XHTML, etc.)
    Note: You can validate your page source code at
      • Site navigation links that are only accessible by JavaScript
      • Password protected pages
    • Submit a sitemap in the Sitemaps tool of Webmaster Center tools and provide the path to the sitemap.xml file(s) in a Robots.txt file  

    Q:  Live Search is crawling my site, so why is it not indexing my pages?

    A: Make sure your content is viewable by MSNbot.

    • Ensure there are text navigation links if main navigation is pictorial-based
    • Ensure there is plenty of indexable content outside of <script> </script>tags
    • Make sure each page has unique <title> and <meta> description tags
    • If possible, avoid frames and meta tag refresh
    • If your site uses session IDs or cookies, make sure they are not needed for crawling
    • Use friendly URLs  whenever possible
    • Review our Webmaster Center blog post, I'm not ranking in Live Search, what can I do?


    There are many design and technical issues that can prevent MSNBot from indexing your site. If you've followed our guidelines and still don't see your site in Live Search, you may want to contact a search engine optimization company. These companies typically provide tools that will help you improve your site so that we can index it. To find site optimization companies on the Web, try searching on search engine optimization company.


    Q: How do I increase the Live Search crawl rate for my site?

    A: Currently, there is no way to manually increase the re-crawl rate for MSNbot. However, submitting an updated sitemap in our Webmaster Center tools may help. First, make sure you are not blocking MSNbot in your Robots.txt file or MSNbot’s IP range. Next, set the sitemap <changefreq> dates to “daily”. This could help initiate a recrawl. You can always change these back to more appropriate dates once the recrawl has taken place.



    As with general sitemap guidelines, submitting a new or updated sitemap is not a guarantee that Live Search will use the sitemap or more fully index your site. This is only a suggested solution to the issue of increasing the crawl rate.

    Q: How do I decrease the Live Search crawl rate?

    A: As stated in our blog post: Robots Exclusion Protocol: Joining Together to Provide Better Documentation,  you can set, “Crawl-Delay: X” where x is the number of seconds to wait between each request. Note that setting the crawl delay reduces the crawler load on your servers, but it also increases the amount of time it will take MSNBot to reindex your website (which is proportional to the length of the delay). As a result, this change might make it more difficult for your customers to find your site on Live Search.

    Another way to help reduce load on your servers is by implementing HTTP compression and Conditional Get. Read more about this on our blog post Announcing Crawler Improvements for Live Search.


    Q: Why are my indexed pages decreasing?

    A: This is possibly due to a combination of factors.

    • Ensure you have unique content on each page
    • Submit a sitemap with <changefeq> dates set to “daily
    • Ensure the page is crawlable


    Q: How do I submit more than one sitemap to Webmaster Center tools?

    A: Currently, the Sitemap tool only uses the last sitemap URL submitted. We strongly recommend using a sitemap index file if you have more than one sitemap.

    Sunday, November 30, 2008 5:00 AM

All replies

  • Ok,

    One question regarding this. We have built new tourist portal and replaced old one.

    The thing is, Yahoo has indexed cca 75000 pages on our portal, Google around 100000 and MSN around 750 only.

    OK, this time I made sitemapindex with 11 sitemaps included (11 languages on site). Direct links in images excluded from sitemap, so sitemap has around 26000 links.

    I uploaded that sitemap 10 days ago (approx) and google started to crawl right away, I even changed crawl rate there. Yahoo is a bit slower with that, and MSN is still telling situation around 750 indexed links.

    It is not possible that 3 major search engines show such different results...

    Site is:

    Can you incresase crawl rate on our site? Take a peak there, you will se that content and site is perfect.

    Thank you in advance,

    Thursday, January 29, 2009 11:14 PM
  • It just seems that the Google crawler is a lot more aggressive. 

    Our site has over 200,000 pages in the sitemap.  Google started crawling immediately after the sitemap was submitted and within weeks over 50,000 pages have been crawled and indexed.  Live on the other hand, didn't start to crawl till months later and so far only 1000 pages have been indexed over a period of four weeks.  We also noticed that Live tends to go directly for the natural navigation existing on the site and igores the sitemap to certain extend.

    Just to share our experiences.
    Thursday, February 12, 2009 7:22 AM
  • I am so tired of Live search. Why should I or anyone ever use Live search if they can't crawl or index sites. It seems that 98% have the same problem with Live search/MSN. They are trying to make it sounds like, the webbmasters are doing a bad job, but thats not true. I have done everything that are mentioned in the FAQ, but nothing helps. Sites are not crawled anyway, only 1 or 2 pages per site.
    • It is obviously Microsoft live search that are lousy at what they are doing. 
    If one at least could have some warnings at the "crawl issues", but no nothing. NADA!. Live search does simple not work. Why should anyone ever use Live search services?

    Are Microsoft committing a slow suicide?
    - Huge marketshare with Windows OS
    - 70% of all people under 40 in Europe and US are using MSN messenger
    - 60% are having hotmail an account (quickly loosing to gmail though).

    - A lousy search engine - (no wonder when crawler doesn't work)
    - A non crawling search engine
    - No internet marketing tools for the public (that are known)
    - No internet ads (that are known)
    - Really bad webmaster tool (crawling issues list empty, despite no crawling = status quo)
    - Confusing name of search tool - Is it MSN search or Live Search ? Or both? Confusing.

    - No blogtools or blogservices that are known.

    If MS have such large marketshare with the OS and internet/chatting tools, and all Youth in their hands. How can they NOT have a working search tool, known Online marketing, known blogtools, working webtools?

    Current indexstaus on one of my sites, about Vietnam
    Google - 300+ pages
    Yahoo - 200+ pages
    Live search 1 page

    I recently added a couple of thousand pages of content and are expecting Google and Yahoo to shortly index them, I have no expectations at all from Live search, no more.

    Is it a totally different strategy compared to Google and Yahoo, or just lack of marketing and engineering skills.
    Why does everyone experience problem with Live search?

    Thursday, February 19, 2009 2:23 PM