locked
Webmaster Tools Under reporting RRS feed

  • Question

  •  

    For the longest while the number of pages indexed was almost as many as Google and never as many as Yahoo. Now the number of pages reportedly indexed is 1/10 of what it was before?

     

    The pages all seem to be indexed as they should or that I can't find anything that should be listed that isn't. Or perhaps only a few areas seem questionable. Traffic is down in terms of daily visits though and in the web reports we are seeing more Yahoo results now than MSN/Live than ever before.

    Friday, April 18, 2008 12:33 PM

Answers

  • Hi,

     

    I understand what you are saying, however, as I mentioned there are many reasons why your pages might have fallen out of the index including age/relevance. Updating your sitemap update date could encourage the crawlers to recrawl your whole site. You may be right about the beta issue, but I honestly don't know. I am only trying to provide ideas that might help the crawlers reindex your pages. As far as sitemap compatability goes, Yahoo, Google, and Live have joined together to create a joint sitemap protocol. The announcement can be found here: Live Search supports the joint sitemap protocol.

     

     

    Brett

    Thursday, April 24, 2008 3:01 PM

All replies

  • Hi,

     

    Have you updated your sitemap lately? There are several reasons why pages might fall out of the index including age/relevance. Not to say that is the issue, but it might help.

     

     

    Thanks,

     

    Brett

    Tuesday, April 22, 2008 4:24 PM
  • Hi Brett,

     

    We use AspDotnetStoreFront as our ecommerce platform and it has features that generate sitemaps. Yahoo and Google are unique sitemaps although http://www.fabricstodyefor.com/sitemap5.xml is used with Google as well as your bot.

     

    If listings were in terms of relevance that would be an interesting question because that would infer in our view, logic.  That you might be shopping for red fabric paint or that you want brand name fabric paint.

     

    We catalog product by distributor, brand and also group products by type. To a bot it is likely duplicate content but so are cars on a road...

     

    For the most part I'm more convinced it is an internal problem caused by the beta test of the "live product catalog" MSN is testing. We submitted a partial list of products and it was not at all satisfactory for us to pursue and ever since the catalogs we submitted expired the pages we had indexed are no longer indexed.

     

    Today we are busy upgrading our site and so we'll be offline a while longer. But regardless of today's downtime, the number of pages indexed at Yahoo and Google are multiple times greater than what is at MSN Live. It wasn't that long ago that MSN Live had indexed our site perfectly. 

     

    Our Google Analytics report shows that Yahoo has passed the results given for MSN Live for the first time. And so, MSN Live is now a distinct 3rd place result for driving unique web traffic to our site. In turn our sales could be slightly better if all of our product pages were indexed. Now though there are fewer pages indexed than the number of product pages we have, let alone the total number of pages in our site.

     

    At this time Yahoo is the model that best indexes our site and it has done some things that are quite intelligent for a bot to have resolved. One of the most relevant features is that the Yahoo bot has been able to ascertain the Distributors product groups as well as the brands of products each distributor carries. We divide our products by Distributor, Brand, and product type.

     

    MSN/Live has been able to subjugate the Brand name catalogs from the product types but now fails to show the products.

     

     

     

    Tuesday, April 22, 2008 7:28 PM
  • Hi,

     

    I understand what you are saying, however, as I mentioned there are many reasons why your pages might have fallen out of the index including age/relevance. Updating your sitemap update date could encourage the crawlers to recrawl your whole site. You may be right about the beta issue, but I honestly don't know. I am only trying to provide ideas that might help the crawlers reindex your pages. As far as sitemap compatability goes, Yahoo, Google, and Live have joined together to create a joint sitemap protocol. The announcement can be found here: Live Search supports the joint sitemap protocol.

     

     

    Brett

    Thursday, April 24, 2008 3:01 PM
  • Hi Again,

     

    The sitemap information is helpful. Our ecommerce system works in real time to post new pages or changes to special files for Yahoo and Google. The MSN bot is not able to read the same file so I have to use a program that crawls (GSiteCrawler) the site and that is the xml file in the webmaster account at MSN that is being read from.

     

    /yahooindex.aspx

     

    /googleindex.aspx

     

    /sitemap5.xml

     

    Since my last post I've updated meta data or cleaned up the site before generating a new sitemap.xml file, for the MSN bot. Only now I've again lost pages from the index. Overall I'm fairly satisified with the pages or the results that are indexed regardless of how many web pages there are. I suppose losing pages is a good thing if the bot wants only fresh content?

     

    What I've done though, is to change the order of how the database driven pages display the products in hopes that will instigate the bot to reconsider the pages it may have determined to be obsolete?

     

    I wrote to the Live Product team and  they ask that I send them one of the XML files that was used for the live beta product catalog. Since those files have expired the products that were indexed in the Live Search directory are no longer indexed as they were before uploading the beta live product catalog files.

     

     

    Saturday, April 26, 2008 6:27 PM