locked
eCommerce Coverage - Why is Live Search Ignoring Product Pages on Large eCommerce Sites? RRS feed

  • Question

  • I asked about this in July 2008, and have received no reply on this forum.  I sent emails to various people in the team and received nothing beyond the boilerplate "read the guidelines" (I actually received two slightly differently pasted-together emails from two different correspondents).

    I had already done everything in the guidelines which can be done on a large site.

    We run a site which has 30,000 products on it, and so has c.90,000 pages, Google indexes an average of 12,000 of these pages every day and every month lands visitors into more than 4,000 different pages based on 7,000 different keywords. 

    MSN and Live Search between them have figures about 2% of these.  Live Search lists 339 pages on the site (up from 140 in July).  Googlebot reads this number of new pages every 40 minutes.

    Running a search for the name of a product or a phrase specific to a certain product on Google.ie, both our site and most of our competitors will appear, listing the relevant product page(s) for all of us.  Try the same thing on Live Search and you get nothing.

    Irish eCommerce vendors cannot use Live Product Search.  Live Search is ignoring 99% of the product pages on our sites.  So our customers in Ireland have no way of finding product information or competitive prices using Live Search.  They go to Google.

    Most of my site's potential customers (non-technical people, office workers, admins) use Live Search or other Microsoft search tools by default - please give them the results they are looking for.
    Friday, October 31, 2008 3:17 PM

Answers

  • Hi Fergal,

     

    I've been looking at this for awhile and I've talked with my manager. There's nothing that you are doing wrong per se. You have decent backlinks, your site is good, I think it boils down to your rank, which is on the low side of good, and the size of our index. I suggest working on your ranking, maybe take a look at our FAQs we posted on the indexing forum and the ranking forum. Sorry I can't help more than that.

     

     

    Brett

     

    Tuesday, January 13, 2009 4:13 PM

All replies

  •  

    Hi,

     

    I am not sure what happened with the other post, but I apologize for not seeing it before today. I would like to help you with this issue. In order to do so, could you please:

     

    1. Provide me with domain name(s)

    2. The top 5 keywords you are targeting

    3. Are you using any redirects and if so, what type

     

    You can post them here or send them to me at lswmc@microsoft.com

     

     

    Thanks!

     

    Brett

    Friday, October 31, 2008 4:12 PM

  • Hi Brett,

    1. http://www.enterprisesales.ie

    2. We are not targeting any keywords.  The site has tens of thousands of pages, yet they are not appearing when users search for product-specific text or keywords.  Your webmaster tools reports only 330 pages indexed.

    3. No redirects.

    My problem is that we are getting less than 1% of our traffic from you, when we know that a huge number of Irish Internet users (our sole market) will be using your search engine rather than Google, so the traffic should be much higher.  Our competitors in this market have the same problem, so I am not looking for special treatment...

    Regards

    Fergal
    Tuesday, November 4, 2008 12:15 PM
  • I took a look at your home, robots, and site map. I looked at them from a search engine optimization standpoint and not from an aesthetics or visitor operability view. Your site needs help to perfom well. Here are just the basics:

    1. The title tag should ALWAYS be the first tag inside the head section.
    2. The title tag doesnt convey your niche market.
    3. The description tag should ALWAYS be the second tag in the head section.
    4. The keywords tag should ALWAYS be the third tag. You don't have one.
    5. It is known that pages that have a good relevancy % between the keywords tag and the words on the page tend to perform better than the pages that dont.
    6. You don't have a robots tag that states index,follow.
    7. Your robots.txt may be hurting you more than it is protecting you. If a competitor wants to comete against you directly, they will anayze your page. Use the robots.txt to exclude pages, not bots.
    8. Your sitemap crashed my browser twice. This is because you have way too many lines of code. The  sitemap.xml should list your important "gateway" pages and pages you think will be hard to find. There is a good possibility that the Live bot that examined your sitemap.xml penalized you for having over 17000 lines of code. And thats where my Mozilla crashed. So you are looking at close to 3000 pages at 17,000 lines. Again, just list the important pages, the catagory pages, the ones that are hard to get to. Keep your sitemap to under 250 pages or so.
    9. Anything in a javascript you should assume that the bot won't be able to figure out what it is. Since you have a lot of script, you better make sure the actual text inside the html conveys the message you are sending to the search engines.

    I could go on but my time is valuable to me. What I gave you was a freebie. I am sure you well get a reply to your post from an official moderator, but don't expect them to give out too much information. The more they divulge, the easier it is to crack their code. Definately read all official guidline from all search engines. This should always be considered BEFORE design starts. If you have any questions feel free to ask.

    Emo

    Sunday, November 16, 2008 10:05 PM
  • Well, thanks for the unsolicited assistance, I was actually addressing myself to the people running the Live Search engine.  My posting is about why Live Search was ignoring all the pages on the site when Google was indexing over 10,000 a day...  The site performs exceptionally well in Google and appallingly badly in Live Search, I want to know why.

    In case anyone else is reading this, let me just say that the above "advice" is misguided, and that's being kind.  Anyone considering clicking on the dropped self-promotional link above should perhaps read the following:

    1. The title tag should ALWAYS be the first tag inside the head section.

    No, it's just has to be inside the HEAD element.  I think most search engines will be able to find it by looking for the telltale <title> and </title> around it.

    2. The title tag doesnt convey your niche market.

    We don't have a niche market.  The title tags on all our tens of thousands of product pages contain product title followed by the category path to the product.  Couldn't be more relevant to each product's target audience.

    3. The description tag should ALWAYS be the second tag in the head section.

    Again, there's no standard which says this.

    4. The keywords tag should ALWAYS be the third tag. You don't have one.

    Keywords tags are largely ignored by Google and MSN, and probably Yahoo! as well.  We'll be putting them back in soon but encouragingly Google forgives our lapse.

    5. It is known that pages that have a good relevancy % between the keywords tag and the words on the page tend to perform better than the pages that dont.

    Mmm. See answer to 4.

    6. You don't have a robots tag that states index,follow.

    I don't need one.  Google finds all the pages on the site, a crawler doesn't need to be told to follow a link, that's its job.

    7. Your robots.txt may be hurting you more than it is protecting you. If a competitor wants to comete against you directly, they will anayze your page. Use the robots.txt to exclude pages, not bots.

    Mmm.  Are you insane?  Robots.txt is an advisory which nice crawlers can choose to comply with or ignore.  Anyone can just read any public page on the site.

    8. Your sitemap crashed my browser twice. This is because you have way too many lines of code. The  sitemap.xml should list your important "gateway" pages and pages you think will be hard to find. There is a good possibility that the Live bot that examined your sitemap.xml penalized you for having over 17000 lines of code. And thats where my Mozilla crashed. So you are looking at close to 3000 pages at 17,000 lines. Again, just list the important pages, the catagory pages, the ones that are hard to get to. Keep your sitemap to under 250 pages or so.

    Yes, well, look up the XML Sitemap standards, it complies perfectly.  Google uses it to decide which 10,000 pages to index every day.  The XML sitemap is for crawlers, not for opening in a browser.  All pages on the site are easy to find, they can all be navigated to in a few clicks.  And the XML sitemap is designed to provide a flat structure for the crawler to quickly list all the pages on the site, not the important ones.  Google seems very happy with our sitemap, Live Search seems to have a problem.

    9. Anything in a javascript you should assume that the bot won't be able to figure out what it is. Since you have a lot of script, you better make sure the actual text inside the html conveys the message you are sending to the search engines.

    Yes, we've gone to some trouble to make sure that the website is there in the HTML.  We've put the javascript stuff into the javascript part.  It's amazing, but Google seems to be able to allow people to land in to the site on 4,000 different pages using 7,000 key phrases this month, so the text must be pretty accessible.


    Friday, November 21, 2008 4:33 PM
  • Hi Fergal,

     

    I've been looking at this for awhile and I've talked with my manager. There's nothing that you are doing wrong per se. You have decent backlinks, your site is good, I think it boils down to your rank, which is on the low side of good, and the size of our index. I suggest working on your ranking, maybe take a look at our FAQs we posted on the indexing forum and the ranking forum. Sorry I can't help more than that.

     

     

    Brett

     

    Tuesday, January 13, 2009 4:13 PM