locked
What is up with MSN/Live Bots and indexing? RRS feed

  • Question

  • Back in June 2008, my site Shop The Lingerie Store changed shopping carts, well it was more of an upgrade. During the process 301 was used to redirect the old urls to the new .aspx pages. This is the problem. MSN/Live bots are not following the robots.txt file. Half of the pages that MSN/Live has indexed is pages that have a "do not index" in the robots.txt and products are not being indexed.

     

    In the past when I was using the old shopping cart, MSN sent me the highest percentage of taffic from any of the search engines. The order was MSN (including Live), Yahoo, and then Google. Now, according to GA depending on the month, Yahoo and Google trade places for the top spot. For December both Google and Yahoo had 3000+ visits and MSN/Live came in with a whopping 101 (one hundred and one) visits.

     

    I have resubmited an xml sitemap (through the webmaster tools) but this hasnt helped. Also, my site has over 600 products which should see if you use the site:, yet MSN is returning just 308 pages, which doesnt include very many of the actual products and includes results that shouldnt be index per robots.txt file.

     

    Can someone at MSN tell me what is going on with the spiders/indexing ?

    Thursday, January 1, 2009 11:57 PM

Answers

All replies

  • Anyone at MSN have any ideas to the above post?

     

    Since G00gle and Yah00 both have an simular number of pages indexed, traffic for both is very close I dont thing there is anything wrong with my site. So...why is it that I can only get 100 visitors a month from MSN?

     

    Why does the MSNBot not respect the "do not follow" in the robots.txt file? Half the pages of my site that is INDEXED on MSN are pages that should not be indexed per the robots.txt file.

     

    Again this is very concerning since MSN used to be my top referrer of search engine traffic.

     

    Regards,

     

    Tuesday, January 6, 2009 2:31 AM
  • Taking offline.

     

    Thursday, January 8, 2009 7:57 PM