locked
Live Search is taking away my indexed pages - Why? RRS feed

  • Question

  •  

    So a few days ago on the Webmasters tools,  MSNbot had indexed 27 pages.  The next day it was 30 pages.  I was thrilled since it seemed like it was biting off a chunk at a time, which was fine.   Yesterday the index was 20 pages!

     

    Now it's 9....

     

    What is causing the MSNbot to de-index me?  

     

    I have a 301 redirect to redirect all jeftek.com requests to www.jeftek.com

    I have a published and submitted fresh sitemaps in the WM tools

    I believe all the links are search engine friendly

     

    I do see some old site links to pages that no longer exist on the site (they were spammed and need to be removed from the index.), but only 1 of "good" content.  How can I get the bad pages removed from the index?

     

    Also, Could it be that MSN is seeing a page on www.jeftek.com and jeftek.com and thinking they are duplicates?   With the 301, they should all be www'd,  but the index still has them without the www.   Is there a way to tell Live Search to merge the results since they are the same domain?

     

    Thanks,

     

    Jef

     

    Tuesday, February 12, 2008 3:40 PM

Answers

  • As far as I know, Live Search still has issues with following 301 or 302 redirects in the first place, not sure about robots.txt. Regarding pages including in index, yeah they vary for us too almost everyday.

    Regards,
    Mark
    Thursday, February 14, 2008 2:38 PM

All replies

  • I think it's fairly natural for it to bounce around.  We vary by as much as 20% on any given day in live for our promotional items site.

     

     

     

    Wednesday, February 13, 2008 3:46 AM
  •  

    Promo,

     

    Thanks for the response.  I guess I shouldn't be so anxious, and need to let it settle in.

     

    It seems that it changes at least every 24 hours, so hopefully tomo will bring some good news.   I am starting to see search engine traffic reach the site as well.

     

    There are still a ton of "bad" pages in the index from this domain, so hopefully they will start to fall out since they are 404s now.

     

    Jef

    Wednesday, February 13, 2008 4:41 AM
  •  

    I woke up this morning t find out that 65 pages are now indexed!   So it is definitely on the right track.

     

    I see both MsnBot 1.0 and msnBot 1.1 both visited the site last night, so maybe that has something to do with it?

     

    The only bad thing is that I see pages in the index now that were explicitly prohibited by my robots.txt.   the Webmaster tools validated my robots.txt, so I wonder why this happened.

     

    Jef

    Wednesday, February 13, 2008 2:16 PM
  • As far as I know, Live Search still has issues with following 301 or 302 redirects in the first place, not sure about robots.txt. Regarding pages including in index, yeah they vary for us too almost everyday.

    Regards,
    Mark
    Thursday, February 14, 2008 2:38 PM
  •  

    Excellent!

     

    I think it's already been reported that msn is ignoring the robots file which they will hopefully correct in time.

     

    As far as 1.0 vs 1.1 we still get both.  Once it "settles" in you should see it hitting your site every day at a consistent level with spikes every few days.

     

    It sounds like your site is working out pretty much like our site did.  Overall we're pleased and are now getting traffic for promotional items searches through live.

    Thursday, February 14, 2008 2:39 PM