locked
Sudden drop in domain/page rank, then 94% of pages delisted! RRS feed

  • Question

  • Good afternoon everyone.

    I sent a couple emails to the Webmaster Tools Support address about these topics, but I see that there's forums now to post issues, so it makes more sense to discuss my issues here.

    First of all, on Thursday, November 15, I logged in to Webmaster Tools and saw that my domain rank had been dropped from five bars to one. Four of my top five pages dropped from 5 bars to 1 bar of rank, and one page stayed at 4 bars. My page had been crawled the day before, though it only updated (as far as I can tell) one page. The crawler seems to be ignoring my <lastmod> entries in my sitemap.xml file, but that's neither here nor there for right now.

    So, that's pretty difficult for me to swallow, seeing as a competitor (whose name rhymes with foogle) has my site highly ranked for a number of search terms that I don't even begin to show up for on Live Search. Totally irrelevant content shows up before my pages do. Again, something that's neither here nor there - for now.

    But, I kept with it.

    Until this morning.

    I logged back in to Webmaster Tools to see if my site had been crawled again, as I updated another couple pages and noted as such in my sitemap.xml, which I resubmitted.

    Instead, I found that 94% (47 of 50) of my pages had been completely removed from Live Search. Thanks!

    Not only that, the site hadn't been crawled in the mean time. And two of the three pages remaining are not special in really any regard (the one that I do consider "special" is my home page).

    What's the deal here?

    I've got Live Search embedded on my site. This is a company site, and visitors use the Live Search embedded search to help them find things on my site. Unfortunately, they can't do that now because Live Search decided to delist 94% of my site for no reason - and didn't even bother to crawl it before, apparently, deciding to drop 47 of my pages.

    Any thoughts before I remove Live Search (permanently?) from my company's site?
    Tuesday, November 20, 2007 8:01 PM

Answers

  • I have been looking into this.  The reality is there are a lot of reasons that this happens to a site.  I can't triage each individual site who is having this issue, so I am looking into the root causes.   Remember, that SEO/SEM practices can improve your index/rank, however, as I have stated before MSNBot crawls billions of web pages, not every page is indexed. For a site to be indexed, it must meet specific standards for content, design, and technical implementation.

     

    I know you are looking for a more direct solution to your problem and I am working on that.  I will post something as soon as I can speak for all sites who are experiencing this issue.   I appreciate your patience while I dig into this.

     

    Jeremiah 

    Tuesday, December 4, 2007 11:29 PM

All replies

  •  tss71 wrote:
    I found that 94% (47 of 50) of my pages had been completely removed from Live Search.

     

    My experience is that if you try again in 2 minutes, the total will change again.

     

    My Indexed Pages count bounces constantly between 150 and 88 for my site with >3000 unique url's in my sitemap. That doesn't count the times that it says "No data is available".

    Wednesday, November 21, 2007 12:56 AM
  • This is GOPStrategies.com. I changed my display name.

    I have not had a similar experience. I have tried again repeatedly throughout the day, and I have had consistent results. What you have suggested is actually the first thing I tried, as I thought it might have been an error. I also resubmitted by sitemap.xml file.

    I have had a couple times where it's listed "No data available," but see - even when I've had that problem, my pages were still available within the search service. The 47 pages that are gone in Webmaster Tools are gone on all of Live Search, too.

    When you have that fluctuation, do you lose those pages on Live Search in the mean time? Or are they still in the search service, but not available/listed in Webmaster Tools?

    When mine disappear, they disappear everywhere.
    Wednesday, November 21, 2007 1:18 AM
  • I would agree that this is pretty serious.  The site relies on LS for site searching.  When a search service only returns three pages total for a site, I can understand completely why a site owner would get upset. 

    "Wait two minutes" is not acceptable from a webmaster standpoint.  Imagine searching Amazon.com for "book" and being told there are only three books for sale.  Then searching again and being told there are 3000 then searching again and being told there are 1200.  This would not instill confidence in the visitor.

    I'm guessing the hard drive that contains the index information for 94% of the site suffered a failure recently and needs to be restored.

    Furthermore -- and this this forum is related to Webmaster Tools -- I would expect webmaster tools to acknowledge that urls have been delisted and provide reasons why pages fell off the index.  For instance "reported as spam" "link farm detected" "hidden links on page" "hardware failure."  If you feel folks would then be able to tweak the system, perhaps a generic "webmaster guidelines violation" message would help.
    Wednesday, November 21, 2007 2:16 PM
  •  carlcamera wrote:
    I would agree that this is pretty serious.  The site relies on LS for site searching.  When a search service only returns three pages total for a site, I can understand completely why a site owner would get upset. 

    "Wait two minutes" is not acceptable from a webmaster standpoint.  Imagine searching Amazon.com for "book" and being told there are only three books for sale.  Then searching again and being told there are 3000 then searching again and being told there are 1200.  This would not instill confidence in the visitor.

    I'm guessing the hard drive that contains the index information for 94% of the site suffered a failure recently and needs to be restored.

    Furthermore -- and this this forum is related to Webmaster Tools -- I would expect webmaster tools to acknowledge that urls have been delisted and provide reasons why pages fell off the index.  For instance "reported as spam" "link farm detected" "hidden links on page" "hardware failure."  If you feel folks would then be able to tweak the system, perhaps a generic "webmaster guidelines violation" message would help.

    Thank you.

    I've removed the Live Search embedded search box from my website, reluctantly. We're now on day two of having only three of fifty pages indexed, and this after I resubmitted my sitemap.xml file twice - well, actually, one resubmission and one use of the ping service. Nothing.

    Other search engines that I don't micromanage like this crawl my site every day. The phx.gbl address that I think is associated with Live Search hits my site every now and again, but none of my problems are fixed and none of my pages are actually crawled. It seems to just be downloading my sitemap.xml file and then doing nothing with it. And I even use some <lastmod> entries to alert the crawler that there have been updates made to some pages! Still nothing.
    Wednesday, November 21, 2007 3:47 PM
  • In case anyone cares, the problem is still going on. The site has not been crawled, nothing has changed. Still 3/50 pages.
    Friday, November 23, 2007 2:13 PM
  • As of this morning I am at "No data available," so we'll see if everything might be fixed whenever that's over.
    Saturday, November 24, 2007 4:30 PM
  • Nevermind. Nothing changed. Still 3/50 pages, still not crawled since 11/14.

    Any of the Live Search people want to reply in this thread?
    Saturday, November 24, 2007 7:00 PM
  •  

    I've been at no data available for 18 hours with identical symptoms.  I am going to guess that a 2nd sitemap submission is the trigger.

     

     

    Sunday, November 25, 2007 5:40 AM
  •  PromoManagers.com wrote:

     

    I've been at no data available for 18 hours with identical symptoms.  I am going to guess that a 2nd sitemap submission is the trigger.

     

     



    I've got data again, but it's still the same 3/50 pages without having been crawled since 11/14. I have only had the one sitemap, though I have resubmitted it several times. I'm not sure if that triggered all of this, but I don't think so.

    What did you do exactly that you think triggered your problem?
    Sunday, November 25, 2007 4:10 PM
  •  

    I'm guessing submitting the same sitemap multiple times is causing problems.  I did it once through the submit button, once through the ping, everything was gone next time I logged on.

     

    I can't even find my site when searching for it with the .com!  18 months we've been online, every page indexed by google and msn doesn't even have the home page?

     

     

    Sunday, November 25, 2007 4:20 PM
  •  PromoManagers.com wrote:

     

    I'm guessing submitting the same sitemap multiple times is causing problems.  I did it once through the submit button, once through the ping, everything was gone next time I logged on.

     

    I can't even find my site when searching for it with the .com!  18 months we've been online, every page indexed by google and msn doesn't even have the home page?

     

     



    Okay, then I think we did the same thing. As you can see by my posts above, I've had periodic "No data available" issues but my main problem is that 47 of my 50 pages were delisted without notice or any sort of explanation.

    That being said, I think we have the same problem and hopefully someone from Live Search will reply in this thread soon to at least let us know that it's being looked into. I find it hard to believe that malformed <meta> tags get a quick reply but this thread doesn't get anything, with my company having used/wanting to use Live Search for embedded search (which serves Live Search ads, I should add) and your company wanting to make a serious advertising commitment.
    Sunday, November 25, 2007 4:57 PM
  •  

    I agree, our site is paying this second for Microsoft Adcenter ads for a site that's not even listed in the search terms.  I've emailed and will respond as soon as I hear something.

     

     

     

    Sunday, November 25, 2007 7:01 PM
  • Thanks. Best of luck to you! I hope this is resolved soon for everyone's sake.

    Keep me up to date with how it goes, if you don't mind.
    Sunday, November 25, 2007 7:49 PM
  • Well, I deleted my old sitemap and then re-added it, also changing my authentication method from a <meta> tag to an XML file. We shall see if my site actually gets crawled again and, if so, picks up all my pages. Will post with more information as soon as I get it.

    If this doesn't change anything I may delete the site from Webmaster Tools, then start over from scratch.
    Monday, November 26, 2007 4:44 PM
  • MSNbot hit my sitemap today, though there have been no updates yet in Webmaster Tools. We shall see.
    Monday, November 26, 2007 9:54 PM
  • I haven't heard back yet but have faith that I will as the contact was sincere in wanting to help.  I would not get overly excited about the MSN Bot, it's come to our site 5-7 times a day for the last 4 days and done nothing.  It appears to grab the sitemap and then bails without going further.  Yahoo Slurp is off and running index more and more each day and Google is intensely scanning the site.  

     

    The only thing changing in my web panel is the number of inbound links.  Other than that our site www.promomanagers.com still seems to be lost in the void as far as live search goes.   There are hundreds of clicks a day from the other engines yet Live is still zero.

     

    I wish there was an official answer to this.  The site was there when I first registered, somewhere between validating the site with an XML file (MSN pings this file multiple times a day) and submitting a sitemap I lost all information in the web panel - even the time and date of last crawl.  The only thing working is inbound links.

     

    Thanks

    Tuesday, November 27, 2007 5:51 AM
  •  

    Actually, the MSNbot crawl did help some. I am up to 42/50 pages now. My domain rank is still terrible, and almost all of my pages have only one bar, and the 3 pages that were previously in the system have not been crawled since 11/9 or 11/14, for reasons I do not know.

     

    So, this is an improvement. But it's still not where it needs to be, I don't think. I'm incredibly shocked that no one from Live Search has replied at all in this thread.

    Tuesday, November 27, 2007 1:40 PM
  • Nothing new for us but it does now show over 10 high ranking links in.  Yet we still have nothing.

     

    Nobody from Microsoft has gotten back to us and we've suspended our ads.  I may actually take the time to send a certified letter to someone at Microsoft.  I'd just like an explanation.

     

    Tuesday, November 27, 2007 3:14 PM
  • There is some good news as the hits yesterday from the bot resulted in 2 pages being indexed.  For whatever reason when searching for promotional items promomanagers www.promomanagers.com is returned.  However searching direct for promomanagers results in nothing.   It's almost like the live search tool wants to break the word into promo managers.

     

    There are 10+ high quality inbound links showing, zero outbound links, and it is not indexing any other pages which is very odd.

     

     

     

    Wednesday, November 28, 2007 6:11 PM
  • I was at 42/50, then jumped up to 48/50 once, then back to 42/50, now 43/50. The older pages that used to be there (the original 3 that stuck around after the weird thing happened where everything got lost) haven't been crawled since 11/9 or 11/14 depending. That's weird if you ask me.

    At some point it'd be super nice if I actually showed up in search results for relevant topics but alas you can search for the name of my company and I'm not even on the front page. I'll never understand how there can be such a huge variance between this algorithm and Google's while Live Search expects to be competitive with them. You can't find what you're looking for with Live Search! Doesn't that kind of defeat the purpose of a search engine?

    But I digress. I'm not adding the embedded search back to my company site until all this is sorted out and behaves consistently. And frankly for a search engine that doesn't put me anywhere near the top in any relevant searches, I don't see why I should serve their ads on my website (within the embedded search) in the first place, but that's neither here nor there.
    Wednesday, November 28, 2007 6:28 PM
  • What's really bizarre is other engines are indexed Microsoft content faster than Live.   I think really they need to take a close look at how they are indexing as it's falling further behind competitors.  

     

    Our site still has only two pages at www.promomanagers.com    I really believe part of the problem is the Live engine is choking on some keywords and really wants to split names like ours into two words.

     

     

     

    Saturday, December 1, 2007 4:34 AM
  • This is a big concern of mine too.

    What's more important to me, though, is the fact that Live Search's results look nothing like the results provided by Google and Yahoo, for example.

    Now, it's one thing if that's the case because Live Search is providing more relevant results. But almost always, they're less relevant results.

    I've found that the ONLY way to really find what I am looking for on Live Search is to use quotes around phrases, and even then I don't tend to get what I want as often as I do if I just went to Google and searched with a query that didn't have any quotes at all.

    It's almost like Live Search looks for each word individually and never puts any context in the search. I search for a baseball player named Jesus Flores and all I get are photos of Jesus Christ. Come on, guys.

    I searched for auto factory one time and got as the #1 result a photo of John F. Kennedy's autopsy.

    Seriously, what are you doing?
    Saturday, December 1, 2007 10:00 PM
  • By the way, my HOMEPAGE just got delisted from Live Search now. Seriously.

    I have 42/49 subpages indexed now, but my homepage is gone.

    Does anyone want to give an explanation of what is going on or maybe just at least tell us that you're looking into this?

    Sometimes I wonder what the point of this forum is if no one's going to respond to problems that multiple users are having.
    Saturday, December 1, 2007 10:04 PM
  • I can't find anything on Live unless I use quotes.  It likes to breakup search terms and provide irreleveant and useless results based on one part or the other.

     

    Sunday, December 2, 2007 12:39 AM
  • New update!

    As of this morning I am down to 2/50 pages indexed. I had 42 yesterday, and 41 of those have been completely wiped out without notice. My homepage, which had been delisted yesterday without reason, is now back. So I have two pages on my fifty-page site listed in Live Search right now.

    The homepage is listed as having been crawled on 12/1/2007.
    The other page has not been crawled since 11/9/2007.

    Irrational.

    And I did nothing.

    I didn't change anything on the site. I didn't change anything in Live Search. I haven't touched a single file on the site. This happened all on its own.

    So Nathan or Jeremiah, would you care to at least tell me you're looking into this - even if it's not true - or should I continue to get jerked around without any acknowledgment that something weird is going on?
    Sunday, December 2, 2007 2:27 PM
  • I have been looking into this.  The reality is there are a lot of reasons that this happens to a site.  I can't triage each individual site who is having this issue, so I am looking into the root causes.   Remember, that SEO/SEM practices can improve your index/rank, however, as I have stated before MSNBot crawls billions of web pages, not every page is indexed. For a site to be indexed, it must meet specific standards for content, design, and technical implementation.

     

    I know you are looking for a more direct solution to your problem and I am working on that.  I will post something as soon as I can speak for all sites who are experiencing this issue.   I appreciate your patience while I dig into this.

     

    Jeremiah 

    Tuesday, December 4, 2007 11:29 PM
  • I ran gopstrategies.com through this validator just now and I put a checkbox in "validate entire site"

    http://www.htmlhelp.com/tools/validator/

    It only found 45 pages.  I thought this could be another tool you use to help get your site indexed.

    Thursday, December 6, 2007 11:54 PM
  • Hi,

    I have been having similar problems to those reported in this thread.

     

    In September 25% of my sites hits came via MSN.
    In October that dropped off a bit.
    in November zero hits came from MSN.
    My website has not undergone any major revisions in that time period.


    If I search MSN for "esoteric science" or "esotericscience" (the url of my site is www.esotericscience.org) my site doesn't even appear in the results.
    These searches always used to list my website in the top five, but now they don't even appear.

    I have recently signed up to Live Webmaster Tools and see that only 3 of my pages (out of about 30) are listed and they each only have one bar!

     

    I reported this problem to Microsoft Global Escalations but they just refered me to this forum.

     

    Two months have passed since this problem began - Surely someone must have some idea of what is going on?

     

    Thanks,

    Lee

    Tuesday, December 25, 2007 11:10 AM
  • My website even worst. It's totally removed from the Live index after I converted to blog format. After re-submit it again I got one page indexed and the last crawl date was a month ago. However I received steady traffic growth from other search engine such as Google and Yahoo.

     

    This is the latest statistic of traffic source comparison between the 3 search engines to my website for last 1 month:

     

    Google = 94.11%

    Yahoo = 4.74%

    Live/MSN = 1.09%

     

    And see this June 2007 statistic:

     

    Live/MSN = 50.91%

    Yahoo = 27.06%

    Search = 19.27%

    Google = 2.75%

    Wednesday, December 26, 2007 9:37 AM