locked
Any word on sites dropping from the index? RRS feed

  • Question

  • So, has anyone figured it out yet? From looking here and elsewhere around the web, it seems that many sites have just dropped out of Live Search completely in the last few months and MSNBot refuses to crawl them, even though they are not listed as blocked. Is this a recognized issue yet?

    For the record, here's what happened with one of my sites ( http://www.mymoneysites.com/ ). It was being crawled, and showed up in Live Search fine prior to November of 2007. November 16 was the last visit by MSNBot. After that it was no longer crawled, and it disappeared from Live Search. In December I signed up with the Webmaster Tools here to investigate. On that day, MSNBot crawled my site, but it hasn't returned since no matter what I do. It says there are only two indexed pages, though actually none show up in Live Search - even searching the literal name of the site "MyMoneySites" in quotes only lists pages with links to my site, but not my site itself. I checked the entire results of this search. It's not just that it doesn't show up first; it doesn't show up at all. Strangely enough, if I change the search to the "advanced" search, without changing anything else, my site suddenly shows up as the first result. That seems very unusual and may help troubleshoot the issue. Searching site:mymoneysites.com also works, though the results oddly say "results 1-1 of 2".

    I'm not sure what the problem is here, as Yahoo and Google crawl my site every day with no issues.

    Friday, January 11, 2008 5:15 AM

Answers

  •  TheophilusCarter wrote:
    So, has anyone figured it out yet? From looking here and elsewhere around the web, it seems that many sites have just dropped out of Live Search completely in the last few months and MSNBot refuses to crawl them, even though they are not listed as blocked. Is this a recognized issue yet?

    For the record, here's what happened with one of my sites ( http://www.mymoneysites.com/ ). It was being crawled, and showed up in Live Search fine prior to November of 2007. November 16 was the last visit by MSNBot. After that it was no longer crawled, and it disappeared from Live Search. In December I signed up with the Webmaster Tools here to investigate. On that day, MSNBot crawled my site, but it hasn't returned since no matter what I do. It says there are only two indexed pages, though actually none show up in Live Search - even searching the literal name of the site "MyMoneySites" in quotes only lists pages with links to my site, but not my site itself. I checked the entire results of this search. It's not just that it doesn't show up first; it doesn't show up at all. Strangely enough, if I change the search to the "advanced" search, without changing anything else, my site suddenly shows up as the first result. That seems very unusual and may help troubleshoot the issue. Searching site:mymoneysites.com also works, though the results oddly say "results 1-1 of 2".

    I'm not sure what the problem is here, as Yahoo and Google crawl my site every day with no issues.

     

    So I checked your site to see how things were going for you across the web.   I noticed you have 48 pages being indexed by Yahoo and 2 pages indexed in Live Search, and 25 pages in Google.  I looked at your site and given that your site is a directory and as a directory you may have trouble increasing that rank.   Being a Directory is not bad, but you need to have something that would distinguish yourself.

     

    Looking at your backlinks to see what might drive rank for your site.  I see mostly more directories pointing to you. 

     

    forums.site-reference.com/topic/6180/Please-Review...
    www.chrismerriman.com/index.php/archives/1389
    www.projectwonderful.com/out_nojs.php?r=1&c=0&id=6... 
    www.projectwonderful.com/out_nojs.php?r=0&c=0&id=6... 
    www.legittorrents.info/details.php?id=761cad6577bf...
    www.joeant.com/DIR/info/get/178/94467 
    www.joeant.com/DIR/cat/178/Get_Paid
    www.fetchcash.com/cash/money_making_ideas.html

     

    You really need a higher quality of links to drive up your rank.  If you can do this you should see improvements in all the SERPs.

     

    Jeremiah

    Monday, January 14, 2008 8:27 PM

All replies

  •  

    I have had the exact same issue. Until today, I didn't know that MSN finally had wemaster tools. My client's site, touchdownhotels.com was indexed by MSN for almost all of its 80+ pages and was getting first page search results for a while....then one day....nothing. I originally noticed that there were no clicks coming from MSN, then I decided to check the keywords and finally realized that there were only 2 pages indexed....the home page and a contest page! I was devistated. I am a webmaster and this site is one of 4 sites my client has asked me to build. It makes me look real bad when one day all of a sudden your site disappears from a search engine. I wish someone would answer this question. I even submitted the site for review for possibly being blocked for some reason, but I never got a response. I used to really enjoy MSN for placement, now....not so much.

     

     

    Alex

    Friday, January 11, 2008 4:02 PM
  •  

    It's not just not be crawled, our site www.promomanagers.com has been getting 50-100 pages per day checked by MSN Bot for the last 7 and yet we still can't be found in the index.  Like most of you Microsoft refuses/does not answer emails sent to email addresses they've published to answer questions regarding live.com issues.  It's all pretty comical at this stage and now that the scope of the problem is clearly much larger than they thought, MS support seems to have vanished from this site as well.

     

    Friday, January 11, 2008 6:11 PM
  • Same here, I've also emailed Microsoft with no response. Things I have tried so far:

     - signed up for Webmaster Tools (as I said, it crawled my site once more on the day I signed up, but has not returned since and that was over a month ago)
     - checked my robots.txt file with the validator on Webmaster Tools - no problems
     - submitted my sitemap.xml to Webmaster Tools
     - used the ping service to submit my sitemap
     - sent an email to MS about the problem

    Friday, January 11, 2008 11:34 PM
  • Hi,

    I run www.simply-europe.co.uk, a european holiday site. I am also having devistating problems like yourself and many other webmasters. I had over 400 pages indexed with msn, with over 90 pages first or second page ranked. In the middle of october 2007 I noticed I was loosing pages from  the index. Within 5 days the indexed pages had gone down to about 50, with only my home page showing up in the search under "European Package Holidays"

    This is first page on yahoo out of 36,000,000 listings. Yahoo and google have nearly 400 pages indexed.

     

    Having contacted MSN help, they advised my to resubmit, where over 400 pages were again indexed,only to be lost again.

     

    We are now into the main holiday booking season, but because I have lost all my first page listings, mainly holiday company pages such as Thomson Holidays,Airtours & Expedia, my page views are down by 80% to what they should be, and  earnings are  also down at least 80%. This hurts more because I went as a full time webmaster in Sept 2007, because I was doing so well until MSN messed up well and proper.

     

    I have now had to start PPC,but on Yahoo and Google to try and recoup some earnings.

     

    MSN....Please sort this problem before you put a lot of webmasters out of business

    Sunday, January 13, 2008 10:49 AM
  •  TheophilusCarter wrote:
    So, has anyone figured it out yet? From looking here and elsewhere around the web, it seems that many sites have just dropped out of Live Search completely in the last few months and MSNBot refuses to crawl them, even though they are not listed as blocked. Is this a recognized issue yet?

    For the record, here's what happened with one of my sites ( http://www.mymoneysites.com/ ). It was being crawled, and showed up in Live Search fine prior to November of 2007. November 16 was the last visit by MSNBot. After that it was no longer crawled, and it disappeared from Live Search. In December I signed up with the Webmaster Tools here to investigate. On that day, MSNBot crawled my site, but it hasn't returned since no matter what I do. It says there are only two indexed pages, though actually none show up in Live Search - even searching the literal name of the site "MyMoneySites" in quotes only lists pages with links to my site, but not my site itself. I checked the entire results of this search. It's not just that it doesn't show up first; it doesn't show up at all. Strangely enough, if I change the search to the "advanced" search, without changing anything else, my site suddenly shows up as the first result. That seems very unusual and may help troubleshoot the issue. Searching site:mymoneysites.com also works, though the results oddly say "results 1-1 of 2".

    I'm not sure what the problem is here, as Yahoo and Google crawl my site every day with no issues.

     

    So I checked your site to see how things were going for you across the web.   I noticed you have 48 pages being indexed by Yahoo and 2 pages indexed in Live Search, and 25 pages in Google.  I looked at your site and given that your site is a directory and as a directory you may have trouble increasing that rank.   Being a Directory is not bad, but you need to have something that would distinguish yourself.

     

    Looking at your backlinks to see what might drive rank for your site.  I see mostly more directories pointing to you. 

     

    forums.site-reference.com/topic/6180/Please-Review...
    www.chrismerriman.com/index.php/archives/1389
    www.projectwonderful.com/out_nojs.php?r=1&c=0&id=6... 
    www.projectwonderful.com/out_nojs.php?r=0&c=0&id=6... 
    www.legittorrents.info/details.php?id=761cad6577bf...
    www.joeant.com/DIR/info/get/178/94467 
    www.joeant.com/DIR/cat/178/Get_Paid
    www.fetchcash.com/cash/money_making_ideas.html

     

    You really need a higher quality of links to drive up your rank.  If you can do this you should see improvements in all the SERPs.

     

    Jeremiah

    Monday, January 14, 2008 8:27 PM
  • Jeremiah,

    Thanks for your response and I understand what you are saying. Of course, since the site I pointed out is (relatively) new I am still in the process of getting backlinks. However, right now I am not really concerned about my site's rank in MSN - I am concered that MSNBot does not crawl my site at all, and can't seem to see more than one page during the rare times it does crawl. Why would it not crawl my site a single time in over a month? Especially considering that it was crawling my site regularly in September and October. If the backlinks were important in determining crawl rate, why would MSNBot have crawled it more often back then, when I had fewer backlinks than I do now? Google and Yahoo, on the other hand, not only crawl the site every day but several times per day.

    So the concern here (and on other forums I have read) is that MSNBot seems to ignore certain sites and refuse to crawl them completely - even sites that it was crawling fine several months ago.
    Monday, January 14, 2008 9:55 PM