Very bad indexing of my site, why RRS feed

  • Question

  • My site www.112-nieuws.nl exists about a year now and since then it's updated daily.
    Now the site contains 4500-5000 pages and is good indexe in G and Y, but not in Live...

    Webmaster Central:
    Blocked: no
    Domain Rank: 5
    Indexed Pages: 55 (Live Search itself gives 98 pages)
    No crawl issues
    205 backlinks

    Overview over time, numbers from Webmaster Central:

    Sept 2007; website live
    29 jan 2008; only 9 out of 1000+ pages indexed (in G & Y no problems)
    06 Feb 2008; only 6 our of 1000+ pages indexed
    27 Mar 2008; only 58 pages of 1700 indexed
    12 Aug 2008; only 86 pages of 2500 indexed
    16 Sep 2008; only 75 pages indexed (So the number is decreasing!!)
    26 Okt 2008; only 55 pages indexed (again decreasing!!)

    What is the problem with indexing the site??
    Sunday, October 26, 2008 12:03 PM


All replies

  • There are a few possibilities:


    1) No sitemap to help robots find the pages.  Visit http://sitemaps.org to learn more.

    2) Robots.txt too restrictive.  Visit http://www.robotstxt.org/ for more info.

    3) Dynamic content makes many pages seem like one.  This is a little trickier...use .htaccess or a program like ISAPI_Rewrite to "trick" the robots into thinking you have static URLs.

    4) No links to the pages.  Your problem may be with variables being passed and scripted pages that there is no physical anchor text to link to your pages.  Robots like to see nice markup with an anchor link and a relevant descriptive text.  This also helps for text readers.

    5) Hosting company has firewalls or other blocks that are reducing robot traffic.  A cheapy host or ISP will do this to save bandwidth.

    6) Updated too frequently.  If updates change the URLs or links on pages, this confuses robots and makes them think the page is no longer relevant.  If you put a link on to one of your pages on a page for a week, then remove it from that page, it is possible a robot will never see it in its cycle.  Make links as permanent as possible.


    Good luck.

    Monday, October 27, 2008 6:58 PM
  • Thanx for your comment, but your comment is copy-pasted I guess, without looking to this case... (sorry)

    1) There is a sitemap, this one is automatically rebuild after new posts are made to the site.
    (not with every post, but about 2 times a day, to prevent spamming)

    2) No pages are blocked by the robots.txt

    3) I use URL-rewrite for URLs, and I payed extra attention to prevent duplicate content, so this is carefully done

    4) Like mentioned in my post the Webmaster Central sees 205 backlinks to my site.

    5) I have my own dedicated server

    6) My site is a news-site, URL's don't change.

    I work in the webdesign webdesign business as online communication specialist with special interest for SEO, Searchengine Marketing (AdWords Qualified) and I also work with different Analytics software (Google Analytics, GetClicky, Woopra) so I think I know what I'm doing, these points are the basic things.

    I also tracked the activity of the different bots, the results of that told me that Googlebot was visiting the site very active, Yahoo a bit less, but Yahoo was also picking up changes, Live only crawled only a few pages....

    The sitemap is submitted on Google Webmaster Tools and also to Yahoo Site Explorer. Both are good at indexing my site, but Live Search somehow isn't. It's very annoying that i'm giving a lot of attention to build a site that should be good crawlable and that Live Search just doesn't crawl my site. I'm hoping for some reaction from someone from Microsoft.... anyone?

    Monday, October 27, 2008 7:36 PM
  • Actually no, not copy-pasted.  These are just basic strategies that typically work and are usual causes so no surprise that others might have the same advice.


    1) that makes sense.

    2) good, as long as no agents are blocked either

    3) depending on how the URL-rewrite is set up, just be sure case-sensitivity or directory setup is not causing an issue (I doubt that would be the case, but just covering bases).

    4) 205 backlinks - are those unique page links or are they 205 links going right to just a few of your pages?  In either case, that's not enough to index all of the pages you said you had.

    5) Good dedicated server, but you have an ISP or host of some type.

    6) Not sure how a news site would preclude it from having changing urls, but sounds good.


    I guess if you've covered all the points and it sounds like you know what you are doing, and only Live is having problems creating links, I would tend to follow with the "Live Sucks" theory.  I'm having problems with my site indexing non-existant pages and not the real ones, despite taking all the same measures as above.


    When it comes right down to it, Live is just ages behind Google.  I would also hope for Microsoft to respond, but I think they are more interested in buying Yahoo! and will let their own engine die in hopes that they can pull that off.


    If you're good on Google and Yahoo!, that's your best bet.  Perhaps (if you haven't already) add Google sitesearch to your site and let MSN/Live visitors just get to your homepage and search the rest from there.  If your homepage is getting all the crawls from MSN, concentrate on getting enough info there for bring people to the door and let your site handle the rest.


    Aside from that, WHOIS cannot find your site, and tracert can't resolve the IP address for your domain.  Check your registration and your DNS Host.  If I can't get to your site, it may explain why a robot can't.

    Monday, October 27, 2008 8:10 PM
  • Thanx again for your comment,

    The WHOIS does actually see my site, try http://who.is (it also shows the IP-address
    I guess also Google / Yahoo is my bet, but I want to score on Live Search also.....
    And I realize that MSN is indeed ages behind... too bad...

    Still hoping for a reply from some Microsoft moderator?
    Tuesday, October 28, 2008 10:23 AM
  • I have exactly the same problem with my site www.anuo.be - the number of indexed pages is decreasing day after day
    and today i have ZERO indexed page. Searches disapear for example in HOTBOT who use live search results and like you visitors are coming from G .. and Y .. I have impression that msn robot erase his work
    Tuesday, October 28, 2008 10:57 AM
  • Hi,


    Are you employing redirects on any of these pages? I will do a little more research on this and get back to you.




    Tuesday, October 28, 2008 4:28 PM
  • Okay, I hope you have a good answer so we can improve some things...
    What kind of redirects do you mean? (301? I don't use those anywhere)
    Thursday, October 30, 2008 8:56 AM
  • Hi bretti read in another post that you have submited urls for a site crawled by msnbot but not indexed in live search - can you give link where i can do the same. If it is another way to be indexed, i will place urls from my index (http://www.anuo.be) because i have always no result with xml file on november 1 (root of site since oktober 10 - http://search.live.com/results.aspx?q=site%3Aanuo.be&FORM=MSNH). You have meaby another solution?  At the moment G.. as indexed 153 pages on 940 and begin to send me visitors.thanks for your time
    Saturday, November 1, 2008 6:51 AM

    What is needed is a customer service rep. The rep provides the service to the customer very easy! Its all about being able to find what you want when you want. I can show you how it is done its really easy ill show you how its done,
    Saturday, November 1, 2008 10:30 AM
  • customer service rep
    what's that ?

    Saturday, November 1, 2008 9:37 PM
  • Hii brett
    on november 4th, i have no one page indexed by msnbot - what can i do ? - on oktober 14th, the home page was indexed and two days later disappear from data's. At the moment it is impossible to put a complete sitemap because we are under construction. If i place a sitemap of homepage, did msnbot follow all links ? - can you give me a way because xml file don't give results. Another thing i do not understand my livesearch account see Outbound links from anuo (???) with zero indexed page - When i do a search on msn (http://search.live.com/results.aspx?q=anuo.be&FORM=MSNH) - live search give as result one of my post on this forum - Thanks for your time
    Tuesday, November 4, 2008 8:13 AM
  • Brett (or somebody else),

    Can you please have a look at the problem I submitted in this subject??
    Over the last year I placed some messages in this forum to solve this problem, but nothing yet....
    Tuesday, November 4, 2008 11:24 AM
  • Hi brett
    can you explain me something : when i ask site:www.anuo.be i see no results in msn.be and when i go whith my login on my lifesearch 'page' anuo is not crawl and not indexed. When i go on MAHALO.COM who open results in a frame it give for live 8 results (600 for google) - do the test http://www.mahalo.com/SpecialTongue Tiedearch?search=site%3Aanuo.be&go=Search - how is possible that mahalo show results and msn.be NOTHING. When i go see cache date of crawling is 11/7 - 4 days ago - and on my dashboard

    Site status 

    Last crawled: N/A Indexed pages: N/A

    can you give ma an explation about those differences - thanks for your time
    Sunday, November 9, 2008 11:44 PM
  • ALLELUIAH - msn bot has indexed 8 pages on 11/10

    I hope that the number of indexed pages grow speedly but he has to work hard because i have 1350 pages for the moment
    Monday, November 10, 2008 4:07 PM
  • Right now the number of indexed pages is 199, so the number is rising, > 4.500 to go.....
    Wednesday, November 19, 2008 10:06 AM
  • Well, seems like live finally is finding my pages.
    Right now 1200 pages indexed! :-)
    Monday, November 24, 2008 9:05 AM
  • Congratulations!


    Friday, December 5, 2008 7:27 PM