locked
Total pages indexed - a bit depressing RRS feed

  • Question

  • Ok I am not sure if this is really where to post this but

    for one of my sites the total pages indexed comes up as 24, however the same site in google comes up as 10,000ish and yahoo has almost double that at 22,000,

    So some sort of diagnostic tool that tells me why pages aren't being indexed would be really helpful, as i have wracked my brain to figure out why the indexed pages total is so low, and all i can come up with is that perhps live search doesn't like dynamic URLs (zencart, phpBB etc)

    Also where should i post this query as this probably isn't anything to do with the tools, (although an errors/problems tool would be nice)
    Wednesday, November 21, 2007 11:14 PM

Answers

  • I have heard from a number of customers that they don't understand why their total pages indexed is low.  The basic answer is that, while MSNBot crawls billions of web pages, not every page is indexed. For a site to be indexed, it must meet specific standards for content, design, and technical implementation. For example, if your site’s link structure doesn't have links to each page on your site, MSNBot may not find all the pages on your site.

     

    Ensure that your site follows our indexing guidelines. These guidelines will help you place important content in searchable elements of the page. Also, make sure that your site doesn't violate any of the technical guidelines that can prevent site ranking. For more information about site ranking in web search results, see About website ranking.

     

    I agree with previous posters that it would be good to submit a sitemap with us as a next step.

     

    Hope this helps a little.

     

    Jeremiah

     

    Tuesday, December 4, 2007 7:57 PM

All replies

  • Ok I'm back, Could a mod move this to the General section of the webmasters forum

    I should add, I have validated my sitemap (thats one of the fastest validation tools i have ever seen)
    Nothing is blocked according to the summary page
    and i have provided the URL to my sitemap index

    The 24 links it seems to be indexing are all the hardcoded urls that are in various parts of the template, No dynamically generated urls seem to be appearing

    Ok, I just went to double check it in live search and apparently there are now 11 pages only in the index

    i used this search string

    site: scrapbookoutletnz.co.nz (yes i know there shouldn't be a space between the : and the s but if i put them next to each other I get a smilie like this Tongue Tied)

    But it still sees 24 pages indexed in webmaster tools

    Thursday, November 22, 2007 10:34 PM
  •  

    On the Sitemap tab, there is a brief explanation of how you can ping Live Search with your Sitemap, using a URL like this:

    http: //webmaster.live.com/ping.aspx?siteMap=[Your sitemap web address]

     

    You need to include your full sitemap URL, including the http-part.

     

    When I did this, my index count jumped from 20-ish to over 200-ish, and it climbs each day. It worked for me, it might work for you.

     

    Monday, November 26, 2007 11:26 AM
  • I'll give that a go, I had tried clicking on the submit button on the  sitemaps page but that hasn't improved things, i am down to 2 pages indexed, which is really not good for a site that does a lot of business around the Xmas period

    Another bizzare thing is that, one of our alternate domain names has 30 links under it, and the majority of the links are too our old shopping cart that was replaced about 6 months ago if not earlier, (there are redirects on the server that will send these links to the relevant location in the new store, but they are marked as permanant redirects so they should have updated by now)

    We own 4 variations on our base domain name (.com ,co.nz plus a variation of the sitename) all of which are pointed at the same domain and have been for a long while

    I'll keep an eye on the page indexes over the next few days
    Monday, November 26, 2007 7:57 PM
  • Any update  - did the ping submit work?! I have a similar issue with a 12,000 page site with a grand total of 28 pages indexed in MSN. Have tried both sitemap submission methods now so I guess will wait and see. Be good to know if you have had any results in the last 72 hours though!

     

    Thursday, November 29, 2007 9:38 AM
  • Yep i have had a 400% increase in indexed pages (there are now 9 pages indexed)

    The alternate domain name (which redirects to the main one above ) has got 61 pages indexed under it, still can't figure that out as it redirects on the server  to the  main one and has done for 4 or more months
    Thursday, November 29, 2007 7:00 PM
  • Many thanks Nigel - the percentage increase sounds impressive at least !! I can only hope for similar results! Cheers

     

    Friday, November 30, 2007 9:13 AM
  • Ok i spoke to soon

    we are down to one page indexed now,

    it could be worse though, the site that is normally miles ahead of us in all rankings has vanished from MSNs index (and it does seem to be denting their online business), although unfortunatly the company that was quite a distance behind us, is rocketing up the rankings and is almost neck and neck with us on a ranking site they all use

    I am utterly stumped and can't figure out what is going on, apparently the last crawl of our site was 19/11/07 (or 11/19/07 for you USA people), if i knew what was causing it i'd fix it, but i just don't know,

    i wonder if i can sue to get hair plugs for rapidly thinning hair hmmmm


    Sunday, December 2, 2007 9:16 PM
  • Looks like we might be having the same problem. Here's a link to page 2 of my thread, my most recent post is at the bottom. I too spoke too soon and ended up getting hosed (again) when another crawl deindexed 94% of my site for the second time.
    Sunday, December 2, 2007 10:05 PM
  • Yes it looks like the same problem

    I also checked my logs and the msnbot is looking at the robots.txt, and my main page and thats it

    wanna know something funny, my personal site, which is basically a repository for rubbish etc has...
     9 pages indexed in google,
    48 pages indexed in yahoo
    but over 100 in live search

    perhaps i should add links from my personal site to try and boost the business site

    Its just the not knowing why the sites are dropping that is bugging me, and for our main search term we seem to have been leapfrogged by the company that was behind us


    Monday, December 3, 2007 2:26 AM
  • I have heard from a number of customers that they don't understand why their total pages indexed is low.  The basic answer is that, while MSNBot crawls billions of web pages, not every page is indexed. For a site to be indexed, it must meet specific standards for content, design, and technical implementation. For example, if your site’s link structure doesn't have links to each page on your site, MSNBot may not find all the pages on your site.

     

    Ensure that your site follows our indexing guidelines. These guidelines will help you place important content in searchable elements of the page. Also, make sure that your site doesn't violate any of the technical guidelines that can prevent site ranking. For more information about site ranking in web search results, see About website ranking.

     

    I agree with previous posters that it would be good to submit a sitemap with us as a next step.

     

    Hope this helps a little.

     

    Jeremiah

     

    Tuesday, December 4, 2007 7:57 PM
  • Thanks Jeremiah

    I thought i had said i had a sitemap, but it looks like i haven't said so, I actually have 3 sitemaps all of which are listed in my sitemapindex.xml, also at the very bottom of my robots.txt i have the line

    SITEMAP: http://www.scrapbookoutletnz.co.nz/sitemapindex.xml  
    to add in autodiscovery of the sitemaps

    I have pinged the sitemap servery thing with all the sitemaps and the sitemapindex is the one that is listed in my Webmaster tools

    I am loath to make any major site changes as my Google, Yahoo and even my Ask indexing is fairly good

    If any of you other webmasters out there have a spare 5 minutes

    http:\\scrapbookoutletnz.co.nz is the site

    as I cannot spot any technical issues, if something appears glaringly obvious to you let me know as a fresh pair of eyes might help

    I do know there are validation issues on my home page regarding the marquee tag (a microsoft invention), and the flash <embed> tags, but surely they can't be throwing it off



    Tuesday, December 4, 2007 9:36 PM
  •  

    Hi Jeremiah,

     

    Basically what all posters on teh thread agree is that Live Webmaster Console shows the smallest number of pages indexed of all Webmaster Consoles (Google and Yahoo). Your explanation:

     

    "For example, if your site’s link structure doesn't have links to each page on your site, MSNBot may not find all the pages on your site."

     

    Is not Live speciffic, but any crawler will not find your pages if thse are not linked from teh pages in the current index of a search engine.

     

    So this is why the answer does not help.

     

    PS. Do not take it personally! Smile

     

    All the best,

    Ivan

    http://www.employireland.ie

     

    Wednesday, December 5, 2007 8:30 PM
  • When I visit my Site summary, it says "Indexed pages: 17,500". But my site only has about 3000 pages. I'm not sure what all else might be counted. If I count JPEGs and other files as "pages", that boosts the number to ~8000. And maybe they get indexed twice, maybe once at mydomain and once at www.mydomain? That would bring us up close to 17,500--but that's a lot of guesswork piled on top of conjecture.

    It might be nice if that "Indexed pages" text was a hyperlink to a wonderful help page that says what "Indexed pages" means in this context.

    I like the Webmaster Center. Keep up the good work!

    -Larry Hosken

    Sunday, December 9, 2007 5:03 AM
  • "And maybe they get indexed twice, maybe once at mydomain and once at www.mydomain?"

    Oh, looking at the list of "Top 5" web pages, I do see some pages from mydomain and some from www.mydomain. That's a little strange since www.mydomain/foo.html is a 301 redirect to mydomain/foo.html

    I don't know whether that suggests that the indexer is doing something silly... or if the crawler is doing the right thing and the Webmaster Tools "site summary" is reporting it in a silly way.
    Sunday, December 9, 2007 5:18 AM
  • Larry, I think the problems are pretty deep at this point with live search and aren't limited to bad counts.  If the algorithms involved give more weight to promo managers (on a job site) than they do www.promomanagers.com when someone actually searches for www.promomanagers.com you have to wonder what is really going on and how accurate the results are going to be for everyone.

     

    Forgetting all this other stuff for a minute, you would think a search engine would return exact search matches before all else, but live is not doing that.  It's interesting to go through and search keywords in the promotional products industry as it shows a massive disparity between Live and the rest.

     

    It's been 3 weeks since we setup live webmaster tools and we regret doing it.  I think something in the setup is preventing proper indexing.  It's been a disaster since the first day we signed up. 

     

    Monday, December 10, 2007 10:28 PM