locked
General Observations With The Bot RRS feed

  • Question

  •  

    It's been a few months and I figured it was time to give a rundown of what has happened.  Specifically what we've found that helps, what we think hurts and the like.  We started with 1 page hidden in MSN through most of 2007.  Despite success building an index elsewhere we were unable to promote our way up the chain at Live.  For reasons still mysterious to us - but we believe it was simply a matter of hitting a date threshold known only to Live we got indexed a few weeks ago.   We went from 1 to 40 pages over the course of a day, then from 40 to 1000 over about a week.  The bot continued to aggressively index the site and we've bounced up and down by 20% since.

     

    There still seems to be no real rhyme or reason, live still chooses to leave out some pages like the Port Authority Legacy Jacket come and go from the index without any obvious reason.  It appears to be there today while the Port Authority Competitor Jacket managed to vanish.  I think MSN is still a little sensitive to what it feels are redundancies on the pages and drops one or the other but as of yet cannot figure out which one it wants.   Overall though it's digging pretty quickly through the site but still appears to be ignoring the robots file.

     

    Overall the advice is to get the site out there through related sites in your industry or niche.  Failing that try Yahoo if you are non-commercial, goguides and others.  They do seem to help.

    Tuesday, February 19, 2008 1:27 AM

Answers

  • I don't have much info for you. You might try updating your sitemap's "updated" date which will encourage the bot to recrawl your pages.

     

    Brett

    Monday, May 5, 2008 7:56 PM

All replies

  •  

    Same thing happened to our site, but i did some minor fixes like ALT attributes, Meta Tags etc and Live bot started to index our pages from 200 to 400 to 1000 and now its 51,000 ...

     

    Up till now i've noticed that Live Bot heavily rely on Sitemap, for testing purpose i've removed the sitemap of our website http://kaanza.com interesting to notice that it has dropped 5000 pages in 5 days Tongue Tied..

    Saturday, April 26, 2008 9:57 PM
  • Hello Guys,

     

    Same is happening to me in fact,

     

    I started the website few months ago, and before i submitted to live webmasters tool, the web site indexed number of pages used to increase dailly, since the day i registered my account the number is decreasing in some how a systimatic way, now after i reached 28000 pages, im getting only 7100!!

     

    Note that a decrease happenes (Big decrease) and then a small increase, then a decrease again.

     

    Notes:

     

    1- i use a recognised robots.txt.

    2- im using an xml sitemap.

    3- i never removed pages from my website till now. and never moved them from the original place.

    4- there is no major changes to my website since a while.

    5- my website works fine with all other search engines and the number of indexed pages are increasing.

    6- my rank is full as i see in webmasters tool, all rectangles are green !

    7- im still doing good in search results for keywords on live search even.

     

    My website is www.12allchat.com  english version www.12allchat.com/en

     

    Note: im living in arab world, we have slow internet connections and there is no hosting servers, so im hosting my website in usa on a dedicated server, how to be as close as possible to my language (arabic) and region?

     

    Thanx in advance.

     

     

    Tuesday, April 29, 2008 4:50 PM
  •  12AllChat wrote:

    Hello Guys,

     

    Same is happening to me in fact,

     

    I started the website few months ago, and before i submitted to live webmasters tool, the web site indexed number of pages used to increase dailly, since the day i registered my account the number is decreasing in some how a systimatic way, now after i reached 28000 pages, im getting only 7100!!

     

    Note that a decrease happenes (Big decrease) and then a small increase, then a decrease again.

     

    Notes:

     

    1- i use a recognised robots.txt.

    2- im using an xml sitemap.

    3- i never removed pages from my website till now. and never moved them from the original place.

    4- there is no major changes to my website since a while.

    5- my website works fine with all other search engines and the number of indexed pages are increasing.

    6- my rank is full as i see in webmasters tool, all rectangles are green !

    7- im still doing good in search results for keywords on live search even.

     

    My website is www.12allchat.com  english version www.12allchat.com/en

     

    Note: im living in arab world, we have slow internet connections and there is no hosting servers, so im hosting my website in usa on a dedicated server, how to be as close as possible to my language (arabic) and region?

     

    Thanx in advance.

     

     

     

     

    I posted a similar problem here (the amount of pages is smaller) 2 days ago and no reply from any forum operator yet.

     

    I guess many people have problem with decreasing of amount of indexed pages, so I hope somebody will reply soon.

     

    Tuesday, April 29, 2008 9:15 PM
  •  

    OMG our pages have reached from 51,000 to 18,900 in three days .. whats happening Tongue Tied
    Wednesday, April 30, 2008 8:52 AM
  •  

    Guys,

     

    I got a new replay from Microsoft team which is something different this time

     

    " Hello Alaa,
     
    Thank you for writing back to Live Search Technical Support. My name is Vivian and I understand that your site's sub pages keep disappearing in Live Search page. I realize the inconvenience this may have caused you.
     
    Here are the details that I used for my investigation:
     
    Website Name: www.12allchat.com
    Search URL: http://search.live.com/results.aspx?q=site%3Awww.12allchat.com&form=QBNO
    Robots.txt File: Present
    MSNBot: Not Blocked
     
    Based on my investigation, your main site is indexed on the Live Search however your sub pages are missing. I also checked on your Robots.txt file found on the top-level (root) directory of your website and there does not seem to be any problems.
     
    What I am going to do now is forward your email to one of our Live Search Product Specialists for review. They will be coordinating with you moving forward.
     
    Alaa, we would like to inform you that while MSNBot crawls billions of web pages, not every page on the Internet can be crawled. For example, if your site's link structure does not have links to each page on your site, MSNBot may not be able to find all of your site's pages.  To determine whether we have encountered links to a particular page or domain, you can use the "link:" or "linkdomain:" search operator. Also, higher quality pages are more likely to be crawled than lower quality pages. For more information about search operators please visit http://search.msn.com/docs/help.aspx?t=SEARCH_REF_AdvSrchOperators.htm .
     
    In the meantime, to ensure successful indexing, please review your site's source code and verify that it meets the standards for content, design, and technical implementation. The Live Search design guidelines can be found on this page:
     
    http://search.msn.com/docs/siteowner.aspx?t=SEARCH_WEBMASTER_REF_GuidelinesforOptimizingSite.htm
     
    To see which pages we have indexed from a site you can use the "site:" search operator. For more information about site indexing please see http://search.msn.com/docs/siteowner.aspx?t=SEARCH_WEBMASTER_REF_GettingSiteIndexed.htm .
     
    We ask for your patience while our product specialists try to investigate this issue. If you have additional questions on Live Search and indexing, please do not hesitate to reply to this email. I will also forward the details you include.
     
    Regarding your inquiry on how to have more than 1 sitemap using Webmasters Tool and if you can set geographic area for your website to get better ranking on targeted area or language, please note that Live Search Webmaster Tools is still in Beta and we do not have assisted support at this time. However, if you have questions about the service, you may join our webmasters forum here:
     
    http://forums.microsoft.com/webmaster .
     
    Members of the Live Search Product Group regularly reply to community posts. You can also browse through the forum as many of the top questions have already been answered there. If you have additional questions that have not been answered, please do not hesitate to post a new thread and wait for the product group or other webmasters to answer.
     
    We appreciate your continued support as we strive to provide you with the highest quality service available. Thank you for using Live Search. "
     

    Finally I beieve the reduction is due to the continued updates they are doing, specially the webmaster tool still in beta version, and i believe of my experience that who signedup an account with webmaster tool, loosed indexed pages, with keeping getting what he deserve of rank, i think we guys who have webmaster tool, will get huge increase in pages indexed when they finish updating or when we move to stable version of webmaster tool.
    Wednesday, April 30, 2008 9:08 AM
  • I don't have much info for you. You might try updating your sitemap's "updated" date which will encourage the bot to recrawl your pages.

     

    Brett

    Monday, May 5, 2008 7:56 PM
  • okay wow it has reached to 1700 pages now Tongue Tied .. whereas Google has increased indexed pages.

    Wednesday, May 7, 2008 2:12 PM