locked
Where are the crawl bots?? RRS feed

  • Question

  • I find this very frustrating that the "other" search engine finds my site but MSNbots do not regularly crawl my web site.

     

    http://www.alltechportal.com

     

    Last index was 07/02/08, yes this is good compared to other posts here but the 'other' search engine is crawling every 3-4 days and as my site is up dated all the time this issue is crucial to my business.

     

    Can I set the crawl rate ?

     

    Is this issue being addressed by the powers that be ??

     

    When can we here see some progress in this issue?

     

     

    Tuesday, April 1, 2008 2:18 AM

Answers

  • This is answered in the fact that I have responded. It is a way for me to keep track of threads and not just a final response (although sometimes it is, but it is fairly obvious in those instances). When you reply, it shows up as unanswered again in my admin tool. In regards to the datacenter info, I am unable to give you that information as we have datacenters located around the world and only a few senior individuals know where the bots originate from.

     

     

    Brett

    Monday, May 5, 2008 5:20 PM
  • Website crawled again on 05/05/08

     

    You must be doing something right.

     

    Answered

     

    Thursday, May 8, 2008 6:32 AM
  • I am happy to report that my site is being crawled at least once a week and my site rank has improved every time.

     

    Thanks to all and good luck.

     

    Thursday, May 22, 2008 1:22 AM

All replies

  • Hi,

     

    Currently you are ranking very low. Please read our blog post:

     http://blogs.msdn.com/webmaster/archive/2008/01/11/i-m-not-ranking-in-live-search-what-can-i-do.aspx

     

    Higher ranked sites generally get crawled more often.

     

    Thanks,

     

    Brett

    Tuesday, April 1, 2008 6:19 PM
  • Hi Brett thanks for you reply,

     

    I have chaged the content and wording putting in lots of keywords but if the crawler won't come to my site then how will my ranking change?

     

    I said in my previous post, why does this question arise time and time again at this forum?

     

    The frequency of crawl rate is abysmal.  Can you please answer the questions in my previous post instead of pushing the generic line of "you rank too low"?

     

    As I said before, this question keeps coming up time and time again. What are the boffins doing about it? Why can the other search engine list my site in top ten in most keywords most of the time and yet Live.com can't even search my Home Page and I cannot find it in the listing in Live.com unless I type in the specific address?

     

    Tuesday, April 1, 2008 9:47 PM
  • Ok, well looks like this post will not be answered properly. So I'll pose another question. Where exactly are the websites crawled from?

     

    Where are the data centres that crawl the sites?

     

    I have a problem with website security regarding fraudulent transactions. I would like to block some countries from accessing my website. Will this have an effect on crawling data? Can someone tell me where the data centres are based?

     

    Does anyone know this information?

     

    Tuesday, April 8, 2008 10:49 PM
  • Ok so these forums are a hive of activity and answers are flying in.

     

    My site was crawled 8/4/08.......but only terms and conditions page and register page....

     

    NOT my home page and still this issue is unresolved.

     

    So if my home page doesn't get crawled no new information can be found in live searches and you still cant find my website.

     

    BTW the other search engine has crawled my site many times since my first post here and I rank very well there.

     

     

     

    Wednesday, April 9, 2008 11:09 PM
  •  

    In my opinion, I think while other search engines value 'smaller' sites, MSN does not.

    Unless your mass-linked on other sites, your not going to get more than a sprinkling from MSNBot

     

    My own logs show that MSN spiders 1% of that compared with Yahoo and 10% of that which Google spiders.

    In fact even Gigabot indexes me three times more than MSNBot.

    Monday, April 14, 2008 11:59 PM
  •  

    Echo Echo Echo............... (Watches a tumble weed blow through these forms).............

     

    SITE LAST CRAWLED ON 07/02/08

    I type into live search this ..........   alltech portal, four pages come up ten links on each page and NONE are my home page. Forty links HAHAHA

    I have finally figured it out, no-one uses live so that’s why I get no hits and if no-one searches on there why bother to crawl sites for updated data.

     

    To Steve-On thanks for your response to this thread and helpful info. (Do you work here? :-))

    Wednesday, April 23, 2008 12:14 AM
  • Hooray my site was crawled on 25/4/08.

     

    My thanks go out to all the people who made this happen " I couldn't have done it without you"

     

    This is a very informative forum and very helpful.

     

    I also wish to thank "mom and dad for being there for me" and also god for his powers to make this happen.

     

    On a side note my page has been crawled by the other search engine at least 25 times in this time period.

     

    Tuesday, April 29, 2008 9:32 PM
  • I am shocked to say that my site was crawled again on 30/4/08.

     

    Live.com is such a good seach engine, it has my site listed very high with my keywords.

     

    and I'm getting hits from there.

     

    Should I call this thread fixed??...............maybe I'll wait Smile

     

    Friday, May 2, 2008 12:03 AM
  •  

    Hi,

     

    We recently updated the crawl bot, so hopefully that will take care of the issue, but I'll keep an eye on the thread.

     

    Thanks,

     

     

    Brett

    Friday, May 2, 2008 8:33 PM
  • Hi Brett, Thanks for your reply however it is a bit presumptuous to call this answered.

     

    I WILL determine as the OP whether or not my issue has been answered/fixed.

     

    More time needed to see if the bot does come to my site regularly and updates my site info.

     

    Can you answer the other questions that I asked??

     

      Where are the data centres that crawl the sites?

     

    I have a problem with website security regarding fraudulent transactions. I would like to block some countries from accessing my website. Will this have an effect on crawling data? Can someone tell me where the data centres are based? 

     

     

    I'm happy that this is being addressed and I appreciate the effort to resolve this issue.

    Saturday, May 3, 2008 1:33 AM
  • Site crawled again on 01/05/08 and 02/05/08.

     

    Hey it might be working.

     

    Good Job.

     

    Sunday, May 4, 2008 2:37 AM
  • not in my case

     

    I even didnt get an answer in this forum...our site www.zoznamrealit.sk has still only 5 from 22000 pages indexed. Somebody wrote here, that the sites arent indexed because nobody uses LIVE, but nobody will it use, if there is no indexed content....at least in smaller countries like Slovakia is.

     

    Sunday, May 4, 2008 4:27 PM
  • This is answered in the fact that I have responded. It is a way for me to keep track of threads and not just a final response (although sometimes it is, but it is fairly obvious in those instances). When you reply, it shows up as unanswered again in my admin tool. In regards to the datacenter info, I am unable to give you that information as we have datacenters located around the world and only a few senior individuals know where the bots originate from.

     

     

    Brett

    Monday, May 5, 2008 5:20 PM
  • Thank you Brett, I am happy with your reply/answer.

     

    I'm still monitoring the crawl rate of my site and as of today the dates mentioned above remain the same (no change).

     

    As for the country of origin for the Bots, Thank you for this answer also, one more "Bot crawl"

     

    and I will be happy to say this post is finished.

     

    Tuesday, May 6, 2008 12:09 AM
  • Website crawled again on 05/05/08

     

    You must be doing something right.

     

    Answered

     

    Thursday, May 8, 2008 6:32 AM
  • I am happy to report that my site is being crawled at least once a week and my site rank has improved every time.

     

    Thanks to all and good luck.

     

    Thursday, May 22, 2008 1:22 AM