locked
Site is found, but not being crawled. RRS feed

  • Question

  • Hello all,

    I see my site is being accessed by the MSNbot but its not effectively crawling my site.  G**gle is crawling it daily and is indexing thousands of pages a day.  I have checked my robots.txt / sitemap.xml files and everything checks out. Even used some public analyzers to test crawlability from sever major search engines (yahoo/msn/google) and everything checks out.  Even 100% html validation by w3.org.

    Yet with all this MSN barely notices my site.  And the site is not being blocked.

    Site: http://www.adcrow.com

    Is there something I am missing, need to correct?  My site has over 700,000 articles but MSN barely gets past the homepage.

    Thanks
    Wednesday, December 12, 2007 7:09 PM

Answers

  • There are a lot of reasons why sites don't get indexed.  One of the big challenges for webmasters is ensuring that the sites that link to them are quality sites.  As a webmaster, you need to ensure not only that there is great content on your site, but that you get the right kind of people and sites linking to you. 

     

    So one of the best ways to ensure that MSNBot finds your website is to include valuable content that other site owners want to link to. MSNBot crawls billions of web pages, not every page is indexed.

     

    For a site to be indexed, it must meet specific standards for content, design, and technical implementation. Ensure that your site follows our indexing guidelines. I recommend reading the techniques section.  A lot of sites that are ranking low, may have a issue that is being caused whether inadvertantly or not by items in this section.

     

    Jeremiah

    Monday, December 17, 2007 7:12 PM

All replies

  • Welcome to the club and I wish I could help you.  Take a look at the list of problems and what you describe has been asked in the past.  Just look at my thread from realdealtools.  We are at a lost of why we are not being indexed.  Live Search team has really not provide any real guidence or answers.

    Steve


    Friday, December 14, 2007 1:46 PM
  • Maybe this is what they meant when the CEO said they were going to kill google?   They figured they couldn't beat them in search technology so they'd save webot crawling costs by only having it index one  page?

     

    I sent out a certified letter this morning to a higher up at Microsoft asking for an explanation and pointing him to this site.  I don't expect much of a response but who knows.  It's obvious the Live team knows the problem and just has no answers.  

    Whether or not a site has quality inbound links is irrelevant.  There are plenty of excellent websites with almost no links for civic organizations and charities.   The inability of the livebot to even follow a basic link from the homepage stems from bad code, not a bad site.

     

    When you look at our dilemna, a search of promomanagers on live search  yields a vist to a club jobs site for a job as promo manager.   There are then 40 indexed pages dealing with our company (but not our site) and then some other gibberish.  Is it any wonder Live's traffic is a fraction of Google when you can't even find a website by typing in the website page?

    We shut down whatever ads we were running this morning, this is just silly that they can't index a page properly.  Why reward them with your advertising dollars?

     

    Friday, December 14, 2007 3:54 PM
  • There are a lot of reasons why sites don't get indexed.  One of the big challenges for webmasters is ensuring that the sites that link to them are quality sites.  As a webmaster, you need to ensure not only that there is great content on your site, but that you get the right kind of people and sites linking to you. 

     

    So one of the best ways to ensure that MSNBot finds your website is to include valuable content that other site owners want to link to. MSNBot crawls billions of web pages, not every page is indexed.

     

    For a site to be indexed, it must meet specific standards for content, design, and technical implementation. Ensure that your site follows our indexing guidelines. I recommend reading the techniques section.  A lot of sites that are ranking low, may have a issue that is being caused whether inadvertantly or not by items in this section.

     

    Jeremiah

    Monday, December 17, 2007 7:12 PM
  • So, fast forward to today...3 weeks since I originally posted this thread and guess whats changed???  Nothing.  Oh, correction, Google and Yahoo are continuing to index more and more of my site while LS continues to find to irnore my site, at most LS indexes one or two pages and then stop.

    To date, Google has over 22,000 pages of my site Indexed and each week that number continues to increase by about 2,000 more indexed pages.  Granted, my site has over 700,000 pages of ORIGINAL content so even google's pace is quite slow - but compared to LS its blazing fast.

    Google: 22,000+ (and growing)
    MSN LS: 73 (and slowly falling)

    MSNBot "was" hitting my site every two days...albeit it wasn't doing anything.  And now that stopped as of 12/24/07...MSNBot hasn't hit my since then...now 9 days ago.  Alternatively, Google is at my site every day, all day long.

    http://www.adcrow.com 

    Jeremiah, why don't you take a look at my site and tell me why LS isn't crawling it?


    Thursday, January 3, 2008 12:23 AM
  •  

    Same here, i'm also waiting when i'm being considered by msn, while site is already indexed by G and Y
    Thursday, January 3, 2008 9:16 AM
  •  $hez wrote:

     

    Same here, i'm also waiting when i'm being considered by msn, while site is already indexed by G and Y


    You guys just don't get it.  Having Google and Yahoo all over your site means nothing to MS and Live Search.  It has been pointed out over and over again by Jeremiah.  Live Search determines if your site is worthy.  It will not be pressured by those other search enginines to index your site.   They only control 80% of the search market.  What do they know?

    You must provide better content or have more inbound links.  Or there is another problem with your site.  It is not a LS issue.  Carry on and thanks for taking the time to write.

    Thursday, January 3, 2008 2:04 PM
  • It sounds to me like, based on the last comment, that MSN doesn't care a whole lot about indexing a whole lot of content out there. I have been reading all sorts of info on how to make my site 'desirable' for MSN and it sounds like an insurmountable task for the average neophyte web guy like me. There are literally thousands of pointers to my site out there and I'm still not getting indexed by MSN. I give up and I'll stick with working with Google which is somewhat more forgiving. I can only spend so much time fighting wind mills.


    RC

    Tuesday, April 7, 2009 6:24 PM