12. prosince 2007 19:09Hello all,
I see my site is being accessed by the MSNbot but its not effectively crawling my site. G**gle is crawling it daily and is indexing thousands of pages a day. I have checked my robots.txt / sitemap.xml files and everything checks out. Even used some public analyzers to test crawlability from sever major search engines (yahoo/msn/google) and everything checks out. Even 100% html validation by w3.org.
Yet with all this MSN barely notices my site. And the site is not being blocked.
Is there something I am missing, need to correct? My site has over 700,000 articles but MSN barely gets past the homepage.
14. prosince 2007 13:46Welcome to the club and I wish I could help you. Take a look at the list of problems and what you describe has been asked in the past. Just look at my thread from realdealtools. We are at a lost of why we are not being indexed. Live Search team has really not provide any real guidence or answers.
14. prosince 2007 15:54
Maybe this is what they meant when the CEO said they were going to kill google? They figured they couldn't beat them in search technology so they'd save webot crawling costs by only having it index one page?
I sent out a certified letter this morning to a higher up at Microsoft asking for an explanation and pointing him to this site. I don't expect much of a response but who knows. It's obvious the Live team knows the problem and just has no answers.
Whether or not a site has quality inbound links is irrelevant. There are plenty of excellent websites with almost no links for civic organizations and charities. The inability of the livebot to even follow a basic link from the homepage stems from bad code, not a bad site.
When you look at our dilemna, a search of promomanagers on live search yields a vist to a club jobs site for a job as promo manager. There are then 40 indexed pages dealing with our company (but not our site) and then some other gibberish. Is it any wonder Live's traffic is a fraction of Google when you can't even find a website by typing in the website page?
We shut down whatever ads we were running this morning, this is just silly that they can't index a page properly. Why reward them with your advertising dollars?
17. prosince 2007 19:12
There are a lot of reasons why sites don't get indexed. One of the big challenges for webmasters is ensuring that the sites that link to them are quality sites. As a webmaster, you need to ensure not only that there is great content on your site, but that you get the right kind of people and sites linking to you.
So one of the best ways to ensure that MSNBot finds your website is to include valuable content that other site owners want to link to. MSNBot crawls billions of web pages, not every page is indexed.
For a site to be indexed, it must meet specific standards for content, design, and technical implementation. Ensure that your site follows our indexing guidelines. I recommend reading the techniques section. A lot of sites that are ranking low, may have a issue that is being caused whether inadvertantly or not by items in this section.
3. ledna 2008 0:23So, fast forward to today...3 weeks since I originally posted this thread and guess whats changed??? Nothing. Oh, correction, Google and Yahoo are continuing to index more and more of my site while LS continues to find to irnore my site, at most LS indexes one or two pages and then stop.
To date, Google has over 22,000 pages of my site Indexed and each week that number continues to increase by about 2,000 more indexed pages. Granted, my site has over 700,000 pages of ORIGINAL content so even google's pace is quite slow - but compared to LS its blazing fast.
Google: 22,000+ (and growing)
MSN LS: 73 (and slowly falling)
MSNBot "was" hitting my site every two days...albeit it wasn't doing anything. And now that stopped as of 12/24/07...MSNBot hasn't hit my since then...now 9 days ago. Alternatively, Google is at my site every day, all day long.
Jeremiah, why don't you take a look at my site and tell me why LS isn't crawling it?
3. ledna 2008 9:16
3. ledna 2008 14:04
You guys just don't get it. Having Google and Yahoo all over your site means nothing to MS and Live Search. It has been pointed out over and over again by Jeremiah. Live Search determines if your site is worthy. It will not be pressured by those other search enginines to index your site. They only control 80% of the search market. What do they know?
You must provide better content or have more inbound links. Or there is another problem with your site. It is not a LS issue. Carry on and thanks for taking the time to write.
7. dubna 2009 18:24
It sounds to me like, based on the last comment, that MSN doesn't care a whole lot about indexing a whole lot of content out there. I have been reading all sorts of info on how to make my site 'desirable' for MSN and it sounds like an insurmountable task for the average neophyte web guy like me. There are literally thousands of pointers to my site out there and I'm still not getting indexed by MSN. I give up and I'll stick with working with Google which is somewhat more forgiving. I can only spend so much time fighting wind mills.