Crawling/Indexing Feedback and Discussion Forum

Our forums have moved and this forum is locked. Please join the new webmaster community!

Mon, 15 Jun 2009 10:46:28 Z

Our forums have moved and this forum is locked. Please join the new webmaster community at http://www.bing.com/community/forums/default.aspx?GroupID=11

Thanks-
Alicia

Program Manager, Server and Tools Online Operations Team

My site not indexing by bing

Thu, 04 Jun 2009 11:07:37 Z

Hallo! My site http://buy-generic-pills.com not indexed now by bing. Before my site always well indexed, and in the last time disappeared from delivery. It is showing as not blocked in the webmaster central but all the site pages are not indexed. What happened? Please return my site to the index.
With kind regards, George.

MSN only indexing 6 pages

Fri, 08 May 2009 19:54:19 Z

Why is msn so bad at indexing sites, I have been using webmaster tools for some time now and over the past month the most of the pages at www.eandhbaxendale.com have vanished from live search, and half of the pages that are left in the index don't even exist any more. Google and yahoo on the overhand are completely different they both have indexed more than 90 of the 130 pages submitted using a sitemap.

According to webmaster tools there are no crawl issues, i have used the robots.txt validator which reports no problems. It's now that bad that I have had to replace the live site search box on the site with a google custom search box, so our visitors searches return results.

MSN not indexing or crawling my site

Thu, 12 Mar 2009 03:33:21 Z

I have submitted my site to webmasters and everything, but nothing. I only see the non-www version of my homepage which should be the www version. I dont see msnbot crawling my site either.

Please can someone look at it. Am I banned, or am I not following some rule I I overlooked? Please someone. My site is:

http://www.premierresumeandwriting.com

How can I speed up the crawling-speed of msnbot

Wed, 10 Jun 2009 08:47:55 Z

Hi,

we have a big site. Currently we have 2,4 Million pages in Google index (+30k pages/day). Google spiders about 200k pages/day. MSNBot only spiders about 500 pages/day. We only have 1,4k pages in bing index. How can we speed up msnbot? Why is it so slow? We do not have crawl-delay set in robots.txt

Need Help, Home Page On First Page of Live.com Search Results then only the home page disappears

Tue, 14 Apr 2009 15:05:50 Z

RE Site: www.movingdayboxes.com

Hello, my site was on the first page, typically in the top five for search term: moving boxes on live.com
However in the last few days the page has totally disappeared from ranking. In the webmaster tools I see all my pages, but the home page has simply disappeared from live search engine results. Again everything looks good in webmaster tools including backlinks, score, and crawls.

Please let me know what you suggest.

Thanks,
Matt

who can tell me how the live search crawle the external links ?

Wed, 03 Jun 2009 15:19:37 Z

who can tell me how the live search crawle the external links ? my site is

http://www.itourbeijing.com

http://www.itourstibet.com

http://www.itourschina.com

how can i choose the key words for my site www.itourbeijing.com www.itourschina.com www.itourstibet.com

Wed, 03 Jun 2009 15:11:12 Z

who can tell me how to choose the key words for my tour website?

www.itourbeijing.com

www.itourschina.com

www.itourstibet.com

Thanks a lot ....

why my site www.chinatravel.com can not be indexed by live search?

Thu, 13 Nov 2008 09:57:49 Z

Hello,

can anyone tell me why my site www.chinatravel.com can not be indexed by live search? I've submited many time and still find no results.

Thanks in advance.

David

Why is this site not indexed?

Thu, 30 Apr 2009 09:18:32 Z

I'm using the webmaster tools to analyze the site: http://www.apprendereinrete.it

If I search on MSN Live "site:http://www.apprendereinrete.it" I can see no result, even if it shows 36 results pages

What can be the problem?

Trying to figure out why my site(s) have old information and will not update.

Thu, 28 May 2009 18:42:00 Z

Hello,
I have 3 sites on one server. All are indexed correctly on Yahoo and Google. I believe the sites should be indexed better on MSN. All 3 of my sites have old text show up when I search in the following manner on live search site:fish--oil.com
It will shows old text "dpmsurf". This was a test page from a few months ago. Same thing happens with my other 2 sites www.krill--oil.com and www.omega--3.com. None of these return anything if I enter site:www.fish--oil.com (with www) into live search.
Why would live search show such old text? Why can't I get the MSNBot to update my site with current content? Please help,
Thanks,
Dan

Please help, msnbot is practically attacking my site - pages being recached every few seconds

Tue, 10 Feb 2009 11:44:58 Z

For some reason, msnbot has a fascination with my /video/ directory, and has been the cause of investigation for our site using so much bandwidth lately. Turns out, since Feb 1st, msnbot has requested the /videos/ page over 240,000 times, using over 25GB of bandwidth. I have changed robots.txt to disallow it to index the folder, but it still persists. Here is a clipping of our raw access log, for just the last 10 minutes.

http://www.rpgamers.net/accesslog.txt

If anyone can advise why it has such great interest in this page, and now to allow it to index it without continously flooding it, I would appreciate your recommendations.

Thank you.

Site disappeared after several year of high rankings!

Wed, 27 May 2009 09:09:50 Z

One of my sites http://www.noprescriptiondrugstore.com disappeared from the MSN/Live index after several years of high rankings. During this recession time the lost of traffic from MSN is killing my income. Please help me figure out what went wrong!? It is showing as not blocked in the webmaster central but all the site information disappeared. What happened?!?!?! :-(

http://www.mingloo.com not fully indexed

Tue, 26 May 2009 06:48:56 Z

Our website has over 800 unique pages but live.com has indexed only 60. The site is hosted on Yahoo Stores and is fully indexed on Yahoo... but for some reason msn is not picking up all the pages. How can I have the search engine crawl all the pages of my website?

Setting the Region & Canonicalization Issue for my Malta website

Mon, 25 May 2009 13:50:19 Z

I have 2 issues related to my Malta blog and will appreciate if Live moderators can look into them ...

(a) Region setting in my Webmaster Center has changed from 'us' to 'cc' just recently. At the same, I started to lose rankings in Live Search US when this happened too. I understand that Live's bot might now be looking at the domain name extension to set region, but this is not always a good indication. In fact my website's registrant/administrative details have always included my local details, i.e. Malta (MT Europe). As regarding hosting, it's been always on a US server, since 2002. How can I remove any association with 'cc' without changing domain name of course?

(b) Homepage has always been showing as www(dot)malta.cc in serps, although I changed it to just malta.cc over a year ago now. A 301 redirect was made back then and whilst all internal urls are correctly listed, the homepage is still to-date displayed with 'www'. Is there a reason why the homepage is not displayed as malta.cc only?

Thanks in anticipation for your feedback.

Windjobs.org ~ Meta info not being read

Fri, 26 Dec 2008 16:32:50 Z

Hi,

My site's (Wind Jobs) meta info is not being read by live.com, I've resubmitted the site to twice in live.com's "Submit your site's URL" Is there something more I need to do?

Sincerely,
David

Compressed Sitemap Index file?

Fri, 22 May 2009 16:21:09 Z

Our site currently has the sitemap index located in the robots.txt, it is also added to the webmaster tools.   It's been about 6 months and not many pages have been picked up. (http://www.preamped.com/PA_SiteMapIndex.XML.gz)

Does the crawler support compressed site map files? Is that the issue?   Do sitemaps really need to be named "Sitemap.XML" even if the changed name is in the robots.txt?

Having 450 indexed pages with a sitemap of over 2 million is pretty sad after 6 months.    We just don't know what to try next.

Thanks,
Ryan - http://www.preamped.com

why does my site disappear to within MSN search?

Thu, 14 May 2009 13:40:35 Z

Saturday, May 02, 2009 9:36 PM

0 votes

My site seems to just no longer exist in MSN searches although my favicon link shows up and a link from sitepoint's forums.
The site is caswell chat.co .uk
Any help would be welcomed very much. http://www.caswellchat.co.uk/index.shtml

Everything was fine a few months ago if that is any help.

Edited fish090 Wednesday, May 06, 2009 3:32 PM
Edited fish090 14 hours 55 minutes ago

Site not indexed, but have sitemap.xml, robots.txt and are included in Google and Yahoo index

Mon, 04 May 2009 11:47:46 Z

Hi.

I created a site and use the webmaster tools on Live.com. I created a valid sitemap.xml and robots.txt file and entered the sitemap.xml location in the webmaster tools.

Since Google and Yahoo visit my site on a daily basis, I'd expect Live.com to do the same, but still no luck.

How can I make sure that my website is included in the Live.com index as well?

URL is http://www.bjornoya.nl

Thanks for your help.

Orginator of Content Getting Penalized for Dupe

Wed, 20 May 2009 21:07:29 Z

Our page, http://www.footballfanatics.com/MLB is being punished for dupe content, it dropped from the first page to page 3 a couple of days ago and i tracked it down to this.

2 sites ripped our meta title, description keyword and content directly.
http://search.live.com/results.aspx?q=%22MLB+Shop%2C+MLB+Apparel%2C+Merchandise%2C+Gear%2C+Clothing%2C+Accessories%2C+Gifts%22&form=QBRE

We contacted the one of the offenders and they promply changed their content, the other however has not changed anything. I changed my title tags in hopes that the spider will hit the page and remove the penalty.

It just seems weird that the Orginiator of content also gets penalized in this situation

Urgent! Why isn't a certain page at my website indexed at MSN?

Thu, 21 May 2009 13:18:51 Z

Hello friends, I run a blog on http://howtogetridofblackheadsfast.com . I noticed that an important news page in my site, http://howtogetridofblackheadsfast.com/2009/05/birth-video-leicester-watch-this-brutal-birth-video-from-leicester/, has not been indexed by MSN. Since this is a popular news story, I am losing quite a bit of traffic I otherwise would have from MSN search.

What can I do in such a case? How do I notify MSN so that it crawls my page again? Thanks a lot.

My site has disappeared from Live search results yet still has a 3/5 rating in webmaster tools

Wed, 20 May 2009 21:42:59 Z

Ok... Curiously I went to live and did a search for my domain name yourhomeprofile.com, and lo and behold, no results.

Google, and Yahoo seem to crawl my site fine. Google has 147 pages indexed and I get 80% of my traffic from them.

I was listed just fine on Live up until about two weeks ago. What's the problem?

I have been trying to get my site indexed for about 2 months now without any luck...Please Help

Wed, 06 May 2009 20:56:18 Z

I have been trying to get MSN to index my site for about 2 months and still no luck. All of the other search engines have indexed the site ( http://veyronbugatti.com ). Can anyone please help.

Thanks in advance.

Andrew

Bad header from MSN search

Mon, 04 May 2009 14:08:45 Z

Severeal requests are coming from 65.55.209.155 with an invalid header.
Usually, the "From:" header contains "msnbot(at)microsoft.com", but some requests contained "If-Modified-Since: Tue, 21 Apr 2009 08:35:20 GMT", like if the actual string was missing (and the new line).
This is blocked by our Web application firewall :-(

Is this a known bug? Can we expect this to be fixed soon?

Indexing issue

Wed, 15 Apr 2009 00:31:27 Z

Hello,

My site has so far just had the homepage of my shop indexed. It has been index by both yahoo and google with relatively high ranking. It has been several months since the site has been up. I did a proper 301 redirect from my old site, and all the other search engines have picked up the new site.

Can someone please help me understand what i'm doing wrong with msn live. FYI i have subscribed to the webmaster tools and have the sitemap submited. I also have proper meta tags.

my site is http://www.breathalyzeralcoholtester.com

My blogspot blog scrawled by msnbot but only index my blog url

Mon, 13 Apr 2009 12:40:47 Z

HI Experts,
I have a blog http://shawpnendu.blogspot.com which i submitted in live. Today i saw that msnbot scrawleed my blog but index only one page(only my blog home page). On the otherhand google indexed almost all of my pages. My question is why msbot index only one. Could you please answer my question. I have also submitted sitemap which i used in google.

THANKS IN ADVANCE.

"Try one of these related suggestions"

Tue, 12 May 2009 19:51:59 Z

I'm curious as to why my website would not show up in the SERPs when entering the query for my full URL, www.woodsonjoneschevrolet.com. When conducting this search Live.com returns this...

We did not find any results for www.woodsonjoneschevrolet.com.

Try one of these related suggestions

My site has 81 pages indexed and ranks well for many terms. What would be the cause of being left out of SERPs for this term and even more alarming, why is MSN suggesting my competitors?? Thanks for any insight you can provide.

My website was last crawled in February

Sat, 02 May 2009 12:19:06 Z

Hi,

I have submitted my site cart.all4sourcing.com since last February. I always re-submit the updated sitemap every week through Webmaster Center. Today, I just realize that my site is only last crawled in 2-8-2009. The "Web Page" in "Crawl Issues" tab is "No data is available.". What is the problem? What else can I do to solve it.

Thanks.

Website not indexed yet?

Tue, 19 May 2009 08:32:28 Z

Hi there,

Sorry if this is a common question, but I've been unable to get my website http://www.scotiasystems.com indexed? The website has been indexed by Google for a couple of months now - but I still can't see any results here?

I've had a look at the tips here in the forums, but all appears fine with the site?

Can you please help?

Thanks,

Nick

Disallowed pages being indexed and sitemap being ignored

Thu, 14 May 2009 08:20:54 Z

My website's been up since toward the end of last year: www.eraofdata.com

It gets regularly crawled and indexed by some other search engines I don't need to mention.

I've been looking at why @ 97% of my search engine referrals are from one particular engine and 0% come from live search for a couple of months, but haven't been able to figure it out, hence this post (the last significant update was in March although I've made some tweaks about a month ago).

According to webmaster tools, my root index page is indexed and one other 'page': a gzipped sitemap file.

1. I've really got no idea why such a file is getting crawled, so I excluded it in robots.txt a few weeks back, yet it's still getting crawled, but only by live search.

2. I can't establish why live search is only indexing the root page and not any other content. This content is explicitly referred to in the sitemap file and the other content referenced in the sitemap is getting picked up by other search engines.

For reasons related to the content of that site it would be appropriate if live search could crawl all the pages listed in the sitemap from time to time.

ajmer dhariwal || eraofdata.com

What does MS use as a User-agent to fetch RSS feeds ?

Tue, 28 Apr 2009 20:51:44 Z

What does MS use as a User-agent to fetch RSS feeds ?

yahoo seems to use YahooFeedSeeker
google seems to use Feedfetcher-Google

what do you use?

Thanks

Website Not Being Indexed!

Tue, 14 Apr 2009 10:06:20 Z

Hi,

I am trying to get my website http://www.usbflash.co.uk indexed by msn / live search and for some reason it will not show up in search results. It has been a few months now and google and other search engines has indexed lots of pages but nothing from here yet!?

I am signed up for the webmaster tools and have submitted the sitemap but still nothing, all that it says in the data is N/A.

Please can you help me to understand why this is the case?

Thanks

Matt

MSN not indexing yet.

Thu, 07 May 2009 00:29:08 Z

I have followed all given instructions and understand that MSN is slow for new sites. Can you let me know if there is an issue or give the bot a push? Thanks. www.youserguide.com

Page being indexed includes a dynamic promo code in the URL

Fri, 24 Apr 2009 18:17:54 Z

I searched for a keyphrase using Live Search. The page that is listed for my site included a promo code in the URL. I don't understand how or why you are doing this as this promo code is only used by our Yahoo Sponsored Search Ads

Robots.txt ignored by msnbot/2.0b

Sat, 16 May 2009 16:52:15 Z

On www.geograph.org.uk we are noticing that msn bot doesnt seem to honour robots.txt .

extract from robots.txt, of relevent bit

User-agent: *

Disallow: /mapbrowse.php

Disallow: /map/

Disallow: /mapper/

Disallow: /maplarge.php

Disallow: /mapprint.php

....

Disallow: /editimage.php

Disallow: /browse.php

...

etc

As far as can see the file is fine, and these commands have been there for a while.

Some example lines from the access log that shouldnt be crawled...

65.55.106.191 - 0 [16/May/2009:07:32:11 +0100] "GET /maplarge.php?t=tolJ5ojXXJ0ojXJFojXXJfoMObJqoVOXJL5405oZZbXNbt8tOXhwZu4 HTTP/1.1" 503 5307 "-" "msnbot/2.0b (+http://search.m

sn.com/msnbot.htm)"

65.55.106.191 - 0 [16/May/2009:07:33:30 +0100] "GET /maplarge.php?t=tolJ5ojXXJ0ojXJFojXXJfoMObJqoVOXJL5405oZZbXNbt8tOXhwZu4 HTTP/1.0" 200 37166 "-" "msnbot/2.0b (+http://search.

msn.com/msnbot.htm)"

65.55.106.236 - 0 [16/May/2009:07:58:03 +0100] "GET /browse.php?p=290852 HTTP/1.1" 200 6164 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.164 - 0 [16/May/2009:08:18:02 +0100] "GET /editimage.php?id=1082059 HTTP/1.1" 200 6063 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.147 - 0 [16/May/2009:08:18:32 +0100] "GET /browse.php?p=382785 HTTP/1.1" 200 6905 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.188 - 0 [16/May/2009:08:18:35 +0100] "GET /mapbrowse.php?t=tolJ5oOXXJ0oOXJFoOXXJfobXNJqoVjMJL5405oVMbNwlMbujNNZZuw&gridref_from=SJ9892 HTTP/1.1" 200 7495 "-" "msnbot/2

.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.207 - 0 [16/May/2009:08:32:07 +0100] "GET /browse.php?p=286331 HTTP/1.1" 200 6069 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.165 - 0 [16/May/2009:08:43:17 +0100] "GET /browse.php?p=283628 HTTP/1.1" 200 6036 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.218 - 0 [16/May/2009:09:05:01 +0100] "GET /reuse.php?id=857043 HTTP/1.1" 200 19990 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.115 - 0 [16/May/2009:13:01:31 +0100] "GET /browse.php?p=644008 HTTP/1.1" 200 5978 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.107 - 0 [16/May/2009:13:28:03 +0100] "GET /browse.php?p=404440 HTTP/1.1" 200 6038 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.241 - 0 [16/May/2009:14:06:33 +0100] "GET /ecard.php?image=643562 HTTP/1.1" 200 6061 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.140 - 0 [16/May/2009:14:28:30 +0100] "GET /browse.php?p=426977 HTTP/1.1" 200 5946 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.114 - 0 [16/May/2009:14:42:22 +0100] "GET /browse.php?p=512392 HTTP/1.1" 200 6134 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.230 - 0 [16/May/2009:15:18:32 +0100] "GET /mapbrowse.php?t=tolJ5oOXXJ0oOXJFoOXXJfobObJqoVjXJL5405o4lVXZwNtNwXjXaMu&gridref_from=SK4384 HTTP/1.1" 200 7609 "-" "msnbot/2

.0b (+http://search.msn.com/msnbot.htm)"

65.55.106.241 - 0 [16/May/2009:15:23:02 +0100] "GET /browse.php?p=154252 HTTP/1.1" 200 6780 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

(according to reverse DNS they look valid msn IPs)

We also get lots of hits from "msnbot/1.1" - which as far as can see obay robots.txt, eg this this is an example of a valid request

65.55.209.228 - 0 [16/May/2009:16:13:26 +0100] "GET /profile/3860?expand=1 HTTP/1.1" 200 39533 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"

My website is being crawled but not indexed

Fri, 01 May 2009 13:11:02 Z

Hi there,

I made a new webside (http://www.webdesign-loos.de) 3-4 weeks ago.
The side is being crawler regularly by the msnbot.
But I see not datas in the webmaster tools and the side is not beeing indexed on live search.

The site is being indexed by G and Y and ranked well.

May you can help me out?

Thx,

Alex

What's up with getting indexed??

Fri, 15 May 2009 14:46:55 Z

I have tried to get my site http://www.bookoosports.com indexed, finally after months my home page is indexed but none of the other 146 pages are showing up, the last time it was crawled was 4/27/2009. Why is it taking so long just to get crawled let alone getting indexed?? Google and Yahoo had no problem indexing my whole site in a matter of a very few weeks, but MSN is taking months just to get a single page indexed let alone the rest of the site! What's up???

Automated feedback

Fri, 15 May 2009 13:17:23 Z

A while ago, while updating with Microsoft.com I was asked if I'd agree to have my computer comunicate silently with Microdoft and send back statistics and feedback.

How do I stop my computer from doing that now?

Some sites indexed, others not

Wed, 06 May 2009 10:49:20 Z

Hi there
I have around 20 sites listed in the MSN webmaster site list and some get indexed and some not.
They all either have a robots.txt files which accepts all bots "User-agent: *" or don't have one at all.
They all have sitemaps which have been submitted some time ago (several months ago)
None are on tricky or adult subjects
All have been indexed by other search engines
They have all been verified via meta tag
but.......
only some are indexed by MSN
These are the ones not making the list
www.ecommcampaign.co.uk x
www.breakthrough-supplement.co.uk
www.leedsvalentinesfair.co.uk
www.marcoberon-magictricks.co.uk x
www.bourne-web-design.co.uk x
www.linkpedia.co.uk
www.ecommweb.co.uk

I have looked and can't find a correlation between the included and excluded sites.

grateful for insights

regards
Gary

Why my website has not being indexed? -Part 2-

Fri, 15 May 2009 08:54:28 Z

Hi Brett,

I've asked you to re-check my website submission ( www.projects-abroad.se ) in the last post ( http://social.microsoft.com/Forums/en-US/webmastercrawling/thread/6fa247c8-c11c-44fb-a255-215a38459ff6 ) and I didn't received an answer.

I tried to search for my website again today on live.com organic and no result came out.

What is to be done in this case?
Thank you in advance,
ChiefRA