locked
Site not crawled since 9/25! RRS feed

  • Question

  • Comments regarding this to Microsoft's email help have not produced any meaningful answers, so can anyone answer this: Why has the site not been crawled since 9/25/2007?  The main URL has been submitted twice to Live Search, the site contains a validated robots.txt along with pertinent META statements and a XML sitemap.  Also, the language column in the profile shows "zz" instead of "en."  I'm not questioning the placement of the search results or the ranking (1 green box - ugh!) but I want to know why the crawler is balking at this site.  http://www.tileboutique.com.  The site is listed in the LS URL index so it is supposedly known to Live Search. And,  as with most of the other sites' documented problems in the forum, Google and Yahoo have absolutely no problems with this site.

     

    Five other sites that I host on the same server are crawled by Live Search without problem.  All have similar robots.txt and META specifications.

     

    Any help or resolution is certainly appreciated!

    Friday, January 4, 2008 4:54 PM

Answers

  • Cybervetures,

     

    I can't explain the difference between your other sites as you only listed the http://www.tileboutique.com.  I think your biggest challenge is the lack of any readable text on the homepage.  We offer the general advice in our indexing guidlines:

    In the visible page text, include words users might choose as search query terms to find the information on your site.

    • Limit all pages to a reasonable size. We recommend one topic per page. An HTML page with no pictures should be under 150 KB.
    • Make sure that each page is accessible by at least one static text link.
    • Don't put the text that you want indexed inside images. For example, if you want your company name or address to be indexed, make sure it is not displayed inside a company logo.
    • Add a site map. A site map helps MSNBot to find all of your pages. Links that are embedded in menus, list boxes, and similar elements are not accessible to web crawlers unless they appear in your site map.
    • Keep your site hierarchy fairly flat. That is, each page should only be from one to three clicks away from the home page.

    Also as you represent a physical store, you should consider listing your site in the search engine local directories.  This will help define for engines what your site is about and build credibility.

     

    ·         You can find live search’s directory located here:

    o   https://llc.local.live.com/ListingCenter.aspx

     

    I have mentioned it, in other posts, but I am always surfing to sites likeYelp and Angies list.  Getting your site listed there can build credibity as well.  

     

    To summarize this site has a couple of SEO tweaks to the design that could help you and I think a little bit of promotional leg work could do the rest.

     

    Best of luck and let me know how it goes.

     

    Jeremiah

    Friday, January 4, 2008 5:39 PM

All replies

  • Cybervetures,

     

    I can't explain the difference between your other sites as you only listed the http://www.tileboutique.com.  I think your biggest challenge is the lack of any readable text on the homepage.  We offer the general advice in our indexing guidlines:

    In the visible page text, include words users might choose as search query terms to find the information on your site.

    • Limit all pages to a reasonable size. We recommend one topic per page. An HTML page with no pictures should be under 150 KB.
    • Make sure that each page is accessible by at least one static text link.
    • Don't put the text that you want indexed inside images. For example, if you want your company name or address to be indexed, make sure it is not displayed inside a company logo.
    • Add a site map. A site map helps MSNBot to find all of your pages. Links that are embedded in menus, list boxes, and similar elements are not accessible to web crawlers unless they appear in your site map.
    • Keep your site hierarchy fairly flat. That is, each page should only be from one to three clicks away from the home page.

    Also as you represent a physical store, you should consider listing your site in the search engine local directories.  This will help define for engines what your site is about and build credibility.

     

    ·         You can find live search’s directory located here:

    o   https://llc.local.live.com/ListingCenter.aspx

     

    I have mentioned it, in other posts, but I am always surfing to sites likeYelp and Angies list.  Getting your site listed there can build credibity as well.  

     

    To summarize this site has a couple of SEO tweaks to the design that could help you and I think a little bit of promotional leg work could do the rest.

     

    Best of luck and let me know how it goes.

     

    Jeremiah

    Friday, January 4, 2008 5:39 PM
  • Maybe you missed it, but there is text navigation with anchors on the Tile Boutique main page (at the bottom) that links to second-tier main pages but these pages do not appear to be indexed or known to Live Search.  I should also mention that most of the site pages were duplicated from another site http://www.tilecollection.net whose pages are indexed and are returned in Live Search.  Tile Boutique happens to be one of the previous Tile Collection stores which was spun off into its own entity.  Unfortunately, Tile Collection appears to be offline at the moment but its pages are cached in Live Search.

     

    Based on your suggestion, I did add some descriptive text on the main page that will be invisible to the browser but can also be ready by MSNbot.  However, this text or the subsidiary pages (this is a flat site with a sitemap listed in robots.txt) won't be seen if the bot doesn't crawl the site which it hasn't done sine 9/25.

     

    Is the "last crawled" timestamp indicative of the last successful crawl attempt or does it represent when the bot visited the site, whether it crawled successfully or aborted? And can anyone answer what the significance of the language indicator "zz" is?

    Sunday, January 6, 2008 7:55 AM
  • I manage some websites, and one of them has not been crawled for more than 3 months (Kemang Village), what makes the crawler come back ? so how the crawler knows that the sites has been changed ?
    Tuesday, January 15, 2008 3:36 PM
  • Hi Jeremiah,

     

    I have a related question and NEED YOU ADVICE.

     

    Some of my confidential personal information came up on a Live Search through a recruiting portal. I had requested the portal guys to delete my information and they have done so. The issue is when I search my name on Live Search it is still

    showing the OLD PAGE in the CACHE. The portal webmaster says that MSN needs to delete the CACHE or recrawl his portal page in order to get the up to date information on Live Search.

     

    How can I get the CACHE deleted or request Live Search to recrawl the page (if at all).

     

    Thanks ahead for your advice,

    AA

     

    Wednesday, January 16, 2008 10:51 AM
  • AA MM,

     

    is this still an issue?

     

    Brett

     

    Monday, March 31, 2008 10:44 PM