locked
problems problems problems RRS feed

  • Question

  •  

    Hi,

    I have a few sites that I cannot get recognized on MSN?  They do well in Yahoo and Google, but I cannot seem to do anything right for MSN.  www.dfwchevydealer.com and www.metroplexchevrolet.com.  I emailed support from Microsoft, and they said that my sites looked fine.  They did say they are having algorhythm problems.  Okay, so why don't they show up say when you type in dfw chevy dealer or metroplex chevrolet in the search bar?  I am number 2 in google with www.metroplexchevrolet.com and anywhere from 3 to 5 in Yahoo with www.dfwchevydealer.com.  Would anyone have the time to take a look at my site and tell me if there is anything wrong?  I really like MSN and use them as my home page, but I do not have any luck at all with getting my sites recognized by them.  Please help.

    One other thing that I noticed is with MSN, is sites that employ cloaking or redirecting their sites seem to do fine.  I thought cloaking was frowned upon.  I am a rookie please advise on how to work with MSN.
    Thanks,
    Jason

    Wednesday, March 5, 2008 8:07 PM

Answers

  • The Live robots.txt validator works differently from Google's, and the Live explanation is misleading.

     

    In the Live robots.txt validator you enter the body of the robots.txt file into the top textarea box, in this case: 

     

    Robots.txt validator:
    Sitemap: http://www.dfwchevydealer.com/sitemap.xml

    User-agent
    : *
    Disallow:

     

     

    [Validate]

     

    Validation results:

    No problems found.

     

     

    Monday, March 10, 2008 5:41 PM

All replies

  • This can happen if your sitemaps are not actually saved in UTF-8 format, or if they have funky line endings.  It's no issue for Google or Yahoo but seems to cause issues with Live.  In typical MS form, IIS doesn't pass charset by default, all versions of VS .Net are infamous for being totally retarded about saving UTF-8 declared xml out as ANSII, and of course Live can't seem to figure out what to do when they get an ANSII file which claims to be UTF-8.

     

    In looking at your sitemaps, they seem to be correct, but I am unsure how to be certain remotely, so thought it would be worth double checking and resubmitting.

     

    One other thing that I noticed is with MSN, is sites that employ cloaking or redirecting their sites seem to do fine. 

    Just like everything in life, cheaters may seem to be winning in the short term, but long term, they rarely come out on top.

    Thursday, March 6, 2008 10:16 AM
  • thanks for the reply.  I have a site map for www.dfwchevydealer.com, I do not have one at www.metroplexchevrolet.com yet.  I am kinda running an experiment.  I have never had a site map until recently, and my sites have done well in google and yahoo.  I feel like I am crossing my t's and dotting my i's, but I just can't be sure.  What is really a big pain is that you have to wait months to see if the changes you make help or do not help your rankings.  My problems with MSN is that my sites do not show up at all period.  I am even using the webmaster tools.  I have had my sites for about 7 or 8 month, and they do well elsewhere.  I don't know if there is anything I can do at all with MSN.  They should at least show up.  Especially when I put quotations around it www.dfwchevydealer.com  I am frustrated and am begining to wonder if I am just wasting my time. 

     

    Thursday, March 6, 2008 7:32 PM
  • Initially, using a sitemap my clients site was getting indexed very quickly and accurately.  Then I had made some changes to the sitemap, and when I resubmitted them I had an issue where only the home page was listed, and the domain rank plummeted.  This went on for weeks, then I resaved out the sitemap file in UTF-8, resubmitted it, and very suddenly (4 hours or so), the domain rank went back to 5 bars, and we once again had 35-40 pages indexed.  Over the past couple days we are having issues again, where the domain rank dropped to 2 bars, and only 7 pages are indexed;  I had made changes to my sitemaps so tried several ways to be sure it was saved out UTF-8 and none of them seemed to work this time.  It now seems possible that it was all a coincidence before, but that still feels unlikely, since the timing seemed way too perfect to be coincidence.
     
    In my experience, Live can react extremely fast when it "likes" everything, unlike Google and Yahoo.  Google and Yahoo are slow, steady, and very consistent; Live can be 100x as fast to react, but it's highly variable and seems super finicky.  This seems to be compounded by sitemaps.  No sitemap is much better than a "bad" sitemap, but it takes a "good" sitemap for it to react quickly.  I am not sure why your site without a sitemap only has the home page listed after all this time.

     

    I would recommend adding a robots.txt for your sites, with sitemap link in the site with a sitemap, but I have no idea if that will fix anything.  I know it seems a little silly since Live is currently ignoring robots.txt, but just because it's not correctly obeying the robots.txt doesn't mean that there couldn't be some wierd indexing bug causing issues with sites that don't have them.  I know it's a total shot in the dark, but might be worth 5 minutes just to see if it helps.

    Thursday, March 6, 2008 11:41 PM
  • I appreciate your time and help greatly.  I hope I can return the favor some time.

     This site map and robots.txt file thing has got me a little confused.  I am having a hard time finding out how to make them correctly.  I used a site that generates a site map for my website. 

     www.xml-sitemaps.com  is the website that will generate the sitemap.  I don't know how good they are??? 

    Here is the link to my sitemap generated for the dfwchevydealer website:       http://www.dfwchevydealer.com/sitemap.xml

     

    If you get time would you take a look at it and let me know what you think?  I really don't know what I am really looking at yet.  I am still learning web design and this sitemap and robots.txt thing has kinda halted my progress.

     

     To me this sitemap does not seem to convey all my links,    My home page is full of links to my inventory.  You can click on any image and most text.

     
    Anyhow, you have helped me shed some light on my problems and I appreciate your help.

    Friday, March 7, 2008 3:39 AM
  • I just hope I am able to help.  www.xml-sitemaps.com should generate a correct sitemap, I have looked at yours several times, and it "seems" like it is saved out as UTF-8, but to be honest, I am not quite sure how to tell remotely.   One thing that might help, is to know what sort of editor you use.  It looks like you probably just published the sitemap it produced, which might have already been saved out as UTF-8.  I  really wish I could be more definitive here, but frankly I am unsure about how to verify it specifically, and not from lack of trying.  I have not been able to figure out a way to test remotely if a file is saved as UTF-8 or ASCII, only if it is readable by the browser, if the webserver declares text/xml as UTF-8 by default, and if the XML declaration is UTF-8 (the latter is, unfortunately, almost totally meaningless).  I guess to make a long story short, if you just used xml-sitemaps, didn't edit the file at all, and published, I think you should be ok.

     

    You are correct that the sitemap doesn't follow all of "your" links, it struggles with the param based urls under a different domain in much the same way a robot might.  That seems to be due to the fact that all the urls to your inventory (or any other somewhat specific data) are directed to http://www.cdmdata.com/, basically any pages not associated with your domain are not considered as "yours".

     

    robots.txt is fairly easy, just create, save, and publish the following text files:
    http://www.dfwchevydealer.com/robots.txt

    Sitemap: http://www.dfwchevydealer.com/sitemap.xml

    User-agent: *
    Disallow:

    http://www.metroplexchevrolet.com/robots.txt

    User-agent: *
    Disallow:

     

    You can find more details about robots.txt here, but basically you are saying that regardless of the bot, you don't wish there to be any restriction on either files, or directories while crawling on your site.  Normally, if you want everything crawled, you shouldn't have to specify anything or even have a robots.txt (as normally robots.txt is to prevent hidden areas of your site from being crawled), but it is possible that since Live ignores the prevention normally enabled by robots.txt, it's theoretically possible that it actually requires explicit permission.
    Friday, March 7, 2008 9:40 AM
  • I am using Dreamweaver CS3 as my HTML editor.  You are correct I just saved the sitemap created by xml-sitemaps.com.  As for being saved as UTF-8, I am not sure I will have to look when I get home from my real job.

     

      When I look at the actual sitemap it has this at the top <?xml version="1.0" encoding="UTF-8" ?> .   Also, do I need to create a sitemap for each page, and a robots.txt file for each page?  My site has a total of 4 pages.  Will I need a total of 12 to include sitemaps and robots.txt files? 

    So the robots.txt example you provided lets them crawl my entire site?  I don't really care to hide anything, so I guess it is fine to crawl the whole thing.  I just want to be sure I don't exclude anything. 

    I just can't believe I cannot find my site no matter what I seem to do.  Maybe it is just not good enough yet.  I know that I have decent content and I have links from my high ranking sites to it. 

    Thanks again,

    P.S.  what are your websites?  Maybe I can put a couple of links in for you.

    Friday, March 7, 2008 4:40 PM
  •  jcameron wrote:
    When I look at the actual sitemap it has this at the top <?xml version="1.0" encoding="UTF-8" ?> .

    Unfortunately the xml declaration doesn't really help us much, it does need to match the proper encoding of the actual page, but it doesn't ensure that the OS saves the file as UTF-8, the server serves the page as UTF-8,  or that the browser (or bot) interperets/displays it as UTF-8.

     

     jcameron wrote:
    Also, do I need to create a sitemap for each page, and a robots.txt file for each page?  My site has a total of 4 pages.  Will I need a total of 12 to include sitemaps and robots.txt files?

     

    Nope, just one sitemap and robots.txt file for each of your two domains, there are some cases where you might need more than one sitemap or robots.txt, but they don't apply to your cases.
     

     

     jcameron wrote:
    So the robots.txt example you provided lets them crawl my entire site?  I don't really care to hide anything, so I guess it is fine to crawl the whole thing.  I just want to be sure I don't exclude anything.

    yep.

     jcameron wrote:
    I just can't believe I cannot find my site no matter what I seem to do.  Maybe it is just not good enough yet.  I know that I have decent content and I have links from my high ranking sites to it.

    It's nothing to do with being good enough, it's just a matter of figuring out what is causing an issue between your sites and MSNBot, then fixing it.  Unfortunately, the folks who can actually look at the real data, and find out the answers for certain, rarely are able to help us out.  Sometimes this is because to do so they would have to reveal information that could be exploited by black hat, sometimes its because they don't know what is wrong (if they did, in many cases they would probably just fix the issue on the bot side), but mostly, I think it's because they simply dont have time.  On small teams, it's tough to balance support and development.  The more time they spend on the boards, the less time they have to be working on the product, but if they lose touch with the customer, the product loses value.


     jcameron wrote:
    Thanks again,

    P.S.  what are your websites?  Maybe I can put a couple of links in for you.

    All I have really done lately is http://www.sylvanwaychristian.org/ which is fairly location based (like your sites) and in a totally different area than you, so a link would not help.  Appreciate you thinking of me though. 
    Friday, March 7, 2008 7:56 PM
  • Thanks for all your help.  I am going to create my robots.txt file and see if that will help.  I will let you know.

    Sunday, March 9, 2008 4:12 PM
  • MSN validator tool keeps giving me this error when I copied and pasted the code and save it as my robots.txt file.  Maybe I am not using the tool right.  Any ideas?

     

    Line #1: www.dfwchevydealer.com
    Error: 'user-agent' -  should be the first tag.
    **************************************************
    Warning: 'sitemap' - tag isn't specified.
    **************************************************

    Sunday, March 9, 2008 4:27 PM
  •  

    Sorry, that was two separate files.  One for www.dfwchevydealer.com, and one for www.metroplexchevrolet.com.

     

    The robots.txt file for http://www.dfwchevydealer.com/ should contain only:

     


    The robots.txt file for www.metroplexchevrolet.com should contain only:

    User-agent: *
    Disallow:

     

     

     

    Sunday, March 9, 2008 8:45 PM
  • Oh, also, in the MSN robots.txt validator you will get "Warning: 'sitemap' - tag isn't specified." for www.metroplexchevrolet.com , but since it doesn't have a sitemap, you can't specify one.
    Sunday, March 9, 2008 8:49 PM
  • Thanks for sticking with me.  I cannot believe this little file is giving me such a headache.  I tried it again with the new code and I keep getting this error.  I tried doing what it said and still got the error???  I switched it back and did some research on the web and still have not come to get a, "you did it right" answer from the MSN validator tool. Sad

    Line #1: www.dfwchevydealer.com
    Error: 'user-agent' -  should be the first tag.

    Sorry for being such a pain.  I just cannot figure out what is wrong.

    Sunday, March 9, 2008 11:54 PM
  • The Live robots.txt validator works differently from Google's, and the Live explanation is misleading.

     

    In the Live robots.txt validator you enter the body of the robots.txt file into the top textarea box, in this case: 

     

    Robots.txt validator:
    Sitemap: http://www.dfwchevydealer.com/sitemap.xml

    User-agent
    : *
    Disallow:

     

     

    [Validate]

     

    Validation results:

    No problems found.

     

     

    Monday, March 10, 2008 5:41 PM
  • Okay.  That is how that works?  Dumb me, I  have been putting in my url to my home page.  Sheesh.  Too easy.  I can't believe that I did not think of that.  Thanks for all you've done.  Take Care,
    Jason

    Monday, March 10, 2008 11:39 PM