Web host doesn't have robots.txt help!! RRS feed

  • Question

  • Please help,


    My site is not being indexed and I think it may be because I  (my canned webhost) does not use robots.txt files.


    Therefore I cannot validate it and am wondering if there is any way to get around this.


    please advise.



    Wednesday, June 11, 2008 1:06 AM


  • what is your domain name and how long has it been active?





    Monday, June 16, 2008 7:34 PM

All replies

  • what is your domain name and how long has it been active?





    Monday, June 16, 2008 7:34 PM
  • Hi Brett,


    My domain name is www.creativethreadarts.com.


    It has been active for approximately 2 years.

    Sunday, June 22, 2008 5:47 PM
  • Do you develop and maintain the website you’re self, and upload it to the server?


    I find it very weird that you don’t have the possibility to use robots.txt file, what about sitemap and an authentication file for the ownership of you’re website. I mean, all you have to do is place it in the root directive.


    Tuesday, July 1, 2008 10:59 AM
  • As it says above I have a 'canned host'.  I don't even know what a root directive is.  I can only add from the pre determined template.  The host I am using allowed me to sitemap with google but my question is do I have to add another page that just says robots.txt? or how can I get that in there.


    I can add my own meta tags, keywords etc and add to the Head .  Where should I put robots.txt and is that the only thing I put in there?





    Thursday, July 3, 2008 12:00 PM

    The robot.txt file is used to give information to the crawlers. It’s a single text sheet made in notepad. This is very often used to tell the crawlers what they don’t want them to index (this can be sensitive materials, images++). If you don’t have anything to “hide”, you can just forget about the robots.txt file. You can read about it here: http://www.robotstxt.org/robotstxt.html


    For the validation you should use the meta tag validation, this should be no problem and has absolutely nothing to do with the robots.txt file.


    If you are talking about the validation of robots.txt tool in admin account, this is only meant to be a tool to check the text in the robots.txt file before you upload it, so you don’t get it wrong.


    You can try to copy/past what’s between the dotted line and you will see that it just validates the text you enter to let you know if it’s correct or not (it has nothing to do with what’s on you’re server):


    User-agent: *



    Sitemap: http://www.dinstartside.no/sitemap.xml


    My advice would be:
    1. don’t give the robots.txt any attention, you will do fine without it.

    2. Spend some time learning SEO, remember, the “title” should not be more than 80 characters, you have 134. Make sure that the keywords are present at the website All this is more important than the robots.txt file...
    You can use this validation to get a better understanding:


    Plz let me know if any is unclear.


    Best regards.

    Wednesday, July 9, 2008 11:56 AM