locked
about robots.txt RRS feed

  • Question

  •  

    this is my robots.txt

     

    # Robots.txt file from http://www.topmarketcn.com
    # All robots will spider the domain
    User-agent: *
    Allow: /
    Sitemap:http://www.topmarketcn.com/sitemap.xml
    msn check up no problem
    google check up  have wrong at line: Sitemap:http://www.topmarketcn.com/sitemap.xml
    anyone can tell me why?TKS
    Thursday, April 24, 2008 2:40 AM

Answers

  • Hi,

     

    This is the correct method according to robotstxt.org to allow all robots access:

     

    User-agent: *
    Disallow:
    

     

     

    Currently there is very little to no support for "allow".

     

     

    Thanks,

     

    Brett

     

     

     

    Thursday, April 24, 2008 2:49 PM

All replies

  • Hi,

     

    This is the correct method according to robotstxt.org to allow all robots access:

     

    User-agent: *
    Disallow:
    

     

     

    Currently there is very little to no support for "allow".

     

     

    Thanks,

     

    Brett

     

     

     

    Thursday, April 24, 2008 2:49 PM
  • my robots is write like this:

     

    # Robots.txt file from http://www.topmarketcn.com
    # All robots will spider the domain
    User-agent: *
    Disallow:
    Sitemap:http://www.topmarketcn.com/sitemap.xml

     

    it is ok now?

    best regards

    Friday, April 25, 2008 6:43 AM