locked
how to stop like http://www.qurini.com:802007/ from crawling and live sreach RRS feed

  • Question

  • i have proplem on live that it crawl this bad link http://www.qurini.com:802007/ and show it the link to my web site
    and crawl the sub link from it as http://www.qurini.com:802007/vb/memberlist.php
    how i can stop there and let live bot crawl the right links

    Thursday, December 11, 2008 10:04 PM

Answers

  • Hi,

     

    Have you tried placing that URL in your robots.txt file?

    Wednesday, December 17, 2008 7:58 PM