No announcements
Found 124608 threads
-
0 VotesRobots.txt Validator, again
Hi, Your validator find now errors on a robots.txt as basic as this one : Code Block User-agent: ...
-
0 Votes
Need MSNbot to follow the robots.txt
robots.txt contains something like # go away ?Answered | 1 Replies | 5273 Views | Created by Jim3ELS - Thursday, May 7, 2009 2:35 PM | Last reply by Brett Yount - Thursday, May 7, 2009 5:07 PM -
0 Votes
MSNbot only crawling robots.txt
Made any changes to your robots.txt file (http and/or https)?Answered | 1 Replies | 4803 Views | Created by homberger - Wednesday, April 1, 2009 3:38 PM | Last reply by Brett Yount - Wednesday, April 8, 2009 9:25 PM -
0 Votes
MSNBot Not Hitting Robots.txt
Thanks Nicky, have added as suggested, although I think the issue is that MSNBot is not even trying to hit the robots.txt file, rather than mis-interpret whats in it.Answered | 9 Replies | 6112 Views | Created by Altnet_00 - Tuesday, December 9, 2008 9:10 AM | Last reply by Brett Yount - Tuesday, February 24, 2009 5:44 PM -
0 Votes
Error in Robots.txt Validator
Well, on the one hand, the robots.txt checker says it only checks for msnbot -- but on the other hand, I would suggest it report something less confrontational as "invalid ...Answered | 3 Replies | 4640 Views | Created by jwolthuis - Wednesday, November 21, 2007 12:45 AM | Last reply by free-survey - Wednesday, December 5, 2007 5:17 PM -
0 Votes
Robots.txt Vs Sitemap.xml
Hi, robots.txt and sitemaps are used for two different things.Answered | 1 Replies | 3754 Views | Created by kio2008 - Wednesday, April 9, 2008 3:03 PM | Last reply by Brett Yount - Friday, April 11, 2008 6:11 PM -
0 Votes
Sitemap url in robots.txt
Hi, i have read that its possible to put the sitemap url in robots.txt file. i have also found a post that mentioned to put the link and disallow bots. something like ...Answered | 1 Replies | 13474 Views | Created by Nelson Monteiro - Friday, January 2, 2009 5:55 PM | Last reply by Brett Yount - Friday, January 2, 2009 8:04 PM -
0 Votes
robots.txt with multiple User-Agent: lines not recognized by MSFT robots.txt checker
User-agent: Googlebot-Image User-agent: Yahoo-MMCrawler User-agent: MSNBot-Media Allow: /photos/ Disallow: ...Answered | 1 Replies | 3896 Views | Created by loupiote - Sunday, December 14, 2008 6:17 AM | Last reply by Brett Yount - Tuesday, December 30, 2008 7:39 PM -
0 Votes
MSNBOT appears to be case sensitive when reading robots.txt
Does anyone have any suggestions about how to resolve the issue I am having, where MSNBOT appears to ignore my robots.txt?Answered | 10 Replies | 7098 Views | Created by SteveC99 - Thursday, July 3, 2008 12:38 PM | Last reply by Brett Yount - Monday, August 25, 2008 8:21 PM -
2 Votes
msnbot is using MY robots.txt to crawl YOUR site
In other words, it THINKS it is getting robots.txt for www.yoursite.com, but it is really reading the robots.txt file that is served for the default host at the IP address for ...Answered | 2 Replies | 18234 Views | Created by bmpub - Saturday, April 11, 2009 4:17 PM | Last reply by Brett Yount - Wednesday, April 15, 2009 5:21 PM -
0 Votes
MSNBOT_Mobile Not honoring robots.txt
I have several sites and one of those sites has a bot trap with a directory excluded in the robots.txt file.Answered | 1 Replies | 4612 Views | Created by atatum - Thursday, January 17, 2008 6:38 AM | Last reply by Jeremiah Andrick Microsoft - Monday, February 4, 2008 3:46 AM -
0 Votes
what all Robots.txt directives FAST supports ?
We want to create a robots.txt file for our sharepoint web application to specifiy what all locations to be crewled by any crewler and what to not.Unanswered | 2 Replies | 8920 Views | Created by mahavir patil - Tuesday, December 30, 2008 9:30 AM | Last reply by mahavir patil - Wednesday, December 31, 2008 7:06 AM -
0 Votes
Web host doesn't have robots.txt help!!
If you don’t have anything to “hide”, you can just forget about the robots.txt file.Answered | 5 Replies | 4166 Views | Created by miteylittle - Wednesday, June 11, 2008 1:06 AM | Last reply by Energy_no_ - Wednesday, July 9, 2008 11:56 AM -
0 Votes
Robots.txt File text that matters most Please Help
I have put up Google adds on my site,I have the problem that is explained hereunder in detail,Please some Guide me How it can be solved.The detail is as ...Answered | 1 Replies | 3827 Views | Created by Dhanjal2 - Friday, March 7, 2008 12:53 PM | Last reply by Brett Yount - Wednesday, May 28, 2008 3:46 PM -
0 Votes
Does pattern matching (wildcards) work in robots.txt?
Does MSN accept wildcards in the robots.txt like Google and Yahoo?Answered | 1 Replies | 4719 Views | Created by thwart - Thursday, February 7, 2008 4:13 PM | Last reply by Brett Yount - Friday, May 30, 2008 8:40 PM -
0 Votes
WHS web server - robots.txt - should default to opt out?
I'm sure it's unlikely, but would it hurt to have a robots.txt file with a disallow all?Unanswered | 2 Replies | 3767 Views | Created by Lyme - Sunday, April 22, 2007 4:40 PM | Last reply by judgeschambers - Monday, April 23, 2007 2:53 AM -
0 Votes
Robots.txt validation tool - no work?
Hi Jeffrey, What you pasted there is your Webmaster tools validation code which is not part of your robots.txt.Answered | 2 Replies | 6896 Views | Created by www.jeffreydavidmorris.com - Saturday, February 28, 2009 6:37 PM | Last reply by www.jeffreydavidmorris.com - Friday, May 8, 2009 12:30 AM -
0 Votes
robots.txt
hi are you sure to write this ? User-agent: * Disallow: Sitemap: ...Answered | 1 Replies | 4282 Views | Created by abyzn - Wednesday, November 12, 2008 11:37 AM | Last reply by Brett Yount - Thursday, November 20, 2008 9:21 PM -
0 Votes
Site Being Indexed Even Though robots.txt Disallowing It
The robots.txt file that we're using looks like this: User-agent: * Disallow: / The Webmaster Tools validation says that's correct, ...Answered | 2 Replies | 3638 Views | Created by zz_andre - Tuesday, May 13, 2008 5:59 PM | Last reply by zz_andre2 - Thursday, May 29, 2008 3:50 PM -
0 Votes
Why is MSNBOT ignoring our robots.txt
Hi, Our robots.txt file is currently: User-agent: msnbot Disallow: ...Answered | 1 Replies | 4301 Views | Created by SteveC99 - Tuesday, August 5, 2008 12:14 PM | Last reply by Brett Yount - Monday, August 25, 2008 8:13 PM
No announcements
