locked
"Web adddress isn't valid" when trying to add a new site to Webmaster Tools RRS feed

  • Question

  • I'm trying to add www.uncorneredmarket.com to the Live Webmaster Tools, but I keep getting the message "Web address isn't valid".  The site is live and has been for months.  I've been able to add other sites.  I'm only having a problem with this one for some reason.

    I've also tried uncorneredmarket.com (without the www) and that doesn't work either.  I would like the site indexed with the www.

    Am I missing something obvious here?
    Thanks,
    Daniel
    Friday, February 8, 2008 4:51 PM

Answers

  • It looks like you server is running security software called Bad Behaviour which is returning 403 (forbidden) errors when webmaster tools tries to validate your website's existance. Try adding your website to webmaster tools after disabling this tool.

    HTTP Request

    GET / HTTP/1.1

    Host: www.uncorneredmarket.com

    User-Agent: Mozilla/5.0

     

    HTTP Response

    HTTP/1.1 403 Bad Behavior

    Proxy-Connection: Keep-Alive

    Connection: Keep-Alive

    Transfer-Encoding: chunked

    Via: 1.1 SEA-PRXY-02

    Date: Sat, 09 Feb 2008 00:17:21 GMT

    Content-Type: text/html

    Server: Apache/2.0.61 (Unix) PHP/4.4.7 mod_ssl/2.0.61 OpenSSL/0.9.7e mod_fastcgi/2.4.2 DAV/2 SVN/1.4.2

    X-Powered-By: PHP/5.2.3

    Vary: Accept-Encoding

    Keep-Alive: timeout=2, max=100

     

     

    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

    <html xmlns="http://www.w3.org/1999/xhtml">

    <head>

    <title>HTTP Error 403</title>

    </head>

    <body>

    <h1>Error 403</h1>

    <p>We're sorry, but we could not fulfill your request for

    / on this server.</p>

    <p>You do not have permission to access this server.</p>

    <p>Your technical support key is: <strong>836b-0060-17f4-e8c8</strong></p>

    <p>You can use this key to <a href="http://www.ioerror.us/bb2-support-key?key=836b-0060-17f4-e8c8">fix this problem yourself</a>.</p>

    <p>If you are unable to fix the problem yourself, please contact <a href="mailto:dan+nospam@nospam.uncorneredmarket.com">dan at uncorneredmarket.com</a> and be sure to provide the technical support key shown above.</p>

    Saturday, February 9, 2008 12:33 AM
  •  

    Sorry, but it's not our policy to give out the IP range for our crawlers. It's also not practical as this is a distributed system with a large number of IPs that change frequently.
    Monday, February 18, 2008 7:08 PM
  •  snekse wrote:
    I'm having the same issue, but I use shared hosting through CrystalTech, so I have no idea if they use Bad Behavior and I highly doubt I'd be able to change the settings.  Is there a work around?

    ERROR: The web address isn't valid.
    Web address: www.gastronomicfightclub.com
    Sitemap address: www.gastronomicfightclub.com/GFC_SiteMap.cfm

     

    I was able to add your website successfully. There may have been a problem accessing your site at the time that you tried it. Try adding it again.

    Monday, February 18, 2008 7:13 PM
  •  

    I did some investigation of your site, http://www.datacrunch.net/. When I tried browsing to the site from my workstation it took 26 seconds to load and resulted in a 404 error. I'm not sure how your site is setup, but it appears that your site root always returns a 404. Luckily, the subpages within the site seem fine. Now although the root is a 404, your webserve still seems to serve the correct content.

     

    Try using fiddler to check what response codes your website is returning to end-users.

    Tuesday, December 30, 2008 7:32 PM

All replies

  • It looks like you server is running security software called Bad Behaviour which is returning 403 (forbidden) errors when webmaster tools tries to validate your website's existance. Try adding your website to webmaster tools after disabling this tool.

    HTTP Request

    GET / HTTP/1.1

    Host: www.uncorneredmarket.com

    User-Agent: Mozilla/5.0

     

    HTTP Response

    HTTP/1.1 403 Bad Behavior

    Proxy-Connection: Keep-Alive

    Connection: Keep-Alive

    Transfer-Encoding: chunked

    Via: 1.1 SEA-PRXY-02

    Date: Sat, 09 Feb 2008 00:17:21 GMT

    Content-Type: text/html

    Server: Apache/2.0.61 (Unix) PHP/4.4.7 mod_ssl/2.0.61 OpenSSL/0.9.7e mod_fastcgi/2.4.2 DAV/2 SVN/1.4.2

    X-Powered-By: PHP/5.2.3

    Vary: Accept-Encoding

    Keep-Alive: timeout=2, max=100

     

     

    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

    <html xmlns="http://www.w3.org/1999/xhtml">

    <head>

    <title>HTTP Error 403</title>

    </head>

    <body>

    <h1>Error 403</h1>

    <p>We're sorry, but we could not fulfill your request for

    / on this server.</p>

    <p>You do not have permission to access this server.</p>

    <p>Your technical support key is: <strong>836b-0060-17f4-e8c8</strong></p>

    <p>You can use this key to <a href="http://www.ioerror.us/bb2-support-key?key=836b-0060-17f4-e8c8">fix this problem yourself</a>.</p>

    <p>If you are unable to fix the problem yourself, please contact <a href="mailto:dan+nospam@nospam.uncorneredmarket.com">dan at uncorneredmarket.com</a> and be sure to provide the technical support key shown above.</p>

    Saturday, February 9, 2008 12:33 AM
  • Eric,
    I temporarily disabled Bad Behavior and Webmaster Tools recognized the site.

    I noticed in the Webmaster Tools summary panel that Live has indexed over 1000 pages on the site, some as recently as February 6.  So, it appears as though the MSNbot has had some ability to crawl the site (though the pages listed were oddly only a "category" type of page).  Perhaps Bad Behavior was only blocking Webmaster Tools...and not the MSNbot?

    Regardless, is there a range of IP addresses that I can safely whitelist in Bad Behavior so that the real MSNbot (there are fake agents, I've heard) is allowed, while spambots pretending to be MSNbot are not?

    Thanks for your help,
    Daniel


    Saturday, February 9, 2008 7:31 AM
  • I'd like to know as well.

    That doesn't seem to work for me when I disable Bad Behavior.
    Monday, February 11, 2008 5:57 AM
  •  

    I'm having the same problem. We are using Go daddy to do our hosting. I didn't have this problem when submitting to other search engines.
    Tuesday, February 12, 2008 6:01 PM
  • I'm having the same issue, but I use shared hosting through CrystalTech, so I have no idea if they use Bad Behavior and I highly doubt I'd be able to change the settings.  Is there a work around?

    ERROR: The web address isn't valid.
    Web address: www.gastronomicfightclub.com
    Sitemap address: www.gastronomicfightclub.com/GFC_SiteMap.cfm
    Wednesday, February 13, 2008 3:22 PM
  •  

    Sorry, but it's not our policy to give out the IP range for our crawlers. It's also not practical as this is a distributed system with a large number of IPs that change frequently.
    Monday, February 18, 2008 7:08 PM
  •  snekse wrote:
    I'm having the same issue, but I use shared hosting through CrystalTech, so I have no idea if they use Bad Behavior and I highly doubt I'd be able to change the settings.  Is there a work around?

    ERROR: The web address isn't valid.
    Web address: www.gastronomicfightclub.com
    Sitemap address: www.gastronomicfightclub.com/GFC_SiteMap.cfm

     

    I was able to add your website successfully. There may have been a problem accessing your site at the time that you tried it. Try adding it again.

    Monday, February 18, 2008 7:13 PM
  • I've tried for several days with the following site.  I don't understand the domain issue if all the sites are on the same server, I didn't have problems adding the others??

     

     

    ERROR:  The web address isn't valid.

    Web address: http://www.cycle-sites.com

    Sitemap address:  http://www.cycle-sites.com/sitemap.xml

     

     

    Saturday, December 20, 2008 3:13 PM
  • I'll throw my 2 cents in here, same result for me.

     

    ERROR:  The web address isn't valid.

    Web address:www.wtflolomgbbq.com

    Sitemap address:www.wtflolomgbbq.com/SiteMap.xml

     

    This is kind of dumb... "Bad Behavior" is bad behavior, isn't it? Google, Yahoo, even Cuil have no problems accessing my site, but Microsoft can't? My robots.txt file allows everybody to search, and I'm even submitting a good sitemap.

     

    I don't have complete control over my server, since it's a shared asp.net server (um, Microsoft box, guys).

     

    At the very least, you should be giving a better description of what's gone wrong when the submission fails.

    Thursday, December 25, 2008 3:44 AM
  •  

    Your site contains a bug in your WTFCMS.MainCMS.HandleVisitorStatLogging() code. If a user-agent is not specified in a request then your the your site will throw an exception, which ASP.NET handles as an 500 HTTP status code. This is probably the reason why your site is not working. However, Webmaster Tools should be specifying a user-agent. I'll file a bug on this behaviour and one of the devs should be able to take a look at it soon.

     

    Also, you should consider changing your customErrors mode in ASP.NET. Currently you are showing a full stacktrace with your exception, which is generally considered to be a security risk. You should add <customErrors mode="On"/> or <customErrors mode="RemoteOnly"/> to your web.config file.

    Friday, December 26, 2008 6:27 PM
  •  Eric Carter (MSFT) wrote:

     

    Your site contains a bug in your WTFCMS.MainCMS.HandleVisitorStatLogging() code. If a user-agent is not specified in a request then your the your site will throw an exception, which ASP.NET handles as an 500 HTTP status code. This is probably the reason why your site is not working. However, Webmaster Tools should be specifying a user-agent. I'll file a bug on this behaviour and one of the devs should be able to take a look at it soon.

     

    Also, you should consider changing your customErrors mode in ASP.NET. Currently you are showing a full stacktrace with your exception, which is generally considered to be a security risk. You should add <customErrors mode="On"/> or <customErrors mode="RemoteOnly"/> to your web.config file.

     

    Thanks.... I haven't done much asp.net stuff (though I've been a software engineer for 25 years), and I'm in the process of building up a CMS package for my own use. I'll fix the bug and get a handle on my error mode (though right now the stack trace is handy).

     

    I don't get a lot of visitors yet... heh... but hopefully, once it gets properly indexed, I'll start seeing some more users.

     

     

     

     

    Friday, December 26, 2008 6:48 PM
  • I'm having the same problem with http://www.datacrunch.net but certainly i do not have same error going out... see logs:

    2008-12-29 18:40:37 GET /jakarta/isapi_redirect.dll - [IP_REMOVED] msnbot-webmaster/1.0+(+http://search.msn.com/msnbot.htm) - 200 0 64 0 181 87

    I'm running windows 2008 server with jakarta isapi filter... the response is OK 200 coming out. Any idea what to do here?

    z

    Monday, December 29, 2008 6:46 PM
  • by the way - running siteBig Smileatacrunch.net query on live.com shows that msn bot has no problem crawling at all
    Monday, December 29, 2008 6:48 PM
  •  

    I did some investigation of your site, http://www.datacrunch.net/. When I tried browsing to the site from my workstation it took 26 seconds to load and resulted in a 404 error. I'm not sure how your site is setup, but it appears that your site root always returns a 404. Luckily, the subpages within the site seem fine. Now although the root is a 404, your webserve still seems to serve the correct content.

     

    Try using fiddler to check what response codes your website is returning to end-users.

    Tuesday, December 30, 2008 7:32 PM
  •  

    Strange. I just stopped in here to say I was able to get past the "Web Address isn't valid" by using the full asp.net URL (http://www.wtfolomgbbq.com/Default.aspx) and I saw the DataCrunch issue - I had no problem surfing to it.

     

    At any rate, thanks for the earlier input. Now if I can just get the Authentication to work.

     

    Oh... and I gave Fiddler 2 a try on his site with the "msnbot/1.1 (Live Search Webmaster Tool)" agent string, and it handled HTTP requests 1.0, 1.1 and 1.2 with no problem (instantly).

    Wednesday, January 7, 2009 2:31 AM