locked
Unhandled exception but not able to determine why RRS feed

  • Question

  • Hi, I'm using Visual studio 2008 and c# programming language to use the live search API 2.0

    In order to get a better understanding of how this all works I've been reading the Live Search Api Documentation sample code and I've used that to successfully query and receive data from the live search API.  However, I notice that sometimes I'm getting unhandled exceptions that Visual studio says are coming from this line here that I received from the API documentation.  I simplified the example for display here

    searchQuery = "hello world";
    SearchRequest request = BuildRequest(searchQuery, offset);
    SearchResponse response = service.Search(request);

    foreach (WebResult result in response.Web.Results)
    {
           result.DisplayUrl)
    }




    However, as a test I wanted to see how long I could go doing that over and over so what I did was I found a wordlist of about 20,000 random words to use for changing the searchQuery and  for each word I'd change that offset value + 50 and it would run through again and then +50 again and then again, etc all the way up to 1000 where after that it would grab a new searchQuery and do it again.

    I think I understand how this all works now, very cool.  However, before I went out for supper last night I left it running and it went for about a half hour then I got the below error.


    Unhandled Exception: System.NullReferenceException: Object reference not set to
    an instance of an object.
       at LiveSearch.Program.Main(String[] args) in E:\LiveSearch\LiveSearch\Program
    .cs:line 63

    Now when I visit line 63 from above, which is where the
    foreach (WebResult result in response.Web.Results) line is.

    So in visual studio I hovered over the response.Web.Results and it says all the properties are set to be null except my search query.  Does this mean that, that particular search was invalid perhaps?

    Last night before I went to bed, I started it up again and when I awoke this morning, this time it lasted for just under 20 minutes before getting the same thing back.  I check the value of my searchQuery and it is set to a good value and so was the offset value as wel.

    Is it normal for this to be happening randomly?  Is there any way to handle it?


    Thanks for reading,

    RickyWh,
    Monday, December 1, 2008 7:56 PM

Answers

  • Hi Rick,

     

    I wanted to let you know that I started a discussion with our Legal team to fix the obvious bug in the Terms of Use that you found.

    We will introduce an explicit paragraph about allowing more freedom to use an AppID for testing purposes but introducing a traffic limitation in that case.

     

    Thank you for drawing our attention to this problem. We had just missed it.

     

    --Alessandro

    Thursday, December 4, 2008 9:19 AM

All replies

  • Well, besides the fact that you are violating the terms of use and that your AppID would be immediately terminated at the first audit (you must use the app in a customer-facing application, you are not; you must send query that are human-generated, you are not; you must display all the results you are returned, you are not), you are also looking like a denial of service attack to the service, so the watchdog is simply throttling down connections from your IP address. 

    Luckily for you, you did not trip it often enough (or long enough) to be permanently blacklisted.

    Such a detection is a fairly complex heuristics on the traffic that looks at bursts of traffic coming from a single IP in a period of time.

     

    When you are identified as a DoS, the service simply stops returning you results. You have no way to know that the transaction failed at the transport layer (where you get an HTTP 200), but your result is empty, hence the null reference: there is no response.Web in your object when this happens.

     

    Please do read the terms of use before using the service.

    We do periodic and frequent audits where we shut down offending AppIDs used for BOTs, scrapers or other usages outside of what is permitted.

     

    HTH

     

    --Alessandro

     

     

     

     

    Tuesday, December 2, 2008 4:56 AM
  • Hi again

    After reading your post and reading mine again, I see what you mean.  I am actually just learning the API now, once i'm familiar with it very well, I'll definitely be honoring the terms of service for the API and be creating some customer-facing applications.

    Thanks for your response


    Rickity Rick,
    Tuesday, December 2, 2008 6:46 AM
  • Hi Rick,

     

    Sorry if I came through as a little harsh.

     

    Unfortunately they wrote our terms of use in the strictest legalese, so I want to take a chance to summarize their spirit as often as I can.

     

    As long as you don't trigger he DoS and keep your traffic reasonable, you will stay in the tolerance band where we will not react. We understand that you need some wiggle room to experiment and kick the tires of the service.

     

    The reason why we are so strict is just that we need to admister the capacity of the service. While very large, it is not infinite, and we need as much as possible to make sure that the queries that we serve through the API further our business needs  (i.e. allow potentially showing an ad). If we let anyone come by and sends us hundreds or thousands of queries per seconds, we would eventually reach the point where this would have an impact on all other users, and we cannot let this happen.

     

    Happy experimenting

     

    HTH

     

    --Alessandro

     

    Tuesday, December 2, 2008 7:10 AM
  • Hi Rick,

     

    I wanted to let you know that I started a discussion with our Legal team to fix the obvious bug in the Terms of Use that you found.

    We will introduce an explicit paragraph about allowing more freedom to use an AppID for testing purposes but introducing a traffic limitation in that case.

     

    Thank you for drawing our attention to this problem. We had just missed it.

     

    --Alessandro

    Thursday, December 4, 2008 9:19 AM