locked
Not able to PUSH 7.2B events to event hub from Azure function RRS feed

  • Question

  • we have a service which is generating 7.2 B events in 30 mins Duration.  Need to store these events in Azure Data Explorer. To send the data we are using Event-Hub since we can stream the data in real-time using the Event-Hubs. We are not able to send these many records to the event hub through AMQP and event hub client SDK. Is there any way to push huge records in nominal Duration(~30mins). 

    We are able to push  2.5 Million Events to HUB in batches(Each batch contains 400 Events) with the following code:

    public static async Task EventHub(string processors
     {     
         var producerOptions = new EventHubProducerClientOptions();
         producerOptions.ConnectionOptions.TransportType = EventHubsTransportType.AmqpWebSockets;
       var producer = new EventHubProducerClient(connectionstring, hubName,    producerOptions);
             using EventDataBatch eventBatch = await producer.CreateBatchAsync();
                _ = eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes(processLogs)));
          await producer.SendAsync(eventBatch);
    }
    

    and the event data follows below:

    {

    "Event_End_Timestamp": "2020-11-06T06:00:01.8700000Z",

    "Event_Duration": 0.163,

    "Scrap": 5.0,

    "OperationCycle": 0.19,

    "WorkOrder_ID": "123450",

    "Sku": "98764",

    "Sequence": "10",

    "Process_ID": "H200",

    "Machine_ID": "M1503",

    "Parent_Event_ID": 10000002,

    "Event_ID": 10000000,

    "Event_Type_Counter": 10,

    "Event_Type": "OpercationCycle",

    "Event_Start_Timestamp": "2020-11-06T06:00:01.7060000Z"

    }



    Friday, November 20, 2020 8:33 AM

All replies

  • azure related forum migrated to mentioned below forum please use this link

    https://docs.microsoft.com/en-us/answers/index.html

    hope answer the question if issue resolve accept answer

    Friday, November 20, 2020 11:39 AM