locked
Web Serivce Architecture/Design for a CRM Integration Service: How to Execute Batch Jobs in a Load Balanced Environment RRS feed

  • Question

  • I've created a web service in C# that I am calling from a SQL Agent scheduled job and that integrates data into CRM 2011 On-Premise. The web server is load-balanced with two servers in the farm at this time.  On the CRM App Servers, they are also load-balanced and the oddity is that on the CRM App Services, it runs about 95% of the processes on one and 5% on the other. The reason for this is that there are four unique process that are importing data into a database, as follows. 

    • 15K records per hour
    • 100 records per hour
    • 20K records per hour
    • 1K records per hour

    Note that the difference between the volumes ran by each are significant.  What we find is that #2/#4 runs on the same server and #1/#3 run on the same server.  Because #2 and #4 take up about 95% of the workload, server-2 is being underutilized. 

    What I would like to do is figure out how to break apart the two larger jobs into small subsets of data. This is running for about 60 offices/branches we have across the country.  So I think that if I can adjust the design a bit, so that I can fire off separate sessions/jobs per branch, the load-balancer would be better able to balance the workload.  However, I am not sure how to go about designing/architecting this, as I am not sure exactly what the CRM App Servers are doing and how I can go about getting them to correctly load-balance updates firing off from my web-service.

    I might need the web service methods to fire off in such a way as to create multiple sessions that the load-balancer will balance for me.  Currently, I am created unique threads per branch. So I think the best option here is to adjust the design to create unique sessions where I am currently creating unique threads.

    So my question is, what does the CRM App Server load-balancer need in order to consider something as a separate session or job?  Would I need to create say a unique method for each session, or could I go into the global.asax for the web-service and create muiltiple sessions in such a way that the load-balancer could pick them up and balance them correctly? 

    I just have no idea where to start looking at this, but I am sure it's a common issue to run into.  So I am hoping someone that has done this can offer some suggestions or point me in the right direction.

    Best regards,

    Jon


    Jon Gregory Rothlander





    • Moved by Caillen Thursday, July 16, 2015 5:11 AM
    • Edited by jonrothlander Friday, July 31, 2015 6:58 PM
    Wednesday, July 15, 2015 3:11 PM

Answers

  • Hello,

    You should ask in the Microsoft Dynamics CRM Forum on the Microsoft Dynamics Community.

    As Microsoft's Dynamics Community forums are on a different platform, we cannot move the question for you.

    Karl


    When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer.
    My Blog: Unlock PowerShell
    My Book: Windows PowerShell 2.0 Bible
    My E-mail: -join('6D73646E5F6B61726C406F75746C6F6F6B2E636F6D'-split'(?<=\G.{2})'|%{if($_){[char][int]"0x$_"}})

    Friday, September 4, 2015 10:06 PM

All replies

    • Proposed as answer by Caillen Thursday, July 16, 2015 5:10 AM
    • Marked as answer by Just Karl Friday, September 4, 2015 10:06 PM
    • Unmarked as answer by Just Karl Friday, September 4, 2015 10:06 PM
    Wednesday, July 15, 2015 3:17 PM
  • I was wondering why no one ever responded to my question.  Now I see that you moved it to the off-topic category without understanding the topic being discussed or without replying to me to asking it if it was off topic. 

    This isn't off topic.  This is a CRM 2011 specific service that has to be designed to work within CRM 2011 and the load-balancers.  No one on the Web Service forum will have any clue as to how CRM 2011 works with it's load-balancers and architecture and how this CRM service needs to function.  Only a senior level CRM developer would have any idea how to address this question. 

    I did update the questions to make this a bit more obvious. But I am unsure how to move this to the CRM Development forum.  I guess I can just post it again.


    Jon Gregory Rothlander




    Friday, July 31, 2015 6:55 PM
  • Hello,

    You should ask in the Microsoft Dynamics CRM Forum on the Microsoft Dynamics Community.

    As Microsoft's Dynamics Community forums are on a different platform, we cannot move the question for you.

    Karl


    When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer.
    My Blog: Unlock PowerShell
    My Book: Windows PowerShell 2.0 Bible
    My E-mail: -join('6D73646E5F6B61726C406F75746C6F6F6B2E636F6D'-split'(?<=\G.{2})'|%{if($_){[char][int]"0x$_"}})

    Friday, September 4, 2015 10:06 PM