Answered by:
Web Serivce Architecture/Design for a CRM Integration Service: How to Execute Batch Jobs in a Load Balanced Environment

Question
-
I've created a web service in C# that I am calling from a SQL Agent scheduled job and that integrates data into CRM 2011 On-Premise. The web server is load-balanced with two servers in the farm at this time. On the CRM App Servers, they are also load-balanced and the oddity is that on the CRM App Services, it runs about 95% of the processes on one and 5% on the other. The reason for this is that there are four unique process that are importing data into a database, as follows.
- 15K records per hour
- 100 records per hour
- 20K records per hour
- 1K records per hour
Note that the difference between the volumes ran by each are significant. What we find is that #2/#4 runs on the same server and #1/#3 run on the same server. Because #2 and #4 take up about 95% of the workload, server-2 is being underutilized.
What I would like to do is figure out how to break apart the two larger jobs into small subsets of data. This is running for about 60 offices/branches we have across the country. So I think that if I can adjust the design a bit, so that I can fire off separate sessions/jobs per branch, the load-balancer would be better able to balance the workload. However, I am not sure how to go about designing/architecting this, as I am not sure exactly what the CRM App Servers are doing and how I can go about getting them to correctly load-balance updates firing off from my web-service.
I might need the web service methods to fire off in such a way as to create multiple sessions that the load-balancer will balance for me. Currently, I am created unique threads per branch. So I think the best option here is to adjust the design to create unique sessions where I am currently creating unique threads.
So my question is, what does the CRM App Server load-balancer need in order to consider something as a separate session or job? Would I need to create say a unique method for each session, or could I go into the global.asax for the web-service and create muiltiple sessions in such a way that the load-balancer could pick them up and balance them correctly?
I just have no idea where to start looking at this, but I am sure it's a common issue to run into. So I am hoping someone that has done this can offer some suggestions or point me in the right direction.
Best regards,
Jon
Jon Gregory Rothlander
- Moved by Caillen Thursday, July 16, 2015 5:11 AM
- Edited by jonrothlander Friday, July 31, 2015 6:58 PM
Wednesday, July 15, 2015 3:11 PM
Answers
-
Hello,
You should ask in the Microsoft Dynamics CRM Forum on the Microsoft Dynamics Community.
As Microsoft's Dynamics Community forums are on a different platform, we cannot move the question for you.Karl
When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer.
My Blog: Unlock PowerShell
My Book: Windows PowerShell 2.0 Bible
My E-mail: -join('6D73646E5F6B61726C406F75746C6F6F6B2E636F6D'-split'(?<=\G.{2})'|%{if($_){[char][int]"0x$_"}})- Marked as answer by Dave PatrickMVP Saturday, September 5, 2015 9:28 PM
Friday, September 4, 2015 10:06 PM
All replies
-
Wednesday, July 15, 2015 3:17 PM
-
I was wondering why no one ever responded to my question. Now I see that you moved it to the off-topic category without understanding the topic being discussed or without replying to me to asking it if it was off topic.
This isn't off topic. This is a CRM 2011 specific service that has to be designed to work within CRM 2011 and the load-balancers. No one on the Web Service forum will have any clue as to how CRM 2011 works with it's load-balancers and architecture and how this CRM service needs to function. Only a senior level CRM developer would have any idea how to address this question.
I did update the questions to make this a bit more obvious. But I am unsure how to move this to the CRM Development forum. I guess I can just post it again.
Jon Gregory Rothlander
- Edited by jonrothlander Friday, July 31, 2015 7:00 PM
Friday, July 31, 2015 6:55 PM -
Hello,
You should ask in the Microsoft Dynamics CRM Forum on the Microsoft Dynamics Community.
As Microsoft's Dynamics Community forums are on a different platform, we cannot move the question for you.Karl
When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer.
My Blog: Unlock PowerShell
My Book: Windows PowerShell 2.0 Bible
My E-mail: -join('6D73646E5F6B61726C406F75746C6F6F6B2E636F6D'-split'(?<=\G.{2})'|%{if($_){[char][int]"0x$_"}})- Marked as answer by Dave PatrickMVP Saturday, September 5, 2015 9:28 PM
Friday, September 4, 2015 10:06 PM