Hi again,
Recently I've written a code, which creates a job with a single task, and that single task submits 1000's of subsequent tasks, which are dependent on this task (i.e. will begin to run right after it terminates).
The submission itself takes a number of seconds, while surprisingly the
transition from Submitted to Queued state takes a long time, which eventually is an exponential function of number of tasks (i.e. on my machine 100th task takes 0.4 sec to queue, 500th task takes 1 sec to queue, 1000th task takes
5 sec to queue, 2000th - 10.5 sec , and so on). Integrating it all, 2000 tasks take
1.5 hours just to queue!
Since we need to work with 100,000s of tasks, we find this performance a real drawback from using Windows HPC, unless you can explain what I'm doing wrong. Perhaps a priority of a job should be specified, or the scheduling policy should be changed?
I have even tried to split the creation of tasks into a tree (i.e. create 10 tasks, each of which will create 10 tasks, each of which will create 10 tasks, and so on) and it did not make the performance any better.
Analyzing resource usage on the HPC head node I found that during this queuing process the system CPU usage is taken mostly by
sqlservr.exe (50% CPU) and HpcScheduler.exe (30% CPU) processes.
I'll appreciate your help very much, since we should soon make a decision whether or not to take the HPC direction.
All best,
Alex.
