locked
Performance drops when adding more cores RRS feed

  • Question

  • Hi

    We have setup an environment with 7 Cisco UCS compute nodes each equipped with 4 socket 12 core CPUs installed with Windows 2012 R2 HPC update 3. 

    Our MPI application seems to be running fine utilizing all CPU cores but we see very different execution times running the job with different ressources. The job is doing a lot of calculations, so it is heavily reying on CPU.

    To be able to have stable execution time we need to reduce cores instead of using all 48 cores. We have found that optimal setup is to reduce cores as seen below:

    2 nodes 40 cores each

    3 nodes 30 cores each

    7 nodes 13 cores each

    If I calculate the total number of cores it is around 80-100, so is there some sort of limit, that we canot find?

    Further we see network trafic running on 3 nodes 30 cores is 4-500 Mbps while running same job with 40 cores each shows 0-500 Kbps. So running in a non-optimal configuration, network trafic drops significantly.

    So at the moment we are not utilizing our hardware at all, so suggestions would be much appreciated.

    Friday, December 15, 2017 9:05 PM

All replies

  • Hi, This sounds like the behavior of your application and its algorithm. You need contact your application developer to get better understanding why "network traffic running on 3 nodes 30 cores is 4-500 Mbps while running same job with 40 cores each shows 0-500 Kbps".

    Qiufang Shi

    Monday, December 18, 2017 1:57 AM
  • We have talked with Microsoft Support about this issue, and we have got Namd set up to run in our environment, and while that is scaling in Microsoft Supports Azure environment, it does not in mine.

    My worries are that something is misconfigured in my environment, but in that case I do not know where to search. But you suppose not thet to be the case?

    Monday, December 18, 2017 9:51 AM