Submitting a job with thousands of tasks RRS feed

  • Question

  • What is the fastest way to submit a job that has 2000-5000 tasks?  It can take up to 5 seconds to submit a job with 2000 tasks from xml.  The HPC cmdlets seem to be even slower.  Anyone have any creative ideas?

    Thursday, July 27, 2017 8:17 PM

All replies

  • The fast way to do that is using parametric sweep task which only needs one records but it will expand during execution.

    So if all your tasks are similar and without dependencies, you can wrapper them into parametric tasks. If your all tasks commands varies a lot, you can wrapper the execution in an execution script, so that your parametric task command will be: myAppwrapper.cmd *

    During execution, the * will be translated to 1,2,...,2000 if your parametric sweep defined as: /parametric:1-2000:1

    Friday, July 28, 2017 9:24 AM
  • That actually appears slower than what I'm doing.  
    Monday, July 31, 2017 2:30 PM
  • Hi Jay, it should only take within a second to submit a parametric sweep task. Could you share more details?

    Qiufang Shi

    Tuesday, August 1, 2017 12:05 AM