none
Change Powershell Parallel limit

    Question

  • Is there a way to change powershell foreach -parallel limit in v3 while running a workflow?


    99upgrade

    • Moved by Bill_Stewart Tuesday, December 31, 2013 11:10 PM Abandoned thread
    Thursday, October 24, 2013 6:45 AM

All replies

  • Hi,

    while v4 introduced a ThrottleLimit parameter on foreach-parallel, I don't think there is a way to do that in v3. If you are into c# you could use dotPeek to check how this has been implemented and try to reproduce it.

    Thursday, October 24, 2013 8:05 AM
  • Do you know if there is a limit in v4? If yes, what is the limit?

    99upgrade

    Thursday, October 24, 2013 8:06 AM
  • there is no limit other than your machine's resources and theoretically speaking the maximum value for an integer ([int]::MaxValue)
    • Marked as answer by 99upgrade Thursday, October 24, 2013 8:13 AM
    • Unmarked as answer by 99upgrade Thursday, October 24, 2013 9:34 AM
    Thursday, October 24, 2013 8:11 AM
  • I upgraded to v4 and when I run foreach -parallel it still executes 5 at a time.

    99upgrade

    Thursday, October 24, 2013 9:35 AM
  • I can't test it myself, have a look at http://www.happysysadm.com/2013/06/how-to-throttle-workflow-activites-in.html and see if you can reproduce the results. What happens if you set the ThrottleLimit to a value higher than 5?

    Thursday, October 24, 2013 9:50 AM
  • Even if I change the throttlelimit to 1000 it still limits to 5 at a time.

    workflow Test-ping
    { 
        param(
      
        [parameter(mandatory=$true)]
        [int]$ThrottleLimit
            
        )

        $servers = Get-Content -Path servers.txt
        foreach -parallel -throttlelimit $ThrottleLimit ($server in $servers){Test-Connection -ComputerName $server -Count 1}

    }


    99upgrade

    Thursday, October 24, 2013 10:24 AM
  • The hard limit seems to be set to 5.  See:

    http://technet.microsoft.com/en-us/library/hh849862.aspx


    ¯\_(ツ)_/¯


    • Edited by jrv Thursday, October 24, 2013 12:51 PM
    Thursday, October 24, 2013 12:50 PM
  • Agreed. The hard limit on v3 is set to 5. However, I upgraded to v4 as suggested by Dirk_74 and still see the limit is set to 5. Any way to change this?

    I'm running the code following on v4

    workflow Test-ping
    { 
        param(
      
        [parameter(mandatory=$true)]
        [int]$ThrottleLimit
            
        )

        $servers = Get-Content -Path servers.txt
        foreach -parallel -throttlelimit $ThrottleLimit ($server in $servers){Test-Connection -ComputerName $server -Count 1}

    }


    99upgrade

    Thursday, October 24, 2013 3:10 PM
  • The limit is set by the OS.  In V4 the limit is still controlled by the thread limit which is still set to 5.  First you have to alter the thread limit on the process you are running.  Once you set that higher the parallel limit can be set higher than 5.

    I haven't installed V4 yet but am upgrading this weekend so I cannot test this yet.


    ¯\_(ツ)_/¯

    Thursday, October 24, 2013 3:52 PM
  • @http://technet.microsoft.com/en-us/library/hh849862.aspx

    It's saying that 5 is the default value not the max value. Maybe it helps if you change the WorkflowExecution options before running your workflow?:

    $wo = New-PSWorkflowExecutionOption -MaxSessionsPerWorkflow 50 -MaxSessionsPerRemoteNode 50 -MaxDisconnectedSessions 50
    Register-PSSessionConfiguration -Name ITWorkflows -SessionTypeOption $wo -Force

    Thursday, October 24, 2013 3:56 PM
  • I think it will still be limited by the Windows default thread limit which is 5. 

    This will be good if it actually does up the limit.


    ¯\_(ツ)_/¯

    Thursday, October 24, 2013 4:28 PM
  • I cannot test this until tomorrow so lets see how it goes.


    99upgrade

    Thursday, October 24, 2013 5:15 PM
  • The limit is set by the OS.  In V4 the limit is still controlled by the thread limit which is still set to 5.  First you have to alter the thread limit on the process you are running.  Once you set that higher the parallel limit can be set higher than 5.

    I haven't installed V4 yet but am upgrading this weekend so I cannot test this yet.


    ¯\_(ツ)_/¯

    Wow so am I the only one on v4 now? ;)

    99upgrade

    Thursday, October 24, 2013 5:18 PM
  • The limit is set by the OS.  In V4 the limit is still controlled by the thread limit which is still set to 5.  First you have to alter the thread limit on the process you are running.  Once you set that higher the parallel limit can be set higher than 5.

    I haven't installed V4 yet but am upgrading this weekend so I cannot test this yet.


    ¯\_(ツ)_/¯

    Wow so am I the only one on v4 now? ;)


    99upgrade

    Yup.  You are the only on in the whole world on V4.  We wanted to see if you survived.  You are the guinea pig, the lab rat.


    ¯\_(ツ)_/¯

    Thursday, October 24, 2013 5:25 PM
  • I tried:

    workflow Test-ping
    {

        $wo = New-PSWorkflowExecutionOption -MaxSessionsPerWorkflow 100 -MaxDisconnectedSessions 200
        Register-PSSessionConfiguration -Name test-ping -SessionTypeOption $wo -Force
        $servers = Get-Content -Path servers.txt
        foreach -parallel ($server in $servers){Test-Connection -ComputerName $server -Count 1}

    }

    and I get this error:

    Cannot convert the "Microsoft.PowerShell.Commands.PSWorkflowExecutionOption" value of type
    "Deserialized.Microsoft.PowerShell.Commands.PSWorkflowExecutionOption" to type
    "System.Management.Automation.PSSessionTypeOption".
        + CategoryInfo          : InvalidArgument: (:) [], ParentContainsErrorRecordException
        + FullyQualifiedErrorId : ConvertToFinalInvalidCastException
        + PSComputerName        : [localhost]


    99upgrade


    • Edited by 99upgrade Friday, October 25, 2013 5:15 AM edited
    Friday, October 25, 2013 5:15 AM
  • Start here and follow the instructions at each step.

    get-PSSessionConfiguration


    ¯\_(ツ)_/¯

    Friday, October 25, 2013 5:30 AM
  • Using Snover and company examples and adding your custom onfiguration:

    $wo = New-PSWorkflowExecutionOption -MaxSessionsPerWorkflow 100 -MaxDisconnectedSessions 200
    Register-PSSessionConfiguration -Name test-ping -SessionTypeOption $wo -Force
    
    $s = New-PSWorkflowSession 
    Invoke-Command $s { 
    workflow Invoke-ParallelForEach{ 
        foreach -parallel ($i in 1..10){ 
            InlineScript{ 
                "foo: $using:i" 
            } 
            Get-Process -Name PowerShell* 
        } 
       } 
    }
    
    # now run the workflow with the custom configuration
    Invoke-Command $s { Invoke-ParallelForEach -PSComputerName localhost -PSConfigurationName test-ping}
    workflow Test-ping{
    
         $servers = Get-Content -Path servers.txt
         foreach -parallel ($server in $servers){Test-Connection -ComputerName $server -Count 1}
    
     }
    

    This must be run elevated to run against the local computer.


    ¯\_(ツ)_/¯

    Friday, October 25, 2013 6:14 AM
  • Jrv, I ran your code but cannot determine whether its limiting to 5 at a time or not. However, when I did measure-command {Your code} v/s measure-command {My original code}they both take the same amount of time to complete. So if you tested this on v4 and it works for you, we can mark it as answer. I will continue to run some more tests to see if your code works as expected. Thank you for the time and help. Much appreciated.

    99upgrade

    Friday, October 25, 2013 5:38 PM
  • It is hard to tell with only small short bits of code.  It takes 10 times or more longer to launch a thread then it does for it to execute.

    The workflow settings are fromV3.  None of this has changed in V4 with the exception of adding a couple of shortcuts.  I don't believe that the WF engine has not changed since 2011.


    ¯\_(ツ)_/¯

    Friday, October 25, 2013 5:51 PM