locked
PowerShell Remoting and WinRM PSRemotingTransportException JobFailure RRS feed

  • Question

  • I have a PowerShell script that starts up backup jobs on a Veeam server and returns the results once the job has complete.
    This script works locally and remotely via Invoke-Command using a similar command to the one below.

    Invoke-Command -ComputerName VeeamServer -ScriptBlock {"BackupJobA","BackupJobB" | C:\Scripts\Start-VeeamJob.ps1}
    My problem is for a some backup jobs, the command executes and starts the backup job but then the WinRM session fails with the error below:
    Processing data for a remote command failed with the following error message: The client cannot connect to the destination 
    specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the 
    logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination 
    is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: "winrm quickconfig". 
    For more information, see the about_Remote_Troubleshooting Help topic.
        + CategoryInfo          : OperationStopped: (System.Manageme...pressionSyncJob:PSInvokeExpressionSyncJob) [], PSRemotingTransportException
        + FullyQualifiedErrorId : JobFailure

    When this happens the Windows PowerShell log has an "Engine state is changed from Available to Stopped" Information event under Source: PowerShell (PowerShell) | Event ID: 403 | Task Category: Engine Lifecycle
     
    I have had a quick look at about_Remote_Troubleshooting at the following link but nothing jumped out at me: http://technet.microsoft.com/en-us/library/hh847850.aspx

    Has anyone come across this sort of issue or could offer a suggestion? Both servers involved in the conversation are running Server 2008 R2 SP1.

    Thanks,
    Mark.

    • Moved by Bill_Stewart Thursday, January 2, 2014 8:40 PM Abandoned
    Thursday, August 22, 2013 3:19 PM

All replies

  • Whenever a PowerShell session ends that message is logged.

    Your error is a system issue.  I suspect something you are trying to run causes an exception or outputs some unusual messages or data.

    The issue is not really a scripting issue.  It may be a network issue.  You can call Microsoft Support fro help.


    ¯\_(ツ)_/¯

    Friday, August 23, 2013 10:10 PM
  • winrm quickconfig is what you need to focus on. winrm quickconfig is a cmd line command to enable remote desktop. Log onto the Veeam backup server and open an Administrative PowerShell prompt and run either 'winrm qc' and\or 'Enable-PSRemoting'.

    Hope this helps,

    Matthew Kerfoot


    Thursday, November 14, 2013 3:25 PM
  • winrm quickconfig is what you need to focus on. winrm quickconfig is a cmd line command to enable remote desktop. Log onto the Veeam backup server and open an Administrative PowerShell prompt and run either 'winrm qc' and\or 'Enable-PSremoting'.

    Hope this helps,

    Matthew Kerfoot

    "Remote Desktop"????

    If remoting was not initialized we would get a message telling us that.  We would also get a refused on RPC on the remote.  It is too bad the OP didn't post back with the resolution.

    Your advice is a good first step for all connectivity issues.  See The Scripting Guys Blog for more methods for validating remoting.  He has covered quite a few scenarios recently.

    This is an old thread.


    ¯\_(ツ)_/¯

    Thursday, November 14, 2013 3:31 PM
  • I didn't end up finding a root cause on this.

    I needed to run some commands on the server I was backing up before and after the backup job. I could recreate the issue at will calling the script to backup this particular server via Invoke-Command. I didn't get the same symptoms calling the script locally or with other backup jobs via the exact same Invoke-Command route.

    Unfortunately the server I was backing up was a monster so rather than flogging a weird problem and triggering a huge backup on each test, it was far simpler to have the schedule live on the backup server and remote the other direction. Thanks for the links jrv, I must have missed the alerts from this post back then.

    From memory, I started suspecting the size of the job may possibly be a contributing factor, perhaps I was hitting a quota or limit somewhere. Nothing jumped out at me in this area from that link in my first post and in the end the juice wasn't worth the squeeze.

    Remoting the other way lead me onto another learning curve the next day on using "Enable-PSRemoting" rather than "winrm quickconfig": http://social.technet.microsoft.com/Forums/en-US/c7c8cc03-8fe7-42d4-853f-6db0d1d29ab6/powershell-remoting-against-x64-os-and-microsoftpowershell32?forum=ITCG

    Mark

    Thursday, November 14, 2013 4:55 PM
  • Mark - it sounds like a timeout bug.  Check with Microsoft support or with Microsoft Connect.  Perhaps post in the WMF forum.  Give them a link to this and anything else you can glean like captures of the log files.

    My guess is that it is a bug.  An active link should not timeout.  Check to be sure the session defaults are not set to drop a session after a certain amount of time.  Sessions can  have limits in time, bandwidth and data volume.  I would hope that exceeding one of these would produce a like message such as "timeout exceeded" or such.

    I had an issue with loading WSUS.  It was the first load after an install and would run for a very long time.  A net tech kept seeing this new long time connection that BITS maintains while downloading.  He kept blocking the connection.  I fixed him and we tried again.  Another tech saw the processor usage was up since we were loading the database.  He kept stopping the WSUS service.  Eventually I set the service to restart infinitely.  The tech complained that the system was infected.  Now that I knew who he was so I fixed time too.

    Look around for something that may be disabling the connection like a firewall or AV or a helpful network technician.

    Thanks for the feedback.


    ¯\_(ツ)_/¯


    • Edited by jrv Thursday, November 14, 2013 5:13 PM
    Thursday, November 14, 2013 5:13 PM
  • Happily I don't think I was a victim of a helpful network technician, it was a pretty locked up system.

    I was leaning towards limits or quota's purely based on the size of the failing job compared to successful ones ... but the about_Remote_Troubleshooting seems to suggest if I was hitting any of those I'd get a polite message telling me what threshold to adjust. With that being said you'd assume it's a bug with a certain threshold being breached and not handled correctly which results in the problem.

    It wasn't a timeout (at least from a WsMan perspective) because the disconnection from the massive backup job was a fraction of the way in compared to the sessions that were invoked from the same host against smaller backup jobs that came back successfully.

    Thinking about it now, the disconnect was so fast it may have been caused by Veeam freezing the guest to back it up and it was that same guest that was receiving the data from invoke-command. I'll see if I can find the time to test this out and watch what phase the backup job is in when the disconnect happens. I may also try and "invoke it" from a guest that isn't being frozen at the time. I can't remember if that occurred to me or I tried either of those test back then though ... OutOfMemory. Neither approach would be very scientific though. A magic bullet that enables a debug log somewhere that doesn't take a week to work out what you are reading would be nice.

    Mark

    Thursday, November 14, 2013 5:52 PM
  • I may be wrong, it's worth to check the Timeout settings

    Get-Item WSMan:\localhost\Shell\IdleTimeout


    Regards Chen V [MCTS SharePoint 2010]

    Thursday, November 14, 2013 6:19 PM
  • Timeouts should be checked but you seem to have covered that.

    Logs are EventLog messages from WsMan WinRM and all other errors that may have occurred at the same time in any EventLog.


    ¯\_(ツ)_/¯

    Thursday, November 14, 2013 10:04 PM
  • I know this is an old post, but I just wanted to comment because we are having the same issue. Backing up 12 SQL instances simultaneously through 12 separate powershell remoting sessions.

    I have had several things in the past cause this issue. One is timing, the other is when an exception is thrown directly from the scriptblock in the invoke-command from a psremoting session.

    When an exception is thrown, depending on the timing, you may or may not get back the actual error message. It seems that it dumps the output streams fairly quickly, so you may or may not get the final error. I found that it greatly helps to disable compression when setting up the remoting session, as your chances of seeing the final error message increase tenfold.  Such as:

    $option = New-PSSessionOption -NoCompression
    $s = New-PSSession SomeComputer -SessionOption $option

    Also, when looping and displaying results from the src/destination sessions, I assume you are doing something like this:

    for($j=0; $j -lt $jobList.Count; $j++) { $jobInfo = $jobList[$j];

         Start-Sleep -Milliseconds 10;

        if($jobInfo.jobSrc.HasMoreData)
         {
          Start-Sleep -Milliseconds 10;
             $Host.PrivateData.VerboseForegroundColor = [ConsoleColor]::DarkRed.ToString();
             $Host.PrivateData.ErrorForegroundColor = [ConsoleColor]::Red.ToString();
             Receive-Job $jobInfo.jobSrc;
         }
    if($jobInfo.jobDest.HasMoreData) { Start-Sleep -Milliseconds 10; $Host.PrivateData.VerboseForegroundColor = [ConsoleColor]::DarkGreen.ToString(); $Host.PrivateData.ErrorForegroundColor = [ConsoleColor]::Green.ToString(); Receive-Job $jobInfo.jobDest; } }

    I found that if I do not include the Sleep, I get a number of errors. It would seem that "Receive-Job" and "HasMoreData" may have conflicting issues on some Powershell versions, so try adding some sleeps in and see what happens.




    • Proposed as answer by Brain2000 Thursday, July 18, 2019 2:55 AM
    • Edited by Brain2000 Thursday, July 18, 2019 9:33 PM
    Thursday, July 18, 2019 2:52 AM