Answered by:
Invoke-RestMethod not disposing of connections properly?

Question
-
I am working with Invoke-RestMethod to do some automation. The first call to the target system returns an XML response that contains a collection of items that I want to do some operations on, so I then iterate through that list of objects and make an Invoke-RestMethod call for each of those items.
The initial call to get the object works just fine, as does the call during the first iteration. On subsequent iterations however the script stalls and eventually I get an error:
Invoke-RestMethod : The operation has timed out
At C:\test\zfa-branch\CreateTCProject.ps1:16 char:2
+ Invoke-RestMethod -Credential $tclogin -Verbose -uri "$serverUrl$buildHref/paus ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : System.Net.WebException,Microsoft.PowerShell.Commands.InvokeRestMethodCommandDoing some research this exact same problem appears to happen when using the HttpWebRequest .Net object and not disposing of the request objects properly (http://stackoverflow.com/questions/5827030/httpwebrequest-times-out-on-second-call). There is not an obvious way to get the Invoke-RestMethod to use the dispose methods using any of the cmdlet's parameters so I added a "Start-Sleep -s 6" and then "[GC]::Collect()" statements to try and force the issue. Then MOST of the calls while iterating succeed and I only time out on 1 or 2 out of 10 iterations. Using the Start-Sleep call alone does not resolve the issue, at least at values up to 30 seconds.
Should I give up on Invoke-RestMethod and use the old WebRequest way of doing things? Is Invoke-RestMethod not disposing objects properly in the background? Other ideas welcomed.
Here's a snippet of the script and if it matters to anyone, the automation is for a TeamCity CI build server.
$tclogin = Get-Credential $serverUrl = "http://some.server.com:9999" $projectName = "Some Project" $shareRoots = $true [System.Reflection.Assembly]::LoadWithPartialName("System.web") $projectNameEncoded = [System.Web.HttpUtility]::UrlPathEncode($projectName) $sourceProject = Invoke-RestMethod -Credential $tclogin -Verbose -uri $serverUrl/httpAuth/app/rest/projects/$projectNameEncoded/buildTypes -Method get foreach ( $buildType in $sourceProject.buildTypes.buildType ) { $buildHref = $buildType.href Invoke-RestMethod -Credential $tclogin -Verbose -uri $serverUrl$buildHref/paused -Method put -Body "true" -ContentType "text/plain" Start-Sleep -s 6 [GC]::Collect() }
Thanks!
- Moved by Bill_Stewart Thursday, August 22, 2013 9:51 PM Abandoned thread
Tuesday, August 6, 2013 5:59 PM
Answers
-
I know this thread is dead, but for completeness, I have a solution to this
Before you call Invoke-Restmethod add this or similar and replace MyURI with your own URI variable:
$ServicePoint = [System.Net.ServicePointManager]::FindServicePoint($MyURI)
Call your Invoke-RestMethod.
After you call it, you can use the method below to close the connections you have.
$ServicePoint.CloseConnectionGroup("")
This is quite brute force so could disconnect things that you don't want disconnected!
The ServicePointManager has a bunch of methods so you can do other things too.
Wednesday, January 8, 2014 4:43 PM -
This has been moved to the dustbin of scripting history because it is not a scripting issue. It is a break/fix issue as you have already proven. It is likely that the call to the web site is causing the session to become corrupted because the service is not compatible with what you are trying to do.
¯\_(ツ)_/¯
Tuesday, January 7, 2014 3:08 PM
All replies
-
You should not be calling the garbage collector in a loop. It is an async process. It will run when needed so why are you executing it. I suspect the repeated calls are causing some unknown issue to occur.
It is also possible that the service does no like being called so frequently. I recommend using the more flexible WebService methods.
¯\_(ツ)_/¯
Tuesday, August 6, 2013 7:04 PM -
Thanks for the reply!
I was executing garbage collection based on various threads talking about the similar problems, and that it did at least partially help to resolve the issue seemed at least somewhat telling.
The service is fine with calls at that frequency as I've been able to replicate the behavior I want by calling to curl.exe instead of Invoke-RestMethod on my local machine, however for irrelevant and complicated reasons that's not really an option for the script implementation.
To disambiguate your reference to "WebService methods" are you referring to the HttpWebRequest .Net classes, the New-WebServiceProxy cmdlet, or something else?Tuesday, August 6, 2013 9:43 PM -
New-WebServiceProxy is the one that is easiest to use and eliminates repeated logins.
Once you have the proxy you can just call methods directly.
¯\_(ツ)_/¯
Tuesday, August 6, 2013 9:51 PM -
You should also try using the sessionvariable parameter to prevent the constant re-auth.
¯\_(ツ)_/¯
Tuesday, August 6, 2013 9:58 PM -
Hi Nathan I have what sounds like the exact same problem. I can call a rest method 3 times before my script hangs. I have tried using the session variable to prevent re-auth and the problem remains. Have you managed to get this working? Thanks in advance.Tuesday, December 24, 2013 12:45 PM
-
Hi Nathan I have what sounds like the exact same problem. I can call a rest method 3 times before my script hangs. I have tried using the session variable to prevent re-auth and the problem remains. Have you managed to get this working? Thanks in advance.
Another thing that needs to be noted. Timeouts are usually caused by the web server or the network. Quick repeated calls to a service may cause the web server to freeze for a short time. If the server is busy it can also cause a timeout. You should first look at the server logs to see what is happening.
Garbage collection here is not a solution.
¯\_(ツ)_/¯
Tuesday, December 24, 2013 5:11 PM -
Hi
Its not the network for a few reasons:-
1. It is ALWAYS the third call that hangs and times out
2.The HTTP request never hits the web server which underpins the REST API at the third attempt
3. No HTTP requests leaves my PC when the invoke-restmethod is called for the third time. (using Wireshark to verify this)
4. It really doesn't smell like a network or server side issue.
Tuesday, January 7, 2014 1:08 PM -
This has been moved to the dustbin of scripting history because it is not a scripting issue. It is a break/fix issue as you have already proven. It is likely that the call to the web site is causing the session to become corrupted because the service is not compatible with what you are trying to do.
¯\_(ツ)_/¯
Tuesday, January 7, 2014 3:08 PM -
You are right, its not a scripting issue per se. It seems to be a windows issue. I wont say problem as it appears to be by design, but in the case of Invoke-restmethod is is very limitting. I am looking for a workaround for it.
More details here:
http://stackoverflow.com/questions/5827030/httpwebrequest-times-out-on-second-call
Wednesday, January 8, 2014 1:29 PM -
I know this thread is dead, but for completeness, I have a solution to this
Before you call Invoke-Restmethod add this or similar and replace MyURI with your own URI variable:
$ServicePoint = [System.Net.ServicePointManager]::FindServicePoint($MyURI)
Call your Invoke-RestMethod.
After you call it, you can use the method below to close the connections you have.
$ServicePoint.CloseConnectionGroup("")
This is quite brute force so could disconnect things that you don't want disconnected!
The ServicePointManager has a bunch of methods so you can do other things too.
Wednesday, January 8, 2014 4:43 PM -
I got to this, because I got a HTTP - 505 error while connecting with INVOKE-RestMethod from Win 2012 R2 powershell 4 to Apache Tomcat server on Cisco Identity Services Engine (ISE).
The connection will stay in "CLOSE_WAIT" until the powershell session will close.
Maybe this only happens with Apache Tomcat. see JBoss Forum, Microsoft Forum
There are some possibilities:
- Set $ServicePoint.ConnectionLimit to an higher value (as fiddler does)
- Use Invoke-WebRequest (as MS Connect suggest as Workaround for delete and put)
I could not reproduce the $ServicePoint.Expect100Continue=$false - It still ends in a timeout because of connections in close_wait status.
Patrick Wahlmüller
- Edited by Patrick WahlmüllerMVP Monday, June 2, 2014 10:22 AM
Monday, June 2, 2014 10:22 AM -
Thank you! This worked for me to make the servicepointmanager calls as directed by LanceC1 in order to avoid re-writing the scripts to use invoke-webrequest... This was a problem when attempting to perform multiple DELETE calls with invoke-restmethod under the same script session... the calls would timeout, and I kept thinking something was wrong in the code, but clearly this falls under the 'known bug' category... No additional issues so far since adding the before and after calls with ServicePointManager. :) ... I ad no issues of course with multiple 'POST' method calls, so there is no need for the additional spm calls there...
Tuesday, May 3, 2016 2:29 PM -
Thanks! This solved my problem too.
Invoke-restmethod would timeout after two deletes but invoke-webrequest worked like a charm!
Tuesday, May 3, 2016 7:30 PM -
This solution worked for me too. And just to clarify since I had the question after reading this-- you don't need to modify your Invoke-RestMethod call at all. Simply at the servicepoint calls before and after the Invoke-RestMethod, using the same URI obviously, and all is fixed.Tuesday, September 25, 2018 6:06 PM