Start-Job hangs Powershell on Windows 2016

Ask your Windows PowerShell-related questions, including questions on cmdlet development!
Forum rules
Do not post any licensing information in this forum.

Any code longer than three lines should be added as code using the 'Select Code' dropdown menu or attached as a file.
User avatar
akincer
Posts: 23
Joined: Fri Mar 28, 2014 10:05 am

Start-Job hangs Powershell on Windows 2016

Post by akincer » Wed Oct 02, 2019 8:45 am

Not sure if anyone else has seen this, but there's what must be a bug that occasionally causes the parent thread to hang when trying to run Start-Job and if you're starting a bunch of jobs on a loop you'll probably hit it for sure. If you're hitting this and are looking for a workaround, I have one but it's a bit complicated.

Create a new sub-thread and launch the job from there. Here's some sample code. There are probably several different ways to detect when the start-job process fails.

Code: Select all

$ChildScriptBlock = {
	$Parameters = $args[0]
	
	$log = '<your path to log files>\' + $Parameters["jobName"] + '.txt'
	"Child Script: Got " + $Parameters["P1"] + " and " + $Parameters["P2"] + " With fallback of " + $Parameters["Fallback"]>> $log
	#$Parameters["P1"] >> $log
	
	
}

$ParentScriptBlock = {
	$ParentParameters = $args[1]
	$ChildScriptBlock = $ParentParameters["ChildScriptBlock"]
	$log = '<your path to log files>\' + $ParentParameters["jobName"] + '.txt'
	#"Got " + $ParentParameters["P1"] + " and " + $ParentParameters["P2"] >> $log
	$ChildParameters = @{
		Fallback = "fallback";
	}
	foreach ($key in $ParentParameters.Keys)
	{
		if ($key -ne "ChildScriptBlock")
		{
			#"Adding key $key to ChildParameters" >> $log
			$ChildParameters[$key] = $ParentParameters[$key]
		}
	}
	$job = Start-Job -Name $ParentParameters["jobName"] -ScriptBlock $ChildScriptBlock -ArgumentList $ChildParameters
}

$ParentParameters = @{
	P1 = "Parameter 1";
	P2 = "Parameter 2";
	ChildScriptBlock = $ChildScriptBlock;
}

$runspacePool = [runspacefactory]::CreateRunspacePool()
$runspacePool.Open()
$taskList = @()
for ($jobnumber = 1; $jobnumber -lt 200; $jobnumber++)
{
	$ParentParameters["jobName"] = "job_$jobnumber"
	$newThread = [powershell]::Create().AddScript($ParentScriptBlock).AddParameter('ParentParameters', $ParentParameters)
	$newThread.RunspacePool = $runspacePool
	$handle = $newThread.BeginInvoke()
	$taskList += New-Object -TypeName psobject -Property @{ "Handle" = $handle; "Thread" = $newThread; }
	Start-Sleep 1
}

$taskList | Where-Object { $_.Handle.IsCompleted } | ForEach-Object {
	$_.Thread.EndInvoke($_.Handle)
	$_.Thread.Dispose()
	$_.Thread = $_.Handle = $Null
}
From there you can deal with the threads in taskList. Most of the code to deal with threads was pulled from the start-parallel module. The rest I threw together.

Anyone else seeing this with Windows 2016?

User avatar
akincer
Posts: 23
Joined: Fri Mar 28, 2014 10:05 am

Re: Start-Job hangs Powershell on Windows 2016

Post by akincer » Thu Oct 03, 2019 3:11 pm

Using process explorer, I discovered that if I kill the sub-process that was spun up when it hung the parent thread takes off again. So it looks like it's happening during the spinning up of the Powershell instance it's creating.

Also -- I'm not really able to recreate this on a Windows 2008 box with WMF 5.1 installed. The Powershell versions are slightly different though, so not sure if there's a bug fix for this between versions or if it's completely specific to 2016.

WMF 5.1 2008 PS Version -- 5.1.14409.1005
WMF 5.1 2016 PS Version -- 5.1.14393.3053

Anyone out there with access to Windows Server 2016 that you can run some test code, please see if Powershell hangs at some point during this:

Code: Select all

$ScriptBlock = {
	$test = "blah"
}
$arguments = "Whatever args you want"
for ($j = 1; $j -lt 500; $j++)
{
	Write-Host "Starting job $j"
	$job = Start-Job -Name "Job_$j" -ScriptBlock $ScriptBlock -ArgumentList $arguments
	Write-Host "Finished starting job $j"
	Start-Sleep 1
	$jobs = Get-Job
	while ($jobs.Count -ge 6)
	{
		Get-Job | Remove-Job -ErrorAction SilentlyContinue
		Start-Sleep 1
		$jobs = Get-Job
	}
}

User avatar
jvierra
Posts: 13793
Joined: Tue May 22, 2007 9:57 am
Contact:

Re: Start-Job hangs Powershell on Windows 2016

Post by jvierra » Thu Oct 03, 2019 3:27 pm

Remove the following and you will begin to see the issue.

[b}-ErrorAction SilentlyContinue[/b]

User avatar
jvierra
Posts: 13793
Joined: Tue May 22, 2007 9:57 am
Contact:

Re: Start-Job hangs Powershell on Windows 2016

Post by jvierra » Thu Oct 03, 2019 4:00 pm

Here is the safe way to do this. The diffe3rence between 2016 and other systems is mostly due to system timing disrupted by your ignored exceptions.

Code: Select all

for ($j = 1; $j -lt 500; $j++){
	$job = Start-Job -ScriptBlock {sleep (Get-Random  -Maximum 30 -Minimum 5)} -ArgumentList Nothing
	Write-Host 'Finished starting job' $job.Name -Fore green
	do{
		Get-Job -State Completed | Remove-Job -Verbose
	}
	while((Get-Job).Count -ge 6)
}

User avatar
akincer
Posts: 23
Joined: Fri Mar 28, 2014 10:05 am

Re: Start-Job hangs Powershell on Windows 2016

Post by akincer » Fri Oct 04, 2019 10:56 am

jvierra wrote:
Thu Oct 03, 2019 3:27 pm
Remove the following and you will begin to see the issue.

[b}-ErrorAction SilentlyContinue[/b]
No. This is not the issue.

To be more clear -- it's not the cleanup of completed jobs that's the issue. Powershell hangs on STARTING a job. I can literally run PSExec and see the hung instance sitting there doing nothing and killing it causes the parent thread to start back up. Diagnostic output shows that it hung DURING the spin-up of the job.

User avatar
jvierra
Posts: 13793
Joined: Tue May 22, 2007 9:57 am
Contact:

Re: Start-Job hangs Powershell on Windows 2016

Post by jvierra » Fri Oct 04, 2019 11:06 am

Use my code to understand why this is happening. When you create 500 processes they will eventually get in trouble and may seem hung but are really just not getting scheduled enough to keep them running.
No one would every crate 500 processes with PowerShell because that would exceed the session limits or create a scheduling issue. If you need hundreds of tasks then use a "RunSpaceFactory" and many "RunSpaces". These can be throttled. The easiest way to do this is to use a Workflow that runs the task in parallel.

User avatar
akincer
Posts: 23
Joined: Fri Mar 28, 2014 10:05 am

Re: Start-Job hangs Powershell on Windows 2016

Post by akincer » Fri Oct 04, 2019 11:20 am

jvierra wrote:
Fri Oct 04, 2019 11:06 am
Use my code to understand why this is happening.
This. Is. Not. The. Issue. Sometimes it fails at job #6, sometimes at job # 100. Most of the time I can see with PSExec there is only one child Powershell process running. I've even had this fail on the very first job.

User avatar
akincer
Posts: 23
Joined: Fri Mar 28, 2014 10:05 am

Re: Start-Job hangs Powershell on Windows 2016

Post by akincer » Fri Oct 04, 2019 11:28 am

jvierra wrote:
Fri Oct 04, 2019 11:06 am
The easiest way to do this is to use a Workflow that runs the task in parallel.
I haven't gotten into workflows. Is that truly multi-threaded so I can saturate the cores on a machine? The code blocks I need to run varies in runtime and the scale of operations, but I need to squeeze as much out of the system as possible.

At the end of the day I don't care how I accomplish this as long as it gets the maximum performance. But there's still something amiss with starting a bunch of jobs at least on two different 2016 machines I have here.

User avatar
jvierra
Posts: 13793
Joined: Tue May 22, 2007 9:57 am
Contact:

Re: Start-Job hangs Powershell on Windows 2016

Post by jvierra » Fri Oct 04, 2019 11:29 am

This is not a scripting issue if it fails on the first job. You need to fix your systems. Of course this all depends on what script is running when it fails.

User avatar
akincer
Posts: 23
Joined: Fri Mar 28, 2014 10:05 am

Re: Start-Job hangs Powershell on Windows 2016

Post by akincer » Fri Oct 04, 2019 11:46 am

I've said from the start I think it's a bug specific to Windows Server 2016 but specifically wanted to see if anyone else saw this behavior to try to figure out if it is specific to my systems.

Interestingly if I use Invoke-Command to create jobs locally but treated as remote jobs this problem doesn't happen. It's specific to Start-Job.

Post Reply