Start-Job hangs Powershell on Windows 2016

Ask your Windows PowerShell-related questions, including questions on cmdlet development!
Forum rules
Do not post any licensing information in this forum.

Any code longer than three lines should be added as code using the 'Select Code' dropdown menu or attached as a file.
jvierra
Posts: 14018
Joined: Tue May 22, 2007 9:57 am
Contact:

Re: Start-Job hangs Powershell on Windows 2016

Post by jvierra » Fri Oct 04, 2019 12:01 pm

Again - for bugs post in UserVoice. You can search there to see if anyone else has posted this issue.

I would also ask why you think you know the job is "hung". "Suspended" does not mean "hung". It means the job is waiting for something like a file or memory or the network.

Remember that PowerShell is throttled. It can only use a certain amount of resources and it will seem to stop while waiting to clean up memory or for handles to be available.

Another thing you need to be sure about is that you have the latest Net Framework installed.

PowerShell and the Net Framework on Win10 and WS2016 are identical. They are not different versions although the build number can be different for various trivial reasons.

User avatar
akincer
Posts: 23
Joined: Fri Mar 28, 2014 10:05 am

Re: Start-Job hangs Powershell on Windows 2016

Post by akincer » Fri Oct 04, 2019 12:19 pm

When I say "hung" I mean all execution stops. I do not know what the underlying source of suspended execution might be. Ending the child process that the parent process is clearly waiting on something before continuing causes it to pick right up where it left off. Even with extremely simple code it doesn't execute a single line of it when a start-job goes awry for whatever that's worth.

I can insert any number of throttles to wait for cleanup to happen. I've tried many. Running on PS 5.1 on 2008 results in as-expected behavior. Running on 2016 does not. Same code. Same version of the framework (with a super minor point difference). Polar opposite results.

jvierra
Posts: 14018
Joined: Tue May 22, 2007 9:57 am
Contact:

Re: Start-Job hangs Powershell on Windows 2016

Post by jvierra » Fri Oct 04, 2019 12:23 pm

What command does it stop on?

jvierra
Posts: 14018
Joined: Tue May 22, 2007 9:57 am
Contact:

Re: Start-Job hangs Powershell on Windows 2016

Post by jvierra » Fri Oct 04, 2019 12:44 pm

I am pretty certain that this will fix your issue.

Code: Select all

$ScriptBlock = { 
	$test = "blah" 
} 
$arguments = "Whatever args you want" 
for ($j = 1; $j -lt 200; $j++){ 
	Start-Sleep 1 
	Get-Job -State Completed| 
		Select-Object -skip 6 |
		Remove-Job 
	Write-Host "Starting job $j" 
	Start-Job -Name "Job_$j" -ScriptBlock $ScriptBlock -ArgumentList $arguments 
	Write-Host "Finished starting job $j" 
}
Be sure to run this exact code without changes or you may re-introduce the race condition from your original code.

User avatar
akincer
Posts: 23
Joined: Fri Mar 28, 2014 10:05 am

Re: Start-Job hangs Powershell on Windows 2016

Post by akincer » Fri Oct 04, 2019 2:21 pm

For now I've just decided to use PoshRSJob and keep it all in runspaces. I cannot seem to replicate it, but I had BLISTERING speed using Invoke-Command as a job in one test. I've got acceptable performance with PoshRSJob. Now it seems to be a matter of finding the right balance of throttling concurrent "jobs" per logical core in the runspace settings. The code is NOT particularly CPU intensive at any one moment because it makes REST calls to an API that slow execution down.

BTW -- I ran your code. Same result. Here's where it stopped this time:

Starting job 93
185 Job_93 BackgroundJob Running True localhost ...
Finished starting job 93
Starting job 94

It always stops execution during the launching of a job.

jvierra
Posts: 14018
Joined: Tue May 22, 2007 9:57 am
Contact:

Re: Start-Job hangs Powershell on Windows 2016

Post by jvierra » Fri Oct 04, 2019 3:14 pm

Again - how can you tell it is hung? The job you are trying to start is a PowerShell instance. That instance is not starting. As a result the Start-Job is waiting for it to complete. You will have to use something like ProcessExplorer to find out state the PS instance is in.

Have you checked you Net Framework for all updates and run the diagnostics to be sure it is working correctly? All of this code is part of the Net Framework.

The only way this can happen is if the system fails to start the PowerShell instance.

jvierra
Posts: 14018
Joined: Tue May 22, 2007 9:57 am
Contact:

Re: Start-Job hangs Powershell on Windows 2016

Post by jvierra » Fri Oct 04, 2019 4:33 pm

Try this as it will diagnose a different race condition.

Code: Select all

$ScriptBlock = { 
	sleep (Get-Random -Minimum 5 -Maximum 30)
	'Exited'
} 
$arguments = "Whatever args you want" 
for ($j = 1; $j -lt 200; $j++){ 
	Start-Sleep 1 
	Get-Job -State Completed| 
		Select-Object -skip 6 |
		Remove-Job 
	Write-Host "Starting job $j" 
	Start-Job -Name "Job_$j" -ScriptBlock $ScriptBlock -ArgumentList $arguments 
	Write-Host "Finished starting job $j" 
}

User avatar
akincer
Posts: 23
Joined: Fri Mar 28, 2014 10:05 am

Re: Start-Job hangs Powershell on Windows 2016

Post by akincer » Mon Oct 07, 2019 10:53 am

I can tell it's "hung" because execution in the parent thread stops.

That code stopped executing where it always does -- starting a job. This time on job 5.

Using process explorer:

powershell.exe parent process state: Wait:UserRequest
conhost.exe parent process state: Wait:UserRequest
powershell.exe child process state: Wait:Suspended
conhost.exe child process state: Wait:UserRequest

It looks like the wheels are coming off on the spinning up of the conhost.exe child process as the only thread listed for it is conhostV2.dll!ConsoleCreateIoThread whereas the conhost.exe for the parent has a few more.

These servers are quite up to date so it's not that. I can spin up a fresh install to test against when I get some time.

User avatar
akincer
Posts: 23
Joined: Fri Mar 28, 2014 10:05 am

Re: Start-Job hangs Powershell on Windows 2016

Post by akincer » Mon Oct 07, 2019 12:12 pm

Figured out why jobs using Invoke-Command were going out to lunch and it's embarrassingly silly. I had to do this because due to how multi-threading is achieved using runspaces, REST API requests were stepping all over each other. The performance boost for this use case using jobs instead of runspaces is staggering.

Of course still don't know why start-job misbehaves on these systems while "remote" jobs do not.

Locked