Using Job Tracker

Ask questions about creating Graphical User Interfaces (GUI) in PowerShell and using WinForms controls.
Forum rules
Do not post any licensing information in this forum.

Any code longer than three lines should be added as code using the 'Select Code' dropdown menu or attached as a file.
This topic is 5 years and 10 months old and has exceeded the time allowed for comments. Please begin a new topic or use the search feature to find a similar but newer topic.
Locked
User avatar
jsira2003@yahoo.com
Posts: 117
Last visit: Tue Jul 11, 2023 6:18 am

Re: Using Job Tracker

Post by jsira2003@yahoo.com »

The good news and the bad news... The good news is I got the job working and it meets my objectives except for performance. It appears Jobs are much slower than the normal script execution. What can be done here to run a job as if it was the main script so I can get my performance back? Keep in mind my initial goal is to never have the application say "Not Responding". I gained some huge advantages in the while loop while waiting for the job completion. I definitely want to keep that.

In the code sample I check for running. This is fine for my application. Should the job fail it is ok despite the fact it is a long job.

This is just an example of my code and how the job is called.

Thanks for any help you can provide,
John

$tsb =
{
Function Totaler
{
param([int]$number, [int]$qty)
[int]$total = 0
[int]$i = 0
for($i = 0; $i -lt $qty; $i++)
{
$total += $number
}
return $total
}
}
cls
[int]$number = 5
[int]$qty = 5
[int]$total=0
$myjob = start-job -Name base -ScriptBlock { totaler $args[0] $args[1] } -InitializationScript $tsb -ArgumentList $number, $qty
while ((Get-Job -Name base).state -eq "Running")
{
[System.Windows.Forms.Application]::DoEvents()
#and perform a lot of other import stuff!
}
$total = receive-job $myjob
$total
get-job| remove-job
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

Re: Using Job Tracker

Post by jvierra »

Jobs are not slower that scripts. A job is just a script run in another PS process.

For very, very short scrips the job will be slower because the setup time can be as long as the script execution time. Jobs are meant to be used for long running scripts.

Your test script is to short to be used in a job effectively.
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

Re: Using Job Tracker

Post by jvierra »

Why are you not using the Job Tracker as suggested?
User avatar
jsira2003@yahoo.com
Posts: 117
Last visit: Tue Jul 11, 2023 6:18 am

Re: Using Job Tracker

Post by jsira2003@yahoo.com »

Well in fact the job I have to run is quite long. before the job it took 10 hours for the script to run. Now it takes 14 after the job. In the example given I have shown the hand-off to a function. I read that a job run's in a lower priority. The fact that it now takes an extra 5 hours to do the same thing after the job suggests a slow down. The job does run correctly just slower. The actual job does a directory enumeration. This is a single line and there there is some post processing in the resulting array list that follows. The key here is that the function did not change. What changed is that it is now a job.

I looked into the job tracker but didn't understand it. I did spend some time studying it but If I can figure how to implement it in my solution I needed to look elsewhere. I used what I could understand and make work. I also tried runspaces and got the same slow results.
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

Re: Using Job Tracker

Post by jvierra »

You have a loop that is wasting time.

while ((Get-Job -Name base).state -eq "Running")

The repeated Get-Job and DoEvents is killing the process efficiency. The JobTracker doe not do this and does not place focus on the job repeatedly. Use JobTracker and set the time to one minute and the performance will be much better.
User avatar
jsira2003@yahoo.com
Posts: 117
Last visit: Tue Jul 11, 2023 6:18 am

Re: Using Job Tracker

Post by jsira2003@yahoo.com »

I want that loop and I will put many commands in there to alter processing, ask the user if they want to exit or abort the job etc, etc, etc. That loop is needed and it important. However that loop should not affect the speed to the job that is created. The job should be a separate entity. The do events is important to make sure no events queue up with the function hand-off. I need that loop. Even though you don't see all the commands I need in there so assume it is important. Any other ideas I try must work similarly to the job example I gave. I need to create a job and let it run. I want to wait for it but allow other things to occur in the loop to alter the processing as necessary.

thank you,
John
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

Re: Using Job Tracker

Post by jvierra »

If you use JobTracker you can do all of this in the UpdateScript and it will be much faster.

\The job is a separate entity until you call Get-Job in a tight loop. "DoEvents" is not good way to do what you are trying to do. The JobTracker article explains this and is one reason the JobTracker was created.

Use the JobTracker. It will do all you need and the job will run quickly assuming the code in the job is written correctly. A script that makes repeated network calls in the background will always run more slowly when the network or called service is busy. You cannot control that easily.

Without knowing what you job is doing it is not possible to know how to direct you.

I can think of no data gathering job that should take 24 hours. It sounds like you are doing something using the wrong tools or in the wrong way.

What is it that you are trying to do.
User avatar
jsira2003@yahoo.com
Posts: 117
Last visit: Tue Jul 11, 2023 6:18 am

Re: Using Job Tracker

Post by jsira2003@yahoo.com »

Dare I make one more comment! I removed the do nothing loop and after the job just put a wait-job statement. I used a smaller data set. I am sure this code is correct. Both produce the same result. The job is much slower. I hope this shows the complete picture.

thanks,
John

without the job - 4 plus minutes
Function Baseline([string]$server, [string]$shareDir, [string]$fileMask, [string]$recurse, [string]$serverLogBaseline)
{
[int]$count = 0

[array]$array = $fileMask.split(",")

#
#was recurse selected?
#
#repoint psdrive to file system
Set-Location c:

IF ($recurse -EQ "Y")
{
[System.Collections.ArrayList]$files = get-childitem -recurse -force -Path $($shareDir) -include $array -ErrorAction SilentlyContinue | select @{ Name = "Server"; Expression = { $server } }, @{ Name = "ShareDir"; Expression = { $ShareDir } }, fullname, name, length, @{ Name = "Hash"; Expression = { get-filehash $_.fullname -Algorithm md5 | select -ExpandProperty hash } }, lastwritetime -ErrorAction SilentlyContinue
}
ELSE
{
IF ($recurse -eq "N")
{
[System.Collections.ArrayList]$files = get-childitem -force -Path $($shareDir + "\*") -include $array -ErrorAction SilentlyContinue | select @{ Name = "Server"; Expression = { $server } }, @{ Name = "ShareDir"; Expression = { $ShareDir } }, fullname, name, length, @{ Name = "Hash"; Expression = { get-filehash $_.fullname -Algorithm md5 | select -ExpandProperty hash } }, lastwritetime -ErrorAction SilentlyContinue
}
}

for ($i = 0; $i -lt $files.Count; $i++)
{

#is the record complete to add to the files database
IF ($files[$i].name -ne "Thumbs.db" -and $files[$i].hash -and !(Extended $files[$i].fullname) -and $files[$i].name -notmatch '~' -and $files[$i].length -ne "0" -and $files[$i].name -notmatch '\+')
{

#special case, do not create hash for the file I am currently writing to
IF ($serverLogBaseline -ne $files[$i].fullname)
{

$count++
}
ELSE
{
$files.RemoveAt($i)
$i--
}

}
ELSE
{
$files.RemoveAt($i)
$i--
}
}
#
# must add \* to the end of the path. will allow non recurse to work. With the \* the recurse parameter can
# be added or removed as needed.
#
return [array]$files, [int]$count
}


$completeFileList, $fileCount = Baseline $server.server $sharePath.sharepath $sharePath.FileMask $sharePath.recurse $serverLogBaseline

versus************************************


with the job - 16 plus minutes
$baselineSB = {
Function Baseline
{
param ([string]$server,
[string]$shareDir,
[string]$fileMask,
[string]$recurse,
[string]$serverLogBaseline)


[int]$count = 0

[array]$array = $fileMask.split(",")

#
#was recurse selected?
#
#repoint psdrive to file system
Set-Location c:

IF ($recurse -EQ "Y")
{
[System.Collections.ArrayList]$files = get-childitem -recurse -force -Path $($shareDir) -include $array -ErrorAction SilentlyContinue | select @{ Name = "Server"; Expression = { $server } }, @{ Name = "ShareDir"; Expression = { $ShareDir } }, fullname, name, length, @{ Name = "Hash"; Expression = { get-filehash $_.fullname -Algorithm md5 | select -ExpandProperty hash } }, lastwritetime -ErrorAction SilentlyContinue
}
ELSE
{
IF ($recurse -eq "N")
{
[System.Collections.ArrayList]$files = get-childitem -force -Path $($shareDir + "\*") -include $array -ErrorAction SilentlyContinue | select @{ Name = "Server"; Expression = { $server } }, @{ Name = "ShareDir"; Expression = { $ShareDir } }, fullname, name, length, @{ Name = "Hash"; Expression = { get-filehash $_.fullname -Algorithm md5 | select -ExpandProperty hash } }, lastwritetime -ErrorAction SilentlyContinue
}
}

for ($i = 0; $i -lt $files.Count; $i++)
{

#is the record complete to add to the files database
IF ($files[$i].name -ne "Thumbs.db" -and $files[$i].hash -and !(Extended $files[$i].fullname) -and $files[$i].name -notmatch '~' -and $files[$i].length -ne "0" -and $files[$i].name -notmatch '\+')
{

#special case, do not create hash for the file I am currently writing to
IF ($serverLogBaseline -ne $files[$i].fullname)
{

$count++
}
ELSE
{
$files.RemoveAt($i)
$i--
}

}
ELSE
{
$files.RemoveAt($i)
$i--
}
}
#
# must add \* to the end of the path. will allow non recurse to work. With the \* the recurse parameter can
# be added or removed as needed.
#
return [array]$files
}
}


$myjob = Start-Job -name base -ScriptBlock { Baseline $args[0] $args[1] $args[2] $args[3] $args[4] } -InitializationScript $baselineSB -ArgumentList $server.server, $sharePath.sharepath, $sharePath.FileMask, $sharePath.recurse, $serverLogBaseline
sleep 2
Wait-Job -Name base

I should have mentioned that an array gets returned. The array almost has 6,000 elements.
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

Re: Using Job Tracker

Post by jvierra »

If you are parked in an event in a loop or sitting at a wait then the main process will get a higher priority in thread selection. This includes threads in all child processes. The job tracker does not use any techniques that raise the priority of the main process. This can allow all threads a more equal priority.
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

Re: Using Job Tracker

Post by jvierra »

I would also note that your code is mysteriously over worked. You should not do selections and formatting in the job. Just get the data. Once you have acquired the data you can filter it in the main. The constant conversion in the select will slow the job down.

It would help if you posted formatted code as what you have posted is mostly unreadable. If their is a lot of code post a files as an attachment.
This topic is 5 years and 10 months old and has exceeded the time allowed for comments. Please begin a new topic or use the search feature to find a similar but newer topic.
Locked