Compiled Service hangs with 'command was found in the module'

This forum can be browsed by the general public. Posting is limited to current SAPIEN license holders with active maintenance and does not offer a response time guarantee.
Forum rules
DO NOT POST LICENSE NUMBERS, ACTIVATION KEYS OR ANY OTHER LICENSING INFORMATION IN THIS FORUM.
Only the original author and our tech personnel can reply to a topic that is created in this forum. If you find a topic that relates to an issue you are having, please create a new topic and reference the other in your post.

Any code longer than three lines should be added as code using the 'Select Code' dropdown menu or attached as a file.
This topic is 11 months and 3 weeks old and has exceeded the time allowed for comments. Please begin a new topic or use the search feature to find a similar but newer topic.
User avatar
CermakPOI
Posts: 41
Last visit: Wed Jan 31, 2024 2:57 am

Compiled Service hangs with 'command was found in the module'

Post by CermakPOI »

Product: PowerShell Studio 2023 (64 Bit)
Build: v5.8.218
OS: Windows 10 Enterprise (64 Bit)
Build: v10.0.19044.0

PowerShell version(s): 5.1.17763.3770

*** Please add details and screenshots as needed below. ***

We are working with the Powershell Studios Windows Service feature since 2017. 3 Services are currently in production on around 15 Servers and all of them are somtimes affected somtimes.

Sometimes (about every 2 to 3 weeks or not at all during a "WSUS-month") we get these kind of event log notifications on one of the servers:

Line 1908: The 'Start-Sleep' command was found in the module 'Microsoft.PowerShell.Utility', but the module could not be loaded. For more information, run 'Import-Module Microsoft.PowerShell.Utility'.

This occurs completely random, after the service is running for weeks or days or sometimes on service start and cannot be reproduced!
Also the commands and modules vary, but they are always builtin modules.

Other powershell scripts are working/running on these servers in the meantime without any issues.

Now the problem with that is:
The service an process is still seen as running from the windows point of view - but the service itself is completely hanging inside!
Which makes handling this issue with powershell code a little bit hard ;-) Especially since lots of build in cmdlets are "gone".

So we cannot use the service restart mechanisms from the builtin windows service management.

I know this is a not so easy, but maybe somebody has an idea?
User avatar
brittneyr
Site Admin
Posts: 1655
Last visit: Thu Mar 28, 2024 1:00 pm
Answers: 39
Been upvoted: 30 times

Re: Compiled Service hangs with 'command was found in the module'

Post by brittneyr »

We are looking into this and will get back to you soon with any information/ideas we have.
Brittney
SAPIEN Technologies, Inc.
User avatar
Alexander Riedel
Posts: 8479
Last visit: Thu Mar 28, 2024 9:29 am
Answers: 19
Been upvoted: 37 times

Re: Compiled Service hangs with 'command was found in the module'

Post by Alexander Riedel »

I was not able to readily reproduce this, most likely because it would require something more complex to run for a longer time than my test services.
I can make some guesses as to what would cause this though.
PowerShell Modules, and that includes internal ones that come with PowerShell, are no locked when in use as a DLL in a traditional compiled applications. So if a module gets updated, e.g. by Windows Update or some other mechanism while a running PowerShell script uses this module could potentially lead to such an effect. I have not verified this yet though.

Another reason for this can be a memory leak. This is a situation where memory gets allocated, e.g. by creating a new object, but never released. This can be in your code, a module you use or within PowerShell itself. Over time, this will make the memory usage of a process grow and fragment, creating all sorts of side effects. You could see that if you look at the memory usage of your service when these types of things happen and compare them to the usages after, say, a day of running.

To alleviate the problem I would suggest to simply restart all of these services any time a Windows update occurred. Also every Sunday at 1500 hours (or something like that).
This will show you if the runtime of a service is a factor and you could then look at the code more closely in that regard.
As a practical step, your code could on occasion check the system log if something happened in there affecting it and restart itself as necessary.

I will start some research into why internal cmdlets could suddenly disappear. If I find anything I will post it here.
Alexander Riedel
SAPIEN Technologies, Inc.
User avatar
Alexander Riedel
Posts: 8479
Last visit: Thu Mar 28, 2024 9:29 am
Answers: 19
Been upvoted: 37 times

Re: Compiled Service hangs with 'command was found in the module'

Post by Alexander Riedel »

This user here seems to experience a similar problem:
https://superuser.com/questions/1613630 ... ot-working
and here:
https://github.com/Azure/azure-function ... issues/377

It would be worthwhile investigating if you have any commonalities with the issues described in these posts.
Alexander Riedel
SAPIEN Technologies, Inc.
User avatar
CermakPOI
Posts: 41
Last visit: Wed Jan 31, 2024 2:57 am

Re: Compiled Service hangs with 'command was found in the module'

Post by CermakPOI »

After Windows update, the servers are restarted anyway. The times this happens do not really match.

The links you posted do not apply to our issue.

We adressed some memory leak scenarios inside this service with cleaning up vaiables and tehn initiating a garbage collection [System.GC]::GetTotalMemory($true)
Also a second "watchdog" service was created, which checks the custom built heartbeat and the memory usage and restarts this service if we think it's required. Since we id this, we are way below 1 GB ram usage for this service and the system ram usage is also not high.

The problem is: Powershell modules produce this issue system-wide - so the watchdog is also not able to perform it's task - since it required basic Powershell functions of course.
Also the Service doesn't just stop - which would make it very easy to restart automatically by windows internal methods, but it just hangs forever (until it's restartet externally).
If the exe-builder could catch such errors and just stop the service or exit or even just crash so the service/process gets triggered as stopped, that would be just great!
User avatar
Alexander Riedel
Posts: 8479
Last visit: Thu Mar 28, 2024 9:29 am
Answers: 19
Been upvoted: 37 times

Re: Compiled Service hangs with 'command was found in the module'

Post by Alexander Riedel »

The C# host cannot catch PowerShell exceptions, you can only do that in the PowerShell code itself.
I would encourage you at this point to contact Microsoft about this issue. Since it is a PowerShell issue they are much better equipped to assist you with this.
I will make a note and see if a "life sign" indicator to the PowerShell code on the inner layer can be added. I cannot promise that this is something that can be added quickly.
Alexander Riedel
SAPIEN Technologies, Inc.
This topic is 11 months and 3 weeks old and has exceeded the time allowed for comments. Please begin a new topic or use the search feature to find a similar but newer topic.