PSRemoting Failure Scenario

Ask your Windows PowerShell-related questions, including questions on cmdlet development!
Forum rules
Do not post any licensing information in this forum.

Any code longer than three lines should be added as code using the 'Select Code' dropdown menu or attached as a file.
Locked
User avatar
rene@nielsenhome.com
Posts: 13
Joined: Fri Sep 05, 2008 4:35 am

PSRemoting Failure Scenario

Post by rene@nielsenhome.com » Tue Nov 09, 2010 3:28 am

I'd like some input on a failure scenario involving PSRemoting.

Basic scenario:
I have an executable which interrogates a database for modular test specifications. By providing arguments to the executable I can ask it to run all or subsets of the tests. I have automated the remote execution of this executable using invoke-command of a PowerShell script which in turn executes a Start-Process of the executable with arguments.

Configuration details:
Domain: All systems are in the same domain and the executing process is using domain administrator credentials
Initiating server: Windows Server 2003 R2, fully patched
Client server: Windows Server 2008 R2, fully patched
Enable-PSRemoting has been run on all systems.
No PSSession option changes have been made, the defaults are involved so far in all diagnostics.

Problem Description:
When the executable is instructed (via the command line arguments) to remotely execute a certain number of test cases it fails with a "Not enough storage is available to complete this operation." error.

Diagnostic information:

Successful tests

When the same request (executable with the same command line arguments) is made LOCALLY (from a PowerShell prompt) on the remote client server (no remoting involved) the process completes successfully. This is regardless of the volume of tests selected.

When the subset of test cases is requested individually using REMOTING they succeed.

Failed tests

When remoting is involved and the volume of tests reaches a certain number (not huge, we're talking about around 7).

Failure behavior is consistent (the result that is, the exact failure point varies between servers and between runs on a particular server) when the test is run using remoting against different client systems (one running Windows Server 2008 R2, the other running Windows Server 2003 R2 32-bit)

Summary:

Remoting between these systems works but fails when the number of tests increases. All tests succeed when run locally.

The clue provided at runtime is "Not enough storage is available to complete this operation."

I have researched the PSSessionOptions but any promising settings I've come across seem to be set to unlimited by default.

If anyone has any suggestions I'd appreciate hearing them.

Thanks

User avatar
jhicks
Posts: 1789
Joined: Sun Jan 21, 2007 11:31 pm

PSRemoting Failure Scenario

Post by jhicks » Wed Nov 24, 2010 12:18 am

Is there any additional information from event logs? Although it seems clear your command is exhausting system resources. If you look at memory perf counters on the machine when trying this remotely, what do you see? There's overhead when using remoting which normally isn't a problem but your test tool seems to really push things beyond a breaking point. It sounds like you are trying to run something like this:test.exe arg1,arg2,arg3I wonder if you would have better luck this way:test.exe arg1test.exe arg2test.exe arg3

User avatar
rene@nielsenhome.com
Posts: 13
Joined: Fri Sep 05, 2008 4:35 am

PSRemoting Failure Scenario

Post by rene@nielsenhome.com » Wed Nov 24, 2010 1:34 am

Thanks for the response.

There are no records in the event logs around the time this process runs. It only runs for a couple minutes max before failing (last run was 1:40 minutes).

Monitoring memory usage shows that on this 2 Gb system the memory usage starts at a pre-run base of around 665 Mb and runs up to 791 Mb at which point the script fails. I have also collected an assortment of performance counters but have not been able to track a correlation to the storage exhaustion.

What is curious is that the entire collection of test (about 200) run to completion in a local process but the remoting fails after a small number (between 10 and 35). There is no consistency regarding the test that fails on each run.

For now I'm stuck with no smoking gun diagnostic that tells me what the mysterious storage is that is insufficient. I only suspect that a resource associated with the WS-Man may be the culprit. I've had no-joy in my research into possible configuration options there that might illuminate this one.

Let me know if you have any other areas I might look for clues or tests that I can run to narrow the field of possible causes.

Thanks.

User avatar
rene@nielsenhome.com
Posts: 13
Joined: Fri Sep 05, 2008 4:35 am

PSRemoting Failure Scenario

Post by rene@nielsenhome.com » Wed Nov 24, 2010 1:57 am

Thanks.

I have researched session limits but it appears that the ones specific to storage are set to unlimited by default. They seem only to apply to scenarios where you want to constrain resource usage for specific sessions. If someone knows of settings that might effect this that are otherwise send me a clue on it.

In this case the remote client system has over 10 Gb of free space, no user quotas involved. The output file for a successful run (redirected console output) is only 16 Kb. A failed output log is only around 5 Kb. The remoting host in this case has 25% of the disk free (about 5 Gb). Runtime monitoring shows no appreciable fluctuation in the physical disk free space.

jvierra
Posts: 13889
Joined: Tue May 22, 2007 9:57 am
Contact:

PSRemoting Failure Scenario

Post by jvierra » Wed Nov 24, 2010 2:17 am

The error is likely due to your paging file being filled up. CHeck paging file size and limit. RunPerfmon on paging file and see if it is filling up.



You should also note that your error message has a lot more information about the error. You have hidden that from us so understranding the exact nature of this error is mostly guesswork.

Please post full error text.
jvierra2010-11-24 09:21:38

jvierra
Posts: 13889
Joined: Tue May 22, 2007 9:57 am
Contact:

PSRemoting Failure Scenario

Post by jvierra » Wed Nov 24, 2010 3:14 am

Here is Microsoft Support on this mesage:

Code: Select all

Error Message:
Not enough storage is available to complete this operation.

User Action:
Do one of the following, then retry the operation: (1) reduce the number of running programs; (2) remove unwanted files from the disk the paging file is on and restart the system; (3) check the paging file disk for an I/O error; or (4) install additional memory in your system.

http://technet.microsoft.com/en-us/libr ... 78736.aspx

User avatar
jhicks
Posts: 1789
Joined: Sun Jan 21, 2007 11:31 pm

PSRemoting Failure Scenario

Post by jhicks » Wed Nov 24, 2010 4:19 am

The issue is not drive space but memory. You are on the right track. Your remote command is exceeding the remote shell memory limits. The max memory per remote shell is 150MB and it sounds like you are exceeding that. You can check the setting on the remote computer from an elevated PowerShell session.PS S:> (get-item WSMan:localhostShellMaxMemoryPerShellMB).Value150You could try setting a new value and trying again.set-item WSMan:localhostShellMaxMemoryPerShellMB -Value 1024This should raise the limit to 1GB. Not sure if you have to restart WinRM but it might not hurt.

User avatar
rene@nielsenhome.com
Posts: 13
Joined: Fri Sep 05, 2008 4:35 am

PSRemoting Failure Scenario

Post by rene@nielsenhome.com » Wed Nov 24, 2010 4:32 am

BINGO!

I did exactly as you recommended and re-ran the test which ran to completion (first time as a remote execution).

Much thanks to both of you for the helpful diagnostics to keep this going. I was completely stuck. You provide an invaluable resource and I know I am not alone in expressing how much your efforts are appreciated. Keep up the good work!

User avatar
rene@nielsenhome.com
Posts: 13
Joined: Fri Sep 05, 2008 4:35 am

PSRemoting Failure Scenario

Post by rene@nielsenhome.com » Wed Nov 24, 2010 6:57 am

Yes, setting the WSMAN shell max memory resolved the issue. Thanks again.

I am going to do some testing to see if this can be successfully dynamically manipulated via the Connect-WSMAN cmdlet or, failing that, at least do test runs to tune the shell max memory so it is a little more in line with what this process really needs (unfortunately this process runs tests which will continue to increase in number).

This is an internal software development test environment so the security aspects are a little less than if it were an exposed production setting.

Locked