Windows server 2019 error when running task with C# script

Octopus Server
1

We have windows server 2016 servers that work fine during our deployments. We have one 2019 server that fails any time it runs into a step that includes a C# script.

The error I get is: ERROR: Script execution failed. [COMException] No logical space left to create more user strings. (Exception from HRESULT: 0x80131198)

To reproduce I have added a simple script step, select run on deployment target, select C#, used the simple script below:
Console.WriteLine(“Testing C# script”);

If I run this through the Script Console for the same deployment target it works fine.
The tentacle service runs under Local System for both the 2016 and 2019 servers.

I have tried installing .net runtimes and sdks for framework, core, and the newer .net. I’ve tried running the service as an admin user. None of these worked.

I did have to install at least .net 4.8 to get around the error mentioned here: Steps run on Win2019 are terminated on the first error · Issue #5755 · OctopusDeploy/Issues · GitHub.

Here is the full raw log from when it occurs:
| == Failed: Deploy Test release 1.208.16-47123.16.3 to Test ==
09:19:23 Verbose | Step 2: Confirm Deployment does not apply to the current environment, and will not be executed
09:20:44 Fatal | The deployment failed because one or more steps failed. Please see the deployment log for details.
|
| == Failed: Step 1: Test Script ==
09:19:57 Fatal | The step failed: Activity Test Script on Test failed with error ‘The remote script failed with exit code 1’.
09:19:57 Verbose | Test Script completed
|
| == Failed: Test ==
09:19:29 Verbose | Octopus Server version: 2021.3.8275
09:19:29 Verbose | Environment Information:
| IsRunningInContainer: False
| OperatingSystem: Microsoft Windows 10.0.14393
| OsBitVersion: x64
| Is64BitProcess: True
| CurrentUser: NT AUTHORITY\SYSTEM
| MachineName: OctoTest
| ProcessorCount: 8
| CurrentDirectory: C:\Windows\system32
| TempDirectory: C:\Windows\TEMP
| HostProcessName: Octopus.Server
| PID: 4400
09:19:29 Verbose | Executing Test Script (type Run a Script) on Test
09:19:29 Verbose | Script isolation level: NoIsolation
09:19:29 Verbose | Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-217083
09:19:29 Verbose | Executable directory is C:\windows\system32\WindowsPowershell\v1.0
09:19:29 Verbose | Executable name or full path: C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe
09:19:29 Verbose | Starting C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory ‘C:\Octopus\Work\20220913141929-217083-126’ using ‘OEM United States’ encoding running as ‘NT AUTHORITY\SYSTEM’
09:19:30 Verbose | Process C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20220913141929-217083-126 exited with code 0
09:19:30 Verbose | Using Calamari.win-x64 20.4.0
09:19:30 Verbose | Script isolation level: FullIsolation
09:19:31 Verbose | Acquiring isolation mutex RunningScript with FullIsolation in ServerTasks-217083
09:19:31 Verbose | Executable directory is C:\windows\system32\WindowsPowershell\v1.0
09:19:31 Verbose | Executable name or full path: C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe
09:19:31 Verbose | Starting C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory ‘C:\Octopus\Work\20220913141930-217083-128’ using ‘OEM United States’ encoding running as ‘NT AUTHORITY\SYSTEM’
09:19:31 Verbose | Calamari Version: 20.4.0
09:19:31 Verbose | Environment Information:
09:19:31 Verbose | OperatingSystem: Microsoft Windows NT 10.0.17763.0
09:19:31 Verbose | OsBitVersion: x64
09:19:31 Verbose | Is64BitProcess: True
09:19:31 Verbose | CurrentUser: NT AUTHORITY\SYSTEM
09:19:31 Verbose | MachineName: Test
09:19:31 Verbose | ProcessorCount: 8
09:19:31 Verbose | CurrentDirectory: C:\Octopus\Work\20220913141930-217083-128
09:19:31 Verbose | TempDirectory: C:\windows\TEMP
09:19:31 Verbose | HostProcess: Calamari (5292)
09:19:32 Verbose | Performing variable substitution on ‘C:\Octopus\Work\20220913141930-217083-128\Script.csx’
09:19:32 Verbose | Executing ‘C:\Octopus\Work\20220913141930-217083-128\Script.csx’
09:19:36 Verbose | Setting Proxy Environment Variables
09:19:36 Verbose | Writing script module ‘SetLastUpdateValue’ as c# module SetLastUpdateValue.csx. Import this module via #load SetLastUpdateValue.csx.
09:19:54 Info | ERROR: Script execution failed. [COMException] No logical space left to create more user strings. (Exception from HRESULT: 0x80131198)
09:19:57 Verbose | Updating manifest with output variables
09:19:57 Verbose | Updating manifest with action evaluated variables
09:19:57 Fatal | The remote script failed with exit code 1
09:19:57 Fatal | The action Test Script on Test failed

3

Hi @pjones,

Thanks for reaching out on our help forum, and welcome to the community! I’m sorry you’re having trouble with your WinServer 2019 deployment. I must say, I haven’t seen this issue before, and from what I found, there isn’t much helpful information about it. From what I did see, the ERROR: Script execution failed. [COMException] No logical space left to create more user strings. (Exception from HRESULT: 0x80131198) error occurs when a C# assembly is too large. Even stranger, this error only occurs on your 2019 server versions and not in 2016. Is it possible that there is no disk space on that 2019 server?

You may want to engage with Microsoft on this error as they will likely be in a better position to help you find the root cause of this seemingly C#/Windows error.

I’m sorry I couldn’t be of more help. If you do decide to engage Microsoft, please let us know if you’re able to find the solution so others who might experience this issue can see the result. Let us know if you have any other questions.

Best,
Brent

4

Thanks for the response @brent_kinney. There is disk space unfortunately, I came to the same conclusion about the nature of that message. Since the script that’s running is just a log out statement it makes me think it’s some wrapping happing within scriptcs that may be causing it but I’m uncertain. I’m going to try to run some c# scripts directly on the server to see if there is an issue with windows running them at all.

5

Hey @pjones,

That is a good idea and hopefully points us to a possible cause. I know the file is just a basic print line, but you may also want to check the RAM of that VM to ensure it’s not tapped out due to another process. Also, this is a long shot, but maybe check that server’s paging file size setting to ensure it’s not set to something odd.

image
image562×928 32.1 KB

Let us know how your testing goes.

Best,
Brent

6

Hey @brent_kinney the paging looked good on the server. It seems that the windows server is not the issue.

What I have done is I went into our project and cloned it, altered nothing in the new project and it deploys the scripts perfectly fine. This makes me think there is some caching occurring at the project level that is possibly causing issues.

Another thing we noticed is the newly cloned project creates releases and runs the scripts very quickly. Our current project is very slow on new release creation and deployments.

Is there anything you can think of that would cause this?

7

@brent_kinney is it possible it’s compounding the new releases scripts/modules with all of our past release scripts/modules and it causes it to go over the limit?

8

Hey @pjones,

Yes, that is possible depending on your deploy settings. Are you using a custom install directory for that package? If so, there is a setting inside the step to purge the directory before deployment. If you don’t have that checkbox marked, that folder can get out of hand fairly quickly, depending on how often you deploy.

image
image1175×534 33.7 KB

The other thing that might affect the deploys’ quickness is your lifecycle retention policies. What does the retention settings look like for the lifecycle on that project?

Best,
Brent

9

Hey @brent_kinney we actually had suspicions that it may be custom install directories. We have 5 packages that use a custom directory but all of them have the purge checkbox checked. Our lifecycle also has a retention of past 3 releases, I tried changing it to 1 release but no difference.

Also want to mention this is a fresh server, there are currently no deployment folders on the box.

Something we have noticed, on our current project, when it is deploying the work directory created has a variables.secrets file that is 14MB large. When we deploy from the cloned project (has the same variables) the variables.secrets file is 350KB. It seems like wires are getting crossed in octopus somehow.

10

Hmm, let me bring this up with my team and see if anyone has seen this before. I’ll get back to you with and update later today.

11

Hey @pjones,

Can you send us an export of the original and cloned projects?

If you could also send the package being used by those projects, that would be helpful as well.

Something appears to be different between those projects, and we’d like to compare them. You can upload those exports using our secure upload here.

Best,
Brent

12

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.