Environment Variable Block in Octopus 2 - max limit exceeded

harrisonmeister · 22 August 2014 13:45

Hi

I have been investigating for a few hours now that we have exceeded the max environment variable limit for the windows process which is executed when powershell is called inside of the tentacle service

This limit error is triggered by us calling rs.exe from within a deployment step which uses a .rss file (reporting services script file) - which in turn compiles that script to vb.net code and calls into the XML Serializer

I believe this is due to the number of package steps, custom variables we have, and the built in octopus variables.

There are about 25 variables for the release, 65 steps.

Find attached the exception we get when rs exe is running:

Do you know if you plan to change the way the octopus variables are used (the built-in ones that is) - we don’t utilize those in our deployment story at all.

Ta, Mark

PS-Error.txt (11 KB)

harrisonmeister · 22 August 2014 14:05

Sorry - I should clarify- it turns out we do use a small number of the built-in octopus variables for steps e.g. to contain the body contents for an email step

Thanks, Mark

Nicholas_Blumhardt · 24 August 2014 23:46

Thanks for the report Mark. I’ve created an issue to track it here: https://github.com/OctopusDeploy/Issues/issues/1166

You can clear all of the Octopus variables except the “system” ones like $OctopusParameters using:

ls variable:"Octopus.*" | rv

It’d be very useful to know if inserting this before the failing line fixes the issue (if it doesn’t we’ll need to dig deeper).

Best regards,
Nick

harrisonmeister · 25 August 2014 16:43

Hi Nick

Thanks for the prompt reply. I will try the removal of the variables as above and let you know how I get on (we’re in the UK and todays a bank holiday so it wont be until at least 24 hours from now until I’ll be able to let you know if it works)

Thanks, Mark

harrisonmeister · 25 August 2014 21:06

Hi Nick,

I’ve tested it and it doesnt seem to have any effect (it still errors with the original attached file)

Does ls variable:“Octopus,*” remove environment variables too?

I list out the variables in the deploy script using this:

"Removing the Octopus.* environment variables" | Write-Host
	ls variable:"Octopus.*" | rv
	"Checking existing environment variables after removal of Octopus.* ones"
	$envVariables = Get-ChildItem env:
	$length = 0
	$concatenatedVars = ""
	foreach($envVariable in $envVariables)
	{
		$envVariableKey = $envVariable.Key
		$envVariableValue = $envVariable.Value
		$concatenatedVars += $envVariableValue
		$length += $envVariableValue.length
		Write-Host "$envVariableKey has value: '$envVariableValue'"
	}
	Write-Host "concatenated envVars length='$length'"
	Write-Host "concatenated envVars='$concatenatedVars'"

and i still get the mass-list of variables.

Thanks Mark

Nicholas_Blumhardt · 25 August 2014 21:57

Hi Mark,

What kinds of variables do you see in env:? At a guess, removing them with rv should also be possible - does that sound viable?

Regards,
Nick

harrisonmeister · 25 August 2014 22:17

Hi Nick,

I think we can certainly try and use this as a workaround, but I do fear
other customers will eventually reach this limit (there’s very little
documentation on msdn/windows about this limit)

I’ll have a play with the script and try that out, and let you know how I
get on.

(We still get the octopus .Action and .Step multiple variables show up in
the output from my script I sent in the previous comment)

Thanks, Mark

Nicholas_Blumhardt · 26 August 2014 04:42

Hi Mark,

You’re no doubt correct - at the moment just getting a handle on the scope of the problem is the first priority, but as you say, a longer-term solution will be needed. Thanks for the extra info.

Regards,
Nick

harrisonmeister · 26 August 2014 06:23

Hi Nick

Thanks - I’ll let you know how I get on later today

Mark

harrisonmeister · 26 August 2014 09:10

Hi Nick

I’ve just tested the following code in our reports deployment:

Write-Host "BEGIN Enumerating 'Process' Environment variables like Octopus.*"
	$envVariables = [Environment]::GetEnvironmentVariables("Process").GetEnumerator() | Where-Object { $_.Name -like "Octopus.*" }
	foreach($envVariable in $envVariables)
	{
		Write-Host (" - [{0}] = '{1}'" -f $envVariable.Name, $envVariable.Value)
	}
	Write-Host "END Enumerating 'Process' Environment variables"
	
	
	"Removing the Octopus.* environment variables" | Write-Host
	foreach($envVariable in $envVariables)
	{
		$envVarName = $envVariable.Name
		Write-Host ("- Setting 'Process' level environment variable [{0}] to NULL" -f $envVariable.Name)
		[Environment]::SetEnvironmentVariable($envVarName,$null,"Process")
	}

and this appears to have worked for this deployment.

This was deployed skipping all steps which I was able to skip (we have some steps which I dont have permission to skip). It looks like that although you have environment variables for all of the action/steps - there are some which state ‘Skipped’ in them.

I dont think the ‘variable’ and environment variables are the same thing (hence the | rv command not having any effect).

Just musing ideas at this point I think there are a few options:

I think from our perspective, it would be ok to probably remove the octopus environment variables if the correlating actions/steps are skipped - but I dont know if that has any internal implications to the Octopus Server/Tentacle behaviour.
Despite this link stating there is no “technical limit” -> http://msdn.microsoft.com/en-gb/library/windows/desktop/ms682653(v=vs.85).aspx
I wonder if the variables should be read in via some other means (file etc) before the deploy script is run. Adding a new child process for each step still wouldnt get around the issue for larger number of steps like we have
Shortening the Octopus environment variable names, though this only works around the limit issue.

Obviously the root cause is an OS limitation, so we understand its ultimately not something you have that much control of.

For now, I’ll see if we can determine certain octopus environment variables we can safely remove - is there a link to the ‘important’ octopus ones that we shouldn’t touch?

Thanks, Mark

harrisonmeister · 26 August 2014 10:25

Hi Nick

I’ve settled on this powershell snippet which removes the Octopus.Environment.MachineInRoles collection and the Octopus.{StepName}.EnabledFeatures - since we dont utilise these in our scripts.

$envVariables = [Environment]::GetEnvironmentVariables("Process").GetEnumerator() | Where-Object { $_.Name -like "Octopus.Environment.*" -or $_.Name -like "Octopus.*.EnabledFeatures" }
"Removing the Octopus.Environment.* and Octopus.*.EnabledFeatures environment variables" | Write-Host
foreach($envVariable in $envVariables)
{
	[Environment]::SetEnvironmentVariable($envVariable.Name,$null,"Process")
}

We’re happy to use this moving forward (providing you see no major issues) since the issue only appears in 15 of our ~350 nuget packages.

Chatting with a colleague he thinks the issue around the environment variable limit is actually scoped to the use of rs.exe where it does its ‘compilation-on-the-fly’ of the .rss file (from the original stack trace, it looks like its the XmlSerializer), so I don’t know if the environment variable limit would cause issues in other usages, though I’d be interested to find out.

Thanks, Mark

Nicholas_Blumhardt · 26 August 2014 23:21

Thanks for the follow-up Mark; we’ll still look into this but glad to hear it’s no longer blocking you.

All the best!
Nick

harrisonmeister · 4 September 2014 08:15

Hi Nick

Just as a quick follow up. We have now migrated any use of reporting services script (.rss) files to an embedded .Net Library which we call from within powershell.

This removes the issue of compile-on-the-fly, and therefore the ennvironment variable block size completely

Cheers, Mark