We have OctopusBypassDeploymentMutex set to True and Octopus.Action.MaxParallelism set to 10.
However for some targets Octopus.Action.MaxParallelism is set to 5, because there’s a lot of actions here, and not enough RAM memory to handle so many PowerShell processes at once.
Deployment steps that are set to run in parallel here (at these machines) are targetted for different roles, which are all assigned to the same machine.
To be honest I don’t see MaxParallelism variable to have any effect, it seems that all steps start running at the same time which is very problematic, because Calamari processes (definitely more than 5 or 10) at this target host consume all memory, which is leading to OutOfMemory exceptions in the deployment process - all steps have to be assigned and retry, sometimes many times before deployment finally success without exceptions.
Octopus Server v2021.1 (Build 7316)
Variables preview for step of deployment, for target machine UAT-API…
Unfortunately, this system variable limits the maximum number of machines, not Roles that the steps will run against at the same time. So, this variable will not apply in your scenario where all of your steps are running on the same machine.
If you had multiple tentacle instances configured on a single machine, and each of those instances existed as a separate “machine” within Octopus with your Roles split between these machines, then this system variable would be applied.
I understand but it’s not the case - there’s single machine with only one tentacle (so it’s still a single target in Octopus Server infrastructure).
That machine (single target/tentacle) has role A, B, C, …, Z.
In process there’re steps to run in parallel:
Step 1 to run at target A
Step 2 to run at target B
etc.
It’s configured like this, because there’re environemnts, where there is more machines, f.ex. machine for roles A-C, different machine for roles D-M, etc.
But in this case it’s environment with single machine with all roles at once.
I understand the scenario, but the system variable doesn’t work the way you’re describing.
If you attempt to deploy Step 1 to 10 machines and set MaxParallelism to 5. Then Octopus will be run Step 1 on 5 machines, wait for completion, then run on the remaining 5 machines.
If your deployment process is configured to run 50 Steps in parallel then the same example would deploy 50 steps to 5 machines, wait for completion, then run 50 steps on the remaining 5 machines.
If you configure the steps to run in parallel, they will always do so, regardless of what the MaxParallelism value is set to.
The only real options here would be to scale up the single machine to allow it to handle the number of tasks running on it or amend the deployment process in some way to reduce the number of tasks it tries to run at the same time. Such as by changing them away from running in parallel.
The problem is that I can’t query these steps f.ex. with custom mutex in these PowerShell steps, because I want to prevent too many processes to even get started. I’m still thinking about alternative solutions.
Thank you for an explanation.
Edit:
Octopus.Action.MaxParallelism :
This variable limits the maximum number of machines on which the action will concurrently execute.
However it’s far away from perfect - the 6th step will run only after all of 5 previous steps will finish, instead of after one (any) of these 5 will finish, which for sure will extend the time of deployment.
And another problem will be at other tenants/environments, where f.ex. 4 out of these 5 steps might be disabled (but the “pause” between 5th and 6th step will be still active!)
In the middle of sequence of many parallel steps I’ve added empty steps which are set NOT to run in parallel with previous step.
These empty “throttle” steps are set to run only for specified environments with low RAM.
And that works exactly as it should for many days at these environments, but the problem I’ve just discovered at other environments (without RAM issue) is that even if “throttle” step is disabled/not executed for them, deployment steps after this middle point are executed after deployment steps before this middle point.
Unfortunately, the “Run in parallel with previous step” option only links that step to the one directly above it within the deployment process, regardless of whether the step is disabled or not. So, the “break” steps that you’ve added will have the same effect across all the deployments, as you’re seeing.
