Hey Jamie, thanks for holding out while I took a closer look - there’s a lot of pieces interplaying here!
Background Info
At the center of this is some terminology that I think will help:
- Action - A unit of work completed by a step (for example, if you query a deployment process, and look at the steps collection, you’ll notice that each step also has an actions collection for the details of the action the step is performing.
- Task - A representation of a deployment or some other activity in the Octopus Server
So, to answer your specific questions -
(1) Is this because OctopusBypassDeploymentMutex does not apply to ‘run on server’ steps (like our first step)?
Not quite - this is more to do with the concurrency evaluation than the mutex, although you are correct that running on the server could cause some strange interactions with that variable.
(2) Do Octopus.Acquire.MaxParallelism and Octopus.Action.MaxParallelism even come into play if OctopusBypassDeploymentMutex = False or are these independent of that setting? Would our settings have limited simultaneous package pushes to 10 overall or 10 per Target (or not at all)?
These settings are more complementary than independent. You don’t see the impact of these because they are scoped to a deployment, and each tenanted deployment is its own deployment, rather than a collection of multiple targets like an untenanted deployment would be.
Example Deployment
And now for some example deployment scenarios. Given a project that runs a PowerShell script and then deploys a package:
- Deploying this project to 20 deployment targets is a single task
- Deploying this project to 20 tenants (regardless of their underlying infrastructure/overlap) is 20 tasks
This behavior is broadly a representation of this caveat from the BypassDeploymentMutex page:
OctopusBypassDeploymentMutex
must be set at the project variable stage. It will allow for multiple processes to run at once on the target. Having said that, deployments of the same project to the same environment (and, if applicable, the same tenant) are not able to be run in parallel even when using this variable.
By splitting tenanted deployments, Octopus helps to ensure that it can run only the tasks it explicitly expects to. This is why even when you would expect to be limited to 36 tasks (by the number of available targets), the concurrency calculation sees those as independent (because there’s a separate tenant attached to each deployment, thus not violating the “same project / same environment/ same tenant” match.
Concrurency Tagging
More explicitly, Octopus uses an internal system variable called Octopus.Task.ConcurrencyTag
. by default, this is usually something like #{Octopus.Tenant.Id}/#{Octopus.Project.Id}/#{Octopus.Environment.Id}
, and is meant to handle the case I outlined above.
It’s possible to override this tag at the project level. For example, you could give your tenants a new common variable like Tenant.DeploymentGrouping
. Then, at the deployment level, you can set Octopus.Task.ConcurrencyTag
to something like #{Tenant.TenantDeploymentGrouping}/#{Octopus.Project.Id}/#{Octopus.Environment.Id}
. This would then group your tenants into matching concurrency tags, and would only allow one from any given group to go at a time. This is a lot to think through in a vacuum/theoretical sense, but if you’re interested in continuing to pursue refinements of this tenanted deployment process, we can schedule some time to go through this in your scenario and evaluate the different options available.
As always, let me know if anything I’ve said above doesn’t make sense, or if you have additional questions or concerns about how the process is modelled. I’m glad to hear you were able to get through your 400 deployments regardless!