Parallel Processes across Tenants on a single Deployment Target

I’m just not grasping the concept here. We’re looking to speed up deployments in a tenanted environment.

  1. If I have a single Project containing 6 Steps.
  2. I want to deploy the project to a single server (Deployment Target) that is home to 10 Tenants.
    Is there a way to make this deployment run for all 10 Tenants simultaneously (or at least a few of them) or does it have to proceed sequentially, running Steps 1 to 6 for Tenant 1, then running Steps 1 to 6 for Tenant 2, etc?

I see the OctopusBypassDeploymentMutex setting but it then says that “deployments of the same project to the same environment (and, if applicable, the same tenant) are not able to be run in parallel even when using this variable.”

Is there a way to increase the deployment speed (in terms of simultaneous execution) across multiple Tenants on the same Target?

Jamie

Hey Jamie, thanks for reaching out!

Have you tested using OctopusBypassDeploymentMutex for your deployment? Since you’re deploying to 10 different tenants, it should still work to deploy concurrently, up to your task cap limit.

For example, I have a simple tenanted IIS deploy to two tenants on the same target. I also have OctopusBypassDeploymentMutex set to true in my project variables. When I deploy the release, I can see both deployment actions running concurrently:

You can also see in the logs that they were deploying to IIS at the same time:

If you’re still running into concurrency issues with the Mutex variable set, we can look a little closer and try to find a better solution. Let me know how it goes, happy to help however we can!

Cory,

I guess the part that was unclear is the statement “deployments of the same project to the same environment (and, if applicable, the same tenant) are not able to be run in parallel even when using this variable.” Is this specifically referring to multiple Deployment processes?
Example: You cannot expect a concurrent Deployment of

  • (Deployment#1) Project#1 to Environment#1 for Tenant#1
  • (Deployment#2) Project#1 to Environment#1 for Tenant#2
    to run simultaneously even with the Mutex set to true?

We did try setting the variable to true and it appeared that for our single Deployment, all Tenants on the same Deployment Target were updated at the same time. This is what we were hoping was possible. Am I correct in assuming that the total number of Tasks (across multiple Deployment Targets) would then be limited by the Task Cap setting on the Octopus Server?

During our test of the OctopusBypassDeploymentMutex we did run into some issues with file access (file in use) issues. We’ll need to do more investigation there.

As always, thank you for the help.

Jamie

Hey @jamie.sidoti, you’re totally right that the docs aren’t exactly as clear as they could be with that. We’re taking that feedback and we’ll try to clarify it (and provide a reasonable example to help illustrate if we can!)

You are correct - the call out is specifically for deployment processes, not the tasks themselves. Octopus does this by default to protect files from needing to be accessed concurrently and hitting resource contention/locking issues. You are also correct that task cap is the true upper limit for your deployment targets.

And as to the final bit - custom installation directories should help eliminate some of your file locking issues. However, it’s always good to review the files that were causing locking errors - if they’re more central to the deployment (like the IIS metabase, not something included in your package), you may still run into issues with locking and have to address via a different method.

Thanks for the questions, even though you already had proper answers for most of them :smile: Happy to help if you run into any additional issues!

Alright, I think we have a handle on it. Thanks for confirming the Mutex and Task Cap items. That’s helpful. As for the file access issue… yup, we just tracked it down to a directory that was getting shared between the processes. Easy fix there.

Thank you!

Jamie

1 Like

I’m sorry to have to revisit this topic again, but what we observe is not in line with what we thought we understood. :slight_smile:

Given the following (admittedly over-the-top) scenario where we were testing server limits:

  1. Octopus Server deploying a single project to 400 Tenants across 36 Deployment Targets
    Project consists of
  • Script Step running a PowerShell script (Run a script on a worker from the Default Worker Pool on behalf of targets in a Role of role)
  • Deploy a Package Step (Deploy package package from Octopus Server built-in to deployment targets in Role role)
  1. Task Cap set to 50
  2. OctopusBypassDeploymentMutex is explicitly set to False in the Project Variables

Octopus.Acquire.MaxParallelism has not been set, so default should be = 10
Octopus.Action.MaxParallelism has not been set, so default should be = 10

Expected behavior:
We expected to see a maximum of 36 (out of a total of 400) deployments to be executed at one time (one deployment per Target since OctopusBypassDeploymentMutex is set to False).

Observed behavior:
We immediately see 50 tasks in a state of executing in the Octopus Tasks Log

Questions:
(1) Is this because OctopusBypassDeploymentMutex does not apply to ‘run on server’ steps (like our first step)?
(2) Do Octopus.Acquire.MaxParallelism and Octopus.Action.MaxParallelism even come into play if OctopusBypassDeploymentMutex = False or are these independent of that setting? Would our settings have limited simultaneous package pushes to 10 overall or 10 per Target (or not at all)?

As the deployment continued we always had 50 shown in a state of executing. If different steps are being throttled by any of the variables it’s difficult to tell. Even with the outrageously high Task Cap the deployment did complete, pushing the packages to 400 tenants.

Thanks

Hey @jamie.sidoti , I’m going to do some more local testing and gut checking on this and get back to you when I have some more solid information to share. Thanks for all of the information and context you were able to provide, that’s incredibly helpful!

Hey Jamie, thanks for holding out while I took a closer look - there’s a lot of pieces interplaying here!

Background Info

At the center of this is some terminology that I think will help:

  • Action - A unit of work completed by a step (for example, if you query a deployment process, and look at the steps collection, you’ll notice that each step also has an actions collection for the details of the action the step is performing.
  • Task - A representation of a deployment or some other activity in the Octopus Server

So, to answer your specific questions -

(1) Is this because OctopusBypassDeploymentMutex does not apply to ‘run on server’ steps (like our first step)?

Not quite - this is more to do with the concurrency evaluation than the mutex, although you are correct that running on the server could cause some strange interactions with that variable.

(2) Do Octopus.Acquire.MaxParallelism and Octopus.Action.MaxParallelism even come into play if OctopusBypassDeploymentMutex = False or are these independent of that setting? Would our settings have limited simultaneous package pushes to 10 overall or 10 per Target (or not at all)?

These settings are more complementary than independent. You don’t see the impact of these because they are scoped to a deployment, and each tenanted deployment is its own deployment, rather than a collection of multiple targets like an untenanted deployment would be.

Example Deployment

And now for some example deployment scenarios. Given a project that runs a PowerShell script and then deploys a package:

  • Deploying this project to 20 deployment targets is a single task
  • Deploying this project to 20 tenants (regardless of their underlying infrastructure/overlap) is 20 tasks

This behavior is broadly a representation of this caveat from the BypassDeploymentMutex page:

OctopusBypassDeploymentMutex must be set at the project variable stage. It will allow for multiple processes to run at once on the target. Having said that, deployments of the same project to the same environment (and, if applicable, the same tenant) are not able to be run in parallel even when using this variable.

By splitting tenanted deployments, Octopus helps to ensure that it can run only the tasks it explicitly expects to. This is why even when you would expect to be limited to 36 tasks (by the number of available targets), the concurrency calculation sees those as independent (because there’s a separate tenant attached to each deployment, thus not violating the “same project / same environment/ same tenant” match.

Concrurency Tagging

More explicitly, Octopus uses an internal system variable called Octopus.Task.ConcurrencyTag. by default, this is usually something like #{Octopus.Tenant.Id}/#{Octopus.Project.Id}/#{Octopus.Environment.Id}, and is meant to handle the case I outlined above.

It’s possible to override this tag at the project level. For example, you could give your tenants a new common variable like Tenant.DeploymentGrouping. Then, at the deployment level, you can set Octopus.Task.ConcurrencyTag to something like #{Tenant.TenantDeploymentGrouping}/#{Octopus.Project.Id}/#{Octopus.Environment.Id}. This would then group your tenants into matching concurrency tags, and would only allow one from any given group to go at a time. This is a lot to think through in a vacuum/theoretical sense, but if you’re interested in continuing to pursue refinements of this tenanted deployment process, we can schedule some time to go through this in your scenario and evaluate the different options available.

As always, let me know if anything I’ve said above doesn’t make sense, or if you have additional questions or concerns about how the process is modelled. I’m glad to hear you were able to get through your 400 deployments regardless!