Question about task concurrency across tenants

I’m trying to run 2 deploys for the same environment but different tenants. I want some steps to only run 1 instance of that step at a time because it affects shared resources, but other steps to be able to run in parallel as they only affect tenant resources.

  1. Deploy a Package step run on targets in role OctopusServer (1 of these) to migrate the tenant db
  • This can be slow and is independent between tenants. I want multiple of these to execute concurrently
  1. Azure Script step run on Octopus Server
  • I want to run only 1 instance of this. Azure rate limits API calls and crashes if I start running too many
  1. Upgrade a Helm Chart step run on the Octopus Server on behalf of Kubernetes deployment targets (only 1 k8s cluster, so 1 target here)
  • I want to run only 1 instance of this. It configures a shared load balancer.

Step 1 initially ran 1 at a time as expected from the below docs. I configured OctopusBypassDeploymentMutex scoped to just step 1 and that seems to have allowed step 1 to run concurrently. Step 1 is working as desired now.

Based on “By default, Octopus will only run one process on each deployment target at a time, queuing the rest” from Run multiple processes on a target simultaneously - Octopus Deploy, I expected steps 2 and 3 to only allow 1 instance as desired by default, but both steps 2 and 3 execute concurrently with other tenants. How do I get steps 1 and 2 to queue instead of running concurrently? Thanks for your help!

Hey @borigas,

Thanks for reaching out!

Have you thought about creating an internal tenant and only running steps 2 & 3 on that tenant leveraging tenant tags?

I’m just going to do some digging, but you can run multiple tenant deployments to the same environments with some tweaking of Octopus system variables. I’ll get back to you later today with further information.

Thanks

Thanks for your help. I don’t think tenant tags work well for us. In our case, we do want steps #2 and #3 to run for each tenant because they’re doing different things based on the tenant.

  • Step #2 is configuring different Azure resources. It’s just that Azure rate limits are shared across resources
  • Similarly, step #3 is configuring the same load balancer resource, but configuration is different because it’s different URLs it’s changing based on the tenant.

Just to clarify from your last sentence, I have multiple concurrent deployments running now with step #1 as desired. The problem is steps #2 and #3 also are running concurrently when I’d like them to queue.

Thanks again for your help!

Thanks for the additional context @borigas

I might have a solution for you, but I haven’t tested it yet. By default, steps running on the octopus server (as the built-in worker) run tasks in parallel and you want this to happen for your first step. Could you try adding another value to your OctopusBypassDeploymentMutex and set it to false with no scoping?

I’m hoping this runs the other steps sequentially across different tenant deployments. I’ll be testing this myself and I’ll feedback again later.

Thanks

Thanks. I had the same idea. It didn’t work when I tested it though. Setting OctopusBypassDeploymentMutex=False for everything still allowed steps 2/3 to run while other tenant deploys had steps 2/3 running

Hey @borigas,

Sorry for the delay in response while I worked this one out.

To run the script steps (2 & 3), are you running them on the built-in worker on the Octopus server? Or have you configured an external worker?

Yes, they’re run on the Octopus server.
Step #2

Step #3