Configure parallel deployment tasks across deployment targets

usability
(Alex Goris) #1

We’ve been using Octopus deploy for about 2 months now, and are really happy with it in general, it really helps visualize the status across multiple applications, versions and tenants.

One thing I’ve been struggling with, is the parallel deployment tasks configuration, I’ve read the documentation here, but as far as I understand it, it’s only possible to configure the number of parallel tasks from the Octopus server perspective.

So we have 2 main applications, with about 50 tenants each (so ~100 deployments). For these applications we have 3 internal testing environments: Alpha, beta and staging. These environments relate to various stages of our release process. For each of these environments we have a dedicated pretty beefy AWS EC2 instance running.

Every night we build our 2 applications, and they have to deployed to each tenant in the alpha environment, in a ‘worst-case scenario’ (depending on the stage of our release cycle) we might have beta and staging builds for either of the 2 applications as well, which also have to be deployed to their respective environments, again for all tenants. In this worst-case scenario we would have to do ~300 deployments during the night, which ideally should be finished before ~9AM so our employees can investigate any errors which might have happened.

A single deployment of one application to one tenant, can range anywhere from 5 minutes to several hours, since it involves loading large data sets, which vary from tenant to tenant. I am fully aware that this is a completely different scenario as your run-of-the-mill .net or Java web application, but it is what it is and for the time being we have to deal with it.

The alpha/beta/staging EC2 instances are spec’ed to deal with about ~5 simultaneous deployments, and we can scale them to finish on time if we get more tenants over time.

However I can’t seem to configure Octopus to run these 5 deployments over the 3 deployment targets continously. I’ve set OctopusBypassDeploymentMutex to true on both projects, but e.g If I first start an alpha deployment, Octopus will start 5 simultaneous deployment tasks on the alpha deployment target, and queue the deployments for the remaining 45 tenants.
Then, f I start a beta deployment of the same project (or of another project, doesn’t really matter), instead of also starting the 5 deployment tasks on the beta deployment target, all 50 deployments will just be added to the end of the queue. So from a deployment target perspective, the deployments are performed sequentially, and not in parallel.

We’re not at a stage where we’re using Octopus to deploy to acceptance/production environments (which our customers have access to), but it’s definitely something we are planning to do, but that would mean a production deployment (which would be on a customer-specific EC2 instance and thus deployment target) could be blocked because Octopus has queued deployments to our internal testing environments, which is definitely not what we want.

I really hope I’m missing something, is it possible to configure Octopus the way I want?

(Kenneth Bates) #4

Hi Alex,

Thanks for getting in touch and providing such a great level of detail! That’s awesome to hear that you’re having a good experience with Octopus so far.

It sounds like you’re after increasing your task cap, which is by default set to 5. This controls the max number of simultaneous tasks Octopus can run. The following doc page shows how you can change this task cap.

It would be worth experimenting with increasing this to find a good sweet spot, since increasing this limit means Octopus would then require more system resources to handle it.

Let me know how you go or if you have any further questions or concerns moving forward. :slight_smile:

Best regards,

Kenny

(Alex Goris) #5

Hi Kenny,

Thanks for the reply. I’m not sure increasing the task cap would fix my problem, as that would just increase the number of simultaneous task which Octopus can run, and what I’m looking for is configuring the number of tasks per deployment target, as I pointed out, our 3 testing servers (alpha/beta/staging) can deal with about 5 simultaneous deployments, so increasing the task cap to 10 would just run the next 10 queued tasks, which would likely be on the same deployment target, thus overloading that target, while still leaving the other targets idle. That’s my interpretation of the document you linked, please correct me if I’m wrong though.

Reading that and related docs though, it sounds like multiple nodes might be what I’m after? Please note that we’re using the Cloud version of Octopus, and I can’t seem to access the ‘Nodes’ configuration page which is mentioned in the docs. Is this feature available at all in the Cloud version?

(Kenneth Bates) #6

Hi Alex,

Thanks for getting in touch! I’m terribly sorry about not getting back to you sooner on this one. My bad for letting it fall through the cracks.

I had a chat with my colleague about this question, and unfortunately I think what you’re after isn’t possible as there’s no way to prioritize tasks in the queue. Would something like simply disabling the test targets while you’re deploying to prod help? You can do this manually by excluding machines on the deployment page, or configure target roles in a way which would mean they’re not applicable in certain environments.

Let me know what you think or if I’m off the mark in any way. :slight_smile:

Best regards,

Kenny