Hi,
We are currently using Octopus Cloud for about a year now to deploy a range of Microservices (over 300 currently), and we, during our busiest part of the day we see quite a large queue of deployment tasks.
We’re considering adding Workers to help with the queueing problem as we are limited to 5 concurrent deployments on Cloud. Will adding Workers help in our scenario? Do we need to make any changes to our existing projects to help get better concurrency?
Hi,
The task limit of 5 concurrent is a hard limit on Octopus Cloud currently, but there are ways to help increase the velocity of your deployments. The task cap is limited by the number of deployments in play, so if you have five projects/releases deploying, the rest will be queued up behind those tasks, and this is the behavior you’re seeing.
You can add multiple workers, and use these workers to run more than one process at a time on each worker. The benefit of using a Worker is that you offload the heavy lifting from the Octopus server which will help run your deployment scripts and tasks quicker to assist with reducing queued tasks and deployments times.
I want to lay this out below for you, on how you can achieve this.
Octopus & Workers Setup
I’ve configured a sample Space and Project(s) to help better answer this question. I added 2 Workers in a single worker pool named “WorkerPool”.
Custom Step Template
To show this, I created a Custom Step Template which is below, which will sleep based on a value of time. I’ve used 6000s, to give 100 minutes.
write-host 'Hello'
Start-Sleep -s #{SleepTime}
write-host 'Goodbye'
Project Setup
I’ve created a Project called Project1 and created a deployment process, which is using 4 of the same steps based on the step template I created. These steps all very similar, however, there is one key thing to be aware of, is that steps 2, 3 and 4 are set to run parallel with the previous steps like below. Step 1 does not have this, as it’s the first in the deployment process. This bit is critical, as, without this, it will only run one deployment process at a time, and if you are doing deployments, then it will only run one process per task, which limits you to 5 in total.
You should have a deployment process, something like below.
Cloning Project
I’ve gone ahead and cloned this project nine times to get a total of 10 Projects. You can clone this under Project -> Settings -> Select Elipsis and then Clone. I then specified the name of this project. Once this is done, you should see something like below:
Creating & Deploying a Release
In each of the Projects, you can then create a release for each one name “0.0.1” or similar.
The next step is to deploy all ten releases. The step below is me deploying each of the ten projects.
When you have kicked off the deployment, your dashboard will show that 5 of your projects are deploying, and five are Queued.
But, when you go into one of the deployments, you should see that all four deployment steps are running.
Tasks
At this point, you can browse to Tasks, and confirm that five deployments are underway and are limited by the five deployment tasks, but each deployment task is running four deployment tasks in parallel, and you can see how this scales. The caveat is that you need to have a deployment process that’s able to be run in parallel and not one after the other.
Caveat
This approach isn’t for everyone as it relies on a deployment process that can be run in parallel, rather than one after the other. Microservices do generally fit this pattern fairly well and it likely won’t work for traditional deployments that need to be run in a set order. We would recommend running a Proof of Concept of this approach before bringing it into a Production deployment
