We have an Octo instance that has several projects that, when they deploy to the production environment, they deploy to two roles (blue and red) in an A/B scenario with 8 targets (take all the red down, deploy to those, bring them up, then take all the blue down, deploy to those, bring them up). Powershell takes them up and down within BigIP. What happens at times is that 1 person will start a deployment, and then another person will come in behind them and not pay attention to the fact that someone is already running a deployment to the same environment (prod). They will kick off a second project deployment to the same environment, and this often causes all the nodes to be down at the same time, which means our site is down. I want to prevent that (obviously).
Is there a way to setup a step in the deployment to check to see if someone is already running a deployment to the same environment and wait until the other step is finished before continuing? Maybe a sort of queuing concept?
It would be nice if the solution for this was easy to implement as well. I’m not a power user (yet).
Hi Mike,
Octopus Deploy actually already has just such a queuing feature (see screenshot attached). When a task is running a concurrency tag made up of <TenantId>/<ProjectId>/<EnvironmentId> (or excluding the TenantId if there are no tenants involved) is recorded and any further tasks that execute with that same tag are paused and wait until the relevant task has completed.
Which version of Octopus Deploy are you currently using as it should have been available for some time. Keep in mind this will only “block” for deployments to the same project and environment.
If you are not seeing this behaviour then perhaps you are encountering a bug. Could you please send through the server logs of the two tasks that are running concurrently but which, through the explanation provided above, should not be.
If this is a problem then hopefully we can get to the bottom of it easily!
Thanks,
Rob
Octopus 3.3.22
You say “same project and environment.” The problem is that different projects to the same environment are the problem. We have multiple projects which target the same environment, and due to the way we take up and down the nodes, two different projects pushing at the same time to the same environment can leave them in an all-down state temporarily. If it helps, in our case, each project is essentially a different website on the same set of boxes. We take the whole box “down” when doing a deployment, so the A set and the B set can each be down at the same time during the deployment of project 1 and project 2.
Switching to a rolling deployment might help, but doesn’t solve the problem, and increases the deployment time substantially.
Mike,
If they are different projects then unfortunately there is no out-of-the-box ability to block concurrent deployments. This is by design since the projects are more often then not, independent processes that can (or should) execute at the same time. To get the ability you are after you will need to do some custom scripting in your project. You can either call back into the Octopus Server via the API (remember it has been written API first, everything you can do in the portal you can do through the API) or via some sort of “locking” mechanism by checking for and writing some form of lock file.
I hope that provides some guidance to get what you are after.
Cheers,
Rob