For each EC2 instance launched by an EC2 autoscaling group (ASG), the ASG Lifecycle Hook on that instance (deployment target) runs a local script that executes the octo.exe cli create-release w/deploy command to create a release and deploy to itself.
An Octopus release can go to multiple targets per deployment. But Octopus limits multiple deployments of the same release to one target at a time. However, this limitation is meaningless for our scenario since each deployment only goes to one target. This is a huge negative impact for us. For example, an ASG launches 6 additional instances. Each instance triggers its own deployment which deploys only to itself. However, Octopus forces these deployments to execute sequentially thereby unnecessarily increasing the total time it takes to scale up. For example, it takes 10 minutes to do a deployment. It should only take 10 minutes for all 6 targets to come online. But instead, it takes an hour before all 6 targets come online.
What I’m asking for is a switch for the octo.exe create-release & deploy cli that bypasses this limitation.
Thanks!
BTW…The release name for each instance is already unique, but for any particular launch, those releases are a clone of the latest release deployed. Octopus, however, uses an internal key that identifies these releases as the same and enforces said limitation.
We had a blog post a little while back, which did what I believe you are asking about but for Azure. I believe that could help provide some inspiration and be a good starting point for adapting this to AWS. The sample is set up in our samples instance, you just need to log in as guest, and then you can review it.
Please have a look over that and let us know how you get along!
The answer to your direct question of if there is a switch for octo.exe currently would be no. Stepping back a bit, it sounds like you are hitting the limitation of deployments of the same project to the same environment (and, if applicable, the same tenant) are not able to be run in parallel even when using this variable.
The basic building block the blog post I posted above used was deployment target triggers. I think this could be a viable solution to the issue you are facing. With deployment target triggers configured, when a target gets added, it will kick off the last successful deployment by default. Usually, depending on the timing of the scaling event, the majority of the new targets will get batched into one deployment. However, this is not guaranteed, which is where that blog post takes it a little further to ensure the scale-out has been completed before deploying.
Deployment target triggers may be all you need in your use case. Do you think this would be a workable solution?
Hello Mark, I can say with confidence that target triggers are not a workable solution. We have been using them and have had a lot of trouble with race conditions and timing problems.
What we have done is use AWS ASG Lifecycle hooks for instance launch and termination; re launching instances, the hook communicates with AWS by machine. Here’s the sequence on each instance launched by the ASG:
Launch hook runs script on instance that executes octo CLI to deploy the latest release to itself
Script monitors deployment,
When deploy successfully completed, script tell hook “all done”
As far as triggers and other custom ways of batching instances launched, we have found the asynchronous/waiting and control hand-offs approach unreliable. We have found AWS ASG lifecycle hooks to be very reliable. The hooks have explicit by machine control at all times .
Here’s the kind of solution triggers and custom batching approaches use afaik; lots of complicated and difficult to maintain code where:
The launch hook for each new instance runs a script to post to a queue when its ready to be deployed to
Once the number of targets queued == number launched, a deploy command is sent to Octopus to deploy the release to all machines launched
Then waits for Octopus to send back that the deployment was successful for each machine
Then completes the launch hook for each machine launched
or, I just pass a flag in the deploy-release CLI on each instance launched that lets Octopus run multiple deployments of the same release on different targets at the same time. Problem solved. No additional code needed.