The response mis-understood our situation. For each AWS EC2 instance launched by an AWS ASG, a script installs and registers the tentacle and then invokes a Calamari update task for that machine. The ASG launches instances concurrently. But Octopus forces the Calamari update task to single-thread (i.e. run one-machine-at-a-time). This significantly slows down the overall time needed to scale up instances. For example, we really use up to 50 instances or more in an ASG. When we stand up a new ASG that use anywhere near that number of instances, and it takes even 1 minute to do the Calamari update, then 40+ minutes is added to the total time needed to scale up.
What we need is to pass in a var to the Calamari update job that will allow it to run in parallel. Here’s how we’re currently invoking the update task:
Write-Host (" Update Calamari")
$findmachine = $repositoryForSpace.Machines.FindByName("$targetName")
If (!$findmachine) {Throw " var: findmachine is null, ERROR at repository.Machines.FindByName(`"$targetName`")"}
$machineid = $findmachine.id
$body = @{
Name = "UpdateCalamari"
Description = "Updating calamari on $targetName"
Arguments = @{
Timeout= "00:45:00"
MachineIds = @($machineId)
}
} | ConvertTo-Json
$out = Invoke-RestMethod "$ssasg_octopusproductionserverurl/api/$spaceId/tasks" -Method Post -Body $body -Headers $header
Write-Host (" {0}" -F $out[0])
Thanks for explaining further and the extra details.
Would you be able to elaborate more on why you require calamari to be up-to-date before the target is used in a deployment? Understanding this may help us guide you to a solution.
Upon the first deployment on a target, the calamari on that machine is automatically upgraded, so doing this before-hand shouldn’t be necessary.
Any more information you can provide would be most appreciate and would hopefully allow us to better understand your use-case.
Background
In our experience over the past few years with AWS Autoscaling Groups, we’ve found that scale-out/in processes work in a significantly more reliable way when AWS and the ASG remain in synchronous control of the launch process, and for AWS, this is by/per machine. Although Octopus offers features to support AWS ASGs they have proven unreliable. Rather, we use the AWS ASG Lifecycle Hooks to customize the process.
The Issue
We used to let Octopus do the Calamari update after the machine was registered as a target, but it disrupted and delayed the scale out process. Specifically, we register the tentacle via the instance userdata script, which is the first custom script to run when the instance comes online. But we have subsequent steps that register the instance to the domain and a last step to trigger deployments, which are then verified to AWS. Only after that does the ASG launch lifecycle hook tell the ASG that the machine is ready for use. The machine reboots between these steps, which at times collided with Octopus’s Calamari Update task, which left the machine in a bad state. Also, Windows machines take a long time to launch so speeding up the scale out is critical to the value it provides as traffic increases. The asynchronous/batch approach that Octopus takes is unreliable, takes longer, and is error prone for AWS. We don’t and won’t use it.
Our Ask
We already obtained the Octopus variable and value that we use to allow the same release to be deployed to separate machines via multiple deployments. I.e. each machine has its own deployment of the same release that deploys at the same time. That works great! Now we find that Octopus still is the bottleneck by not allowing each machine to run its own Calamari Update task at the same time, which is what we are asking for via this ticket. For example, if we could pass in an argument via the code shown above that would allow these updates to run all at the same time. Is there some ID or key that’s preventing this?
Hello Finnian, this is great! Can you make this release available to us? Also, are there upgrade parameters or variables involved in allowing concurrent Calamari update tasks?
Sorry it has taken awhile to get back to you on this, just to let you know that 2023.4 is not available for our Octopus Server customers at the moment (it is an Octopus Cloud specific version and has yet to be fully tested on our Server product) so we are unable to get you a release link for that but hopefully 2023.4 will be coming out to our Server customers soon.
I cannot get you a date for that unfortunately but keep an eye out on our downloads page as that will update when 2023.4 is available to our Octopus Server customers.