We recently performed a fleet-wide upgrade of Calamari and the Tentacle software. (Over 1500 tentacles). Based on that experience I believe we either misunderstood how Task Cap and MaxParallism work together, or we missed something important.
As an example of our prior understanding, consider a specific deployment of ProjectX to 400 tentacles. In the default configuration that task would begin, and the deployment would proceed in a single task to those tentacles as defined by the parallelism setting. If that deployment takes precisely 2 minutes per tentacle, it would take about 1 hour 20 minutes to update all of those tentacles. While that is occurring, there would -still be 4 tasks available- for other deployment activities.
However, while our Calamari/Tentacle task upgrade was running no project deployment tasks would run to any tentacles. They were instead queued. Is there a configuration/option we missed that would have allowed project deployments to proceed while the automatic tentacle software upgrades were also in process? Or can you more clearly describe how these configurations work together.
FYI - the Task Cap and MaxParallism values we are using are all defaults.
Thanks for getting in touch, and sorry for the confusion here.
I think your general understanding of Task Cap and MaxParallelism is on the money, specifically the Task Cap is how many tasks can run concurrently, and MaxParallelism determines how many machines each task runs on. To put that in context if you have set
Task Cap = 5 , each with
MaxParallelism = 5 then you can have up to 25 machines processing deployments at any one moment.
Where this stops is with Tentacle upgrades. That task takes an exclusive lock over the task pipeline and prevents any other tasks from running regardless of Task Cap. Why? Because as part of the upgrade the Tentacle needs to restart, so if the upgrade runs interleaved with a project deployment on a machine it will cause that deployment to fail and potentially leave that machine in a fairly ugly state. To avoid this we prevent other tasks from running while the upgrade task is running, although I would say that we can call this out better, I will raise this internally and see what we can do.
If you have any questions please let me know.
Thanks for the response! I guess that raises another concern…
We were addressing a large backlog of tentacles that were behind on Tentacle and Calamari versions. We updated our server configuration to automatically upgrade Tentacle and Calamari. It took most of the day to complete those upgrades, and we’re only about 1/8th the way through installing tentacles on our endpoints.
I’m concerned about the next time we upgrade our Octopus server (something we haven’t done yet with this configuration).
- Assuming there is an Tentacle and Calamari upgrade in the new version, will a task to upgrade all the tentacles automatically start… thusly blocking the pipeline again?
- If so, is there any mechanism to break that upgrade task into chunks? Perhaps even at a Tennant level? When we reach all our endpoints, my math puts us at over 2 days just to upgrade calamari and tentacle software.
- If we revert the automatic upgrade settings we’ll then end up a huge proportion of tentacles in a “Healthy with Warnings” state perpetually until they receive a deployment which (as I understand it) will push the Calamari/Tentacle upgrade. That seems to leave us in a state where there is no reasonable mechanism to address those warning states short of manually upgrading each tentacles…
I understand the reasoning you describe for locking down the pipeline, but it seems there should be a middle ground. Blocking the entire pipeline when already upgraded tentacles or tentacles that don’t require upgrades would be available to deployments seems like a rather significant scale issue…
I understand the concern, and this is where we struggle a little to define the difference between Calamari and Tentacle.
While Tentacle upgrades do block other deployments from running, they are (except in rare cases) not required to be updated in lock step with your Octopus Server. As we (somewhat awkwardly) show in our compatibility matrix Tentacle versions are both forwards and backwards compatible for long periods of time. It was only the introduction of our Spaces feature that broke this compatibility, and I don’t see anything on the roadmap at the moment that would require you to upgrade Tentacle anytime soon. The UI also doesn’t show any warnings on each target here as this mismatch is by design (although there is a warning that can be dismissed advising that an update is available.
The warnings that you are seeing are around Calamari being mismatched between Server and deployment target. Unlike Tentacle, Calamari upgrades are required as part of every Octopus Server upgrade, however they are not a blocking task so can be run anytime. By default they will get updated upon first deployment to a deployment target when a new version of Calamari is available, but that doesn’t have to be the case. If you would prefer to keep Calamari up to date (and keep the warnings clean) then I would recommend changing your Machine Policy in
Infrastructure > Machine Policies from
Update when involved in a deployment to
Sorry for the wall of words here, I hope that all makes some form of sense.
Happy to answer any further questions that you have,
Thank you for the wall of words!! That is very helpful.
Long-term I guess I would still be concerned about Tentacle upgrades blocking, but knowing we won’t face it every LTS upgrade will save a lot of consternation.
No problems, happy to help!
Let me know if there is anything else you need.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.