We have configured our servers to always keep Calamari up to date so that we do not have a large number of warnings simply reporting that Calamari is out of date. After upgraded the Octopus server a fleet-wide upgrade of Calamari happened. That’s fine, but it took a very long time to complete. Is
Is there a way to increase the number of tentacles that concurrently receive Calamari updates? I believe we were only processing 8 at a time, and each took between 1 and 2 minutes. We have a large number of tentacles.
Thanks for the heads up on the upgrade performance, we do limit the number of tentacle upgrades based on your servers processor count, and at this time it isn’t able to be changed. We limit this to ensure that we don’t cause other performance issues on the server or network.
In your case, would changing to “automatically update Calamari when a deployment targets is involved in a deployment” be an acceptable outcome?
We’re interested to know your reasoning for choosing this setting - is it because of the warnings about calamari being up to date? We are considering whether we should remove the “Calamari is out of date” warning from Octopus completely.
The only benefit of this warning in our minds is that it gives you an opportunity to update calamari before actually running a deployment, which will mean your deployments will be slightly quicker. Would be great to get your thoughts on that!
We’re trying to optimize our Octopus server configuration to support thousands of geographically disperse tentacles. See this thread as well:
Yes, we enabled ‘Always keep Calamari up to date’ specifically to eliminate the warnings. We also have “Automatically update Tentacle” enabled. We did this because our operations team wants to use the Infrastructure page and see problems that are meaningful and actionable. Having thousands of nodes with the out-of-date warning made for a low signal to noise ratio. The downside is that this change forces very long running tasks when Octopus software updates occur.
The out-of-date warning doesn’t have any value to me when Octopus is configured to upgrade Tentacle/Calamari in any automatic fashion.
Fundamentally, I think these updates should simply be entirely transparent. As a user I want to be able to upgrade my Octopus server without negatively impacting deployment schedules nor do I want to cause undue consternation among the operations support folks.
we do limit the number of tentacle upgrades based on your servers processor count
Can you give me some guidance on this? I’m not sure I’ve seen this documented. In the near term this may relieve some pressure.
Jim, I just commented on your GitHub issue, but another issue we (I work with Jeff) have is that we don’t want to manage the lifecycle of the Tentacle separate from the version of the Octopus Server. When we upgrade our Octopus Server, we don’t want to have to evaluate what versions of the Tentacles are out there and make sure they are still supported. We’d like to only have to worry about the old one and the new one.
Can you describe the logic or point me to the code for the logic to determining the limit based on processor count? Maybe we can just add some more processors and get to where we want to be.
Other than that, we are considering other ideas, one of which is to write our own Tentacle Upgrader that would call your APIs to initiate Tentacle Upgrades in smaller batches. Thoughts on this?
Hi there @tgillitzer and sincere apologies for the delay in my response,
Thankyou for letting us know about how the ‘healthy with warnings’ icons are causing unwarranted concerns for your operations staff. I’ve updated the github issue to reflect your views, again, apologies for the delay here. We’ve placed this item on our backlog for now, and will address it as soon as priority of other items allow.
In terms of the upgrade limit, the code for this unfortunately is proprietary, however it is a calculation based on this value provided by the .NET framework which is your logical processor count.
The number of concurrent upgrades will be double the Octopus Servers logical processor count. It will be a minimum of 2 and will not exceed 32.
I’ve updated our documentation to reflect this for you here.
Thanks @Jim.Burger . We have access to the repo, granted via our license terms. Was mostly just curious.
If it is as straight forward as double the processor count, I think that is good enough. We can at least add more processors and increase the number of concurrent upgrades.
I will add that it seems odd to bind the concurrent tentacle upgrade limit to the processor count. It seems to me that it is unlikely that the bottle neck is processor but maybe I’m missing something.