Increase number of concurrent tentacle updates

feature
(jwdean) #1

We have configured our servers to always keep Calamari up to date so that we do not have a large number of warnings simply reporting that Calamari is out of date. After upgraded the Octopus server a fleet-wide upgrade of Calamari happened. That’s fine, but it took a very long time to complete. Is

Is there a way to increase the number of tentacles that concurrently receive Calamari updates? I believe we were only processing 8 at a time, and each took between 1 and 2 minutes. We have a large number of tentacles.

Thank you,
Jeff

(Jim Burger) #5

Hi @jwdean,

Thanks for the heads up on the upgrade performance, we do limit the number of tentacle upgrades based on your servers processor count, and at this time it isn’t able to be changed. We limit this to ensure that we don’t cause other performance issues on the server or network.

In your case, would changing to “automatically update Calamari when a deployment targets is involved in a deployment” be an acceptable outcome?

We’re interested to know your reasoning for choosing this setting - is it because of the warnings about calamari being up to date? We are considering whether we should remove the “Calamari is out of date” warning from Octopus completely.

The only benefit of this warning in our minds is that it gives you an opportunity to update calamari before actually running a deployment, which will mean your deployments will be slightly quicker. Would be great to get your thoughts on that!

Hope that helps,

Kind regards,

(jwdean) #6

We’re trying to optimize our Octopus server configuration to support thousands of geographically disperse tentacles. See this thread as well:

Yes, we enabled ‘Always keep Calamari up to date’ specifically to eliminate the warnings. We also have “Automatically update Tentacle” enabled. We did this because our operations team wants to use the Infrastructure page and see problems that are meaningful and actionable. Having thousands of nodes with the out-of-date warning made for a low signal to noise ratio. The downside is that this change forces very long running tasks when Octopus software updates occur.

The out-of-date warning doesn’t have any value to me when Octopus is configured to upgrade Tentacle/Calamari in any automatic fashion.

Fundamentally, I think these updates should simply be entirely transparent. As a user I want to be able to upgrade my Octopus server without negatively impacting deployment schedules nor do I want to cause undue consternation among the operations support folks.

we do limit the number of tentacle upgrades based on your servers processor count
Can you give me some guidance on this? I’m not sure I’ve seen this documented. In the near term this may relieve some pressure.

Thanks,
Jeff

(Jim Burger) #7

Thanks very much for the considered feedback Jeff,

I’ve raised this with the team and we’re in agreement that the warnings are no longer valuable, so I’ve created a Github issue to resolve this here: https://github.com/OctopusDeploy/Issues/issues/5643

I’ll move towards removing this over the next day or so.

As for the limit, you’re right I don’t think there is documentation regarding this as yet, I’ll address that too. I’ll reach out when that is updated.

Hope that helps

Jim

(tgillitzer) #8

Jim, I just commented on your GitHub issue, but another issue we (I work with Jeff) have is that we don’t want to manage the lifecycle of the Tentacle separate from the version of the Octopus Server. When we upgrade our Octopus Server, we don’t want to have to evaluate what versions of the Tentacles are out there and make sure they are still supported. We’d like to only have to worry about the old one and the new one.

Can you describe the logic or point me to the code for the logic to determining the limit based on processor count? Maybe we can just add some more processors and get to where we want to be.

Other than that, we are considering other ideas, one of which is to write our own Tentacle Upgrader that would call your APIs to initiate Tentacle Upgrades in smaller batches. Thoughts on this?