After upgrading from v3.something to v4.0.11 in production, many of our tentacles were flagged as having an out-of-date Calamari version, saying they needed v4.0.4.
Our machine policies are set to always keep Calamari up to date. At the end of the audit, each tentacle was successfully upgraded to a current Calamari, and on the next health check they were fine. But we still got spammed with 600 spurious “Machine XYZ became healthy but has warnings” messages, followed by 600 “Machine XYZ became healthy” messages.
Shouldn’t the health check look at the Calamari version and report on it being out-of-date after doing the upgrade (if the machine policy allows it)?
Are you saying that you had to manually run a Calamari upgrade task for Calamari to be upgraded on the Tentacles?
Looking at the code, we first do a health check to figure out what the current state of the Tentacle is and if we determine that it needs a Calamari upgrade (and the Always keep Calamari up to date option is selected) we run the Calamari upgrade before committing the health check result.
Upgrade Octopus server so it knows there’s a new version of Calamari out there.
Health check starts.
Health check noticed that Calamari was out of date, and flagged a warning.
Health check updated Calamari.
Health check completes - machine status is “Healthy with Warnings”.
Machine Health Changed subscription triggered, sending me an email.
1 hour delay.
Health check starts.
Health check completes - machine status is “Healthy”.
Machine Health Changed subscription is triggered, sending me an email.
Except that happens on all registered machines simultaneously.
Since that first health check updated the Calamari version, I don’t think it should have gone into “Healthy with Warnings” status; the warning had already been resolved by the time the health check completed.
Thank you for the breakdown of events! And I agree with you that the health check should not report a machine as having warnings if it’s due to Calamari being out of date and Calamari being automatically upgraded as part of the health check.
I’ve raised this GitHub issue to have this investigated and fixed as soon as possible.