URGENT: Calamari Upgrade has broken Service Fabric Upgrade

(Stuart Jones) #1

Hi,

We’re currently unable to upgrade any of our Service Fabric applications, since the weekend, when I assume Calamari has been upgraded on our hosted instance.

I’ve done some troubleshooting, and I think I’ve isolated the issue to this commit:

Specifically, line 126:
$UpgradeEnabled = $UpgradeEnabledInProfile -and -not $OverrideUpgradeBehavior -eq ‘VetoUpgrade’

This doesn’t evaluate properly - you can check quickly in PowerShell as follows:
e.g.
$UpgradeEnabledInProfile = $true
$OverrideUpgradeBehavior = ‘None’
$UpgradeEnabled = $UpgradeEnabledInProfile -and -not $OverrideUpgradeBehavior -eq ‘VetoUpgrade’

In this circumstance, $UpgradeEnabled is False - so the script ends up trying to RegisterAndCreate - this fails because an earlier version of the application already exists.

A pair of brackets will fix this, as follows:
$UpgradeEnabled = $UpgradeEnabledInProfile -and -not ($OverrideUpgradeBehavior -eq ‘VetoUpgrade’)

Please could you arrange for this to be hotfixed on our hosted instance ASAP?

Many thanks,

Stuart Jones

1 Like
(Shane Gill) #3

Hi Stuart,

Thanks for getting in touch and doing the detective work.

We are in the process of fixing the issue, you can track our progress here: https://github.com/OctopusDeploy/Issues/issues/5657

We will patch your instance when the release is available.

To get unstuck immediately you can override the Service Fabric deploy script. Please see our Service Fabric documentation.

I hope this helps.

Cheers,
Shane

(Stuart Jones) #6

Hi Shane,

Thanks for that, and I can see that the issue has been closed off in GitHub.

However the issue still persists on our server and I can’t easily implement the workaround you suggested. I’ve got 19 separate microservice projects, seven separate environments, and a whole host of packages tied to releases which are in-flight, moving through from Dev -> Test -> Prod. The amount of effort required to recreate all those packages with a custom script in the root of the package is colossal, compared to how long it would take to hot-fix the script in situ on our instance. I can put a custom instance of that script in for any new packages, however that doesn’t solve our problem for the existing packages and releases.

I’m really disappointed that this hasn’t been fixed on our instance yet - the change to that script in the last Calamari release had clearly never been tested, because it cannot ever have worked - and as a result you’ve upgraded our instance (and no doubt countless others) with fundamentally defective, untested code, and left us in the position where we’ve already lost a full working day, unable to deploy anything to our Service Fabric clusters.

I’m sure that someone in your organisation must have admin credentials to be able to log on to our hosted instance; please can you get someone to log on and manually patch that file on our server as a matter of urgency.

Kind regards,

1 Like
(Ben Pearce) #7

Hi Stuart,

Apologies for headaches this has caused.
I have replaced the Calamari Cloud package on your instance with the latest version, 4.21.1, which includes the fix that Shane mentioned. The new package should be unpacked the next time you run a deployment.

Let me know if you have any further troubles or if you require the Calamari Cloud package to be rolled back for some reason.

Regards
Ben

1 Like
(Stuart Jones) #8

Hi Ben,

Many thanks for this - I can confirm that it’s working as intended now. Really appreciate you getting us back up and running again.

Many thanks,

Stuart