Ran into problems last night with most of our Azure Web App targets - suddenly they appeared unhealthy and periodically healthy. Been forcing updates to Calemari for all targets this morning (had to retry maybe 10 times per target!) and it seemed to work initially. However after a little while, nearly all targets are unhealthy again and for some reason Calemari update is now pending again. Anyone seen this issue before? In the log i get the following error: "Operation returned an invalid status code ‘Forbidden’ ". When in the healthy state we have no problems actually deploying to the targets. Confirmed with ops team and no changes has been made to network, Azure tenant etc. Any input on how to troubleshoot this?
Hi Jasper,
Thanks for getting in touch! I’m sorry to hear you’re hitting this confusing behavior here. Firstly, Calamari shouldn’t really be in play here. It doesn’t get updated on the Azure targets, but instead it’s on the server/worker that’s doing the deployment.
The Forbidden error in your stack trace screenshot almost looks like a 403 error. Is it possible the Service Principal account’s permissions have been altered in some way? Are you able to test with an Azure PowerShell script using this account to see if that shows the web apps it should have access to?
I hope this helps narrow it down, and I look forward to hearing back!
Best regards,
Kenny
Hi Kenneth,
Yeah - long shot it was something regarding Calamari - however they stayed green for a little while after the update. The service principal is still working and if we retry 8-10 times, we actually get access and can deploy (just like when I updated Calamari).
Latest status is that this morning every target is now healthy, with no additional changes and haven’t had unhealthy targets since 2100 last night. So no sure what to conclude from this.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.