Azure App Service zip deployment keeps failing

I have a deployment step that does a new style zip deployment of an Azure app service. The step deploys a site packaged as a nupkg to 6 azure app services. It continually fails to deploy to 5 out of the 6 with a “Conflict” error. When I check the deployment logs at Azure, it says the deployment was a success. When I rerun the deployment, it always works.

We’re using 2022.1 (2584).

Simon

Conflict 
June 6th 2022 15:57:08
Error
System.Exception 
June 6th 2022 15:57:08
Error
   at Calamari.AzureAppService.Behaviors.AzureAppServiceBehaviour.UploadZipAsync(PublishingProfile publishingProfile, HttpClient client, String uploadZipPath, String targetSite) in C:\buildAgent\work\cdb95c8a359b9bc9\source\Calamari\Behaviors\AzureAppServiceBehaviour.cs:line 155 
June 6th 2022 15:57:08
Error
   at Calamari.AzureAppService.Behaviors.AzureAppServiceBehaviour.Execute(RunningDeployment context) in C:\buildAgent\work\cdb95c8a359b9bc9\source\Calamari\Behaviors\AzureAppServiceBehaviour.cs:line 114 
June 6th 2022 15:57:08
Error
   at Calamari.Common.Plumbing.Pipeline.PipelineCommand.ExecuteBehaviour(RunningDeployment context, IBehaviour behaviour) 
June 6th 2022 15:57:08
Error
   at Calamari.Common.Plumbing.Pipeline.PipelineCommand.Execute(ILifetimeScope lifetimeScope, IVariables variables) 
June 6th 2022 15:57:08
Error
   at Calamari.Common.Plumbing.Pipeline.PipelineCommand.Execute(ILifetimeScope lifetimeScope, IVariables variables) 
June 6th 2022 15:57:08
Error
   at Calamari.Common.CalamariFlavourProgramAsync.Run(String[] args) 
June 6th 2022 15:57:09
Verbose
Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in e:\Octopus\Work\20220606145141-133900-4988 exited with code 100 
June 6th 2022 15:57:09
Verbose
Updating manifest with output variables 
June 6th 2022 15:57:09
Verbose
Updating manifest with action evaluated variables 
June 6th 2022 15:57:09
Fatal
The remote script failed with exit code 100 

Hi Simon,

Thanks for posting your question, and sorry to hear you’re experiencing that issue.

I’m not sure yet what could be causing this, but I was wondering if you’d be willing to send me a raw task log of when the deployment fails, and one where you ran it after the failure and it completed?

You can upload these to me securely using the following link: Support - Octopus Deploy

Let me know once you’ve done that (unfortunately I don’t receieve any notification when files are sent), and I’ll review them and let you know if I find anything.

I’m looking forward to hearing back from you!

Best,
Patrick

I’ve uploaded the raw log file for the deployment failing. I can also upload the log file from Azure as well, if that would help.

I’ve also attached the log file for me retrying the failed deployment & it succeeding.

Hi Simon,

Thanks for sending those logs through. I’ve reviewed them and I’m not seeing much pointing to this being on the Octopus side of things, however, I did find the following open GitHub issue that seems to align with what you’re seeing: Receiving "Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)" when deploying our app · Issue #3042 · projectkudu/kudu · GitHub

I’m curious what you think, but from what I’ve gathered in the issue, this seems to happen when multiple deployments go out quickly. That looks to me like it fits your scenario, so if the other details match (e.g. S1 tier on app service, or deploying to a Linux function app) I’m wondering if you’d be willing to test some of the workarounds mentioned toward the end of the issue see if any of them alleviates the problem?

Let me know what you think and we can go from there!

Best,
Patrick

From my investigation, I can’t see any errors logged either. Our deployment differs in that it’s a Windows Web app. All the deployment logs at Azure report a successful deployment - I’ve attached the corresponding logs, although they don’t offer much, other that indicating everything worked.

The 5 that fail do share the same app service plan, which is suspicious. I suspect I may need to raise a ticket with Microsoft.

Hi Simon,

Thanks for having a look through that issue, and for getting back to me. I think I saw some of those reports came from Windows Web Apps, and they were on the same App Service Plan (S1 likely), so I agree that sounds suspicious as well.

I also agree those logs look clean and unfortunately, I’m not quite sure where or how this would show up in the Azure logs as I’m not as familiar with that end of things. It’s probably a good idea if you’re willing to raise a ticket with Microsoft to see if they’re able to help out here, but keep me posted on how that goes.

Also, feel free to reach out in the meantime with any further questions or findings.

Best,
Patrick