Hi,
So TL;DR:
1. The new task is twice as slow as the old Deploy Azure Web App task
2. It also sometimes fails with a “Conflict” error. Full error later in this post.
I was quite excited to see the new app service deploy task that was announced in your blog: Improved Azure App Service deployments - Octopus Deploy
I upgraded Octopus Server to 2021.1, and found that it timed out (or so it seemed). I saw this was fixed in 2021.1.7316, so I upgraded again. Now, the deployment sits there reporting “Uploading package to (app service name)” for about 7 minutes, then fails with a “Conflict”. Looking in the Azure portal, I can see the deployment still being in progress, and eventually it succeeds, but a couple of minutes after Octopus has already reported failure.
Everything works great with the old “Deploy an Azure Web App (Web Deploy)” task – but I need the features of the new task, so that’s why I switched. The old task completes successfully in a little more than 3 minutes, which is less than half the time it takes for the new task to fail.
Uploading package to (appname)
June 18th 2021 07:15:20
Error
Running rollback behaviours…
June 18th 2021 07:15:20
Error
Conflict
June 18th 2021 07:15:20
Error
System.Exception
June 18th 2021 07:15:20
Error
at Calamari.AzureAppService.Behaviors.AzureAppServiceBehaviour.UploadZipAsync(HttpClient client, String uploadZipPath, String targetSite) in C:\buildAgent\work\cdb95c8a359b9bc9\source\Calamari\Behaviors\AzureAppServiceBehaviour.cs:line 161
June 18th 2021 07:15:20
Error
at Calamari.AzureAppService.Behaviors.AzureAppServiceBehaviour.Execute(RunningDeployment context) in C:\buildAgent\work\cdb95c8a359b9bc9\source\Calamari\Behaviors\AzureAppServiceBehaviour.cs:line 116
June 18th 2021 07:15:20
Error
at Calamari.Common.Plumbing.Pipeline.PipelineCommand.ExecuteBehaviour(RunningDeployment context, IBehaviour behaviour)
June 18th 2021 07:15:20
Error
at Calamari.Common.Plumbing.Pipeline.PipelineCommand.Execute(ILifetimeScope lifetimeScope, IVariables variables)
June 18th 2021 07:15:20
Error
at Calamari.Common.Plumbing.Pipeline.PipelineCommand.Execute(ILifetimeScope lifetimeScope, IVariables variables)
June 18th 2021 07:15:20
Error
at Calamari.Common.CalamariFlavourProgramAsync.Run(String[] args)
June 18th 2021 07:15:23
Fatal
The remote script failed with exit code 100
June 18th 2021 07:15:23
Fatal
The action Deploy an Azure App Service on (appname) failed
Any tips?
Hi @hallgeir.lien
Sorry to hear that you’re having an issue with the new Azure App Service step. This does seem like unusual behaviour, and I’d like to investigate further to see what could be happening here.
You mention that the conflict error happens sometimes; how often would you say it happens? I have seen the conflict error before when there is another deployment in progress. Could you confirm that no other deployments are running simultaneously to the web app when you see this happen?
If that isn’t the case - would it be possible to send over a copy of the full task log for both a successful deployment and a failed deployment, please? There may be something in those that could help us get to the bottom of this. Feel free to PM them to me if you don’t want them on a public forum.
Regards,
Thanks for getting back to me!
I can confirm that there’s no other deployments happening at the same time. One pattern that I do notice is that when it happens, it takes about 7 minutes to fail. When it completes, it’s in about 5.5 minutes.
Perhaps there’s some internal retry or something, and some timeout triggering the retry, causing the conflict?
Here’s the full log output for a failed run: Octopus app service error - Pastebin.com
Here’s for a successful one: Task ID: ServerTasks-14419Related IDs: Deployments-1864, Channels-45 - Pastebin.com
It seems to happen almost every time on first deploy, AND if I have previously deployed to the site some other way (e.g. with webdeploy). If I do repeated deploys, it’s usually fine.
Thanks for sending over those logs! It’s good to hear that you’re not stuck and are able to deploy after another attempt.
I can’t see anything that would help explain why this is happening. It’s interesting that it happens mostly on a first deployment to an existing site that you’ve used webdeploy on.
I’ll do some testing and attempt to reproduce the behaviour, I may need to raise the issue with our developers but I’ll let you know how I get on.
Regards,
1 Like
It’s worth mentioning that the deploy package is quite big (about 300MB or thereabouts). Not sure if this helps triggering the issue or not.
And I did a new deploy today - failed again with conflict. Last time I deployed was before the weekend, using the new app service deploy task.
Perhaps some warmup issues causing it to take longer the first time, causing the task to retry or something?
Seems to be very consistent - every morning when I do the first deploy, it fails with the same error. So I deploy in the evening to my environment, then go home for the day, come back the next day, run the deploy again, and it fails.
This is obviously a problem unless we do deploys every hour or so. Which we won’t do for the time being.
Hi @hallgeir.lien
Sorry to hear this is still happening. I’ve so far been unable to reproduce the error. If you don’t mind, could you export the process JSON and PM it to me, please? I’d like to take a look and see if I’m able to get a better understanding of what could be happening here.
Regards,
Sure thing. I will send it right away. Thanks!
Hi @hallgeir.lien
Thanks for sending over the process JSON. Unfortunately, I’ve still been unable to reproduce the issue.
Would it be possible for you only the ‘Deploy an Azure App Service’ step, please? I’d be interested to see if the conflict still happens when the other steps are disabled.
Regards,
Hi,
I’ve tried that earlier actually. The other step that writes to key vault is a recent addition, and the other steps are skipped for the Azure environment, so the App service deploy step has run in isolation with the same result.
Hi @hallgeir.lien
Thanks for confirming that you’ve tried that already. This really is odd behaviour. I’ve had a look over the logs again and noticed the following:
> 12:07:05 Verbose | Unable to get the system proxy settings due to not running under .Net Framework. Calamari will not use any proxy settings.
Now, this is just a theory; but if you have the Octopus Manager configured to ‘Use the proxy server configured in Internet Explorer’, could you change it to ‘Use a custom proxy server’ and manually enter the proxy details, please?
The reason I ask you to try this is to rule out any bugs that may be preventing Octopus from pulling the correct proxy information.
Please let me know how you get on with the above.
Regards,
We don’t use a proxy server:
In any case if there was an issue with a proxy, I’d think it would fail every time?
Ok - good to know. Of course, under normal circumstances, I would expect it to fail every time, but it’s good to rule that out as a possible cause.
I wonder if something is interfering with the transfer, possibly an anti-virus or similar? I’ll do some further testing with a larger package and let you know how I get on.
Regards,
Thanks!
Well it does succeed when I retry - I can’t disable the AV to test this theory though, but would find it strange that it works more or less everytime after it fails once, but ALWAYS fails after not doing any deploys for a while.
We can definitely not use this as it is now for anything other than some testing… Did you try with a larger deploy package, with a lot of files?
Our package is about 4500 files, and 300MB in size.
Hi @hallgeir.lien
Yes, I’ve also been trying with packages of various sizes.
I’ve been looking further into this, and it appears that other (non-Octopus) users have hit the same ‘conflict’ message for various reasons. The most common resolution I’ve been seeing is to upgrade the Azure App service plan.
You can read more about that on the GitHub issue submitted to Kudo here, this StackOverflow page, and a ticket that was opened with Microsoft here.
Many of the comments on the links above explain the same intermittent issue that you’re seeing. One even said that scaling up and back down again got rid of the issue (link). I think at this stage, it could be worth reaching out to Microsoft for them to confirm everything is running correctly in your service plan or even temporarily upgrading the service plan to check if this resolves it for you.
Please let me know how you’d like to proceed.
Regards,
Interesting!
I will try upgrading the app service plan to see if this helps. I might not be able to confirm that before monday though, as it works fine now, and probably will until it’s been idle for a while.
It is odd though that the old app service deploy task works flawlessly though. Is there a big change in how they are deploying, that would explain that difference?
An update - I upgraded from Standard S1 to Premium P1V2 (about the same price, so easy choice). The deployment time for a successful deploy was reduced from 5.5 minutes til about 2.5. So far it’s a good sign.
I’ll report back on Monday if it also solves the conflict issues.
Thanks for the update!
I found another interesting comment here saying his issue was due to storage limits of the app service plan; it could be worth investigating that side of things too.
Looking forward to hearing how it goes on Monday. Have a great weekend!
Regards,
Happy to report that it seems perfectly fine now! I don’t think the issue was storage limits to be honest - we were using Standard S1, which should have 50GB available. Our package is 300MB zipped, about 600 uncompressed. May be some timeouts triggering this perhaps…
Anyway, works great now it seems so thanks for the help! If anything changes I’ll report back to this thread.
