Deploy an Azure App Service

This step was working for a while, deploying a container to the App Service, and it has suddenly stopped, and I’m not sure where to look to find the actual error as all I get reported back is this in the step and logs

Updating web app to use image <redacted>.azurecr.io/rm-ui:develop from registry <redacted>.azurecr.io 
Updating app settings with container registry 
Updating configuration with container image 
Running rollback behaviours... 
Operation returned an invalid status code 'BadRequest' 
Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponseException 
   at Microsoft.Azure.Management.WebSites.WebAppsOperations.UpdateConfigurationWithHttpMessagesAsync(String resourceGroupName, String name, SiteConfigResource siteConfig, Dictionary`2 customHeaders, CancellationToken cancellationToken) 
   at Calamari.AzureAppService.Extensions.UpdateConfigurationAsync(IWebAppsOperations operations, TargetSite targetSite, SiteConfigResource config, CancellationToken cancellationToken) in C:\buildAgent\work\cdb95c8a359b9bc9\source\Calamari\Extensions.cs:line 108 
   at Calamari.AzureAppService.Behaviors.AzureAppServiceDeployContainerBehavior.Execute(RunningDeployment context) in C:\buildAgent\work\cdb95c8a359b9bc9\source\Calamari\Behaviors\AzureAppServiceDeployContainerBehavior.cs:line 74 
   at Calamari.Common.Plumbing.Pipeline.PipelineCommand.ExecuteBehaviour(RunningDeployment context, IBehaviour behaviour) 
   at Calamari.Common.Plumbing.Pipeline.PipelineCommand.Execute(ILifetimeScope lifetimeScope, IVariables variables) 
   at Calamari.Common.Plumbing.Pipeline.PipelineCommand.Execute(ILifetimeScope lifetimeScope, IVariables variables) 
   at Calamari.Common.CalamariFlavourProgramAsync.Run(String[] args) 
The remote script failed with exit code 100 
The action Deploy UI to Azure on <redacted> failed 

Good afternoon @JamesH,

Thank you for contacting Octopus support and sorry to hear you are having issues with the ‘Deploy an App Service’ step.

Are you able to provide us with the raw logs for that falling deployment please so we can see the full log.

I have created you a secure link here so you can upload them (feel free to omit any sensitive values but it would only be Octops Support staff that would see the raw task log).

Are you able to connect to that Webapp via a health check through the Octopus UI?

If not are you able to connect to that web app outside of Octopus using Azure CLI? This would let us know if the Octopus server machine is able to connect to the web app, if the step was working before and now it suddenly isn’t without you changing any of the step settings it would indicate a potential environmental problem so it would be good to know if you can communicate with the webapp from outside of Octopus.

Let me know if you can upload the raw logs and when you have uploaded them (we don’t get a notification when a customer uploads a file to our secure site) and the outcome of the Azure CLI testing too.

If there is anything else I can help you with in the meantime please reach out,

Kind Regards,

Clare Martin

Hi Clare

So the process works fine if I deploy the same step using a package to a different web app. It just seems the container is breaking now for some reason.

Interestingly if I clone the container step, the process adds a v prefix to the container tag, so rm-ui:develop transforms into rm-ui:vdevelop which is odd. So I have to manually re-add the step and then that part works fine. Probably a new topic needs opening.

I’ll upload the raw logs for you now. I’ve created single step runbook for this now.

We were on a couple of versions out of date but I upgraded to 2021.3.12313 last night. I see there’s a new version out now.

Many thanks

James.

The file has been uploaded now

1 Like

Just an FYI, I’ve created a new step that deploys a container to a new App Service and that woks fine. So It’s something with this App Service possibly - but it would be good to have a more detailed error message.

I’ll see if the CLI throws back more info

Hi @JamesH,

Sorry for the late response here, I was just going over your logs after I got back from lunch and there is nothing that jumps out at me, I was going to ask if you had any luck with the Azure CLI so I look forward to hearing from you with regards to how your testing goes with that.

It seems like you have confirmed that it is a specific App Service that’s having the issue though (some good fault finding from your end there, thank you for sharing it with us).

I agree, the error message is extremely vague and doesn’t give you anything to go off, I assume we get that from Azure itself as we are usually very descriptive with our error messages (most of them even come with handy links to documentation!).

I will await the results of the Azure CLI testing but reach out in the meantime if you have any other issues,

Kind Regards,

Clare Martin

I pushed a new image to the app and it seemed to work fine, I used

az webapp config container set --name <redacted> --resource-group <redacted> --docker-custom-image-name <redacted>/rm-ui:develop

Any more ideas ?

Hey @JamesH,

Just for my understanding of this ticket are you able to confirm the following is correct:

  • You are trying to push a package to an azure webapp via the Deploy an Azure App Service Step.
  • It is erroring out with Operation returned an invalid status code 'BadRequest'.
  • On the same step you changed it to deploy to a different Webapp and it worked fine.
  • You then created a completely new step that deploys a package to another webapp and that works fine…
  • You have just tried to connect to that webapp (the one that wasn’t working) with Azure CLI and it connects to the webapp fine?

The webapp itself in Octopus (on the Infrastructure page) is it passing its health check? I know I mentioned it before but I am not sure whether we answered that or not.

Have you restarted the webapp (I know this sounds like a silly request but it would be good to rule out any potential ‘service issues’ with the webapp itself).

Last thing, if you have restarted the webapp and if the health check on the webapp then passes in Octopus are you able to try and push the new image you had running in the Azure CLI to the webapp via Octopus (as we know that full test works outside of Octopus).

I am keen to try and get this working for you and I understand running these troubleshooting steps could be frustrating but we still don’t have much to go off other than we think Octopus has an issue with that webapp and the only error message we are getting is ‘Bad request’, which isn’t very helpful as you said!

Let me know the outcome of the additional tests and we can go from there,

Kind Regards,

Clare Martin

You are trying to push a package to an azure webapp via the Deploy an Azure App Service Step.
It is erroring out with Operation returned an invalid status code ‘BadRequest’.

Yes

On the same step you changed it to deploy to a different Webapp and it worked fine.

Yes but as a package and not a container - to a different deployment target

You then created a completely new step that deploys a package to another webapp and that works fine…

Yes It did, but that now also fails with the same error. I seem to have a day’s grace period where they work fine and then stop.

You have just tried to connect to that webapp (the one that wasn’t working) with Azure CLI and it connects to the webapp fine?

The CLI seems to work fine

The webapp itself in Octopus (on the Infrastructure page) is it passing its health check? I know I mentioned it before but I am not sure whether we answered that or not.

Yes

Have you restarted the webapp (I know this sounds like a silly request but it would be good to rule out any potential ‘service issues’ with the webapp itself).

Yes

Actually the web app activity log is showing a bad request error with lots of data. Can I use the previous link to upload it for you ?

Hey @JamesH,

Is it an Azure log from the webapp or an Octopus log?

Send it over using the same link I posted earlier and I will see if I can advise you on the errors you are seeing,

Let me know once you have uploaded them,

Kind Regards,

Clare Martin

Hi Clare

It’s an azure log.

I think it boils down to

Required parameter AccessKey is missing.

So by the looks of things, first time I use the step to deploy is fine and then subsequest ones for the container fail. Only a guess though

It’s uploaded now.

James.

Hey @JamesH,

So I have had a look at the log, since it is an Azure log I can’t comment with certainty on what is going on. Ideally, you now need to speak to Azure to diagnose this issue as I cannot recommend any fault-finding tips with your web app, this needs to be done via Azure.

If Octopus gets you to change any details on that web app and it breaks the web app Azure will not help you fix it as another company has instructed you to change something.

I did find the websites I have listed below though, I have no idea if they will help but I thought I would at least try and be helpful rather than just send you to Azure.

It seems there were a few issues with web apps that have mounts attached to them, all the websites below report a 400 error code and the errors you are seeing in the Azure logs, which I have shown below for other users experiencing this issue (sensitive values omitted of course).

"status": {
        "value": "Failed",
        "localizedValue": "Failed"
    },
    "subStatus": {
        "value": "BadRequest",
        "localizedValue": "Bad Request (HTTP Status Code: 400)"
    },
    "submissionTimestamp": "2022-03-03T11:26:46.4541513Z",
    "subscriptionId": "xxxxxxxx",
    "tenantId": "xxxxxxxx",
    "properties": {
        "statusCode": "BadRequest",
        "serviceRequestId": null,
        "statusMessage": 
"{\"Code\":\"BadRequest\",\"Message\":\"Required
 parameter AccessKey is missing.\",\"Target\":null,\"Details\":[{\"Message\":\
"Required parameter AccessKey is missing.\"},{\"Code\":\"BadRequest\"},
{\"ErrorEntity\":{\"ExtendedCode\":\"01010\",\"MessageTemplate\"
:\"Required parameter {0} is missing.\",\"Parameters\":[\"AccessKey\"],\"Code\"
:\"BadRequest\",\"Message\":\"Required
 parameter AccessKey is missing.\"}}],\"Innererror\":null}",

I am sorry I cannot help you further here but I hope I have given you some direction in which to go to Azure with those logs and get them to diagnose them for you.

Please let me know if there is anything we can do in the meantime and please reach out if you have any other issues, I hope you get some answers from Azure on a fix for this.

Kind Regards,

Clare Martin

So I’m using the cotpus deploy docker image for the worker pool

octopusdeploy/worker-tools:3.3.2-ubuntu.18.04

So I’m not sure how I can change what SDK is used short of building my own image.

Ther is no option to add a mount path in the Octopus Step, so are you saying I should create my own Step Template ? Probably easier to get it working I guess.

James.

Actually I could probably remove the mount path, run the step and then re-add it as part of the deployment. In fact I’ll try that

It’s also interesting that the cli command works fine without mentioning the mount path

Hey @JamesH,

Yea try remove the mount path and see if that works, from what I have managed to read it seems like that issue is specific to mount paths but you would have to talk to Azure to get the most up to date information on that issue and how it affects webapps.

I am not sure how your Azure Webapps are setup, I know when I run tests for customers I just create a bog-standard webapp in Azure with no mounts, Octopus then just deploys to it as a target once created so I have not had any exposure to mounts inside Azure with regards to Webapps.

It might not be that though, without getting info from Azure on what the issue could be with the app itself I am just going off a typical google search so I may be completely in the wrong ballpark with my website links.

Let me know your findings, it would be interesting to see if removing the mount allows you to successfully deploy with the step inside Octopus. If it does this would confirm our suspicions.

Kind Regards,

Clare Martin

Hi Clare

Thanks. Unfortunately removing a mount path restarts the app, so my new plan is to deploy the new container to a deployment slot, mount the path and then switch the deployment slots for the app.

Once I get this working I’ll let you know

James.

1 Like

For anyone else reading this. It’s reasonably straight forward to delete the path mapping, deploy the container and re-add the path mapping. I believe this also restarts the container, so not ideal for production.

It’s easier to deploy the webapp to a deployment slot, and then add the path mapping afterwards. Then switching the slots as per your deployment process.

Obviously keep in mind that deployment slots are only available using a Production App Service plan. So for Basic App Service plans you’ll need to delete the mount path first.

This resource has the az command details - az webapp config storage-account

1 Like