Deploying Azure Web Apps in multiple region failures

(bil.simser) #1

Hi guys,

I have an Azure web app that’s running in two different regions. I’ve setup both apps under the same deployment target so when I deploy the app it goes to both regions. I’ve also got the deployment setup as a rolling deployment. A few options options on the step: we remove files that are not part of the deployment and we do enable the appoffline feature.

The problem is that (sometimes) the deploy fails because of a locked file. E.G:

Deploying to Azure WebApp ‘WEBAPP’ in Resource Group ‘RESOURCEGROUP’, using subscription-id ‘SUBSCRIPTION’ Microsoft.Web.Deployment.DeploymentDetailedClientServerException: Web Deploy cannot modify the file ‘msvcr120.dll’ on the destination because it is locked by an external process.

Retrying the deploy works (usually).

This will probably happen in one region, not both.

I thought the AppOffline feature would resolve this but it doesn’t. We were using deplolyment slots, but those are also temperamental as the swapping doesn’t always work (and we can still get this same error in a slot).

The way I solved this with another application in Azure is to run a “shutdown app” (using PowerShell) then start it up again after the deploy. That resulted in an outage but it always worked.

On the new app I can do this without an outage because we’re running a load balancer in front of them. The problem is I only have a single step so I can’t how to run a custom script/command as part of the rolling development.

Or is there another way to fix this and have reliable deploys?

Thanks

(Daniel Fischer) #3

Hi @bil_simser,

Thanks for getting in touch! It looks like you are on the right track using our Rolling Deployment feature. This feature is generally what we suggest for load balancer situations where you need to remove a target form the load balancer, run the entire deployment on a single target, then do the same to the next target.

My suggestion would be to add the script step into the rolling deployment. The rolling deployment allows you to add multiple different steps and step types including a script step.

If I have missed something here, or misinterpreted your issue, please let me know.

Best regards,
Daniel

(bil.simser) #4

Hi Daniel,

Tried a bunch of things but nothing is working. Referring to the rolling development page I created two child steps, Stop App Service and Start App Service to bookend the Deploy Packages step. These are both Azure PowerShell steps that
I’ve used in the past to stop a web application.

So the Stop step is simply:

Stop-AzureWebSite -Name $AppService

Where $AppService is a variable I have setup scoped to the target and environment for each App Service in Azure.

All steps run under Azure Accounts (which are of type AzureAccount bound to the environments as we have one for each environment).

Here’s the log dump on the failure. The error at 8:27:41 tries to run the Stop-AzureWebSite cmdlet but fails because it says no default subscription has been designated and to use Select-AzureSubscription to set one. I’ve tried calling
Select-AzureSubscription -Default with the Subscription name, id, etc. (all text variables) but they always fail saying they can’t find the subscription (and I’ve confirmed there’s no typo). When you look at the script it does log into Azure correctly and
come back with the service principal information, so it is signing in correctly to Azure and even knows the SubscriptionId, however I can’t seem to use these values anywhere else. I’ve tried information from the page

https://octopus.com/docs/deployment-process/variables/azure-account-variables
to use these in PowerShell but nothing comes back as it’s described on the page.

Here’s the scrubbed log from Octopus. Let me know if you have any ideas why this failure is happening and what we might be missing.

Failed: [REDACTED]

08:27:42 Fatal | The action Stop App Service on [REDACTED] failed

  Failed: Stop App Service (on the Octopus Server)

08:27:14 Verbose | Octopus Server version: 2018.8.4+Branch.master.Sha.1136e3e30f9ec4e8c4c8f2beb7688e7d95915fff

08:27:14 Verbose | Environment Information:

    OperatingSystem: Microsoft Windows NT 10.0.14393.0
    OsBitVersion: x64
    Is64BitProcess: True
    CurrentUser: [REDACTED]
    MachineName: [REDACTED]
    ProcessorCount: 2
    CurrentDirectory: C:\Windows\system32
    TempDirectory: C:\Temp\
    HostProcessName: Octopus.Server
    PID: 3544

08:27:14 Verbose | Executing Stop App Service (type Run an Azure PowerShell Script) on Octopus Server

08:27:14 Verbose | Using account ID ‘[REDACTED]’

08:27:14 Verbose | Account variables are being contributed by the target

08:27:14 Verbose | Using Calamari.Cloud 4.9.1

08:27:14 Verbose | Using Octopus.Dependencies.AzureCmdlets 5.7.0

08:27:14 Verbose | Using Octopus.Dependencies.AzureCLI 2.0.42

08:27:14 Verbose | Running this script in the Octopus Server security context ([REDACTED])

08:27:14 Verbose | Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory ‘C:\Octopus\Work\20190617142714-59393-7050’ using ‘OEM United States’ encoding running as ‘[REDACTED]’ with the same environment
variables as the launching process

08:27:16 Verbose | Octopus Deploy: Calamari version 4.9.1

08:27:16 Verbose | Environment Information:

08:27:16 Verbose | OperatingSystem: Microsoft Windows NT 10.0.14393.0

08:27:16 Verbose | OsBitVersion: x64

08:27:16 Verbose | Is64BitProcess: True

08:27:16 Verbose | CurrentUser: [REDACTED]

08:27:16 Verbose | MachineName: [REDACTED]

08:27:16 Verbose | ProcessorCount: 2

08:27:16 Verbose | CurrentDirectory: C:\Octopus\Work\20190617142714-59393-7050

08:27:16 Verbose | TempDirectory: C:\Temp\

08:27:16 Verbose | HostProcessName: Calamari

08:27:16 Verbose | Performing variable substitution on ‘C:\Octopus\Work\20190617142714-59393-7050\Script.ps1’

08:27:16 Verbose | Executing ‘C:\Octopus\Work\20190617142714-59393-7050\Script.ps1’

08:27:17 Verbose | Name Value

08:27:17 Verbose | ---- -----

08:27:17 Verbose | PSVersion 5.1.14393.2879

08:27:17 Verbose | PSEdition Desktop

08:27:17 Verbose | PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}

08:27:17 Verbose | BuildVersion 10.0.14393.2879

08:27:17 Verbose | CLRVersion 4.0.30319.42000

08:27:17 Verbose | WSManStackVersion 3.0

08:27:17 Verbose | PSRemotingProtocolVersion 2.3

08:27:17 Verbose | SerializationVersion 1.1.0.1

08:27:18 Verbose | PowerShell Environment Information:

08:27:18 Verbose | OperatingSystem: Microsoft Windows NT 10.0.14393.0

08:27:18 Verbose | OsBitVersion: x64

08:27:18 Verbose | Is64BitProcess: True

08:27:18 Verbose | CurrentUser: [REDACTED]

08:27:18 Verbose | MachineName: [REDACTED]

08:27:18 Verbose | ProcessorCount: 2

08:27:18 Verbose | CurrentDirectory: C:\Octopus\Work\20190617142714-59393-7050

08:27:18 Verbose | CurrentLocation: C:\Octopus\Work\20190617142714-59393-7050

08:27:18 Verbose | TempDirectory: C:\Temp\

08:27:18 Verbose | HostProcessName: powershell

08:27:18 Verbose | TotalPhysicalMemory: 8385564 KB

08:27:18 Verbose | AvailablePhysicalMemory: 5237776 KB

08:27:20 Verbose | Authenticating with Service Principal

08:27:21 Verbose | Account : [REDACTED]

08:27:21 Verbose | SubscriptionName : [REDACTED]

08:27:21 Verbose | SubscriptionId : [REDACTED]

08:27:21 Verbose | TenantId : [REDACTED]

08:27:21 Verbose | Environment : AzureCloud

08:27:26 Verbose | Azure CLI: Authenticating with Service Principal

08:27:31 Verbose | [

08:27:31 Verbose | {

08:27:31 Verbose | “cloudName”: “AzureCloud”,

08:27:31 Verbose | “id”: “[REDACTED]”,

08:27:31 Verbose | “isDefault”: true,

08:27:31 Verbose | “name”: “[REDACTED]”,

08:27:31 Verbose | “state”: “Enabled”,

08:27:31 Verbose | “tenantId”: “[REDACTED]”,

08:27:31 Verbose | “user”: {

08:27:31 Verbose | “name”: “[REDACTED]”,

08:27:31 Verbose | “type”: “servicePrincipal”

08:27:31 Verbose | }

08:27:31 Verbose | }

08:27:31 Verbose | ]

08:27:31 Verbose | Azure CLI: Setting active subscription to [REDACTED]

08:27:35 Verbose | Successfully authenticated with the Azure CLI

08:27:35 Verbose | Invoking target script “C:\Octopus\Work\20190617142714-59393-7050\Script.ps1” with parameters

08:27:41 Error | Stop-AzureWebSite : No default subscription has been designated. Use Select-AzureSubscription -Default

08:27:41 Error | to set the default subscription.

08:27:41 Error | At C:\Octopus\Work\20190617142714-59393-7050\Script.ps1:19 char:1

08:27:41 Error | + Stop-AzureWebSite -Name $AppService

08:27:41 Error | + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

08:27:41 Error | + CategoryInfo : CloseError: (:slight_smile: [Stop-AzureWebsite], ApplicationException

08:27:41 Error | + FullyQualifiedErrorId : Microsoft.WindowsAzure.Commands.Websites.StopAzureWebsiteCommand

08:27:41 Verbose | Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20190617142714-59393-7050 exited with code 1

08:27:42 Verbose | Updating manifest with output variables

08:27:42 Verbose | Updating manifest with action evaluated variables

08:27:42 Fatal | The remote script failed with exit code 1

(Ben Pearce) #5

Hi Bil,

Can I ask you to try some troubleshooting steps to try and narrow down the problem?

Firstly, do you have this problem on any other Azure script steps?
Are these steps running on the server or an external worker?
In your log, do the subscription values you are getting appear to be the ones you expect?

Can I get you to add the cmdlet Get-AzureRmContext to the top of the scripts you are running and see if the output from this cmdlet gives you back the expected subscription?

I noticed that you are running an older version of Octopus Server, we have made some changes to the Azure login process since this version, so upgrading may help. One change specifically deals with isolating the Azure login in PowerShell so that it doesn’t interfere with any other saved Azure contexts on the machine. You can test this by running Get-AzureRmContext directly on the machine or via the Script Console in the Octopus UI. If this cmdlet returns a valid context this may be the issue and you will need to log it out.

As a workaround in the meantime, you could also stop and start the Azure Web App using the Azure CLI stop and start commands.

Also, if you continue to have trouble can you also export the raw log with variables. If you remove any values for security, would you be able to leave some of the values visible, so I can tell if they are the same or different to other values in the log file. You can also email the log file directly to me, ben.pearce@octopus.com.

Hopefully, some of this helps you out.

Regards
Ben

(Ben Pearce) #7

Hi Bil,

I’m sorry that I didn’t notice this in my previous response, but I think Stop-AzureWebSite is the wrong cmdlet.
When dealing with Azure Resource Manager, which you are since you are using an Azure Service Principal account, all the cmdlets will be *-AzureRm*, so your script should be using Stop-AzureRmWebApp and Start-AzureRmWebApp. The same goes for the select subscription command, which should be Select-AzureRmSubscription.

The version of the Azure PowerShell modules that shipped in your version of Octopus is getting a bit old as well, you can upgrade the PowerShell module version by installing the latest version on your Octopus server and following our documentation to use the installed version rather than the bundled version.

Hopefully, this helps you get your deployment moving again.

Regards
Ben

(system) closed #10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.