Remove-AzureRmWebAppSlot Hang

Hi,

Do you have any advice for how I might diagnose why invoking Remove-AzureRmWebAppSlot (AzureRM.Websites Module v5.2.1) succeeds only on occasions, and otherwise simply seems to hang indefinitely.

I am migrating to Octopus Server v2021.3 (Build 8275) from a currently reliable working Octopus Server v2018.7.7.

The same Azure Script task runs on both servers and succeeds reliably on the old server.

Successful slot removal log:
Authenticating with Service Principal
WARNING: Because the Az modules have all the capabilities of AzureRM modules
and more, we will retire the AzureRM modules on 29 February 2024. Learn more
about the migration to Az: Migrate Azure PowerShell scripts from AzureRM to Az | Microsoft Docs.
Environments : {[AzureChinaCloud, AzureChinaCloud], [AzureCloud, AzureCloud],
[AzureGermanCloud, AzureGermanCloud], [AzureUSGovernment,
AzureUSGovernment]}
Context : Microsoft.Azure.Commands.Profile.Models.PSAzureContext
Invoking target script W:\Octopus\Work\20211216141259-167761-26\Octopus.FunctionAppenderContext.ps1 with parameters
Invoking target script W:\Octopus\Work\20211216141259-167761-26\Script.ps1 with parameters.
WARNING: Because the Az modules have all the capabilities of AzureRM modules
and more, we will retire the AzureRM modules on 29 February 2024. Learn more
about the migration to Az: Migrate Azure PowerShell scripts from AzureRM to Az | Microsoft Docs.
[ServerTasks-167761] [RunningScript] [Read Lock] [“ServerTasks-167761” (has a read lock)] Releasing lock.
Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in W:\Octopus\Work\20211216141259-167761-26 exited with code 0
Updating manifest with output variables
Updating manifest with action evaluated variables
Successfully finished Delete Staging Slot on WebApp

Hanging slot removal log:
Authenticating with Service Principal
WARNING: Because the Az modules have all the capabilities of AzureRM modules
and more, we will retire the AzureRM modules on 29 February 2024. Learn more
about the migration to Az: Migrate Azure PowerShell scripts from AzureRM to Az | Microsoft Docs.
Environments : {[AzureChinaCloud, AzureChinaCloud], [AzureCloud, AzureCloud],
[AzureGermanCloud, AzureGermanCloud], [AzureUSGovernment,
AzureUSGovernment]}
Context : Microsoft.Azure.Commands.Profile.Models.PSAzureContext
Invoking target script W:\Octopus\Work\20211216141259-167761-30\Octopus.FunctionAppenderContext.ps1 with parameters
Invoking target script W:\Octopus\Work\20211216141259-167761-30\Script.ps1 with parameters.
WARNING: Because the Az modules have all the capabilities of AzureRM modules
and more, we will retire the AzureRM modules on 29 February 2024. Learn more
about the migration to Az: Migrate Azure PowerShell scripts from AzureRM to Az | Microsoft Docs.
[ServerTasks-167761] [RunningScript] [Read Lock] [“ServerTasks-167761” (has a read lock)] Releasing lock.
Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in W:\Octopus\Work\20211216141259-167761-26 exited with code 0

Thanks,

Brian

Good afternoon @InstandaDeployments,

Thank you for contacting us and sorry to hear you are having issues with your deployments.

Thankyou for sending over those logs, are you able to send us over the full log of a task that works and one that doesn’t so we can get a better understanding of what is going on. I have created you a secure link so you can upload the files, only Octopus Support staff will be able to see these so your logs will be kept secure.

Are you able to run that command outside of Octopus and see if that works successfully or not, we can then see if it is environmental or an issue with Octopus itself.

Finally, are you able to check what powershell modules you have installed. It looks like you are invoking an AzureRM command which Microsoft is depreciating in 2024.

If you have AzureRM and AzureCLI powershell modules installed on the same box it can cause conflicts, are you able to confirm this is not the case.

I look forward to hearing from you but in the meantime please reach out if you have any other queries.

Kind Regards,

Clare Martin

Hi Clare,

I suspect the issue might be that issuing the command fails, either to establish an exclusive lock, or fails to poll for the exclusive lock status, on the resource. I found an error logged on Azure for the app service deployment slot and have uploaded the error details.

Full logs uploaded.

The command does succeed when executed in PowerShell on the server directly, i.e. not from within an Octopus deployment task.

I am aware there are issues with having multiple versions of the same module installed, or having legacy and current modules installed. I believe this not to be case.

Output from Get-Module -ListAvailable -Name Az* on Octopus server:

ModuleType Version Name


Script 5.3.1 Azure
Script 0.5.4 Azure.AnalysisServices
Script 4.6.1 Azure.Storage
Script 6.13.2 AzureRM
Script 0.6.15 AzureRM.AnalysisServices
Script 6.1.8 AzureRM.ApiManagement
Script 0.1.9 AzureRM.ApplicationInsights
Script 6.1.2 AzureRM.Automation
Script 4.1.6 AzureRM.Batch
Script 0.14.7 AzureRM.Billing
Script 5.0.7 AzureRM.Cdn
Script 0.9.13 AzureRM.CognitiveServices
Script 5.9.2 AzureRM.Compute
Script 0.3.8 AzureRM.Consumption
Script 0.2.13 AzureRM.ContainerInstance
Script 1.0.11 AzureRM.ContainerRegistry
Script 5.0.4 AzureRM.DataFactories
Script 0.5.12 AzureRM.DataFactoryV2
Script 5.1.5 AzureRM.DataLakeAnalytics
Script 6.2.2 AzureRM.DataLakeStore
Script 4.0.10 AzureRM.DevTestLabs
Script 5.1.1 AzureRM.Dns
Script 0.3.8 AzureRM.EventGrid
Script 0.7.1 AzureRM.EventHub
Script 4.1.9 AzureRM.HDInsight
Script 5.1.6 AzureRM.Insights
Script 3.1.9 AzureRM.IotHub
Script 5.2.2 AzureRM.KeyVault
Script 4.1.5 AzureRM.LogicApp
Script 0.18.6 AzureRM.MachineLearning
Script 0.2.8 AzureRM.MarketplaceOrdering
Script 0.10.5 AzureRM.Media
Script 6.11.2 AzureRM.Network
Script 5.0.4 AzureRM.NotificationHubs
Script 5.0.7 AzureRM.OperationalInsights
Script 1.1.1 AzureRM.PolicyInsights
Script 4.1.11 AzureRM.PowerBIEmbedded
Script 5.8.4 AzureRM.profile
Script 4.1.10 AzureRM.RecoveryServices
Script 4.5.3 AzureRM.RecoveryServices.Backup
Script 0.2.13 AzureRM.RecoveryServices.SiteRec…
Script 5.1.1 AzureRM.RedisCache
Script 0.3.13 AzureRM.Relay
Script 6.7.4 AzureRM.Resources
Script 0.6.14 AzureRM.ServiceBus
Script 0.3.16 AzureRM.ServiceFabric
Script 1.0.1 AzureRM.SignalR
Script 4.12.2 AzureRM.Sql
Script 5.2.1 AzureRM.Storage
Script 4.0.11 AzureRM.StreamAnalytics
Script 4.0.6 AzureRM.Tags
Script 4.1.4 AzureRM.TrafficManager
Script 4.0.6 AzureRM.UsageAggregates
Script 5.2.1 AzureRM.Websites

Thanks,

Brian

I think this might be a long standing bug with the AzureRM modules. By adding the “-AsJob” flag, the deployment step proceeds since it no longer waits upon the outcome of deleting the deployment slot.

I checked deployments done from the old Octopus server and the staging deployment slots mostly remain undeleted. This suggests to me that both the new and old Octopus servers are inconsistently deleting the deployment slots. The only difference being that the new server waits indefinitely for the command to complete and old server presumably timeouts out silently and then progresses to the next deployment step.

A workaround might be to add a timeout to the command. My only concern is that our deployment process might actually publishing to the same existing staging slot instead of actually deleting it and re-creating a new one.

Good morning @InstandaDeployments,

Your comment does seem to make sense, from your logs I can see it’s trying to delete the staging slot. I don’t want to post the name of your staging slot on here but if you look from line 521 to 528 in your main log you sent over where it discusses ‘VERBOSE: Performing the operation “Removing web app slot” on target’. It waits 35 minutes before it finds the staging slot, from the other logs you sent over it states there is a lock on that staging slot, so I think you are correct in that it is timing out as there is a lock on that slot.

Are you running that script on those two boxes at the same time to the same web slot? We have seen it before where it will hang if running two of the same script on the same web app.

The workaround was to set the variable OctopusBypassDeploymentMutex to False which will allow both scripts to proceed without hanging forever within Octopus.

Does this help you at all?

Anything else you need in the meantime please reach out!

Kind Regards,

Clare Martin

No, deployments are not being done on both servers simultaneously, so there should be no conflict. I am testing deployments on the new server against applications which are exclusively available to my deployment testing.

I have raised this issue with Microsoft and hope they conclude there is a bug with the AzureRM modules.

I am considering implementing some timeout logic around this command. Although it wont alleviate my concerns about deploying to the same slot over and over again.

I think I am now confident the issue lies with Microsoft.

Thank you for your assistance.

@InstandaDeployments,

Thanks for getting back to me, sorry I could not be more help but hopefully Microsoft agree with you and confirm it is a bug. I imagine they will ask you to edit your script to use AzureCLI instead as that is their new module.

If they don’t provide any help please feel free to post back in this topic and we can look at a workaround together if you get stuck with the timeout edits.

Anything else you need too please reach out!

Kind Regards,

Clare Martin

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.