Azure cmdlets causing deployment failurs

The azure powershell cmdlets such as Set-AzureSubscription, Select-AzureSubscription and Remove-azureSubscription use common file across all powershell instances for managing the azure subscriptions. For the system user (account that runs the tentacles), this path is C:\Windows\system32\config\systemprofile\AppData\Roaming\Windows Azure Powershell. We run a tentacle for each service we deploy to azure (currently 22 services) which has led to a situation where deployments fail if two or more kick off close to each other. This seems to be happening more and more often for us as we ramp up our use of Octopus. The exceptions thrown are due to file locking issues (often for set-azuresubscription) as well as subscription entries ‘going missing’ (often for remote-azuresubscription) from the common files.
We have been able to reproduce one of the issue by editing the ‘deploytoazure.ps1’ for two projects to just sleep for 60 seconds and then kicking off a deployment for the two projects off about 20 seconds apart. One will also fail with an exception in the set-azuresubscription command.
To work around this we have had to edit the bootstrapdeploytoazure script to add some retry / try catch around ‘set-azuresubscription’ and adding the –SubscriptionDataFile option to all *–azuresubscription cmdlets so each deploy uses its only subscription file.
Attached is our hacked up script if anyone is interested.
Hoping Octopus might add some retry / fault handling to future releases.

BootstrapDeployToAzure.ps1 (3 KB)

Hi Tim,

Thanks for reaching us. A couple of questions to understand your environment a bit more:

  • Are you always using the same Azure subscription with Set-AzureSubscription?

  • When you say “We run a tentacle for each service we deploy to azure (currently 22 services)” does this mean you have 22 servers running as tentacles and you deploy 1 service from each? or you deploy the 22 services from only one tentacle/server in 22 different steps?

Dalmiro

Most of our services share the same Azure subscription for a given environment so often one subscription is used for more than one simultaneous deployment.
We have 22 octopus projects each with their own tentacle, all tentacles run on the same server, each using a different port. Each deployment has many steps (send email, deploy to azure, vip swap script, ect)
Thanks
tim

Hi Tim,

Thanks for all the information you have provided. We have wanted to work better with the cmdlets but have never been given enough information to replicate and or produce issues, but you have given us a start!
So I have created a GitHub issue which you can track here: https://github.com/OctopusDeploy/Issues/issues/1345

Thanks again!
Vanessa