Azure powershell works, Octopus fails

Hi

I have a problem with the Octopus Azure functionality.

When I use the step “Run an Azure Powershell script” it has started to fail in Octopus. It runs and does what it should, but the state in Octopus is failure.

Observations:

  1. The scripts actually run successfully, but they end up with a failure status in Octopus deploy. I use them to swap Azure slot and run SQL - everything works. But the deploy fails in Octopus.

  2. It seems that it is the initial Azure authentication that fails. It says that the Azure subscription doesn’t exist. The subscription in Azure has the exact same GUID.

  3. The problems started somewhere between August 15th and August 23rd, 2018 - I am almost certain that nothing has been changed in my Azure account nor Octopus setup. Only Octopus Deploy has been regularly updated. The windows installation might have been updated.

  4. The same account (which is registred in Infrastructure/Accounts) works with no errors in “Deploy an Azure Web App” and “SQL - Execute Script file” (Community template) steps

Details below…

Can you help?

Cheers,
Christian


In an example where the step based on “Run an Azure Powershell script”, named “Do nothing” and contains nothing but a comment:

#Do nothing

…the log from the execution is:

Error: %(D was unexpected at this time.
Error: ERROR: The subscription of ‘YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY’ doesn’t exist in cloud ‘AzureCloud’.
Fatal: The remote script failed with exit code 1
Fatal: The action Do nothing on myproduct Prod WebApp Test slot failed
The subscription exists and the step actually works, but it ends up with a failure status.

The full verbose log:

Verbose: Octopus Server version: 2018.8.0+Branch.master.Sha.c206660c295650f47189fab7d64d3ac71b0309a9
Verbose:
Environment Information:
OperatingSystem: Microsoft Windows NT 10.0.17134.0
OsBitVersion: x64
Is64BitProcess: True
CurrentUser: NT AUTHORITY\SYSTEM
MachineName: SRV-BUILD-01
ProcessorCount: 4
CurrentDirectory: C:\WINDOWS\system32
TempDirectory: C:\WINDOWS\TEMP
HostProcessName: Octopus.Server
PID: 15528
Verbose: Executing Do nothing (type Run an Azure PowerShell Script) on Octopus Server
Verbose: Using account ID ‘azureserviceprincipal-myproduct’
Verbose: Account variables are being contributed by the target
Verbose: Using Calamari.Cloud 4.8.3
Verbose: Using Octopus.Dependencies.AzureCmdlets 5.7.0
Verbose: Using Octopus.Dependencies.AzureCLI 2.0.42
Verbose: Running this script as the Octopus Server (NT AUTHORITY\SYSTEM)
Verbose: Starting C:\WINDOWS\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory ‘C:\Octopus\Work\20180903123948-22063-30’ using ‘Western European (DOS)’ encoding running as ‘NT AUTHORITY\SYSTEM’ with the same environment variables as the launching process
Verbose: Octopus Deploy: Calamari version 4.8.3
Verbose: Environment Information:
Verbose: OperatingSystem: Microsoft Windows NT 10.0.17134.0
Verbose: OsBitVersion: x64
Verbose: Is64BitProcess: True
Verbose: CurrentUser: NT AUTHORITY\SYSTEM
Verbose: MachineName: SRV-BUILD-01
Verbose: ProcessorCount: 4
Verbose: CurrentDirectory: C:\Octopus\Work\20180903123948-22063-30
Verbose: TempDirectory: C:\WINDOWS\TEMP\
Verbose: HostProcessName: Calamari
Verbose: Performing variable substitution on ‘C:\Octopus\Work\20180903123948-22063-30\Script.ps1’
Verbose: Executing ‘C:\Octopus\Work\20180903123948-22063-30\Script.ps1’
Verbose: Name Value
Verbose: ---- -----
Verbose: PSVersion 5.1.17134.228
Verbose: PSEdition Desktop
Verbose: PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
Verbose: BuildVersion 10.0.17134.228
Verbose: CLRVersion 4.0.30319.42000
Verbose: WSManStackVersion 3.0
Verbose: PSRemotingProtocolVersion 2.3
Verbose: SerializationVersion 1.1.0.1
Verbose: PowerShell Environment Information:
Verbose: OperatingSystem: Microsoft Windows NT 10.0.17134.0
Verbose: OsBitVersion: x64
Verbose: Is64BitProcess: True
Verbose: CurrentUser: NT AUTHORITY\SYSTEM
Verbose: MachineName: SRV-BUILD-01
Verbose: ProcessorCount: 4
Verbose: CurrentDirectory: C:\Octopus\Work\20180903123948-22063-30
Verbose: CurrentLocation: C:\Octopus\Work\20180903123948-22063-30
Verbose: TempDirectory: C:\WINDOWS\TEMP\
Verbose: HostProcessName: powershell
Verbose: TotalPhysicalMemory: 16776756 KB
Verbose: AvailablePhysicalMemory: 11099356 KB
Verbose: Authenticating with Service Principal
Verbose: Account : XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Verbose: SubscriptionName : myproduct
Verbose: SubscriptionId : YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY
Verbose: TenantId : ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ
Verbose: Environment : AzureCloud
Verbose: Azure CLI: Authenticating with Service Principal
Verbose: C:\Octopus\Work\20180903123948-22063-30> “C:\Octopus\OctopusServer\Tools\Octopus.Dependencies.AzureCLI\2.0.42\AzureCLI\wbin\…\python.exe” -IBm azure.cli login --service-principal -u XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX -p ******** --tenant ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ
Error: %(D was unexpected at this time.
Verbose: Azure CLI: Setting active subscription to YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY
Error: ERROR: The subscription of ‘YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY’ doesn’t exist in cloud ‘AzureCloud’.
Verbose: Successfully authenticated with the Azure CLI
Verbose: Invoking target script “C:\Octopus\Work\20180903123948-22063-30\Script.ps1” with parameters
Verbose: Process C:\WINDOWS\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20180903123948-22063-30 exited with code 1
Verbose: Updating manifest with output variables
Verbose: Updating manifest with action evaluated variables
Fatal: The remote script failed with exit code 1
Fatal: The action Do nothing on myproduct Prod WebApp Test slot failed

Hi Christian, thanks for getting in touch.

We currently seem to be having some issues with the new Azure CLI functionality that was introduced as part of 2018.7.11. You should be able to work around this issue in the interim by adding a new variable to your project called OctopusDisableAzureCLI with a value of True.

We aren’t entirely sure why this is failing for you in this particular case. We would appreciate it if you could provide us with some additional information, if possible. If you add a new key to your app registration and use that instead, do you get the same error? The current thought is that we may not be handling the parameters to the Azure CLI appropriately so there may be something we are missing in the login command which we aren’t able to see. You may be better able to answer that question for us :slight_smile:

“C:\Octopus\OctopusServer\Tools\Octopus.Dependencies.AzureCLI\2.0.42\AzureCLI\wbin\…\python.exe” -IBm azure.cli login --service-principal -u XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX -p ******** --tenant ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ

Is there anything in that specific command which could explain why we are seeing this error message such as special characters outside of normal alpha numeric characters? Usually, you would get a base64 encoded value from Azure, so is possibly a long shot.

Are you using a different cloud environment other than AzureCloud or possibly behind a proxy? We have a few outstanding issues with the Azure CLI in those situations, so would just like to eliminate those as possibilities as well.

Regards,
Shaun

Hi Shaun,

Thank you for the quick response.

You should be able to work around this issue in the interim by adding a new variable to your project called OctopusDisableAzureCLI with a value of True.

I can confirm that setting OctopusDisableAzureCLI to true fixes the problem

I have tried to determine if my azure subscription had problems, and actually I guess this is the real problem. I had problems logging in with the subscription with Azure CLI (az login), so I tried to reset my credentials using :

az ad sp credential reset --name APP_ID --password NEW_PASSWORD

APP_ID is the subscription id (I guess 99% of azure user are getting bald because of azure terms :slight_smile:)
After resetting the credentials, I could login with the subscription Azure CLI and the deploy process (wtih CLI enabled) stopped failing.

But why can a deployment work if the credential is corrupt in one or the other way?. The steps actually worked, they just ended up with a failed state.

Cheers,
Christian

Hi Christian, I am glad that resolved things for you.

I am still a bit concerned by the previous error as we tried to isolate the Azure CLI as much as possible, yet the same credentials failed for the Azure CLI but succeeded for the Azure PowerShell commands, which seems to defy logic. There is the possibility that there may still be a bug in either the Azure CLI or in how we are using it, so we will keep investigating to make sure we don’t have a problem.

In regards to your question as to how the step could have worked but still result in a failed state; that is possible due to the Azure CLI login failure returning a non zero error code which we didn’t account for. The credentials were still valid in other scenarios which would have resulted in the deployment running through to completion, but having a failure state. If you did however try to use the Azure CLI as part of a pre-deployment or deployment script the deployment would have failed in its entirety.

Hopefully the above clarifies things, but happy to elaborate if it doesn’t.

Regards,
Shaun