Azure deployment fails with TLS 1.0/1.1 disabled

andrew.rackcheev · 2 October 2020 06:49

Hello,

We have configured Azure deployment but recently we switched of a client protocol TLS 1.0 and TLS 1.1 and Azure deployment stopped to work. We tried different strategies including the update to the latest Octopus Deploy, but it still fails if client side TLS 1.0/1.1 is turned off on a tentacle. Is it a known issue? Any workaround?

Regards,
Andrew

paul.calvert · 2 October 2020 08:42

Hi Andrew,

Thanks for getting in touch!

Am I correct in thinking that the Azure target is a virtual machine with the tentacle installed directly on it?
When you disabled TLS1.0/1.1 was this done on both the Octopus Server and the Tentacle, or just the Tentacle? If it was only on the tentacle, could you try disabling on both sides and re-testing?

Also, would you be able to provide the deployment log or error produced in the deployment log where this fails?

Regards,
Paul

andrew.rackcheev · 2 October 2020 09:02

Hi Paul,

Sorry for original vague description. Here is how it’s done

We have Octopus Server and a tentacle running on the same machine
We are using a Service Principal account to connect to Azure
We are deploying AppServices and functions to Azure

When we hit deploy button, Octopus tries to connect to Azure, but fails immediately, so no any deployment step is executed. Here is the error message

08:52:39 Verbose | 08:52:39 Verbose | 08:52:40 Verbose | 08:52:43 Verbose | 08:52:43 Verbose | 08:52:43 Verbose | 08:52:43 Verbose | 08:52:43 Verbose | 08:52:43 Verbose | 08:52:43 Verbose | 08:52:43 Verbose | 08:52:43 Verbose | 08:52:43 Verbose | 08:52:44 Verbose | 08:52:51 Verbose | 08:53:08 Verbose | 08:53:10 Error | 08:53:10 Error | details
08:53:10 Error | 08:53:10 Error | 08:53:10 Error | 08:53:10 Error | 08:53:10 Error | 08:53:10 Error | 08:53:10 Error | 08:53:10 Error | mand
08:53:10 Fatal | 08:53:10 Verbose | Octopus Server version: 3.3.9+Branch.master.Sha.35de9da28e9759d98a404c505b882114f10e2e44
Using account ID ‘azureserviceprincipal-azureserviceprincipal’
Octopus Deploy: Calamari.Azure version 3.3.9+Branch.master.Sha.b0e61033e755006a1d6083c75e649335f3676752
Name Value
---- -----
PSVersion 5.1.14409.1018
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
BuildVersion 10.0.14409.1018
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
Importing Azure Service Management PowerShell module
Importing Azure Resource Manager PowerShell modules
Authenticating with Service Principal
Login-AzureRmAccount : Service returned error. Check InnerException for more
At C:\OctopusData\Work\20200929075239-1\Octopus.AzureContext.ps1:70 char:2
+ Login-AzureRmAccount -Credential $creds -TenantId $OctopusAzureAD …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: ( [Add-AzureRmAccount], AdalServ
iceException
+ FullyQualifiedErrorId : Microsoft.IdentityModel.Clients.ActiveDirectory.
AdalServiceException,Microsoft.Azure.Commands.Profile.AddAzureRMAccountCom
The step failed: The remote script failed with exit code 1
Stop web app completed

Regards,
Andrew

paul.calvert · 2 October 2020 09:17

Thanks for the additional information.

A few more questions to follow up with though
What version of Octopus were you running initially, and what version are you now running?
Does the machine have .NET 4.5 or higher installed?

With TLS 1.0/1.2 disabled, would you be able to test the Login-AzureRmAccount command directly in Powershell and see if you encounter the same error?

Lastly, the Azure step doesn’t seem to be catching the specific error, to try and catch the correct error would you be able to use a standard Run a Script step with a script body of:

try {
  $securePassword = ConvertTo-SecureString $azurePassword -AsPlainText -Force
  $creds = New-Object System.Management.Automation.PSCredential ($azureClient, $securePassword)
  Disable-AzureRMContextAutosave -Scope Process
  Login-AzureRmAccount -Credential $creds -TenantId $azureTenantId -SubscriptionId $azureSubscriptionNumber -ServicePrincipal
  Get-AzureRmResourceGroup
} catch [System.Exception] {
  Write-Host $_.Exception|format-list -force
}

You’ll also need to add a project variable named azure with a value set to your Azure account.
You can run this test in a fresh project if you like.

andrew.rackcheev · 2 October 2020 09:20

After being hit by the issue we tried even the latest Octopus 2020.4 and easily reproduced the problem there on a fresh Windows 2016 DataCenter VM with the latest .NET installed . I will check the script you suggested later today and let you know

Regards,
Andrew

andrew.rackcheev · 5 October 2020 09:06

Hi Paul,

Just tried to execture this script in as simple “Run a Script” step and it errors out

But if gives this error

09:02:53 Info | System.Management.Automation.ValidationMetadataException: The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
09:02:53 Info | at System.Management.Automation.ValidateNotNullOrEmptyAttribute.Validate(Object arguments, EngineIntrinsics engineIntrinsics)
09:02:53 Info | at System.Management.Automation.ParameterBinderBase.BindParameter(CommandParameterInternal parameter, CompiledCommandParameter parameterMetadata, ParameterBindingFlags flags)
09:02:53 Verbose | Process C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20201005090246-42-1 exited with code 0

What am I doing wrong?

If you’d like I can share credentials to a test server thru email where you can see the problem.

Regards,
Andrew

paul.calvert · 5 October 2020 09:17

Other than checking that you had a project variable named azure linked to the correct account, I’m not sure what else could fail that script in that way.
e.g.

Have you been able to test this outside of Octopus by running the AzureRM login commands from Powershell directly?

andrew.rackcheev · 5 October 2020 09:36

Hi Paul,

Yes, my bad. Too many test Octopus instances

Here is the output now

09:31:21 Info | Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException: Service returned error. Check InnerException for more details —> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. —> System.ComponentModel.Win32Exception: The client and server cannot communicate, because they do not possess a common algorithm
09:31:21 Info | at System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface SecModule, String package, CredentialUse intent, SecureCredential scc)
09:31:21 Info | at System.Net.Security.SecureChannel.AcquireCredentialsHandle(CredentialUse credUsage, SecureCredential& secureCredential)
09:31:21 Info | at System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint)
09:31:21 Info | at System.Net.Security.SecureChannel.GenerateToken(Byte[] input, Int32 offset, Int32 count, Byte[]& output)
09:31:21 Info | at System.Net.Security.SecureChannel.NextMessage(Byte[] incoming, Int32 offset, Int32 count)
09:31:21 Info | at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
09:31:21 Info | at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
09:31:21 Info | at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
09:31:21 Info | at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
09:31:21 Info | at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
09:31:21 Info | at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
09:31:21 Info | at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
09:31:21 Info | at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
09:31:21 Info | at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
09:31:21 Info | at System.Net.ConnectStream.WriteHeaders(Boolean async)
09:31:21 Info | — End of inner exception stack trace —
09:31:21 Info | at System.Net.HttpWebRequest.GetResponse()
09:31:21 Info | at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpWebRequestWrapper.d__2.MoveNext()
09:31:21 Info | — End of stack trace from previous location where exception was thrown —
09:31:21 Info | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
09:31:21 Info | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
09:31:21 Info | at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpHelper.d__01.MoveNext() 09:31:21 Info | --- End of inner exception stack trace --- 09:31:21 Info | at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.RunAsyncTask[T](Task1 task)
09:31:21 Info | at Microsoft.Azure.Commands.Common.Authentication.ServicePrincipalTokenProvider.GetAccessToken(AdalConfiguration config, String promptBehavior, Action1 promptAction, String userId, SecureString password, String credentialType) 09:31:21 Info | at Microsoft.Azure.Commands.Common.Authentication.Factories.AuthenticationFactory.Authenticate(IAzureAccount account, IAzureEnvironment environment, String tenant, SecureString password, String promptBehavior, Action1 promptAction, IAzureTokenCache tokenCache, String resourceId)
09:31:21 Info | at Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient.AcquireAccessToken(IAzureAccount account, IAzureEnvironment environment, String tenantId, SecureString password, String promptBehavior, Action1 promptAction) 09:31:21 Info | at Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient.Login(IAzureAccount account, IAzureEnvironment environment, String tenantId, String subscriptionId, String subscriptionName, SecureString password, Boolean skipValidation, Action1 promptAction, String name, Boolean shouldPopulateContextList)
09:31:21 Info | at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.<>c__DisplayClass87_0.b__0(AzureRmProfile localProfile, RMProfileClient profileClient, String name)
09:31:21 Info | at Microsoft.Azure.Commands.Profile.Common.AzureContextModificationCmdlet.ModifyContext(Action2 contextAction) 09:31:21 Info | at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.SetContextWithOverwritePrompt(Action3 setContextAction)
09:31:21 Info | at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.ExecuteCmdlet()
09:31:21 Info | at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
09:31:21 Info | ErrorCode: service_returned_error
09:31:21 Info | StatusCode: 501
09:31:21 Verbose | Process C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20201005093114-62-1 exited with code 0
09:31:21 Verbose | Updating manifest with output variables
09:31:21 Verbose | Updating manifest with action evaluated variables
09:31:21 Verbose | Successfully finished Debug on the Octopus Server

Note: if I add

[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12

at the very begining then the reply is different

v09:34:05 Info | Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException: AADSTS700016: Application with identifier ‘****’ was not found in the directory ‘*****’. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.
09:34:05 Info | Trace ID: 1ba0cf6e-1fa1-48a5-a046-17086bebe901
09:34:05 Info | Correlation ID: 38c94e25-bf04-4f12-9bc6-e6e32b8dbc76
09:34:05 Info | Timestamp: 2020-10-05 09:34:05Z —> System.Net.WebException: The remote server returned an error: (400) Bad Request.
09:34:05 Info | at System.Net.HttpWebRequest.GetResponse()
09:34:05 Info | at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpWebRequestWrapper.d__2.MoveNext()
09:34:05 Info | — End of stack trace from previous location where exception was thrown —
09:34:05 Info | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
09:34:05 Info | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
09:34:05 Info | at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpHelper.d__01.MoveNext() 09:34:05 Info | --- End of inner exception stack trace --- 09:34:05 Info | at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.RunAsyncTask[T](Task1 task)
09:34:05 Info | at Microsoft.Azure.Commands.Common.Authentication.ServicePrincipalTokenProvider.GetAccessToken(AdalConfiguration config, String promptBehavior, Action1 promptAction, String userId, SecureString password, String credentialType) 09:34:05 Info | at Microsoft.Azure.Commands.Common.Authentication.Factories.AuthenticationFactory.Authenticate(IAzureAccount account, IAzureEnvironment environment, String tenant, SecureString password, String promptBehavior, Action1 promptAction, IAzureTokenCache tokenCache, String resourceId)
09:34:05 Info | at Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient.AcquireAccessToken(IAzureAccount account, IAzureEnvironment environment, String tenantId, SecureString password, String promptBehavior, Action1 promptAction) 09:34:05 Info | at Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient.Login(IAzureAccount account, IAzureEnvironment environment, String tenantId, String subscriptionId, String subscriptionName, SecureString password, Boolean skipValidation, Action1 promptAction, String name, Boolean shouldPopulateContextList)
09:34:05 Info | at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.<>c__DisplayClass87_0.b__0(AzureRmProfile localProfile, RMProfileClient profileClient, String name)
09:34:05 Info | at Microsoft.Azure.Commands.Profile.Common.AzureContextModificationCmdlet.ModifyContext(Action2 contextAction) 09:34:05 Info | at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.SetContextWithOverwritePrompt(Action3 setContextAction)
09:34:05 Info | at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.ExecuteCmdlet()
09:34:05 Info | at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
09:34:05 Info | ErrorCode: unauthorized_client
09:34:05 Info | StatusCode: 400
09:34:05 Verbose | Process C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20201005093402-63-3 exited with code 0
09:34:06 Verbose | Updating manifest with output variables
09:34:06 Verbose | Updating manifest with action evaluated variables
09:34:06 Verbose | Successfully finished Debug on the Octopus Server

Regards,
Andrew

paul.calvert · 5 October 2020 09:58

Ok, so that suggests that it isn’t using TLS1.2 unless forced to do so.
It may be worth checking what TLS settings are still enabled using IISCrypto or similar. The only way for Octopus to use something below TLS1.2 is if there is something below that still enabled.

Also, is this script step running directly on the Octopus Server or does it run on a Worker or deployment target?

andrew.rackcheev · 5 October 2020 10:03

Hello,

yes, that’s the cause as I explained in the original post. If we disable TLS 1.0/1.1 client protocols then Octopus stops deploying to Azure. It looks like a bug, since these protocols are outdated and Azure will stop supporting them soon.

The real issue is that we cannot add

[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12

To Azure deployment step, since it’s never reached. It has to be done in Octopus during initial connection to Azure. Any plans to fix this?

Regards,
Andrew

paul.calvert · 5 October 2020 10:15

Octopus is fully compatible with TLS 1.2 and has been since around version 3.3. The fact that it fails when TLS 1.0/1.1 is disabled suggests that a different machine it is connecting to is trying to use those protocols. Are there any proxy servers or other networking devices that Octopus has to traverse in order to reach Azure that may not have TLS 1.0/1.1 disabled?

Have you tried running the Azure script on this machine but outside of Octopus?

andrew.rackcheev · 5 October 2020 10:20

There is Client Side protocols and Server side protocols. If we turn off Server side protocols, then yes , everything works. If we turn off Client side protocols then Octopus stops deploying to Azure. The test instance speficially is Azure VM deploying to Azure, so I doubt there is any blocker in between. You can try it yourself using the famous IISCrypto utility. If you turn off client side TLS 1.0/1.1 you won’t be able to deploy to Azure.

paul.calvert · 5 October 2020 10:25

OK, and with only the client side protocols disabled, if you run your script (or just the Login-AzureRmAccount command) outside of Octopus, does it work?

andrew.rackcheev · 5 October 2020 10:27

if I explicitly set

[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12

then yes.

paul.calvert · 5 October 2020 10:54

If the same behaviour is occurring outside of Octopus then it doesn’t look like this is a bug with the software.

It seems that it will be necessary for the server-side protocols to also be disabled in order for this to work correctly.

andrew.rackcheev · 5 October 2020 11:26

I figured out the root cause. Powershell 5 is using TLS 1.0 by default and you need to explicitly set it to a higher TLS. I’ve managed to workaround this by adding registry keys to force Powershell to use a higher TLS. But I believe you need to apply a fix to Calamari, so it is set TLS correctly on EACH use of Powershell

Regards,
Andrew

system · 5 November 2020 11:26

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.