Octopus Worker - Azure Web App target - Unhealthy unable to connect to remote server

Hi,

We have the following scenario.

An On-Premise Octopus Server instance which we are looking to deploy to Azure Pass Services via an on premise Octopus worker which connects to Azure over VPN into Azure vWan/vNet within which the PaaS service resides.

Have the Azure Web App Service targets set-up but currently coming back as Unhealthy.

Health check task log and online posts suggest it is coming from the worker and it is connectivity issue. Pretty much confirmed by following entries in task log.

“Unable to connect to the remote server”
“No connection could be made because the target machine actively refused it xxx.xxx.xxx.xxx:15871”

Have been assuming that comms were over 1872 for Azure Web App PaaS as currently understand it users MS Deploy.

Is this correct?

Is there any other info of what other ports may be involved as far as Octopus is concerned?

Regards

John

Hi,

A bit more information on what we are endeavouring to do - to be honest don’t know if it is possible.

So we have two workers set-up on premise with the intention of sending production deployments through one and non-prod through the other - we have set-up a VPN between ourselves and Azure.

Here we are assuming (perhaps wrongly) that Octopus would be communicating with azure web app with over port 1872 and SQL database using port 1433 - hence the VPN. We are trying to avoid VMs in the cloud.

Now we have the Azure Service Principal accounts set-up on our on premise instance and have configured a couple of Azure Web App targets. The health check on both targets is currently giving the following stack trace.

Is what we are trying to achieve even possible? Or do we need to have a bit of a re-think.

Service returned error. Check InnerException for more details
Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException
at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpHelper.d__01.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.<SendHttpMessageAsync>d__15.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.<SendTokenRequestAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.<RunAsync>d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.<AcquireTokenForClientCommonAsync>d__69.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.<AcquireTokenAsync>d__1a.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Calamari.Azure.Integration.Security.ServicePrincipal.GetAuthorizationToken(String tenantId, String applicationId, String password, String managementEndPoint, String activeDirectoryEndPoint) at Calamari.Azure.Accounts.AzureServicePrincipalAccountExtensions.GetAuthorizationToken(AzureServicePrincipalAccount account) at Calamari.Azure.Accounts.AzureServicePrincipalAccountExtensions.CreateWebSiteManagementClient(AzureServicePrincipalAccount account) at Calamari.Azure.HealthChecks.WebAppHealthChecker.ConfirmWebAppExists(AzureServicePrincipalAccount servicePrincipalAccount, String resourceGroupName, String siteAndSlotName) at Calamari.Azure.HealthChecks.WebAppHealthChecker.ExecuteHealthCheck(CalamariVariableDictionary variables) at Calamari.Commands.HealthCheckCommand.Execute(String[] commandLineArguments) at Calamari.Program.Execute(String[] args) --Inner Exception-- Unable to connect to the remote server System.Net.WebException at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpWebRequestWrapper.<GetResponseSyncOrAsync>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpHelper.<SendPostRequestAndDeserializeJsonResponseAsync>d__01.MoveNext()
–Inner Exception–
No connection could be made because the target machine actively refused it 10.50.99.22:15871
System.Net.Sockets.SocketException
at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
The remote script failed with exit code 100

Well another one bites the dust. Health check now working as expected - firewall/proxy issue - if anyone else comes across this I believe we have worked out that Azure Web App step works over https:\:443 in the scenario defined. Looking to still use VPN for SQL database and SSDTs … fingers crossed.

Hey John,

Thanks for getting back to me! We were discussing this late yesterday afternoon and I’m so glad that you got this resolved. Fingers crossed for your SQL connection.

Let me know if you have any more questions.

Regards,

Dane

Hi,

Just adding some more info in case anyone else comes this way.

So just to recap … we had an on-prem worker an ExpressRoute VPN connection to Azure and wanted to deploy to PaaS App Services as well as PaaS Azure SQL using SSDTs (sqlpackage.exe).

Now we found that the Web App Service step - would just go over https://…:443 - it was our proxy giving us the unhealthy targets so the VPN was useless in this respect.

In terms of DB deployment, for our Azure SQL resource the FQDN (ourresource.database.windows.net), needed a DNS entry mapping it to the internal IP (10.x.x.x) of the Azure resource over the VPN, port 1433 needed to be allowed.

For the Octopus DB project we then had two standard powershell scripts.

  1. The first authenticated against Azure AD to get a token for an Azure service principal with sufficient rights to upgrade the DB
  2. The second added the token as the AccessToken parameter for sqlpackage.exe.

Below are the key excerpts for both scripts

Script 1 for Octopus step "Get token"

$TenantID = $OctopusParameters["AzureAccount.TenantId"]
$clientId = $OctopusParameters["AzureAccount.Client"]
$clientSecret = $OctopusParameters["AzureAccount.Password"]
$resourceAppIdURI = 'https://database.windows.net/'

$tokenResponse = Invoke-RestMethod -Method Post -UseBasicParsing `
    -Uri "https://login.windows.net/$($TenantID)/oauth2/token" `
    -Body @{
        resource=$resourceAppIdURI
        client_id=$clientId
        grant_type='client_credentials'
        client_secret=$clientSecret
    } -ContentType 'application/x-www-form-urlencoded'

if ($tokenResponse) {
    Write-debug "Access token type is $($tokenResponse.token_type), expires $($tokenResponse.expires_on)"
	
    Set-OctopusVariable -name AccessToken -value $tokenResponse.access_token -sensitive
}

Script 2 apply token to sqlpackage.exe

SqlPackage.exe /a:Publish /sf:$workingDirectory$sourceDacpac /TargetDatabaseName:$DestinationDatabase /TargetServerName:$DestinationServer /AccessToken:"$($OctopusParameters[“Octopus.Action[Get token].Output.AccessToken”])"

Note

  • The $DestinationServer was the FQDN for our Azure SQL resource

  • Key to it working was ensuring the token was recognised as a string, the output parameter without the surrounding “$(…)” would result in:…

    Login failed for user ‘<token-identified principal>’.

Finally we were using Octopus Server 2019.13

John,

That is incredibly helpful - I’m sure there will be people very happy to stumble upon this thread!

Thanks for all the information!

Regards,

Dane.