Azure WebApp Deployment Failure

vikramchirum · 10 June 2019 21:44

Hi All

We are having issues deploying from octopus using azure web app. The current version of octopus is 2019.3.1

Around Feb/March, we switched from using management certificates to service principal to deploy to azure web apps. Things were working great until last Friday. We get the following error. I verified the azure account Test and Save works fine and was able to pull the webapps asscoiated with the service account in the process step of deploy web app, which means that octopus was able to talk to azure and get the apps. I also added the deployment target to do a health check i get the same error.

The underlying error is
Unable to connect to the remote server
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 40.126.7.98:443

What am i missing here.

The logs from deployment step are

Deploying to Azure WebApp ‘XXXXXXXXXXX’ in Resource Group ‘xxxxxxxx’, using subscription-id ‘xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx’
June 10th 2019 14:52:03Error
Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException: Service returned error. Check InnerException for more details —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 40.126.7.98:443
June 10th 2019 14:52:03Error
at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
June 10th 2019 14:52:03Error
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
June 10th 2019 14:52:03Error
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
June 10th 2019 14:52:03Error
— End of inner exception stack trace —
June 10th 2019 14:52:03Error
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
June 10th 2019 14:52:03Error
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult)
June 10th 2019 14:52:03Error
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
June 10th 2019 14:52:03Error
— End of stack trace from previous location where exception was thrown —
June 10th 2019 14:52:03Error
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
June 10th 2019 14:52:03Error
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
June 10th 2019 14:52:03Error
at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpWebRequestWrapper.d__8.MoveNext()
June 10th 2019 14:52:03Error
— End of stack trace from previous location where exception was thrown —
June 10th 2019 14:52:03Error
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
June 10th 2019 14:52:03Error
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
June 10th 2019 14:52:03Error
at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpWebRequestWrapper.d__2.MoveNext()
June 10th 2019 14:52:03Error
— End of stack trace from previous location where exception was thrown —
June 10th 2019 14:52:03Error
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
June 10th 2019 14:52:03Error
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
June 10th 2019 14:52:03Error
at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpHelper.d__01.MoveNext() June 10th 2019 14:52:03Error --- End of inner exception stack trace --- June 10th 2019 14:52:03Error at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpHelper.<SendPostRequestAndDeserializeJsonResponseAsync>d__01.MoveNext()
June 10th 2019 14:52:03Error
— End of stack trace from previous location where exception was thrown —
June 10th 2019 14:52:03Error
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
June 10th 2019 14:52:03Error
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
June 10th 2019 14:52:03Error
at Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.d__15.MoveNext()
June 10th 2019 14:52:03Error
— End of stack trace from previous location where exception was thrown —
June 10th 2019 14:52:03Error
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
June 10th 2019 14:52:03Error
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
June 10th 2019 14:52:03Error
at Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.d__9.MoveNext()
June 10th 2019 14:52:03Error
— End of stack trace from previous location where exception was thrown —
June 10th 2019 14:52:03Error
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
June 10th 2019 14:52:03Error
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
June 10th 2019 14:52:03Error
at Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.d__0.MoveNext()
June 10th 2019 14:52:03Error
— End of stack trace from previous location where exception was thrown —
June 10th 2019 14:52:03Error
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
June 10th 2019 14:52:03Error
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
June 10th 2019 14:52:03Error
at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.d__69.MoveNext()
June 10th 2019 14:52:03Error
— End of stack trace from previous location where exception was thrown —
June 10th 2019 14:52:03Error
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
June 10th 2019 14:52:03Error
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
June 10th 2019 14:52:03Error
at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.d__1a.MoveNext()
June 10th 2019 14:52:03Error
— End of stack trace from previous location where exception was thrown —
June 10th 2019 14:52:03Error
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
June 10th 2019 14:52:03Error
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
June 10th 2019 14:52:03Error
at Calamari.Azure.Integration.Security.ServicePrincipal.GetAuthorizationToken(String tenantId, String applicationId, String password, String managementEndPoint, String activeDirectoryEndPoint)
June 10th 2019 14:52:03Error
at Calamari.Azure.Integration.Websites.Publishing.ResourceManagerPublishProfileProvider.GetPublishProperties(AzureServicePrincipalAccount account, String resourceGroupName, AzureTargetSite azureTargetSite)
June 10th 2019 14:52:03Error
at Calamari.Azure.Deployment.Conventions.AzureWebAppConvention.GetPublishProfile(VariableDictionary variables)
June 10th 2019 14:52:03Error
at Calamari.Azure.Deployment.Conventions.AzureWebAppConvention.Install(RunningDeployment deployment)
June 10th 2019 14:52:03Error
at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
June 10th 2019 14:52:03Error
at Calamari.Deployment.ConventionProcessor.RunConventions()
June 10th 2019 14:52:03Error
ErrorCode: service_returned_error
June 10th 2019 14:52:03Error
StatusCode: 501
June 10th 2019 14:52:03Error
Running rollback conventions…
June 10th 2019 14:52:03Error
Service returned error. Check InnerException for more details
June 10th 2019 14:52:03Error
Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException
June 10th 2019 14:52:03Error
at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpHelper.d__01.MoveNext() June 10th 2019 14:52:03Error --- End of stack trace from previous location where exception was thrown --- June 10th 2019 14:52:03Error at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() June 10th 2019 14:52:03Error at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) June 10th 2019 14:52:03Error at Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.<SendHttpMessageAsync>d__15.MoveNext() June 10th 2019 14:52:03Error --- End of stack trace from previous location where exception was thrown --- June 10th 2019 14:52:03Error at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() June 10th 2019 14:52:03Error at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) June 10th 2019 14:52:03Error at Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.<SendTokenRequestAsync>d__9.MoveNext() June 10th 2019 14:52:03Error --- End of stack trace from previous location where exception was thrown --- June 10th 2019 14:52:03Error at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() June 10th 2019 14:52:03Error at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) June 10th 2019 14:52:03Error at Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.<RunAsync>d__0.MoveNext() June 10th 2019 14:52:03Error --- End of stack trace from previous location where exception was thrown --- June 10th 2019 14:52:03Error at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() June 10th 2019 14:52:03Error at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) June 10th 2019 14:52:03Error at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.<AcquireTokenForClientCommonAsync>d__69.MoveNext() June 10th 2019 14:52:03Error --- End of stack trace from previous location where exception was thrown --- June 10th 2019 14:52:03Error at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() June 10th 2019 14:52:03Error at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) June 10th 2019 14:52:03Error at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.<AcquireTokenAsync>d__1a.MoveNext() June 10th 2019 14:52:03Error --- End of stack trace from previous location where exception was thrown --- June 10th 2019 14:52:03Error at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() June 10th 2019 14:52:03Error at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) June 10th 2019 14:52:03Error at Calamari.Azure.Integration.Security.ServicePrincipal.GetAuthorizationToken(String tenantId, String applicationId, String password, String managementEndPoint, String activeDirectoryEndPoint) June 10th 2019 14:52:03Error at Calamari.Azure.Integration.Websites.Publishing.ResourceManagerPublishProfileProvider.GetPublishProperties(AzureServicePrincipalAccount account, String resourceGroupName, AzureTargetSite azureTargetSite) June 10th 2019 14:52:03Error at Calamari.Azure.Deployment.Conventions.AzureWebAppConvention.GetPublishProfile(VariableDictionary variables) June 10th 2019 14:52:03Error at Calamari.Azure.Deployment.Conventions.AzureWebAppConvention.Install(RunningDeployment deployment) June 10th 2019 14:52:03Error at Calamari.Deployment.ConventionProcessor.RunInstallConventions() June 10th 2019 14:52:03Error at Calamari.Deployment.ConventionProcessor.RunConventions() June 10th 2019 14:52:03Error at Calamari.Azure.Commands.DeployAzureWebCommand.Execute(String[] commandLineArguments) June 10th 2019 14:52:03Error at Calamari.Program.Execute(String[] args) June 10th 2019 14:52:03Error --Inner Exception-- June 10th 2019 14:52:03Error Unable to connect to the remote server June 10th 2019 14:52:03Error System.Net.WebException June 10th 2019 14:52:03Error at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context) June 10th 2019 14:52:03Error at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult) June 10th 2019 14:52:03Error at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) June 10th 2019 14:52:03Error --- End of stack trace from previous location where exception was thrown --- June 10th 2019 14:52:03Error at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() June 10th 2019 14:52:03Error at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) June 10th 2019 14:52:03Error at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpWebRequestWrapper.<GetRequestStreamSyncOrAsync>d__8.MoveNext() June 10th 2019 14:52:03Error --- End of stack trace from previous location where exception was thrown --- June 10th 2019 14:52:03Error at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() June 10th 2019 14:52:03Error at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) June 10th 2019 14:52:03Error at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpWebRequestWrapper.<GetResponseSyncOrAsync>d__2.MoveNext() June 10th 2019 14:52:03Error --- End of stack trace from previous location where exception was thrown --- June 10th 2019 14:52:03Error at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() June 10th 2019 14:52:03Error at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) June 10th 2019 14:52:03Error at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpHelper.<SendPostRequestAndDeserializeJsonResponseAsync>d__01.MoveNext()
June 10th 2019 14:52:03Error
–Inner Exception–
June 10th 2019 14:52:03Error
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 40.126.7.98:443
June 10th 2019 14:52:03Error
System.Net.Sockets.SocketException
June 10th 2019 14:52:03Error
at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
June 10th 2019 14:52:03Error
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
June 10th 2019 14:52:03Error
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
June 10th 2019 14:52:03Fatal
The remote script failed with exit code 100
June 10th 2019 14:52:03Fatal
The action Deploy Azure API STG TEST on a Worker failed

benpearce · 11 June 2019 04:29

Hi,

Thanks for getting in touch.

Looking at the logs, it appears that the deployment is running from a worker. When you run the Azure Account test and the Azure Web App list lookup they are bot actually running directly on the Octopus server, but the deployment task and the health check are running from the worker.
It could be that the worker is not configured correctly for a corporate proxy required to access the internet.
To setup a Web Request proxy on your Worker, you will need to follow the documentation or you can also configure it via the command line on the worker.

Let me know if this solves the problem.

Regards
Ben

vikramchirum · 11 June 2019 23:47

Hi @benpearce

You are spot on about the issue. We had only one worker process instance which was getting overloaded with a ton of deployments and so our server team decided to spin up a new test instance of worker process to share the load and they did not configure the required stuff in the new test instance

They reverted back to the original worker process instance and they are gonna find the necessary things to set up the new worker instance. Our deployments are working as they were before of course with old instance.

Thanks for your help