WebApp Deployment - Connection terminated issue

mandest · 21 July 2018 15:59

When deploying to a WebApp instance on Azure, Octopus Deploy (Cloud Version) suffers the following error for one of two WebApps - the other deploys successfully. Both WebApps are configured similarly.

The deployment succeeds from Visual Studio (local) and from CI services.

Full log:

Task ID:        ServerTasks-1165
Task status:    Failed
Task queued:    Saturday, July 21, 2018 3:14:14 PM
Task started:   Saturday, July 21, 2018 3:14:14 PM
Task completed: Saturday, July 21, 2018 3:14:53 PM
Task duration:  39 seconds
Server version: 2018.6.14+Branch.projects-nancy-module-tests.Sha.b7693755b553f2d6eb8c3e1f564bdc1065a98c8d
Server node:    EC2AMAZ-UNAAJA8

                    | == Failed: Deploy release 0.0.13 to Staging ==
15:14:14   Verbose  |   Guided failure is not enabled for this task
15:14:53   Fatal    |   The deployment failed because one or more steps failed. Please see the deployment log for details.
                    | 
                    |   == Success: Acquire packages ==
15:14:14   Info     |     Acquiring packages
15:14:14   Info     |     Making a list of packages to acquire
15:14:14   Verbose  |     No packages are required on any deployment targets
15:14:14   Verbose  |     Acquire Api.2018.7.20.60 on Octopus Server
                    |     Acquire Functions.2018.7.20.60 on Octopus Server
                    |     Acquire Web.2018.7.20.60 on Octopus Server
15:14:14   Verbose  |     Checking package cache for package Api v2018.7.20.60
15:14:14   Info     |     Package Api v2018.7.20.60 was found in cache. No need to download.
15:14:14   Verbose  |     Using file: E:\Octopus\Packages\Api\Api.2018.7.20.60.nupkg
15:14:14   Verbose  |     Checking package cache for package Functions v2018.7.20.60
15:14:14   Info     |     Package Functions v2018.7.20.60 was found in cache. No need to download.
15:14:14   Verbose  |     Using file: E:\Octopus\Packages\Functions\Functions.2018.7.20.60.nupkg
15:14:14   Verbose  |     Checking package cache for package Web v2018.7.20.60
15:14:14   Info     |     Package Web v2018.7.20.60 was found in cache. No need to download.
15:14:14   Verbose  |     Using file: E:\Octopus\Packages\Web\Web.2018.7.20.60.nupkg
15:14:14   Info     |     All packages have been acquired
15:14:14   Verbose  |     Acquire Packages completed
                    |   
                    |     Success: Octopus Server
                    |     
                    |       Success: Download package Api v2018.7.20.60 from Octopus Server (built-in)
15:14:14   Info     |         Successfully downloaded Api 2018.7.20.60 (48.596 MB)
15:14:14   Verbose  |         Stored package in E:\Octopus\Packages\Api\Api.2018.7.20.60.nupkg
                    |       
                    |       Success: Download package Functions v2018.7.20.60 from Octopus Server (built-in)
15:14:14   Info     |         Successfully downloaded Functions 2018.7.20.60 (5.65 MB)
15:14:14   Verbose  |         Stored package in E:\Octopus\Packages\Functions\Functions.2018.7.20.60.nupkg
                    |       
                    |       Success: Download package Web v2018.7.20.60 from Octopus Server (built-in)
15:14:14   Info     |         Successfully downloaded Web 2018.7.20.60 (23.732 MB)
15:14:14   Verbose  |         Stored package in E:\Octopus\Packages\Web\Web.2018.7.20.60.nupkg
                    |       
                    |   == Success: Step 1: API ==
15:14:27   Verbose  |     API completed
                    |   
                    |     Success: Octopus Server on behalf of [Staging] API
15:14:15   Verbose  |       Octopus Server version: 2018.6.14+Branch.projects-nancy-module-tests.Sha.b7693755b553f2d6eb8c3e1f564bdc1065a98c8d
15:14:15   Verbose  |       Environment Information:
                    |       OperatingSystem: Microsoft Windows NT 10.0.14393.0
                    |       OsBitVersion: x64
                    |       Is64BitProcess: True
                    |       CurrentUser: EC2AMAZ-UNAAJA8\svcOctopusServer
                    |       MachineName: EC2AMAZ-UNAAJA8
                    |       ProcessorCount: 2
                    |       CurrentDirectory: C:\Windows\system32
                    |       TempDirectory: C:\Users\svcOctopusServer\AppData\Local\Temp\
                    |       HostProcessName: Octopus.Server
                    |       PID: 3232
15:14:15   Verbose  |       Executing API (type Deploy an Azure Web App) on Octopus Server
15:14:15   Verbose  |       Using account ID 'azureserviceprincipal-azure'
15:14:15   Verbose  |       Account variables are being contributed by the target
15:14:15   Verbose  |       Using Calamari.Cloud 4.7.21
15:14:15   Verbose  |       Using Octopus.Dependencies.AzureCmdlets 5.7.0
15:14:15   Verbose  |       Running this script with a custom user account (EC2AMAZ-UNAAJA8\svcScriptRunner)
15:14:15   Verbose  |       Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory 'C:\Octopus\Work\20180721151415-1165-41' using 'OEM United States' encoding running as '\svcScriptRunner' with that user's environment variables plus 3 custom variable(s)
15:14:16   Verbose  |       Octopus Deploy: Calamari version 4.7.21
15:14:16   Verbose  |       Environment Information:
15:14:16   Verbose  |       OperatingSystem: Microsoft Windows NT 10.0.14393.0
15:14:16   Verbose  |       OsBitVersion: x64
15:14:16   Verbose  |       Is64BitProcess: True
15:14:16   Verbose  |       CurrentUser: EC2AMAZ-UNAAJA8\svcScriptRunner
15:14:16   Verbose  |       MachineName: EC2AMAZ-UNAAJA8
15:14:16   Verbose  |       ProcessorCount: 2
15:14:16   Verbose  |       CurrentDirectory: C:\Octopus\Work\20180721151415-1165-41
15:14:16   Verbose  |       TempDirectory: C:\Users\svcScriptRunner\AppData\Local\Temp\
15:14:16   Verbose  |       HostProcessName: Calamari
15:14:16   Info     |       Deploying package:    E:\Octopus\Packages\Api\Api.2018.7.20.60.nupkg
15:14:16   Verbose  |       Extracting package to: C:\Octopus\Work\20180721151415-1165-41\staging
15:14:22   Verbose  |       Extracted 427 files
15:14:22   Info     |       Deploying to Azure WebApp 'api-staging' in Resource Group, using subscription-id 'SUBSCRIPTION_ID'
15:14:22   Verbose  |       Authentication Context: https://login.windows.net/___
15:14:23   Verbose  |       Looking up siteAndSlotName api-staging in resourceGroup
15:14:24   Verbose  |       Retrieving publishing profile from https://management.azure.com/subscriptions/SUBSCRIPTION/resourceGroups/ResourceGroup/providers/Microsoft.Web/sites/api-staging/config/publishingCredentials/list?api-version=2015-08-01
15:14:25   Verbose  |       Retrieved publishing profile. URI: https://$api-staging:PW@api-staging.scm.azurewebsites.net  UserName: $api-staging
15:14:25   Verbose  |       Using siteAndSlot api-staging
15:14:25   Verbose  |       Using siteName api-staging
15:14:25   Verbose  |       Using ID 'fb95465c-b2e1-4416-8401-7f5b2c94736f' for connections to the remote server.
15:14:25   Verbose  |       Pre-authenticating to remote agent URL 'https://$api-staging:PW@api-staging.scm.azurewebsites.net/msdeploy.axd?site=api-staging' as '$api-staging'.
15:14:25   Verbose  |       Performing synchronization pass #1.
15:14:25   Verbose  |       Pre-authenticating to remote agent URL 'https://$api-staging:PW@api-staging.scm.azurewebsites.net/msdeploy.axd?site=api-staging' as '$api-staging'.
15:14:26   Verbose  |       Delete operation on dirPath (api-staging\app_data\jobs\continuous\Emails) skipped because of rule SkipDeleteDataDir.
15:14:26   Verbose  |       Delete operation on dirPath (api-staging\app_data\jobs\triggered) skipped because of rule SkipDeleteDataDir.
15:14:26   Verbose  |       The dependency check 'DependencyCheckInUse' found no issues.
15:14:26   Verbose  |       Received response from agent (HTTP status 'OK').
15:14:26   Verbose  |       The synchronization completed in 1 pass(es).
15:14:26   Info     |       Successfully deployed to Azure. 0 objects added. 0 objects updated. 0 objects deleted.
15:14:26   Verbose  |       Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20180721151415-1165-41 exited with code 0
15:14:27   Verbose  |       Updating manifest with output variables
15:14:27   Verbose  |       Updating manifest with action evaluated variables
15:14:27   Verbose  |       Successfully finished API on [Staging] API
                    |     
                    |   == Failed: Step 2: Web ==
15:14:52   Fatal    |     The step failed: Activity Web on [Staging] Web failed with error 'The remote script failed with exit code 100'.
15:14:52   Verbose  |     Web completed
                    |   
                    |     Failed: Octopus Server on behalf of [Staging] Web
15:14:27   Verbose  |       Octopus Server version: 2018.6.14+Branch.projects-nancy-module-tests.Sha.b7693755b553f2d6eb8c3e1f564bdc1065a98c8d
15:14:27   Verbose  |       Environment Information:
                    |       OperatingSystem: Microsoft Windows NT 10.0.14393.0
                    |       OsBitVersion: x64
                    |       Is64BitProcess: True
                    |       CurrentUser: EC2AMAZ-UNAAJA8\svcOctopusServer
                    |       MachineName: EC2AMAZ-UNAAJA8
                    |       ProcessorCount: 2
                    |       CurrentDirectory: C:\Windows\system32
                    |       TempDirectory: C:\Users\svcOctopusServer\AppData\Local\Temp\
                    |       HostProcessName: Octopus.Server
                    |       PID: 3232
15:14:27   Verbose  |       Executing Web (type Deploy an Azure Web App) on Octopus Server
15:14:27   Verbose  |       Using account ID 'azureserviceprincipal-azure'
15:14:27   Verbose  |       Account variables are being contributed by the target
15:14:27   Verbose  |       Using Calamari.Cloud 4.7.21
15:14:27   Verbose  |       Using Octopus.Dependencies.AzureCmdlets 5.7.0
15:14:27   Verbose  |       Running this script with a custom user account (EC2AMAZ-UNAAJA8\svcScriptRunner)
15:14:29   Verbose  |       Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory 'C:\Octopus\Work\20180721151427-1165-42' using 'OEM United States' encoding running as '\svcScriptRunner' with that user's environment variables plus 3 custom variable(s)
15:14:30   Verbose  |       Octopus Deploy: Calamari version 4.7.21
15:14:30   Verbose  |       Environment Information:
15:14:30   Verbose  |       OperatingSystem: Microsoft Windows NT 10.0.14393.0
15:14:30   Verbose  |       OsBitVersion: x64
15:14:30   Verbose  |       Is64BitProcess: True
15:14:30   Verbose  |       CurrentUser: EC2AMAZ-UNAAJA8\svcScriptRunner
15:14:30   Verbose  |       MachineName: EC2AMAZ-UNAAJA8
15:14:30   Verbose  |       ProcessorCount: 2
15:14:30   Verbose  |       CurrentDirectory: C:\Octopus\Work\20180721151427-1165-42
15:14:30   Verbose  |       TempDirectory: C:\Users\svcScriptRunner\AppData\Local\Temp\
15:14:30   Verbose  |       HostProcessName: Calamari
15:14:30   Info     |       Deploying package:    E:\Octopus\Packages\Web\Web.2018.7.20.60.nupkg
15:14:30   Verbose  |       Extracting package to: C:\Octopus\Work\20180721151427-1165-42\staging
15:14:32   Verbose  |       Extracted 135 files
15:14:32   Info     |       Deploying to Azure WebApp 'staging' in Resource Group, using subscription-id 'SUBSCRIPTION'
15:14:32   Verbose  |       Authentication Context: https://login.windows.net/AUTH_CONTEXT
15:14:32   Verbose  |       Looking up siteAndSlotName staging in resourceGroup
15:14:33   Verbose  |       Retrieving publishing profile from https://management.azure.com/subscriptions/SUBSCRIPTION/resourceGroups/ResourceGroup/providers/Microsoft.Web/sites/staging/config/publishingCredentials/list?api-version=2015-08-01
15:14:34   Verbose  |       Retrieved publishing profile. URI: https://$staging:PW@staging.scm.azurewebsites.net  UserName: $staging
15:14:34   Verbose  |       Using siteAndSlot staging
15:14:34   Verbose  |       Using siteName staging
15:14:34   Verbose  |       Using ID 'c4639a4a-4aca-45e6-b026-da796b97f464' for connections to the remote server.
15:14:34   Verbose  |       Pre-authenticating to remote agent URL 'https://$staging:PW@staging.scm.azurewebsites.net/msdeploy.axd?site=staging' as '$staging'.
15:14:34   Verbose  |       Performing synchronization pass #1.
15:14:34   Verbose  |       Pre-authenticating to remote agent URL 'https://$staging:PW@staging.scm.azurewebsites.net/msdeploy.axd?site=staging' as '$staging'.
15:14:35   Verbose  |       Received response from agent (HTTP status 'OK').
15:14:35   Verbose  |       Retry #1 on Azure deploy. Exception: Web Deploy experienced a connection problem with the server and had to terminate the connection.  Contact your server administrator if the problem persists.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_CONNECTION_TERMINATED.
15:14:37   Verbose  |       Using siteAndSlot staging
15:14:37   Verbose  |       Using siteName staging
15:14:37   Verbose  |       Using ID '2f381a3f-debd-48ad-b493-a4b2ce458167' for connections to the remote server.
15:14:37   Verbose  |       Pre-authenticating to remote agent URL 'https://$staging:PW@staging.scm.azurewebsites.net/msdeploy.axd?site=staging' as '$staging'.
15:14:38   Verbose  |       Performing synchronization pass #1.
15:14:38   Verbose  |       Pre-authenticating to remote agent URL 'https://$staging:PW@staging.scm.azurewebsites.net/msdeploy.axd?site=staging' as '$staging'.
15:14:38   Verbose  |       Received response from agent (HTTP status 'OK').
15:14:38   Verbose  |       Retry #2 on Azure deploy. Exception: Web Deploy experienced a connection problem with the server and had to terminate the connection.  Contact your server administrator if the problem persists.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_CONNECTION_TERMINATED.
15:14:42   Verbose  |       Using siteAndSlot staging
15:14:42   Verbose  |       Using siteName staging
15:14:42   Verbose  |       Using ID '40662e4e-7d7b-4d5b-b499-4ba583f998c5' for connections to the remote server.
15:14:42   Verbose  |       Pre-authenticating to remote agent URL 'https://$staging:PW@staging.scm.azurewebsites.net/msdeploy.axd?site=staging' as '$staging'.
15:14:43   Verbose  |       Performing synchronization pass #1.
15:14:43   Verbose  |       Pre-authenticating to remote agent URL 'https://$staging:PW@staging.scm.azurewebsites.net/msdeploy.axd?site=staging' as '$staging'.
15:14:43   Verbose  |       Received response from agent (HTTP status 'OK').
15:14:43   Verbose  |       Retry #3 on Azure deploy. Exception: Web Deploy experienced a connection problem with the server and had to terminate the connection.  Contact your server administrator if the problem persists.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_CONNECTION_TERMINATED.
15:14:51   Verbose  |       Using siteAndSlot staging
15:14:51   Verbose  |       Using siteName staging
15:14:51   Verbose  |       Using ID '94beda27-11cf-4dc6-b2a4-dfb241d4614c' for connections to the remote server.
15:14:51   Verbose  |       Pre-authenticating to remote agent URL 'https://$staging:PW@staging.scm.azurewebsites.net/msdeploy.axd?site=staging' as '$staging'.
15:14:52   Verbose  |       Performing synchronization pass #1.
15:14:52   Verbose  |       Pre-authenticating to remote agent URL 'https://$staging:PW@staging.scm.azurewebsites.net/msdeploy.axd?site=staging' as '$staging'.
15:14:52   Verbose  |       Received response from agent (HTTP status 'OK').
15:14:52   Error    |       Microsoft.Web.Deployment.DeploymentDetailedException: Web Deploy experienced a connection problem with the server and had to terminate the connection.  Contact your server administrator if the problem persists.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_CONNECTION_TERMINATED. ---> System.Xml.XmlException: There is an unclosed literal string. Line 1, position 95.
15:14:52   Error    |       at System.Xml.XmlTextReaderImpl.Throw(Exception e)
15:14:52   Error    |       at System.Xml.XmlTextReaderImpl.ParseAttributeValueSlow(Int32 curPos, Char quoteChar, NodeData attr)
15:14:52   Error    |       at System.Xml.XmlTextReaderImpl.ParseAttributes()
15:14:52   Error    |       at System.Xml.XmlTextReaderImpl.ParseElement()
15:14:52   Error    |       at System.Xml.XmlTextReaderImpl.ParseElementContent()
15:14:52   Error    |       at Microsoft.Web.Deployment.TraceEventSerializer.Deserialize(Stream responseStream, DeploymentBaseContext baseContext, DeploymentSyncContext syncContext)
15:14:52   Error    |       --- End of inner exception stack trace ---
15:14:52   Error    |       at Microsoft.Web.Deployment.StatusThreadHandler.CheckForException()
15:14:52   Error    |       at Microsoft.Web.Deployment.AgentClientProvider.RemoteDestSync(DeploymentObject sourceObject, DeploymentSyncContext syncContext, Nullable`1 syncPass, String syncSessionId)
15:14:52   Error    |       at Microsoft.Web.Deployment.DeploymentObject.SyncToInternal(DeploymentObject destObject, DeploymentSyncOptions syncOptions, PayloadTable payloadTable, ContentRootTable contentRootTable, Nullable`1 syncPassId, String syncSessionId)
15:14:52   Error    |       at Microsoft.Web.Deployment.DeploymentObject.SyncTo(DeploymentProviderOptions providerOptions, DeploymentBaseOptions baseOptions, DeploymentSyncOptions syncOptions)
15:14:52   Error    |       at Calamari.Azure.Deployment.Conventions.AzureWebAppConvention.DeployToAzure(RunningDeployment deployment, String siteAndSlotName, CalamariVariableDictionary variables, SitePublishProfile publishProfile)
15:14:52   Error    |       at Calamari.Azure.Deployment.Conventions.AzureWebAppConvention.Install(RunningDeployment deployment)
15:14:52   Error    |       at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
15:14:52   Error    |       at Calamari.Deployment.ConventionProcessor.RunConventions()
15:14:52   Error    |       Running rollback conventions...
15:14:52   Error    |       Web Deploy experienced a connection problem with the server and had to terminate the connection.  Contact your server administrator if the problem persists.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_CONNECTION_TERMINATED.
15:14:52   Error    |       Microsoft.Web.Deployment.DeploymentDetailedException
15:14:52   Error    |       at Microsoft.Web.Deployment.StatusThreadHandler.CheckForException()
15:14:52   Error    |       at Microsoft.Web.Deployment.AgentClientProvider.RemoteDestSync(DeploymentObject sourceObject, DeploymentSyncContext syncContext, Nullable`1 syncPass, String syncSessionId)
15:14:52   Error    |       at Microsoft.Web.Deployment.DeploymentObject.SyncToInternal(DeploymentObject destObject, DeploymentSyncOptions syncOptions, PayloadTable payloadTable, ContentRootTable contentRootTable, Nullable`1 syncPassId, String syncSessionId)
15:14:52   Error    |       at Microsoft.Web.Deployment.DeploymentObject.SyncTo(DeploymentProviderOptions providerOptions, DeploymentBaseOptions baseOptions, DeploymentSyncOptions syncOptions)
15:14:52   Error    |       at Calamari.Azure.Deployment.Conventions.AzureWebAppConvention.DeployToAzure(RunningDeployment deployment, String siteAndSlotName, CalamariVariableDictionary variables, SitePublishProfile publishProfile)
15:14:52   Error    |       at Calamari.Azure.Deployment.Conventions.AzureWebAppConvention.Install(RunningDeployment deployment)
15:14:52   Error    |       at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
15:14:52   Error    |       at Calamari.Deployment.ConventionProcessor.RunConventions()
15:14:52   Error    |       at Calamari.Azure.Commands.DeployAzureWebCommand.Execute(String[] commandLineArguments)
15:14:52   Error    |       at Calamari.Program.Execute(String[] args)
15:14:52   Error    |       --Inner Exception--
15:14:52   Error    |       There is an unclosed literal string. Line 1, position 95.
15:14:52   Error    |       System.Xml.XmlException
15:14:52   Error    |       at System.Xml.XmlTextReaderImpl.Throw(Exception e)
15:14:52   Error    |       at System.Xml.XmlTextReaderImpl.ParseAttributeValueSlow(Int32 curPos, Char quoteChar, NodeData attr)
15:14:52   Error    |       at System.Xml.XmlTextReaderImpl.ParseAttributes()
15:14:52   Error    |       at System.Xml.XmlTextReaderImpl.ParseElement()
15:14:52   Error    |       at System.Xml.XmlTextReaderImpl.ParseElementContent()
15:14:52   Error    |       at Microsoft.Web.Deployment.TraceEventSerializer.Deserialize(Stream responseStream, DeploymentBaseContext baseContext, DeploymentSyncContext syncContext)
15:14:52   Verbose  |       Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20180721151427-1165-42 exited with code 100
15:14:52   Verbose  |       Updating manifest with output variables
15:14:52   Verbose  |       Updating manifest with action evaluated variables
15:14:52   Fatal    |       The remote script failed with exit code 100
15:14:52   Fatal    |       The action Web on [Staging] Web failed
                    |     
                    |   Canceled: Step 3: Functions
15:14:52   Verbose  |     Step "Functions" runs only when all previous steps succeeded; skipping
                    |

MarkSiedle · 23 July 2018 02:07

Hi Mandest,

Thanks for getting in touch and sorry to hear you’re having this issue.

From what we can see in the logs, this is a communication issue between WebDeploy and Azure. So, strangely, WebDeploy can talk to Azure for one of your WebApps, but not for the other. Could you confirm if these WebApps are both in the same Azure region? Are there any other differences we should be aware of? I.e. Is one using slots but the other is not?

Unfortunately we have had customers report this same issue and it’s never been resolved (because no one could reproduce this). In some cases, the problem suddenly went away after a week or so. In other cases, users setup another WebApp with the same configuration in Azure and it suddenly started working.

When you say the deployment succeeds from VS, where is your Octopus Server installed? Is it somewhere you could easily install Visual Studio (and your source code) so you could test if a right-click-deploy from Visual Studio works from the very same server where Octopus is executing WebDeploy?

If Visual Studio succeeds, but Octopus still gets an exception, then the issue will be related to the way Octopus and Calamari syncs your package against your Azure web app. Calamari is open-source, so you can see the calls Calamari makes to the Microsoft.Web.Deployment.DeploymentManager here.

It is this call to the Microsoft library that is returning the exception: Web Deploy experienced a connection problem with the server and had to terminate the connection and is unfortunately outside of our control.

We’ve also found various forums where people have encountered this issue when deploying to Azure, but none offer a consistent reason/resolution.

Some links that may provide some guidance

The comments in this forum suggest that DNS changes fixed the problem, while others mentioned that having Fiddler running caused this problem (I don’t think this one applies to you, since you have one deploying successfully).

The comments in this github repo suggested problems deploying to various Azure regions and a problem with the instance counts being set in Azure.

Sorry we can’t be of more help, but unfortunately once we reach the call to execute WebDeploy, it’s a communication problem with Azure and WebDeploy and is outside of our control

If WebDeploy continues to be problematic and you’d like to discuss some alternate ways to deploy to Azure (FTP or PowerShell) via Octopus, let me know and I’ll describe some options that might help.

Cheers
Mark

Robert_Glickman · 17 January 2019 19:19

Hi Mark,

We just ran into this issue this past week, with nearly identical log messages. A deployment to Azure succeeded on January 7th @ 4:52pm. The same deployment, with no known changes failed on January 8th @ 10:20am, and it continues to fail today. We upgraded from Octopus version 2018.7.13 to 2018.11.1 on January 7th @ 5:30pm.

As you can see, the upgrade of Octopus seems suspicious. Obviously, working against this is that 2018.7.13 was released after this post was created. Also, in trying to resolve the issue, I downgraded Octopus to 2018.7.13, but that doesn’t seem to have helped. My current theory is that Azure Powershell (or related modules) caused this to break and somehow the rollback of Octopus didn’t rollback the version of Azure Powershell used. Do you know how reliable an Octopus downgrade is in relation to this issue? Can you think of any other connection, or reason this might have cropped up?

Thanks,
Robert

John_Simons · 17 January 2019 21:43

Hi Robert,

Thanks for getting in touch and sorry to hear you’re having a similar issue to this one.

So I can provide the best support can you please send me your raw logs:

before the first upgrade (when it was all working with v2018.7.13)
after the upgrade (when it failed against v2018.11.1)
and also after the downgrade (when you downgraded back to v2018.7.13)

So in total you need to send me 3 raw logs, so I can be sure the issue you are seeing is related to this.

Also, we’ve released a patch in v2018.11.3 that fixed this issue with Azure deployments, are you able to upgrade to this version ?

Regarding downgrading, it all depends, have a read of https://octopus.com/docs/administration/upgrading/guide#Upgrading-DowngradingorRollingBackanUpgrade

Regards
John

Robert_Glickman · 18 January 2019 02:09

Hi John,

What’s the best way to send those logs? Also, I upgraded in the course of reversing the downgrade, so we’re on 2018.11.3 now.

Thanks,
Robert

John_Simons · 18 January 2019 02:16

Hi Robert,

You can upload the logs to https://file.ac/1R52hzHmSh8/

Regards
John

Robert_Glickman · 18 January 2019 02:29

Hi John,

I uploaded the files (twice, because I mislabeled two of them).

Thanks,
Robert

John_Simons · 18 January 2019 06:25

Hi Robert,

We are still investigating why your deployment in failing now.
Do you have any Azure specific application settings for the “staging” slot, eg. WEBSITE_WEBDEPLOY_USE_SCM ?

Regards
John

Robert_Glickman · 18 January 2019 18:24

Hi John,

Thanks for continuing to look at this. We do not use WEBSITE_WEBDEPLOY_USE_SCM. Although we do have some application settings there, none that seem related to Web Deploy. The setting that looks most relevant would be WEBSITE_LOAD_CERTIFICATES, but I don’t think that’s likely related to the issue.

Thanks,
Robert

MarkSiedle · 25 January 2019 06:30

Hi Robert,

Unfortunately we haven’t had any luck reproducing this yet.

Could you please confirm if you’ve noticed any improvements after the upgrade to 2018.11.3? As in, are you noticing it happening any more or less often?

More importantly, have you noticed any patterns emerging, in terms of this only occurring for certain projects/packages and not others? Or perhaps only certain app services in certain subscriptions/regions in Azure (if you have projects deploying to different subscriptions/regions).

Cheers
Mark

alexdangelo · 26 January 2019 01:59

Hi Mark,

Robert will be back next week so I’m chiming in with some updates.

Octopus was downgraded to a previous version and then re-upgraded to 2018.11.3. This bug still reproduces consistently for the specific package we’re trying to deploy.
An engineer on the team tried deploying using Visual Studio 2017 to the Azure staging slot and swapped that into production in Azure, and that worked perfectly.
I went back to Octopus and promoted the prestaging build to production and had the same error still happen.

The package we’re deploying to Azure Web Apps is the only one that has been failing. We deploy lots of other packages and they all succeeded (they are all going to the same region and have the same subscription). It’s just this one we have in particular and only going from prestaging -> staging/production that has the problem, and deployments had worked previously.

I will need to look more here at the steps that succeeded via VS and also re-review the notes that you have earlier in this thread.

Thank you,
Alex

Robert_Glickman · 28 January 2019 20:00

Hi Mark,

I’ll add a little bit of color to what Alex said above. No other deployments are facing this issue that we know of. Other deployments to the same Resource Group (different web app) and the same App Service Plan are succeeding. Additionally, other deployments to this app in other stages are succeeding (same code, different App Service). The problem is specifically deploying this app to our Staging slot.

Thanks,
Robert

Shaun_Marx · 29 January 2019 02:26

Hi Robert,

Since this seems to be something related to this particular website/slot, could you please confirm whether the exact same issue persist if you create a new slot and deploy to that? Similarly, does the same issue persist if you were to create a new Azure Web App and slot?

I would also just like to confirm whether the publish action performed by your engineer via Visual Studio was using Web Deploy? Could you upload the logs for this as well, please?

Regards,
Shaun

Robert_Glickman · 29 January 2019 02:40

Hi Shaun,

The team will be creating a new slot tomorrow to see if it successfully deploys. I’ll see if we can get the logs for the VS deploy.

Thanks,
Robert

Robert_Glickman · 30 January 2019 02:27

This seems to have been fixed now by deleting the Staging slot and recreating it.