We’re using Octopus 3.17.2 and have a Powershell Task setup (both a raw PS task and one with the Azure Subscription and Azure Modules) and in both the call to Publish-AzWebApp hangs:
I’m not sure what to do to troubleshoot it because there’s no trace after the Publish-AzWebApp call.
Logs:
PowerShell 7.2.4
Copyright (c) Microsoft Corporation.
Type ‘help’ to get help.
We have the Azure Modules installed on the server and executing this command while logged into the server does work. Any tips much appreciated, thanks.
Does this seem like it could match what is going on at your end? Are there any other Tasks running against that WebApp at the time you noticed this behaviour?
I’d be happy to take a further look into what’s going on, however as you’ve mentioned, the logs files won’t be too helpful here. Would you be able to please take a process dump of any Tentacle/Calamari exe’s while the deployment is hung? You should be able to upload any files securely here, please let me know if there are any issues with it at all.
Feel free to let me know if you have any questions!
Safe to say we’re not in a deadlock situation with two conflicting jobs.
I may take you up on the Memory Dump (been a long time) but before I do, the reason I’m trying this vanilla PS script is because our task to Deploy Azure Web App has recently started failing.
Deploying to Azure WebApp 'appsvc-MemberDetails-v1(Staging)' in Resource Group confidential, using subscription-id 'confidential
May 26th 2022 15:25:18
Error
Microsoft.Web.Deployment.DeploymentAgentUnavailableException: Could not complete the request to remote agent URL
'https://CONFIDENTIAL@OURSERVICEADDRESS.p.azurewebsites.net/msdeploy.axd?site=appsvc-MemberDetails-v1(Staging)'. --->
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. --->
System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
May 26th 2022 15:25:18
Error
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
May 26th 2022 15:25:18
Error
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
May 26th 2022 15:25:18
Error
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
May 26th 2022 15:25:18
Error
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
It was working 2 weeks ago and ‘apparently’ nothing has changed to config, network or environment.
We’ve added the Cert to the server and no luck and interestingly made a low-level script using Invoke-WebRequest to avoid any libraries and again we saw the same: The underlying connection was closed: An unexpected error occurred on a send
We ran a network trace and see packets are being dropped, we completely whitelisted the site and still are seeing packets drop. So at this stage its looking like a change somewhere, perhaps a hot fix. Octopus Deploy isn’t doing any calls to any other sites that could get blocked?
Any tips of tools to try to diagnose it further would be much appreciated!
Thanks for that extra info, I agree that something else definitely seems to be at play here!
Typically if these errors aren’t from an invalid certificate, there are typically from TLS negotiation issues. I know that Azure has been making moves to deprecate older protocols and now expects Tls1.2 (at least for their Devops Service).
I noticed right at the bottom of that article you linked that someone also indicated that TLS issues would also throw this error: Additionally, invalid TLS settings can cause a similar or identical issue. If the server has disabled for example TLS1.0 but your client (Visual Studio) tries to use 1.0, it will fail with the same message.
We introduced TLS1.2 in Octopus 3.1 so it should be available to use for your version also, however I wonder if the machine hasn’t been configured to use it and now Azure is forcing it. Could you please check the available Ciphers/Suites via a tool like IISCrypto? I also saw a brief mention that programs such as WireShark running in the background could potentially cause errors like this.
Let me know how you get on or if you have any questions!
Cause of the issue: As noted, Octopus Deploy Current old version and the old windows server it’s running on is using TLS 1.0 by default. Enabling TLS 1.2 won’t work as it will continue to use TLS 1.0.
I have tried disabling TLS 1.0 to force to use TLS 1.2, but that broke the other steps . So I have developed a PowerShell script that calls the Kudu API deployment and I have added a PowerShell variable that forces the script to use TLS1.2 and It worked!!!
That’s great to hear you managed to get it working, thanks for the update!
If you were able to upgrade one day, newer versions of Octopus will include a built-in step for leveraging Kudu for deployments. Check out our comparison tool here to see exactly what’s changed for each version.
Feel free to reach out again anytime if you ever have any questions or further issues!
. So I have developed a PowerShell script that calls the Kudu API deployment and I have added a PowerShell variable that forces the script to use TLS1.2 and It worked!!!