We’ve recently encountered an error on two of our servers where both tentacles suddenly lost connection. I’ve gone through the troubleshooting steps for the listening tentacles but see no difference. It happening on both of our servers with two different tentacles tells me it’s probably a network/firewall issue, I’ve done my best to troubleshoot that part as well. The most recent error in the logs is this:
System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> System.ComponentModel.Win32Exception: An unknown error occurred while processing the certificate
--- End of inner exception stack trace ---
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Halibut.Transport.SecureListener.<ExecuteRequest>d__23.MoveNext()
Thank you for contacting Octopus Support and sorry to hear you are seeing issues with two of your tentacles.
You mentioned you did check out our troubleshooting pages but I am wondering if you managed to see this one about Schannel and TLS as the error you are seeing is explicitly referenced in that documention.
Have you changed any TLS protocols on those servers recently or applied any windows patches that might have impacted TLS or Schannel?
Are you able to make use of IISCrypto on the tentacles to make sure the TLS cyphers match those on the Octopus Server.
I look forward to hearing from you,
Kind Regards,
Clare
Sorry, I’ve also gone through those steps, with IISCrypto and without. No success unfortunately. The health check produces the same result of:
An error occurred when sending a request to 'xxxxx:10933/', before the request could begin: The client was unable to establish the initial connection within 00:01:00.
Thank you for confirming you took a look at that document and thanks for providing information from your health check. That error does point towards network I am afraid so there is not much Octopus can do to resolve that.
The only things you can do is to:
Make sure all of your TLS settings match what is on the Octopus Server.
Ports 10933 and 443 are open on the internal (VM / machine) and external firewalls and accepting traffic (10933 is what the tentacle uses to communicate on and 443 is what it uses for its initial communication when setting up a tentacle). You can run netstat -a to find all the listening ports on that machine.
Check the tentacles can ping the Octopus Server (you can also use our tentacle ping tool which you would have seen in our troubleshooting documentation as that allows you to actually ping through port 10933 (or a port of your choosing).
Check all firewall, Load Balancer, and networking logs (do you have wireshark or a more advanced network log available for use) for those tentacle ports being blocked.
Do you have tentacles that do connect to the Octopus Sever, if so I would use those as a baseline, compare all settings network related and TLS (using IIS Crypto) and make sure the failing tentacles match the working ones.
Were those tentacles connecting to Octopus before and they have suddenly stopped? Has anything changed with those machines, are they on a different VLAN to the machines that are working, have they recently been moved, had anything changed on the load balancer if you have one or external firewall?
I would usually get some logs from you but all they will show is the connection either timed out or the ‘target machine actively refused the connection’ similar to what the health check shows, which again points to network. I would triple check your Octopus Tentacle and Server logs though as they might be a bit more specific with the error, sometimes if its a TLS issue we do log that rather than a standard refusal of connection.
I am sorry I could not be of further help here but hopefully my suggestions above lead you to a positive outcome and you are able to pinpoint the issue. Let us know if you do.