Issues setting up connection from Octopus server to Octopus Tentacle

Hello,
Any help here is appreciated as we are just about stuck on getting Octopus Deployments executing in our prod environment. We have reviewed all the troubleshooting documentation on your site.

When installing the tentacle on our web server we are able to bring up the Tentacle test page on port 10933 but are receiving an error when checking the health of the tentacle through the Octopus web portal.

The error in the Octopus Web Portal:
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host…

Error in the Octopus Tentacle log:

2021-09-30 12:22:02.4967 6252 6 INFO listen://[::]:10933/ 6 Accepted TCP client: [::ffff:172.24.0.17]:41893
2021-09-30 12:22:02.4967 6252 3 ERROR listen://[::]:10933/ 3 Unhandled error when handling request from client: [
System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Halibut.Transport.SecureListener.d__24.MoveNext()
2021-09-30 12:22:07.5797 6252 10 INFO listen://[::]:10933/ 10 Accepted TCP client: [::ffff:172.24.0.17]:5809
2021-09-30 12:22:07.5797 6252 3 ERROR listen://[::]:10933/ 3 Unhandled error when handling request from client: [
System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

Hi @Dan-303,

Thank you for contacting Octopus Support.

I would recommend running through our troubleshooting documentation for listening tentacles. I would also try the Tentacle Ping tool.

Do you have a proxy and/or load balancer between the Octopus Server and Tentacle?

Let me know at your earliest convenience.

Best Regards,
Donny

I did look into those articles. Ill take another look and see if I missed something. We are behind a load balancer, but we are able to bring up the test page on port 10933. Do you hhave any suggestions knowing were are behind a load balancer? Thanks!

Hi @Dan-303,

Thank you for getting back to me.

Have you checked the Octopus Server logs to see if there is a corresponding entry for the connection attempt from the tentacle? If you are able share them, I can have a look through the Tentacle and the Octopus Server logs.

Here is a secure link you may use to upload those:
https://octopus.com/support/individual/xctgk5mbpfsnrqtnctu8r4muqp1sa5nycf4ze55ucp3g1auff3ts65jnxw$5348847$ygcg61qtc5u7d9a9e7rx39tieipsi7r5

I look forward to hearing back from you.

Best Regards,
Donny

Thanks Donny. I’ve sent the log files using that link. We were receiving an error earlier today where the thumbprint did not match, but think we have resolved that one. Now trying to get through the “unable to read data from transport connection” issue. One example is around 2021-09-30 13:31.

Hi @Dan-303,

Thank you for getting back to me and providing the requested logs.

Upon checking both the sets of logs, it does appear that something environmental is causing the issue. The connection attempts logged on both sides are showing the following frequent and correlating errors:

  • Server: An existing connection was forcibly closed by the remote host
  • Tentacle: Authentication failed because the remote party has closed the transport stream

With something between the Octopus Server and Tentacle is terminating the connection, I would recommend checking the logs of any network appliances between the two (such as firewalls, load balancers, etc).

If you have any additional questions, please don’t hesitate to task.

Best Regards,
Donny

Are you able to tell if this is certificate related or can we safely say that the ssl communication is successful? Just trying to rule things out to give my network team more to go on. Any additional information you can provide would help. Thanks!

Hi @Dan-303,

Thank you for getting back to me.

It is difficult to say one way or the other, unfortunately. The only thing we know with certainty is that neither the Octopus Server nor the Tentacle appear to be terminating the connection.

I hope your network team is able to get this resolved for you quickly. If you have any additional questions, please let me know.

Best Regards,
Donny

Hi Donny,

Just updating you on our progress with troubleshooting this issue. I spoke to my network team and we spent a good amount of time working through the issue today and are stuck with the same error when initiating the health check against the tentacle -
An error occurred when sending a request to ‘https://.com:10933/’, before the request could begin: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host… Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host… Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host… An existing connection was forcibly closed by the remote host.

We can successfully hit the tentacle test page at port 10933 which gives us the Octopus Tentacle configured successfully message. We also receive this message as expected when running a curl command. In addition we have run the ping / pong app in your troubleshooting guide and the tests pass. As mentioned previously our log files on the tentacle are recording the error that we see in the octopus server ui for the health check.

Could you provide some information on what is going on under the hood when this health check is initiated? Since we are able to access the test page and run the ping pong test successfully it appears the health check is performing some other action that is failing. Do you have any other documentation or advice that could help us pinpoint the issue or would a support call be possible? Thanks.

Hi @Dan-303,

I am stepping in while Donny is offline. It does look like the network is communicating properly between your Server and Tentacle.

I am wondering if you have any Endpoint/AV protection Running on either the Tentacle machine or the Octopus Server machine that might be killing the healthcheck scripts? We’ve seen success in the past by whitelisting the Octopus home folders.
We have an example script for Windows defender and adding exclusions here: Hardening Octopus - Octopus Deploy

Please let me know if that helps at all.

Regards,
Garrett

Thanks Garrett. I will check with my team on what you suggested. In the meantime could you provide any more information on what is actually happening during that health check that is not occurring during our other successful test page and ping/pong tests? If we know what the health check is doing under the hood it might point us in the direction of the problem.

Hi @Dan-303,

Thank you for getting back to us.

When Endpoint/AV protection blocks health checks, it is usually when calling bootstrapper.ps1 on the tentacle that triggers it.

Whitelisting the Tentacle working folders and the Tentacle executable is usually enough to fix this. We see this most often on Server 2012 R2 machines running Endpoint/AV protection with AI/heuristics (such as SentinelOne, TrendMicro, etc).

If you have any additional questions, please let us know.

Best Regards,
Donny

Hey Guys,

I did confirm with my team and I looked on the server as well and windows firewall is turned off and there is no antivirus running. I did make sure that the Octopus Deploy service has appropriate permission to the Octopus Deploy directory and the Octopus deployment directory. Any other ideas?

Hi @Dan-303,

Thank you for getting back to us.

Is it possible to bypass the load balancer between the Octopus server and the tentacle’s machine? If you are still seeing the same error messages,

  • Server: An existing connection was forcibly closed by the remote host
  • Tentacle: Authentication failed because the remote party has closed the transport stream

something between the Octopus Server and the Tentacle is interfering.

Unfortunately, these network issues can be quite tricky, especially in complex network environments.

I would also suggest asking if your network team can do a packet trace while demonstrating the issue to confirm where the connection is being terminated.

If you have any additional questions, please let us know.

Best Regards,
Donny