Hello,
Any help here is appreciated as we are just about stuck on getting Octopus Deployments executing in our prod environment. We have reviewed all the troubleshooting documentation on your site.
When installing the tentacle on our web server we are able to bring up the Tentacle test page on port 10933 but are receiving an error when checking the health of the tentacle through the Octopus web portal.
The error in the Octopus Web Portal:
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host…
Error in the Octopus Tentacle log:
2021-09-30 12:22:02.4967 6252 6 INFO listen://[::]:10933/ 6 Accepted TCP client: [::ffff:172.24.0.17]:41893
2021-09-30 12:22:02.4967 6252 3 ERROR listen://[::]:10933/ 3 Unhandled error when handling request from client: [
System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Halibut.Transport.SecureListener.d__24.MoveNext()
2021-09-30 12:22:07.5797 6252 10 INFO listen://[::]:10933/ 10 Accepted TCP client: [::ffff:172.24.0.17]:5809
2021-09-30 12:22:07.5797 6252 3 ERROR listen://[::]:10933/ 3 Unhandled error when handling request from client: [
System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
I did look into those articles. Ill take another look and see if I missed something. We are behind a load balancer, but we are able to bring up the test page on port 10933. Do you hhave any suggestions knowing were are behind a load balancer? Thanks!
Have you checked the Octopus Server logs to see if there is a corresponding entry for the connection attempt from the tentacle? If you are able share them, I can have a look through the Tentacle and the Octopus Server logs.
Thanks Donny. I’ve sent the log files using that link. We were receiving an error earlier today where the thumbprint did not match, but think we have resolved that one. Now trying to get through the “unable to read data from transport connection” issue. One example is around 2021-09-30 13:31.
Thank you for getting back to me and providing the requested logs.
Upon checking both the sets of logs, it does appear that something environmental is causing the issue. The connection attempts logged on both sides are showing the following frequent and correlating errors:
Server: An existing connection was forcibly closed by the remote host
Tentacle: Authentication failed because the remote party has closed the transport stream
With something between the Octopus Server and Tentacle is terminating the connection, I would recommend checking the logs of any network appliances between the two (such as firewalls, load balancers, etc).
If you have any additional questions, please don’t hesitate to task.
Are you able to tell if this is certificate related or can we safely say that the ssl communication is successful? Just trying to rule things out to give my network team more to go on. Any additional information you can provide would help. Thanks!
It is difficult to say one way or the other, unfortunately. The only thing we know with certainty is that neither the Octopus Server nor the Tentacle appear to be terminating the connection.
I hope your network team is able to get this resolved for you quickly. If you have any additional questions, please let me know.
Just updating you on our progress with troubleshooting this issue. I spoke to my network team and we spent a good amount of time working through the issue today and are stuck with the same error when initiating the health check against the tentacle -
An error occurred when sending a request to ‘https://.com:10933/’, before the request could begin: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host… Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host… Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host… An existing connection was forcibly closed by the remote host.
We can successfully hit the tentacle test page at port 10933 which gives us the Octopus Tentacle configured successfully message. We also receive this message as expected when running a curl command. In addition we have run the ping / pong app in your troubleshooting guide and the tests pass. As mentioned previously our log files on the tentacle are recording the error that we see in the octopus server ui for the health check.
Could you provide some information on what is going on under the hood when this health check is initiated? Since we are able to access the test page and run the ping pong test successfully it appears the health check is performing some other action that is failing. Do you have any other documentation or advice that could help us pinpoint the issue or would a support call be possible? Thanks.
I am stepping in while Donny is offline. It does look like the network is communicating properly between your Server and Tentacle.
I am wondering if you have any Endpoint/AV protection Running on either the Tentacle machine or the Octopus Server machine that might be killing the healthcheck scripts? We’ve seen success in the past by whitelisting the Octopus home folders.
We have an example script for Windows defender and adding exclusions here: Hardening Octopus - Octopus Deploy
Thanks Garrett. I will check with my team on what you suggested. In the meantime could you provide any more information on what is actually happening during that health check that is not occurring during our other successful test page and ping/pong tests? If we know what the health check is doing under the hood it might point us in the direction of the problem.
When Endpoint/AV protection blocks health checks, it is usually when calling bootstrapper.ps1 on the tentacle that triggers it.
Whitelisting the Tentacle working folders and the Tentacle executable is usually enough to fix this. We see this most often on Server 2012 R2 machines running Endpoint/AV protection with AI/heuristics (such as SentinelOne, TrendMicro, etc).
If you have any additional questions, please let us know.
I did confirm with my team and I looked on the server as well and windows firewall is turned off and there is no antivirus running. I did make sure that the Octopus Deploy service has appropriate permission to the Octopus Deploy directory and the Octopus deployment directory. Any other ideas?
Is it possible to bypass the load balancer between the Octopus server and the tentacle’s machine? If you are still seeing the same error messages,
Server: An existing connection was forcibly closed by the remote host
Tentacle: Authentication failed because the remote party has closed the transport stream
something between the Octopus Server and the Tentacle is interfering.
Unfortunately, these network issues can be quite tricky, especially in complex network environments.
I would also suggest asking if your network team can do a packet trace while demonstrating the issue to confirm where the connection is being terminated.
If you have any additional questions, please let us know.
Hi Donny- Just updating you on our progress. We are now getting the following errors
Within the Octopus UI -
Halibut.Transport.Protocol.ProtocolException: Unable to receive the remote identity; the identity line was empty. at Halibut.Transport.Protocol.MessageExchangeStream.ReadRemoteIdentity() at Halibut.Transport.Protocol.MessageExchangeStream.ExpectServerIdentity() at Halibut.Transport.Protocol.MessageExchangeProtocol.PrepareExchangeAsClient() — End of inner exception stack trace — at Halibut.Transport.Protocol.MessageExchangeProtocol.PrepareExchangeAsClient() at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsClient(RequestMessage request) at Halibut.HalibutRuntime.<>c__DisplayClass39_0.b__0(MessageExchangeProtocol protocol) at Halibut.Transport.SecureListeningClient.ExecuteTransaction(Action`1 protocolHandler, CancellationToken cancellationToken)
In the Tentacle Logs -
2021-10-25 17:28:51.0022 2592 9 INFO listen://[::]:10933/ 9 A client at [::ffff:172.24.0.18]:61253 connected, and attempted a message exchange, but it presented a client certificate with the thumbprint ‘592*****334’ which is not in the list of thumbprints that we trust
2021-10-25 17:29:05.5473 2592 10 INFO listen://[::]:10933/ 10 Accepted TCP client: [::ffff:172.24.0.17]:33407
2021-10-25 17:29:05.5473 2592 9 ERROR listen://[::]:10933/ 9 Unhandled error when handling request from client: [::ffff:172.24.0.17]:33407
System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
We are running an F5 and it looks like the cert/thumbprint being presented by Octopus Server is being overwritten by the F5. Is this something you have come across before?
Hi @Dan-303
Thanks for the extra information which helps narrow down the issue for us.
My guess is that your F5 is set to allow HTTPS traffic for Server to Tentacle communicaitons. However this will likely cause an issue if the F5 is also set for SSL offloading. The F5 is probably intercepting the SSL traffic and presenting its own cert to the Tentacle.
To get around this, you could try changing the protocol the F5 is set to for Server to Tentacle traffic, from HTTPS to TCP to allow the Octopus server speak directly to the Tentacle with no interception for SSL traffic.
Thanks Paraic. To help us progress on this, do you know what information is sent to the tentacle during the health check? Is a host header being sent? My network team is asking because of the unknown identity error.
error in ui:
An error occurred when sending a request to ‘url:10933/’, before the request could begin: Unable to receive the remote identity; the identity line was empty. Unable to receive the remote identity; the identity line was empty. Unable to receive the remote identity; the identity line was empty.
tentacle log:
2021-10-26 11:57:32.8963 2592 58 INFO listen://[::]:10933/ 58 Accepted TCP client: [::ffff::53325
2021-10-26 11:57:32.9122 2592 55 INFO listen://[::]:10933/ 55 A client at [::ffff:]:53325 connected, and attempted a message exchange, but it presented a client certificate with the thumbprint ‘592F7***334’ which is not in the list of thumbprints that we trust
Hi Donny - So we are using the F5 with an irule that will redirect to an internal ip based on the host header value. Since you are saying the host header value is not available in the request, does octopus have another recommendation on how to accomplish this? I don’t think we are the first ones to want to utilize an F5 with Octopus in this manner. Thanks.
I’m not sure if the host header is an issue here.
Judging from the tentacle logs, the Octopus Server is reaching the tentacle and attempting to establish a connection.
2021-10-26 11:57:32.8963 2592 58 INFO listen://[::]:10933/ 58 Accepted TCP client: [::ffff::53325
2021-10-26 11:57:32.9122 2592 55 INFO listen://[::]:10933/ 55 A client at [::ffff:]:53325 connected, and attempted a message exchange, but it presented a client certificate with the thumbprint ‘592F7***334’ which is not in the list of thumbprints that we trust
The issue being encountered is that the certificate that the tentacle is expecting isn’t correct. This is typically caused by SSL offloading on the F5. Enabling SSL passthrough over TCP usually resolves this problem.
If this still doesn’t resolve the problem, another option may be to consider using Polling tentacles instead. This would reverse the communication direction making the tentacle the client and the octopus server the TCP server.