We are experiencing that the octopus tentacles (In Listening Mode) are going offline and health checks are failing with below exception.
Can you please help? This is Urgent.
2020-10-21 00:01:31.7597 28 INFO listen://[::]:10933/ 28 Accepted TCP client: [::ffff:10.161.56.18]:54952
2020-10-21 00:01:31.7753 52 INFO listen://[::]:10933/ 52 Performing TLS server handshake
2020-10-21 00:01:31.8378 52 INFO listen://[::]:10933/ 52 Secure connection established, client is not yet authenticated, client connected with Tls12
2020-10-21 00:01:31.8378 52 INFO listen://[::]:10933/ 52 Client at [::ffff:10.161.56.18]:54952 authenticated as 10011693581C9F7FA144D6277A092706A7CFFF17
2020-10-21 00:15:34.3232 52 INFO listen://[::]:10933/ 52 Unhandled error when handling request from client: [::ffff:10.161.56.18]:54952
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadLine()
at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 93
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 67
at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func`2 incomingRequestProcessor) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 110
at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 126
2020-10-21 01:01:32.9670 28 INFO listen://[::]:10933/ 28 Accepted TCP client: [::ffff:10.161.56.18]:55190
2020-10-21 01:01:32.9826 50 INFO listen://[::]:10933/ 50 Performing TLS server handshake
Is this one tentacle or many? Has anything recently changed in your infrastructure or network that could be causing the connection to be altered? When was the last time the tentacle(s) worked properly?
There are many tentacles failing health checks on many servers.
Some of these servers also have more than one tentacle. But all are in Listening mode.
Some standard windows patches and security software are installed on these servers.
When was the last time they were working correctly? Did a certificate expire? Did something on your network/infrastructure change? Did you upgrade Octopus?
Hi Jeremy, how would the tentacle find out the octopus server ? is this info about the servername is saved in a config file or cert ? Currently the communication from Octopus Server to tentacle is happening… so i was wondering if the tentacle is not able to respond back to server. to find it out, i need to know the places i should look for. BTW, i did all the troubleshooting in the above link and still no luck.
Listening tentacles don’t store the server anywhere, all they do is listen for communications from the server on a specific port. You can find that port out by remoting into the tentacle machine and running tentacle.exe show-configuration --instance==INSTANCENAME in an admin command prompt from the tentacle directory. Docs: https://octopus.com/docs/octopus-rest-api/tentacle.exe-command-line/show-configuration
Was there anything in the tentacle logs that might indicate the issue?
Are you able to follow the network traffic from the tentacle back to the server? This appears to be network related. Do you have any functioning listening tentacles you can compare network settings with?
yes tentacle is going back to server and vice versa. But the health check is timing out from UI in octopus deploy… but the tcp connections are ok and we found this as part of tracking the network traffic.
ty, is there a way for me to check the database to find out a given machine health checks that i can query and find out… At least we can know from when the checks were failing…
To get the history of health checks the easiest way might be to do the Audit Trail on a machine. If you go to one of the affected machines within Infrastructre -> Deployment Targets, then click into an affected machine, and in the upper right click the 3 dots and click Audit Trail. This will let you track the target and you should see when it has gone unhealthy.
Is that the sort of information you were looking for? Please let me know if I misunderstood.
Appreciate your prompt feedback Jeremy. But since we use Octopus version 3.3.2, i dont have infrastructure link in the UI in the Admin Console of Octopus Server.