Tentacle service crashing and cannot find cause in Event logs

Legacy Uncategorised
1

Last week, many of the Tentacle services apparently crashed in our environment, and I have not been able to find any reasons in the Application or System Event logs.

Below appears to be the last information logged from last Tuesday, Nov 8.
We are running v3.4.11, and all Tentacles have been updated to this as well

Any suggestions to help determine what caused this, or what to enable in order to capture more information if this were to occur in the future?

2016-11-08 20:20:49.9583 80 INFO listen://[::]:10933/ 80 Unhandled error when handling request from client: [::ffff:10.97.8.115]:49307
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadLine()
at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 93
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd() in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 67
at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func`2 incomingRequestProcessor) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 110
at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureListener.cs:line 140

2

Hi Jim,

Thanks for getting in touch! I’m sorry to hear you had some issues with your Octopus setup.

One thing you can do is to change the logging level on the Tentacle to log more information that might hint at what happened. To change it, follow the instructions on this page.

Hope that helps!

Thank you and best regards,
Henrik

3

Hi Henrik,

Is there a way to do this to multiple Tentacles remotely?
Also, since it appears this happened during the patching and reboot process
(which I execute and manage as well) of the Tentacle server, do you suggest
I just keep the additional logging set just before times we have special
system maintenance (like patching) occurring? And then turn it off until
just before we have other systems maintenance?

I think the service either failed to start, or else started and crashed
afterward.
Thanks.

  • Jim Locke.
4

Hi Jim,

Unfortunately there is no way to automate the update of logging level remotely, maybe next time you have system maintenance to complete, you could update the logging on one of the Tentacle servers and complete the maintenance and see if it experience the same issue as previously. We do recommend only having the logging change during activities being investigated as it will impact the performance of your server otherwise.

Thank you and best regards,
Henrik