OctopusDeploy Tentacle service doesn't auto-start after server restart

Hi,

We have 8 identical servers with OctopusDeploy Tentacle service running there. These servers are rebooted every morning. The service auto-starts successfully on 7 of them, but always fail on one of them. If I start the service manually everything works. How to fix it so that the service auto-starts successfully? Note that this has been working in past and we never had this issue in an older version of Octopus (2.x).

Octopus 3.3.20

Errors from the Windows System Logs:

Event 7009, Service Control Manager
A timeout was reached (30000 milliseconds) while waiting for the OctopusDeploy Tentacle service to connect.

Event 7000, Service Control Manager
The OctopusDeploy Tentacle service failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion.

Errors from Logs\OctopusTentacle.txt

2016-07-20 06:37:34.0563 13 INFO listen://[::]:10933/ 13 Unhandled error when handling request from client: [::ffff:10.204.4.27]:19930
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadLine()
at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 93
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 67
at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func`2 incomingRequestProcessor) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 110
at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 126

Thanks,
Miro

Hi Miro,

Thanks for reaching out!

  • Is the Tentacle service set to Automatic or Automatic (Delayed start). If its the first one, you might wanna try with the 2nd.

  • How are you rebooting your servers? We have an open issue for Tentacles not being started when the machine gets rebooted using SCCM/SCOM: https://github.com/OctopusDeploy/Issues/issues/2375

Thanks,
Dalmiro

Hi Dalmiro

Thank you for your advise. The service is set to “Automatic”. I’ve tried to set it to “Automatic (Delayed start)” so will be able to tell tomorrow if that works.

We are rebooting servers by using Task Scheduler in Windows, which runs a command similar to “shutdown /r /t 0”.

Thanks,
Miro

Hi,

I have set the service to “Automatic (Delayed start)” and it looks like as a workaround it has solved the problem.

Thanks,
Miro