This is breaking our deployments about 50% of the time.
Common observations:
If a deployment fails once, its usually halfway through the process.
The targets are Azure Windows VMs.
Reattempting the deployment from just the step where it failed last time usually works.
The health checks on the targets are all completely fine.
There does not appear to be any kind of resource saturation anywhere in the chain (neither network, CPU, RAM, etc).
The failure seems to occur when the process moves from deploying to one set of targets to a different set.
I have tried increasing the timeout values in the Octopus server config as follows, to no avail:
<add key="Logging.Level" value="Debug" />
<add key="SqlServerInstance" value="(local)\SQLEXPRESS" />
<add key="Halibut.ConnectionErrorRetryTimeout" value="00:15:00"/>
<add key="Halibut.TcpClientConnectTimeout" value="00:02:00"/>
<add key="Halibut.TcpClientHeartbeatSendTimeout" value="00:02:00"/>
<add key="Halibut.TcpClientHeartbeatReceiveTimeout" value="00:02:00"/>
Typical deployment run:
2019-07-29 16:32:45.6560 2084 161 ERROR https://tb-l-agnt-1.westeurope.cloudapp.azure.com:10933/ 161 Connection initialization failed while connecting to https://tb-l-agnt-1.westeurope.cloudapp.azure.com:10933/
Halibut.Transport.Protocol.ConnectionInitializationFailedException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.NetworkStream.Read(Byte buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte buffer, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadLine()
at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in C:\buildAgent\work\fe2b45bbd4978f75\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 124
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectProceeed() in C:\buildAgent\work\fe2b45bbd4978f75\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 113
at Halibut.Transport.Protocol.MessageExchangeProtocol.PrepareExchangeAsClient() in C:\buildAgent\work\fe2b45bbd4978f75\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 61
— End of inner exception stack trace —
at Halibut.Transport.Protocol.MessageExchangeProtocol.PrepareExchangeAsClient() in C:\buildAgent\work\fe2b45bbd4978f75\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 66
at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsClient(RequestMessage request) in C:\buildAgent\work\fe2b45bbd4978f75\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 35
at Halibut.HalibutRuntime.<>c__DisplayClass34_0.b__0(MessageExchangeProtocol protocol) in C:\buildAgent\work\fe2b45bbd4978f75\source\Halibut\HalibutRuntime.cs:line 157
at Halibut.Transport.SecureListeningClient.ExecuteTransaction(Action`1 protocolHandler) in C:\buildAgent\work\fe2b45bbd4978f75\source\Halibut\Transport\SecureListeningClient.cs:line 65