Deployment fails on uploading packages to tentacle after upgrade to 3.0

We are deploying to 2 different datacenters (different vendors). On 2.6 we always had the error message during upload of packages when deploying to one of them, but octopus retried up to 4 times and it usually worked. We now upgraded to 3.0 and it doesn’t appear to be retrying when it fails and now we are unable to deploy to any servers in that datacenter, so fairly urgent… It’s only one package that fails, the other seem to upload correctly.

Error is:

An error occurred when sending a request to 'https://IP_REMOVED:10933/', after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.

Tentacle log below:
   at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 91
   at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 65
   at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func`2 incomingRequestProcessor) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 110
   at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsServer(Func`2 incomingRequestProcessor, Func`2 pendingRequests) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 90
   at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 122
2015-08-19 16:07:33.9078      9  INFO  listen://0.0.0.0:10933/           9  Unhandled error when handling request from client: IP_REMOVED:45487
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadLine()
   at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 91
   at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 65
   at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func`2 incomingRequestProcessor) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 110
   at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsServer(Func`2 incomingRequestProcessor, Func`2 pendingRequests) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 90
   at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 122
2015-08-19 16:07:44.2002     10  INFO  listen://0.0.0.0:10933/          10  Unhandled error when handling request from client: IP_REMOVED:45486
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---

Any ideas? I tried setting maxparallelism in the project to 1 but that didnt help.

Hello,

You are correct. In 3.0 we aren’t retrying on connection errors as we did in 2.6. This wasn’t a deliberate decision, just a side-effect of changing our communication stack.

I’ve raised an issue to (re)implement a retry policy in 3.0; which we will implement as soon as we can.

I realize this doesn’t help you this moment. Is it possible to address why the connection is intermittently failing?

Regards,
Michael