Error during Acquire Packages

Legacy Uncategorised
1

we run about 20 deploys every morning to our QA and Staging environments for our multiple tenants. Most work without a problem, but each morning, one or two will fail during Acquire Packages.

each server gets multiple packages, and this doesn’t necessarily occur on the first package, but often later, after several have uploaded successfully. the error is this:

Uploading package Encompass.DatabaseLoggingService 2015.2.2 Beginning streaming transfer of Encompass.DatabaseLoggingService.2015.2.2.nupkg-6414876f-426e-4993-a79f-274563d71954 An error occurred when sending a request to 'https://66.162.195.202:10933/', after the request began: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. Halibut.HalibutClientException: An error occurred when sending a request to 'https://66.162.195.202:10933/', after the request began: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) --- End of inner exception stack trace --- at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count) at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count) at System.IO.Compression.DeflateStream.Read(Byte[] array, Int32 offset, Int32 count) at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count) at System.IO.BinaryReader.FillBuffer(Int32 numBytes) at System.IO.BinaryReader.ReadInt32() at Newtonsoft.Json.Bson.BsonReader.ReadNormal() at Newtonsoft.Json.Bson.BsonReader.ReadInternal() at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, JsonContract contract, Boolean hasConverter) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader) at Halibut.Transport.Protocol.MessageExchangeStream.ReadBsonMessage[T]() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 198 at Halibut.Transport.Protocol.MessageExchangeStream.Receive[T]() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 152 at Halibut.HalibutRuntime.<>c__DisplayClass6.<SendOutgoingHttpsRequest>b__5(MessageExchangeProtocol protocol) in y:\work\7ab39c94136bc5c6\source\Halibut\HalibutRuntime.cs:line 115 at Halibut.Transport.SecureClient.ExecuteTransaction(Action 1 protocolHandler) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureClient.cs:line 60 --- End of inner exception stack trace --- Server stack trace: at Halibut.Transport.SecureClient.HandleError(Exception lastError, Boolean retryAllowed) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureClient.cs:line 154 at Halibut.Transport.SecureClient.ExecuteTransaction(Action 1 protocolHandler) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureClient.cs:line 94 at Halibut.HalibutRuntime.SendOutgoingHttpsRequest(RequestMessage request) in y:\work\7ab39c94136bc5c6\source\Halibut\HalibutRuntime.cs:line 117 at Halibut.ServiceModel.HalibutProxy.Invoke(IMessage msg) in y:\work\7ab39c94136bc5c6\source\Halibut\ServiceModel\HalibutProxy.cs:line 37 Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Octopus.Shared.Contracts.IFileTransferService.UploadFile(String remotePath, DataStream upload) at Octopus.Worker.Tentacles.TentacleRemoteEndpointFacade.UploadFile(String fileName, DataStream package) in Y:\work\refs\tags\3.1.4\source\Octopus.Worker\Tentacles\TentacleRemoteEndpointFacade.cs:line 73 Octopus.Server version 3.1.4 (3.1.4+Branch.master.Sha.3510f96241b0bb85f7189448bb7fef417340ebf8)

it then says

File upload failed. Retry attempt 1 of 5...

and actually retries 5 more times, before finally giving up.

I have attached a log of a deployment with this error.

any help would be appreciated.

ServerTasks-13483.log.txt (59 KB)

2

Hi Mike,

I have looked at your log file; my suspicion is that something (firewall, proxy, etc) is killing the connections once they have been open for a specified amount of time. This would fit the symptoms. The library (Halibut) we use for communication between Tentacles and the server will re-use connections, rather than open new ones. So your deploys run fine for a while, until something decides to kill the connections between the server and the Tentacle. We then attempt to retry up to 5 times (as you noticed), but we are using pooled connections which in this case have also been killed. At least that’s my theory.

There are some config settings we can play with to see if it resolves the problem. The first I think is worth trying is:
<add key="Halibut.TcpClientPooledConnectionTimeout" value="00:00:00"/>

The config line above should be added to the appSettings section of Octopus.Server.exe.config, located by default at C:\Program Files\Octopus Deploy\Octopus. This determines the how long between uses until a pooled connection expires. Setting it to zero (as above), effectively means connections are not pooled. It would be interesting to see if this resolves the problem, and also if it adversely affects performance. You can always tweak the value to find a balance if necessary.

Please let me know if this helps? If not, we’ll dig further.

Regards,
Michael

3

made the change. we’ll see how things go tomorrow morning.

4

well, we’ve gone through two daily deployments and the error has not recurred, so the fix appears to be effective. thank you very much!

5

You’re most welcome. I’m glad to hear it helped.

Just one thing to keep in mind: currently when you upgrade your Octopus server, those settings are wiped, and you will have to re-add them. We are working on improving this.

Happy Deployments!
Michael R

6

Hi,
I had the same issue and it was also fixed by adding that setting. Thanks heaps!