Deployment for one enviroment stopped working

bjoern.rieken · 8 January 2021 11:22

Hi all,

starting from December we have a strange issue occurring in one of our deployments. Or setup is:

Octopus server 3.13.2
Polling Tentacle (we tried 3.13.2 and 6.0.383)
Most steps based on Powershell
Integration environment (running in the same network as the Octopus server, three different targets)
Test environment (running at a customer, three different targets)
Production environment (running in the same network as Test, three different targets)

The process we have created is running since 2018 on all three environments (Test and Production are running the same steps, there are some differences for Integration), but starting in December, the release to the Production environment started to fail, while Test is running fine. The eror we are seeing in the Octopus log is:
The step failed: Activity Deploy Build Result on environment target name failed with error 'Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

Server exception: 
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Halibut.Transport.Protocol.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count)
   at System.IO.Compression.DeflateStream.Read(Byte[] array, Int32 offset, Int32 count)
   at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
   at System.IO.BinaryReader.ReadInt32()
   at Newtonsoft.Json.Bson.BsonReader.ReadNormal()
   at Newtonsoft.Json.Bson.BsonReader.Read()
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at Halibut.Transport.Protocol.MessageExchangeStream.ReadBsonMessage[T]()
   at Halibut.Transport.Protocol.MessageExchangeStream.Receive[T]()
   at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessReceiverInternal(IPendingRequestQueue pendingRequests, RequestMessage nextRequest)'.

The error occurs randomly, so in one deployment it will show up for step 1, in the next in step 3 etc., but always in one of the first few steps.

When I make a health check for the targets in the environment, the mentioned error also randomly shows up, so basically a target is reachable, not reachable (with the mentioned error) and reachable again.

When we check the Tentacle logs, we see the following issue (I’ve removed the server URL and the thumbprint):
8 INFO https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 INFO https://octopus.server.url:10943/ 8 INFO https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 TRACE https://octopus.server.url:10943/ 8 INFO https://octopus.server.url:10943/ System.Security.Authentication.AuthenticationException: XYZ
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectProceeed()
at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsSubscriber(Uri subscriptionId, Func2 incomingReq 8 Opening a new connection 8 Connection established 8 Performing TLS handshake 8 Secure connection established. Server at [::ffff:127.0.0.4]:10943 identified by thumbprint: xxx, using protocol Tls12 8 Received: IScriptService::GetStatus[11] / 7cefb031-9f45-4461-b210-11d753b4d048 8 Sent: Halibut.Transport.Protocol.ResponseMessage 8 Received: 8 Received: IScriptService::GetStatus[12] / 5c64dc2a-09d9-4649-8072-887a97ca82fc 8 Sent: Halibut.Transport.Protocol.ResponseMessage 8 Received: IScriptService::GetStatus[13] / 66b5e734-7d31-44d8-8a6d-7f23a593f663 8 Sent: Halibut.Transport.Protocol.ResponseMessage 8 Received: IScriptService::GetStatus[14] / 0e8afc96-e8ee-4454-9128-8e07603a1d65 8 Sent: Halibut.Transport.Protocol.ResponseMessage 8 Received: IScriptService::GetStatus[15] / 6910756a-a5bb-4963-9835-9ae5a11c90fe 8 Sent: Halibut.Transport.Protocol.ResponseMessage 8 Received: IScriptService::GetStatus[16] / 1d7053f7-80c0-489f-9572-7a5c84a167bf 8 Sent: Halibut.Transport.Protocol.ResponseMessage 8 Received: IScriptService::GetStatus[17] / 88ed866f-5b4a-4941-8e5a-e533653d2769 8 Sent: Halibut.Transport.Protocol.ResponseMessage 8 Received: IScriptService::GetStatus[18] / 0194306b-3f73-4b8f-b921-8cd19939fecd 8 Sent: Halibut.Transport.Protocol.ResponseMessage 8 Received: IScriptService::GetStatus[19] / 033cd190-dcbb-45c1-93ca-3d4d301b7a3a 8 Sent: Halibut.Transport.Protocol.ResponseMessage 8 Received: IScriptService::GetStatus[20] / 994dbe68-8682-4830-8f41-391312998f5d 8 Sent: Halibut.Transport.Protocol.ResponseMessage 8 Authentication failed while setting up connection to https://octopus.server.url:10943/ uestProcessor, Int32 maxAttempts) at Halibut.Transport.SecureClient.ExecuteTransaction(Action1 protocolHandler)

But that error is not related to a deployment, it also shows up in times without any deployment.

We have tried the following:

Reinstalling the tentacle
Upgrading the tentacle to the latest version
Checking the firewalls
Checking the TLS version

Right now, we would appreciate any help or idea for finding the issue. The issue in Octopus might point to a network problem, but shouldn’t that issue also show up for the Test environment, since it is the same network?

charles.hague · 8 January 2021 13:22

Hi Bjoern,

Thank you for reaching out to us with your query.

The most likely explanation for this would be an issue with the production machines as it sounds like they are dropping the connection at apparently random intervals. It could also be a network issue where an intermediate device is interrupting the connection for some reason.

Could you please check to see if any changes have recently been made? For example, operating system updates installed, anti-virus or other security software added/changed/updated, changes to the network configuration and so on.

Best Regards,

Charles

system · 8 February 2021 13:22

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.