OD 2.6 Tentacle Connection Issues

jasonw · 2 February 2015 16:58

After upgrading to version 2.6, we seem to get this connectivity error 40% of the time during deployments.

Operation: Acquire package [PackageName] failed with error: Operation: Upload package to https://192.168.3.9:10933/ failed with error: One or more items in the current operation failed.
Fatal 09:48:45
The step failed

Exception:
The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided.
Pipefish.PipefishException: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided. —> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Security._SslStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Octopus.Shared.FileTransfer.FileSender.b__5(Stream stream) in y:\work\refs\heads\release\source\Octopus.Shared\FileTransfer\FileSender.cs:line 110
at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request, Action1 response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 83 --- End of inner exception stack trace --- at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request, Action1 response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 114
at Pipefish.Transport.SecureTcp.Streaming.Client.StreamClient.Send(Action`1 streamWriter, Int64 contentLength) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Streaming\Client\StreamSender.cs:line 60
at Octopus.Shared.FileTransfer.FileSender.Receive(SendStreamRequest message) in y:\work\refs\heads\release\source\Octopus.Shared\FileTransfer\FileSender.cs:line 122
at Pipefish.Actor.OnReceivingTyped[TBody](Message message) in y:\work\3cbe05672d69a231\source\Pipefish\Actor.cs:line 113
Octopus.Server version 2.6.0.778

If we click retry, sometimes it just works. We hadn’t run into this before the upgrade and it doesn’t seem to be affected by the ‘Polling’ versus ‘Listening’ configuration.

This is dramatically effecting our build-deploy cycle and causing build-failures left and right. Any help will be greatly appreciated.

Dalmiro · 3 February 2015 17:12

Hi Jason,

Thanks for reaching out. All we can see from that error is that the problem gets triggered when the package gets uploaded to your tentacle. What is the network relationship between your tentacle and the Octopus server? (is there a firewall, proxy or load balancer in the middle?) Is it possible that you have some sort of network restriction that prevents you from uploading certain amount of data to your tentacle server?

Thanks

Dalmiro

jasonw · 3 February 2015 18:04

They are on a private network with no firewall restrictions, proxies or
anything between them. This error is intermittent.

I can try again and it will work, so it doesn’t seem to be a setting that
would prevent it from working.

Dalmiro · 11 February 2015 17:03

Hi Jason,

Please stay put for our next release of 2.6.1 on which we fixed a thing or two that were causing deployments to randomly stall (and eventually timeout). It should be out before the end of the week and it’ll hopefully help you with this issue.

Thanks!

Dalmiro.

jasonw · 11 February 2015 18:53

Thank you!

jasonw · 12 February 2015 17:14

Hi Dalmiro,

Thanks for your response. Our maintenance licence expired today on the day
it rolled out! Is there anyway we can get the latest update to get our
systems running again?

markus · 16 February 2015 20:48

Hi, we are having the same issues in version 2.6.2, even with retries:

Failed to send stream: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided.
Info    13:40:51
Retry attempt #1
Info    13:40:51
Failed to send stream: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided.
Info    13:40:53
Retry attempt #2
Info    13:40:53
Failed to send stream: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided.
Info    13:40:55
Retry attempt #3
Info    13:40:55
Failed to send stream: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided.
Info    13:40:57
Retry attempt #4
Fatal    13:40:57
The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided.
Pipefish.PipefishException: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided. ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Security._SslStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Octopus.Shared.FileTransfer.FileSender.<SendStream>b__5(Stream stream) in y:\work\refs\heads\master\source\Octopus.Shared\FileTransfer\FileSender.cs:line 145
   at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request, Action`1 response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 87
   --- End of inner exception stack trace ---
   at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request, Action`1 response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 120
   at Pipefish.Transport.SecureTcp.Streaming.Client.StreamClient.Send(Action`1 streamWriter, Int64 contentLength) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Streaming\Client\StreamSender.cs:line 60
   at Octopus.Shared.FileTransfer.FileSender.SendStream() in y:\work\refs\heads\master\source\Octopus.Shared\FileTransfer\FileSender.cs:line 125
   at Octopus.Shared.FileTransfer.FileSender.Receive(SendStreamRequest message) in y:\work\refs\heads\master\source\Octopus.Shared\FileTransfer\FileSender.cs:line 107
   at Pipefish.Actor.OnReceivingTyped[TBody](Message message) in y:\work\3cbe05672d69a231\source\Pipefish\Actor.cs:line 113
Tentacle version 2.6.2.845

jasonw · 16 February 2015 22:15

We are good. The update seemed to have fixed our issue.

Eric_Johnson · 17 February 2015 18:17

Hey gang, I am also seeing the same thing. I am on 2.6.2.845. If I re-install the tentacle, it fixes it for maybe one to two attempts. After that, we are back to these errors.

Uploading SP.Web.1.0.10 (233.44 MB) to SQ-STAGING2-B5693F2A

Info 11:13:16
Failed to send stream: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided.

Info 11:13:19
Retry attempt #1

Info 11:13:40
Failed to send stream: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided.

Info 11:13:42
Retry attempt #2

Info 11:14:02
Failed to send stream: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided.

Info 11:14:04
Retry attempt #3

Info 11:14:28
Failed to send stream: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided.

Info 11:14:30
Retry attempt #4

Fatal 11:14:50
The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided.
Pipefish.PipefishException: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided. —> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Security._SslStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Octopus.Shared.FileTransfer.FileSender.b__5(Stream stream) in y:\work\refs\heads\master\source\Octopus.Shared\FileTransfer\FileSender.cs:line 145
at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request, Action1 response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 87 --- End of inner exception stack trace --- at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request, Action1 response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 120
at Pipefish.Transport.SecureTcp.Streaming.Client.StreamClient.Send(Action`1 streamWriter, Int64 contentLength) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Streaming\Client\StreamSender.cs:line 60
at Octopus.Shared.FileTransfer.FileSender.SendStream() in y:\work\refs\heads\master\source\Octopus.Shared\FileTransfer\FileSender.cs:line 125
at Octopus.Shared.FileTransfer.FileSender.Receive(SendStreamRequest message) in y:\work\refs\heads\master\source\Octopus.Shared\FileTransfer\FileSender.cs:line 107
at Pipefish.Actor.OnReceivingTyped[TBody](Message message) in y:\work\3cbe05672d69a231\source\Pipefish\Actor.cs:line 113
Octopus.Server version 2.6.2.845

Eric_Johnson · 17 February 2015 18:24

To provide more info, while watching the task monitor, it seems that the file is getting partially uploaded but times out before completion. It will get anywhere from 25% to 85% uploaded in the four tries. Is there a way to set a longer timeout period?

Dalmiro · 18 February 2015 19:08

Jason - Glad to hear that

Cheers!

Dalmiro · 18 February 2015 19:14

Hi Eric,

Thanks for reaching out. Your upload seems to be failing 4-5 seconds after starting. We currently have a default timeout of 30mins (which you cant increase), so it shouldn’t fail that fast from an Octopus perspective. This leads me to believe there might some sort of network restriction/rule killing the connection as soon as it begins. Would you be able to check this from a network perspective and let us know your findings?

Regards,

Dalmiro

markus · 20 February 2015 19:08

Anything new on this issue? We still have this problem.

What kind of network restriction or rule are you talking about?
We are using polling tentacles.

Regards,
Markús

Eric_Johnson · 20 February 2015 23:08

Hey Markus,

So I spent a great deal of time on the phone with Backspace, my hosting provider, and the communication between the servers is unrestricted and performing as expected. However, I am still having issues with the file transfers dying roughly 65% of the time.

Eric

Dalmiro · 24 February 2015 18:47

Hi Markus and Eric,

We have a tool called TentaclePing and TentaclePong that we use to simulate and troubleshoot Octopus/Tentacle communications. You can read about how to use them and download them here:

Try to setup [datasizes] similar to your packages sizes and a [filechunk] of 4mb. Let us know how it works and also feel free to share this info with your hosting provider/network dept.

Please keep us posted on how it goes

Dalmiro.

markus · 24 February 2015 20:58

Hi Dalmiro, thank you, but it is only “polling tentacles” that are not working. Do you have a similar tool for testing polling tentacles?

Eric_Johnson · 25 February 2015 20:42

So I ran the test and it does good up until a 50mb file, then it starts to stumble. However, I am able to transfer a 200mb file between the two servers without issue. The network connectivity only seems to be an issue in the context of octopus. ANY help with this would be greatly appreciated.

Eric

Eric_Johnson · 25 February 2015 21:33

Hey guys, I need to retract my earlier statement. I have found my issue. Windows 2012 and apparently 2012 R2 have an issue with TCP Offload. By turning that off on the network, my network was able to transfer without issue.

Hope this helps!

Eric Johnson

markus · 25 February 2015 21:39

Hi Eric, did you turn TCP Offload off on the on the Octopus Server or on the Tentacle side?

Markús

Eric_Johnson · 25 February 2015 21:55

Both.