Issue Setting Up Polling Tentacles

Octopus Server
1

Hello,

My company has Octopus licenses that we purchased and we are having problems connecting one of our client environments to our Octopus Server which is self-hosted. This is only happening for a specific environment, all our other clients were able to be connected to the Octopus Server successfully. We are connecting via polling Tentacles.

When attempting to install the Octopus Tentacle on a Windows Server 2019 machine we get the following error thrown.

Checking connectivity on the server communications port 10943...
Checking that server communications are open failed with message The underlying connection was closed: An unexpected error occurred on a send.. Retrying (1/5) in 00:00:00.7500000.
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
   at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.ConnectStream.WriteHeaders(Boolean async)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetResponse()
   at Octopus.Tentacle.Communications.OctopusServerChecker.<>c__DisplayClass2_0.<CheckServerCommunicationsIsOpen>b__1()
   at Octopus.Tentacle.Communications.OctopusServerChecker.Retry(String actionDescription, Action action, Int32 retryCount, TimeSpan initialDelay, Double backOffFactor)

When I navigate to the Octopus server URL from the client machine I am able to access the website successfully. I also successfully did a telnet from the client machine to the Octopus server via port 10943.

I took a look at the TLS settings and both server and client machines have both TLS 1.1 and 1.2 enabled in the registry.

I then took a look at the Octopus Server logs and saw this being thrown, which tells me that the request from the client machine is getting to the Octopus Server but the server is refusing it.

2022-08-16 15:09:26.0649   4128     58  INFO  "listen://[::]:10943/"           84  "Accepted TCP client: [::ffff:71.5.104.228]:22394"
2022-08-16 15:09:26.0649   4128     58 ERROR  "listen://[::]:10943/"           84  "Socket IO exception: [::ffff:71.5.104.228]:22394"
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ReceiveAsync(Socket socket, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.ReceiveAsync(Memory`1 buffer, SocketFlags socketFlags, Boolean fromNetworkStream, CancellationToken cancellationToken)
   at System.Net.Sockets.NetworkStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.FillHandshakeBufferAsync[TIOAdapter](TIOAdapter adapter, Int32 minSize)
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Security.SslStream.ProcessAuthentication(Boolean isAsync, Boolean isApm, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.AuthenticateAsServerAsync(SslServerAuthenticationOptions sslServerAuthenticationOptions, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.AuthenticateAsServerAsync(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
   at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client)
   at Halibut.Transport.SecureListener.HandleClient(TcpClient client)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Halibut.Transport.SecureListener.HandleClient(TcpClient client)
   at Halibut.Transport.SecureListener.<>c__DisplayClass20_0.<<Accept>b__2>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Halibut.Transport.SecureListener.<>c__DisplayClass20_0.<Accept>b__2()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
--- End of stack trace from previous location ---

Can we please have some help/assistance in getting this issue resolved? Am I missing something obvious? My team spent all day trying to triage with no luck. Please let me know if there’s any other information I can send you to make your investigation easier.

Thanks,
Mike

3

Hi Mike,
Thanks for reaching out and sorry to see you having issues with your polling tentacles.

First off if you haven’t already I would go through the Tentacle Troubleshooting

I would also suggest the use of the Tentacle ping tool Troubleshooting Tentacles - Octopus Deploy

Specifically on what you have done thus far, I would double check that the polling tentacle machine can reach the Octopus server on port 443, not just 10943. Its not often clear that both are needed.

I would also check for the presence of AntiVirus on both Tentacle and Octopus server as well as any proxy servers in the network chain. This is quite common as a blocker for our scripts.

You don’t mention if this is an HA setup but this can be an issue also.

As a last resort using Wireshark or similar network scanning tool can narrow down the specific traffic that is being blocked.

If you get any interesting results from the Troubleshooting do let us know if we can help out.

Kind Regards,
Paraic

4

Hi Mike,
Just to add to the previous, we have had several issues with load balancers/firewalls that are between tentacle and Octopus server and in many cases SSL offloading is occurring and intercepting the cert being presented via the tentacle connection.

Typically SSL offloading would need to be turned off in that case or a host header of some description coded into the traffic rules.

Just thought I’d mention that one as that error does match that use case.

Kind Regards,
Paraic

5

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.