Can't add new tentacle client to the server

[gregoire.pascal]

Hi for some reason ,

We are building a new environment based on 2019 servers.
Our octopus server is running 2016.
Everything looks fine regarding network, the Server can ping the client and Telnet the port 10933.
We’ve copied the tentacle config file from a working environment.
We also have already working environment working using 2019 server.
But here every time we try to add the new target in octopus we have this error:

Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host… Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host… An existing connection was forcibly closed by the remote host.

And we have this on the tentacle logs:

2023-07-05 16:09:16.7395 7508 124 INFO listen://[::]:10933/ 124 Accepted TCP client: [::ffff:@IP_server]:53218
2023-07-05 16:09:16.7395 7508 119 ERROR listen://[::]:10933/ 119 Socket IO exception: [::ffff:@IP_server]:53217
System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.BeginReceive(Byte buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
at System.Net.Sockets.NetworkStream.BeginRead(Byte buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)

We’ve spend the day trying to fix it but we can’t find any reason why.
Also interesting fact we can’t open the tentacle URL from the server
https://@IP_client:10933

But this work locally on the client server.

[clare.martin]

Good morning @gregoire.pascal,

Thank you for contacting Octopus Support and sorry to hear you are struggling to connect a tentacle to your Octopus Server.

Thank you for providing us with so much detail from your logs and how you have set this tentacle up as it really helps us understand the background of the issue.

I will say I think the issue here stems from the fact you mentioned you copied over a configuration file from a working tentacle. If you did that I assume you copied the whole thing including that other tentacles thumbprint? Tentacle thumbprints have to be unique otherwise you will get conflicts in Octopus and network issues like you are seeing.

Its one of the reasons we do not recommend (or support) a tentacle install being put on a ‘gold’ or ‘master’ OS base image as when you build new servers based on that OS image it copies all the tentacle configs onto other machines and conflicts happen which then result in the kind of issues you are seeing here.

I recommend you manually uninstall the tentacle on that machine following this guide here and re-install it from scratch, if you want to copy some of the config file over from another machine you can but do not touch the Tentacle.CertificateThumbprint in the config file.

You can see below an example of a tentacle worker config file and you can see the tentacle certificate thumbprint which has to be unique per machine.

The port number also needs to be unique on the machine itself between tentacles - ie if you have two tentacles on a machine they cannot use the same port number.

  <set key="Tentacle.CertificateThumbprint">63467B4CB43234A1456DF1018A94A18649408413</set>
  <set key="Tentacle.Communication.TrustedOctopusServers">[{"Thumbprint":"17297940A90EDB2F5FF7ABB1ECBFB70392648A56","CommunicationStyle":1,"Address":null,"Squid":null,"SubscriptionId":null}]</set>
  <set key="Tentacle.Deployment.ApplicationDirectory">C:\Octopus\Applications\Main Computer Tentacle Worker</set>
  <set key="Tentacle.Services.NoListen">false</set>
  <set key="Tentacle.Services.PortNumber">10956</set>

If you were looking to automate your tentacle installs we have a guide here on utilising DSC to achieve that.

Let me know if the manual uninstall and re-install works for you, also let me know if I am on the wrong track and you did not copy the tentacle thumbprint over from the other config file.

Kind Regards,
Clare

[gregoire.pascal]

Hi,

Thanks for the answer, i just reinstall a new server on the same environment and installed only DotNet 4.8 and tentacle 6.4 still the same issue. (i haven’t copy the config file)
I did some Wireshark.
When i do a telnet i have this:
server > SYN > client
Client > SYN ACK > Server
Server > ACK > Client
When i try to contact the tentacle
server > SYN > client
Client > SYN ACK > Server
Server > ACK > Client
Server > RST > Client

I have no idea why octopus send a reset packet.

[clare.martin]

Hey @gregoire.pascal,

Thanks for the quick reply, I assume you have checked out our Tentacle Troubleshooting page as you mentioned you tried to connect to it via https://@IP_client:10933 and this does not work.

I know you can telnet in through 10933 but telnet uses different protocols to how Octopus communicates with the tentacles so I would try our Tentacle Ping tool as that is the gold standard for network connections and will tell you if you have a connection to your Octopus Server from your tentacle.

Let me know the outcomes of the tentacle ping commands and we can go from there.

Kind Regards,
Clare

[gregoire.pascal]

Didn’t know about this tool.
This is the result:

PS C:\Users\administrator.CEE\Desktop\TentaclePing.1.1.0> .\TentaclePing.exe @IP_Client
Using SSL Protocol: Tls
Pinging @IP_Client on port 10933
2023-07-06T09:38:17 Connect: Failed! 128ms; connected: True; SSL: True
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by
the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte buffer, Int32 offset, Int32 count)
at System.Net.Security.SslState.StartReceiveBlob(Byte buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncReq
uest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte buffer, AsyncProtocolRequest asyncRe
quest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at TentaclePing.Program.SendRequest(String hostname, Int32 port, SslProtocols sslProtocol, Int32& bytesRead, Boolean&
connected, Boolean& sslEstablished, String data)
at TentaclePing.Program.ExecutePing(String hostname, Int32 port, Int32 dataSize, SslProtocols sslProtocol)

[clare.martin]

Hey @gregoire.pascal,

Yea that tool is really useful for figuring out network issues such as this!

If the tentacle ping tool is failing to connect then unfortunately you have a networking issue here which is preventing your Octopus tentacles to connect to the Octopus Server.

This is what you should see:

We do have some documents on hardening Octopus with regards to deployment targets which does mention networking but the main culprits are usually things such as:

• Internal and / or external firewalls not allowing ports 443 (for the initial tentacle registration) and port 10933 - TCP if you require a protocol in your firewall (default tentacle listening port) through, you can read more about this here.

• Proxy Server on your network not routing the tentacle packets properly.

• The machine itself is not listening on port 10933 - this should not be the case as you can telnet using that port but check by running netstat -a in a command prompt and it will show you all the ports that are listening.

• Or something else such as a load balancer on your network which is routing the tentacle requests incorrectly.

One thing to look at is this document here regarding SSL protocols. It may be the case that you have disabled TLS 1.2 on your tentacle boxes and that is the protocol Octopus Server uses to communicate with the tentacles. So can you make sure your machines are using TLS 1.2 by following that document and using a tools like IIS Crypto to see what protocols you have available on the tentacles.

Other than that I am afraid we cant be much more help here and you will need to get your networking team to see where the requests are being blocked, it seems you have access to wireshark so you should see the requests from the Tentacle ping tool in there and hopefully that will show you where its failing or being blocked as the tentacle does communicate using TCP not whatever protocol Telnet uses so its not a like for like if you can Telnet in or ping the servers.

Hopefully you can get this working but let me know if you are still struggling, we can only advise on networking issues as we don’t know your network and are not authorised to tell you how to set it up other than what I have mentioned here so I hope what I have gone though helps.

Kind Regards,
Clare

[gregoire.pascal]

Hi thanks.

I did a wireshark on both server and i’m missing a packet on the client.
The TLS Hello packet never arrive on client.
So i suppose this is the problem.
Let’s hope we find the guilty network device.

[clare.martin]

Hey @gregoire.pascal,

Great news you have managed to hopefully narrow this down, let me know if you get stuck and I will do my best to help.

Kind Regards,
Clare