Linux tentacle: OpenSSL error - ee key too small

[altyne]

Premise

Tentacle

• Linux Tentacle: 6.1.1391 (default port 10933)
• RedHat Enterprise Linux 8.4 (Ootpa)

Octopus Server

• Version: 2021.1 (Build 7236)
• Windows Server 2016 Data Center (10.0.14393)
• Thumbprint using SHA256RSA

Notes

Port 10933 is open, I can telnet.

Using Linux tentacle agent:

Error from Octopus Server log:

An error occured when sending a request to …, before the request could begin: Authentication failed because the remote part has closed the transport stream.
at System.Net.Security.SslStream.StartReadFrame(Byte buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)… An error occured when sending a request to …, before the request could begin: Authentication failed because the remote part has closed the transport stream.
at System.Net.Security.SslStream.StartReadFrame(Byte buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)…

This is error from agent : /var/log/secure

Interop+Openssl+SetSslCertificate(SafeSslContextHandle contextPtr, SafeX509Handler certPtr, SageEvpPKeyHandler keyPtr)…The error throw in : Interop+Openssl+SetSslCertificate(SafeSslContextHandle contextPtr, SafeX509Handler certPtr, SageEvpPKeyHandler keyPtr)…

I cannot acces https://my-machine:10993 in the browser too. But I have another agent in windows this type of URL is accessbile.

Using SSH:

from Octopus Sever log:

Could not connect to SSH endpoint. An existing connection was forcibly closed by remote host.Could not connect to SSH endpoint. An existing connection was forcibly closed by remote host.

[clare.martin]

Good morning @altyne,

Welcome to the Octopus Community and sorry to hear you are having issues with one of your Linux Tentacles.

Thank you for providing such detailed background information of your setup a it paints a good picture of what might be occurring here.

One thing I noticed off the bat is you mentioned you cant connect via https://my-machine:10993 in the browser. Is this a typo at all as you ideally need to connect to the tentacle via https://my-machine:10933 not 10993 as you suggested you tried?

Are you able to try connecting over https://my-machine:10933 to see if this works.

From your SSH logs it does look like something is blocking Octopus here from connecting to the Linux Tentacle. I imagine you have seen this already but we have some documentation on troubleshooting tentacles that may help?

Would you like to use SSH for the Linux target or would you like to use the Octopus Linux Tentacle manager?

We have some documentation on setting Octopus Linux targets to connect with SSH but its a bit more complicated than using the Tentacle manager so it would depend on your preference as how you need to set the tentacle up and configure firewalls etc.

Our recommended way of installing Octopus Tentacle on your Linux machines is through Tentacle manager and the documentation for this is here.

Are you using a polling tentacle or listening? If it is polling you want to use it is a different port but from your wording, it looks like you want to use a listening tentacle?

Your error logs suggest either;

• You have a networking issue somewhere that is blocking access from the Octopus Server to the Tentacle, you mentioned you can telnet in but it would be worth installing and using Tentacle Ping (you don’t need pong) and running that as that’s our ‘gold standard’ for connections. If you cant connect using that there is definitely a network issue.

• If you can connect through Tentacle ping, if you could make sure your Linux machine is actually listening on port 10933 (using the netstat command) and it’s only the Octopus Service that is listening on that port we can look into your logs a bit further but from what you have sent over so far this does look like a network issue I am afraid.

• There may be an anti-virus application blocking the script used to communicate initially from your Server to your tentacle so if you confirm networking is not the issue it might be worth temporarily disabling AV on your Linux machine to see if that is the cause of this issue.

I look forward to hearing if anything on that troubleshooting page helps you out, especially the Tentacle Ping and netstat commands.

Get in touch in the meantime if there is anything else we can help you with,

Kind Regards,

Clare Martin

[altyne]

@clare.martin Thanks for quick reply.

This is typo error

https://my-machine:10993
It should be https:my-machine:10933

Browser error : ERR_CONNECTION_CLOSED

More Information:
I configured my tentacle as ‘listening mode’ @ 10993 using ‘Tentacle’ as instance name and configured as service as well.

We have tentacle clients to other RHEL machines (OpenSSL 1.0.2k-fips 26 Jan 2017) as SSH targets but it works perfectly. Only this hardened by ‘Security’ team. They are not sharing what they did except - they applied some guidelines from RHEL security hardening checklist.

Ports 22 and 10933 are open. I check the process using

ss -tlunp

process   state  send-q  local address:port   peer Address:port
tcp       listen  128    *:10933               *:*

The current version : OpenSSL 1.1.1g FIPS 21 Apr 2020

I followed all the guides before raising this concern. But none of them help with my current issue,

I will try this one:

Tentacle Ping

I will let you know the result.

[altyne]

Another typo error :

I configured my tentacle as ‘listening mode’ @ 10993 using ‘Tentacle’ as instance name and configured as service as well.

Correct: I configured my tentacle as ‘listening mode’ @ 10933 using ‘Tentacle’ as instance name and configured as service as well.

The result of Tentacle ping:

Connect: Failed! 4ms; connected: True: SSL: True
Authentication failed because the remote party has closed the transport stream
..

What authentication is this using? My RHEL Linux tentacle is running under running root?

In addition, I stopped the linux tentacle but I get the same error above. But should be expecting:
Connect: Failed! Xms; connected: FALSE: SSL: True

[clare.martin]

Good morning @altyne,

I figured that was a typo error, thank you for letting us know the results of the Tentacle Ping, unfortunately, that is our ‘gold standard’ on network connections between the Octopus Server and the Tentacle so if that is failing it means you have a network issue somewhere and you will need to speak to your network team in order to rectify this.

The remote party has closed the transport stream error usually means traffic is getting blocked by something so either a firewall or load balancer. There is nothing Octopus Support can really do to help you here as your Networking team should be able to see where the connection is getting blocked. Using a program such as Wireshark usually helps our customers pinpoint the issue.

Just as a side note, what commands did you run when doing the tentacleping on your Octopus Server to your Linux Tentacle? I just found something interesting…

I have a Tentacle running on Ubuntu (firewall is disabled on both server and tentacle) and the tentacle passes health checks on Octopus Server.

If I use TentaclePing from the Octopus server to my Linux Tentacle and run the command shown in the screenshot I get that error message (same as you are seeing):

If I specify the TLS version but only use TLS I get the same message:

However, if I specify TLS1.2 I get a successful connection:

If you didn’t run the command with the TLS version are you able to do that just to make sure this is not the reason you are not seeing a success. If you look in our GitHub repository for the TentaclePing it will give you the commands you need to run:

TentaclePing <ip/hostname> [<port>] [<datasize>] [<chunksize>] [<ssl_protocol> {None|Ssl2|Ssl3|Tls|Tls11|Tls12|Default}]

Let me know the outcome of the new ping command, I would try them all so None|Ssl2|Ssl3|Tls|Tls11|Tls12|Default and see if you can connect with any of them, if you cannot it is as I mentioned above (a network issue).

Listening tentacles will run under the highest version of TLS installed on the OS so you need to try all the versions to see which one your Tentacle might use.

We have some documentation on hardening TLS so it might be worth looking at that and making sure your Linux box doesn’t have any TLS versions disabled or, if it does, which ones are disabled.

On another note to answer your question about disabling the Tentacle Service, I think the “Connect” part is referring to “I can hit that port successfully” rather than “there is a tentacle accepting my request”. So, seeing it showing as true when the tentacle service is stopped feels normal.

I look forward to hearing from you about the outcome of the ping with TLS commands,

Kind Regards,

Clare Martin

[altyne]

@clare.martin

I tried with different protocols (all of them) but still get the same error.

In Linux tentacle, do have scripts/commands to execute on how the tentacle know what TLS/SSL protocol or versions might pick?

I check RHEL OpenSSL supports (openssl ciphers -v):

• TLSv1.2
• TLSv1.3

[clare.martin]

Good morning @altyne,

Thanks for the update, you are in a pretty tough spot here I think.

It does look like there are network restrictions in place if you cannot connect at all with Tentacle ping on any of the TLS ports. The tentacle will use whatever TLS version is on the OS but it depends what OS release you have as to what the tentacle uses, regardless though you cannot connect on any of the TLS ports so this is a network issue, not a Tentacle one.

There isn’t much we can advise other than the server and tentacle need compatible TLS versions and ciphers, so, you need to work with your network and security team to figure out what those would be.

It’ll likely be a bunch of trial and error.

I think the best way forward for you to get you working as quickly as possible would be to spin up a clean, non-network restricted Linux box, get the tentacle working and then have security begin their hardening process but test the tentacle after each change and see what breaks it.

I am so sorry I cannot help you any further here, unfortunately, we don’t know your environment and what you have locked down so the only thing we can really advise is to start with a brand new unrestricted (from a networking point of view) box, make sure it works, then start restricting it step by step, testing each time to see if it still connects. You should be able to pinpoint what breaks it then.

I hope you get to the bottom of this soon and you are able to start deploying to Linux Tentacles.

If there is anything else we can help you with let us know,

Kind Regards,

Clare Martin