Linux Octopus Tentacle not installing

Octopus Server
1

On a Linux server, Error unpacking rpm package tentacle-6.1.668-1.x86_64. We found reference in your help, telling us to uninstall and try with gzip and the base installer. We now can see a fatal error in the log file. I have uploaded the log file to my support login. Any recommendations on how to resolve this?

1 INFO ==== NewCertificateCommand ====
2021-05-28 14:59:08.0724 900980 1 INFO CommandLine: /opt/octopus/tentacle/Tentacle.dll new-certificate --instance Tentacle --if-blank
2021-05-28 14:59:08.7219 900980 1 ERROR ===============================================================================
2021-05-28 14:59:08.8308 900980 1 FATAL error:23076072:PKCS12 routines:PKCS12_parse:parse error

3

Hi @cmuesing ,

Thanks for getting in touch with us over your question, and sorry you’re having issues installing a tentacle.

We should be able to better troubleshoot once we know a few more details, so could you provide us with the Linux distro you’re using, as well as the process you’re following to attempt setting up the tentacle?

Also, is this a new tentacle you’re trying to configure, or are you adding a tentacle to an existing instance?

Looking forward to hearing back from you.

Best,
Patrick

5

Good morning Patrick.

This is a new tentacle we are trying to configure on a Splunk Linux server running RHEL 8.4. However, we have attempted it several times.

Our System Admin has tried installing the tentacle-6.1.690-1.x86_64.rpm and tentacle-6.1.670-linux_x64.tar.gz

He was able to finally get things unpacked using the following:

tar xvzf tentacle-linux_x64.tar.gz -C /opt/octopus

The error occurs when he attempt to configure the tentacle after the install.

/opt/octopus/tentacle/configure-tentacle.sh

Name of Tentacle instance (default Tentacle):

Invalid characters will be ignored, the instance name will be: ‘Tentacle’

What kind of Tentacle would you like to configure: 1) Listening or 2) Polling (default 1):

Where would you like Tentacle to store log files? (/etc/octopus):

Where would you like Tentacle to install applications to? (/home/Octopus/Applications):

Enter the port that this Tentacle will listen on (10933):

Enter the thumbprint of the Octopus Server: ######

The following configuration commands will be run to configure Tentacle:

sudo /opt/octopus/tentacle/Tentacle create-instance --instance “Tentacle” --config “/etc/octopus/Tentacle/tentacle-Tentacle.config”

sudo /opt/octopus/tentacle/Tentacle new-certificate --instance “Tentacle” --if-blank

sudo /opt/octopus/tentacle/Tentacle configure --instance “Tentacle” --app “/home/Octopus/Applications” --port 10933 --noListen False --reset-trust

sudo /opt/octopus/tentacle/Tentacle configure --instance “Tentacle” --trust #####

sudo /opt/octopus/tentacle/Tentacle service --install --start --instance “Tentacle”

Press enter to continue…

Configuration file at /etc/octopus/Tentacle/tentacle-Tentacle.config already exists.

Setting home directory to: /etc/octopus/Tentacle

Saving instance: Tentacle

6

Hi @cmuesing ,

Thank you for getting back to us and detailing your steps as others may find them helpful. Just to confirm, your system administrator was able to get the tentacle installed successfully with the above method?

I can confirm that the above worked fine for me as well using RHEL 8.4, though unfortunately, I wasn’t able to reproduce the original error.

Best,
Patrick

7

Unfortunately not. Our System Admin got the tentacle unpacked, but got an error on a certificate. Now it is telling us that the tentacle already exists, but when trying to add it to Octopus infrastructure, it gets a connection refused. We have attempted to uninstall using the instructions in your documentation.

/opt/octopus/tentacle/Tentacle service –stop –uninstall and delete-instance, but both commands tell us there is no Tentacle installed or running. I tried sudo /opt/octopus/tentacle/Tentacle delete-instance and got a message that it deleted the instance: Tentacle.

Then trying to sudo /opt/octopus/tentacle/configure-tentacle.sh takes me through the wizard again, with the same results.

image001.jpg
image001.jpg1565×443 76.3 KB

The log looks very similar to the screen display.

image002.jpg
image002.jpg1356×598 144 KB

8

Hi @cmuesing,

As a best practice I imagine you have already run any yum updates especially for any Security updates though it seems unlikely to be the issue.

As a precaution, can you re-download the rpm and install it, but first run the uninstall from this link:

To verify the requirements, can you run the openssl commands and copy it here?

openssl version; openssl ciphers

Also can you check if SELinux is running?

cat /etc/selinux/config; getenforce

If the correct latest openssl version is installed we can look at debugging the scripts that run the Tentacle installation. If you can enlist the aid of you sysadmin again and ask them to run the following at this step:

strace -f -o /tmp/outfile sudo /opt/octopus/tentacle/configure-tentacle.sh

…and attach the /tmp/outfile it creates, that would be very helpful.

Kind Regards,
Paraic

9

​​​​​ We uninstalled after gathering the below data for you. Then yum installed the tentacle with this failure.

yum install tentacle
Updating Subscription Management repositories.
Last metadata expiration check: 2:04:18 ago on Wed 02 Jun 2021 07:51:43 AM MDT.
Dependencies resolved.

image002.jpg
image002.jpg1007×260 44.6 KB

10

Hi @cmuesing ,

Thanks for getting back to us with the results of the yum installation method.

Sorry to press on the other results, but were you able to get an output from the other suggestions above (e.g. OpenSSL details, SELinux, strace)? With those, we should be able to further troubleshoot this issue.

Best,
Patrick

11

I pasted the results inside the other email, but it looks like they got deleted in the response. Here you go.

To verify the requirements, can you run the openssl commands and copy it here?

`openssl version; openssl ciphers```

OpenSSL 1.1.1g FIPS 21 Apr 2020

TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-CCM:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-CCM:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-CCM:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-CCM:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:PSK-AES256-GCM-SHA384:PSK-AES128-GCM-SHA256:PSK-AES256-CBC-SHA:PSK-AES128-CBC-SHA256:PSK-AES128-CBC-SHA:DHE-PSK-AES256-GCM-SHA384:DHE-PSK-AES256-CCM:DHE-PSK-AES128-GCM-SHA256:DHE-PSK-AES128-CCM:DHE-PSK-AES256-CBC-SHA:DHE-PSK-AES128-CBC-SHA256:DHE-PSK-AES128-CBC-SHA:ECDHE-PSK-AES256-CBC-SHA:ECDHE-PSK-AES128-CBC-SHA256:ECDHE-PSK-AES128-CBC-SHA

Also can you check if SELinux is running?

`cat /etc/selinux/config; getenforce```

This file controls the state of SELinux on the system.

SELINUX= can take one of these three values:

enforcing - SELinux security policy is enforced.

permissive - SELinux prints warnings instead of enforcing.

disabled - No SELinux policy is loaded.

SELINUX=enforcing

SELINUXTYPE= can take one of these three values:

targeted - Targeted processes are protected,

minimum - Modification of targeted policy. Only selected processes are protected.

mls - Multi Level Security protection.

SELINUXTYPE=targeted

root@ots-corp-sploga.open-techs.local:/home/cmuesing$ getenforce

Enforcing

If the correct latest openssl version is installed we can look at debugging the scripts that run the Tentacle installation. If you can enlist the aid of you sysadmin again and ask them to run the following at this step:

strace -f -o /tmp/outfile sudo /opt/octopus/tentacle/configure-tentacle.sh

…and attach the /tmp/outfile it creates, that would be very helpful.

strace -f -o /tmp/outfile sudo /opt/octopus/tentacle/configure-tentacle.sh

strace: decode_nlattr: [xlat 0x55d8fb187f40, dflt “AF_???”, decoders 0x7fff66215550] size is zero (going to pass nla_type as decoder argument), but opaque data (0x7fff66215600) is not - will be ignored

strace: decode_nlattr: [xlat 0x55d8fb187f40, dflt “AF_???”, decoders 0x7fff66215550] size is zero (going to pass nla_type as decoder argument), but opaque data (0x7fff66215600) is not - will be ignored

12

Hi @cmuesing
Thanks for supplying the additional info. I feel we are making progress.

For the openssl version, it is very recent and I don’t believe we have done testing on this exact version. To help with compatibility you can install the compat-openssl toolkit and see if it helps with the install:

yum -y install compat-openssl10

I would also like to rule out the SELinux factor if I can. The easiest way to do this would be to run the setenforce command and set it to permissive, so its still recording but not enforcing:

setenforce Permissive

(This does not need a reboot as it will only survive the current login session.)

Prior to retrying the install I would remove the /etc/octopus and /opt/octopus directories entirely and then re-run the installation.
If it still fails then it might be an issue with SELinux.

To check if SELinux is enforcing a block on the Tentacle install, I would run the following:

yum install -y setroubleshoot-server setools

(these may be installed already)
Once installed they will help parse the audit logs to extract any errors. You can then run the command to search for errors:

sealert -a /var/log/audit/audit.log

This will print out current errors and you can check if any reference Octopus Tentacle in the output.
It will also recommend actions to allow the actions to go ahead but I would check with your Sysadmin on this part.

in the meantime if I can ask for the outfile you ran previously with the strace command that would help us diagnose further. Please attach the whole file as an attachment if you can.

Thanks for helping us work through this - we are keen to fix the problem and get you up and running.

Kind Regards,
Paraic

13

Paraic,

This is from my sys admin. The logs are attached to this email.

OctopusTentacle.txt (8.03 KB)

strace output.txt (4.65 KB)

strace.log (901 Bytes)

14

Hi @cmuesing
Updating this ticket with some more info.
I have setup a few reproductions on clean Centos and RHEL 8.4 systems with openssl 1.1.1g and I have been unable to reproduce the error.

Can you confirm that you installed the compat lib for backwards compatibility:

yum -y install compat-openssl10

This seems to be a solution for some folks who have a similar problem. Let me know if you can try this.

Kind Regards
Paraic

16

We have done the compat and it was already installed. That is not solving our issue.​​​​​
Cindy Muesing | Senior DevOps Engineer

17

Hi Cindy
Thanks for reporting back on that.

It does appear to be related to the openssl configuration as the ciphers provided by openssl ciphers in your case differs from the default Redhat 8 install that I have setup on my VMs.

Could you run this command and send me the output when you get a chance - this will show if there is a different crypto policy from the default on your VM? (this can be run as a non-root user)

update-crypto-policies --show

If the output is other than DEFAULT, it may explain the incompatibility, specifically around SHA-1.
Also if you sysadmin has made any hardening changes around ciphers, please do let us know.

Kind Regards,
Paraic

20

It is FIPS, not Default and yes, our Sys Admins have changed the ciphers for hardening.

What do we need to re-enable?

21

Hi Cindy
Thanks for getting that last piece of the puzzle.

It looks like the crypto settings are the issue. As a workaround you can reset the FIPS mode back to default. To do this run as root:

fips-mode-setup --disable

You will need to reboot to fully apply the settings. This will reset the crypto settings back to DEFAULT. I have successfully reproduced this in our test labs.

The install should complete then. You did say that your admins had applied some other cipher hardening - that may still be an issue if you run into the same problem when re-installing. You will need to ask them to revert those changes. Though its not 100% certain, its likely that the SHA-1 ciphers are needed for the Tentacle install. I will need to confirm this with our Dev team and get back to you.

Let me know how you get on.

Kind Regards,
Paraic

22

Per my Sys Admin:

I was able to get the tentacle installed and configured once we disabled FIPS, but it broke as soon as I turned it back on. The non-Splunk machines have their Octopus tentacles working just fine with FIPS enabled. We need FIPS enabled per security policy.

It sounds to me that this is a SHA Cipher issue. Can we get a list of the Octopus tentacle ciphers in the preferred order from support?

23

Hi Cindy,
Thanks for the updates.

Investigating the source of our preferred cipher list, it appears that it originates from our .NET for Linux/Windows installation and therefore is not something we can control directly. I am testing a workaround for using FIPS after Tentacle is installed to see if we can make it work. Officially we don’t yet test extensively against RHEL 8 so our support is thin on that front for now.

Can you let me know which Splunk product you are installing on the Linux clients? I can try and replicate that issue also. If the Splunk install creates a log file I would like to see what errors they are showing if you can send them along also.

Again thanks for your patience on this issue. Hopefully we can get a resolution for you soon.

Kind Regards,
Paraic

24

Hi Cindy
My colleague put together the comparison chart between the ciphers we normally see on RHEL 8 and the ones you have under FIPS. This shows the AES ciphers that we most likely are using for encrypting our connections.

Hope this helps if you are trying to add in ciphers.

Kind Regards
Paraic
CipherComparison.pdf (42.5 KB)

25

Paraic,

Thank you for the cipher comparison. That lead us the resolution. We had a consultant assist us with the installation of Splunk 8.1 and apparently have an unusual and reduced configuration of the ciphers.

I appreciate you sticking with me on this issue. It was a difficult one to figure out.