Polling linux tentacle don't use proxy during register-with command

henrik.von.gunten · February 5, 2021, 5:27am

Hello, I’m trying to set up a linux polling tentacle which needs to use a proxy for outbound communication to octopus server, but the register-with command fails due to communication timeout (towards Octopus Cloud server), even though I’ve configured the instance with the proxy command.

Any advice how to proceed on this one? Is there a way to enable more debug logging to check if proxy is being used? There is no trace in proxy server log of the tentacle client attempting to use the proxy.

The tentacle config do contain proxy host info:

  <set key="Octopus.Proxy.ProxyHost">bf-proxy********.net</set>
  <set key="Octopus.Proxy.ProxyPort">8888</set>
  <set key="Octopus.Proxy.UseDefaultProxy">false</set>

tentacle register-with error:

Checking connectivity on the server communications port 10943...
{0} failed with message {1). Retrying ({2}/{3}) in {4}. (Checking that server communications are open,The operation has timed out.,1,5,00:00:00.7500000) => Input string was not in a correct format.
System.Net.WebException: The operation has timed out.
   at System.Net.HttpWebRequest.GetResponse()
   at Octopus.Tentacle.Communications.OctopusServerChecker.<>c__DisplayClass2_0.<CheckServerCommunicationsIsOpen>b__1() in /opt/buildagent/work/f090214449efab33/source/Octopus.Tentacle/Communications/OctopusServerChecker.cs:line 48
   at Octopus.Tentacle.Communications.OctopusServerChecker.Retry(String actionDescription, Action action, Int32 retryCount, TimeSpan initialDelay, Double backOffFactor) in /opt/buildagent/work/f090214449efab33/source/Octopus.Tentacle/Communications/OctopusServerChecker.cs:line 85

tina.bamford · February 3, 2021, 6:35pm

Hi Henrik,
Thanks for getting in touch. I’m sorry to hear you’re running into this connectivity issue setting up a Linux polling tentacle. I hope you don’t mind if I request a little more information to track down what might be the source of the problem here. If possible, can you send over the following:

Can the machine access the proxy server outside of the setup script?
Please send over the full script that includes the “register-with” command?
If you telnet to bf-proxy on 8888, does it work?

Looking forward to your reply.
Kind Regards,
Tina

henrik.von.gunten · February 4, 2021, 2:24pm

Hi,

we’ve had some progress since yesterday, and the proxy can now be used after using the polling-proxy command. We also had to open up on port 443 (in addition to 10943) on the proxy towards the octopus server.

Now we’re reaching the octopus server, but there seem to be a certificate issue:
Checking connectivity on the server communications port 10943…
Connected successfully
Registering the tentacle with the server at https://*****.octopus.app/
Detected automation environment: NoneOrUnknown
The following certificate errors were encountered when establishing the HTTPS connection to the server: RemoteCertificateChainErrors
Certificate subject name: CN=.octopus.app
Certificate thumbprint: 7F8897DDF3E4AB07B7F2E027EA7D233E99A3FD1A
The following certificate errors were encountered when establishing the HTTPS connection to the server: RemoteCertificateChainErrors
Certificate subject name: CN=.octopus.app
Certificate thumbprint: 7F8897DDF3E4AB07B7F2E027EA7D233E99A3FD1A
The following certificate errors were encountered when establishing the HTTPS connection to the server: RemoteCertificateChainErrors
Certificate subject name: CN=*.octopus.app

Is it required to install some certificates on either Octopus Server or on the host where the tentacle client is running? If so, where can we find some documentation how to configure?

tina.bamford · February 4, 2021, 7:50pm

Hey Henrik,
That’s good news. Glad to hear it!
Is it at all possible that you’re running into Sectigo Issue?

I hope this helps.

Kind Regards,
Tina

henrik.von.gunten · February 8, 2021, 3:05pm

I’ve read the Sectigo Issue, and if I understand this correctly we might need to install Sectigo new root CA certificate on the host where we want to install the tentacle?

There is no problem using curl to fetch the frontpage of the octopus server on port 443, so the host machine seem to be installed with CA certificates with valid chain for the https://****.octopus.app:443/ server.

henrik.von.gunten · February 8, 2021, 3:15pm

This post do seem to report an identical issue: Unable to register a polling tentacle with Octopus Cloud

How was this resolved?

Justin_Walsh · February 8, 2021, 4:06pm

Hi Henrik,

Just jumping in for Tina here, since she’s out of the office today. In that case, we needed to reprovision the instance to include the new certificate with the updated intermediate certificate. As all instances get the new certificate, this, sadly, won’t fix it on your end.

If you’re running an older distro, it is possible that you might need to update the certs. Depending on how old of a distro you’re running, you may be able to get away with updating the ca-certificates package on your tentacle machine.

Some other thoughts here:

When you mention that you can curl the octopus server correctly, is that going through the same proxy as the Tentacle configuration script is configured to use, or direct?
Is your proxy (or any other network device) doing SSL termination/re-encoding or any other scenarios where it is interrupting the SSL chain?

Look forward to hearing from you soon!

henrik.von.gunten · February 8, 2021, 4:16pm

Hi,

thanks for the update. The machine is running Red Hat Enterprise Linux Server release 7.9, so hardly an old distro.

Right now, we’re not using the proxy, but connecting directly from the tentacle server, as we tried to rule-out if proxy was the cause of the issue. As of now, there is no other network device doing SSL termination/re-encoding.

Justin_Walsh · February 8, 2021, 6:52pm

Hrrm, very strange indeed!

Are you able to post up the output of the following command (substituting samples.octopus.app with your instance name):

openssl s_client -host samples.octopus.app -port 443

Should give you a long output containing the SSL connection negotiation - perhaps that can shed some light on what’s occurring there.

henrik.von.gunten · February 8, 2021, 7:15pm

Here’s the output:

openssl.txt (5.1 KB)

Justin_Walsh · February 8, 2021, 7:30pm

Looks like the certificate matches what’s expected - I just spun up a repro on my end with a fresh Centos 7.9 and was able to connect to my cloud instance without any problems, so I’m a little puzzled here as well. Here’s my command stack to install tentacle:

yum install krb5-libs libicu openssl-libs wget
wget https://rpm.octopus.com/tentacle.repo -O /etc/yum.repos.d/tentacle.repo
yum install tentacle
/opt/octopus/tentacle/configure-tentacle.sh

Which worked exactly as expected:

The following configuration commands will be run to configure Tentacle:
sudo /opt/octopus/tentacle/Tentacle create-instance --instance "Tentacle" --config "/etc/octopus/Tentacle/tentacle-Tentacle.config"
sudo /opt/octopus/tentacle/Tentacle new-certificate --instance "Tentacle" --if-blank
sudo /opt/octopus/tentacle/Tentacle configure --instance "Tentacle" --app "/home/Octopus/Applications" --noListen "True" --reset-trust
sudo /opt/octopus/tentacle/Tentacle register-with --instance "Tentacle" --server "https://octopus-yyy.octopus.app" --name "JW-Centos7-Repro" --comms-style "TentacleActive" --server-comms-port "10943" --apiKey "API-XXXXXXXXXXXXXXXXXXXXXXXXXX" --space "Justin" --environment "Dev"  --role "Linux"
sudo /opt/octopus/tentacle/Tentacle service --install --start --instance "Tentacle"
Press enter to continue...
Saving instance: Tentacle
Setting home directory to: /etc/octopus/Tentacle
A certificate already exists, no changes will be applied.
Removing all trusted Octopus Servers...
Application directory set to: /home/Octopus/Applications
Tentacle will not listen on a port
These changes require a restart of the Tentacle.
Checking connectivity on the server communications port 10943...
Connected successfully
Registering the tentacle with the server at https://yyy.octopus.app/
Detected automation environment: NoneOrUnknown
Machine registered successfully
These changes require a restart of the Tentacle.
Service installed: Tentacle
Service started: Tentacle

I’m at a bit of a loss here - do these install instructions line up with what you did on your end?

henrik.von.gunten · February 9, 2021, 8:19am

Hi, yes, the installation instructions are the same. I’m also having no issues on a fresh CentOS 7 instance.

I think we’ll have to follow this up with our hosting provider.

Thanks for your help troubleshooting this.

henrik.von.gunten · February 10, 2021, 8:37am

Hi again,

we were not able to figure out why we get the RemoteCertificateChainErrors error, but after trying to work around the failing “register-with” command, by updating the tentacle configuration using these commands:

sudo /opt/octopus/tentacle/Tentacle configure --trust THUMBPRINT --instance instanceName
sudo /opt/octopus/tentacle/Tentacle server-comms --instance instanceName --host *****.octopus.app --port 10943 --style TentacleActive

And installing the tentacle service and starting it, it seem like the tentacle is attempting to connect to Octopus server, but now we get the following error in log:

2021-02-09 16:19:26.0487 89953 1 INFO CommandLine: /opt/octopus/tentacle/Tentacle.dll run --instance=instanceName --noninteractive
2021-02-09 16:19:26.1399 89953 1 INFO Agent will trust Octopus Servers with the thumbprint: 7189F8CFD99E53BF8237DC8A61DF128F2ACF8C42
2021-02-09 16:19:26.1447 89953 1 INFO Agent will poll Octopus Server at https://example.octopus.app:10943/ for subscription poll://gcfxyh7wq4rhk5c65n6b/ expecting thumbprint 7189F8CFD99E53BF8237DC8A61DF128F2ACF8C42
2021-02-09 16:19:26.2051 89953 1 INFO Agent will not listen on any TCP ports
2021-02-09 16:19:26.2718 89953 6 INFO https://example.octopus.app:10943/ 6 Opening a new connection to https://example.octopus.app:10943/
2021-02-09 16:19:26.4240 89953 6 INFO https://example.octopus.app:10943/ 6 Secure connection established. Server at [::ffff:52.178.72.181]:10943 identified by thumbprint: 7189F8CFD99E53BF8237DC8A61DF128F2ACF8C42, using protocol Tls12
2021-02-09 16:19:26.4728 89953 6 ERROR https://example.octopus.app:10943/ 6 Unexpected exception executing transaction.
Halibut.Transport.Protocol.ProtocolException: Unable to receive the remote identity; the identity line was empty.
at Halibut.Transport.Protocol.MessageExchangeStream.ReadRemoteIdentity()
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectServerIdentity()
at Halibut.Transport.Protocol.MessageExchangeStream.IdentifyAsSubscriber(String subscriptionId)
at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsSubscriber(Uri subscriptionId, Func2 incomingRequestProcessor, Int32 maxAttempts) at Halibut.Transport.PollingClient.<ExecutePollingLoop>b__12_0(MessageExchangeProtocol protocol) at Halibut.Transport.SecureClient.ExecuteTransaction(Action1 protocolHandler, CancellationToken cancellationToken)

Is there perhaps something else happening during “register-with” which is now lacking?

paul.calvert · February 10, 2021, 8:45am

Hi Henrik,

We have sometimes seen that error appear when the Octopus Server is using the older SHA1 algorithm for the thumbprint rather than SHA256.

Within the Octopus UI, if you navigate to Configuration > Thumbprint does it state SHA1 or SHA256?
e.g.

If it is using a SHA1, then you will need to regenerate the certificate being used, however, this will break the trust with all tentacles and therefore needs to be planned accordingly.

We have a guide that details the suggested steps to update this and all tentacles with as little manual work as possible.

Regards,
Paul

henrik.von.gunten · February 10, 2021, 8:52am

The Octopus server: “The server certificate uses the sha256RSA algorithm.”

Is it even possible to re-generate Octopus server certificate on the cloud version?

paul.calvert · February 10, 2021, 8:59am

Apologies, I had missed that this was a Cloud instance. The server certificate is fine then.

After re-reading your last message, when you say

work around the failing “register-with” command

Does that mean that the register-with command hasn’t been run at all?

If so, this will likely mean that the Octopus Server doesn’t recognise the tentacle and is refusing the connection.

henrik.von.gunten · February 10, 2021, 9:09am

Does that mean that the register-with command hasn’t been run at all?

Yes, the “register-with” command was not able to complete, with the following error looping:

Checking connectivity on the server communications port 10943…
Connected successfully
Registering the tentacle with the server at https://*****.octopus.app/
Detected automation environment: NoneOrUnknown
The following certificate errors were encountered when establishing the HTTPS connection to the server: RemoteCertificateChainErrors
Certificate subject name: CN=.octopus.app
Certificate thumbprint: 7F8897DDF3E4AB07B7F2E027EA7D233E99A3FD1A
The following certificate errors were encountered when establishing the HTTPS connection to the server: RemoteCertificateChainErrors

henrik.von.gunten · February 10, 2021, 9:22am

What is a bit strange is that the thumbprint seem to belong on certificate on port 443, not the one on port 10943.

Do the register-with command require connecting to both port 443 and 10943?

$ openssl s_client -servername ****.octopus.app -connect ***.octopus.app:10943 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin
SHA1 Fingerprint=71:89:F8:CF:D9:9E:53:BF:82:37:DC:8A:61:DF:12:8F:2A:CF:8C:42

$ openssl s_client -servername ****.octopus.app -connect ****.octopus.app:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin
SHA1 Fingerprint=7F:88:97:DD:F3:E4:AB:07:B7:F2:E0:27:EA:7D:23:3E:99:A3:FD:1A

paul.calvert · February 10, 2021, 9:28am

That’s correct, the register-with command connects to the Octopus Server on 443 and uses the REST API to configure the new tentacle on the Octopus Server. During this process, it will also test connectivity on port 10943. Once this command completes, all further communication between the tentacle and server will take place on port 10943.

henrik.von.gunten · February 10, 2021, 9:30am

Also noticing that when viewing the certificate on port 10943, the SHA-256 thumbprint seem to be different than the one listed in Octopus Server:

In Octopus Server (seem to list the SHA-1 fingerprint):