Unable to receive the remote identity; the identity line was empty

supplier.contracts · 29 September 2020 15:54

We’ve are using Octopus deploy self hosted, and today whilst adding a new deployment target I’ve come across an issue we haven’t seen before. No other tentacles are having or had this issue before.

The server is a newly built Ubuntu server LTS 20.04
The tentacle is installed up working when browsing to it’s internal url
Octopus Deploy OS Version Microsoft Windows 10.0.14393
CLR Version NET Core 3.1.7

When checking the Health of the tentacle inside of Octopus deploy I get this error:
(thumbprint and url deliberately XXXXX’d out by me)

Performing TLS handshake
September 29th 2020 16:07:06Info
Secure connection established. Server at [::ffff:xxx.xxx.xx.xx]:10933 identified by thumbprint: XXXXXXXXXXXXXXXXXXXXXXXXXXXX, using protocol Tls12
September 29th 2020 16:07:06Info
Identifying as a client
September 29th 2020 16:07:06Error
Connection initialization failed while connecting to https://XXXXXXXX:10933/ Halibut.Transport.Protocol.ConnectionInitializationFailedException: Unable to receive the remote identity; the identity line was empty.
—> Halibut.Transport.Protocol.ProtocolException: Unable to receive the remote identity; the identity line was empty.
at Halibut.Transport.Protocol.MessageExchangeStream.ReadRemoteIdentity()
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectServerIdentity()
at Halibut.Transport.Protocol.MessageExchangeProtocol.PrepareExchangeAsClient()
— End of inner exception stack trace —
at Halibut.Transport.Protocol.MessageExchangeProtocol.PrepareExchangeAsClient()
at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsClient(RequestMessage request)
at Halibut.HalibutRuntime.<>c__DisplayClass39_0.b__0(MessageExchangeProtocol protocol)
at Halibut.Transport.SecureListeningClient.ExecuteTransaction(Action`1 protocolHandler, CancellationToken cancellationToken)

jeremy.miller · 29 September 2020 17:50

Hi,

Welcome to the Octopus Forums!

Sorry to hear you’re having issues with your Linux tentacle.

Could you please check if you’re using a sha256 certificate?

To do this go to your server and click Configuration, then Thumbprint.

Please let me know.

Thanks,
Jeremy

supplier.contracts · 30 September 2020 08:21

Hi Jeremy

Thank you for your time.

This is what it says:
" The server certificate uses the sha1RSA algorithm."

So we’ve had Octopus for about 3 years now and never had an issue, and have always updated Octopus as LTS comes out. We normally use Windows server, but we also have a few Linux targets. One built just a few weeks ago worked fine. This one built yesterday is the first to have this issue.

Paul

paul.calvert · 30 September 2020 08:41

Hi Paul,

The certificate used by the Octopus Server to authenticate with Tentacles is generated during the initial install, it then won’t have been touched by any upgrades, as doing so would break the connection to all Tentacles.
So, in the last 3 years when we changed the certificate algorithm to use sha256, this would only apply to new installs of Octopus Server, or when a user manually regenerates the certificate.

In our experience it seems that Windows does still permit sha1 for new targets, however, we are seeing more and more Linux distros starting to block it. Which is what you’re now experiencing.

In order to resolve this issue, you will need to generate a new certificate for the Octopus Server (https://octopus.com/docs/octopus-rest-api/octopus.server.exe-command-line/new-certificate).
Once this certificate is generated, all currently connected listening tentacles will cease communicating with the Octopus Server.
To bring the tentacles back online you will need to update their trust with the new certificate thumbprint (https://octopus.com/docs/octopus-rest-api/tentacle.exe-command-line/update-trust).

A recent user came up with these steps to allow him to keep his current tentacles online during this process. It is a little more complex though.

export current server certificate ( Octopus.Server.exe export-certificate ) to .pfx file
generate and import new certificate ( Octopus.Server.exe new-certificate --export-pfx ) - this will break Tentacles’ trust
import back old certificate ( Octopus.Server.exe import-certificate ) to maintain already trusted Tentacles
execute Octopus Task for ALL Tentacles to add second, new, exported certificate’s thumbprint ( Tentacle.exe configure --trust ) - all Tentacles will trust both old and new certificate
import new, already generated certificate to Octopus Server ( Octopus.Server.exe import-certificate )

Regards,
Paul

supplier.contracts · 30 September 2020 09:35

Hi Paul

Sounds like we’ve got our work cut out there to replace those.

A couple of questions:

For generating a new Octopus instance where do I retrieve the current instance name that’s needed in the command line?

Where do I get the thumbprint of the new certificate? I’m assuming I need to make a note of the existing thumbprint to so one can be replaced by the other?

It’s a shame you don’t have something built in to do this directly into Octopus as an update button? Surely Octopus could do this, generate a new certificate, send that new thumbprint to all the listening tentacles, then disable the old Octopus certificate. Deployments is exactly what Octopus is made for.

Some clients are going to potentially have hundreds of tentacles that need updating manually, and at the moment Octopus deploy doesn’t have anything built in to deploy to those tentacles that is already knows about. Anything on the cards for that?

Paul

paul.calvert · 30 September 2020 09:59

The Octopus instance name will be present in the top right of the Octopus Manager or by running octopus.server list-instances

The thumbprint can be found by running octopus.server show-thumbprint

I imagine that this process hasn’t been added to Octopus as an automated process is because it isn’t something that would be run on a regular basis.

If following the steps that the other user listed, step 4 could be done using the Octopus script console (can be found in the Tasks page) and sent to all targets at once. However, once the trust is added the tentacle service needs to be started, which could be initiated using the same script console. It would likely return a failure as the connection to the tentacle would be cut during the restart of the service, but the service would still restart.

supplier.contracts · 30 September 2020 10:13

Thanks Paul

It wouldn’t be done often but it’s a pretty big breaking change for us. We have numerous tentacles and growing, in several different environments, some of which we don’t control. e.g potentially clients servers. Therefore to do these steps manually on the individual servers would be quite a substantial task for us.

The OS’s are only just catching up, like the change with the latest Ubuntu server just made, and I see this as a problem that’s going to suddenly escalate in the coming months as other users of Octopus start having this issue with new targets and OS upgrades.

I think Octopus Deploy should be doing more here to help their customers, and not relying on a post from another user to help them.

Could we have a more details example from support as to how to do step 4 via the Octopus script console in the meantime?

Paul

jeremy.miller · 30 September 2020 13:09

Hi Paul,

We actually have a more detailed guide with examples here in our documentation: https://github.com/OctopusDeploy/docs/blob/enh-cert-rotation/docs/how-to/how-to-regenerate-certificates-with-octopus-server-and-tentacle.md

Can you please take a look and let me know if that is sufficiently detailed or if you need more information?

Thanks,
Jeremy

supplier.contracts · 30 September 2020 13:57

Thank you Jeremy,

On the final step below, do we also have to do this in reverse on the Tentacle and update it in Octopus too as that wasn’t mentioned previously? I thought it was the Octopus cert that was being replaced, not the one on the Tentacle?

“update the Thumbprint for the updated target.” Where is that Thumbprint obtained from?

C:\Program Files\OctopusDeploy\Tentacle\Tentacle.exe new-certificate C:\Program Files\OctopusDeploy\Tentacle\Tentacle.exe service --restart

After this is generated on the Tentacle, it can be updated on the Octopus server. Navigate to {{Infrastructure,Deployment Targets}} and update the Thumbprint for the updated target.

jeremy.miller · 30 September 2020 14:54

Hi Paul,

That final section is if you need to create a new tentacle certificate, which you shouldn’t need to do in this case. The instructions are just both in the same documentation because of their similar subject matter. You should only need to follow the first section.

Please let me know if that makes sense or if you have any other questions. And please let me know when you’re in a good state afterward.

Thanks,
Jeremy

supplier.contracts · 5 October 2020 14:18

Well it now seems that the instructions on the link that you provided have now been deleted, and gives a 404 error.

Has the page been moved or renamed?

Paul

jeremy.miller · 5 October 2020 14:31

Hey Paul,

Sorry about that. It’s in the midst of getting added to permanent documentation so any links I send you will be temporarily functional.

I’ll copy paste the contents of the current document here(which should be final). For anyone finding this in the future via searching, please go to our documentation to find the up to date most current instructions.

Please let me know if you get it going or need more help.

Thanks,
Jeremy

INSTRUCTIONS:

Octopus uses self-signed certificates to securely communicate between Tentacles and the Octopus Server. Prior to Octopus 3.14, the certificates were SHA1, and following the shattered exploit, the certificate generation was upgraded to SHA256.

This guide walks through the process of regenerating certificates to use the new algorithm. This is useful for upgrading from SHA1 to SHA256, and if you want to rotate certificates.

For more information on why Octopus uses self-signed certificates, please see the blog post Why Octopus uses self-signed certificates.

You can view the algorithm used by the Server certificate on the {{Configuration,Thumbprint}} page. If the algorithm contains sha1 , we recommend regenerating your certificate.

:::warning Updating an existing Octopus Server or Tentacle It’s important to consider the impact of updating an existing Octopus Server or Tentacle as changes are required to ensure each component trusts the other. If there is a mismatch between the certificate and the expected thumbprint, communication between the components will not be possible and must be resolved manually. Read the information below carefully. :::

Configuring Octopus Server to use a new certificate {#ConfiguringOctopusServerToUseANewCertificate}

At a high level, changing the certificate on an Octopus Server involves the following steps:

Backup your existing certificate.
Generate a new certificate to a file.
Make the Tentacles trust the new certificate.
Replace the certificate on the Octopus Server.
Remove the old trusted certificate from the Tentacles.

:::hint At present, this process is more manual than we would prefer, and we are aiming to improve this process over time. :::

Backup your existing certificate by executing the following statement at an elevated command-line on the server, from the directory where Octopus Deploy is installed ( C:\Program Files\Octopus Deploy\Octopus by default):

Octopus.Server.exe export-certificate --instance OctopusServer --export-pfx=“C:\PathToCertificate\oldcert.pfx” --pfx-password MySecretPassword

./Octopus.Server export-certificate --instance OctopusServer --export-pfx="/tmp/oldcert.pfx" --pfx-password MySecretPassword

This will display output similar to the following:

Checking the Octopus Master Key has been configured.
...
Exporting certificate...
The certificate has been written to C:\PathToCertificate\oldcert.pfx.

Save this certificate and the specified password somewhere secure.

:::hint If you see a warning message about The X509 certificate CN=Octopus Portal was loaded but the private key was not loaded. , you are most likely not running with elevated permissions. :::

Execute the following statement at a command-line on the same server:

Octopus.Server.exe new-certificate --instance OctopusServer --export-pfx=“C:\PathToCertificate\newcert.pfx” --pfx-password MySecretPassword

./Octopus.Server.exe new-certificate --instance OctopusServer --export-pfx="/tmp/newcert.pfx" --pfx-password MySecretPassword

This will display output similar to the following:

Checking the Octopus Master Key has been configured.
...
Generating certificate...
The Octopus Server currently uses a certificate with thumbprint:
1111111111111111111111111111111111111111
A new certificate has been generated with thumbprint:
1234567890123456789012345678901234567890
The new certificate has been written to C:\PathToCertificate\newcert.pfx.

Take a note of the thumbprint of the new certificate ( 1234567890123456789012345678901234567890 in the output above). We will use this thumbprint when we update the Tentacles to trust the new certificate.

The next step is to update all of the Tentacles to trust the new certificate. At present, this functionality is not exposed in the UI; it has to be done via the command-line.

On each Tentacle machine, execute the following command to trust the thumbprint of the newly-created certificate in the directory that the Tentacle agent is installed ( C:\Program Files\OctopusDeploy\Tentacle\ by default):

Tentacle.exe configure --trust=“1234567890123456789012345678901234567890”

./Tentacle configure --trust=“1234567890123456789012345678901234567890”

This will display output similar to the following:

Adding 1 trusted Octopus Servers
These changes require a restart of the Tentacle.

Now that the Tentacles all trust the new certificate, we can update the Octopus Server certificate to the new one we created earlier. In the command prompt on the Octopus Server run:

Octopus.Server.exe import-certificate --instance OctopusServer --from-file=“C:\PathToCertificate\newcert.pfx” --pfx-password MySecretPassword Octopus.Server.exe service --instance OctopusServer --restart

./Octopus.Server import-certificate --instance OctopusServer --from-file="/tmp/newcert.pfx" --pfx-password MySecretPassword ./Octopus.Server service --instance OctopusServer --restart

This will display something like the following:

Importing the certificate stored in PFX file in C:\PathToCertificate\newcert.pfx using the provided password...
Checking the Octopus Master Key has been configured.
...
The certificate CN=Octopus Portal was updated; old thumbprint = 1111111111111111111111111111111111111111, new thumbprint = 1234567890123456789012345678901234567890
Certificate imported successfully.
These changes require a restart of the Octopus Server.

Run a healthcheck on the associated Tentacles and confirm they are all healthy.
Now we are trusting the new certificate, we can now stop the Tentacles trusting the old certificate. On each of the Tentacle machines run:

C:\Program Files\OctopusDeploy\Tentacle\Tentacle.exe configure --instance Tentacle --remove-trust C:\Program Files\OctopusDeploy\Tentacle\Tentacle.exe service --instance Tentacle --restart

./Tentacle configure --instance Tentacle --remove-trust ./Tentacle service --instance Tentacle --restart

Run a healthcheck, and confirm all Tentacles are healthy.
Confirm on the {{Configuration,Thumbprint}} page that the new certificate is using the sha256 algorithm.

Configuring a Tentacle to use a new certificate {#ConfiguringATentacleToUseANewCertificate}

To update the certificate that is used by a Tentacle, run the following commands on the Tentacle machine:

C:\Program Files\OctopusDeploy\Tentacle\Tentacle.exe new-certificate C:\Program Files\OctopusDeploy\Tentacle\Tentacle.exe service --restart

./Tentacle new-certificate ./Tentacle service --restart

After this is generated on the Tentacle, it can be updated on the Octopus Server. Navigate to {{Infrastructure,Deployment Targets}} and update the Thumbprint for the updated target.

system · 5 November 2020 14:31

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.