An error occurred when sending a request to 'https://172-29-X-X.core.cvent.org:10933/', before the request could begin: Unable to receive the remote identity

sarthak.chatterjee · 30 March 2022 08:41

Use case:

Tentacle is configured on my windows nodes and then the nodes are shut off for a few days. During the instance requirement the nodes are turned on and an API call is made to register the node as a deployment target in Octopus. I am facing this failure while doing the registration:

Performing TLS handshake
March 30th 2022 04:24:12Info
Secure connection established. Server at [::ffff:172.29.40.171]:10933 identified by thumbprint: A5AF3690E316FC6713585770A2A085F23751BAF7, using protocol Tls12
March 30th 2022 04:24:12Info
Identifying as a client
March 30th 2022 04:24:12Error
Connection initialization failed while connecting to https://172-29-40-171.core.cvent.org:10933/ Halibut.Transport.Protocol.ConnectionInitializationFailedException: Unable to receive the remote identity; the identity line was empty.
—> Halibut.Transport.Protocol.ProtocolException: Unable to receive the remote identity; the identity line was empty.
at Halibut.Transport.Protocol.MessageExchangeStream.ReadRemoteIdentity()
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectServerIdentity()
at Halibut.Transport.Protocol.MessageExchangeProtocol.PrepareExchangeAsClient()
— End of inner exception stack trace —
at Halibut.Transport.Protocol.MessageExchangeProtocol.PrepareExchangeAsClient()
at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsClient(RequestMessage request)
at Halibut.HalibutRuntime.<>c__DisplayClass41_0.b__0(MessageExchangeProtocol protocol)
at Halibut.Transport.SecureListeningClient.ExecuteTransaction(ExchangeAction protocolHandler, CancellationToken cancellationToken)

I tried following the workaround #2 mentioned in this article:

github.com/OctopusDeploy/Issues

Polling tentacle fails to present client certificate following 2020.1.4 upgrade

opened 06:32PM - 18 Mar 20 UTC

closed 06:41AM - 23 Mar 20 UTC

Justin-Walsh

kind/bug

# Prerequisites - [X] I have verified the problem exists in the latest versio…n - [X] I have searched [open](https://github.com/OctopusDeploy/Issues/issues) and [closed](https://github.com/OctopusDeploy/Issues/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aclosed) issues to make sure it isn't already reported - [X] I have written a descriptive issue title - [X] I have linked the original source of this report - [X] I have tagged the issue appropriately (area/*, kind/bug, tag/regression?) # The bug Following the cloud rollout of 2020.1.4, we've seen a small number of cases where tentacles fail to authenticate to the Octopus Server, with a `Halibut.Transport.Protocol.ProtocolException: Unable to receive the remote identity; the identity line was empty.` error. At this time, we believe this issue stems from older tentacle installs that would have generated a certificate with a weaker cipher, which is presenting issues when trying to negotiate a secure connection over modern encryption protocols (TLS). ## How do I know if I'll be affected? You may be affected by this bug if you use Octopus Cloud, and you use a polling tentacle that was created in the distant past (e.g. v3.0.25). ## Steps to reproduce TBD ## Related bugs We saw a similar issue affecting _listening_ tentacles in the same release, but with slightly different symptoms: #6266. ### Log exerpt #### Tentacle log ``` https://snip.octopus.app:10943/ 4 Unexpected exception executing transaction. Halibut.Transport.Protocol.ProtocolException: Unable to receive the remote identity; the identity line was empty. at Halibut.Transport.Protocol.MessageExchangeStream.ReadRemoteIdentity() in C:\buildAgent\work\fe2b45bbd4978f75\sourc e\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 169 at Halibut.Transport.Protocol.MessageExchangeStream.ExpectServerIdentity() in C:\buildAgent\work\fe2b45bbd4978f75\sou rce\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 242 at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsSubscriber(Uri subscriptionId, Func`2 incomingRequest Processor, Int32 maxAttempts) in C:\buildAgent\work\fe2b45bbd4978f75\source\Halibut\Transport\Protocol\MessageExchangePr otocol.cs:line 75 at Halibut.Transport.SecureClient.ExecuteTransaction(Action`1 protocolHandler) in C:\buildAgent\work\fe2b45bbd4978f75 \source\Halibut\Transport\SecureClient.cs:line 66 ``` #### Server log ``` listen://[::]:10943/ 82 A client at [::ffff:10.1.0.115]:52944 connected, and attempted a message exchange, but did not present a client certificate ``` ## Affected versions **Octopus Server: 2020.1.2 -> (Suspected)** ## Workarounds ### Workaround 1 (recreate tentacle) 1. Log into the Octopus server and note the target name, environments, and target roles. 2. Open tentacle manager on the impacted machine. 3. Take note of the "Home Directory" setting for your instance (C:\Octopus by default) 4. Delete the Tentacle instance via the Tentacle Manager: ![image](https://user-images.githubusercontent.com/49404281/76994464-cbf82d80-6924-11ea-97e4-954c20ad3aea.png) 5. Ensure that the `Tentacle.config` file has been removed from the home directory noted in step 2. 7. Recreate the tentacle instance with the previously noted name, environments, and target roles. Click the "Overwrite existing" checkbox on the registration page to prevent duplicate targets being created. ### Workaround 2 (new certificate) 1. Run `tentacle new-certificate` on the target (https://octopus.com/docs/octopus-rest-api/tentacle.exe-command-line/new-certificate 1) 2. Update the thumbprint for the target accordingly on the Server as shown: ![image](https://user-images.githubusercontent.com/1407488/77127331-c66f2480-6a97-11ea-84d5-1ecb557514c2.png) ### Workaround 3 (patch server) 1. Ensure that your servers running Tentacle are updated with the Windows Cumulative updates for March 2020 2. Reboot the servers running Tentacle ## Internal links ### Reported cases - https://secure.helpscout.net/conversation/1112186056/58987 (recreated polling tentacle) - https://secure.helpscout.net/conversation/1113171485/59057 (new cert for polling tentacle) - https://secure.helpscout.net/conversation/1113360699/59108 (recreated polling tentacle) ### Slack discussion - https://octopusdeploy.slack.com/archives/CUCG4LAF8/p1584465465243200

by running the commands:

C:\Program Files\Octopus Deploy\Tentacle>tentacle new-certificate
A new certificate has been generated and installed. Thumbprint:
FCEE0BABE421A80F30537FA46E5509022C3FEF7A
These changes require a restart of the Tentacle.

C:\Program Files\Octopus Deploy\Tentacle>tentacle service --restart
Restarting service OctopusDeploy Tentacle
Stopping service…
Waiting for service to become Stopped. Current status: StopPending
Waiting for service to become Stopped. Current status: Stopped
Service stopped
Waiting for service to become Running. Current status: StartPending
Waiting for service to become Running. Current status: Running
Service Started.

On retrying registration it still failed with this error:

An error occurred when sending a request to ‘https://172-29-40-171.core.cvent.org:10933/’, after the request began: The server at https://172-29-40-171.core.cvent.org:10933/ presented an unexpected security certificate. We expected the server to present a certificate with the thumbprint ‘A5AF3690E316FC6713585770A2A085F23751BAF7’. Instead, it presented a certificate with a thumbprint of ‘FCEE0BABE421A80F30537FA46E5509022C3FEF7A’ and subject ‘CN=Octopus Tentacle’. This usually happens when the client has been configured to expect the server to have the wrong certificate, or when the certificate on the server has been regenerated and the client has not been updated.

One weird observation:

After updating the thumbprint on the server, removing the deployment target and attempting registration again, i could notice that the thumbprint got updated to the current one but the underlined reference is still the old one.

Can someone please help?

clare.martin · 30 March 2022 09:40

Good morning @sarthak.chatterjee,

Welcome to the Octopus Community and sorry to hear you are having issues connecting one of your Tentacles.

I was just about to ask if you managed to change the thumbprint on the Tentacle in the Octopus UI but you have just confirmed you did. The thumbprint you have highlighted is the Octopus Server thumbprint which you put into the Tentacle install wizard when it asks you. I assume you have not changed that so as long as when you installed the Tentacle on your Deployment Target Server you made sure you put that Octopus Server Thumbprint in you should be fine.

One way to check this is through Tentacle Manager on the deployment target:

So your Tentacle manager should show the 44F0 thumbprint in there. If it does not you will need to re-install the Tentacle and input the correct thumbprint for the Octopus Server.

Did you manage to run a health check on the Tentacle now you changed the thumbprint? Does it pass now or is it still failing?

I look forward to hearing from you,

Kind Regards,

Clare Martin

sarthak.chatterjee · 30 March 2022 09:50

Hi @clare.martin !

Thanks for getting back.

This is the status on my node’s tentacle manager:

It shows the cert thumbprint : FCEE0BABE421A80F30537FA46E5509022C3FEF7A which is same as what i can see in the deployment target in octopus UI.

However the highlighted portion value is different
Octopus UI:

Tentacle manager:

I assume these values should be same for it to work? Isn’t there any other way than to delete and reconfigure tentacle ( which is not quite an option here as my tentacle will be configured a few days before than the actual registration always)?

clare.martin · 30 March 2022 10:24

Hi @sarthak.chatterjee,

Yes you are correct the Octopus Server thumbprint needs to be correct on the Tentacle for the Tentacle to be trusted, almost like a handshake, if it doesn’t have the correct Octopus Server thumbprint the connection will be refused by the Octopus Server.

The Octopus Server certificate never gets changed though unless you recently upgraded from a really old version of Octopus (where we were using certificates with sha1rsa) to a newer version of Octopus (where we are now using sha256). Or unless you regenerated it manually for some reason.

Are you able to just have a quick read of our Documentation surrounding this as it will help you understand what I mean here.

Can you confirm you have not upgraded your Octopus Instance recently from a really old version (I am talking 3 years old) to a newer version.

We do have a script you can run which will change the value of the Octopus Server Thumbprint on your tentacles which is shown in the documentation I linked above (only run the script in the picture below - do not follow the whole documentation as that will regenerate your Octopus Server Certificate, all we want to do is change the thumbprint the Tentacle has for the Octopus Server):

Replace ‘1234567890123456789012345678901234567890’ with the 44F0 thumbprint in your Octopus Server.

You will need to restart the Tentacle Service once you have run the command and then check the Tentacle Manager to ensure the Octopus Server certificate has been changed.

This is shown in this forum post where one user experienced the same issue as you with regards to the first error message you encountered.

Do you have any other Tentacles that are experiencing the same issue? Are you able to take a look at other deployment targets and make sure they pass their health checks and also have the same Octopus Server Thumbprints in the Tentacle Manager on that machine?

It seems a bit odd if only this one Tentacle has a different Octopus Server Thumbprint unless you have not switched this deployment target on for a while.

One other thing to note, the GitHub issue you mentioned in your first post is only for polling tentacles and not listening tentacles, from the port number of yours (10933) it seems you are using a listening Tentacle so the issue here was the Octopus Server thumbprint being incorrect in the Tentacle manager on the deployment target and not the Tentacle certificate thumbprint itself.

I only mention this because if you do have issues with your other Tentacles (and you notice the Octopus Server thumbprint is incorrect in Tentacle Manager) you will not have to update their certificate as you did, you would just need to run the tentacle.exe configure command to ensure the correct Octopus Sever certificate thumbprint is loaded into the Tentacle Manager.

I hope this helps but feel free to reach out if you have any other queries.

Once you have updated the Octopus Server thumbprint on that Tentacle you should not have to do that again even if it is switched off and on etc.

Kind Regards,

Clare Martin

system · 30 April 2022 13:06

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.