Migrate from version 2019 to upper : Unable to receive the remote identity; the identity line was empty

Hi,

We are using Octopus deploy self hosted in version 2019.13.7, and I’m trying to migrate to the last version.

We are using polling tentacles over websockets, and after the migration to a 2020 version or upper we got an error on each Tentacle: “Halibut.Transport.Protocol.ProtocolException: Unable to receive the remote identity; the identity line was empty.”.

The octopus server is hosted in a private network behind a reverse proxy that handle https certification.
When configuring a certificate in localhost and adding a tentacle on that same machine, I’m not getting that error.
I also tried to renew the certificate of the octopus server and re install all tentacles but without success.
I think there is something about the mismatch of the certificate exposed by the reverse proxy and the one configured inside octopus server bindings.

2021-12-23 14:29:57.3305   7752      1  INFO  ==== ServiceCommand ====
2021-12-23 14:29:57.3305   7752      1  INFO  CommandLine: C:\Program Files\Octopus Deploy\Tentacle\Tentacle.exe service --instance Tentacle --start
2021-12-23 14:29:57.9545   6528      1  INFO  ================================================================================
2021-12-23 14:29:57.9545   6528      1  INFO  Changed log folder from C:\windows\system32\config\systemprofile\AppData\Local\Octopus\Logs to D:\Octopus\Logs
2021-12-23 14:29:57.9545   6528      1  INFO  Tentacle.exe version 5.0.12 (5.0.12+Branch.master.Sha.c9d9f3853598cf83d0b569251728147b5fb79768) instance Tentacle
2021-12-23 14:29:57.9545   6528      1  INFO  Environment Information:
  OperatingSystem: Microsoft Windows XXXX
  OsBitVersion: x64
  Is64BitProcess: True
  CurrentUser: XXXX
  MachineName: FE2-XXXX
  ProcessorCount: 4
  CurrentDirectory: C:\windows\system32
  TempDirectory: C:\windows\TEMP\
  HostProcessName: Tentacle
  PID: 6528
2021-12-23 14:29:58.0190   6528      7  INFO  ==== RunAgentCommand ====
2021-12-23 14:29:58.0190   6528      7  INFO  CommandLine: C:\Program Files\Octopus Deploy\Tentacle\Tentacle.exe run --instance=Tentacle
2021-12-23 14:29:58.0190   7752      1  INFO  Waiting for service to become Running. Current status: StartPending
2021-12-23 14:29:58.1234   6528      7  INFO  Agent will trust Octopus Servers with the thumbprint: XXXX
2021-12-23 14:29:58.1440   6528      7  INFO  Agent will poll Octopus Server at wss://octopus.dev.xxxxx.com/OctopusComms for subscription poll://xxxxxxxxxxx/ expecting thumbprint XXXX
2021-12-23 14:29:58.1440   6528      7  INFO  Agent will not use a proxy server
2021-12-23 14:29:58.1440   6528      7  INFO  Agent will not listen on any TCP ports
2021-12-23 14:29:58.1440   6528      7  INFO  The Windows Service has started
2021-12-23 14:29:58.2006   6528      8  INFO  wss://octopus.dev.xxxxx.com/octopuscomms    8  Opening a new connection
2021-12-23 14:29:58.2797   6528      8  INFO  wss://octopus.dev.xxxxx.com/octopuscomms    8  Performing handshake
2021-12-23 14:29:58.2846   6528      8  INFO  wss://octopus.dev.xxxxx.com/octopuscomms    8  Secure connection established. Server at wss://octopus.dev.xxxxx.com/OctopusComms identified by thumbprint: XXXX
2021-12-23 14:29:58.3178   6528      8 ERROR  wss://octopus.dev.xxxxx.com/octopuscomms    8  Unexpected exception executing transaction.
Halibut.Transport.Protocol.ProtocolException: Unable to receive the remote identity; the identity line was empty.
   à Halibut.Transport.Protocol.MessageExchangeStream.ReadRemoteIdentity()
   à Halibut.Transport.Protocol.MessageExchangeStream.ExpectServerIdentity()
   à Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsSubscriber(Uri subscriptionId, Func`2 incomingRequestProcessor, Int32 maxAttempts)
   à Halibut.Transport.SecureWebSocketClient.ExecuteTransaction(Action`1 protocolHandler, CancellationToken cancellationToken)
2021-12-23 14:29:58.3295   6528      8 ERROR  wss://octopus.dev.xxxxx.com/octopuscomms    8  An error occurred when sending a request to 'wss://octopus.dev.xxxxx.com/OctopusComms', after the request began: Unable to receive the remote identity; the identity line was empty. Retrying in 7,5 seconds

Thanks,

Etienne

Hi @edebard,

Thanks for reaching out and welcome to the community. Sorry to see you’re having issues with your polling tentacles.

I think the easiest first step would be to verify your Thumbprints are matching on both the polling tentacle manager side and the Octopus server side.


Could you take a look and see if these are still what is expected? Looking forward to hearing back.

Regards,
Garrett

Hi @garrett.dass,

Thanks for the fast reply.

Here are some screens of my config



As I mentionned there is a mismatch between thumbrpint octopus server and my domain certificate thumbprint.

Here is details about certificate of my octopus domain (octopus.dev.xxxxxx.com) that I get when I navigate to htps://octopus.dev.xxxxxx.com/OctopusComms

This config was working on Octopus 2019.13.7

Hi @edebard,

Thank you for getting back to us.

It looks like you may be running into this bug: Polling tentacle fails to present client certificate following 2020.1.4 upgrade · Issue #6262 · OctopusDeploy/Issues · GitHub

Can you confirm what version of Octopus you have upgraded to?

Let us know at your earliest convenience.

Best Regards,
Donny

Hi,

I tried to upgrade to the last version available (2021.3).
I rolled back to version 2019 as I don’t want to block my team.
I will try to upgrade to version 2020.1.0 to see if I have the same problem when I will have more time.

When I was in version 2021 I tried to install a new Tentacle to replace old ones with this error but as I remember I think I had the same error.

Regards
Etienne

Hi @edebard,

Thanks for that information I just have a few more questions for clarification if you don’t mind.

  • Which version of 2020 did you attempt to upgrade to?
  • What happened with the 2021.3 upgrades, was it the same thumbprint issue?
  • Were you able to try the workarounds listed in the GH issue?
  • Would you be able to send a copy of an affected Tentacle and Server logs? I’ve created a secure upload link for you here.

As a side note, I would recommend 2021.2.8001 as an upgrade path instead of any of the 2020 builds.

Looking forward to hearing back.

Regards,
Garrett

Hi @edebard,

We’ve had another customer report with what looks like the same upgrade scenario and issue you are running into, would it be possible for us to get a System Integrity check run from your instance? You can locate the check here:

If that returns any errors, please let us know. Thank you!

Regards,
Garrett

1 Like

Hi @garrett.dass,

I sequentially tested versions 2020.1.0, 2020.1.1, 2020.1.2, 2020.1.3, 2020.1.4 and 2020.1.22. I had no issues. Then upgrading to version 2020.6, the error appeared again.

Here is the result of my health check in version 2020.6:

I will try to update tentacle certificates and install new ones to find any solution.

As a complement, here are logs from my tentacle and octopus server:

  • Tentacle:
2022-01-17 09:46:30.0946   8472      9  INFO  wss://xxxxxxxxxx.com/octopuscomms    9  Opening a new connection
2022-01-17 09:46:30.0946   8472      9  INFO  wss://xxxxxxxxxx.com/octopuscomms    9  Opening a new connection
2022-01-17 09:46:30.1402   8472      9 TRACE  wss://xxxxxxxxxx.com/octopuscomms    9  Connection established
2022-01-17 09:46:30.1402   8472      9 TRACE  wss://xxxxxxxxxx.com/octopuscomms    9  Connection established
2022-01-17 09:46:30.1402   8472      9  INFO  wss://xxxxxxxxxx.com/octopuscomms    9  Performing handshake
2022-01-17 09:46:30.1402   8472      9  INFO  wss://xxxxxxxxxx.com/octopuscomms    9  Performing handshake
2022-01-17 09:46:30.1432   8472      9  INFO  wss://xxxxxxxxxx.com/octopuscomms    9  Secure connection established. Server at wss://xxxxxxxxxx.com/OctopusComms identified by thumbprint: 466AC4D120B37D57C8F3DE4xxxxxxxx
2022-01-17 09:46:30.1432   8472      9  INFO  wss://xxxxxxxxxx.com/octopuscomms    9  Secure connection established. Server at wss://xxxxxxxxxx.com/OctopusComms identified by thumbprint: 466AC4D120B37D57C8F3DE4xxxxxxxx
2022-01-17 09:46:30.1432   8472      9 ERROR  wss://xxxxxxxxxx.com/octopuscomms    9  Unexpected exception executing transaction.
Halibut.Transport.Protocol.ProtocolException: Unable to receive the remote identity; the identity line was empty.
   à Halibut.Transport.Protocol.MessageExchangeStream.ReadRemoteIdentity()
   à Halibut.Transport.Protocol.MessageExchangeStream.ExpectServerIdentity()
   à Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsSubscriber(Uri subscriptionId, Func`2 incomingRequestProcessor, Int32 maxAttempts)
   à Halibut.Transport.SecureWebSocketClient.ExecuteTransaction(Action`1 protocolHandler, CancellationToken cancellationToken)
2022-01-17 09:46:30.1432   8472      9 ERROR  wss://xxxxxxxxxx.com/octopuscomms    9  Unexpected exception executing transaction.
Halibut.Transport.Protocol.ProtocolException: Unable to receive the remote identity; the identity line was empty.
   à Halibut.Transport.Protocol.MessageExchangeStream.ReadRemoteIdentity()
   à Halibut.Transport.Protocol.MessageExchangeStream.ExpectServerIdentity()
   à Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsSubscriber(Uri subscriptionId, Func`2 incomingRequestProcessor, Int32 maxAttempts)
   à Halibut.Transport.SecureWebSocketClient.ExecuteTransaction(Action`1 protocolHandler, CancellationToken cancellationToken)
2022-01-17 09:46:30.1432   8472      9 ERROR  wss://xxxxxxxxxx.com/octopuscomms    9  An error occurred when sending a request to 'wss://xxxxxxxxxx.com/OctopusComms', after the request began: Unable to receive the remote identity; the identity line was empty. Retrying in 29,5 seconds
2022-01-17 09:46:30.1432   8472      9 ERROR  wss://xxxxxxxxxx.com/octopuscomms    9  An error occurred when sending a request to 'wss://xxxxxxxxxx.com/OctopusComms', after the request began: Unable to receive the remote identity; the identity line was empty. Retrying in 29,5 seconds
  • Server:
2022-01-17 09:46:00.8868   9788      4  INFO  https://+:443/OctopusComms/       4  Accepted Web Socket client: xx.xx.xx.xx:40690
2022-01-17 09:46:00.8868   9788      4  INFO  https://+:443/OctopusComms/       4  Accepted Web Socket client: xx.xx.xx.xx:40690
2022-01-17 09:46:00.9190   9788      6  INFO  https://+:443/OctopusComms/       6  A client at xx.xx.xx.xx:40690 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:46:00.9190   9788      6  INFO  https://+:443/OctopusComms/       6  A client at xx.xx.xx.xx:40690 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:46:19.3527   9788     45  INFO  https://+:443/OctopusComms/      45  Accepted Web Socket client: xx.xx.xx.xx:40868
2022-01-17 09:46:19.3527   9788     45  INFO  https://+:443/OctopusComms/      45  Accepted Web Socket client: xx.xx.xx.xx:40868
2022-01-17 09:46:19.3751   9788     42  INFO  https://+:443/OctopusComms/      42  A client at xx.xx.xx.xx:40868 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:46:19.3751   9788     42  INFO  https://+:443/OctopusComms/      42  A client at xx.xx.xx.xx:40868 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:46:19.6374   9788     47  INFO  https://+:443/OctopusComms/      47  Accepted Web Socket client: xx.xx.xx.xx:40870
2022-01-17 09:46:19.6374   9788     47  INFO  https://+:443/OctopusComms/      47  Accepted Web Socket client: xx.xx.xx.xx:40870
2022-01-17 09:46:19.6421   9788     42  INFO  https://+:443/OctopusComms/      42  A client at xx.xx.xx.xx:40870 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:46:19.6421   9788     42  INFO  https://+:443/OctopusComms/      42  A client at xx.xx.xx.xx:40870 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:46:30.1362   9788      5  INFO  https://+:443/OctopusComms/       5  Accepted Web Socket client: xx.xx.xx.xx:41014
2022-01-17 09:46:30.1362   9788      5  INFO  https://+:443/OctopusComms/       5  Accepted Web Socket client: xx.xx.xx.xx:41014
2022-01-17 09:46:30.1585   9788     42  INFO  https://+:443/OctopusComms/      42  A client at xx.xx.xx.xx:41014 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:46:30.1585   9788     42  INFO  https://+:443/OctopusComms/      42  A client at xx.xx.xx.xx:41014 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:47:00.0128   9788      5  INFO  https://+:443/OctopusComms/       5  Accepted Web Socket client: xx.xx.xx.xx:41176
2022-01-17 09:47:00.0128   9788      5  INFO  https://+:443/OctopusComms/       5  Accepted Web Socket client: xx.xx.xx.xx:41176
2022-01-17 09:47:00.0462   9788     42  INFO  https://+:443/OctopusComms/      42  A client at xx.xx.xx.xx:41176 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:47:00.0462   9788     42  INFO  https://+:443/OctopusComms/      42  A client at xx.xx.xx.xx:41176 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:47:28.2999   9788     35  INFO  https://+:443/OctopusComms/      35  Accepted Web Socket client: xx.xx.xx.xx:41326
2022-01-17 09:47:28.2999   9788     35  INFO  https://+:443/OctopusComms/      35  Accepted Web Socket client: xx.xx.xx.xx:41326
2022-01-17 09:47:28.3227   9788     37  INFO  https://+:443/OctopusComms/      37  A client at xx.xx.xx.xx:41326 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:47:28.3227   9788     37  INFO  https://+:443/OctopusComms/      37  A client at xx.xx.xx.xx:41326 connected, and attempted a message exchange, but did not present a client certificate
2022-01-17 09:47:54.3758   9788      4  INFO  https://+:443/OctopusComms/       4  Accepted Web Socket client: xx.xx.xx.xx:41406
2022-01-17 09:47:54.3758   9788      4  INFO  https://+:443/OctopusComms/       4  Accepted Web Socket client: xx.xx.xx.xx:41406

I also uploaded log files to the secure link you shared me.

Regards
Etienne

[Update]
I tried to renew/update certificates of my existing tentacles and installing new tentacles, but the error is still the same.

[Update 2]
Working in version 2020.2.20 and not working in version 2020.3.10

Hi @edebard,

Thank you for the additional information!

I’ll pass this along to @garrett.dass and the team.

Out of curiosity, would you be able to test one of the 2021.2 versions we have available to see if the issue persists on there or if it has been resolved since your last test with a 2021.1 version?

It may be that this issue has been addressed since then, but I will pass along the results of your testing.

Kind Regards,
Adam