Health check failing for one server

Legacy Uncategorised
1

Hi

I have added a listening server as a deployment target. I can add the server successfully, but the health check immediately fails afterwards. This is only affecting one server, every other one is fine.

-Running Octopus 3.0.24.0
-I can browse to https://servername:10933 from another machine and get the successful configuration message (only thing I haven’t tried is locally from the Octopus server as I don’t have local logon rights).
-I have tried both Tentacle 3.0.17 and 3.0.21 (clean uninstall between each one, confirming files/folders/reg keys were removed)
-Restarted the Tentacle service/server multiple times
-Tried different ports, confirmed that the tentacle is listening on that port, and nothing else is. There are no firewalls between the servers, and the Windows firewall is off on the target.
-Deleted and re-added to Octopus several times

Logfile output is below:

2015-10-27 08:51:00.7866 8 INFO listen://0.0.0.0:10933/ 8 Accepted TCP client: 10.92.64.78:51480
2015-10-27 08:51:00.7866 61 INFO listen://0.0.0.0:10933/ 61 Performing SSL (TLS 1.0) server handshake
2015-10-27 08:51:13.8909 61 INFO listen://0.0.0.0:10933/ 61 Secure connection established, client is not yet authenticated
2015-10-27 08:51:13.8909 61 INFO listen://0.0.0.0:10933/ 61 Client at 10.92.64.78:51480 authenticated as 1AE49B652089E7AE1507C3A166411C44AC613EF3
2015-10-27 09:01:14.4751 61 INFO listen://0.0.0.0:10933/ 61 Unhandled error when handling request from client: 10.92.64.78:51480
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadLine()
at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 94
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 65
at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func2 incomingRequestProcessor) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 110 at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsServer(Func2 incomingRequestProcessor, Func`2 pendingRequests) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 90
at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 122

Any advice is appreciated.

Thanks

Tom

2

Hi Tom,

Thanks for getting in touch and going to some length to diagnose the issue yourself. My best guess is that there is some intermediary network firewall or HTTP proxy or something doing SSL offloading between the Octopus Server and Tentacle. The best way to diagnose this, as you mentioned, is to log on to the Octopus Server and try the HTTPS test.

My other suggestions would be to:

  1. Investigate the network and make sure the right firewall burns are in place, also noting we do not support HTTP proxies for the Octopus Server to Tentacle communications protocol.
  2. Try configuring a Polling Tentacle and see if you have the same or similar issue.

Our recommendation is to try and get a Listening Tentacle working because it offers the best of class performance and behaviour.

Further reading in case it helps:

Hope that helps.
Mike

3

Hi Tom,

Whoops, I just realised you said there are no firewalls involved, so much for the easy response! I do think it would be worth going over the suggestions I made though.

Taking a closer look the error message it appears the handshake has started, and the Octopus Server’s client certificate has been inspected. This indicates there is network connectivity between the two machines. It would still be good if you could confirm that by looking through the troubleshooting guide I linked.

I’ll investigate more on my side too, but would appreciate if you can confirm a few things for me:

  1. Is the Octopus Server Certificate Thumbprint: 1AE49B652089E7AE1507C3A166411C44AC613EF3?
  2. Is the Octopus Server IP Address: 10.92.64.78?

Thanks!
Mike

4

Hi Mike

Thanks for the reply. Yes I wish it was something easy like a firewall issue!

Firstly, the Octopus server thumbprint and IP are correct. Also, that there are no errors under “Connectivity” for the target in Octopus. The results appear the same as a successful health test:

Opening a new connection

2015-10-28 09:17:39

Info
Connection established

2015-10-28 09:17:39

Info
Performing SSL (TLS 1.0) handshake

2015-10-28 09:17:51

Info
Secure connection established. Server at 10.14.130.73:10933 identified by thumbprint: 22D548A7C979EDA2F3016DA13D8835F5E71C9A22

2015-10-28 09:17:51

Info
Identifying as a client

2015-10-28 09:18:03

Info
Sent: IScriptService::StartScript[1] / 9b4d7bfd-0ac7-4e47-8a47-95eae6746539

2015-10-28 09:18:03

Info
Received: Halibut.Transport.Protocol.ResponseMessage

Setting up as a Polling server:
I deleted the target in Octopus, and created a new tentacle instance to poll the Octopus server. It was set up successfully, and outputted the below to the tentacle log. The deployment target appeared in Octopus, but was still offline.

I was able to browse to the Octopus server url from the target server.

Tentacle log:

2015-10-28 09:07:28.6155 5 INFO Agent will trust Octopus servers with the thumbprint: 1AE49B652089E7AE1507C3A166411C44AC613EF3
2015-10-28 09:07:28.7091 5 INFO Agent will poll Octopus server at https://webappoctopus.asbbank.co.nz:10943/ for subscription poll://4l8gdhegwbq3w5dv8hps/ expecting thumbprint 1AE49B652089E7AE1507C3A166411C44AC613EF3
2015-10-28 09:07:28.7247 5 INFO Agent will not listen on any TCP ports
2015-10-28 09:07:28.7247 5 INFO The Windows Service has started
2015-10-28 09:07:28.7247 5 INFO Tentacle version: 3.0.4 / 9f326327/refs/heads/master
2015-10-28 09:07:28.7559 7 INFO https://webappoctopus.asbbank.co.nz:10943/ 7 Opening a new connection
2015-10-28 09:07:28.7871 7 INFO https://webappoctopus.asbbank.co.nz:10943/ 7 Performing SSL (TLS 1.0) handshake
2015-10-28 09:07:49.3960 7 INFO https://webappoctopus.asbbank.co.nz:10943/ 7 Secure connection established. Server at 10.92.64.78:10943 identified by thumbprint: 1AE49B652089E7AE1507C3A166411C44AC613EF3

Under Connectivity in Octopus, it logged:

2015-10-28 09:06:27

Info
Request IScriptService::StartScript[1] / 90740732-f072-4a3c-8e41-66cb374094cd was queued

2015-10-28 09:07:50

Info
Request IScriptService::StartScript[1] / 90740732-f072-4a3c-8e41-66cb374094cd was collected by the polling endpoint

I tried everything else under the link you provided and it all looks OK:
-Tentacle service is running as Local System
-Both IP and hostname tried as target
-I can connect locally to the tentacle
-Only one IP address configured on the server
-Unchecked “Check for publisher’s certificate revocation”

The only thing I need to check is browsing to the server url from the Octopus server - will update once I have done this.

Thanks

Tom

5

Hi Tom,

Thanks for being so tenacious on your end.

Am I correct in saying that you have other Tentacles running just fine? If so I wonder what is different about this particular server? At this point the one thing that comes to mind is the potential of a particular cryptography library or cipher suite being disabled, thinking TLS 1.0 etc. Do you think that could be the case?

In Octopus 3.1/Tentacle 3.1 we moved to .NET 4.5 so you can use TLS 1.2, but any versions of Octopus/Tentacle prior to 3.1 only require .NET 4.0 and only use TLS 1.0.

Otherwise I’ll wait to hear back from you.
Mike

6

Hi Mike

Yes we have multiple other servers with tentacles that run without issue.

Checking with IIS Crypto, TLS 1.0 is enabled. And all of the same ciphers are enabled as a known-working machine. I’m not an expert in this area - is there anything else to check?

We are upgrading to Octopus 3.1 over the weekend, so I will try tentacle 3.1 with/without TLS 1.2 to see if this makes any difference next week.

Thanks

Tom

7

Hi

Octopus server is now running 3.1.4, I installed Tentacle 3.1.5 and the same issue occurs.

Thanks

8

Hi Tom,

Thanks for getting back to me. You’re reaching the end of my experience getting Tentacle working when the environment isn’t cooperating.

I’ve gone over the logs you’ve posted and like you say, it appears this Tentacle is working. Can you confirm a couple more details with me:

  1. When you run a health check, does it succeed or fail? Can you send through the logs.
  2. When the health check completes, is the machine seen as being offline? (We cache the result of Health Checks in the SQL Database in the Machine table)
  3. Can you run an ad-hoc script against the Machine (under Tasks -> Script Console)?

Could you attach the raw logs as files for anything that may help, I’m happy to read through them in detail.

If I can’t get to the bottom of the problem we can book an interactive support session.

One final question: Since you’ve got other Tentacles running successfully, it seems like there is something different about this server, potentially driven by group policy or a security policy. Perhaps there is someone else in the organisation who can let you know what the differences are?

Hope that helps.
Mike

9

Hi

This ended up resolving itself today once we had deployed to the server. The health check now runs and completes successfully, and the server shows as online.

We had never attempted to deploy to the server previously as it was at the end of a long lifecycle, so this may have been a much quicker resolution had it been tried earlier.

Thanks

10

Hi Tom,

Thanks for keeping me in the loop, and I’m glad the problem has been resolved.

Happy Deployments!
Mike