Issue connecting to 2nd Tentacle on server

Octopus Server
, , , ,
1

image
image1269×252 11.2 KB

We set up a second listening tentacle on Octopus and on the Client Server. We tried this a number of times with different ports (default tentacle is on the default 10933 port - we tried 10934, 10935, 10944).
The tentacle tentacle started correctly each time and was visible as a service, but not visible from the Octopus Infrastructure Target setup. We tried connecting automatically and by entering details manually. We also tried as a target and as a worker (worker is what we want).
All the ports we mentioned have been opened inbound on the client server.
Not able to connect in either case. The default tentacle for that server can connect fine and healthcheck is good.

This is the report from the healthcheck :
Opening a new connection to https://awsinteg01:10944/
February 23rd 2022 08:11:29Info
Opening a new connection to https://awsinteg01:10944/
February 23rd 2022 08:11:41Error
The client was unable to establish the initial connection within 00:01:00. Retrying in 1.0 seconds.
February 23rd 2022 08:11:42Info
Retrying connection to https://awsinteg01:10944/ - attempt #1.
February 23rd 2022 08:11:42Info
Opening a new connection to https://awsinteg01:10944/
February 23rd 2022 08:11:50Error
The client was unable to establish the initial connection within 00:01:00. Retrying in 1.0 seconds.
February 23rd 2022 08:11:51Info
Retrying connection to https://awsinteg01:10944/ - attempt #1.
February 23rd 2022 08:11:51Info
Opening a new connection to https://awsinteg01:10944/
February 23rd 2022 08:12:03Error
The client was unable to establish the initial connection within 00:01:00. Retrying in 1.0 seconds.
February 23rd 2022 08:12:04Info
Retrying connection to https://awsinteg01:10944/ - attempt #2.
February 23rd 2022 08:12:04Info
Opening a new connection to https://awsinteg01:10944/
February 23rd 2022 08:12:12Error
The client was unable to establish the initial connection within 00:01:00. Retrying in 1.0 seconds.
February 23rd 2022 08:12:13Info
Retrying connection to https://awsinteg01:10944/ - attempt #2.
February 23rd 2022 08:12:13Info
Opening a new connection to https://awsinteg01:10944/
February 23rd 2022 08:12:25Error
The client was unable to establish the initial connection within 00:01:00. Retrying in 1.0 seconds.
February 23rd 2022 08:12:26Info
Retrying connection to https://awsinteg01:10944/ - attempt #3.
February 23rd 2022 08:12:26Info
Opening a new connection to https://awsinteg01:10944/
February 23rd 2022 08:12:34Error
The client was unable to establish the initial connection within 00:01:00. Retrying in 1.0 seconds.
February 23rd 2022 08:12:35Info
Retrying connection to https://awsinteg01:10944/ - attempt #3.
February 23rd 2022 08:12:35Info
Opening a new connection to https://awsinteg01:10944/
February 23rd 2022 08:12:47Error
The client was unable to establish the initial connection within 00:01:00. Retrying in 1.0 seconds.
February 23rd 2022 08:12:48Info
Retrying connection to https://awsinteg01:10944/ - attempt #4.
February 23rd 2022 08:12:48Info
Opening a new connection to https://awsinteg01:10944/
February 23rd 2022 08:12:56Error
The client was unable to establish the initial connection within 00:01:00. Retrying in 1.0 seconds.
February 23rd 2022 08:12:57Info
Retrying connection to https://awsinteg01:10944/ - attempt #4.
February 23rd 2022 08:12:57Info
Opening a new connection to https://awsinteg01:10944/
February 23rd 2022 08:13:09Error
The client was unable to establish the initial connection within 00:01:00. Retrying in 1.0 seconds.
February 23rd 2022 08:13:18Error
The client was unable to establish the initial connection within 00:01:00. Retrying in 1.0 seconds.
February 23rd 2022 08:14:40Info
Opening a new connection to https://awsinteg01:10944/
February 23rd 2022 08:15:01Error
The client was unable to establish the initial connection within 00:01:00. Retrying in 1.0 seconds.

3

Good morning @dmccann,

Welcome to the Octopus forums! Thank you for getting in touch and sorry you are having issues with one of your tentacles.

If you have not seen these yet we have some documentation on setting up multiple tentacle instances (on the same machine). It does look like you have taken port numbers into consideration here, ie given the second Tentacle a different port number to the first so you should have no issues there.

When you set up your multi-tentacle instance as a worker inside Octopus are you making sure you are using the correct thumbprint for the second tentacle, if so are you able to try connecting to it using its IP address and not hostname and see if that makes a difference?

We do have a handy troubleshooting tentacles guide which a lot of our customers use if you haven’t come across it yet, are you able to try and perform some of the actions listed in that guide and see if any help?

I look forward to hearing from you,

Kind Regards,

Clare Martin

4

Hi Clare,
Thanks for getting back so quickly. We’ve installed multiple tentacles on various servers - we’re currently running 7 on one server.
We’ve tried different ports on this one, and ensured the firewall was open for each. We tried the IP address too, plus the original, default tentacle is still working and reachable when we run a healthcheck on it from Octopus.

Thanks,

David

5

Hi @dmccann,

Thank you for confirming that info, are you able to upload the raw logs from the failed health check please and also the server and tentacle logs for the client, I can have a look through them and see if there is anything I can spot.

I have provided a secure link where you can upload the files, are you able to let me know when those files have been uploaded as we don’t get notifications when customers upload files.

I look forward to hearing from you,

Kind Regards,

Clare Martin

6

Thanks Clare - I’ve sent those files via the secure transfer.

Thanks,

David

7

Hey David,

Thanks for sending over those logs, I have had a quick look but unfortunately, they don’t provide much information other than what you have already posted:

The client was unable to establish the initial connection within 00:01:00

06:59:47   Info     |     Offline:
06:59:47   Info     |     - [Tentacle_AWS_SSIS](~/app#/Spaces-1/infrastructure/machines/Machines-705/settings) at https://awsinteg01:10944/, error: An error occurred when sending a request to 'https://awsinteg01:10944/', before the request could...
06:59:47   Fatal    |     One or more machines were not available. Please see the output log for details.

I am really sorry to ask you this but have you managed to restart that tentacle instance service since you changed it to the new port of 10944 to see if that made any difference?

Have you checked that the Octopus Server thumbprint shown in light gray in the Tentacle manager matches the one shown in the Configuration ➜ Thumbprints screen in the Octopus Web Portal.

Can you also check the tentacle thumbprint matches that of the one in the Octopus UI for that tentacle.

Can you connect to your tentacle via the web browser (details located here).

Finally if all those are correct and you can access the browser are you able to set up a tentacle ping and let me know if it allows you to connect to the Octopus Server.

I realise you have many tentacle instances set up so this is not your first time installing one, but I wanted to get the basic troubleshooting out of the way so we know that the ‘easy’ ways to solve tentacle connection issues have been completed before we move onto the ‘harder’ options!

Let me know how your testing goes, I will ask some of our other senior support staff and see if they have seen anything like this before, since your other tentacle on that machine is connecting to the server we know it is not a bug in your version of Octopus so its either the Tentacle install itself or some environmental issue that’s occurring here.

Kind Regards,

Clare Martin

8

Hi Clare - thanks again.
The test on the browser of the client worked fine. The TentaclePing and Pong part I can’t easily do, as the client machine is a PROD system and we need a CAB to install software.
However, there is an existing tentacle on that client server that has a working connection to Octopus and the healthcheck on it is fine. Also deployments are working from it.

9

Hi @dmccann,

Thanks for getting back to me, I did see you mentioned you have a tentacle on there already that is connecting fine but that is on a different port, whilst I am sure you do have that port open as you suggested it never hurts to do a full check to make sure, that is why I recommended the tentacle ping as it would prove 100% you have a full connection from the tentacle itself into Octopus.

The only other thing I can suggest at the moment to rule it out is to uninstall that tentacle and then re-install it, whilst I realise this is a ‘noddy’ step to suggest it has fixed a few niggling issues for many so it never hurts to try.

I did not manage to ask around about this ticket yesterday but we have a meeting at 3 pm (UK time) today so I will bring this ticket up in that and see if anyone else has any ideas.

If you can get a tentacle ping going that would be beneficial for us (you don’t need the pong, that is for older tentacles and Octopus versions) but I sympathise with getting a CAB to install any software on a domain and know it can sometimes take a while to get through so you can install the software.

I will keep you updated on any discussions I have, please reach out in the meantime and see if the re-install works, I do not have high hopes but it would be nice to rule it out.

Kind Regards,

Clare Martin

10

Hi @dmccann,

Sorry to SPAM this thread here, I did not want to edit my comment above just in case you had already read it so will post this as an extra comment.

My colleague has just reminded me of something I missed out which will really help us here without needing the tentacle ping.

You said connecting to the tentacle via the web browser from the tentacle machine works but are you able to please try this from the Octopus Server, this will rule out connection issues.

image
image1559×1063 186 KB

I am going to get our documentation looked at for the listening tentacles as I missed out the ‘try from octopus server’ bit due to it being right underneath the ‘if it doesn’t work do this’. Since it worked I skipped that part and missed the ‘do the same but from the server’ bit. (The polling tentacles section is formatted better so I will get the listening section changed to match the formatting of the polling section).

Let me know if you can connect to that tentacle via the web browser on the octopus server.

Kind Regards,

Clare Martin

11

Hi Clare - thanks - yes I can connect:

image
image869×144 8.44 KB

12

Hi Clare - just saw your previous post. We have restarted and reinstalled the tentacle multiple times.
Oh - and don’t worry about spamming Clare - the more the merrier :slight_smile:
Thanks,

David

13

Hi @dmccann,

Thank you for testing that and confirming it can connect, and also for confirming about the restart and reinstall, I have spoken to some of my other colleagues and they can’t think of anything else to suggest, our US-based team comes online soon and we have our 3 pm meeting with them so I will bring this ticket up then and see if one of them has any ideas.

Unfortunately, the logs do not give us much information and you have tested pretty much everything we can test for connections (which have all been successful)…let me do some more digging and I will get back to you!

Please get in touch if you need anything else in the meantime,

Kind Regards,

Clare Martin

14

Hey @dmccann,

Just got out of our meeting and we have had a discussion about this ticket, the only thing we can think is that you may be using a proxy and the service account that is running the tentacle that’s failing may not have the correct network privileges to be able to connect to the Octopus Server on that port (but the account you were logged in with to try the connection from the browsers does).

Can you confirm the service account you are running that tentacle on has permissions to access the Octopus Server on that port, if it does then the only other thing we can do, if you wouldn’t mind, is to get you to put in the CAB request and install Tentacle ping I am afraid.

We can’t see anything else in the logs or from the tests we have tried so far that points us in any other direction other than network connectivity and the only way now to test that out is via a tentacle ping.

I know that is not the news you wanted to hear but we are a bit stuck in where to direct you other than tentacle ping.

Let me know what you think and the outcome of testing the tentacle service account to see if that has any network blocks on it (specifically from that failing tentacle port).

Kind Regards,

Clare Martin

15

HI Clare.
Thanks again. I changed the user under which the service was running to be the same as the other tentacle and we still got the same results. I am on leave all next week - can I nominate someone else in my company to take over? We hope to get CAB today and deploy TentaclePing and TentaclePong at the weekend.
Thanks,
David

16

Hey @dmccann,

Brilliant thank you for letting me know, we will help out any customer that is having issues so feel free to grab as many people to assist you as possible!

Yes if we can get tentacle ping installed and running that would be really beneficial for us, are you running on a proxy? That is the only way the user account would make a difference network wise.

Good to note that changing the account made no difference, I think the port would come into play there though so if you had some network blocks for user accounts that only allowed certain ports through that might have played into it.

We will see what the tentacle ping comes back with though as that is our definitive ‘it connects’ test.

I look forward to hearing from your nominated colleague next week on the results of the test and we can go from there, have a lovely week off! Hope you are going somewhere sunny, or just having a cozy week with the family!

Reach out if there are any other issues,

Kind Regards,

Clare Martin

17

Hello Clare,

As per your suggestion we have tested both the TentaclePing and TentaclePong. PFA log files.

I have ran the following command, Pls let me know if have done the right thing or not.

TentaclePing.exe 172.16.0.139 10934
(This is the IP where we tried to add second octopus, we are running this command from Build server where we installed Octopus Deploy)

TentaclePong.exe 10934
(We are running this command from server where we adding second octopus)
Ping.txt (39.2 KB)
Pong.txt (60 Bytes)

18

Hi @hbhokare,

Welcome to the forums and thank you for providing us with the tentacle ping and pong (you have done the right thing by running the ping from the Octopus server with the tentacle IP and port - though you can run it from the tentacle too and use the Octopus server IP and port).

When looking at the tentacle ping I can see the below error:

Pinging 172.16.0.139 on port 10934
2022-03-01T07:13:30 Connect: Failed! 21,018ms; connected: False; SSL: False
System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 172.16.0.139:10934
   at System.Net.Sockets.TcpClient.Connect(String hostname, Int32 port)
   at TentaclePing.Program.SendRequest(String hostname, Int32 port, SslProtocols sslProtocol, Int32& bytesRead, Boolean& connected, Boolean& sslEstablished, String data)
   at TentaclePing.Program.ExecutePing(String hostname, Int32 port, Int32 dataSize, SslProtocols sslProtocol)
2022-03-01T07:13:52 Connect: Failed! 20,996ms; connected: False; SSL: False

This, unfortunately, backs up the previous comments in that something somewhere on your network is blocking the connection between the tentacle and Octopus server on port 10934, do you have a more advanced network trace tool on your network such as Wireshark? I am wondering if you are able to use that to see where the traffic is getting blocked?

The only other two things I can suggest (if you have not done so already) is to check that port is actually open on the tentacle and Octopus Server (in an elevated command prompt run ‘netstat -ab’ on the tentacle and then on the Octopus Server). This will give you a list of all the ports that are open on the tentacle and server and show if 10934 is ‘Listening’ or not and if it is associated with the tentacle.

Have you also made sure 10934 is allowed through the firewall on both the tentacle and the octopus server?

As the final test you could temporarily completely disable the antivirus on the tentacle and Octopus Server (both at the same time - some AV applications have the ability to block network ports) and see if that makes a difference?

Unfortunately, we are unable to help you any further on this, though I am very curious as to what is blocking this connection we do not have the ability to fault find on your network, all of the tools we have used to our disposal so far have indicated you do have some connection to the tentacle from the server, the tentacle ping is the ‘holy grail’ of troubleshooting connections and if that says there is an issue then it proves 100% a networking issue is occurring at some stage in the connection process.

We have customers with load balancers and proxies where their networking team have put in the correct rules but connection issues are still occurring, when using Wireshark they can see where their connection is getting blocked and they usually end up putting a rule in somewhere that they forgot they had to do.

I had a good google of the error code in the tentacle ping log:

System.Net.Sockets.SocketException (0x80004005)

This points me to a few ‘check your network’ websites but this one I found interesting. Towards the bottom, it explains what that error code means and even throws the same exception you are seeing.

I know I was not much help here but I and the rest of the team are out of ideas on this one, if your networking team has put all the correct rules into the firewalls, you have checked both server and tentacle firewalls and the rules are correct in there, AV is not blocking ports, the tentacle service account has no networking restrictions, you can ping both server and tentacle from each other, you have checked the ports are open on both server and tentacle and that they are listening and not associated with another program (other than tentacle).

The only other option is to get some more advanced network monitoring tool installed to view the traffic and where it is getting blocked, or allow ALL connections from the Octopus Server and Tentacle (essentially temporarily disable firewalls on both those machines) and test (this way you can prove it is not the firewall at fault here). We cannot advise you to do that though, that would be a conversation for your networking team and yourselves to have.

Let me know your thoughts on all this, sorry it is such a long post but I wanted to make sure I had everything covered that we have gone through and the extra steps you could take to fault find.

I hope one of these leads you to a solution, please reach out if there are any other queries you have,

Kind Regards,

Clare Martin

19

Hello @clare.martin ,

Sure, I will first check whether port 10934 is open on both tentacle and then on the Octopus server or not.

Thanks,
Hardik Bhokare

1 Like
20

Hi @clare.martin
Yes - you were right - it seems there was an issue with the port being closed.
We have now resolved that and I just wanted to let you know and to thank you for all your help!

Regards,
David

21

Good afternoon @dmccann,

This is great news that you got to the bottom of it, hopefully now you can deploy with no issues, it is quite difficult to diagnose potential networking issues so thank you for being so patient whilst trying all the different fault finding tools I posted up!

Reach out anytime if you need help, we are always on hand to help customers where we can!

Thank you for getting back to us too, it helps other customers who are having similar issues if the original poster updates their threads with a confirmation something has worked.

Kind regards and happy deployments!

Clare Martin