Troubleshooting a Linux Polling Tentacle

Octopus Server
1

I have 3 Kubernetes Clusters. Each of them have 3 Tentacles running on them. (For a total of 9 Tentacles.)

Today, the Polling Tenacles on my NonProd Cluster started failing the health check with a “timeout” error. (I presume that this means it is no longer checking in.)

I logged into one of the pods that hosts the Tentacle, and everything seems fine.

I tried to go to this page to follow some troubleshooting tips: Troubleshooting Tentacles - Octopus Deploy But I quickly realized that it was for windows only.

I went to the Linux side of the Tentacle documentation, but there are no troubleshooting steps that I can see. So I thought I would ask here.

Basically here are my symptoms:

  1. The last successful heath check ran 16 hours ago.
  2. Now health checks and deploys all time out.
  3. The Pods that run the Tentacles seem to be working fine. (I can SSH in and look at files and run commands.)
  4. The other 2 Clusters can still connect just fine.
  5. I don’t know of any changes to the NonProd cluster, but I am digging into it.

I looked at the Tentacle log files and this is what they show:

OctopusTentacle.txt 
PS /etc/octopus/Logs> cat ./OctopusTentacle.txt
2021-12-16 21:27:05.4401     48      1  INFO  ================================================================================
2021-12-16 21:27:05.4431     48      1  INFO  Changed log folder from Octopus/Logs to /etc/octopus/Logs
2021-12-16 21:27:05.4460     48      1  INFO  Tentacle version 6.1.670 (6.1.670) instance Tentacle
2021-12-16 21:27:05.4460     48      1  INFO  Environment Information:
  OperatingSystem: Linux 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021
  OsBitVersion: x64
  Is64BitProcess: True
  CurrentUser: root
  MachineName: cont-octopus-tentacles-bd46f4b7c-x8xk5
  ProcessorCount: 8
  CurrentDirectory: /
  TempDirectory: /tmp/
  HostProcessName: tentacle
  PID: 48
2021-12-16 21:27:05.5338     48      1  INFO  ==== ConfigureCommand ====
2021-12-16 21:27:05.5341     48      1  INFO  CommandLine: /opt/octopus/tentacle/Tentacle.dll configure --instance Tentacle --app /home/Octopus/Applications
2021-12-16 21:27:05.5703     48      1  INFO  Application directory set to: /home/Octopus/Applications
2021-12-16 21:27:05.5780     48      1  WARN  These changes require a restart of the Tentacle.
2021-12-16 21:27:09.7172     83      1  INFO  ================================================================================
2021-12-16 21:27:09.7172     83      1  INFO  Changed log folder from Octopus/Logs to /etc/octopus/Logs
2021-12-16 21:27:09.7205     83      1  INFO  Tentacle version 6.1.670 (6.1.670) instance Tentacle
2021-12-16 21:27:09.7205     83      1  INFO  Environment Information:
  OperatingSystem: Linux 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021
  OsBitVersion: x64
  Is64BitProcess: True
  CurrentUser: root
  MachineName: cont-octopus-tentacles-bd46f4b7c-x8xk5
  ProcessorCount: 8
  CurrentDirectory: /
  TempDirectory: /tmp/
  HostProcessName: tentacle
  PID: 83
2021-12-16 21:27:09.8175     83      1  INFO  ==== ConfigureCommand ====
2021-12-16 21:27:09.8175     83      1  INFO  CommandLine: /opt/octopus/tentacle/Tentacle.dll configure --instance Tentacle --noListen True
2021-12-16 21:27:09.8921     83      1  INFO  Tentacle will not listen on a port
2021-12-16 21:27:09.9008     83      1  WARN  These changes require a restart of the Tentacle.
2021-12-16 21:27:13.8282    118      1  INFO  ================================================================================
2021-12-16 21:27:13.8282    118      1  INFO  Changed log folder from Octopus/Logs to /etc/octopus/Logs
2021-12-16 21:27:13.8325    118      1  INFO  Tentacle version 6.1.670 (6.1.670) instance Tentacle
2021-12-16 21:27:13.8325    118      1  INFO  Environment Information:
  OperatingSystem: Linux 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021
  OsBitVersion: x64
  Is64BitProcess: True
  CurrentUser: root
  MachineName: cont-octopus-tentacles-bd46f4b7c-x8xk5
  ProcessorCount: 8
  CurrentDirectory: /
  TempDirectory: /tmp/
  HostProcessName: tentacle
  PID: 118
2021-12-16 21:27:13.9272    118      1  INFO  ==== ConfigureCommand ====
2021-12-16 21:27:13.9272    118      1  INFO  CommandLine: /opt/octopus/tentacle/Tentacle.dll configure --instance Tentacle --reset-trust
2021-12-16 21:27:13.9294    118      1  INFO  Removing all trusted Octopus Servers...
2021-12-16 21:27:14.0082    118      1  WARN  These changes require a restart of the Tentacle.
2021-12-16 21:27:17.9700    153      1  INFO  ================================================================================
2021-12-16 21:27:17.9700    153      1  INFO  Changed log folder from Octopus/Logs to /etc/octopus/Logs
2021-12-16 21:27:17.9734    153      1  INFO  Tentacle version 6.1.670 (6.1.670) instance Tentacle
2021-12-16 21:27:17.9734    153      1  INFO  Environment Information:
  OperatingSystem: Linux 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021
  OsBitVersion: x64
  Is64BitProcess: True
  CurrentUser: root
  MachineName: cont-octopus-tentacles-bd46f4b7c-x8xk5
  ProcessorCount: 8
  CurrentDirectory: /
  TempDirectory: /tmp/
  HostProcessName: tentacle
  PID: 153
2021-12-16 21:27:18.0559    153      1  INFO  ==== NewCertificateCommand ====
2021-12-16 21:27:18.0559    153      1  INFO  CommandLine: /opt/octopus/tentacle/Tentacle.dll new-certificate --instance Tentacle --if-blank
2021-12-16 21:27:19.3044    153      1  INFO  A new certificate has been generated and installed. Thumbprint:
2021-12-16 21:27:19.3129    153      1  INFO  D216C___________REDACTED___________B1C41
2021-12-16 21:27:19.3129    153      1  WARN  These changes require a restart of the Tentacle.
2021-12-16 21:27:23.4275    188      1  INFO  ================================================================================
2021-12-16 21:27:23.4281    188      1  INFO  Changed log folder from Octopus/Logs to /etc/octopus/Logs
2021-12-16 21:27:23.4312    188      1  INFO  Tentacle version 6.1.670 (6.1.670) instance Tentacle
2021-12-16 21:27:23.4312    188      1  INFO  Environment Information:
  OperatingSystem: Linux 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021
  OsBitVersion: x64
  Is64BitProcess: True
  CurrentUser: root
  MachineName: cont-octopus-tentacles-bd46f4b7c-x8xk5
  ProcessorCount: 8
  CurrentDirectory: /
  TempDirectory: /tmp/
  HostProcessName: tentacle
  PID: 188
2021-12-16 21:27:23.5366    188      1  INFO  ==== RegisterWorkerCommand ====
2021-12-16 21:27:23.5466    188      1  INFO  CommandLine: /opt/octopus/tentacle/Tentacle.dll register-worker --workerpool KubernetesNonProdWorkerPool --instance Tentacle --server http://octoserver/ --space IT Kubernetes --policy Default Machine Policy --force --comms-style TentacleActive --server-comms-port 10943 --apiKey ********
2021-12-16 21:27:23.6507    188      1  INFO  Checking connectivity on the server communications port 10943...
2021-12-16 21:27:24.3082    188      1  INFO  Connected successfully
2021-12-16 21:27:24.3082    188      1  INFO  Registering the tentacle with the server at http://octoserver/
2021-12-16 21:27:24.6823    188      1  INFO  Detected automation environment: NoneOrUnknown
2021-12-16 21:27:25.6893    188     13  INFO  Machine registered successfully
2021-12-16 21:27:25.6914    188      1  WARN  These changes require a restart of the Tentacle.
2021-12-16 21:27:29.5940    227      1  INFO  ================================================================================
2021-12-16 21:27:29.5940    227      1  INFO  Changed log folder from Octopus/Logs to /etc/octopus/Logs
2021-12-16 21:27:29.5973    227      1  INFO  Tentacle version 6.1.670 (6.1.670) instance Tentacle
2021-12-16 21:27:29.5973    227      1  INFO  Environment Information:
  OperatingSystem: Linux 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021
  OsBitVersion: x64
  Is64BitProcess: True
  CurrentUser: root
  MachineName: cont-octopus-tentacles-bd46f4b7c-x8xk5
  ProcessorCount: 8
  CurrentDirectory: /
  TempDirectory: /tmp/
  HostProcessName: tentacle
  PID: 227
2021-12-16 21:27:29.6813    227      1  INFO  ==== RunAgentCommand ====
2021-12-16 21:27:29.6813    227      1  INFO  CommandLine: /opt/octopus/tentacle/Tentacle.dll agent --instance Tentacle --noninteractive
2021-12-16 21:27:29.8239    227      1  INFO  Agent will trust Octopus Servers with the thumbprint: A0D31___________REDACTED___________5C311
2021-12-16 21:27:29.8375    227      1  INFO  Agent will poll Octopus Server at https://octoserver:10943/ for subscription poll://4y537qz42kuqf55dqyjl/ expecting thumbprint A0D31___________REDACTED___________5C311
2021-12-16 21:27:29.9906    227      1  INFO  Agent will not listen on any TCP ports
2021-12-16 21:27:30.3171    227      9  INFO  https://octoserver:10943/       9  Opening a new connection to https://octoserver:10943/
2021-12-16 21:27:30.7027    227      9  INFO  https://octoserver:10943/       9  Secure connection established. Server at [::ffff:10.90.68.175]:10943 identified by thumbprint: A0D31___________REDACTED___________5C311, using protocol Tls12

I am unsure how to go about troubleshooting this loss of connection. Any tips or suggestions would be appreciated.

2

I updated the docker image and redeployed the Tentacle pods and the health checks started working again.

While I don’t need help on this issue anymore, I would like to suggest that a Linux based Tentacle troubleshooting guide would be a useful bit of documentation.

4

Hey @OctopusSchaff,

That’s great to hear that you were able to resolve it, apologies for being a bit slow on the uptake of this one!

I 100% agree that our documentation could use a guide for troubleshooting Linux Tentacles, I’ve reached out internally about getting this added and will post a link here when it’s released!

Best Regards,

1 Like
6

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.