Octopus Tentacles not healthy anymore

Since 26 November 2021, Octopus Tentacles have connection but they show unhealth. When did health check this was output.

Performing health check on machine

December 3rd 2021 16:50:33

Verbose

Script isolation level: NoIsolation

December 3rd 2021 16:50:33

Verbose

Executable directory is C:\Windows\system32\WindowsPowershell\v1.0

December 3rd 2021 16:50:33

Verbose

Executable name or full path: C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe

December 3rd 2021 16:50:33

Verbose

No user context provided. Running as current user.

December 3rd 2021 16:50:33

Verbose

Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory ‘C:\Octopus\Work\20211203155033-112663-5’ using ‘OEM United States’ encoding running as ‘EUR\ccservice’ with the same environment variables as the launching process

December 3rd 2021 16:50:34

Verbose

Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20211203155033-112663-5 exited with code 1

December 3rd 2021 16:50:34

Verbose

Exit code: 1

December 3rd 2021 16:50:34

Fatal

The remote script failed with exit code 1

December 3rd 2021 16:50:34

Verbose

The remote script failed with exit code 1 Octopus.Server.Orchestration.Targets.Tasks.ActionHandlerFailedException: The remote script failed with exit code 1 at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.SuccessArbitrator.ThrowIfNotSuccessful(IActionHandlerResult result) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/SuccessArbitrator.cs:line 22 at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation actionHandler, ITaskLog taskLog, VariableCollection variables) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 61 at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.TentacleMachineHealthController.CheckMachineHealth(Machine machine, ITaskLog taskLog) in ./source/Octopus.Server/Orchestration/ServerTasks/HealthCheck/Controllers/TentacleMachineHealthController.cs:line 48 at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.MachineHealthController.CheckHealth(Machine machine, ITaskLog taskLog) in ./source/Octopus.Server/Orchestration/ServerTasks/HealthCheck/Controllers/MachineHealthController.cs:line 56 at Octopus.Server.Orchestration.ServerTasks.HealthCheck.HealthCheckService.PerformHealthCheck(Machine machine, ITaskLog machineTaskLog, ITaskLog serverTaskLog, IHealthResultCollator healthResultCollator, CancellationToken cancellationToken, ExceptionHandling exceptionHandling, Action`2 customAction) in ./source/Octopus.Server/Orchestration/ServerTasks/HealthCheck/HealthCheckService.cs:line 69 Octopus.Server version 2021.3.8275 (2021.3.8275)

December 3rd 2021 16:50:34

Verbose

Recording health check results

Any idea whats going wrong here?

We have Octopus Server with version 2021.3 build 8275

HI @Yogesh_Bhole,

Thanks for reaching out and I am sorry to see you’re having issues.
I just have a few questions that might help us get to the bottom of this quickly:

  • Would it be possible for you to run a System Integrity Check for any issues? You can find it under Configuration → Diagnostics.
  • Which OS is running on the failing Tentacle targets?
  • Are health checks failing on just one target or all targets?
  • Were any updates or AV/Endpoint configured around the November 26th timeframe? In the past, we’ve seen health check scripts being canceled by AV and heuristics monitoring.

Looking forward to hearing back.

Regards,
Garrett

  • System health check seems fine see log here
Unexpected related document id format....................... [ Passed ] 
December 4th 2021 09:25:17Info
Schema...................................................... [ Passed ] 
December 4th 2021 09:25:17Info
Release channel belongs to the appropriate project.......... [ Passed ] 
December 4th 2021 09:25:17Info
Orphaned tenanted deployments............................... [ Passed ] 
December 4th 2021 09:25:17Info
Orphaned channels........................................... [ Passed ] 
December 4th 2021 09:25:17Info
Orphaned releases........................................... [ Passed ] 
December 4th 2021 09:25:17Info
Only one unfrozen deployment process per project............ [ Passed ] 
  • The OS is Windows Server 2012 R2 Standard for both Octopus server and tentacles
  • Health check is failing on all tentacles
  • I will check with IT if around 26 is there AV update?. Connection to tentacles seems fine as also tested by TentaclePing TentaclePong tool
  • I also unintsalled and installed again Tentacle service (Tentacle version 6.0.536) on Tentacle servers but problem still exists

Hi @Yogesh_Bhole,

Just stepping in for Garrett while he’s offline.

Thanks for confirming that information, it definitely seems like something strange is going on and I’d like to have a deeper look into the log files.

Could you please send through the Server and Tentacle logs of when these Health Checks are failing as well as the Task logs? Files can be uploaded securely here.

Feel free to let me know if there are any issues with the upload or you have any questions!

Best Regards.

Thanks for the feedback. Our IT disabled antivirus and suddently OctopusDeploy tentacles showing green for Health Check. Strange that there was connection to Tentacles but it seems antivirus blocking access to file read-write and error from tentacles as shared above was not clear, to come to that decision.

Anyway thanks for your time.

Best regards!

1 Like

Morning @Yogesh_Bhole,

Fantastic news that disabling AV managed to get your Tentacles healthy again! Thank you for letting us know, please reach out if there is anything else you need!

Kind Regards,

Clare Martin

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.