We have a previously functioning Octopus Server set up with a number of environments that we were successfully deploying too.
However recently two tentacles in the same environment have stopped behaving. They respond to the connectivity check (if I restart a tentacle, it shows as disconnected on the server then connected) but if we try a deployment or a healthcheck, it just hangs without response.
Logs on each of the tentacles don’t know anything of interest however the log on the server shows:
2018-05-23 12:54:35.3846 68 WARN It looks as though conversations with XXXX-FAILING-TENTACLE are leaking
Google hasn’t aided me in locating the reason for this message. It only shows that message for the two failing tentacles and not for the other 8 successful tentacles.
I’ve attached logs from server, a tentacle which is working nicely (GoodOctopusTentacle.txt) and one that is not working (BadOctopusTentacle.txt) as there are splats of text in both the Tentacle logs which I discarded seeing as they appear on the good and the bad tentacle.
I have not been able to figure out what is causing this issue yet.
While I keep investigating the issue, could you try the Reinstall option for your Tentacle service in the Octopus Tentacle Manager application and see if that fixes the issue?
I’ve already tried that as part of my earlier testing steps.
Do you know what the “It looks as though conversations with XX are leaking” means? As I’m suspicious because that message only happens on the failing tentacle that it might at least guide us towards the problem, even if it’s just a symptom of something generally benign.
The It looks as though conversations with ... are leaking message is printed when there are > 100 messages on the go at a time between the Octopus server and Tentacle. Which makes sense as the Tentacle receives the message, but fails to process it (the errors in the bad Tentacle log) and never returns the response.
I’ll keep investigating if I can figure out what else we can try and do to get to the root cause of this issue.