Listening tentacle leaking TCP connections

reliability
(Lee) #1

Hi,

We’ve recently noticed an unusual pattern of TCP connections on 2 servers that are running Tentacles in Listening mode (we also have 2 servers with Polling Tentacles that are not exhibiting the same problem). The tentacles are otherwise working correctly and we’ve been able to deploy to them (for many months) without any issues whatsoever, but it appears they are periodically (1 per hour) not closing a TCP connection (presumably opened during a natural heartbeat in operation), that connection then lingers in the TIME_WAIT status indefinitely.

This was first spotted while running 2019.6.2 LTS and has persisted through an update to 2019.6.7 LTS and then also 2019.7.13 & 2019.8.1.

We’ve tried the latter two Fast Lane releases on the thinking that the cause may be tied to the following issue noted in release notes for 2019.7.9:
https://github.com/OctopusDeploy/Issues/issues/5777

We have also applied the Tentacle updates with each version. However, in our case, they have not yielded any improvement. It seems unlikely to be caused by our network/firewall configurations given it otherwise functions perfectly, but obviously happy to look into that further if you’re unable to reproduce yourselves. Given the issue is relatively benign, please do categorize this as low priority and let me know if you need any more detail.

Files hopefully attached (inclusive of tentacle log) to provide some context.

Many thanks in advance for your assistance.

Edit to add: looking back to the origin of that connection chart I attached suggests that this started happening when we updated from 2018.6.7 to 2018.6.12 in early July.


OctopusTentacleNetstat.txt (442 Bytes)
OctopusTentacle.txt (10.9 KB)

(Michael Noonan) #3

Hi @leftfield,

Thanks for getting in touch! I neglected to get back to you earlier - sorry. We’ve done a bunch of work in this area recently and I’m hoping an upgrade of your Tentacle will fix the problem.

Upgrading to Tentacle 5.0.3 fixes the leak we were able to reproduce for Listening Tentacles.

Upgrading to Octopus Server 2019.8.2 fixes the leak for Polling Tentacles.

Please let me know if this fixes the problem in your scenario, or if it doesn’t.

Hope that helps!
Mike

1 Like
(Lee) #4

Hi Mike,

No worries, thanks for getting back to me. That’s great, we’ll give it a try shortly and report back.

Kind regards,
Lee

(Lee) #5

Hi Mike,

I can confirm that the Tentacle 5.0.3 fixes have definitely resolved this issue entirely for us.
Fantastic work, much appreciated :clap:

Thanks again,
Lee