Database connection spike to 100+

Setup: HA connected to SQL Server AWS RDS

I have seen over the last two days that our SQL Server has hit over 100+ connections which has brought down the node, the server and service is still running, but the node in the configuration is offline. This has happened three times over the last two days but hasn’t been seen before this.

I have temporarily resolved the issue by restarting the affected node and just now I have bumped the max connections on each node to 200.

But I’m concerned that this is just a band-aid and there is a more serious underlying issue.

I have looked through the logs at the times the problem occurs, but don’t see anything obvious. I have also looked at what was being deployed at that time to see if there is anything in common, but don’t see anything.

Can I have some pointers on how to track down what is occurring?

Hi Jon,

Thanks for getting in touch! I can certainly see why that would be concerning. We’d like to dig a bit more into it with some additional information. Could you confirm:

Is the node that’s going down the SQL server node, or the Octopus node?
Which version of Octopus are you currently running?
Could you also send us through your server logs (C:\Octopus\Logs in standard installations)?

I look forward to hearing back and getting to the bottom of this one!

Best regards,

Kenny

Hey Kenny,

The node will be one of the Octopus nodes. But it’s not always the same node.

These are the spikes:

I have sent you the logs via Slack due to the size limitation and potentially sensitive data.

Thanks

Hi Jon,

Thank you kindly for keeping in touch and providing that extra information. I’ve received those log files via Slack, and I’m confident it has pointed us in the right direction. Fortunately it looks like a previously known and fixed bug in 2019.9.10 LTS (where it looks like you’re currently on 2019.9.8 LTS). The bug was regarding the projects/all endpoint opening unnecessary connections to the database, and you can reference additional info in the bug report below.

Would you be able/willing to apply the upgrade to at least 2019.9.10 (preferably latest) and let me know how much improvement you see as a result? If you need any non-current version, you can find any of them here.

I hope this helps, and I look forward to hearing back!

Best regards,

Kenny

Hi Kenny,

Upgrading the version is possible, but not fully automated yet, so a bit of a pain. But maybe now gives me a chance to fully automate it.

Before I do, I want to be more confident that this is the problem. As we have been running the 2019.9.8 for several months and haven’t seen this issue before, but it is possible that the connections have spiked and never hit > 100 as I don’t have any alarms on the connection count.

Is there any way to get better confidence in that this is the cause? Is it the common case that this connection bug only manifests some time after installation?

Connection metrics for the last two weeks:

We also have a second 2019.9.8 instance, which has never spiked either to my knowledge.

Thanks
Jon

Hi Jon,

Thanks for keeping in touch! This specific bug was reported by a user after some time of their version being run, though in their case they had a large number of projects which seemed to be related to how many database connections were being opened.

How many projects do you have in this node? Is there a big difference in the total number of projects when compared to the second instance (that hasn’t spiked)? Do you see any performance issues when navigating through them in the web portal? If so, that would most likely be proof of this specific bug being the one biting you. Though if that’s not the case, I would still say there’s a good chance it’s correlated somehow and upgrading would be the first thing to try when possible.

I look forward to hearing back!

Best regards,

Kenny

Hi Kenny,

You’re right. The instance that has started to fail has 1300 ish projects, whereas the second instance is much smaller with less than 500.

As far as performance, the failing instance is much slower display the dashboard (but I guess because it’s showing more projects?) But navigation after that such as into a project or process looks comparable.

Let’s try the upgrade and see.

Thanks for your support

Jon

Hi Jon,

You’re very welcome. Thanks for following up and confirming those details. I’m very interested to hear how much of an impact the upgrade provides! Don’t hesitate to reach out if you have any questions or concerns along the way. :slight_smile:

Best regards,

Kenny