Optimizing Octopus Server performance on AWS

Hello all,

I have a general performance question in regards to running Octopus Server on an AWS EC2 instance. Currently, we are running Octopus Server on a t2.small AWS EC2 instance. The database is on an m3.medium AWS EC2 instance running SQL Server 2014 - this SQL Server also holds our Shared and Dev databases for various projects, totaling around 30 databases, including our TeamCity database.

We have Octopus set up to use around 5 different channels, version rules for each channel, 41 different projects and associated roles, 60 different ‘machines’. It’s not slow all the time, but every now and then it’ll really slow down, to the point you can’t even load a project page.

Essentially, I am looking to optimize the dashboard performance of Octopus Server - in order to do that, I need to find out where the bottleneck is! My initial feeling was that any slowdowns could be a result of the large number of connections to the SQL Server. However, SQL Server should be optimized for this…?

Do we need a larger EC2 instance running Octopus Server?

Does Octopus Server generally use more CPU or Memory?

I’m not really looking for help specifically related to my instance of Octopus Server, more general tips on how other people are running Octopus on AWS, and where the bottlenecks have been in your experience!

Thanks in advance,

Hi @Hotdogsandfrenchfries, (What a great name)

Thanks for getting in touch! To give you the best answer we can, I am going to need to grab some more information from you. To start with, it would really help to know what version of Octopus you are currently using. I know that we had some performance issues surrounding our dashboard in older versions that has been greatly improved apon.

Looking forward to hearing from you. :slight_smile:

Best regards,
Daniel

Hi Daniel!

I’m glad you like my name! Unfortunately it always makes me hungry whenever I have to type it in.

I am running 3.3.5. I’m keen to upgrade to 3.4.15, however we have just entered a very busy season for the company, so need to be careful with changing the CI pipeline too much.

Thanks,
hotdogsandfrenchfries

Hi @Hotdogsandfrenchfries,

Thanks for getting back! Sadly the work we have done around improving the dashboard performance, even logging for performance issues, was all done after 3.3.5. I can link you to a couple of the changes that highlight the areas surrounding the dashboard we have worked on.

SQL Timeout when viewing Dashboard or Project Overview
https://github.com/OctopusDeploy/Issues/issues/2505

Add logging for long running requests and queries
https://github.com/OctopusDeploy/Issues/issues/2529

Unfortunately to see any improvement around the dashboard you are going to need to upgrade to at the very least 3.3.14, Preferably 3.3.15 to get adequate logging surrounding SQL queries. We would recomend however, updating to the latest 3.3 version (3.3.27) As there were no major feature addition but we did continue to implement bug fixes.

Let me know if this is an available option. If you were interested in exactly what has changed from 3.3.5 to 3.3.27 I have linked a comparison below.

Best regards,
Daniel

Hi Daniel,

Thanks for getting back to me. I have upgraded to 3.3.27 without any issues, however still seeing slowdowns on the Tasks and Release pages in particular.

Are there any performance fixes in the later versions, like 3.5.1?

Thanks,

Hi Hotdogsandfrenchfries,

Thanks for getting back! Unfortunately, performance issues generally tend different each time. From 3.3.27-3.5.1 there have been many small changes to performance, some may fix the issues you are having or they may not. The best way to identify where the performance hit is occurring would be to get some server logs now that you have upgraded.

You can find these under C:\Octopus\Logs in a default installation. As I mention in my previous post, in 3.3.15 we brought out some new logging to help identify issues around queries and requests.

If you are unsure about upgrading to 3.5.1 at the moment, attaching the new logs since the upgrade will be the next best thing to do.

Looking forward to hearing from you. :slight_smile:

Best regards,
Daniel