Octopus Server Resources - Task Cap Load Limitations?

[DocOck (Alex)]

Hi Octopus Support,

Here is our scenario:

• Our automatic deployments (we have set up a ServiceNow integration process) are going to poll till the next day before continuing at the service now ticket time (this task is essentially making a call to service now for information)
• This will remain as a task and count against the current task cap of 10
• Deployments that hit the intervention step switched to a ‘queued’ status which removed them from the task cap count while they waited for a click.
• When the cap is reached all additional deployments are ‘queued’

Questions:

• What is a safe cap to increase the environment without needing to adjust server resources?
• Does increasing the cap increase load on the application server to the same level it would the SQL server?
• Is there any guidance on how much to increase server resources? (CPU, memory)
• How might High Availability affect this?

We are on Octopus v2019.3.8

Specs layout per your Q&A:

1. Are you running Octopus as an HA cluster, or single node? Single node.
2. Is the SQL Database Server on the same machine as Octopus or a different machine? Separate SQL 2016 server.
3. Are you hosting any other applications on the same machine as Octopus or its SQL database? No.
4. What kind of server specs are you running for Octopus and SQL Server?
   OS Name Microsoft Windows Server 2012 R2 Standard
   Processor Intel® Xeon® CPU E5-2698 v3 @ 2.30GHz, 2295 Mhz, 2 Core(s), 2 Logical Processor(s)
   Installed Physical Memory (RAM) 4.00 GB
   C Drive: 20 GB free of 44 GB, E Drive: 70 GB free of 124 GB
5. Approximately how many users do you have using Octopus? 130+
6. Approximately how many projects and machines do you have? 808 Projects and 321 Machines/Deployment Targets
7. Approximately how many deployments do you perform at the same time? Average 3-6, but up to 8-10. With the new process, this may spike to 15-20.
8. Do you notice any correlation between deployments of certain projects and the performance problem? Other than tasks waiting (health checks or tentacle upgrades pushing task cap), we haven’t had many issues, but with the new process, we are concerned with performance.
9. Do you notice any correlation between other Octopus Server tasks (like package retention policy processing) and the performance problem? No.
10. Does the Octopus Server ever become unresponsive and how frequently does it become unresponsive? Rarely.
11. Does the Octopus Server recover after the performance degrades, or does it need to be manually restarted in order to recover? If we ever had an issue, it required a service restart.

NOTE: We do not use the internal Octopus package repo–only an external package repository.

Thank you,
Alex

[Justin_Walsh]

Hi @DocOck!

Sadly, there’s no definite formula for increasing the task cap on your machines, as it’s highly dependent on the deployments that you’re doing. There is quite a bit of trial-and-error/iteration to find a setting that works for your deployment processes, frequency, and target counts. We generally recommend bumping it up in increments of, say, +50%, and see how your server resources look.

Increasing the task cap will increase both the application server and the SQL server usage, so they should both be monitored when making adjustments.

High-availability will help to spread the load over multiple application servers, but you will likely need a higher-spec SQL server to run multiple nodes. One option here for spreading the load further, without going to a HA setup (or as well as), is making use of Workers to offload some of the deployment tasks onto other, potentially lower-specced machines.

I hope this helps, and please don’t hesitate to reach out if you have any further questions!74