I have a Kubernetes cluster setup with 2 nodes, 1 windows, 1 linux.
I am scaling the pods based on the number of queued/executing tasks in Octopus.
The only issue is both nodes scale when really I need to get information about which node is having an increase in tasks and then scale that node. I.E 20 tasks queued for linux and 5 for windows should result in 20 linux pods and 5 windows pods being scaled…
There doesn’t seem to be anything like this easily available. I have read the article below and it seems to be able to gather some data relating to pod usage but it’s going to require a lot of troubleshooting to get working and I want to know if there’s any other way to do it.
Hey there Ben! I wanted to see if you could elaborate on what problem you’re wanting to solve with this? If I’m reading this correctly, it seems that you want to scale a worker per the number of tasks being executed. The workers feature is designed to be able to do multiple things at once so I wanted to make sure I was understanding correctly.
I want a HA scaling worker pool running on kubernetes. I understand that a worker can run multiple tasks on one worker but then that worker can become overloaded. The containers provide the ability to make sure no one worker gets overloaded.
I just want to be able to scale the kubernetes worker pools based on the workload, which I’ve managed to get working. Only problem is, like I mentioned before there isn’t a simple way to do so and was hoping someone could suggest something.
Hey there Ben, thank you for clarifying. I spoke with some colleagues, there isn’t a built-in way to do what you’re wanting, the only idea we came up with was using the Subscriptions feature to scale up/down based on Deployment Queued or Runbook Queued events.
However, I wanted to make sure you understood how workers were selected. When a worker has been leased by a task, the lease is not exclusive meaning the same worker can be selected for an entirely different task (see Workers explained - Octopus Deploy for more details). There are a few situations where a worker has exclusive access, but it’s usually for a short time and doesn’t prevent other tasks from selecting it.
Yeah I tested how the logic of the workers are a while back and noticed it did a round-robin. I think we are talking about two different scaling scenarios. The issue with yours is there’s only so many tasks they can take with the resources they have so would require scaling vertical. These days it makes more sense to scale horizontal, I.E using a scale set / K8 cluster.
Would you have any documentation or steps to achieve this using the subscription feature?
We don’t have anything specifically for that scenario, but here’s the documentation on the subscriptions feature Subscriptions - Octopus Deploy. For you, you’d most likely need to create a webhook to fire off the scale events.