Hi guys, we have a set of 85+ tenants that we sometimes want to deploy all at once however we do have some tasks that run a role with a single machine inside because this action must be done once only (like LB active directory updates)
The problem is that when we run this project for all tenants, we get a massive delay in execution because all steps compete to get access to this machine even at package upload time.
Do you have any suggestion to improve this?
Is there a way to make a deploy only once kind of step that we could set on a role with many machines inside?
Hi Arnaud,
This is an interesting question.
One possible suggestion: if your step is a Script Step then you could set the step to Run on: Octopus Server and in On behalf of target roles supply the role which contains your single machine. (See the attached image for an example).
The effect of this would be that the step is executed on the Octopus Server rather than the target. This may remove some contention around package acquisition, but without knowing more about your scenario I can’t promise it will improve the situatuion. It may be worth trying though?
Just to clarify, you want this step to run once per tenant?
If so, the other suggestion that comes to mind is to have multiple machines with the role, but partition the machines by tenant (our docs on scoping machines to tenants).
The step will only execute on targets that match the role and tenant. The effect of this would hopefully be to spread the contention across a number of machines.
What do you think? Please let me know if these options don’t make sense, or need further clarification. Or if I have mis-understood your scenario.
Regards,
Michael
Hi thanks for these ideas.
Pitifully the octopus server can’t be used because it has no access to the resources that the target machines is handling.
Your second option seems interesting but from tests we have done before, tenant tags scopes are ‘AND’ on machine level meaning that I would have to deploy dedicated tentacles on machines to be able to use this scope in conjunctions with the other scopes we already have in place or am I wrong with this?
You are correct, the tenant tags are ‘AND’. But if used in combination with the unique role that you currently use for the step, I think it should be work.
For example, let’s assume the role for your step is AD, which currently only one machine belongs to. Now if you wish to increase the number of machines which can execute the step to 3, we could:
- Add two more machines with the
AD role.
- Create a new tenant tag set, called
AD partition, and have three tags: Partition1, Partition2, and Partition3.
- Assign each of your tenants to one of these three tags (divide them roughly equally)
- Assign each of the 3 machines with the
AD role to one of the 3 partitions
Now when you run a deployment, your AD step will only be able to execute on a machine with the AD role, and with a matching tenant tag (there shoulld only be one machine matching both these criteria for each tenant). This should have effectively expanded the pool.
I realize this is not the simplest solution.
If you would be willing to supply a deployment task log for a deployment which experienced the contention, we would like to have a look? There may be another way to improve the situation.
Regards,
Michael
