Massive parallel deployment

[gebai]

Hello,

We are considering to install Tentacle on all the hosts in our development environment. Let’s say 3000 boxes.

If we want to push a package to all 3000 boxes at the same time (parallel execution), will that cause any problem to the Octopus master server?

Is there anything we should be aware of when doing this? What is the best practice/recommended approach for this type of deployment scenario?

Thanks,

Ge

[Shane_Gill]

Hi Ge,

Thanks for getting in touch.

We have tested and have customers who are currently deploying to over 3000 Tentacles. When doing a deployment to that many machines there are safeguards in place on the Octopus server to ensure that performance isn’t impacted. For example: when pushing packages to 3000 machine the package push will be done in batches of 10 machines. You can elect to override the batch size if you find there is spare capacity on your Octopus server.

There are techniques that will help performance of large deployments. We recommend using listening Tentacles as this communication style is more efficient. You could use Tenants to divide the deployment into many smaller deployments which opens up greater parallelism and reduces the impact of failures. For example:

Assign machines in groups of 50 to Tenants. Machines 1-50 to tenant MachineGroupA, machines 51-100 to MachineGroupB etc etc. With 3000 machines and 50 machines per tenant: creating a deployment to all tenants will break the deployment up into 3000 / 50 = 60 deployments. The default task cap on a node is 5, so 5 deployments will be executed at once. Given a package transfer parallelism of 10, you will now have 5 * 10 = 50 package transfers happening at once. You could then tweak the task cap until your Octopus server has reached full resource utilisation. Another option would be to create an Octopus server cluster and have many Octopus servers executing the deployments. 3 nodes with task caps of 5 and package parallelism of 10 would be 3 * 5 * 10 = 150 package transfers in parallel. As you can see this approach opens up a lot of options to scale out the deployments.

Another benefit of spreading machines across tenants is that if the deployment to a single machine were to fail it would only impact the other machines for that tenant. So instead of the deployment failing for 3000 machines it would only fail for the group of 50 associated with the tenant.

I hope this helps provide some direction for doing large scale deployments, please let me know if you have questions or there is anything else I can help with.

Cheers,
Shane

[gebai]

Thanks, Shane!