We are running Octopus 3.4.4, and have a ~200 step deployment process using ~50 packages and in a development deployment we deploy ~40 web sites and ~6 database to ~10 machines. Currently this process takes 2-3 hours and longer when deploying to upper environments that have substantially more machines. We are looking for ways to speed this process up.
What we are seeing is that the time gap between a step and the next step appears not to be commensurate to the work being performed. Simple steps that take 3-4 seconds when run in the ‘script console’ take substantially longer when running in the context of the deployment project. Are there any tricks or tips to overcome this slowdown.
We have tried parallel steps with mixed success. The version we are on has a parallel task bug that makes the deployment unreliable. We intend to upgrade to a version with the fix and are currently targeting a move to Octopus 3.8.1. The parallel tasks improve our deployment times tremendously, but until we can move to a newer version what things can we look at to improve our deployment times?
Hi,
Thanks for getting in touch. There can be a few things going on here. The first, and main, one is that Octopus is doing a lot more work during a deployment than it is when just running a script in the script console. In particular around updating variable sets and state from step to step and coordinating across multiple machines.
One thing you can look at is the Server Task cap in Configuration -> Nodes. The default setting is 5 which is quite conservative, increasing this may speed things up. However there is some tuning required to find a suitable value for your hardware and deployments. If you put it high enough that the server is regularly maxing out the CPU or HDD throughput that will cause far worse performance. Try increasing it to 10 and then watch your server metrics during a deployment and go from there.
Finally, starting earlier this year we have a team focusing on performance bottlenecks, particularly related to larger deployments and Octopus installations, so hopefully you will see improvements with 3.8.1 and other newer releases if you can upgrade to them.
Regards,
Mark