Running steps in parallel

Hi,

I’m looking to make some performance improvements to one of my projects and during my investigation I’ve copied the project and set all the steps to run in parallel. I’m currently only deploying a subset of the steps but it looks to be a significant improvement. One thing I’m struggling with is limiting how many of these steps execute at the same time. For example if I have a deployment with 20 steps and all are set to run in parallel then they all start running. I was hoping I’d be able to say all the steps are configured to run in parallel BUT only 5 can be running at any one time. From the below page in the documentation I thought one of the MaxParallelism variables might allow me to do this but it doesn’t appear to:

https://octopus.com/docs/administration/managing-infrastructure/run-multiple-processes-on-a-target-simultaneously

Details:

  • Octopus Server Locally Hosted - Version: 2020.4.4
  • All steps run on the same machine
  • All steps have the same designated role
  • All steps are basically the same but different services
  • All steps work from the same template and deploy an IIS site with custom scripting

I do have some workaround options but none are as good as having a variable to set how many execute at the same time in terms of scoping and quickly making changes.

Kind Regards,

Tom

Hi Tom,

Yeah, there isn’t a single variable you can set which will control the max parallelism on a single machine. The most direct way would be to configure this through your step setup by breaking up the steps which run in parallel. Similar to below. I realize this isn’t really what you were looking for, but it’s the best way to control it today.

Also, I’m sure you’ve read the warnings, but tread carefully when performing steps in parallel on the same machine. :slight_smile:

Thanks,
Mark Reeder

Thanks Mark,

That was one of my options, the other was to group the deployments in sets of parent/child steps. So if I wanted 3 running at any one time I’d divide the services evenly between 3 groups since the parent child setup still runs in series. Honestly this one is probably the least hassle to maintain, so I’ll most likely go with this.

I did take note of the warnings :slight_smile: The purpose here is to increase the deployment time in specific testing environments and my initial tests have shown up to a 50% reduction in deployment times depending on the number of steps run in parallel.

Thanks for getting back to me on this,

Tom

Yep, that is another very good option! Just depends on the type of step you are doing, but sounds like this will work well for you.

Nice speed improvement! Always feels good to make such an improvement!