Parallelization in Powershell Scripts

tony_prudente · 26 July 2016 18:42

Hello,

I’ve got an Octopus deploy set up that uses the Redgate Powershell cmdlets to deploy to a list of databases after extracting the package. My Powershell script basically does a foreach over the list of databases and runs the necessary commands.

What I need to do now is effectively the equivalent of a Parallel.Foreach in C#. Our company has hundreds of identical databases that need to be deployed to with some degree of parallelism in order to complete a release in a reasonable amount of time.

Do you have any suggestions on the best approach to achieve this? In my mind, I’m picturing looking at the Octopus UI task log, and under a particular target, somehow seeing an expandable region for each database which gets deployed to on that target. This might not be realistic, but I would appreciate any thoughts you’d have on the matter in general.

I’ve attached my current script.

Thanks,
Tony

Dalmiro · 27 July 2016 17:05

Hi Tony,

Thanks for reaching out. The only way around this I can think of implicates having very big amount of deployment targets (Tentacles) which is pretty much the only way you can do parallel jobs like this in Octopus while expecting to be able to see the status of each specific DB as the deployment goes.

In 3.4 we’ll be introducing the concept of Multi-Tenancy, which initially was developed to think of Tenants as “different customers that might need the same code deployed on their env”. I’m guessing you could work your way around that concept and treat each DB as a different tenant. Please bear in mind that I’m constructing this idea as I type!

You can read more about Multi-tenancy here:

Origins and basics: https://octopus.com/blog/multi-tenant-deployments-origins

Usage of variable with tenants: https://octopus.com/blog/defining-variable-templates (this would be quite important if you decide to go down this road).

Thanks,
Dalmiro

Dalmiro · 27 July 2016 17:11

Idea construction continues…

An un-tenanted approach could be:

Lets say you have 3 DB servers and each has 2 databases where you wanna run scripts on. You could have a Tentacle in each server and run a Powershell Script in it in parallel. This PS script could take a parameter which will reference to an Octopus variable. This Octopus Variable (lets call it “Databases”) would have a copy per Tentacle, and its value would be the names of the databases you’ll be updating from that particular Tentacle.

This way when the script Runs in the first Tentacle, It’ll only update the Databases that are supposed to be updated from that Tentacle.

Hope that makes sense and gives you some ideas

tony_prudente · 27 July 2016 17:40

Re-attaching scripts with different extension

From: Prudente, Tony
Sent: Wednesday, July 27, 2016 10:30 AM
To: ‘Dalmiro Grañas’
Subject: RE: Parallelization in Powershell Scripts [Questions #8589]

Thanks for getting back!

Indeed, my current setup already has one tentacle per database server and my script handles figuring out which databases the tentacle should deploy to (which are the ones on the server the tentacle runs on). So this provides the first level of parallelization simply by way of having more than one db server, but unfortunately that isn’t enough. A single server could have 80 customer databases on it, and at 2-3 minutes per database, I need another level of paralellization still.

I might be close to a solution. I’ve used the Invoke-Parallel cmdlet from the Technet Galleryhttps://gallery.technet.microsoft.com/scriptcenter/Run-Parallel-Parallel-377fd430, which takes the list of databases as an input, and essentially operates like a Parallel.Foreach in C#.

The only snag I’m running into is that, because Invoke-Parallel uses runspaces as a method of parallelization, I need to explicitly import the Octopus cmdlets within the parallel script block. For whatever reason, the Redgate cmdlets are already present in the runspace context, but the Octopus ones are not. In my deploy script (attached), you’ll see where I invoke New-OctopusArtifact, and the script blows up at this point saying it is an unrecognized command. The Invoke-Parallel documentation says you need to import the snapin or module where these things are defined since they’re not automatically imported into the background runspace.

Where would I find the Octopus powershell cmdlets - in other words how to import them in the $scriptBlock?

I’ve attached the following files:

OctopusDbDeploy.ps1 (my custom Octopus deploy script)
Invoke-Parallel.ps1 (the script from TechNet, for reference)
Server

Thanks for your help - I should mention I’m a C# / SQL / Web developer so Powershell is not my top skill. Also wanted to mention that we’re a paid (Enterprise) customer.

Regards,
Tony

OctopusDbDeploy.txt (4 KB)

Invoke-Parallel.txt (26 KB)

tony_prudente · 27 July 2016 21:08

Actually I think a better solution here might be to create temporary files during the parallelized code block, then I can create the octopus artifacts from those files once execution returns to the main thread….

From: Prudente, Tony
Sent: Wednesday, July 27, 2016 10:31 AM
To: 'Dalmiro Grañas’
Subject: RE: Parallelization in Powershell Scripts [Questions #8589]

Re-attaching scripts with different extension

From: Prudente, Tony
Sent: Wednesday, July 27, 2016 10:30 AM
To: 'Dalmiro Grañas’
Subject: RE: Parallelization in Powershell Scripts [Questions #8589]

Thanks for getting back!

Indeed, my current setup already has one tentacle per database server and my script handles figuring out which databases the tentacle should deploy to (which are the ones on the server the tentacle runs on). So this provides the first level of parallelization simply by way of having more than one db server, but unfortunately that isn’t enough. A single server could have 80 customer databases on it, and at 2-3 minutes per database, I need another level of paralellization still.

I might be close to a solution. I’ve used the Invoke-Parallel cmdlet from the Technet Galleryhttps://gallery.technet.microsoft.com/scriptcenter/Run-Parallel-Parallel-377fd430, which takes the list of databases as an input, and essentially operates like a Parallel.Foreach in C#.

The only snag I’m running into is that, because Invoke-Parallel uses runspaces as a method of parallelization, I need to explicitly import the Octopus cmdlets within the parallel script block. For whatever reason, the Redgate cmdlets are already present in the runspace context, but the Octopus ones are not. In my deploy script (attached), you’ll see where I invoke New-OctopusArtifact, and the script blows up at this point saying it is an unrecognized command. The Invoke-Parallel documentation says you need to import the snapin or module where these things are defined since they’re not automatically imported into the background runspace.

Where would I find the Octopus powershell cmdlets - in other words how to import them in the $scriptBlock?

I’ve attached the following files:

OctopusDbDeploy.ps1 (my custom Octopus deploy script)
Invoke-Parallel.ps1 (the script from TechNet, for reference)
Server

Thanks for your help - I should mention I’m a C# / SQL / Web developer so Powershell is not my top skill. Also wanted to mention that we’re a paid (Enterprise) customer.

Regards,
Tony

Dalmiro · 27 July 2016 21:28

Hi Tony,

That temporary file solution makes a lot of sense

Let me know how it goes.

Dalmiro

Octopus_Support · 4 January 2017 06:31

Notice:

This issue has been closed due to inactivity. If you encounter the same or a similar issue and require help, please open a new discussion (if we asked for logs or extra details in this thread, consider including them in the new thread). If you are the creator of this thread and believe it should not be closed let us know via our support email.