How can I set a unique delay for each machine in a rolling deployment in Octopus Deploy?

deployment-config
rolling-deployment
how-to
(Mark Harrison) #1

I deploy to multiple machines as part of a rolling deployment in Octopus, and one of the steps creates a Scheduled Task.

If they run at the same time, they can often cause conflicts with one another. The issue I have is that I need to find a way to schedule the tasks to start 15 minutes apart to ensure that each one starts at different times to reduce the chance of a conflict.

Is this possible within a rolling deployment in Octopus?

(Mark Harrison) #2

That’s a really interesting question!

Whilst it probably wasn’t one of the primary use-cases considered when adding support for rolling deployments in Octopus, it’s possible to achieve using a combination of the following features:

Increment Rolling Counter

The key to getting this to work is to have a counter which is incremented as each deployment target is being deployed to. With this, you can then safely increment a start time which is staggered by a different amount of time for each scheduled task.

:warning: Window Size of 1 :warning:
The following solution has only been tested with a Window Size of 1.

Required Steps

For this to work we’ll create the following script steps outside of the rolling deployment:

  1. Set a Global Start Time - This will just be a date that captures the time the step runs. It will be used as a base time for the Scheduled Tasks to be created from.
  2. Generate Initial Counter - This will generate a counter with an initial value of 0.

Note: These two steps could be combined.

Then we’ll add a number of script steps as child steps to the rolling deployment:

  1. Increment Counter - This will increment the counter by 1 as each machine is deployed to.
  2. Calculate staggered start time - This will calculate the staggered time for the task for the machine.
  3. Create the Scheduled Task - This will create the scheduled task with the staggered time.

All of the relevant steps are illustrated here:

Step: Set Global StartTime

This script step which takes today’s date, and creates an Octopus Output variable called DeployStartTime. This date value will be used as the base value to be incremented for the staggered Task start time. The complete script looks like this:

$now = Get-Date -Format "s"
Set-OctopusVariable -name "DeployStartTime" -value $now

Step: Generate Counter

This script step simply creates an Output variable called Counter with a value of 0:

Set-OctopusVariable -name "Counter" -value "0"

Child-Step: Increment Counter

The next script step increments the counter that was generated in the previous Generate Counter step, and this time, it’s part of the rolling deployment.

However, to increment the counter correctly, the step needs to take into account that this may be the first or subsequent machine to be deployed to.

It does this simply by checking for the existence of an Output variable called:

Octopus.Action[Increment Counter].Output.Counter.

Note that the name of the variable includes the name of the child step we are actually already running - Increment Counter.

This is because:

  • If the value exists, this means that it is not the first time the child script step has been run, and it should use the Octopus.Action[Increment Counter].Output.Counter variable as the source for the current value of the counter.
  • If the value doesn’t exist, then it is the first time the child script step has run, and we need to create the Octopus.Action[Increment Counter].Output.Counter variable for the first time. We use the value from the output variable created in the Generate Counter step.

In either case, we increment the value by 1.

The complete script looks like this:

$InitialCounter =  $OctopusParameters["Octopus.Action[Generate Counter].Output.Counter"]
$IncrementedCounter =  $OctopusParameters["Octopus.Action[Increment Counter].Output.Counter"]
Write-Host "InitialCounter is $InitialCounter"
Write-Host "IncrementedCounter is $IncrementedCounter"

if([string]::IsNullOrWhiteSpace($IncrementedCounter) -eq $false) {
	$Counter = ([int]$IncrementedCounter) + 1
    Write-Highlight "Incrementing counter to $Counter"
    Set-OctopusVariable -name "Counter" -value $Counter
}
else {
	if([string]::IsNullOrWhiteSpace($InitialCounter) -eq $false) {
    	$Counter = ([int]$InitialCounter) + 1
    	Write-Highlight "Incrementing counter to $Counter"
    	Set-OctopusVariable -name "Counter" -value $Counter
    }
}

Child-Step: Calculate task staggered start time

The next child script step takes the incremented counter, referenced by a Project variable called CounterValue and multiplies that value with a second Project variable called StaggeredDelayMinutes.

Note: This step makes use of a convenient feature within Octopus. You can create Project variables which reference values created in the deployment process itself.

The Project variable CounterValue is actually just a reference to the output variable Octopus.Action[Increment Counter].Output.Counter.

In this case, this is simply done to make the variable name simpler.

The variables can be seen here:

Once the delay in minutes has been calculated, the script then adds that value to the base start time we generated in the previous Set Global StartTime step. Finally that staggered start time value is stored as an output variable called TaskStaggeredTime.

The entire script looks like this:

Write-Host "Deploy started at: $DeployStartTime"
Write-Host "StaggeredDelayMinutes: $StaggeredDelayMinutes"
Write-Host "CounterValue: $CounterValue"

$counter = [Convert]::ToInt32($CounterValue)
$minsDelay = [Convert]::ToInt32($StaggeredDelayMinutes)
$DeployStart = [DateTime]::ParseExact($DeployStartTime, "s", $null)
$calculatedStaggerMins = $minsDelay * $counter

Write-Host "Calculated Mins Delay: $calculatedStaggerMins"

$TaskStartTime = $DeployStart.AddMinutes($calculatedStaggerMins)
$TaskStaggeredTime = $TaskStartTime.ToString("s")
$machineName = $OctopusParameters["Octopus.Machine.Name"]

Write-Highlight "${machineName}: Calculated delay of $calculatedStaggerMins mins, staggered task start: $TaskStaggeredTime"

Set-OctopusVariable -name "TaskStaggeredTime" -value $TaskStaggeredTime

Child-Step: Create scheduled task

Finally, you can create your scheduled task with your staggered start time. This script step just prints the staggered task time to demonstrate using the TaskStaggeredTime calculated on the previous child-step.

Write-Highlight "Simulating Scheduled Task creation with time of: $TaskStaggeredTime"

Deployment in action

Here is the deployment of the project, showing the staggered deployment times for each task to be created across each deployment target:

  • Test01 has a calculated delay of 15 minutes, as the counter is 1.
  • Test02 has a calculated delay of 30 minutes, as the counter has been incremented to 2

Sample project:
You can see an example of this in our samples instance (log in as Guest): https://g.octopushq.com/PatternRollingSampleIncrementCounter

(Mark Harrison) listed #3