Custom validation for projects

marchenko_alexandr · 1 May 2023 07:36

Hello, I was wondering if we can have an dedicated validation step that will perform custom checks for any project before deployment, something similar to admissionwebhook from Kubernetes

Here are few restriction examples I wish to check:

channel version rules are configured
project has well known variables in correct format
resource requests and limits set for kubernetes deployments
…

Majority of this checks are doable via API but the problem here is how to wireup everything together?

Things I did already tried:

subscriptions - may be triggered after deployment starts
dedicated process step - may be disabled by operator
triggers - does not have anything related
…

Wondering if I am missing something else or may be someone did tried something similar

jeremy.miller · 1 May 2023 14:24

Hi @marchenko_alexandr,

Thanks for reaching out and for all of the information. There are two ways I believe you could achievet this, but I think the easiest way to achieve this is to have an informational step as Step 1 and then Step 2 is a Manual Intervention step.

For example in Step 1, you could have powershell code do API calls against your project and get the information for channels, variables, etc, and write-highlight those pieces of data. Step 2 would then stop the deployment so you can see that data and let you decide if everything is in good working order before allowing the deployment to proceed. Documentation: Manual intervention and approval step - Octopus Deploy

You can also potentially do this with Output Variables and Run Conditions on steps. You would do all of your checks in step 1, and based on the results you would save an Output Variable, then base your Run Condition of the next step on the result of that.

For example in Step 1, you could have powershell code do API calls against your project to check if it has channel version rules, and all the other requirements, and only if all of these pass, you would set an output variable like this, Set-OctopusVariable -name "TestResult" -value "True", and then later use that as a run condition for the next step like this: #{if Octopus.Action[Step01].Output.TestResult== "True"}True#{/if}

Supplementary output variable docs: Output variables - Octopus Deploy
Supplementary run condition docs: Conditions - Octopus Deploy

It’s two ways to approach it depending on how you would prefer to have it work.

Please let me know if you have any questions about it or if that helps.

Best,
Jeremy

marchenko_alexandr · 2 May 2023 05:42

Thank you for advise, I did consider additional step in a first place, it can be even simpler - aka run script on octopus server and just fail if anything went wrong

The problem with all this approaches is that someone who performs deployment can just skip it

Ok, you may complain that we may mark it as required

But in our case team allowed to edit projects on their own, which means they can just uncheck it from required steps

And this is done by intent, otherwise, if no one can create/edit projects it will be never ending train of requests from teammates to edit something in their projects, but if that’s the case, then there will be no need for such automated checks

The end goal is to allow teams to be empowered while having some rules in place

Like with Kubernetes - anyone can deploy to it, but because of admission webhook policies are applied

I was wondering if there is some kind of webhooks before project process being saved, or release being created or deployment task is going to be queued any of such things may be good starting point to wireup such checks

Another alternative may be to configure roles in a such a way that while allowing create projects teammates can not edit this step with checks (but not sure if thats possible with current roles)

Kenneth_Bates · 2 May 2023 06:17

Hi @marchenko_alexandr,

Thanks for following up with that additional information. This is a really good question, though unfortunately I don’t imagine there being any good solution to achieve this directly. You’re correct that even marking the step as required, a teammate with ProcessEdit permission on this project simply has the permission to modify this setting on the step, and the permissions aren’t granular enough to allow you to select which steps a user with the permission can edit and those they can’t. Additionally, there’s no way I’m aware of to send a webhook notification before your mentioned process change event.

I think you’ve just hit the limitation of these features. I think the closest solution is what has already been discussed, which of course leaves the possibility open for users who have ProcessEdit permission granted to them to modify this required setting on the step. You could put in a note on the applicable step to try to bring attention to your users to not skip the step, which would end up looking like this on the Process page.

I’m sorry I can’t give you the news you were hoping for. Let me know what you think, or if you have any further questions at all!

Best regards,

Kenny

marchenko_alexandr · 2 May 2023 07:26

True

As a workaround what I am thinking about is to have:

an dedicated external script that will make sure that each and every project has such “custom checks” required step
an subscription listening for deployments and checking if step was skipped

if such event occur we are going:

send slack notification
reconfigure project so this step is required again

Considerations: indeed it does not solve the problem, but sooner or later teammates should be tiered of disabling this step instead of fixing issues, plus notifications to slack won’t allow them to skip it often

At least it is better than nothing

Little bit later will give it a try and come back with details of how it might be implemented

PS: hope in future something like that but more reliable may be done without such tricks

marchenko_alexandr · 3 May 2023 06:13

@Kenneth_Bates thank you for ideas and brainstorming, here is what I have so far:

check

for checks we are going to use step template with script like this one:

if ($OctopusParameters) { # we are inside octopus
  $headers = @{"X-Octopus-ApiKey" = $OCTOPUS_APIKEY}
  $projectId = $OctopusParameters["Octopus.Project.Id"]
} else { # local run
  $headers = @{"X-Octopus-ApiKey" = $env:OCTOPUS_APIKEY}
  $projectId = "Projects-1"
}

$project = Invoke-RestMethod "https://robota.octopus.app/api/projects/$projectId" -Headers $headers
$failed = 0

# VARIABLES

$variables = Invoke-RestMethod "https://robota.octopus.app$($project.Links.Variables)" -Headers $headers | Select-Object -ExpandProperty Variables
$owner = $variables | Where-Object Name -EQ 'Owner'
if (-not $owner) {
  Write-Host "- 'Owner' variable is missing"
  $failed += 1
} else {
  # TODO: check if owner is a valid email
  # TODO: cehck if owner is not deactivated
}

# TODO: other checks

if ($failed) {
  Write-Host ""
  Write-Host "$failed checks failed"
  exit 1
} else {
  Write-Host "All checks passed"
}

For script to work we need to add parameter for Octopus Deploy API Key (in my case it has default value from common variables)

install

Now we need somehow add this script as required step to all projects, here is starting point:

$headers = @{"X-Octopus-ApiKey" = $env:OCTOPUS_APIKEY}

$templates = Invoke-RestMethod "https://example.octopus.app/api/actiontemplates/all" -Headers $headers
$template  = $templates | Where-Object Name -EQ 'octochecks'

$step = @{
  Name    = $template.Name
  Actions = @(@{
    Name               = $template.Name
    ActionType         = $template.ActionType
    IsRequired         = $true
    WorkerPoolVariable = 'octoworker' # NOTE: if using default workers remove me
    Properties = @{
      "Octopus.Action.Script.ScriptSource" = $template.Properties.'Octopus.Action.Script.ScriptSource'
      "Octopus.Action.Script.Syntax"       = $template.Properties.'Octopus.Action.Script.Syntax'
      "Octopus.Action.Template.Version"    = $template.Version
      "Octopus.Action.Script.ScriptBody"   = $template.Properties.'Octopus.Action.Script.ScriptBody'
      "Octopus.Action.RunOnServer"         = $true
      "Octopus.Action.Template.Id"         = $template.Id
    }
  })
}

$projects = Invoke-RestMethod "https://example.octopus.app/api/projects/all" -Headers $headers

foreach($project in $projects) {
  # $project = $projects | Where-Object Name -EQ 'Prometheus'
  $process = Invoke-RestMethod "https://example.octopus.app$($project.Links.DeploymentProcess)" -Headers $headers
  if ($process.Steps.Count -eq 0) { continue } # project without any step

  $steps = @()
  $found = $process.Steps | Where-Object { $_.Actions[0].Properties.'Octopus.Action.Template.Id' -eq $template.Id }
  if ($found) {
    $steps += $found
  } else {
    $steps += $step
  }
  foreach($item in $process.Steps) {
    if ($item.Actions[0].Properties.'Octopus.Action.Template.Id' -ne $template.Id) {
      $steps += $item
    }
  }

  $changed = $false
  if ($steps[0].Actions[0].IsDisabled) {
    $steps[0].Actions[0].IsDisabled = $false
    $changed = $true
  }
  if (-not $steps[0].Actions[0].IsRequired) {
    $steps[0].Actions[0].IsRequired = $true
    $changed = $true
  }
  if ((ConvertTo-Json -Depth 100 -InputObject $steps) -ne (ConvertTo-Json -Depth 100 -InputObject $process.Steps)) {
    $changed = $true
  }

  if ($changed) {
    try {
      $process.Steps = $steps
      Invoke-RestMethod -Method Put "https://example.octopus.app$($project.Links.DeploymentProcess)" -Headers $headers -Body (ConvertTo-Json -Depth 100 -Compress -InputObject $process) | Out-Null
      Write-Host $project.Name -ForegroundColor Green
    } catch {
      Write-Host $project.Name -ForegroundColor Red
    }
  } else {
    Write-Host $project.Name -ForegroundColor Cyan
  }
}

Notes:

this script is ment to be running externally, aka github scheduled workflows or kubernetes cronjob
be careful and do not copy paste it as is, it will be different for each setup, for example in my case we are using dedicated worker pool variable
be even more careful before running such scripts because they may corrupt current projects steps, double check everything semi manually first

subsription

very last one - is a subscription listening for deployment events

with fallback code like this

#!/usr/bin/pwsh

Write-Host 'content-type: text/plain'
Write-Host 'cache-control: no-store'
Write-Host ''
$body = $env:REQUEST_BODY | ConvertFrom-Json
<#
$body = @{
    Payload = @{
        Event = @{
            RelatedDocumentIds = @(
                "Deployments-69481"
            )
        }
    }
}
#>
$headers = @{"X-Octopus-ApiKey" = $env:OCTOPUS_APIKEY}
$deployment = $body.Payload.Event.RelatedDocumentIds | Where-Object { $_.StartsWith("Deployments-") }
$test = $body.Payload.Event.RelatedDocumentIds.Count -eq 1
$deployment = Invoke-RestMethod "https://example.octopus.app/api/deployments/$deployment" -Headers $headers
$project = Invoke-RestMethod "https://example.octopus.app/api/projects/$($deployment.ProjectId)" -Headers $headers
$process = Invoke-RestMethod "https://example.octopus.app$($project.Links.DeploymentProcess)" -Headers $headers
$skipped = $process.Steps | Select-Object -ExpandProperty Actions | Where-Object Id -in $deployment.SkipActions | Select-Object -ExpandProperty Name
$templates = Invoke-RestMethod "https://example.octopus.app/api/actiontemplates/all" -Headers $headers
$template  = $templates | Where-Object Name -EQ 'octochecks'

$step = @{
  Name    = $template.Name
  Actions = @(@{
    Name               = $template.Name
    ActionType         = $template.ActionType
    IsRequired         = $true
    WorkerPoolVariable = 'octoworker' # NOTE: if using default workers remove me
    Properties = @{
      "Octopus.Action.Script.ScriptSource" = $template.Properties.'Octopus.Action.Script.ScriptSource'
      "Octopus.Action.Script.Syntax"       = $template.Properties.'Octopus.Action.Script.Syntax'
      "Octopus.Action.Template.Version"    = $template.Version
      "Octopus.Action.Script.ScriptBody"   = $template.Properties.'Octopus.Action.Script.ScriptBody'
      "Octopus.Action.RunOnServer"         = $true
      "Octopus.Action.Template.Id"         = $template.Id
    }
  })
}


$steps = @()
$found = $process.Steps | Where-Object { $_.Actions[0].Properties.'Octopus.Action.Template.Id' -eq $template.Id }
if ($found) {
  $steps += $found
} else {
  $steps += $step
}
foreach($item in $process.Steps) {
  if ($item.Actions[0].Properties.'Octopus.Action.Template.Id' -ne $template.Id) {
    $steps += $item
  }
}

$changed = $false
if ($steps[0].Actions[0].IsDisabled) {
  $steps[0].Actions[0].IsDisabled = $false
  $changed = $true
}
if (-not $steps[0].Actions[0].IsRequired) {
  $steps[0].Actions[0].IsRequired = $true
  $changed = $true
}
if ((ConvertTo-Json -Depth 100 -InputObject $steps) -ne (ConvertTo-Json -Depth 100 -InputObject $process.Steps)) {
  $changed = $true
}

if ($changed) {
  try {
    $process.Steps = $steps
    Invoke-RestMethod -Method Put "https://example.octopus.app$($project.Links.DeploymentProcess)" -Headers $headers -Body (ConvertTo-Json -Depth 100 -Compress -InputObject $process) | Out-Null
    Write-Host "$($project.Name) - updated"
  } catch {
    Write-Host "$($project.Name) - failed"
  }
}

if ($skipped -contains 'octochecks') {
    Invoke-RestMethod -Method Post "https://slack.com/api/chat.postMessage" -ContentType 'application/json' -Headers @{ Authorization = "Bearer $($env:SLACK_TOKEN)" } -Body (@{channel = '@mac'; text = "$($project.Name) deployed without checks by $($deployment.DeployedBy)"} | ConvertTo-Json)
}

Notes:

technically it does repeats install script but for concrete project
its job is to make sure that required step is in place
also notify about deployments where this step was missing
in my case this script was running behind fastcgi and nginx
complete config examples can be found here

recap

Once again, it does not solve the original goal but still better than nothing, and I do believe that sooner or later issues will be fixed, and notifications in slack wont allow engineers to bypass checks often