Retention policy in the presence of tenants

Legacy Scale
1

We use tenants to help us manage the deployment of git topic branches. The creation and deletion of tenants in this case is highly dynamic and one of the challenges we are facing right now is the retention policies. The major problem is that as soon as a tenant is deleted, there is no more deployments for this tenant and as a consequence – no cleaning up of the deployment artifacts on tentacles.

Before jumping into the implementation of the custom solution, I would like to ask you, guys, if you have recommendations on how to deal with this problem.

Thanks

2

Hi Rosti,

Thanks for getting in touch! Currently Octopus will not cleanup anything on a Tentacle outside of deployment of a project. There is no uninstall or remove concept when we delete either tenants or projects.
However we do see the use of a feature like this, so one of the developers and myself had a bit of a brainstorm about what we could offer in this space for Tentacle cleanup.
We came up with the following idea: https://octopusdeploy.uservoice.com/forums/170787/suggestions/18644866
Please visit this suggestion on UserVoice and give it some votes and comments. We would really like to see some community support behind the idea as it has been asked for in the past.

Potentially in the mean time, as you do have information such as which Tenants you are deleting, you could write a project that connects to machines and looks for those tenants in the deployment journals to remove the files you are interested in.
If you want to go down this path I a sure we could assist with the logic we use.

Let me know what you think.
Vanessa

3

Thanks, Vanessa. I just voted for this idea.

In the meantime, we’ll implement our custom solution. The only uncertain thing for us is how to handle the deployment journal. Looks like we have to delete the deployment entries for deployments we are about to clean up. Is it as simple as just rewriting the existing journal without those entries or are there pitfalls?

4

Hi Rosti,

Deleting them from the deployment journal would be optimal, and the only pitfalls that I would see would be if you changed the permissions so Octopus could not write or read from the file. If you run your script via Tentacle this will not be an issue.
The second is if you had the file open when it was needed by another process it may throw or error in that deployment/process.

Calamari is OSS and here are a few of the functions that are called for our retention policies:


Let me know if you run into any issues or if you have any questions.
Vanessa

5

Hi Rosti,

Any chance you’ve implemented this and would be willing to post the source? We’re running up against the exact same thing and would love to collaborate on it if you’ve got something started.

I’ve also gone and voted for the solution :slight_smile:

Jacob

6

Yes, we implemented the custom solution. In our case tenants are git topic branches and the deletion/cleaning procedure is triggered by a git server hook. In response to the topic branch deletion the hook creates a release of the dedicated Octopus project (we named it “git-topics-cleanup”) and adds deleted ref names as release notes. The deployment step in this project parses ref names and comes up with corresponding tenant names (we use jira ticket keys as part of tenant names). Then it queries the tenants and creates a list of projects and environments for each tenant. So far it is pretty straightforward.

Each project that is connected to those tenants has a dedicated channel (we call it “Branch Topic Cleanup”) and a dedicated cleanup step that executes only for this channel. “git-topics-cleanup” deployment step creates release for each project associated with those tenants and deploys it for “Branch Topic Cleanup” channel and corresponding environments. This way we guarantee that the cleanup step will run on all machines containing artifacts for the tenants.

Now, we do some extra work during this step which may not be interesting for you (like, removing web sites, databases, etc.) but here is the part that takes care of the deployment artifacts:

# Deployment artifacts
Write-Host "------------------------------"
Write-Host "Cleaning deployment artifacts"
$tenantName = $OctopusParameters["Octopus.Deployment.Tenant.Name"]
$appDir = Join-Path $OctopusParameters["Octopus.Tentacle.Agent.ApplicationDirectoryPath"] $tenantName

if (Test-Path $appDir) {
  Write-Host "Deleting tenant '$tenantName' extraction directory '$appDir'"
  Remove-Item $appDir -Force -Recurse
}
else {
  Write-Warning "Could not find extraction directory '$appDir' for tenant '$tenantName'"
}

# Deployment journal
Write-Host "------------------------------"
Write-Host "Cleaning deployment journal"
$journalPath = $OctopusParameters["Octopus.Tentacle.Agent.ApplicationDirectoryPath"] -replace "(\\|/)Applications"
$journalName = "DeploymentJournal.xml"
$journal = Join-Path $journalPath $journalName

if (Test-Path $journal) {
  [xml]$deploymentJournal = Get-Content $journal
  $deploymentJournal.Deployments.Deployment | % {
    if (-not (Test-Path $_.ExtractedTo)) {
      if (Test-Path $_.ExtractedFrom) {
        Write-Host "Deleting orphaned package file '$($_.ExtractedFrom)'"
        Remove-Item $_.ExtractedFrom -Force
      }
      Write-Host "Deleting orphaned journal entry for '$($_.ExtractedTo)'"
      $_.ParentNode.RemoveChild($_) | Out-Null
    }
  }
  $deploymentJournal.Save($journal)
}
else {
  Write-Warning "Could not find deployment journal '$journal'"
}

Please note that this code runs on every target machine that has been used during the regular deployment process.

And here is the “git-topics-cleanup” project step that orchestrate everything:

$branches = $OctopusParameters["Octopus.Release.Notes"]

# find tenants and associated releases
$tenants = New-Object System.Collections.ArrayList($null)
$refs_re = "([^/]*/)*(?<topic>([A-Za-z]+-\d+|[\w_-]{0,7}))"
$refs = ($branches -split '[\r\n]') |? {$_}

$refs | % {
  if($_ -match $refs_re) {
    $topicId = $matches["topic"]
    $tenantName_re = [regex]"topic #$topicId$"
    $tenant = Get-FromOctopus "/api/tenants/all" | ? {$_.Name -match $tenantName_re}
    if ($tenant.Count) {
      $msg = "Found more than one tenant for git ref '$_':`n"
      $tenant | % {$msg += "  " + $_.Name + "`n"}
      StopWithError $msg
    }
    elseif ($tenant) {
      # collect tenant's projects and related releases
      $rels_re = [regex]"[0-9\.]+-feat\.$topicId\..+"
      $tenantProject = $tenant.ProjectEnvironments.PSObject.Properties | % {$_.Name}
      if ($tenantProject.Count -and $tenantProject.Count > 1) {
        $msg = "Found more than one project for tenant '$($tenant.Name)':`n"
        $tenantProject | % {$msg += "  " + (Get-FromOctopus "/api/projects/$_" | % {$_.Name}) + "`n"}
        Write-Warning $msg
        return
      }
      $tenantEnv = $tenant.ProjectEnvironments.$tenantProject
      if ($tenantEnv.Count -and $tenantEnv.Count > 1) {
        Write-Warning "Found more than one environment for tenant's project '$tenantProject'"
        return
      }
      
      $tenantReleases = Get-FromOctopus "/api/projects/$tenantProject/releases" | % {$_.Items} | ? {$_.Version -match $rels_re} | % {@{Id=$_.Id; Version=$_.Version}}
      $tenant = @{Releases=$tenantReleases; Name=$tenant.Name; Id=$tenant.Id; Project=$tenantProject; Env=($tenantEnv | ? { $_ }); Alias=$topicId; GitRef=$_;}
      $tenants += $tenant
    }
    else {
      Write-Host "Could not find tenant for git ref '$_'`n"
    }
  }
}

Write-Host "Found following tenants:"
Write-Host ($tenants | Convertto-Json -Depth 10)

$tenants | % {
  $tenant = $_
  Write-Host "------------------------------"
  Write-Host "Creating clean up deployment for '$($tenant.Name)' tenant"

  $relNum = "0.0.0-cleanup-$($tenant.Alias)"
  $release = (Get-FromOctopus "/api/projects/$($tenant.Project)/releases") | % { $_.Items } | ? { $_.Version -eq $relNum }
  # delete release if it already exists
  if ($release -ne $null) {
    Write-Host "Deleting existing release '$relNum' in the tenant's target cleanup project"
    Delete-FromOctopus "/api/releases/$($release.Id)" | Out-Null
  }
  
  # create new release
  # find channel
  $channelName = "Branch Topic Cleanup"
  $channel = (Get-FromOctopus "/api/projects/$($tenant.Project)/channels") | % { $_.Items } | ? { $_.Name -eq $channelName } | % { $_.Id }
  if ($channel -eq $null) {
    $projName = Get-FromOctopus "/api/projects/$($tenant.Project)" | % {$_.Name}
    Write-Warning "Channel $channelName does not exist in project '$projName'"
    Write-Warning "Skipping '$($tenant.Name)' tenant"
    return
  }
  $relNotes = "Initiated by deleting the branch '$($tenant.GitRef)'"
  $body =  @{ Projectid = $tenant.Project;
              version = $relNum;
              ChannelId = $channel;
              ReleaseNotes = $relNotes}
              
  Write-Host "Creating release:"
  Write-Host ($body | ConvertTo-Json -Depth 10)
  $release = Post-ToOctopus "/api/releases" $body
  if ($release -eq $null) {
    Write-Warning "Failed to create release '$relNum'"
    Write-Warning "Skipping '$($tenant.Name)' tenant"
    return
  }

  # create deployment
  $dep = @{
    ReleaseId=$release.Id;
    EnvironmentId=$tenant.Env;
    TenantId=$tenant.Id;
    #ForcePackageDownload=$false;
    #UseGuidedFailure=$false
  }
  Write-Host "Starting deployment:"
  Write-Host ($dep | ConvertTo-Json -Depth 10)
  $deployment = Post-ToOctopus "/api/deployments" $dep
  if ($deployment -eq $null) {
    Write-Warning "Failed to start deployment for release '$($release.Version)'"
    Write-Warning "Skipping '$($tenant.Name)' tenant"
    return
  }
  
  $taskId = $deployment.TaskId
  $task = Get-FromOctopus "/api/tasks/$taskId"
  $duration = 0
  $wait = 2 #sec
  $limit = 1200
  Write-Host "Waiting for deployment to complete (threshold is $limit sec)..."
  while(-not $task.IsCompleted) {
    if ($duration > $limit) {
      Write-Warning "Canceling deployment (more than $limit sec passed)"
      Post-ToOctopus "/api/tasks/$taskId/cancel"
      Write-Warning "Skipping '$($tenant.Name)' tenant"
      return
    }
    Start-Sleep -Seconds $wait
    $duration += $wait
    $task = Get-FromOctopus "/api/tasks/$taskId"
  }
  
  if (-not $task.FinishedSuccessfully) {
    Write-Warning "Deployment failed. Skipping '$($tenant.Name)' tenant"
    return
  }
  elseif ($task.HasWarningsOrErrors) {
    Write-Warning "Deployment completed with warnings"
  }
  Write-Host "Deployment completed in $duration sec"
  
  Write-Host "------------------------------"
  $tenant.Releases | % {
    Write-Host "Purging release '$($_.Version)'"
    Delete-FromOctopus "/api/releases/$($_.Id)" | Out-Null
  }
  
  Write-Host "------------------------------"
  Write-Host "Purging tenant '$($tenant.Name)'"
  Delete-FromOctopus "/api/tenants/$($tenant.Id)" | Out-Null
}