Override Deployment Target (On Behalf Of) when Deploying via API

Advice
1

I am working with Kubernetes Clusters and am developing a way to mirror my Installs to my Backup Cluster when I run a specific RunBook (designed to do this backup install). I have created the script that will iterate through all my Octopus Deploy Projects, find the latest release in each of them (for each tenant) and re-deploy it.

Where I am hitting issues is getting that deployment to target my backup cluster instead of the normal cluster that it would when doing a normal deploy. (The actual projects need to target only the active cluster.)

What I am hoping for:

Is there a way, when I make a POST call to the rest API deployments endpoint, to pick a different “On Behalf Of” value?

That way I could setup targets for my backup cluster, and only use them via API when I run the RunBook.

NOTE: This is not a “Must Have” feature. If there is no way to do it, I will look into programmatically swapping my deployment targets to the backup cluster, run the RunBook, the swap them back. But this is not ideal as it forces a downtime for normal deploys every time I run that RunBook. (Because the targets are not pointing to the active cluster.)

3

Hey @OctopusSchaff , thanks for reaching out!

I have a couple of questions/clarifications to try and ensure I understand your use case.

  • You have standard deployments that go to Kubernetes clusters
  • You also have backup clusters where you deploy additional copies of these projects
  • You have a runbook that finds deltas between your two clusters and deploys to your backup cluster(s) to bring it to full parity with what’s in your active cluster

If that’s correct, here are a couple of questions:

  • Is it possible to deploy to all of the servers at once? Rather, is there a specific aspect of your use case that requires deploying to your backup cluster separately rather than simultaneously with your active node?
  • Have you looked at specifying specific machines when you create a deployment? Using something like this API script sample, you could deploy an existing release to a specific subset of machines (in this case, your backup cluster) using your runbook.
  • Have you looked at using a separate role for your backup cluster? If your active cluster has the role MyApp, you could use a MyApp-Backup role on the corresponding backup server, which would allow you to more granularly target those clusters during deployments. You could then duplicate the relevant steps and scope them to an Active and Backup channel, which would allow you to deploy to specific server sets from your runbook.

At the root of it, it sounds like what you’re hoping for is the ability to override target roles at deployment time to select specific machines. It’s also entirely possible I’ve misunderstood some or all of your setup - feel free to correct anything I’ve missed. Happy to keep discussing further!

4

Is it possible to deploy to all of the servers at once?

The use case we have is more of a “Blue / Green” style. The “backup cluster” is really the the currently inactive cluster that may need to be re-built, upgraded or otherwise worked on. When we are ready to swap it back as the active cluster, we need to ability to “True-up” the applications that are hosted on it. (By “True-up”, I mean we need to get the latest versions of all the Octopus deploy projects deployed to it.) Once we do that, we can swap the load balancer entries to make traffic start going to the “backup cluster”, thus converting it to the active cluster.

Because the backup cluster could be in the middle of being rebuilt during some of the deployments, concurrent deployments to both active and backup clusters is not a feasible solution.

Have you looked at specifying specific machines when you create a deployment?

As I understand it, when specifying a specific machine for a deployment, it has to be in the list of machines that the deployment would normally consider for the deployment. (Meaning that the MachineId has to be on the list of Deployment Targets created by specifying the the “On Behalf of” in the deployment’s process step.) Do to that, I would need to add both the Backup Cluster and the Active Cluster to the list of deployment targets.

If I put both clusters in the list of Deployment Targets, then, since I can’t deploy to the back up on each Deployment, I would have to customize each and every normal deployment to remove the backup cluster from the list. This is too much overhead for my operations team to have to take on. (And it is too error prone.)

However, if I am mistaken, and I can use the API to just add in any deployment target I like (regardless of if it is part of the step’s Target Role or not), then this would be a workable solution. Do you know if that is how it works?

Have you looked at using a separate role for your backup cluster?

I would be happy to create a separate role for my backup cluster. My question is, once I do that, how to use the API to target that role instead of the one that is build on the deployment step?

You could then duplicate the relevant steps and scope them to an Active and Backup channel

Duplicating the steps would be an explosion of work. Each and every project would then have two copies of the steps to maintain. Even with step templates, this would be very cumbersome. I am really hoping there is an API way to switch deployment targets.

At the root of it, it sounds like what you’re hoping for is the ability to override target roles at deployment time to select specific machines.

This is absolutely what I am looking for. The key part that is hanging me up is that it needs to be a role that is not built on the step for normal deployments and it needs to be done via the deployment API (if at all possible).

UPDATE:
I tried using SpecificMachineIds to select a deployment target that is not in the list of deployment targets that the “On Behalf Of” setting builds for the deployment. It did not work. It gives the following error:

You must deploy to at least one machine in the environment.

I went and updated one of my projects to have the backup cluster in its “On Behalf Of” setting, and then it worked fine (for that one project). But as I have stated above, this is not a viable solution since the backup cluster will not always be available.

Hoping there is some other way…

5

Currently, there’s no way to override the target roles via API. Those are scoped at the step level, and unfortunately they aren’t available to be bound to a variable at this time. This request has come up previously, but it’s not something we’ve committed to.

That said, I’ve got a couple of other ideas that may be useful in making this function how you’re hoping -

  1. One idea, using only your current process, could be adding the backup role to your main deployment and including a health check to your deployment process, which would allow you to skip targets that aren’t healthy (backup targets that are being rebuilt). After those standard deploys, you could still use your runbook to true up the deployments, and trigger those deployments via the API using just the SpecificMachineIds of your targets that missed the deployment the first time around.

  2. Following along the idea of a separate role for your backup clusters, you could programmatically enable/disable targets at deployment time. For example, you would have your deployment scoped to run on both the backup and active targets, but since the backups would regularly be disabled, your process would continue on as normal. Then, at backup deployment time:

    • Enable the machines in the backup role
    • Trigger your deployment via runbook with the SpecificMachineIds of your backup servers (which you can pull via the API to help make it smoother). They would be part of the expected deployment targets based on the role, which would allow you to deploy to them specifically reusing an existing known good release.

  3. The other option is still a little convoluted, but possibly workable. Instead of enabling/disabling targets, you would instead shuffle roles as part of your deployment, without having to add an explicit backup role to your deployment process. Your runbook would look something like this:

    • API query to get machines that are identified as backup targets (by naming convention)
    • Add the active role for the deployment to those identified machines
    • Trigger your deployment, using that list as the SpecificMachineIds
    • Remove the role from your backup targets so they aren’t deployed to again

There’s a little bit to unpack there, but should allow you to reasonably continue with your existing process and still leverage a more blue/green approach. Let me know your thoughts, interested to hear more about how you approach this.

1 Like
6

Option 3 looks like it will work out great!

Thank you for the help!

7

Glad to hear it! Feel free to reach out if you run into any issues and need a hand. Good luck and happy deployments!

8

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.