Versioning variable snapshots: hotfixes and feature flags

Context

  • At any one time, our dotnet 6 web app has several feature flags in its appsettings.json
  • When the feature is ready and tested in the test environment, we change the Octopus variable for the live environment to “true” and re-snapshot the variables.
  • If an issue was noticed with the feature just after release, we would disable the feature flag and redeploy.
  • Occasionally we need to hotfix which involves: Branch from the currently live code, make a very small change, push that through the environments all the way to live.

The convenience of being able to toggle a variable without going through the build process and all environments is key. But we need to be able to keep track of some settings and ideally actually move them through environments.

Problem

The hotfix is a new release so it takes the latest config. Hence an old version of the code has several incomplete features activated. Worse still, it’s very hard to tell which ones have changed from within Octopus. It’s even harder when those variables are within library sets.

Ideal solution

Octopus UI reads/writes chosen variable values to the relevant branch in git.
It seems like there’s a chance this will happen in a future early access build Configuration as Code - Octopus Deploy
However, I need a solution that works robustly right now…

Other solutions

It seems really weird in an immutable release based system that I can’t find a way to say “create a new release from that immutable release with one change”.
Does Octopus allow me to clone a release, but specify one thing to change? e.g. Just change the referenced package version, keep everything else i.e. process/variables the same.
Does Octopus allow me to specifically diff, or copy the snapshot of variables it takes for a release?

I’d be happy to somehow segregate these feature flag variables that essentially move through environments from others (e.g. DB connection strings) that don’t.

Hey @graham.helliwell , thanks for reaching out and welcome to the forum!

You gave a lot of information and context around your approach (Which I appreciate!), I’m going to try and relate each of the important points in context with Octopus as a product overall.

Feature flags are definitely a very popular tactic for managing features that are in development. They make it easy to implement A/B testing, as well as keep non stable code safely built into the codebase without exposing it to users, which is incredibly helpful as teams try to become more and more agile with their deployment cycles.

The overarching concept here is that fundamentally, Octopus isn’t intended to be a tool for managing feature flags. This is due primarily to a lot of the things you outlined in your OP -

The convenience of being able to toggle a variable without going through the build process and all environments is key. But we need to be able to keep track of some settings and ideally actually move them through environments.

Octopus’ ultimate goal is to make a deployment easily repeatable and reliable. This is accomplished at the base level by creating release snapshots that dictate what the release should look like so it can be applied consistently to any and all other deployments of that release.

However, I do see where you’re coming from - it would be nice to be able to “branch” your deployment from an existing release to take advantage of the state of the variables at a known point in time. Unfortunately, that’s not how Octopus is built to function today. I did some local testing to see if it was possible to alter a Release after creation to update the ProjectDeploymentProcessSnapshotId , LibraryVariableSetSnapshotIds , ProjectVariableSetSnapshotId properties to instead point to the sets from prior releases, but the API skips over those properties in the PUT request body.

Given the above, there are two general approaches you can take moving forward.

The first, and our current recommendation, would be to invest in a discrete feature flagging tool. LaunchDarkly is the easy one to recommend due to familiarity, but there are other options in the space as well, including CloudBees, Split, and even some Open Source tools that have been built for this purpose. The benefit you’ll get is that Octopus is a great integrator with feature flagging tools - for example, we have a community contributed step template for managing LaunchDarkly feature flags. This type of mechanism with a step template interacting with a third party service that supports your application’s lifecycle is very common in the world of Octopus, and community step templates are just one of the ways to approach it.

The second approach would be to reconsider how you’re mapping your feature flags in Octopus. This can look like a couple of different things:

  • Like you mentioned in your OP, the next steps for Configuration as Code will include non sensitive text variables as an applicable item to be stored in Version Control. This will have some benefits - chiefly, if you deploy from branch feature-test-1 that has feature flags enabled, you can still deploy from the tip of your main branch without having to manipulate the variables. When you merge in feature-branch-1, you’ll update your variables at that time, and your successive deployments from main will continue to match what’s expected. However, this will rely heavily on your branching strategy, and (at least initially) isn’t planned to cover library variable sets.
  • Another option that leverages your existing approach for library variable sets would be looking at the Octopus Deploy Terraform Provider. The Terraform provider would allow you to version control your library variable sets, giving you a clear line of accountability on the state of your variables and enabling you to see previous values and “roll back” those values as needed for hot fix deployments.
  • A final option to consider would be organizing your variable sets to be as explicit as possible for feature flagging. For example, if you’re following the documentation recommendation to use channels for your hotfix deployments, you can scope your feature flag variables to be true/false per environment AND channel, allowing you to more granularly control the behavior.

Like I said initially, Octopus isn’t the best tool for managing feature flags at the moment, due to a primary misalignment with the approach and purpose of the tool as designed. Hopefully the options I’ve laid out offer you a path forward to enable better/simpler feature flag management - if you have any other questions or concerns, don’t hesitate to ask. Happy to help however we can!

Thanks for the full answer here. I hadn’t thought of checking if the API allowed me to do more than the UI for this.

To clarify, my feature flags are currently never in library sets. I mentioned that more in reference to it being virtually impossible to find the difference between two variable snapshots within Octopus (making it terrifying to overwrite the existing snapshot, especially with no way to go back).

When I “update variable snapshot” right now it actually violates the repeatability (for the sake of convenience - fair enough). It’s a really common pattern with immutable models to say “clone but with this change”. Aside from my main issue, it’d be great to even just be able to “clone release with latest variables” - that would be more repeatable since I could deploy the original or new one. It sounds like I could probably manage that through the API right now, but since it sounds very in line with Octopus’ ultimate goal so may be worth future UI consideration.

Knowing that there are snapshot ids available but ignored makes it seem even more of a shame to have the mental overhead of a whole other tool to manage what’s already in Octopus.

In the medium term I believe the Configuration as Code work is likely to cover my needs. Hence in the meantime I will likely either use channels (which feels a bit manual and error prone), or integrate the most lightweight of the libraries/tools you mentioned.

Thanks for all your help and suggestions,
Graham.

Thanks for the feedback Graham, it’s greatly appreciated!

For what it’s worth, I’ve brought your use case up in internal channels for further discussion. DevOps is contextually driven, so one feature flag + Octopus use case will very rarely match another. Having more data points means we have more input to consider as we shape the next phase of Octopus and how it integrates within the CD pipeline.

If you run into any trouble integrating your solutions, feel free to reach out anytime either here or via advice@octopus.com - I know that implementing backup plans can be frustrating, we’re more than happy to help you overcome any of those pain points as you meet them!