I’m aware that Octopus wants to treat releases and their snapshots (including variables) as immutable artifacts. However, you have ancillary things that are part of the deployment process and not the application configuration that should always be up to date.
A common example of this is integration credentials, especially ones that rotate regularly. You want to rollback to an older release, with its snapshotted configuration, but you can’t because some credentials you need during the deployment have rotated. You could “Update Variables” in the release, but that is an all or nothing approach.
We have been bumping into this pain point more and more frequently since we’ve upgraded our security posture. We regularly rotate credentials and the updated credentials are automatically pushed to Variable Sets/Project Variables, but you run into the scenario above.
What is the suggested workaround to this? We are long time users of Octopus, but there might be a feature we are missing? Is there any appetite for a subset of variables that are snapshotted/captured for each deployment, rather than each release?
Thank you for contacting Octopus Support. I’m sorry to hear about your frustrations with Octopus Variables.
Have you considered using an orchestration Project in order to “section out” the different components of your deployments that may have different requirements? We have an excellent blog post on doing this here: Better release management with Octopus Deploy - Octopus Deploy
Obviously you would likely need to re-tool your current setup by adding these additional layers. However, this should provide you the freedom to maintain Releases and update Variables as needed with a more surgical approach.
Let me know if you have any additional questions about this or anything else.
Unfortunately, breaking up the deployment into smaller deployments doesn’t really solve this problem.
To clarify, lets say for example we have the following variables:
Project.Deployment.AclToken -> Required to deploy, rotates frequently
Project.SomeConfig1 -> App config, might change between releases
Project.SomeConfig2 -> App config, might change between releases
...
You can see that snapshotting Project.Deployment.AclToken as part of a Release is actually counterproductive, you will end up having to Update Variables every time you want to rollback, which can have unintended configuration consequences (updating variables that you do want from the snapshot).
You would want Project.Deployment.AclToken to always be the latest value, it has a very different change cadence to the rest of the variables.
In a CI pipeline runner, this would be the equivalent of a Secret or Environment Variable that is accessible to a pipeline and is always up to date.
Thanks for providing that additional clarification around this pain point - I’m just stepping in for Donny as he has gone offline for the day, but I’m happy to help!
To get the best guidance on this, I have escalated your request to our engineering team for a deeper review and will let you know their perspective and potential recommendations for this scenario.
Our team will be in touch soon via this thread, but let us know if we can be of any more help in the meantime.
Thank you for your patience. I’m jumping in for Donny and Britton to pass along the input from the engineering team.
Our suggested approach to handling rotating credentials in this case, as you correctly pointed out release and variable snapshots are immutable and updating the variable snapshot is all or nothing, is to use a secret management tool like Azure Key Vault or AWS KMS and retrieve them as part of the deployment process. We have a blog post around using Azure Key Vault in Octopus with the library step Azure Key Vault - Retrieve Secrets, in case that helps.
Let us know what you think, or if you have any further questions or concerns.
Storing the rotating credentials/tokens in Azure Key Vault/AWS Secrets Manager presents a similar problem. Octopus would need credentials to retrieve those values from the cloud vaults, those credentials should rotate regularly.
Now I’m aware there is the Accounts feature in Octopus for Azure/AWS accounts. I haven’t tested it yet, but do these get snapshotted as part of a release or do they always use the most up to date credentials stored in the account?
Accounts are only referenced in Snapshots/Releases. If the credentials for the account are changed after the Snapshot/Release was created, the Deployment/Runbook will still use the current credentials for that Account. You can see this in the JSON when looking at a Variable set as the Value for the Variable is the AccountID.
That does raise the question then, why not allow Username/Password and Token Account types to be used in variables? Variables are restricted to using AWS/Azure/GCP accounts.
I can certainly see the benefit to adding this functionality to Octopus. If you have a moment, I recommend submitting this suggestion via the + Submit idea button on the top right of our Roadmap page.
Suggestions made here go straight to our Developers and are the best way to have features considered for future versions of Octopus.
Let us know if there is anything else we can assist with.
