New variable change went to Production prematurely in unrelated Emergency Release

Hi,

I’m working on an application where we have a regular deployment channel that goes QA -> Staging -> Production. A month ago I had a new ticket in which I had to alter a variable in the web.config file. Since this variable would have different value in Production and the non-Production environments I created two different values for it in Octopus. The changes in this ticket were subsequently deployed to the QA environment as part of a new release and are now in the Staging environment.

In the meantime we had an Emergency ticket that required a fix to the same application in an unrelated area of the code. This was deployed via a slightly different channel QA2 -> Staging -> Production and went into Production last week. I have just realized that the updated Octopus variable has been applied to this Emergency release and is now in Production before it has been fully tested.

The root of this problem is that in order to set the Production value of the variable correctly for the regular release the variable needed to be created at the time of that release being created prior to deployment to QA (even though it might not hit Production for a long time). This then leaves it being open to being picked up by Emergency releases that are created from scratch in the meantime.

This seems like a common enough problem that there should be a recommended solution for it. However I don’t know what that is. Can anyone help and advise on this matter?

Hi Sperick,

Thanks for getting in touch,

I’m sorry to hear you experienced this issue, I understand this can be frustrating.

It’s definitely possible within Octopus to accommodate this,

Within the Project Variables, you are able to provide scoping to specific environments, for instance, you can set up a Variable that is only applicable to your Testing/UAT Environments, ensuring that deployments to your Production Environment do not use these variables unless you’ve updated the scoping of these variables.

I’ve included a screenshot below of some basic examples of Variable scoping to help highlight this;

In this example, the variable for Username provides unique values for the associated environments (i.e. OctopusUser1 for my Testing/UAT environments and OctopusUser2 for my Production environments.

It’s also worthwhile to mention, you can provide scoping based; Environments, Target Roles, Deployment Targets, Steps, and Channels.

I’ve linked below some documentation we have available on this topic, I’d recommend reading through this to familiarise yourself with the feature.

I hope this helps!

If I’ve misunderstood your query in any way or you require further assistance, please let me know :slight_smile:
Kind Regards,

Reece

Hi Reece,

Thank you for responding. I think I may not have been articulate enough in my first message. We have been using variable scoping within our project. We have this particular variable set up for different values for the different environments.

I’ll try instead explaining my issue using a simplified example. So for all of the environments except for Production let’s say this variable has a value of “B” set in Octopus. For the production environment then the value is set to “A”.

Ok so imagine that there’s a new ticket that requires this value to be changed so that in the non-Prod environments it will be set to “C” and in Production it will be set to “D”. This ticket is included in a Release along with some other tickets and the Project is Deployed to one of the QA environments where it will be tested for a few weeks before moving on to a Staging environment for another week before finally hitting Production. In order to create that build correctly in Octopus we needed to alter the value of the variables for both the Prod and Non-Prod environments. (This channel will go QA1 -> Staging -> Production)

Now in the meantime we get an unrelated Production issue that requires an immediate fix. We resolve this issue locally and create a new Release in Octopus to fast track through the environments. This channel will be QA2 -> Staging -> Production and will hit Production within 3 days.

The problem here is that since we have already changed the value of the variable to “D” for the Production environment in Octopus for this Project this ends up being picked up in the Emergency release because the emergency release was created AFTER the regular release was created. This value for the variable then hits the production environment by accident before the date that it was meant to have changed (3 days instead of 3 weeks).

I hope that example makes more sense to you.

Hi Sperick,

Thanks for getting back to me regarding this,

I appreciate your time putting this information together, this really helps out!

In regards to handling this current situation, since you’ve outlined that you already have unique channels for handling Production fixes (QA2 -> Staging -> Production) and a channel for value changes (QA1 -> Staging -> Production) you could opt to have multiple variable values scoped to these respective channels.

For instance, having your QA2 channel be scoped to ‘B’ which is your already working/tested value and having the QA1 channel be scoped to the ‘D’ value, in doing this you’ll be able to rollout Production fixes without introducing untested variables to the Production environment.

I hope this helps!

Please let me know if this is helpful :slight_smile:

Kind Regards,

Reece