I am interested in peoples thoughts on the best strategy for deploying config updates…
We have a tightly controlled dev->uat->prod->training workflow environment and I am setting up Octopus to deploy using that workflow. We are finding that particularly during the initial setup or .net version upgrade our deployed applications often fail because of config transform bugs. It’s a bit of a pain having to rebuild (tfs) and deploy the whole package just to tweak a few config settings. And to have to push it back through the workflow.
Once our packages are successfully approved in Prod we then promote it into the training environment, if we have a config issue at this point it seems crazy to have to re-commit and deploy to dev then uat and prod before we can get the fix in training. A deploy to prod would mean our business testers would have to retest the entire application, even though the update has nothing to do with production. (note our training environment has to be exactly the same code as prod, hence our workflow rules)
At this point I am thinking we will cheat and directly edit the config files on the servers if we have an issue. But is there a better way?
Thanks for getting in touch! The first thing I would mention would be the following line. We are finding that particularly during the initial setup or .net version upgrade our deployed applications often fail because of configuration transform bugs.
What kind of bugs are you experiencing when you are running your transformations for the initial setup / upgrade?
Unfortunately, there is no simple way to “Update” the configuration transformations once they are deployed, you are limited with your options. We have a few ideas below however.
Ideally, fixing the configuration transformation so that it doesn’t fail would be the best idea to have things running smoothly. Seeing as you have such a tightly controlled Lifecycle the other options I can suggest may not be viable.
The first suggestion would be to consider creating another channel. You could configure this channel to be a hot-fix channel so that if/when there is an issue with the transformation, you can quickly redeploy following the channels Lifecycle.
We have some great documentation on how you can configure channels and use them to deploy hot-fixes: http://docs.octopus.com/display/OD/Channels
The next suggestion covers what you could do if you had variable issues causing the transformations to break. You are able to update the variables in the project and force an older release to have its snapshot replaced with a new snapshot containing those variables. This feature is only intended to be used as a last resort as it has the potential to break stuff that was already tested.
See screenshot1 to show you how you can update the variable snapshot for a previous release.
Lastly, if the configuration files in the package are wrong, you would need to delete the package from your feed and caches, and replace them with the fixed packages. This is not at all a good practice in Octopus and should only be used if absolutely necessary.
Each of these ideas requires a new release to be created for them to take effect. So unfortunately there is no quick automated update / fix for your scenario here. As I stated above though, if you were to provide us with some more information around the issues you are having with the transformations, we may be able to help you get it running smoothly with the initial deployment.
Yeah all your suggestions sound like what we were considering. I think probably having a hotfix channel/workflow so we can directly apply patches might help us.
In answer to your question on what sort of bugs we have, it’s mainly because we can’t fully test our prod transformed config file changes until they are deployed live, so sometimes we miss something, a wcf endpoint setup wrong or an api path wrong. We have automated integration tests that pickup an issue. Updates have to be done outside business hours so counter staff have access to the applications, so we are always on a tight timeframe.
Before octopus we would work with operations to manually fix up the config settings. We need a way to do this without going back to square one and re-building and deploying up the full lifecycle.
A couple of other ideas I have had are:
Setup a separate visual studio project that just contains config files not all the code, setup a separate channel that just deploys the config files from this project.
Take a copy of config files from the octopus packages during deploy and put them in a temp backup area, if they need fine-tuning, we could do them from this area.
Our TFS build server is part of the issue, it’s pretty slow. Are there better ways we could push files into a release if a build is not required? Perhaps a powershell script.
Thanks for getting back! I’m sorry for the late reply here. I ran this past some of our developers to see if there were any options available for you here. It looks like the best option to perform what you need here is to use channels as per my initial response. However, there are a couple of things that were brought up that may help you have things running a bit smoother.
The first thing to mention would be the performance of your TFS server:
Our TFS build server is part of the issue, it’s pretty slow
Having a fast build process will really help here. Increasing the performance of the TFS build server should make a noticeable difference.
Another idea, seeing as you have such a short deployment window and a long build time. You could even try refactoring your settings so that they’re editable from an admin section of your application UI to avoid a redeploy
The final thought is that you could try using a blue/green deployment strategy to enable you to deploy at any time.
I won’t talk too much about Blue/Green in this ticket but I will link you to the documentation if it interests you.
