Hi.
I’ve been using Octopus Deploy for quite some time and I’ve noticed that quite often the updates for the software, has show stopper bugs.
For the most part the bugs has consisted of user permissions bugs, where the user can have permissions before the update and vanished after. In some cases the same permission bugs are back in a later update. My colleagues have begun to point to the software itself when something goes wrong, by default. In a response to that frustration, I rewrote the “Upgrade Octopus Server” Community Step Template so it could install a specific version, just to make it easier to downgrade in case of a major bug.
Therefore I’m wondering if you are running integration tests before you release a new version? Is there at any point a version of Octopus Deploy that is considered to be stable? Is there something I should think about before upgrading to the latest version of Octopus Deploy?
Thanks for getting in touch! We have noticed the same thing, and we are disappointed in the result on your behalf. I’m personally very sad to hear the reaction of your colleagues too.
Yes, we do run automated testing across the whole of Octopus, right up to complex deployment orchestrations. We have a high level of confidence in every release that our deployment engine is solid. One of the areas we don’t have fantastic coverage yet is permissions.
We are in the middle of rebuilding our entire permissions system as part of a larger project. The permission system in Octopus is very flexible and powerful, but it is also too difficult for both our customers and our developers to understand and implement reliably. We don’t have thorough test coverage because permissions is so complex with so many permutations. We are deliberately fixing that now, and in the process we are bound to uncover some latent bugs, and even make some mistakes along the way.
I don’t want to offer a hollow promise of perfection, but what I can promise is we learn from our mistakes. When we find a bug, we red-green refactor so the same bug never happens again without us knowing. When we find a series of similar bugs, we deliberately hunt down the root cause and make changes to stabilise that area. Sometimes this means we need to take risks, and that’s what we are doing with the permissions system work. We absolutely don’t want to hand that risk on to our customers, but some mistakes will inevitably slip through.
In the meantime I would recommend deciding on an upgrade strategy which suits you best. Some of our customers choose the latest patch of the previous monthly release and upgrade monthly. We try to provide the right level of detail in our release notes so you can see what has changed in each release/patch.
Eventhough Octopus sometimes comes with critical bugs, I doubt any of my collegues wants to go back to manually deploying f.ex. BizTalk.
I’ve decided to go with your advice; to just update Octopus server with the latest patch of the previous monthly release. Till now, I’ve updated more or less the same day a new patch has come out. Now that more people is depending on the software, updating several times a week is not a good strategy.
I do notice that you are quick to send out fixes, so bugs aren’t that major of an issue, however it depends on how often the person in charge of updating Octopus Deploy is available to do so.
If I were you, I’d look into identifying a current release that can be sort of recognized as officially stabile. As it is now, customers do not know the difference between a 0 or a 7 release number. Actually, most people will download the latest version, which in worse case can be a 0 version that might contain a serious bug. The quick solution could be to add a new Json feed which only gets the latest patch from any previous major release. In Octopus Deploy, you can add a feature for the update notifier which lets the administrator to be alerted about either the latest version or the latest stabile version.