Upgrade HA Multi Node Setup Octopus Deploy Server

Legacy Operations
1

Hi,
We are currently running Octopus Deploy Servers in our production environment with HA multi node setup. Below are our HA configuration.

2 X Windows 2012 Servers (Both nodes running Octopus version V2018.3.7)
1 X SQL Database Server
1 X SAN Drive Network share path.
Load Balancing via F5.

We are currently running Octopus Deploy Version V2018.3.7 in both Nodes.
We are trying to upgrade both nodes to Version v2018.3.8.
Our question is:
We want to upgrade one node at a time, ensuring our customers can still use our Octopus Deploy Service while we are upgrading. That is zero downtime. Is this possible to do this with the Octopus HA setup we currently have in our production environment? Will Octopus get confused if 2 different versions are running on both nodes during the upgrade time?

To achieve this, we are planning to carry out the below tasks.

  1. From Admin web portal, drain only the ‘Primary(Leader)’ node. Disable Primary node drain. This will ensure the ‘Follower’ node will become the primary node running v2018.3.7.
  2. Log into Octopus server which is NOT the primary. Stop octopus windows service.
  3. Run the latest Octopus Deploy Installer v2018.3.8. This will upgrade the in-active node to v2018.3.8.
    At this point, the Other primary node is still online and running v2018.3.7.
  4. Bring up the upgraded node. Not sure this is possible or not since now we have 2 different versions running in both nodes.
  5. Drain and make the other node in-active and upgrade the second node to v2018.3.8
    At this point, the previously upgraded node would have become the leader node and will be running V2018.3.8.
  6. Bring up the second upgraded node. Now both nodes should be running the same version V2018.3.7
3

Hi Surendba,

Thanks for getting in touch!

Unfortunately upgrading individual HA nodes with no downtime is not a feature that we currently have available (although we are definitely working on this). I have spoken to the team that are running our Octopus Hosted Alpha and they run into a similar problem (how do we upgrade our instances with minimal downtime) and they have a very similar approach to what you are proposing:.

  • Put it into maintenance mode, drain tasks
  • Take first node offline (in the load balancer, making it invisible to clients)
  • Upgrade first node
  • rejoin first node
  • Immediately take node 2 offline in the LB
  • Upgrade node 2
  • rejoin node 2

There will be a small amount of “maintenance” time as the schema upgrades. The newer node will upgrade the DB schema, the old one might get a bit confused. The UI will probably freak out a bit but this will be resolved by a quick refresh post upgrade.

Please let me know if you have any other questions or queries,

Regards
Alex

4

Do we have a better process now?
This process is still not documented, only for windows update.
or I am missing the page for this process in a HA octopus setup.

5

Hi @AndersAndersson,

That is still the current process, and you are right that we don’t have an official doc’s page on this process.

I’ll get that sorted as soon as I can,

Regards,
Alex