We have a 2 node HA environment
after upgrading both nodes to 3.15.5 it reports that there are 4 nodes and says we are in violation of the license. See attached file.
As you can see each node is showing up twice. I checked each node and there is only the default instance
Please advise
Hi Leslie,
Sorry this has occurred. We made a change that re-formats the nodes ID which causes issues in HA if the nodes are connected while one has upgraded. To resolve the duplication you will need to delete some records from the database. The OctopusServerNode will have records with OctopusServerNode-name as the ID and records with just name as the ID. Delete the rows with just name as the ID.
Vanessa
In an attempt to fix it I deleted 2 nodes in the UI.
Doing a select from that table this is what I see
Id Name LastSeen Rank JSON MaxConcurrentTasks IsInMaintenanceMode
OctopusServerNodes-PRD-OCTFPV01 PRD-OCTFPV01 2017-07-13 04:15:44.1715964 +00:00 Follower {} 5 0
OctopusServerNodes-PRD-OCTFPV03 PRD-OCTFPV03 2017-07-13 04:28:37.1039401 +00:00 Leader {} 5 0
Does that mean it should be ok?
Hi Leslie,
You responded as the same time as me. I assume everything has been running smoothly since. The data in this table was updated to have a new ID, as you had nodes running while the data was upgraded they could not find their record any longer and created a new one. Our recommendation when upgrading is to have all services stopped as at minimum your UI could throw errors due to a database mismatch, and at worse corrupt data on save if the schema changes underneath the node.
We are sorry this occurred. It was not a scenario that was tested.
Vanessa
Hi Vanessa,
We also experienced and issue here and were able to resolve with the information provided. I see you recommend stopping all services prior to upgrading. That is not in the current upgrade instructions for HA instances. If that has changed, can the documentation be updated?
One of the awesome aspects of using Octopus are the frequent bug-fix and feature releases. Our production Octopus consists of a 2 nodes in HA. It might be the most common HA setup for those that purchased it (wild assumption on my part). Are there any plans to expand testing to ensure HA upgrades are problem free?
Thanks, love Octopus, but our upgrade issue today left us a little upgrade-shy.
Hi William,
We are in the middle of updating the documentation and our recommendations. This issue brought a few things to light internally as a few of us had different interpretations of both the process and the documentation. We aren’t happy it occurred this way, we really are sorry it happened.
My wording was a little clumsy. We do test HA and upgrades, but an assumption was made and both nodes were offline when it was tested - that was our error. We are going to absolutely recommend that for HA upgrades where a major or minor version number changes all nodes are offline to allow for database schema changes, then the lead node that was updated first can be on while the others are all upgraded. They will run much faster as the DB portion will already have been run.
Making our upgrades much smoother has been a goal and we are all disappointed we missed this, and do hope you will gain your confidence back.
I am happy to update this thread when the documentation is completed if you like.
Vanessa
