I updated from Octopus 4.1.0 to 2019.6.3 LTE, and now the Octopus service crashes on startup. The following error is logged:
2019-07-08 17:07:02.0791 616 7 FATAL Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at Octopus.Core.Initialization.BuiltIn.EnsureServerIsNotRunningAgainstAnIncompatibleDatabase.Initialize(OctopusServerInstallationHistory latest, NuGetVersion applicationVersion)
at Octopus.Core.Initialization.BuiltIn.EnsureServerIsNotRunningAgainstAnIncompatibleDatabase.Initialize(IRelationalStore store)
at Nevermore.StoreInitializer.Initialize()
at Octopus.Server.OctopusServerEngine.Start()
at Octopus.Server.Commands.RunCommand.Start()
at Octopus.Shared.Startup.AbstractCommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions)
at Octopus.Shared.Startup.WindowsServiceHost.<>c__DisplayClass1_0.b__0()
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService()
I have tried rolling back, and installing other versions. Nothing seems to get Octopus into a usable state. Now our pipeline is broken and everyone is mad at me for doing an upgrade.
I hacked the database and fixed it. I did a clean install of 2019.6.3 and used that DB as a functional source-of-truth.
In the table OctopusServerInstallationHistory, the Node column was empty on all rows. I set this to the name of the host machine, and it didn’t seem to help.
One of the rows in the table had the Version column set to null. I took this value from the clean install source-of-truth: “2019.6.3+Branch.tags-2019.6.3.Sha.309fb85ffe2054c1803cb3b67dfdca7aceefd88c”
Basically, I ran this SQL command: UPDATE [OctopusServerInstallationHistory] SET Version = ‘2019.6.3+Branch.tags-2019.6.3.Sha.309fb85ffe2054c1803cb3b67dfdca7aceefd88c’ WHERE Version IS NULL
Once this was done, there were no null values in the table, and Octopus started without error. Hopefully I didn’t create any other problems.
Hi Nick, sorry to hear you were running into this,
We have managed to reproduce this issue and the fix will be available as part of 2019.7.1 with patches for both LTS versions forthcoming.
In regards to the query used to fix the problem; This should not cause you any problems, however it may have skewed the installation history somewhat and although may not be a problem right now could possibly be an issue in the future. In the future, if you are able to take a backup of the database prior to upgrading, you will be able to roll back if there is a problem with the upgrade, allowing your team to keep working and taking pressure off you.
I was focused on finding and fixing the root issue, and now I can see you were working on your end to solve the problem. Is there a way I could have done a better job to help you, short of “don’t ship bugs”?