Server stops immediately after starting on upgrade

zeiddev · 13 January 2020 08:59

I just upgraded to the latest version and when I go to the Octopus Manager it says that the Octopus Windows Service is currently stopped. When I try to start it, it appears to start then stops straight away. Looking at the logs I get the following:

2020-01-13 08:48:50.6183 5556 6 FATAL Unhandled AppDomain exception occurred: Unable to determine type to deserialize. FeedType BuiltIn does not map to a known type
System.Exception
at Nevermore.Serialization.InheritedClassConverter1.GetTypeInfoFromDerivedType(String derivedType) at Nevermore.Serialization.InheritedClassConverterBase2.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Nevermore.RelationalTransaction.Stream
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at Octopus.Core.Initialization.BuiltIn.BuiltInFeedInitializer.EnsureFeed(IRelationalTransaction transaction)
at Octopus.Core.Initialization.BuiltIn.BuiltInFeedInitializer.Initialize(IRelationalStore store)
at Nevermore.StoreInitializer.Initialize()
at Octopus.Server.OctopusServerEngine.Start()
at Octopus.Server.Commands.RunCommand.Start()
at Octopus.Shared.Startup.AbstractCommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions)
at Octopus.Shared.Startup.WindowsServiceHost.<>c__DisplayClass1_0.b__0()
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

Any suggestions?

Thanks

David

Justin_Walsh · 13 January 2020 15:14

Hi @zeiddev.

Thanks for getting in touch, and sorry to hear you’re encountering this follow your upgrade. I’ll get to work trying to replicate this on my end, can you let me know what version you installed? Just want to make sure that I can reproduce it as accurately as possible. The latest LTS is 2019.9.12 and the latest Fastlane is 2019.11.3

Look forward to hearing from you soon.

zeiddev · 13 January 2020 15:33

Hi,

Now I am not sure that I downloaded the correct version. When I looked at the file name it said Octopus.2018.10.4-x64.msi which seemed odd so I went to download it again and got 2019.9.12-x64 which seems more reasonable.

I went to run that one and got the following errors instead:

Using database connection string from configuration file: data source=(local)\SQLEXPRESS;initial catalog=Octopus;integrated security=True
========== SQL Database Connection Details ========
SQL Server: (local)\SQLEXPRESS
SQL Database: Octopus
Authentication Type: Windows Integrated Security
User: EC2AMAZ-7160HGQ\Administrator

===================================================
Preparing to upgrade the database ‘Octopus’ on SQL Server at ‘(local)\SQLEXPRESS’ to the expected schema running as ‘EC2AMAZ-7160HGQ\Administrator’…
Making sure it’s safe to upgrade the database schema…
Checking to see if database schema upgrade is required…

Database schema changes are required. Execution plan:

Planned pre-upgrade scripts (non-transactional):

Octopus.Core.UpgradeScriptsPre.Script0000 - Database options.sql

Planned schema upgrade scripts (single transaction for all scripts):

Octopus.Core.UpgradeScripts.Script0188 - Delete Artfifact RelatedDocumentIds.sql
Octopus.Core.UpgradeScripts.Script0191 - PackageMetadata - Create table.sql
Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.cs
Octopus.Core.UpgradeScripts.Script0194ConvertMachinePolicies.cs
Octopus.Core.UpgradeScripts.Script0194NormalizeVCSRootURLs.cs
Octopus.Core.UpgradeScripts.Script0195 - Add DataVersion column to spaces.sql
Octopus.Core.UpgradeScripts.Script0195SetWorkerPoolType.cs
Octopus.Core.UpgradeScripts.Script0196 - Team and UserRole DataVersion.sql
Octopus.Core.UpgradeScripts.Script0197PreSeedFeedKeyAllocationTable.cs
Octopus.Core.UpgradeScripts.Script0198 - Remove ProjectGroupId from DeploymentResource.sql
Octopus.Core.UpgradeScripts.Script0200 - Add BlockConcurrentDeployments to interruption.sql
Octopus.Core.UpgradeScripts.Script0201EnsureSystemAndSpacePermissionAreCorrect.cs
Octopus.Core.UpgradeScripts.Script0202EnsureArtifactScopesAreConsistent.cs
Octopus.Core.UpgradeScripts.Script0202EnsureMachineHealthCheckPoliciesAreNotNull.cs
Octopus.Core.UpgradeScripts.Script0203 - Add OS details to machine table.sql
Octopus.Core.UpgradeScripts.Script0203 - AddClonedFromTenantIdColumnToTenants.sql
Octopus.Core.UpgradeScripts.Script0204CopyOSDetailsFromHealthCheckJson.cs
Octopus.Core.UpgradeScripts.Script0205 - Add OS details to worker table.sql
Octopus.Core.UpgradeScripts.Script0206CopyOSDetailsFromWorkerHealthCheckJson.cs
Octopus.Core.UpgradeScripts.Script0207 - Set OS Detail Columns To Unknown When Null.sql
Octopus.Core.UpgradeScripts.Script0208AddLTSLicenseHashColumnToMachines.cs

Planned post-upgrade scripts (non-transactional):

Octopus.Core.UpgradeScriptsPost.Script0001 - Rebuild indexes.sql
Octopus.Core.UpgradeScriptsPost.Script0002 - Refresh IdsInUse view.sql

Ensuring pre-conditions for upgrading the database are satisfied…
Searching for indexes that might upset the database upgrade process…

PASS: All columns use the default collation.
PASS: Your Octopus Server will be compliant with your license after upgrading.
PASS: We’ve done our best to remove any unexpected database indexes.
====================================================================================
Starting database schema upgrade…
====================================================================================

Executing pre-upgrade scripts…
Executing TSQL Database Server script ‘Octopus.Core.UpgradeScriptsPre.Script0000 - Database options.sql’
Enabling read committed snapshot isolation on Octopus
Read committed snapshot isolation is already on
Executing schema upgrade scripts…
Beginning transaction
Checking whether journal table exists…
Executing TSQL Database Server script ‘Octopus.Core.UpgradeScripts.Script0188 - Delete Artfifact RelatedDocumentIds.sql’
Checking whether journal table exists…
Executing TSQL Database Server script ‘Octopus.Core.UpgradeScripts.Script0191 - PackageMetadata - Create table.sql’
Executing .NET Database Server script ‘Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.cs’
Error occured when executing scripts, transaction will be rolled back
Error: Upgrade failed due to an unexpected exception:
Error: System.ArgumentNullException: Value cannot be null.
Error: Parameter name: value
Error: at Newtonsoft.Json.Linq.Extensions.Value[T,U](IEnumerable1 value) Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.UserRoleData.get_GrantedSpacePermissions() in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 71 Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.RoleNeedsUpgrading(UserRoleData role) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 36 Error: at System.Linq.Enumerable.WhereListIterator1.MoveNext()
Error: at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
Error: at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.Execute(IDbCommandExecutor commandExecutor, IUpgradeLog log) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 24 Error: at Octopus.Core.Initialization.DbCommandUpgradeScriptResolver.Execute(Func1 commandFactory, IUpgradeLog log) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DbCommandUpgradeScriptResolver.cs:line 25
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.<>c__DisplayClass6_0.b__0(Func1 commandFactory) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 319 Error: at DbUp.Engine.Transactions.SingleTrasactionStrategy.Execute(Action1 action)
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.ExecuteDbCommandUpgradeScript(IConnectionManager connectionManager, IJournal journal, IUpgradeLog log, IDbCommandUpgradeScript upgradeScript) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 307
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.Execute(SqlScript script, IDictionary2 variables) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 287 Error: at DbUp.Engine.UpgradeEngine.PerformUpgrade() Error: We encountered an error during the schema upgrade. The schema upgrade was stopped and rolled back. Don't worry, this means we haven't made any changes to your data, and you should be able to get back up and running quickly. Error: Here is the summary of the problems we encountered: Error: ----------------------------------------- Error: Value cannot be null. Error: Parameter name: value Error: System.ArgumentNullException Error: at Newtonsoft.Json.Linq.Extensions.Value[T,U](IEnumerable1 value)
Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.UserRoleData.get_GrantedSpacePermissions() in Script0192 - Add PackageMetadataPush Permission.cs:line 71
Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.RoleNeedsUpgrading(UserRoleData role) in Script0192 - Add PackageMetadataPush Permission.cs:line 36
Error: at System.Linq.Enumerable.WhereListIterator1.MoveNext() Error: at System.Collections.Generic.List1…ctor(IEnumerable1 collection) Error: at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.Execute(IDbCommandExecutor commandExecutor, IUpgradeLog log) in Script0192 - Add PackageMetadataPush Permission.cs:line 24
Error: at Octopus.Core.Initialization.DbCommandUpgradeScriptResolver.Execute(Func1 commandFactory, IUpgradeLog log) in DbCommandUpgradeScriptResolver.cs:line 25 Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.<>c__DisplayClass6_0.<ExecuteDbCommandUpgradeScript>b__0(Func1 commandFactory) in DatabaseUpgrader.cs:line 319
Error: at DbUp.Engine.Transactions.SingleTrasactionStrategy.Execute(Action1 action) Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.ExecuteDbCommandUpgradeScript(IConnectionManager connectionManager, IJournal journal, IUpgradeLog log, IDbCommandUpgradeScript upgradeScript) in DatabaseUpgrader.cs:line 307 Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.Execute(SqlScript script, IDictionary2 variables) in DatabaseUpgrader.cs:line 287
Error: at DbUp.Engine.UpgradeEngine.PerformUpgrade()
Error: -----------------------------------------
Error: Beginning transaction
Error: Checking whether journal table exists…
Error: Executing TSQL Database Server script ‘Octopus.Core.UpgradeScripts.Script0188 - Delete Artfifact RelatedDocumentIds.sql’
Error: Checking whether journal table exists…
Error: Executing TSQL Database Server script ‘Octopus.Core.UpgradeScripts.Script0191 - PackageMetadata - Create table.sql’
Error: Executing .NET Database Server script ‘Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.cs’
Error: Error occured when executing scripts, transaction will be rolled back
Error: Upgrade failed due to an unexpected exception:
Error: System.ArgumentNullException: Value cannot be null.
Error: Parameter name: value
Error: at Newtonsoft.Json.Linq.Extensions.Value[T,U](IEnumerable1 value) Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.UserRoleData.get_GrantedSpacePermissions() in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 71 Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.RoleNeedsUpgrading(UserRoleData role) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 36 Error: at System.Linq.Enumerable.WhereListIterator1.MoveNext()
Error: at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
Error: at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.Execute(IDbCommandExecutor commandExecutor, IUpgradeLog log) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 24 Error: at Octopus.Core.Initialization.DbCommandUpgradeScriptResolver.Execute(Func1 commandFactory, IUpgradeLog log) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DbCommandUpgradeScriptResolver.cs:line 25
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.<>c__DisplayClass6_0.b__0(Func1 commandFactory) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 319 Error: at DbUp.Engine.Transactions.SingleTrasactionStrategy.Execute(Action1 action)
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.ExecuteDbCommandUpgradeScript(IConnectionManager connectionManager, IJournal journal, IUpgradeLog log, IDbCommandUpgradeScript upgradeScript) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 307
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.Execute(SqlScript script, IDictionary`2 variables) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 287
Error: at DbUp.Engine.UpgradeEngine.PerformUpgrade()
Error:
Error: -----------------------------------------
Error: You have two options from this point:
Error: OPTION 1: Re-run the installer for the version of Octopus Server you were using just before trying this upgrade. Your data hasn’t been changed and everything will go back to how it was beforehand. Please send this log to support@octopus.com and we will work with you to get you upgraded.
Error: OPTION 2: If you know what the problem is and how to fix it, please fix the problem and then try the upgrade again by running C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe database --upgrade
Error:
Error: The previous command returned a non-zero exit code of: 1
Error: The command that failed was: “C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe” database --instance “OctopusServer” --upgrade

Justin_Walsh · 13 January 2020 15:49

Thanks for the additional data, David!

Do you have a backup/snapshot of your Octopus server from before you performed the upgrade earlier today? If there was a downgrade caused by 2018.10 being installed, it may have put your database and filesystem into a corrupted state, and the fastest and easier way forward will likely be to restore your backup/snapshot and then just install 2019.9.12 over the top of your known-good install.

zeiddev · 13 January 2020 16:05

Hindsight is a wonderful thing…

I don’t think that there is one…

Justin_Walsh · 13 January 2020 21:30

Well, that definitely makes things more interesting

The first thing I would attempt would be to download the MSI installed for the version of Octopus you were using prior to today, and attempt to install that - it should hopefully get this back into a working state, and then from there, try and install 2019.9.12 after it’s back up and running. This has worked previously in some cases, and hopefully it gets you up and deploying.

Let me know how this goes for you!

zeiddev · 13 January 2020 21:54

Is there a way of determining the exact version I was on before? I can only assume that the version that I was on was the 2008 version that was in the downloads folder (that I started by mistake). Although I may have upgraded since that version…

Alex.Rolley · 13 January 2020 22:32

Hi @zeiddev

You can check this by looking at the OctopusServerInstallationHistory table in your Octopus Database.

Regards,
Alex

zeiddev · 14 January 2020 08:53

OK, I think that we are back up and running!

I ran the setup for the last known working version and that seemed to do the trick. I have now upgraded to the latest version 2019.12.0 LTS and everything appears to be working fine. (About to run a backup!).

I realise what I did by running an older version was very stupid but it really should not be that easy to corrupt your system in the way that I did. Can this be fed back to the developers please?

Thank you for all your help in getting this resolved!