Server stops immediately after starting on upgrade

reliability
unknown
(David Zeidman) #1

I just upgraded to the latest version and when I go to the Octopus Manager it says that the Octopus Windows Service is currently stopped. When I try to start it, it appears to start then stops straight away. Looking at the logs I get the following:

2020-01-13 08:48:50.6183 5556 6 FATAL Unhandled AppDomain exception occurred: Unable to determine type to deserialize. FeedType BuiltIn does not map to a known type
System.Exception
at Nevermore.Serialization.InheritedClassConverter1.GetTypeInfoFromDerivedType(String derivedType) at Nevermore.Serialization.InheritedClassConverterBase2.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Nevermore.RelationalTransaction.Stream
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at Octopus.Core.Initialization.BuiltIn.BuiltInFeedInitializer.EnsureFeed(IRelationalTransaction transaction)
at Octopus.Core.Initialization.BuiltIn.BuiltInFeedInitializer.Initialize(IRelationalStore store)
at Nevermore.StoreInitializer.Initialize()
at Octopus.Server.OctopusServerEngine.Start()
at Octopus.Server.Commands.RunCommand.Start()
at Octopus.Shared.Startup.AbstractCommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions)
at Octopus.Shared.Startup.WindowsServiceHost.<>c__DisplayClass1_0.b__0()
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

Any suggestions?

Thanks

David

(Justin Walsh) #5

Hi @zeiddev.

Thanks for getting in touch, and sorry to hear you’re encountering this follow your upgrade. I’ll get to work trying to replicate this on my end, can you let me know what version you installed? Just want to make sure that I can reproduce it as accurately as possible. The latest LTS is 2019.9.12 and the latest Fastlane is 2019.11.3

Look forward to hearing from you soon.

(David Zeidman) #6

Hi,

Now I am not sure that I downloaded the correct version. When I looked at the file name it said Octopus.2018.10.4-x64.msi which seemed odd so I went to download it again and got 2019.9.12-x64 which seems more reasonable.

I went to run that one and got the following errors instead:

Using database connection string from configuration file: data source=(local)\SQLEXPRESS;initial catalog=Octopus;integrated security=True
========== SQL Database Connection Details ========
SQL Server: (local)\SQLEXPRESS
SQL Database: Octopus
Authentication Type: Windows Integrated Security
User: EC2AMAZ-7160HGQ\Administrator

===================================================
Preparing to upgrade the database ‘Octopus’ on SQL Server at ‘(local)\SQLEXPRESS’ to the expected schema running as ‘EC2AMAZ-7160HGQ\Administrator’…
Making sure it’s safe to upgrade the database schema…
Checking to see if database schema upgrade is required…

Database schema changes are required. Execution plan:

Planned pre-upgrade scripts (non-transactional):

  • Octopus.Core.UpgradeScriptsPre.Script0000 - Database options.sql

Planned schema upgrade scripts (single transaction for all scripts):

  • Octopus.Core.UpgradeScripts.Script0188 - Delete Artfifact RelatedDocumentIds.sql
  • Octopus.Core.UpgradeScripts.Script0191 - PackageMetadata - Create table.sql
  • Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.cs
  • Octopus.Core.UpgradeScripts.Script0194ConvertMachinePolicies.cs
  • Octopus.Core.UpgradeScripts.Script0194NormalizeVCSRootURLs.cs
  • Octopus.Core.UpgradeScripts.Script0195 - Add DataVersion column to spaces.sql
  • Octopus.Core.UpgradeScripts.Script0195SetWorkerPoolType.cs
  • Octopus.Core.UpgradeScripts.Script0196 - Team and UserRole DataVersion.sql
  • Octopus.Core.UpgradeScripts.Script0197PreSeedFeedKeyAllocationTable.cs
  • Octopus.Core.UpgradeScripts.Script0198 - Remove ProjectGroupId from DeploymentResource.sql
  • Octopus.Core.UpgradeScripts.Script0200 - Add BlockConcurrentDeployments to interruption.sql
  • Octopus.Core.UpgradeScripts.Script0201EnsureSystemAndSpacePermissionAreCorrect.cs
  • Octopus.Core.UpgradeScripts.Script0202EnsureArtifactScopesAreConsistent.cs
  • Octopus.Core.UpgradeScripts.Script0202EnsureMachineHealthCheckPoliciesAreNotNull.cs
  • Octopus.Core.UpgradeScripts.Script0203 - Add OS details to machine table.sql
  • Octopus.Core.UpgradeScripts.Script0203 - AddClonedFromTenantIdColumnToTenants.sql
  • Octopus.Core.UpgradeScripts.Script0204CopyOSDetailsFromHealthCheckJson.cs
  • Octopus.Core.UpgradeScripts.Script0205 - Add OS details to worker table.sql
  • Octopus.Core.UpgradeScripts.Script0206CopyOSDetailsFromWorkerHealthCheckJson.cs
  • Octopus.Core.UpgradeScripts.Script0207 - Set OS Detail Columns To Unknown When Null.sql
  • Octopus.Core.UpgradeScripts.Script0208AddLTSLicenseHashColumnToMachines.cs

Planned post-upgrade scripts (non-transactional):

  • Octopus.Core.UpgradeScriptsPost.Script0001 - Rebuild indexes.sql
  • Octopus.Core.UpgradeScriptsPost.Script0002 - Refresh IdsInUse view.sql

Ensuring pre-conditions for upgrading the database are satisfied…
Searching for indexes that might upset the database upgrade process…

  • PASS: All columns use the default collation.
  • PASS: Your Octopus Server will be compliant with your license after upgrading.
  • PASS: We’ve done our best to remove any unexpected database indexes.
    ====================================================================================
    Starting database schema upgrade…
    ====================================================================================

Executing pre-upgrade scripts…
Executing TSQL Database Server script ‘Octopus.Core.UpgradeScriptsPre.Script0000 - Database options.sql’
Enabling read committed snapshot isolation on Octopus
Read committed snapshot isolation is already on
Executing schema upgrade scripts…
Beginning transaction
Checking whether journal table exists…
Executing TSQL Database Server script ‘Octopus.Core.UpgradeScripts.Script0188 - Delete Artfifact RelatedDocumentIds.sql’
Checking whether journal table exists…
Executing TSQL Database Server script ‘Octopus.Core.UpgradeScripts.Script0191 - PackageMetadata - Create table.sql’
Executing .NET Database Server script ‘Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.cs’
Error occured when executing scripts, transaction will be rolled back
Error: Upgrade failed due to an unexpected exception:
Error: System.ArgumentNullException: Value cannot be null.
Error: Parameter name: value
Error: at Newtonsoft.Json.Linq.Extensions.Value[T,U](IEnumerable1 value) Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.UserRoleData.get_GrantedSpacePermissions() in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 71 Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.RoleNeedsUpgrading(UserRoleData role) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 36 Error: at System.Linq.Enumerable.WhereListIterator1.MoveNext()
Error: at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
Error: at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.Execute(IDbCommandExecutor commandExecutor, IUpgradeLog log) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 24 Error: at Octopus.Core.Initialization.DbCommandUpgradeScriptResolver.Execute(Func1 commandFactory, IUpgradeLog log) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DbCommandUpgradeScriptResolver.cs:line 25
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.<>c__DisplayClass6_0.b__0(Func1 commandFactory) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 319 Error: at DbUp.Engine.Transactions.SingleTrasactionStrategy.Execute(Action1 action)
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.ExecuteDbCommandUpgradeScript(IConnectionManager connectionManager, IJournal journal, IUpgradeLog log, IDbCommandUpgradeScript upgradeScript) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 307
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.Execute(SqlScript script, IDictionary2 variables) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 287 Error: at DbUp.Engine.UpgradeEngine.PerformUpgrade() Error: We encountered an error during the schema upgrade. The schema upgrade was stopped and rolled back. Don't worry, this means we haven't made any changes to your data, and you should be able to get back up and running quickly. Error: Here is the summary of the problems we encountered: Error: ----------------------------------------- Error: Value cannot be null. Error: Parameter name: value Error: System.ArgumentNullException Error: at Newtonsoft.Json.Linq.Extensions.Value[T,U](IEnumerable1 value)
Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.UserRoleData.get_GrantedSpacePermissions() in Script0192 - Add PackageMetadataPush Permission.cs:line 71
Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.RoleNeedsUpgrading(UserRoleData role) in Script0192 - Add PackageMetadataPush Permission.cs:line 36
Error: at System.Linq.Enumerable.WhereListIterator1.MoveNext() Error: at System.Collections.Generic.List1…ctor(IEnumerable1 collection) Error: at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.Execute(IDbCommandExecutor commandExecutor, IUpgradeLog log) in Script0192 - Add PackageMetadataPush Permission.cs:line 24
Error: at Octopus.Core.Initialization.DbCommandUpgradeScriptResolver.Execute(Func1 commandFactory, IUpgradeLog log) in DbCommandUpgradeScriptResolver.cs:line 25 Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.<>c__DisplayClass6_0.<ExecuteDbCommandUpgradeScript>b__0(Func1 commandFactory) in DatabaseUpgrader.cs:line 319
Error: at DbUp.Engine.Transactions.SingleTrasactionStrategy.Execute(Action1 action) Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.ExecuteDbCommandUpgradeScript(IConnectionManager connectionManager, IJournal journal, IUpgradeLog log, IDbCommandUpgradeScript upgradeScript) in DatabaseUpgrader.cs:line 307 Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.Execute(SqlScript script, IDictionary2 variables) in DatabaseUpgrader.cs:line 287
Error: at DbUp.Engine.UpgradeEngine.PerformUpgrade()
Error: -----------------------------------------
Error: Beginning transaction
Error: Checking whether journal table exists…
Error: Executing TSQL Database Server script ‘Octopus.Core.UpgradeScripts.Script0188 - Delete Artfifact RelatedDocumentIds.sql’
Error: Checking whether journal table exists…
Error: Executing TSQL Database Server script ‘Octopus.Core.UpgradeScripts.Script0191 - PackageMetadata - Create table.sql’
Error: Executing .NET Database Server script ‘Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.cs’
Error: Error occured when executing scripts, transaction will be rolled back
Error: Upgrade failed due to an unexpected exception:
Error: System.ArgumentNullException: Value cannot be null.
Error: Parameter name: value
Error: at Newtonsoft.Json.Linq.Extensions.Value[T,U](IEnumerable1 value) Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.UserRoleData.get_GrantedSpacePermissions() in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 71 Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.RoleNeedsUpgrading(UserRoleData role) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 36 Error: at System.Linq.Enumerable.WhereListIterator1.MoveNext()
Error: at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
Error: at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) Error: at Octopus.Core.UpgradeScripts.Script0192AddPackageMetadataPushPermission.Execute(IDbCommandExecutor commandExecutor, IUpgradeLog log) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\UpgradeScripts\Script0192 - Add PackageMetadataPush Permission.cs:line 24 Error: at Octopus.Core.Initialization.DbCommandUpgradeScriptResolver.Execute(Func1 commandFactory, IUpgradeLog log) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DbCommandUpgradeScriptResolver.cs:line 25
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.<>c__DisplayClass6_0.b__0(Func1 commandFactory) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 319 Error: at DbUp.Engine.Transactions.SingleTrasactionStrategy.Execute(Action1 action)
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.ExecuteDbCommandUpgradeScript(IConnectionManager connectionManager, IJournal journal, IUpgradeLog log, IDbCommandUpgradeScript upgradeScript) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 307
Error: at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.Execute(SqlScript script, IDictionary`2 variables) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 287
Error: at DbUp.Engine.UpgradeEngine.PerformUpgrade()
Error:
Error: -----------------------------------------
Error: You have two options from this point:
Error: OPTION 1: Re-run the installer for the version of Octopus Server you were using just before trying this upgrade. Your data hasn’t been changed and everything will go back to how it was beforehand. Please send this log to support@octopus.com and we will work with you to get you upgraded.
Error: OPTION 2: If you know what the problem is and how to fix it, please fix the problem and then try the upgrade again by running C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe database --upgrade
Error:
Error: The previous command returned a non-zero exit code of: 1
Error: The command that failed was: “C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe” database --instance “OctopusServer” --upgrade

(Justin Walsh) #7

Thanks for the additional data, David!

Do you have a backup/snapshot of your Octopus server from before you performed the upgrade earlier today? If there was a downgrade caused by 2018.10 being installed, it may have put your database and filesystem into a corrupted state, and the fastest and easier way forward will likely be to restore your backup/snapshot and then just install 2019.9.12 over the top of your known-good install.

(David Zeidman) #8

Hindsight is a wonderful thing…

I don’t think that there is one…

(Justin Walsh) #9

Well, that definitely makes things more interesting :slight_smile:

The first thing I would attempt would be to download the MSI installed for the version of Octopus you were using prior to today, and attempt to install that - it should hopefully get this back into a working state, and then from there, try and install 2019.9.12 after it’s back up and running. This has worked previously in some cases, and hopefully it gets you up and deploying.

Let me know how this goes for you!

(David Zeidman) #10

Is there a way of determining the exact version I was on before? I can only assume that the version that I was on was the 2008 version that was in the downloads folder (that I started by mistake). Although I may have upgraded since that version…

#11

Hi @zeiddev

You can check this by looking at the OctopusServerInstallationHistory table in your Octopus Database.

Regards,
Alex

(David Zeidman) #12

OK, I think that we are back up and running!

I ran the setup for the last known working version and that seemed to do the trick. I have now upgraded to the latest version 2019.12.0 LTS and everything appears to be working fine. (About to run a backup!).

I realise what I did by running an older version was very stupid but it really should not be that easy to corrupt your system in the way that I did. Can this be fed back to the developers please?

Thank you for all your help in getting this resolved!

1 Like