OctopusDeploy Docker container failed to start

I am using docker-compose and portainer to manage the octopus-deploy ‘stack’. The application started fine for version 2022.3.10723. I pulled the latest release image 2023.1.9767 and redeployed the container to upgrade the instance but this resulted in the app crashing. I reverted to the original 2022.3.10723 version but it is still failing to load. All environment variables look okay.
I am able to ping the database server and access the octopus config files from within the container (using bash interactive shell). Below is the log output:

2023-03-30T15:28:22.550555060Z ======== Config already detected ========
2023-03-30T15:28:22.550583224Z ======== Starting Server =======
2023-03-30T15:28:26.015156095Z Instance OctopusServer of OctopusServer has not been configured on this machine. Available instances: .
2023-03-30T15:28:26.015246889Z Octopus.Shared.ControlledFailureException: Instance OctopusServer of OctopusServer has not been configured on this machine. Available instances: .
2023-03-30T15:28:26.015255191Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceStore.GetNamedRegistryRecord(String instanceName) in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceStore.cs:line 168
2023-03-30T15:28:26.015259718Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceStore.LoadInstanceDetails(String instanceName) in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceStore.cs:line 59
2023-03-30T15:28:26.015261316Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceSelector.LocateApplicationPrimaryConfiguration() in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceSelector.cs:line 111
2023-03-30T15:28:26.015263304Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceSelector.LoadInstance() in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceSelector.cs:line 67
2023-03-30T15:28:26.015265127Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceSelector.LoadCurrentInstance() in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceSelector.cs:line 58
2023-03-30T15:28:26.015266657Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceSelector.get_Current() in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceSelector.cs:line 35
2023-03-30T15:28:26.015268172Z    at Octopus.Shared.Startup.AbstractStandardCommand.Start() in ./source/Octopus.Shared/Startup/AbstractStandardCommand.cs:line 42
2023-03-30T15:28:26.015269637Z    at Octopus.Server.Commands.RunCommand.Start() in ./source/Octopus.Server/Commands/RunCommand.cs:line 80
2023-03-30T15:28:26.015271125Z    at Octopus.Shared.Startup.AbstractCommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) in ./source/Octopus.Shared/Startup/AbstractCommand.cs:line 101
2023-03-30T15:28:26.015272689Z    at Octopus.Shared.Startup.OctopusProgram.Start(ICommandRuntime commandRuntime) in ./source/Octopus.Shared/Startup/OctopusProgram.cs:line 504
2023-03-30T15:28:26.015274164Z    at Octopus.Shared.Startup.NoninteractiveHost.Run(Action`1 start, Action shutdown) in ./source/Octopus.Shared/Startup/NoninteractiveHost.cs:line 12
2023-03-30T15:28:26.015277779Z    at Octopus.Shared.Startup.OctopusProgram.RunHost(ICommandHost host) in ./source/Octopus.Shared/Startup/OctopusProgram.cs:line 221
2023-03-30T15:28:26.015323696Z    at Octopus.Shared.Startup.OctopusProgram.Run() in ./source/Octopus.Shared/Startup/OctopusProgram.cs:line 167
2023-03-30T15:28:26.036977831Z Unhandled exception. Autofac.Core.DependencyResolutionException: An exception was thrown while activating Octopus.Server.OctopusServerEngine -> λ:Octopus.Server.Extensibility.Authentication.Extensions.IAuthenticationProvider[] -> Octopus.Server.Extensibility.Authentication.DirectoryServices.DirectoryServicesAuthenticationProvider -> Octopus.Server.Extensibility.Authentication.DirectoryServices.Configuration.DirectoryServicesConfigurationStore -> Octopus.Core.Repositories.RawConfigurationStore -> λ:Octopus.Core.RelationalStorage.IRawRelationalStore -> Octopus.Core.RelationalStorage.RelationalStoreFactory -> Octopus.Core.RelationalStorage.MappedSqlDatabaseConnectionString -> Octopus.Core.Configuration.OctopusServerStorageConfiguration -> λ:Octopus.Configuration.IKeyValueStore.
2023-03-30T15:28:26.036999759Z  ---> Octopus.Shared.ControlledFailureException: Instance OctopusServer of OctopusServer has not been configured on this machine. Available instances: .
2023-03-30T15:28:26.037001814Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceStore.GetNamedRegistryRecord(String instanceName) in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceStore.cs:line 168
2023-03-30T15:28:26.037003421Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceStore.LoadInstanceDetails(String instanceName) in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceStore.cs:line 59
2023-03-30T15:28:26.037004986Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceSelector.LocateApplicationPrimaryConfiguration() in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceSelector.cs:line 111
2023-03-30T15:28:26.037006523Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceSelector.LoadInstance() in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceSelector.cs:line 67
2023-03-30T15:28:26.037008032Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceSelector.LoadCurrentInstance() in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceSelector.cs:line 58
2023-03-30T15:28:26.037009970Z    at Octopus.Shared.Configuration.Instances.ApplicationInstanceSelector.get_Current() in ./source/Octopus.Shared/Configuration/Instances/ApplicationInstanceSelector.cs:line 35
2023-03-30T15:28:26.037011566Z    at Octopus.Shared.Configuration.ConfigurationModule.<>c.<Load>b__3_2(IComponentContext c) in ./source/Octopus.Shared/Configuration/ConfigurationModule.cs:line 79
2023-03-30T15:28:26.037013265Z    at Autofac.RegistrationExtensions.<>c__DisplayClass5_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)
2023-03-30T15:28:26.037014821Z    at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
2023-03-30T15:28:26.037016401Z    at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
2023-03-30T15:28:26.037017860Z    at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget)
2023-03-30T15:28:26.037019367Z    --- End of inner exception stack trace ---
2023-03-30T15:28:26.037020760Z    at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget)
2023-03-30T15:28:26.037022266Z    at Autofac.Core.Resolving.InstanceLookup.<>c__DisplayClass6_0.<Execute>b__0()
2023-03-30T15:28:26.037023801Z    at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
2023-03-30T15:28:26.037031030Z    at Autofac.Core.Resolving.InstanceLookup.Execute()
2023-03-30T15:28:26.037032523Z    at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
2023-03-30T15:28:26.037036146Z    at Autofac.Core.Resolving.ResolveOperation.ResolveComponent(IComponentRegistration registration, IEnumerable`1 parameters)
2023-03-30T15:28:26.037037670Z    at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters)
2023-03-30T15:28:26.037039153Z    at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable`1 parameters)
2023-03-30T15:28:26.037040636Z    at Autofac.Features.LazyDependencies.LazyRegistrationSource.<>c__DisplayClass5_1`1.<CreateLazyRegistration>b__1()
2023-03-30T15:28:26.037042762Z    at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
2023-03-30T15:28:26.037044192Z    at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
2023-03-30T15:28:26.037045671Z    at System.Lazy`1.CreateValue()
2023-03-30T15:28:26.037047099Z    at System.Lazy`1.get_Value()
2023-03-30T15:28:26.037048517Z    at Octopus.Server.Commands.RunCommand.Stop() in ./source/Octopus.Server/Commands/RunCommand.cs:line 170
2023-03-30T15:28:26.037050233Z    at Octopus.Shared.Startup.AbstractCommand.Octopus.Shared.Startup.ICommand.Stop() in ./source/Octopus.Shared/Startup/AbstractCommand.cs:line 139
2023-03-30T15:28:26.037051826Z    at Octopus.Shared.Startup.OctopusProgram.Shutdown() in ./source/Octopus.Shared/Startup/OctopusProgram.cs:line 563
2023-03-30T15:28:26.037053352Z    at Octopus.Shared.Startup.NoninteractiveHost.Stop(Action shutdown) in ./source/Octopus.Shared/Startup/NoninteractiveHost.cs:line 18
2023-03-30T15:28:26.037054919Z    at Octopus.Shared.Startup.OctopusProgram.<>c__DisplayClass20_0.<RunHost>b__1(Object _, EventArgs _) in ./source/Octopus.Shared/Startup/OctopusProgram.cs:line 219
2023-03-30T15:28:26.037056574Z    at System.AppContext.OnProcessExit()
2023-03-30T15:28:26.048539023Z [createdump] Gathering state for process 1 Octopus.Server
2023-03-30T15:28:26.048553651Z [createdump] Crashing thread 0000000d signal 00000006

Hi @oliver.d.cooper,

Thanks for posting your question, and sorry to hear you’re encountering issues when trying to upgrade your container-hosted Octopus instance.

It looks like from the log you’ve provided the Octopus service is picking up the existing config file from the container, which is indicating there’s an Octopus instance called OctopusServer already installed an running in the container however Octopus is finding no installed instances.

It’s possible that the service running the OctopusServer instance stopped when the upgrade was attempted, and since it’s no longer running the Octopus startup is unable to find it.

I have a few questions I hope you don’t mind answering:

  • Did you restore a previous version of the Octopus database prior to attempting to use the 2022.3.10723 version?
  • Do you have your Master Key for the Instance or did you specify your own in the .env file? I don’t need it, but just curious.
  • Are you using persistent storage of any sort for the Octopus configuration files?
  • Do you have logs from when you attempted the upgrade to 2023.1.9767?

Once I know the above I can let you know what the best next steps might be.

Best,
Patrick

Thank you for your prompt reply!

Following your explanation I understand that Octopus writes to the /config directory when it is starting up and at some point an empty lock file is written called ‘configured’ which indicates that there is an instance already running? Based on this assumption, I proceeded to delete the contents of the /config folder and hey-presto, when I started the container again the files were recreated in the /config directory and it started up fine! I just need to perform the upgrade again somehow.

I restored old databases and Octopus config files, carefully checked file permissions, created a duplicate docker VM from a backup to see what went wrong etc. and it looks like it was a simple lock file that was causing the issue!

What do you think went wrong for this to happen and how can I avoid this in the future?

  • I did not restore the database after downgrading because it had not managed to run the migration scripts when it errored (which I confirmed by looking in the OctopusServerInstallationHistory table).
  • My master key is specified in the env of the docker compose file
  • As below, you can see persistent storage being used
  • I didn’t look at the logs prior to the upgrade, but they are probably on the filesystem somewhere

Here is my docker-compose:

version: '3'
services:
   octopus-server:
    image: octopusdeploy/octopusdeploy:2022.3.10723
    container_name: octopus-server
    user: octopus
    environment:
      ACCEPT_EULA: 'Y'
      DB_CONNECTION_STRING: ${DB_CONNECTION_STRING}
      ADMIN_USERNAME: ${ADMIN_USERNAME}
      ADMIN_PASSWORD: ${ADMIN_PASSWORD}
      ADMIN_EMAIL: ${ADMIN_EMAIL}
      OCTOPUS_SERVER_BASE64_LICENSE: ${OCTOPUS_SERVER_BASE64_LICENSE}
      MASTER_KEY: ${MASTER_KEY}
      ADMIN_API_KEY: ''
      DISABLE_DIND: 'Y'
      OCTOPUS_SERVER_CONFIGURATION_DIRECTORY: '/config'
      OCTOPUS_SERVER_NODE_NAME: ${OCTOPUS_SERVER_NODE_NAME}
      ENABLE_USAGE: 'N'
    ports:
      - 18080:8080
      - 10943:10943
    volumes:
      - /mnt/data/octopus-deploy/config:/config
      - /mnt/data/octopus-deploy/repository:/repository
      - /mnt/data/octopus-deploy/artifacts:/artifacts
      - /mnt/data/octopus-deploy/taskLogs:/taskLogs
      - /mnt/data/octopus-deploy/cache:/cache
      - /mnt/data/octopus-deploy/import:/import

Just tried upgrading again and the same issue occurs. If I delete the contents of the /config directory, the new config is written and the upgrade process starts fine.

Hey @oliver.d.cooper,

Thanks for getting back to me and I’m glad to hear you were able to get unblocked!

It’s interesting you’re hitting this issue each time you try to upgrade, but I think I understand what might be causing this. I notice you’ve mapped your /config directory into persistent storage, and Octopus keeps picking that up when it starts up. When Octopus starts up it’s looking for that existing instance referenced by the config file. However, since Octopus is being run from a new container each time, there’s no existing Octopus instance running and Octopus fails the upgrade when it is unable to find it.

I don’t believe the /config directory is something you need in your persistent storage since you’re managing this instance’s configuration with docker-compose, but I’m curious if you have a use case to do this.

From what I can find the OCTOPUS_SERVER_CONFIGURATION_DIRECTORY environment variable was added to support running Octopus in a read-only root file system: Running Linux Octopus docker with read only root file system · Issue #6838 · OctopusDeploy/Issues · GitHub

In addition, the config files are generally useful for instances where the service isn’t rebuilt with upgrades (which isn’t the case with the docker-compose method), and since you have all your configuration in your docker compose files you might test removing this from your volume mounts to see if it fixes the issue.

Let me know what you think!

Best regards,
Patrick

Perfect! Your suggestion worked, great detective work!

Ideally I would remove all sensitive environment variables from the configuration, so having a persistent config folder would help as this is where the database connection string and master key are stored.

For now I have removed the /config volume, the OCTOPUS_SERVER_CONFIGURATION_DIRECTORY and any other redundant environment variables (ADMIN_USERNAME, ADMIN_PASSWORD, ADMIN_EMAIL, OCTOPUS_SERVER_BASE64_LICENSE). I have kept DB_CONNECTION_STRING and MASTER_KEY, although it would be nice to remove these too.

After the changes, Octopus is now creating the config in non-persistent storage under /home/octopus/.octopus/ as you suggested it would.

Thank you for you help!

Kind Regards,
Oliver.

2 Likes

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.