Master key issues in Octopus Linux Containers

Hello, I am trying to run MSSQL + Octopus in kubernetes, specifically in Openshift.

I have created MSSQL Service and mounted PVC to /var/opt/mssql.
I have also created Octopus Service and mounted 3 PVCs to /artifacts, /repository, /taskLogs accordingly.

When MSSQL pod was up, I scalled Octopus. The database was created and I was given a Master Key. And then with following command I verified it:
/Octopus/Octopus.Server show-master-key --console --instance OctopusServer

After that, I have updated ENV variable MASTER_KEY=“PASTED_24_CHAR_MASTER_KEY”

Unfortunatelly, at the end I got following message and due to that pod restarts again and again:
Failed to decrypt the Octopus Server certificate. This usually indicates the wrong master key is being used to read encrypted data.

Here is part of the log:

======== Writing Configuration File =======
find: ‘/run/secrets/rhsm’: Permission denied
error: attempting to run rootless dockerd but missing ‘rootlesskit’ (perhaps the ‘docker:dind-rootless’ image variant is intended?)
Creating empty configuration file: /home/octopus/.octopus/OctopusServer/Server/Server.linux.config
Saving instance: OctopusServer
Setting home directory to: /home/octopus/.octopus/OctopusServer/Server
Setting server node name to ‘octopus’.

======== Initializing DB ========
Using supplied Master Key
Setting master key
Setting database connection string to: server=…MY_CONNECTION_STRING…
========== SQL Database Connection Details ========
SQL Server: CLUSTER_IP,1433
SQL Database: OctopusDeploy
Authentication Type: SQL Server Credentials
User: sa

===================================================
Testing connection to the ‘master’ database…
Successfully connected to ‘master’ database!
Database ‘OctopusDeploy’ already exists on SQL Server at ‘CLUSTER_IP,1433’, no need to create it.
Checking the Octopus Master Key has been configured.
Making sure it’s safe to upgrade the database schema…
Ensuring pre-conditions for upgrading the database are satisfied…
Searching for indexes that might upset the database upgrade process…

  • PASS: All columns use the default collation.
  • PASS: Your Octopus Server will be compliant with your license after upgrading.
  • PASS: We’ve done our best to remove any unexpected database indexes.
  • PASS: The version of your SQL Server satisfies Octopus Server installation requirements.
    Executing always run pre scripts…
    Executing TSQL Database Server script ‘Octopus.Core.UpgradeScriptsAlwaysPre.Script0000 - Set highest available compatibility level.sql’
    Current COMPATIBILITY_LEVEL for OctopusDeploy is set to 140
    Ensuring COMPATIBILITY_LEVEL for OctopusDeploy is set to 140
    COMPATIBILITY_LEVEL for OctopusDeploy is already 140 or higher
    Checking to see if database schema upgrade is required…
    Database already has the expected schema. No changes are required.
    Executing always run post scripts…
    Executing TSQL Database Server script ‘Octopus.Core.UpgradeScriptsAlwaysPost.Script0000 - Refresh Views.sql’
    Refreshing view dbo.Dashboard
    Refreshing view dbo.IdsInUse
    Refreshing view dbo.MultiTenancyDashboard
    Refreshing view dbo.Release_WithDeploymentProcess
    Refreshing view dbo.RunbookSnapshot_WithRunbookProcess
    Refreshing view dbo.TenantProject
    Process reader took 806ms in transaction ‘ThrowIfOctopusServerHasBeenDowngraded’: SELECT TOP 1 *
    FROM [dbo].[OctopusServerInstallationHistory]
    WHERE ([Version] is not null)
    ORDER BY [Installed] DESC
    Process reader took 499ms in transaction ‘BuiltInTeamInitializer’: SELECT TOP 1 * FROM [dbo].[UserRole] WHERE [Id] = @Id
    Update took 395ms in transaction ‘BuiltInTeamInitializer’: UPDATE [dbo].[UserRole] SET [Name] = @Name, [JSON] = @JSON WHERE [Id] = @Id
    Process reader took 697ms in transaction ‘BuiltInTeamInitializer’: SELECT TOP 1 * FROM [dbo].[Team] WHERE [Id] = @Id
    Failed to decrypt the Octopus Server certificate. This usually indicates the wrong master key is being used to read encrypted data.`

Hi @avizen.j,

Thanks for reaching out to us at Octopus Support!

I’m working to reproduce this issue, so I should have an update for you soon. Could you confirm first what version of Octopus Server you’re running in your container?

Regards,
Mallory

Hi,
Thanks for a quick response, hopefully you we’ll be able to help me.
I am using following Octopus Server linux docker image octopusdeploy/octopusdeploy:2020.5.2

Hi @avizen.j,

Thanks for keeping in touch and letting us know. I’ll jump in here for Mallory to hopefully provide some helpful information.

I raised this internally for more ideas on this, and we think it might be a case of initial startup and subsequent startups using different master keys.

The easiest way to address that would be to clear the existing database and then create a master key manually (details on doing this outlined in this doc page) to ensure both the initial and subsequent startups use the same master key. This means you’d lose your existing config, however, so this is assuming you’re okay with that.

I hope this helps! Please let me know what you think and how you go.

Best regards,

Kenny

Hi @Kenneth_Bates

I have tried following: deleted database PVC and redeployed it. Here is part of SQL Server log after startup:

DBSTARTUP (model, 3): RemapSysfiles1Time took 885 ms
2021-01-05 13:13:23.44 spid13s     Polybase feature disabled.
2021-01-05 13:13:23.54 spid13s     Clearing tempdb database.
2021-01-05 13:13:26.34 Server      Long Sync IO: Scheduler 9 had 2 Sync IOs in nonpreemptive mode longer than 1000 ms
2021-01-05 13:13:29.34 spid30s     A self-generated certificate was successfully loaded for encryption.
2021-01-05 13:13:29.55 spid30s     Server is listening on [ 'any' <ipv6> 1433].
2021-01-05 13:13:30.53 spid30s     Server is listening on [ 'any' <ipv4> 1433].
2021-01-05 13:13:29.94 Server      Server is listening on [ ::1 <ipv6> 1434].
2021-01-05 13:13:29.98 Server      Server is listening on [ 127.0.0.1 <ipv4> 1434].
2021-01-05 13:13:30.04 Server      Dedicated admin connection support was established for listening locally on port 1434.
2021-01-05 13:13:30.14 spid30s     SQL Server is now ready for client connections. This is an informational message; no user action is required.
2021-01-05 13:13:34.27 spid13s     Starting up database 'tempdb'.

Then I have deployed Octopus Server manually setting MASTER_KEY variable to generated string which is 24 char long using following command: openssl rand 16 | base64

However I get the same error. Here is part of Octopus Server log:

======== Initializing DB ========
Using supplied Master Key
Setting master key
Setting database connection string to: server=CLUSTER_IP,1433;database=OctopusDeploy;user=‘MY_USER’;password=‘PASSWORD’
========== SQL Database Connection Details ========
SQL Server: CLUSTER_IP,1433
SQL Database: OctopusDeploy
Authentication Type: SQL Server Credentials
User: ‘MY_USER’

===================================================
Testing connection to the ‘master’ database…
Successfully connected to ‘master’ database!
Creating database ‘OctopusDeploy’ on the SQL Server at ‘CLUSTER_IP,1433’
Created database OctopusDeploy in CLUSTER_IP,1433
Database successfully created. Testing we can connect to it successfully…
Successfully connected to the newly created database.

Pre-upgrade scripts, Post-upgrade, Database schema update,

===================================================
Database schema upgrade completed taking 45 seconds

===================================================
Executing always run post scripts…
Executing TSQL Database Server script ‘Octopus.Core.UpgradeScriptsAlwaysPost.Script0000 - Refresh Views.sql’
Refreshing view dbo.Dashboard
Refreshing view dbo.IdsInUse
Refreshing view dbo.MultiTenancyDashboard

Creating config for GitHubConfiguration, with id issuetracker-github
Initializing Jira integration settings
Creating config for JiraConfiguration, with id jira-integration
These changes require a restart of the Octopus Server.

======== Configuring Paths =======
Checking the Octopus Master Key has been configured.
Skipping database schema upgrade checks. Be warned that the schema may be out of sync with that expected by the Server.
Skipping database compatibility check as requested.
Execute reader took 793ms in transaction ‘MaintenanceModeInitializer’: SELECT TOP 1 * FROM [dbo].[Configuration] WHERE [Id] = @Id
Process reader took 305ms in transaction ‘MaintenanceModeInitializer’: SELECT TOP 1 * FROM [dbo].[Configuration] WHERE [Id] = @Id
Failed to decrypt the Octopus Server certificate. This usually indicates the wrong master key is being used to read encrypted data.

I am thinking maybe that could be caused by this line in SQL Server? If yes, could you provide a workaround?

2021-01-05 13:13:29.34 spid30s     A self-generated certificate was successfully loaded for encryption.

Hi @avizen.j,

Thanks for following up and sending that additional information. I’m sorry to hear you’re still hitting this unexpected issue. We’ve gone through a couple attempts to reproduce this outcome, however we’ve been unable to do so. The process was to use these Docker files, generate the master key using openssl rand 16 | base64, enter it into the env vile and finally run docker-compose.

My colleague just wrote a knowledgebase article, and I’m wondering if comparing his process, and looking at the Linux example he linked in there, might point out any missing piece?

I hope this helps narrow it down, and please let us know how you go!

Best regards,

Kenny

Well, I guess it is worth to mention that I am able to start both MSSQL and Octopus locally using provided dockerfile and docker-compose.yml.

Also I have no problem with MASTER_KEY locally. I am able to start new Octopus container using old MASTER_KEY using which OctopusDeploy database was created.

The problem I am experiencing is happening in Kubernetes. I was trying to run it using this tutorial:

Hi @avizen.j,

Thanks for following up! Is it possible you’ve not set the master key in your kubernetes config? This would cause each replica to create different master keys.

If you’re still hitting this issue, and you have a copy of your YAML, would you be willing to send that through so we can test it on a kubernetes cluster?

I look forward to getting to the bottom of this one! :slight_smile:

Best regards,

Kenny

Okay, problem is gone, sharing here the solution, so that it would not be a disaster for those who are trying to have their Octopus Server running in Kubernetes.

  1. Be sure that container user has permissions to following directories:
    /Octopus, /home/octopus, /etc/octopus
    For those, who use OpenShift sharing the reference. You should be interested in “Support arbitrary user ids” section.
    https://docs.openshift.com/container-platform/4.2/openshift_images/create-images.html
  2. If permissions were not a problem for you then try this. Sometimes, HOME by default is set to /root. Therefore, while trying to compare certificates using MASTER_KEY, Octopus might be looking to other directory and the only thing that it is telling you is this exception (which is not really accurate - it cannot decrypt, it cannot find - at least in my case):
    Failed to decrypt the Octopus Server certificate. This usually indicates the wrong master key is being used to read encrypted data.

SOLUTION: Just to be sure that everything goes smooth, set HOME env variable to /home/octopus/ and you should be good to go.

Hi @avizen.j,

Great to hear you’ve found the solution here, and thank you for sharing the details! I’m sure it’ll be helpful in the future.

Please don’t hesitate to reach out anytime with any other questions or concerns.

Best regards,

Kenny