Hello, I am trying to run MSSQL + Octopus in kubernetes, specifically in Openshift.
I have created MSSQL Service and mounted PVC to /var/opt/mssql.
I have also created Octopus Service and mounted 3 PVCs to /artifacts, /repository, /taskLogs accordingly.
When MSSQL pod was up, I scalled Octopus. The database was created and I was given a Master Key. And then with following command I verified it: /Octopus/Octopus.Server show-master-key --console --instance OctopusServer
After that, I have updated ENV variable MASTER_KEY=“PASTED_24_CHAR_MASTER_KEY”
Unfortunatelly, at the end I got following message and due to that pod restarts again and again: Failed to decrypt the Octopus Server certificate. This usually indicates the wrong master key is being used to read encrypted data.
Here is part of the log:
======== Writing Configuration File =======
find: ‘/run/secrets/rhsm’: Permission denied
error: attempting to run rootless dockerd but missing ‘rootlesskit’ (perhaps the ‘docker:dind-rootless’ image variant is intended?)
Creating empty configuration file: /home/octopus/.octopus/OctopusServer/Server/Server.linux.config
Saving instance: OctopusServer
Setting home directory to: /home/octopus/.octopus/OctopusServer/Server
Setting server node name to ‘octopus’.
======== Initializing DB ========
Using supplied Master Key
Setting master key
Setting database connection string to: server=…MY_CONNECTION_STRING…
========== SQL Database Connection Details ========
SQL Server: CLUSTER_IP,1433
SQL Database: OctopusDeploy
Authentication Type: SQL Server Credentials
User: sa
===================================================
Testing connection to the ‘master’ database…
Successfully connected to ‘master’ database!
Database ‘OctopusDeploy’ already exists on SQL Server at ‘CLUSTER_IP,1433’, no need to create it.
Checking the Octopus Master Key has been configured.
Making sure it’s safe to upgrade the database schema…
Ensuring pre-conditions for upgrading the database are satisfied…
Searching for indexes that might upset the database upgrade process…
PASS: All columns use the default collation.
PASS: Your Octopus Server will be compliant with your license after upgrading.
PASS: We’ve done our best to remove any unexpected database indexes.
PASS: The version of your SQL Server satisfies Octopus Server installation requirements.
Executing always run pre scripts…
Executing TSQL Database Server script ‘Octopus.Core.UpgradeScriptsAlwaysPre.Script0000 - Set highest available compatibility level.sql’
Current COMPATIBILITY_LEVEL for OctopusDeploy is set to 140
Ensuring COMPATIBILITY_LEVEL for OctopusDeploy is set to 140
COMPATIBILITY_LEVEL for OctopusDeploy is already 140 or higher
Checking to see if database schema upgrade is required…
Database already has the expected schema. No changes are required.
Executing always run post scripts…
Executing TSQL Database Server script ‘Octopus.Core.UpgradeScriptsAlwaysPost.Script0000 - Refresh Views.sql’
Refreshing view dbo.Dashboard
Refreshing view dbo.IdsInUse
Refreshing view dbo.MultiTenancyDashboard
Refreshing view dbo.Release_WithDeploymentProcess
Refreshing view dbo.RunbookSnapshot_WithRunbookProcess
Refreshing view dbo.TenantProject
Process reader took 806ms in transaction ‘ThrowIfOctopusServerHasBeenDowngraded’: SELECT TOP 1 *
FROM [dbo].[OctopusServerInstallationHistory]
WHERE ([Version] is not null)
ORDER BY [Installed] DESC
Process reader took 499ms in transaction ‘BuiltInTeamInitializer’: SELECT TOP 1 * FROM [dbo].[UserRole] WHERE [Id] = @Id
Update took 395ms in transaction ‘BuiltInTeamInitializer’: UPDATE [dbo].[UserRole] SET [Name] = @Name, [JSON] = @JSON WHERE [Id] = @Id
Process reader took 697ms in transaction ‘BuiltInTeamInitializer’: SELECT TOP 1 * FROM [dbo].[Team] WHERE [Id] = @Id
Failed to decrypt the Octopus Server certificate. This usually indicates the wrong master key is being used to read encrypted data.`
I’m working to reproduce this issue, so I should have an update for you soon. Could you confirm first what version of Octopus Server you’re running in your container?
Hi,
Thanks for a quick response, hopefully you we’ll be able to help me.
I am using following Octopus Server linux docker image octopusdeploy/octopusdeploy:2020.5.2
Thanks for keeping in touch and letting us know. I’ll jump in here for Mallory to hopefully provide some helpful information.
I raised this internally for more ideas on this, and we think it might be a case of initial startup and subsequent startups using different master keys.
The easiest way to address that would be to clear the existing database and then create a master key manually (details on doing this outlined in this doc page) to ensure both the initial and subsequent startups use the same master key. This means you’d lose your existing config, however, so this is assuming you’re okay with that.
I hope this helps! Please let me know what you think and how you go.
I have tried following: deleted database PVC and redeployed it. Here is part of SQL Server log after startup:
DBSTARTUP (model, 3): RemapSysfiles1Time took 885 ms
2021-01-05 13:13:23.44 spid13s Polybase feature disabled.
2021-01-05 13:13:23.54 spid13s Clearing tempdb database.
2021-01-05 13:13:26.34 Server Long Sync IO: Scheduler 9 had 2 Sync IOs in nonpreemptive mode longer than 1000 ms
2021-01-05 13:13:29.34 spid30s A self-generated certificate was successfully loaded for encryption.
2021-01-05 13:13:29.55 spid30s Server is listening on [ 'any' <ipv6> 1433].
2021-01-05 13:13:30.53 spid30s Server is listening on [ 'any' <ipv4> 1433].
2021-01-05 13:13:29.94 Server Server is listening on [ ::1 <ipv6> 1434].
2021-01-05 13:13:29.98 Server Server is listening on [ 127.0.0.1 <ipv4> 1434].
2021-01-05 13:13:30.04 Server Dedicated admin connection support was established for listening locally on port 1434.
2021-01-05 13:13:30.14 spid30s SQL Server is now ready for client connections. This is an informational message; no user action is required.
2021-01-05 13:13:34.27 spid13s Starting up database 'tempdb'.
Then I have deployed Octopus Server manually setting MASTER_KEY variable to generated string which is 24 char long using following command: openssl rand 16 | base64
However I get the same error. Here is part of Octopus Server log:
======== Initializing DB ========
Using supplied Master Key
Setting master key
Setting database connection string to: server=CLUSTER_IP,1433;database=OctopusDeploy;user=‘MY_USER’;password=‘PASSWORD’
========== SQL Database Connection Details ========
SQL Server: CLUSTER_IP,1433
SQL Database: OctopusDeploy
Authentication Type: SQL Server Credentials
User: ‘MY_USER’
===================================================
Testing connection to the ‘master’ database…
Successfully connected to ‘master’ database!
Creating database ‘OctopusDeploy’ on the SQL Server at ‘CLUSTER_IP,1433’
Created database OctopusDeploy in CLUSTER_IP,1433
Database successfully created. Testing we can connect to it successfully…
Successfully connected to the newly created database.
…
Pre-upgrade scripts, Post-upgrade, Database schema update,
…
===================================================
Executing always run post scripts…
Executing TSQL Database Server script ‘Octopus.Core.UpgradeScriptsAlwaysPost.Script0000 - Refresh Views.sql’
Refreshing view dbo.Dashboard
Refreshing view dbo.IdsInUse
Refreshing view dbo.MultiTenancyDashboard
…
Creating config for GitHubConfiguration, with id issuetracker-github
Initializing Jira integration settings
Creating config for JiraConfiguration, with id jira-integration
These changes require a restart of the Octopus Server.
======== Configuring Paths =======
Checking the Octopus Master Key has been configured.
Skipping database schema upgrade checks. Be warned that the schema may be out of sync with that expected by the Server.
Skipping database compatibility check as requested.
Execute reader took 793ms in transaction ‘MaintenanceModeInitializer’: SELECT TOP 1 * FROM [dbo].[Configuration] WHERE [Id] = @Id
Process reader took 305ms in transaction ‘MaintenanceModeInitializer’: SELECT TOP 1 * FROM [dbo].[Configuration] WHERE [Id] = @Id
Failed to decrypt the Octopus Server certificate. This usually indicates the wrong master key is being used to read encrypted data.
I am thinking maybe that could be caused by this line in SQL Server? If yes, could you provide a workaround?
2021-01-05 13:13:29.34 spid30s A self-generated certificate was successfully loaded for encryption.
Thanks for following up and sending that additional information. I’m sorry to hear you’re still hitting this unexpected issue. We’ve gone through a couple attempts to reproduce this outcome, however we’ve been unable to do so. The process was to use these Docker files, generate the master key using openssl rand 16 | base64, enter it into the env vile and finally run docker-compose.
My colleague just wrote a knowledgebase article, and I’m wondering if comparing his process, and looking at the Linux example he linked in there, might point out any missing piece?
I hope this helps narrow it down, and please let us know how you go!
Well, I guess it is worth to mention that I am able to start both MSSQL and Octopus locally using provided dockerfile and docker-compose.yml.
Also I have no problem with MASTER_KEY locally. I am able to start new Octopus container using old MASTER_KEY using which OctopusDeploy database was created.
The problem I am experiencing is happening in Kubernetes. I was trying to run it using this tutorial:
Thanks for following up! Is it possible you’ve not set the master key in your kubernetes config? This would cause each replica to create different master keys.
If you’re still hitting this issue, and you have a copy of your YAML, would you be willing to send that through so we can test it on a kubernetes cluster?
I look forward to getting to the bottom of this one!
Okay, problem is gone, sharing here the solution, so that it would not be a disaster for those who are trying to have their Octopus Server running in Kubernetes.
If permissions were not a problem for you then try this. Sometimes, HOME by default is set to /root. Therefore, while trying to compare certificates using MASTER_KEY, Octopus might be looking to other directory and the only thing that it is telling you is this exception (which is not really accurate - it cannot decrypt, it cannot find - at least in my case): Failed to decrypt the Octopus Server certificate. This usually indicates the wrong master key is being used to read encrypted data.
SOLUTION: Just to be sure that everything goes smooth, set HOME env variable to /home/octopus/ and you should be good to go.
