Octopus.Server.exe export-certificate

Hi.

I am trying to export my current Octopus Server certificate like described in How to regenerate certificates with Octopus Server and Tentacle - Octopus Deploy

I Run into the below error. Can someone point me to the reason of the below error?

PS C:\Program Files\Octopus Deploy\Octopus> .\Octopus.Server.exe export-certificate --instance OctopusServer --export-pfx=“E:\Octopus\Certs\oldcert.pfx” --pfx-password SecretPW
Checking the Octopus Master Key has been configured.
Making sure it’s safe to upgrade the database schema…
Ensuring pre-conditions for upgrading the database are satisfied…
Searching for indexes that might upset the database upgrade process…

  • “PASS”: “All columns use the default collation.”
  • “PASS”: “Skipping the license compliance check since the database already has the expected schema. No changes are required.”
  • “PASS”: “The version of your SQL Server satisfies Octopus Server installation requirements.”
  • “PASS”: “There aren’t any unexpected indexes in the database.”
    Executing always run pre scripts…
    Executing TSQL Database Server script ‘Script0000 - Set highest available compatibility level.sql’
    Current COMPATIBILITY_LEVEL for Octopus is set to 150
    Ensuring COMPATIBILITY_LEVEL for Octopus is set to 150
    COMPATIBILITY_LEVEL for Octopus is already 150 or higher
    Executing TSQL Database Server script ‘Script0001 - Remove Namespace Prefix From Script Names.sql’
    Preparing to remove namespace prefix from script names in the [SchemaVersions] table…
    Table [SchemaVersions] exists, inspecting script names…
    Script names are not prefixed, nothing to do.
    Always-run pre scripts “succeeded” after 029ms.
    Checking database schema upgrade history…
    Checking to see if database schema upgrade is required…
    Database already has the expected schema. No changes are required.
    Executing always run post scripts…
    Executing TSQL Database Server script ‘Script0000 - Refresh Views.sql’
    Refreshing view dbo.Dashboard
    Refreshing view dbo.IdsInUse
    Refreshing view dbo.MultiTenancyDashboard
    Refreshing view dbo.Release_WithDeploymentProcess
    Refreshing view dbo.RunbookSnapshot_WithRunbookProcess
    Always-run post scripts “succeeded” after 038ms.
    Database upgrade “succeeded” after 383ms.
    Exporting certificate…
    Key not valid for use in specified state.
    Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Key not valid for use in specified state.
    at Internal.Cryptography.Pal.StorePal.Export(X509ContentType contentType, SafePasswordHandle password)
    at Internal.Cryptography.Pal.CertificatePal.Export(X509ContentType contentType, SafePasswordHandle password)
    at System.Security.Cryptography.X509Certificates.X509Certificate.Export(X509ContentType contentType, String password)
    at Octopus.Server.Commands.ExportCertificateCommand.Start() in ./source/Octopus.Server/Commands/ExportCertificateCommand.cs:line 58
    at Octopus.Shared.Startup.AbstractCommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) in ./source/Octopus.Shared/Startup/AbstractCommand.cs:line 100
    at Octopus.Shared.Startup.OctopusProgram.Start(ICommandRuntime commandRuntime) in ./source/Octopus.Shared/Startup/OctopusProgram.cs:line 504
    at Octopus.Shared.Startup.ConsoleHost.Run(Action`1 start, Action shutdown) in ./source/Octopus.Shared/Startup/ConsoleHost.cs:line 34
    at Octopus.Shared.Startup.OctopusProgram.RunHost(ICommandHost host) in ./source/Octopus.Shared/Startup/OctopusProgram.cs:line 221
    at Octopus.Shared.Startup.OctopusProgram.Run() in ./source/Octopus.Shared/Startup/OctopusProgram.cs:line 167
    "-------------------------------------------------------------------------------
    Terminating process with exit code 100
    Full error details are available in the log files at:
    C:\Octopus\Logs

Kind regards
Kasper

Hey Kasper,

Thanks for reaching out and for all the information. Are you by chance running this command vs a newly stood up backup of your DB/Instance?

We typically see this when trying to access an Octopus instance using a backup on a different machine. We use Windows DPAPI to encrypt the master key in the config file, so it will fail to decrypt on a different machine. If so, we will need to get the master key from your other instance with this command, and then set it on the new instance with this command.

Please let me know if that’s the case or if you have any questions.

Best,
Jeremy

Hey Jeremy.
I just migrated the instance to a new server and shut down the old server. The database is located on a seperate sql server. I just checked the master key. It is identical on both the old and new server.

Hey Kasper,

Just to confirm, did you check via the octopus.server.exe command, or by checking the config manually? If you used the config, could you please use the octopus.server.exe command just to confirm as it will be a different value than the config.

Best,
Jeremy

Hi Jeremy. Thank you for getting back to me.

I can confirm that I used “octopus.server show-master-key” and that the keys are the same.

Hey Kasper,

Thanks for confirming.

Since you have the old one booted up, as a test, does the export certificate work on that one?

Best,
Jeremy

Hi Jeremy. I can confirm that it works on the old server.

Hey @kbl_sg,

Just jumping in here for Jeremy who is currently offline as part of our US based team.

You mentioned you migrated your instance to a new server and the SQL DB is on a separate box. When you migrated over the new server did you copy the Octopus folders over by any chance?

As Jeremy mentioned in the config files for Octopus there is a master key and it shows as an encrypted string in the config file. That master key is actually encrypted with the Machine ID of the machine that the Octopus Server is running on.

If you migrate the Octopus Server to a new machine and copy the old config file over you may see issues with the master key as it does not quite match what your DB has and also the encryption in the config file as it will point to the old machines Machine ID.

We have also seen it where a customer has installed Octopus on C on their old VM, they then install it on D (for example) on the new VM, they then copy the files from the old server onto the C drive. When they go to check the config files they look in the C drive not the D drive so they are looking at the wrong config files. To make sure you are seeing the correct one you can navigate to C:\ProgramData\Octopus\OctopusServer\Instances and you will see an octopusserver.config file in there, if you open that you will see the location of the config file Octopus is actually using ie:
"ConfigurationFilePath": "C:\\Octopus\\OctopusServer.config"

I am not quite sure why that error would be thrown on the certificate export, I know it includes the Octopus thumbprint and I assume it would contain some information on the VM itself (possibly the machine ID), but I can see we had another ticket in from a user who migrated their server onto a new VM and copied the Octopus folder over with the old config file.

They got this error in their logs:

Unable to parse configuration key 'Octopus.Storage.MasterKey' as a 'Byte[]'.
System.FormatException

--Inner Exception--
Key not valid for use in specified state.
Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException

Whilst this was on an install and boot up of their server I am wondering if you possibly installed Octopus on the new VM by installing the MSI for Octopus and then pointing it to your external DB and then you copied the Octopus folder over from the old VM with the old config files?

I may be off base here, when I googled the error it mentioned the error occurs when the certificate in the windows store is not set with the private key being exportable but since I have not seen this error before with certificate exports I assume that is not the case unless you physically changed the Octopus Certificate which you will not have done.

Are you able to confirm that when you migrated Octopus onto the new VM you did not copy any of the old config files over and also you only have Octopus installed in one place (ie on the C drive) and your config file in programdata is pointing to the one you are looking at for this issue.

If this is not the case and you have not copied anything over and you are using the usual C drive for the install we will continue to troubleshoot with you.

I look forward to hearing from you,
Kind Regards,
Clare

Hi @clare.martin.

I did the migration according to your instructions on Moving the Octopus Server - Octopus Deploy

I have copied the 3 folders but not the configuration file.

I can also confirm “ConfigurationFilePath”: “C:\Octopus\OctopusServer.config”

kind regards
Kasper

Hey Kasper,

Are there any environmental differences between the 2 boxes such as OS, or IISCrypto may be locked down to not read certain types of SHA that the other box is allowed to?

Best,
Jeremy

The old box was Server 2012 R2 and the new one is Server 2022 with default IIScrypto configuration.

Thanks, while I investigate that, within Octopus, what type of SHA is this certificate you’re trying to export?

(Sorry, you’ll find this in Configuration->Thumbprint)

I am trying to eksport a SHA1 certificate.

Hey Kasper,

Thanks for confirming that. I’m going to do some testing on this, but in the event of needing to roll back you should be able to use the certificate you exported from the old server. Are you unblocked at this point to move forward or do you have other questions or concerns.

Best,
Jeremy

Hey @jeremy.miller

Can you confirm that I won’t risk locking myself out or getting in an inconsistant db state when following the rest of the guide at How to regenerate certificates with Octopus Server and Tentacle - Octopus Deploy? Wouldn’t I risk the same issue when I import the old certificate during a rollback?

SHA1 + Windows Server 2022 is not something we’ve explicitly tested against. Modern versions come with SHA256. If you want to do a dry run to be sure that you won’t be in a bad position potentially in the future, you could:

  1. Do a database backup
  2. Try to import the certificate
  3. If it succeeds you should be good to go, if it fails (it will likely fail before it tries to import, so you won’t be in a bad state), and puts you in a bad state then revert to the backup.

Please let me know if you do go ahead and test that scenario. We are still digging in to see if the export failure is something we can account for on our side but I do think it’s due to it being an older cert combined with Windows 2022.

Best,
Jeremy

I made a database backup and imported the old certificate on the new server. It completed without errors, but since it was already installed it did not change anything.

The existing certificate “CN=Octopus Portal” had the same thumbprint as the new certificate (“[thumbprint]”). Nothing to do.

Thanks for confirming it didn’t error out. You should be good to go with the rest of your process.

Please let me know if you have any other questions or concerns.

Best,
Jeremy

Thank you for your assistance so far. Can you confirm that the proces outlined in How to regenerate certificates with Octopus Server and Tentacle - Octopus Deploy only applies to listening tentacles? It looks like polling tentacles only support trusting 1 thumbprint at the same time? With Tentacle.exe update-trust? In other words I can’t update the server certificate until I have verified that all polling tentacles have had a service restart?

Hi Kasper,
Just stepping in for Jeremy while he is offline.

You’re right, the docs are not very clear on this topic with respect to polling tentacles. In fact I had to consult with our engineers to confirm the best way to update the polling tentacles and it turns out we are opening an issue on this as we speak so we can address polling tentacle updates in a better way.

For now the best I can recommend is to reinstall the polling tentacles fully as the most foolproof method to update the cert. While it is actually possible to have 2 thumbprints on a polling tentacle the behaviour is not as predictable as we’d like and can cause more issues than it solves.

I hope this isn’t too much of an issue for you and you get all the targets updated successfully.

Kind regards,