Certificate management feature seems to add duplicate

I just updated Octopus to 3.14.15926 and tested the Certificate management feature. It seems to add a duplicate (but hidden) entry in the certificate store if you publish twice using the import step.

What I did:

  1. Added a X509 certificate into Octopus certificate management.
  2. Created a test project and added the “Import Certificate” step template
  3. Specified to install into a custom user , store=my
  4. Deployed to a test server running a .net application that uses the certificate.
  5. Tested the application successfully
  6. Deployed a second time.
  7. When the application runs, it hits an error because it finds multiple certificates.
  8. Opened MMC and cannot see any duplicate certificates.
  9. Deleting the cert allows me to replicate the steps.

The .NET method that is hitting the error in our application is System.ServiceModel.Security.X509CertificateInitiatorClientCredential.SetCertificate() with the FindType set as X509FindType.FindByThumbprint

The error thrown is:
System.InvalidOperationException: Found multiple X.509 certificates using the following search criteria: StoreName ‘My’, StoreLocation ‘CurrentUser’, FindType ‘FindByThumbprint’, FindValue ‘7e5642d4c3ee04755d910cd3b0f3436bce0c2e99’. Provide a more specific find value.

Hi,

We are currently investigating this issue.

We haven’t diagnosed the exact cause, but I believe we are close.

My testing so far seems to indicate that the problem doesn’t occur when reading the certificate using the X509Store class. e.g.

var store = new System.Security.Cryptography.X509Certificates.X509Store(StoreName.My, StoreLocation.CurrentUser);
store.Open(OpenFlags.ReadOnly);
var foundCertificates = store.Certificates.Find(X509FindType.FindByThumbprint, {{Your thumbprint}}", false);
var count = foundCertificates.Count; // == 1 

It would appear that System.ServiceModel.Security.X509CertificateInitiatorClientCredential.SetCertificate() uses a slightly different implementation to find certificates.

I would be interested if you were able to attempt to find the certificate in your application using code similar to above (using X509Store), to know whether you had similar results?

I’ll update this thread with more information as we dig further.

Regards,
Michael

I tried to replicate today and had different results than I relayed earlier. It appears I am only having an issue if the certificate was installed once via a means other than octopus. In that case it seems like the Octopus installation doesn’t pick up that the cert has already been installed if done through a different manner. The code you provided did return a count of 2 the the case mentioned.

Thank-you for the update. That is interesting.

I have spent some time experimenting with this too. But I must admit I was not able to get that code to return a count greater than one.

The problem seems to be that the certificate file is written to different places, depending on the situation.

For example, it seems that when Octopus imports the certificate for another user (i.e. not the Tentacle) it writes the certificate to the user’s registry.

HKEY_USERS
   UserName
      Software
         Microsoft
            SystemCertificates

Then, when accessing the certificate as that user, the certificate is moved to their roaming profile:

C:\Users\UserName\AppData\Roaming\Microsoft\SystemCertificates\My\Certificates

At this point, importing the certificate again with Octopus will result in the situation you describe; effectively two certificates in the store.

Unfortunately the Windows Crypto API (CAPI) code is not open-source. And I can’t find any definitive documentation for the behaviour in situations like this.

Given we are relying on the Windows libraries, I’m not sure exactly how we would resolve this. As they are always going to be the source of truth for which certificates are installed.

Are your deployments currently in a stable state that you are happy with?

A few other work-arounds I can think of:

Install another instance of the Tentacle service running as the user you wish the certificates to be imported for

Then you could change your Import Certificate step to Current User, and I think all would be well.

This would mean creating a new role for the new Tentacle instance, and modifying your deployment process so that the Import Certificate step only runs on the newly created role.

Set the certificate credential in code rather than config

Using something similar to the approach shown here, you could read the certificate from the store using the X09Store class, rather than relying on System.ServiceModel.Security.X509CertificateInitiatorClientCredential.

One thing I haven’t tried is using a certificate store other than My. I wonder if the behaviour would be different?

Please let me know what you think? If this issue is still causing a problem for you, we’re more than happy to continue to investigate.
But if you’re in a place that you are happy with, we may let it be unless anyone else reports difficulties.

The workaround I have is manageable and I am moving forward with it, but is still inconvenient. I can’t imagine that other Octopus users are not going to run into this.

For any user that we are going to be deploying certificates via octopus, we need to manually remove certificates prior to Octopus certificate implementation to ensure that they are not installed twice. I think that once we are deploying the certificates fully via Octopus we shouldn’t have any problems.

As to your solutions, I’m not sure that they help, but I will have to investigate further.

Install another instance of the Tentacle service running as the user you wish the certificates to be imported for

For security reasons, we run every service as its own user account. So I’m not sure if installing a tentacle for each account would be manageable. We might have 6 different services deployed to the same server via octopus, all using different accounts. I will have to look at this option more because its been awhile since I looked at the implications of having multiple tentacles.

Set the certificate credential in code rather than config
I’m confused by your comment and example. The code sample in the example essentially mirrors the code that I am using.

Let me know if you need me to clarify any detail of how I am hitting the issue. Thanks for all your help. Octopus is great!

Thank-you, we appreciate your honest feedback.

And I take your point.

I have created an issue for this (which you can follow). We are going to dedicate some more time to it, to see if we can find a reliable way to detect when the certificate has already been imported in this case.

I will attempt to also update this thread with any progress.

Please don’t hesitate to reach out if there’s anything we can assist with in the meantime.

And thank-you for the kind words!