UserDS certificate store causing deployment with a certificate to fail

Hi,

I’m seeing some strange failure when deploying an IIS website with a certificate. Previously worked fine, Octopus managed certificate, deployed a few weeks ago no issues, then last night, about 6 out 16 tenant deployments (separate servers) have suddenly failed with the same error.

On the servers, if I enumerate the store with Get-ChildItem Cert:\LocalMachine I get all the normal stores, but theirs also a UserDS store on the failed deployment servers, but not on the good ones.

When I use

Get-ChildItem Cert:\LocalMachine -Recurse

as Octopus.Features.IISWebSite_BeforePostDeploy.ps1 does, I see an error

“Get-ChildItem : The system cannot find the file specified”

when it gets to the UserDS store. I’m fairly sure this is what’s killing the Octopus deploy script when it tried to setup the bindings.

I have no idea where the UserDS store comes from, or how to reproduce this. It appears to be “Active Directory User Object” in the certlm GUI - however the server isn’t domain joined. The server isn’t 100% under my control so I’m not sure if something else has caused the UserDS to magically appear (or if it was there before and something else has caused problems).

Any suggestions welcome! (Yes I’ve tried rebooting :slight_smile: )

One thing that might be worth implementing in the Octopus deploy script is to ignore certificate / certificate store errors so it stands a chance of still finding the appropriate cert when these sort of errors happen.

Log:

21:37:25 Verbose | Detected IIS Version 10.0
21:37:25 Info | Making sure a Website “” is configured in IIS…
21:37:25 Info | Finding SSL certificate with thumbprint
21:37:46 Error | NotSpecified: The system cannot find the file specified
21:37:46 Error | At \Octopus.Features.IISWebSite_BeforePostDeploy.ps1:485 char:18
21:37:46 Error | + … $certificate = Get-ChildItem Cert:\LocalMachine -Recurse | Where …
21:37:46 Error | + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
21:37:46 Error | at , \Octopus.Features.IISWebSite_BeforePostDeploy.ps1: line 485
21:37:46 Error | at , \Octopus.Features.IISWebSite_BeforePostDeploy.ps1: line 469
21:37:46 Error | at , \Bootstrap.Octopus.Features.IISWebSite_BeforePostDeploy.ps1: line 3257
21:37:46 Error | at , : line 1
21:37:46 Error | at , : line 1

Thanks,

Steve.

Hi Steve,

Thanks for getting in touch!. I’m sorry about the big delay in responding to this question. I noticed your email mentioning this conversation and thought I would copy our response for public visibility on the issue.

We’ve raised this internally and we plan to take a look at whether we can handle this situation more gracefully, along the lines of what you suggested.

Ideally I’d like Octopus to handle a failure to open a certificate store when searching for certificates and not kill the entire process but move onto the other certificate stores

If you’re curious, here is the line of code that is failing on your affected target servers: Calamari/source/Calamari/Scripts/Octopus.Features.IISWebSite_BeforePostDeploy.ps1 at master · OctopusDeploy/Calamari · GitHub

I suspect we’ll need to look at changing this call to Get-ChildItem -Recurse to iterate over each of the certificate stores, in turn, skipping any stores that cannot be opened.

Sorry again for missing this question initially. If you have any further questions or thoughts, please don’t hesitate to reach out.

Best regards,
Daniel

Hi Steve,

On your servers, could you please compare the output of these two commands?

Firstly, the command you mentioned previously that results in the error:

Get-ChildItem Cert:\LocalMachine -Recurse

…versus this alternative (or similar):

Get-ChildItem 'Cert:\LocalMachine\' `
  | Foreach-Object { Get-ChildItem "Cert:\LocalMachine\$($_.Name)" -ErrorAction Ignore }

What we would expect to see on any given server is that the results of the two commands are equivalent (e.g. the same number of certificates with the same thumbprints returned by each command), with the exception that the latter should skip over any inaccessible certificate stores (like UserDS) on any of your servers affected by this issue.

If you could please let us know whether that appears to work as described on your servers, that will help us decide whether to make the corresponding change in Octopus.

Best regards,
Roy

Hi Roy,

Thanks for looking into this. I had a hunch that “-ErrorAction Ignore” might be a suitable work around :slight_smile:

I’ve run both the scripts on one of our offending servers.

Without ErrorAction Ignore:

The listing of certificates breaks with an error when it gets to the UserDS store, which happens to be before WebHosting and My so none of the certificates that Octopus needs to use are listed.

The error is:

Get-ChildItem : The system cannot find the file specified
At line:1 char:1

  • Get-ChildItem Cert:\LocalMachine -Recurse

With ErrorAction Ignore:

The same error is still shown, but as expected it continues to list the other stores, and right at the end lists the contents WebHosting and My stores where our certificates are :smiley:

I’d say that the alternative command (ignoring errors) would work well to resolve the problem we are seeing.

Thanks,

Steve.

Thanks for the update, Steve! We plan to do a bit more testing on our side and we’ll keep you up to date when we make some more progress.

Thanks again,
Roy

Hi Steve,

We’ve started work on a fix, which is expected to roll-out in an upcoming 2020.2.* release of Octopus.

For further updates, you can subscribe to this GitHub issue: https://github.com/OctopusDeploy/Issues/issues/6378.

Please let me know if there’s anything else we can help with in the meantime.

Kind regards,
Roy

Hi Steve,

I’m happy to let you know that the fix has been made in Octopus 2020.2.8, which is currently gradually rolling out to Octopus Cloud and should be ready to download soon, via https://octopus.com/downloads.

If there’s anything else we can help with, please let us know.

Kind regards,
Roy

That’s magic, thanks for your help!

(Yes I’ve got a few other things I need “help” with - well, more feedback on UX stuff around Tenants and variables that either keeps annoying me or has “tricked” me into making mistakes - the same thing repeatedly - Once I get my act together you’ll see a couple of posts from me on the subject, but nothing urgent right now).

Thanks!

Steve.

No problem Steve, happy to help :slight_smile:
Cheers,
Roy