Could not set security on private-key when deploying package to IIS

pnolan · 22 January 2020 04:04

Hi,

In one of our environments we’re getting the following error when deploying a package to IIS.

I have found another post with what appears to be the same\similar issue but the resolution doesn’t work for us. Could not set security on private-key

Both the app pool and octopus service accounts have system admin rights. They also have full control over the folder C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys.

I have also tried removing the certificates from the certificate store and re-adding.

Do you have any ideas what might be causing this issue?

ERROR
IIS configuration complete
14:48:43 Verbose | 14:48:43 Verbose | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Verbose | 14:48:43 Verbose |
14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Error | 14:48:43 Verbose | 14:48:43 Verbose | 14:48:43 Verbose | 14:48:43 Fatal | Deleting ‘c:\Program Files\Dataract\e5\WebServices\Octopus.Features.IISWebSite_BeforePostDeploy.ps1’
Executing feature-class ‘Calamari.Deployment.Features.IisWebSiteAfterPostDeployFeature’
System.Exception: Could not set security on private-key —> System.Security.Cryptography.CryptographicException: Access is denied.
at Calamari.Integration.Certificates.WindowsX509CertificateStore.SetCspPrivateKeySecurity(SafeCertContextHandle certificate, ICollection1 accessRules) 14:48:43 Error | at Calamari.Integration.Certificates.WindowsX509CertificateStore.AddPrivateKeyAccessRules(ICollection1 accessRules, SafeCertContextHandle certificate)
— End of inner exception stack trace —
at Calamari.Integration.Certificates.WindowsX509CertificateStore.AddPrivateKeyAccessRules(ICollection1 accessRules, SafeCertContextHandle certificate) 14:48:43 Error | at Calamari.Integration.Certificates.WindowsX509CertificateStore.AddPrivateKeyAccessRules(String thumbprint, StoreLocation storeLocation, String storeName, ICollection1 privateKeyAccessRules)
at Calamari.Deployment.Features.IisWebSiteAfterPostDeployFeature.EnsureApplicationPoolHasCertificatePrivateKeyAccess(VariableDictionary variables)
at Calamari.Deployment.Features.IisWebSiteAfterPostDeployFeature.Execute(RunningDeployment deployment)
at Calamari.Deployment.Conventions.FeatureConventionBase.ExecuteFeatureClasses(RunningDeployment deployment, String feature)
at Calamari.Deployment.Conventions.FeatureConventionBase.Run(RunningDeployment deployment)
at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
at Calamari.Deployment.ConventionProcessor.RunConventions()
Running rollback conventions…
Adding journal entry:
Could not set security on private-key
System.Exception
at Calamari.Integration.Certificates.WindowsX509CertificateStore.AddPrivateKeyAccessRules(ICollection1 accessRules, SafeCertContextHandle certificate) 14:48:43 Error | at Calamari.Integration.Certificates.WindowsX509CertificateStore.AddPrivateKeyAccessRules(String thumbprint, StoreLocation storeLocation, String storeName, ICollection1 privateKeyAccessRules)
at Calamari.Deployment.Features.IisWebSiteAfterPostDeployFeature.EnsureApplicationPoolHasCertificatePrivateKeyAccess(VariableDictionary variables)
at Calamari.Deployment.Features.IisWebSiteAfterPostDeployFeature.Execute(RunningDeployment deployment)
at Calamari.Deployment.Conventions.FeatureConventionBase.ExecuteFeatureClasses(RunningDeployment deployment, String feature)
at Calamari.Deployment.Conventions.FeatureConventionBase.Run(RunningDeployment deployment)
at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
at Calamari.Deployment.ConventionProcessor.RunConventions()
at Calamari.Commands.DeployPackageCommand.Execute(String[] commandLineArguments)
at Calamari.Program.Execute(String[] args)
--Inner Exception–
Access is denied.
System.Security.Cryptography.CryptographicException
at Calamari.Integration.Certificates.WindowsX509CertificateStore.SetCspPrivateKeySecurity(SafeCertContextHandle certificate, ICollection1 accessRules) 14:48:43 Error | at Calamari.Integration.Certificates.WindowsX509CertificateStore.AddPrivateKeyAccessRules(ICollection1 accessRules, SafeCertContextHandle certificate)
Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20200122034833-62252-198 exited with code 100
Updating manifest with output variables
Updating manifest with action evaluated variables
The remote script failed with exit code 100

paul.calvert · 22 January 2020 07:48

Hi @pnolan,

Thanks for getting in touch!

I’ve responded to your email you sent through, so we can continue to troubleshoot this there if that is ok?

Regards,
Paul

Geoff.fiedler · 1 October 2021 19:03

I am seeing this same error on one server during deploy. What is/was the resolution?

System.Exception: Could not set security on private-key —> System.Security.Cryptography.CryptographicException: Access is denied.
October 1st 2021 01:02:16
Error
at Calamari.Integration.Certificates.WindowsX509CertificateStore.SetCspPrivateKeySecurity(SafeCertContextHandle certificate, ICollection1 accessRules) October 1st 2021 01:02:16 Error at Calamari.Integration.Certificates.WindowsX509CertificateStore.AddPrivateKeyAccessRules(ICollection1 accessRules, SafeCertContextHandle certificate)
October 1st 2021 01:02:16
Error
— End of inner exception stack trace —
October 1st 2021 01:02:16
Error
at Calamari.Integration.Certificates.WindowsX509CertificateStore.AddPrivateKeyAccessRules(ICollection1 accessRules, SafeCertContextHandle certificate) October 1st 2021 01:02:16 Error at Calamari.Integration.Certificates.WindowsX509CertificateStore.AddPrivateKeyAccessRules(String thumbprint, StoreLocation storeLocation, String storeName, ICollection1 privateKeyAccessRules)
October 1st 2021 01:02:16
Error
at Calamari.Deployment.Features.IisWebSiteAfterPostDeployFeature.EnsureApplicationPoolHasCertificatePrivateKeyAccess(IVariables variables)
October 1st 2021 01:02:16
Error
at Calamari.Deployment.Features.IisWebSiteAfterPostDeployFeature.Execute(RunningDeployment deployment)
October 1st 2021 01:02:16
Error
at Calamari.Deployment.Conventions.FeatureConventionBase.ExecuteFeatureClasses(RunningDeployment deployment, String feature)
October 1st 2021 01:02:16
Error
at Calamari.Deployment.Conventions.FeatureConventionBase.Run(RunningDeployment deployment)
October 1st 2021 01:02:16
Error
at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
October 1st 2021 01:02:16
Error
at Calamari.Deployment.ConventionProcessor.RunConventions()
October 1st 2021 01:02:16

paul.calvert · 1 October 2021 19:18

Hi Geoff,

I believe the issue was caused by AV (specifically McAfee) blocking access to MachineKeys.

Regards,
Paul

Geoff.fiedler · 1 October 2021 19:22

Hi Paul,

Thanks for the quick response. It does not appear to be the same issue for us, as it works as expected on two other servers with identical configuration (Virus scan, permissions, etc.).

Please advise on the best way to troubleshoot and resolve this.

Regards,
Geoff

paul.calvert · 1 October 2021 19:49

If possible it would be worth temporarily disabling the AV on this server and testing a deploy.

You could also check event viewer to see if there are any warnings or errors being logged there.

Geoff.fiedler · 1 October 2021 19:51

Thanks Paul, I’ll see if that is an option. Also, FYI, we don’t use McAfee and the Virus Scan is configured identically to the other servers which succeeded. I rebooted after initial fails, to no avail.

Geoff.fiedler · 6 October 2021 13:59

Hello Again Paul,

We disabled the Virus Scanning on the server exhibiting this error, and have tried numerous other approaches, including removing the sites and rebinding. There are no errors logged on the system for these errors, further indicating the issue is specific to Octopus.

Please advise how we can get additional support for this issue, we have taken the offending sever out of the load balancer as a result and need this back in ASAP. thank you! -Geoff

paul.calvert · 6 October 2021 17:38

If you’d like to send through some deployment task logs and some information on how the step is configured to support@octopus.com we can take a look into this further.

The fact that this is only occurring on one of your targets does suggest that it is environmental in some way. Is the tentacle service on this target running as the same user as on the other targets? And are the permissions for those users all identical?

Another item to check would be Windows Updates, are all of the machines updated identically?

Geoff.fiedler · 9 March 2022 19:24

Hi Paul,

Reaching out again as this was not resolved and is now affecting all target servers in all environments (instead of just one target).

Please let me know what information would be most helpful in troubleshooting at this point.

Regards,
Geoff

paul.calvert · 10 March 2022 08:49

Hi Geoff,

If the issue has started affecting other targets then that does suggest that whatever change was made on the first target has now propagated to the rest.

The error is being generated from the OS when Octopus runs this code to check that the IIS application pool has access to the certificate private key.

With access being blocked by the system, it would suggest that either the user that the tentacle service user has had a permission change applied or security software is blocking access.

Other than tracking down what changes have been made to these machines since the last time they worked you could also try running tools such as process explorer or process monitor on the target when the deployment runs to see if you can identify what is blocking access.

Regards,
Paul