In a recent deployment to our production environment Octopus did not apply SSL certificates to the sites. The deployment succeeded, but the application would not run because of SSL requirements. When looking in the logs, I saw messages similar to this one for each binding in IIS:
Finding SSL certificate with thumbprint MYTHUMB
Found certificate: CN=origin.********.ca, OU=PositiveSSL, OU=Domain Control Validated
A different binding exists for the IP/port combination, replacing…
SSL Certificate successfully deleted
SSL Certificate add failed, Error: 1312
I was able to log into each production server and manually associate the certificate with the sites.
A search through the forums brought up http://help.octopusdeploy.com/discussions/problems/16491-fail-to-bind-a-self-signed-ssl-certificate, but it’s pretty old and we’re at a much higher version (2.5.7.384) now.
What else can I do to hunt down this issue and prevent downtime with our next deployment?
Hi Jake,
Thanks for getting in touch - sorry about the difficulties.
There seem to be a few possible causes of this particular error; is it possible there’s some small variation between the thumbprint you’re using, and the one shown for the cert in MMC/Certificate Manager?
The installation script prints:
A different binding exists for the IP/port combination, replacing...
…when the port is bound to a different cert; to avoid any future problems one simple approach might be to check the thumbprint and ensure the binding details are exactly the same in Octopus. With that the case, I don’t think Octopus will attempt to re-bind the certificate as it has done here.
We did make changes to this area in 2.5 so it is also entirely possible the problem is with the script - you may turn up some more useful information by running:
netsh http show sslcert
…on the target machine. Octopus is looking for an entry here with a matching IP address, port and thumbprint (“Certificate Hash”) value. Do you see in the output any reason this might not match up?
Best regards,
Nick
netsh http show sslcert returns the expected thumbprint. For IP:Port it’s showing 0:0:0:0:443 which makes sense. In Octopus we’ve set it up to generate several bindings for the site to IP “*” with different hostnames.
So assuming 0:0:0:0:443 is what Octopus is using when it says “*” for the ip address, it should match up, right?
Yes, that’s correct.
Is there any server you can attempt the deployment to again in order to verify that it is a persistent issue? If so, let me know and I’ll raise a bug for this.
Regards,
Nick
We performed another deployment this morning and the issue persisted. After deployment I had to go into IIS Management on each server and manually associate the certificate with the binding.
One more bit of information – each server has two certificates available. Of course only one matches the requested thumbprint.
Please let me know if I can provide any further information.
Thanks for the follow-up Jake. I’ve raised https://github.com/OctopusDeploy/Issues/issues/1187 - which we’ll update with information as we work through it.
Best regards,
Nick
This is also an issue in my production environment with a cluster of web servers hosting multiple sites. For the moment Octopus cannot really be used for deployments as it renders the sites unavailable due to certificate warnings.
I’m digging into this further today. I should have asked originally:
- Which OS are you running?
- PowerShell version?
- Are you using SNI, or anything similar?
I’ve also prepared a version of our IIS configuration script that writes much more data to the log; is there any machine you can run this on for debugging purposes?
Best regards,
Nick
Some more notes, based on what I’ve been able to find online about this error.
First up, is the certificate in the Local System -> Personal store under MMC? There are a few suggestions on this page: http://social.technet.microsoft.com/Forums/windowsserver/en-US/68452008-a89b-40ba-9927-472efcfafc99/ssl-certificate-add-failed-error-1312 - if the store is correct, are there any other possibilities there that might apply?
On Windows Server 2008 R2 a hotfix may be required: http://support.microsoft.com/kb/981506 (guessing that one is a long shot).
Finally - a bit more of a stretch - the second answer listed on this page suggests a missing configuration item (“Unique container name”) may be the issue and a repair operation required: http://stackoverflow.com/questions/13076915/ssl-certificate-add-failed-when-binding-to-port/19766650#19766650
Just thought I’d share this information since it might help identify something simple; running the instrumented configuration script and sending the details requested above will still help us track this down. Thanks!
Hey Nicholas
Thank you for getting back to me.
First of all I have solved the problem for now by simply letting our Netscaler (load balancer) offload the https connections to one of the two sites involved. This way the IIS only have to handle one certificate, but the end users can access both sites through https.
I have the certificates in localmachine\my and the bindings are correct.
But perhaps I am also asking too much from Octopus, since the problem is also related to IIS configuration.
If I try to setup the bindings manually I have to be able to differentiate the sites on IP address. Binding on hostname alone doesn’t appear to work.
It seems the certificates are bound in the OS and directed by IIS. So if I execute “netsh http show sslcert” I can see that there is only one cert bound on the box, and that this certificate changes depending on the last deployment. This in time is picked up by IIS, so my https bindings are always tied to the certificate from the latest deployment.
As mentioned I have a work around, so this is not urgent. But it would be nice having a solution though.
KR
Morten
C:\Windows\system32>netsh http show sslcert
SSL Certificate bindings:
IP:port : 0.0.0.0:443
Certificate Hash : c36891e5d104d81c9c35800cca29cfae0bf7dbc8
Application ID : {26587bc3-04ae-4f00-833f-e2ac21dc6d48}
Certificate Store Name : (null)
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check : Enabled
Revocation Freshness Time : 0
URL Retrieval Timeout : 0
Ctl Identifier : (null)
Ctl Store Name : (null)
DS Mapper Usage : Disabled
Negotiate Client Certificate : Disabled
IP:port : 0.0.0.0:8172
Certificate Hash : 14172f02bd81768e69a47b6d10380e82b0e46d22
Application ID : {00000000-0000-0000-0000-000000000000}
Certificate Store Name : MY
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check : Enabled
Revocation Freshness Time : 0
URL Retrieval Timeout : 0
Ctl Identifier : (null)
Ctl Store Name : (null)
DS Mapper Usage : Disabled
Negotiate Client Certificate : Disabled
Hey
Tentacles are executing on Windows server 2008.
PS C:\Windows\system32> $psversiontable
Name Value
PSVersion 3.0
WSManStackVersion 3.0
SerializationVersion 1.1.0.1
CLRVersion 4.0.30319.18444
BuildVersion 6.2.9200.16481
PSCompatibleVersions {1.0, 2.0, 3.0}
PSRemotingProtocolVersion 2.2
Regarding SNI: There are hostnames bound to the bindings as some are pm *.domainA.dk and others are *.domainB.dk, but I may not understand your question entirely.
I have a test server where I can run almost anything.
KR
Morten
Our servers are windows server 2012.
PS C:\Users\jakes> $psversiontable
Name Value
PSVersion 3.0
WSManStackVersion 3.0
SerializationVersion 1.1.0.1
CLRVersion 4.0.30319.18449
BuildVersion 6.2.9200.16628
PSCompatibleVersions {1.0, 2.0, 3.0}
PSRemotingProtocolVersion 2.2
The cert is not in the personal store, it’s in Web Hosting.
Unfortunately the issue is only occurring on our production servers, and I cannot deploy to them off-schedule. I’ll see if I can isolate one production server for testing, but that may not be feasible.
We are also looking at removing the SSL certificates from the instances and only doing SSL at the load balancer. We’ll probably be rolling this change out in the next two weeks, effectively solving the issue for us.
Thank you both for following up.
Morten - I believe the binding configuration in Octopus allows the IP address to be specified explicitly - does this resolve the issue for you?
Jake, glad you have an alternative to pursue - is your setup similar to Morten’s in that an explicit IP address might need to be specified? Without a non-prod machine to test on I appreciate it’s a bit tricky to investigate further.
Regards,
Nick
Hey Nicholas
My apologies. I guess IP-binding would solve the problem.
Sincerely
Morten
I suppose we could bind to specific IPs, but our deployments go out to a growing number of instances. Does octopus expose the current tentacle’s IP address(es) through a variable that we could use in the binding?
is there any follow up to this discussion as it is of interest - the SSL bandwidth is not significant enough to use an off-loading/SSL Termination elsewhere and fixing this in Octopus would be excellent.
Hi,
HTTPS requests typically don’t specify a host name, just an IP address, which is why certificates need to be bound to a given IP. However, if your server supports SNI, it is possible to have multiple certificates - we’ve added support for this in Octopus 2.6. Hope that helps!
Paul
Thanks Paul - that’s great to know and we can take the necessary action to integrate automatic certificate distribution.