I’m attempting to update the SSL cert that the Octopus service uses. In Octopus Manager, I clicked “Change bindings…” I removed the old bindings and created a new binding, specifying the new cert, which I’ve already added to the certificate store. However, Octopus continues to use the old cert, as verified by two different browsers.
I’ve even removed the old cert from the machine entirely, set up the new binding using the new cert in Octo Manager, only to find that the old cert comes back (both in the certificate store and used by Octo server as reported by the browser).
I’ve even…
Uninstalled Octo
Removed the cert from the machine:
dir cert:\ -recurse |? {$_.SerialNumber -eq $serialNum} | Remove-Item
Restarted the machine
installed Octo
Didn’t have to do any configuring, so clearly the data are still living somewhere
set up the binding using the correct cert
Somehow, it’s still using the wrong cert and…
dir cert:\ -recurse |? {$_.SerialNumber -eq $serialNum}
… shows that the old cert is back.
I’m guessing maybe the cert is persisted somewhere other than the Certificate Store? I see that I could use this article to clear out data in a few different spots, but I’m not seeing anything about the cert that the Octo service itself uses (as opposed to the certs it uses for deployments).
Is there any light that you can shed upon this for me? I’d hate to uninstall / reinstall again and obliterate all of my current configuration.
Thanks for getting in touch, I’d be happy to help with rotating your Octopus Server’s HTTPS certificate!
Our documentation page Expose the Octopus Web Portal over HTTPS goes into detail about how to rotate the current TLS certificate, including how to import an exisiting certificate.
I’d like to double check that the after making the changes to the bindings, that they are being applied afterwards? There should be a window after clicking the Next button that allows for applying the changes or viewing the script that will get applied:
Essentially what happens is that the SSL certificate thumbprint is stored in the Octopus.Config file, which isn’t removed when uninstalling Octopus and so it’s picking up the old certificate thumbprint value:
Clearing any existing SSL certificates on 0.0.0.0:443.
Adding SSL certificate with thumbprint {redacted} on 0.0.0.0:443.
Updating thumbprint on 0.0.0.0:443;{redacted}
Listen Prefixes after:
0.0.0.0:443;{redacted}
Successfully registered SSL certificate.
Moreover, that redacted thumbprint does show up in the OctopusServer.config, as you indicated. So far, so good.
However, the browser is still seeing the incorrect cert. I had already tried in Chrome and Firefox. I also checked with Edge to make sure I’m not crazy and found the same result.
No problem at all, cheers for providing that extra info!
Is the Octopus Service restarting ok afterward changing the certificate? A restart of the VM would surely do this however, could you please confirm that the correct certificate is being used when running netsh http show sslcert?
I’d be happy to check over your OctopusServer logs to see if anything strange is going on, you should be able to upload them to our secure upload portal here.
The service is restarting OK after changing the cert. To my surprise netsh http show sslcert is showing the correct configuration, as far as I understand:
IP:port : 0.0.0.0:{random-seeming non-http port}
Certificate Hash : {another thumbprint}
Application ID : {an app ID that all the bindings share, except for the binding above, so I suppose these are IIS bindings and the above is Octo}
… the above is making me think to mention that a couple other sites are hosted on this box, but I’ve had IIS stopped throughout our dialog and tests.
Yes, the server logs, great idea: I’m a bit embarrassed that I haven’t thought to comb through that myself. I’m about to upload the log from 5/2. I’m seeing some interesting output:
2023-05-02 10:03:41.1011 30636 180 INFO NonCloningRawFullTableCache GetTableInternal WhenTableHasChanged Subscription "succeeded" after 004ms.
2023-05-02 10:03:41.1251 30636 180 WARN Server certificate is not valid
2023-05-02 10:03:41.1368 30636 180 WARN Connection attempt to SMTP host failed
MailKit.Security.SslHandshakeException: An error occurred while attempting to establish an SSL or TLS connection.
Wow, so our SMTP relay might have an expired cert. I’ll definitely speak to a colleague about that. That seems like it’d be an odd cause of what I’m seeing, so maybe there’s something else at issue too?
I’m also going to upload the log from 5/4, where I see output to the effect of the successful installation of the cert.
In any case, I’ll look into the SMTP relay cert, and I look forward to your analysis of the two log files. Please let me know if I can provide any further data or context.
Cheers for the update and uploading the logs, confirming we’ve received them ok!
I agree that while it could be the issue, it sounds like there is something else at play still as certs on different ports shouldn’t be impacting the other certs on different ports. I’ve also had a quick search for that error in past Tickets and it normally surfaces as a just a warning without causing any access issues.
Do you happen to have HSTS configured on the Octopus Server? We have a warning that indicates that it needs to be removed from each browser manually if reverting back to HTTP, I wonder if this is applying during your certificate rotation:
Reverting from HTTPS to HTTP will not be simple - each browser will need to be manually reconfigured to remove the HSTS entry.
If you are able to send through a HAR file of when you are trying to access Octopus, that should tell us more about which service is using the incorrect certificate.
Looking forward to getting to the bottom of this one!
I’ve uploaded two files. One is the har file that you requested. The other is the full output of netsh http show sslcert . I think you’re right that everything is fine there, but I figured more data couldn’t hurt.
I attempted to clear out old SSL certs on the client side.
Per this article, in Chrome, removed the following from the browser history:
Cached images and files
Hosted app data
Content licenses
Cookies and other site data, for Chrome version 64
I attempted to navigate to my Octopus URL, and it was here that I did the capture to generate the har file. No change; still the wrong cert.
Continuing with the clearing out old certs, I’m on a Windows machine, so I tried the recommendation here to “Clear SSL State” from Internet Options. Still seeing the wrong cert via both Firefox and Edge. (Didn’t try with Chrome, because I didn’t want to do a browser restart at the moment.)
You asked about HSTS: it’s not configured.
Another set of facts, which don’t feel relevant, but for completeness’ sake, since we’re really running out of troubleshooting avenues: the URL for this Octopus instance is octodeploy.local.sa.ucsb.edu, which doesn’t have a proper DNS entry; I just have a Windows hosts file entry for it: 127.0.0.2 octodeploy.local.sa.ucsb.edu. I don’t think this is relevant, though, as other colleagues have the same setup with Octopus running locally, a hosts file entry allowing the host name to resolve to the local machine, and they’re using the same cert.
Thanks for the confirming that and uploading those files, confirming we’ve received them ok!
I’ll dig into them and see if I can spot what’s going on but from a quick glance I can see unique ports being used however I did see that you are using a Central Certificate Store for 443 at the very end of the output of the netsh http show sslcert command:
Central Certificate Store : 443
Certificate Hash : (null)
Turns out the old / wrong cert was present in the Central Certificate Store in IIS. (Sorry, it must have been I who did that, but I have no recollection of doing it.) Once I deleted that, I just deleted the old binding and created the new binding in Octopus Manager, and the browser was seeing the correct cert.
I really appreciate your thorough and friendly help!
