We have a scheduled script that picked up the 2021.1.7149 update this morning, and tried to install it. However, when it tried to restart the Octopus service, it failed with:
2021-05-19 07:02:48.2203 5324 8 INFO The Octopus Server is starting...
2021-05-19 07:02:48.5807 5324 8 FATAL SQL Error -2146893016 - A connection was successfully established with the server, but then an error occurred during the login process. (provider: SSL Provider, error: 0 - The received certificate has expired.)
The received certificate has expired.
Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: SSL Provider, error: 0 - The received certificate has expired.)
---> System.ComponentModel.Win32Exception (0x80090328): The received certificate has expired.
So it turns out the SQL Server certificate was expired, but had likely been so for several months. So, was there perhaps something present in the 2021.1.7149 release that caused this to suddenly be an issue?
We’ve mitigated the problem whilst a new cert is issued by adding “TrustServerCertificate=True” to the connection string.
Thanks for reaching out, and sorry to hear that you had issues following your upgrade. Since we use SQL connection pooling, I think what happened here is that we likely had a connection established while the cert was still valid, and we were still using those pool connections until you restarted the service as part of the upgrade process. When it attempted to reconnect, and the cert was now expired, and it started throwing the error.
I hope this helps explain the likely cause of this scenario! Please don’t hesitate to reach out if you have any further questions.
Hi Justin!
Thanks very much for your swift reply. Yes, that was my theory too, except that we switched to using the current shared DB server in February and it appears the certificate that the DB server is using would have already have been expired by then. (It seems most services aren’t concerned about an expired certificate!) Plus Octopus Deploy does restart each week, so it should logically have been an issue on an earlier version. That’s why it appears the problem was only surfaced by this particular Octopus update. Or perhaps a .NET update?
Thanks again!
That’s very interesting indeed. I don’t see anything in our changelogs for 2021.1 that would indicate any changes that would be the case specifically in 2021.1, but I will run it past our product engineers when they’re back on-deck in a few hours.
What version of Octopus were you running prior to your upgrade today?
Just a quick update here, we have had another report of this come through this morning, and it does seem like the SQL connection is defaulting to encrypt=true in 2021.1 in some cases. I’m thinking this is likely stemming from an update to the SqlClient package.