Upgrade Linux Container

ianburrows.sa · 13 July 2021 09:17

Hi,
We are trying to upgrade from 2020.6.4855 to 2021.1.7500 but have run into an issue where the upgraded container cannot connect to the external database. I exec into the pod and have confirmed that the connection string is still the same, master key has not changed. When I rollback to 2020.6.4855 it connects again without any problems. I spoke to the DB admins and they are not sure what could be blocking the connections. Has something changed in the script which tests the DB connection? I looked at the install.sh script but was unable to find where the connection is tested. It only checks the master key.

The error we see is:

======== Initializing DB ========
Using supplied Master Key
Setting master key
Setting database connection string to: server=SERVER_ADDRESS;database=T_OCTOPUS;user=Octopus_Admin;password=********
========== SQL Database Connection Details ========
SQL Server: SERVER_ADDRESS
SQL Database: T_OCTOPUS
Authentication Type: SQL Server Credentials
User: Octopus_Admin

===================================================
Testing connection to the 'master' database...
Connection test to master timed out after trying 17 times over 00:01:00.9699846.
Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in /opt/buildagent/work/c42ca9d2b8233f1c/source/Octopus.Core/RelationalStorage/SqlServerManager.cs:line 326
ClientConnectionId:00000000-0000-0000-0000-000000000000
We are having trouble working with your SQL Database. This will usually be something you can fix. If not please contact support@octopus.com for help. Here are the error details: SQL Error 0 - A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
Microsoft.Data.SqlClient.SqlException
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in SqlServerManager.cs:line 326
   at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in SqlServerManager.cs:line 346
   at Octopus.Server.Commands.DatabaseCommand.Start() in DatabaseCommand.cs:line 110

When we rollback we see a successful connection:

======== Initializing DB ========
Using supplied Master Key
Setting master key
Setting database connection string to: server=SERVER_ADDRESS;database=T_OCTOPUS;user=Octopus_Admin;password=********
========== SQL Database Connection Details ========
SQL Server: SERVER_ADDRESS
SQL Database: T_OCTOPUS
Authentication Type: SQL Server Credentials
User: Octopus_Admin

===================================================
Testing connection to the 'master' database...
Successfully connected to 'master' database!
Database 'T_OCTOPUS' already exists on SQL Server at 'SERVER_ADDRESS, no need to create it.
Checking the Octopus Master Key has been configured.

The DB admins only said that connecting to the master db shouldn’t happen as this user has no access there but it works on 2020.6.4855.

Thanks for the assistance.
Kind Regards,
Ian

jeremy.miller · 13 July 2021 16:15

Hey @ianburrows.sa,

Thanks for reaching out and for all of the information.

I spun up version 2020.6.4855 in a container and upgraded to 2021.1.7500 both with a custom docker network and without and had no issues.

Here are the commands I used:

docker run --name SQLServerLinux --hostname SQLServerLinux -d -p 1433:1433 -e "SA_PASSWORD=***********;" -e "ACCEPT_EULA=Y" --net octonet mcr.microsoft.com/mssql/server 

docker run --interactive --detach --name OctopusDeploy --hostname OctopusDeploy --publish 8080:8080 --env ACCEPT_EULA="Y" --env DB_CONNECTION_STRING="Server=172.17.0.1,1433;Initial Catalog=Octopus;Persist Security Info=False;User ID=sa;Password=***********;MultipleActiveResultSets=False;Connection Timeout=30;ConnectRetryInterval=1;ConnectRetryCount=3;Min Pool Size=2" --env ADMIN_USERNAME=admin --env ADMIN_PASSWORD=*********** --env ACCEPT_OCTOPUS_EULA=Y --net octonet octopusdeploy/octopusdeploy:2020.6.4855

docker exec -u root -t -i OctopusDeploy /bin/bash
./Octopus.Server show-master-key

docker kill OctopusDeploy
docker rm OctopusDeploy

docker run --interactive --detach --name OctopusDeploy --hostname OctopusDeploy --env MASTER_KEY="*****************" --publish 8080:8080 --publish 10943:10943 --env ACCEPT_EULA="Y" --env DB_CONNECTION_STRING="Server=172.17.0.1,1433;Initial Catalog=Octopus;User ID=sa;Password=***********" --env ADMIN_USERNAME=admin --env ADMIN_PASSWORD==*********** --env ACCEPT_OCTOPUS_EULA=Y --net octonet octopusdeploy/octopusdeploy:2021.1.7500

I did the same with no custom network and had no issue. Is there anything in my reproduction I am possibly missing?

Please let me know.

Best,
Jeremy

ianburrows.sa · 13 July 2021 17:56

Sorry I forgot to mention we are using the helm chart. I tried it locally on a minikube cluster and changing the image to the new version running helm upgrade and it upgraded locally but it isn’t working on our test deployment. Is there any way to debug this?

jeremy.miller · 13 July 2021 18:21

Hey @ianburrows.sa,

Would you be able to either post your helm chart here or direct message it to me so I can take a look please?

Let me know.

Best,
Jeremy

finnian.dempsey · 14 July 2021 00:43

Hi Ian,

Just stepping in for Jeremy as he’s signed off for the day.

Thanks for sending the Helm chart through, confirming we have received it and will look into it to see if there are any obvious issues.

We’ve seen some issues recently where a change to the SQL Client, which was updated in the 2021.1 release, has resulted in some connection issues. I believe it’s this change that might be effecting your ability to connect to the DB, where SQL connections are defaulting to encrypt=true in 2021.1. Would you please be able to share any flags configured for the connection string you are using? Previous issues were resolved by adding a flag “TrustServerCertificate=Yes” to the string.

I’ll keep you posted if I find any potential causes in the Helm chart, otherwise please let me know about the flags used for the connection string and I’ll look into them further.

Best Regards,

ianburrows.sa · 14 July 2021 06:31

Hi Finnian,
Unfortunately that was not successful.

======== Initializing DB ========
Using supplied Master Key
Setting master key
Setting database connection string to: server=SERVER;database=T_OCTOPUS;user=Octopus_Admin;password=********;trustservercertificate=Yes
========== SQL Database Connection Details ========
SQL Server: SERVER
SQL Database: T_OCTOPUS
Authentication Type: SQL Server Credentials
User: Octopus_Admin

===================================================
Testing connection to the 'master' database...
Connection test to master timed out after trying 17 times over 00:01:00.9821786.
Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in /opt/buildagent/work/c42ca9d2b8233f1c/source/Octopus.Core/RelationalStorage/SqlServerManager.cs:line 326
ClientConnectionId:00000000-0000-0000-0000-000000000000
We are having trouble working with your SQL Database. This will usually be something you can fix. If not please contact support@octopus.com for help. Here are the error details: SQL Error 0 - A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
Microsoft.Data.SqlClient.SqlException
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in SqlServerManager.cs:line 326
   at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in SqlServerManager.cs:line 346
   at Octopus.Server.Commands.DatabaseCommand.Start() in DatabaseCommand.cs:line 110

Kind Regards

jeremy.miller · 14 July 2021 15:41

Hey @ianburrows.sa,

I, unfortunately, haven’t been able to get to the bottom of this one yet. I’m going to reach out to our SME for Helm and have him take a look at this. He signs on in about 7 hours so I will likely have an answer for you tomorrow.

Please feel free to reach out in the meantime.

Best,
Jeremy

ianburrows.sa · 14 July 2021 16:12

Thanks Jeremy,
I have also asked the DB admins for information on what the setup is and will also have answer tomorrow.
Cheers,
Ian

jeremy.miller · 14 July 2021 16:52

Sent you a DM with the helm charts we used that worked on our end.

jeremy.miller · 14 July 2021 21:29

Hi @ianburrows.sa,

I had one of our engineers take a look at your helm chart and he didn’t see anything that might be causing this. Once you’ve got the pod running, can you exec in and run these commands?

apt-get update
apt-get install telnet
telnet <sql server address> 1433

This will give us a look at the connection while bypassing any Octopus settings.

Please let me know how it goes.

Best,
Jeremy

jeremy.miller · 15 July 2021 14:04

Hey @ianburrows.sa,

Another thing we could test if our telnet test isn’t productive, is having you move up to a later version of 2020.6 rather than move to 2021.1 and see if the same problems occur.

Please let me know how the tests go.

Best,
Jeremy

ianburrows.sa · 15 July 2021 14:41

Hi Jeremy,
I tried telnet and it returned connection refused from within the pod. I tried it locally and it also can’t connect. I have successfully upgraded to 2020.6.5259 however. It connected successfully and did the database upgrades.
Best,
Ian

jeremy.miller · 15 July 2021 15:36

Hey Ian,

Thanks for the information.

Is there an appliance that sits outside of your DB that we could check the logs to see why that connection is getting refused?

Please let me know.

Best,
Jeremy

system · 15 August 2021 15:36

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.