DBUp package deployment failed - But deployment task was still running, until manually canceled

Legacy Execution
1

We have this weird issue, when a database deployment package step fails for whatsoever reason during a deployment, the deployment process not stop with failure. I have to manually cancel the deployment - until then it just keeps running (or rather polling for task status?). Octopus version is 3.1.6.2 - Some of the failures are due to login issues or some script failed to execute due to PK constraint etc., In this case - Deployment task was running for about 18 hours after the step failure, before I canceled it manually.

Upgrade failed due to an unexpected exception:
August 22nd 2017 17:07:47Info
System.Data.SqlClient.SqlException (0x80131904): Login failed for user 'XXXXXX'.
August 22nd 2017 17:07:47Info
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling)
August 22nd 2017 17:07:47Info
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
August 22nd 2017 17:07:47Info
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
August 22nd 2017 17:07:47Info
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
August 22nd 2017 17:07:47Info
   at System.Data.SqlClient.SqlConnection.Open()
August 22nd 2017 17:07:47Info
   at DbUp.Engine.Transactions.DatabaseConnectionManager.OperationStarting(IUpgradeLog upgradeLog, List`1 executedScripts)
August 22nd 2017 17:07:47Info
   at DbUp.Engine.UpgradeEngine.PerformUpgrade()
August 22nd 2017 17:07:47Info
ClientConnectionId:c959fc85-3deb-43c5-b2bd-c57000c50dba
August 22nd 2017 17:07:47Info
Error Number:18456,State:1,Class:14
August 22nd 2017 17:07:47Info
System.Data.SqlClient.SqlException (0x80131904): Login failed for user 'XXXXX'.
August 22nd 2017 17:07:47Info
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling)
August 22nd 2017 17:07:47Info
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
August 22nd 2017 17:07:47Info
   at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
August 22nd 2017 17:07:47Info
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
August 22nd 2017 17:07:47Info
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
August 22nd 2017 17:07:47Info
   at System.Data.SqlClient.SqlConnection.Open()
August 22nd 2017 17:07:47Info
   at DbUp.Engine.Transactions.DatabaseConnectionManager.OperationStarting(IUpgradeLog upgradeLog, List`1 executedScripts)
August 22nd 2017 17:07:47Info
   at DbUp.Engine.UpgradeEngine.PerformUpgrade()
August 22nd 2017 17:07:47Info
ClientConnectionId:c959fc85-3deb-43c5-b2bd-c57000c50dba
August 22nd 2017 17:07:47Info
Error Number:18456,State:1,Class:14
2

Hi Kuran,

Thank you for getting in touch. It looks like the upgrader is not exiting and keeps running. You can check this by seeing whether the upgrader process is still running on the machine that is running the step.

Could you please check your upgrader code to make sure you are not waiting for user input when a failure occurs?

If this is not the case, could you send me:

  • the task log
  • a screen shot of the step that runs the upgrader and any scripts for that step if they are not visible in the screenshot
  • the parts of the upgrader source code showing the main method and the creation of the UpgradeEngine, including the PerformUpgrade() call and everything in between?

Robert W