SQL Connection Pool Error in 3.4.13

Legacy Uncategorised
1

We were unable to log into our Octopus server this morning and had to restart the service to get it back up.

Looking at the logs showed this repeated over and over since about 7pm last night.

2016-11-02 00:03:16.3016     16 ERROR  An unexpected error occurred while attempting to retrieve and execute a task: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Octopus.Shared.TransientFaultHandling.RetryPolicy.<>c__DisplayClass26_0.<ExecuteAction>b__0() in Y:\Work\refs\tags\3.4.13\source\Octopus.Shared\TransientFaultHandling\RetryPolicy.cs:line 172
   at Octopus.Shared.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) in Y:\Work\refs\tags\3.4.13\source\Octopus.Shared\TransientFaultHandling\RetryPolicy.cs:line 213
   at Octopus.Core.RelationalStorage.RelationalTransaction..ctor(String connectionString, RetriableOperation retriableOperation, IsolationLevel isolationLevel, ISqlCommandFactory sqlCommandFactory, JsonSerializerSettings jsonSerializerSettings, RelationalMappings mappings, IKeyAllocator keyAllocator, IRelatedDocumentStore relatedDocumentStore) in Y:\Work\refs\tags\3.4.13\source\Octopus.Core\RelationalStorage\RelationalTransaction.cs:line 69
   at Octopus.Core.RelationalStorage.RelationalStore.BeginTransaction(RetriableOperation retriableOperation) in Y:\Work\refs\tags\3.4.13\source\Octopus.Core\RelationalStorage\RelationalStore.cs:line 62
   at Octopus.Server.Orchestration.TaskQueueLimiter.HasTooManyRunningTasks() in Y:\Work\refs\tags\3.4.13\source\Octopus.Server\Orchestration\TaskQueueLimiter.cs:line 28
   at Octopus.Server.Orchestration.TaskQueue.PollForMoreWorkOnBackgroundThread(Object state) in Y:\Work\refs\tags\3.4.13\source\Octopus.Server\Orchestration\TaskQueue.cs:line 57

I understand there have been some previous issues with connection leaks but I don’t see any open for 3.4.13.


https://github.com/OctopusDeploy/Issues/issues/2348

2

Hi David,

Thanks for getting in touch! Are you able to send through the corresponding Octopus Server logs (found in c:\octopus\logs in standard installations).
Feel free to mark this thread as private or email them through to support@ octopus.com. We have specific logging around connections that I am hoping to see in the logs.

Vanessa

3

I’ve emailed the logs in to support@octopus.com.

4

Hi David,

We have a couple of leads that we are hoping is a resolution. I will keep you posted when we have a release out with the changes.
Will you be able to upgrade to 3.5 when it is out?

Vanessa

5

Yes we have been keeping up with the upgrades and will go to 3.5 when it is
released.

6

Hi David,

We have released a few patches into 3.5 now. I have had another developer check and we do think this is the issue that should resolve the problem for you:

So please after you upgrade report back if you see similar or the same occurrence.

Vanessa