2019.6.7 LTS - Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool

reliability
(Lucio Falco) #1

Hello,
we are facing the Timeout issue. The Octopus server freezes during the deployments and there is anything we can do but just wait the timeout error (or kill the server process obv).
I have restarted the Octopus server already.

Logs detail:

2019-09-03 22:14:09.7775 3028 142 ERROR Unhandled error on request: http://octopus.xxx.xxx.biz/api/Spaces-4/deployments ihsixxxxx by : Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource1 retry) at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Nevermore.Transient.RetryPolicy.<>c__DisplayClass26_0.b__0()
at Nevermore.Transient.RetryPolicy.ExecuteAction[TResult](Func1 func) at Nevermore.RelationalTransaction..ctor(RelationalTransactionRegistry registry, RetriableOperation retriableOperation, IsolationLevel isolationLevel, ISqlCommandFactory sqlCommandFactory, JsonSerializerSettings jsonSerializerSettings, RelationalMappings mappings, IKeyAllocator keyAllocator, IRelatedDocumentStore relatedDocumentStore, String name, ObjectInitialisationOptions objectInitialisationOptions) at Octopus.Core.RelationalStorage.RawRelationalStore.BeginTransaction(RetriableOperation retriableOperation, String name) at Octopus.Server.Web.Infrastructure.OctopusRelationalStore.BeginTransaction(RetriableOperation retriableOperation, String name) at Octopus.Server.Web.Api.Actions.DeploymentCreateAction.ExecuteRegistered() at Octopus.Server.Web.Infrastructure.Api.Responder1.Respond(TDescriptor options, NancyContext context)
at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass14_0.<get_Routes>b__1(Object o, CancellationToken x)
at Nancy.Routing.Route`1.d__7.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Nancy.Routing.DefaultRouteInvoker.d__2.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Nancy.Routing.DefaultRequestDispatcher.d__5.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Nancy.NancyEngine.d__22.MoveNext()

Thank you for your time.
Regards.
L.

(Tina) #4

Hi Lucio,
Thanks for reaching out. Sorry for the delayed response and to hear you’re getting these timeout errors.

The error you reported may indicate two possible scenarios:

  1. Your SQL Queries are taking a long time, exhausting the SQL Connection Pool. Investigate what might be making your SQL Queries take longer than they should and fix that if possible.

  2. Your Octopus Server may be under heavy load. If your SQL Server can handle more load from Octopus, you can increase the SQL Connection Pool size of your Octopus Server node. This will increase the number of active connections Octopus is allowed to open against your SQL Server, allowing Octopus to handle more concurrent requests.

Take a look at our Performance Troubleshooting Guide, specifically #6.

I hope this helps and let me know if you have any further questions or concerns.