2019.6.7 LTS - Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool

reliability
(Lucio Falco) #1

Hello,
we are facing the Timeout issue. The Octopus server freezes during the deployments and there is anything we can do but just wait the timeout error (or kill the server process obv).
I have restarted the Octopus server already.

Logs detail:

2019-09-03 22:14:09.7775 3028 142 ERROR Unhandled error on request: http://octopus.xxx.xxx.biz/api/Spaces-4/deployments ihsixxxxx by : Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource1 retry) at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Nevermore.Transient.RetryPolicy.<>c__DisplayClass26_0.b__0()
at Nevermore.Transient.RetryPolicy.ExecuteAction[TResult](Func1 func) at Nevermore.RelationalTransaction..ctor(RelationalTransactionRegistry registry, RetriableOperation retriableOperation, IsolationLevel isolationLevel, ISqlCommandFactory sqlCommandFactory, JsonSerializerSettings jsonSerializerSettings, RelationalMappings mappings, IKeyAllocator keyAllocator, IRelatedDocumentStore relatedDocumentStore, String name, ObjectInitialisationOptions objectInitialisationOptions) at Octopus.Core.RelationalStorage.RawRelationalStore.BeginTransaction(RetriableOperation retriableOperation, String name) at Octopus.Server.Web.Infrastructure.OctopusRelationalStore.BeginTransaction(RetriableOperation retriableOperation, String name) at Octopus.Server.Web.Api.Actions.DeploymentCreateAction.ExecuteRegistered() at Octopus.Server.Web.Infrastructure.Api.Responder1.Respond(TDescriptor options, NancyContext context)
at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass14_0.<get_Routes>b__1(Object o, CancellationToken x)
at Nancy.Routing.Route`1.d__7.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Nancy.Routing.DefaultRouteInvoker.d__2.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Nancy.Routing.DefaultRequestDispatcher.d__5.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Nancy.NancyEngine.d__22.MoveNext()

Thank you for your time.
Regards.
L.

(Tina) #4

Hi Lucio,
Thanks for reaching out. Sorry for the delayed response and to hear you’re getting these timeout errors.

The error you reported may indicate two possible scenarios:

  1. Your SQL Queries are taking a long time, exhausting the SQL Connection Pool. Investigate what might be making your SQL Queries take longer than they should and fix that if possible.

  2. Your Octopus Server may be under heavy load. If your SQL Server can handle more load from Octopus, you can increase the SQL Connection Pool size of your Octopus Server node. This will increase the number of active connections Octopus is allowed to open against your SQL Server, allowing Octopus to handle more concurrent requests.

Take a look at our Performance Troubleshooting Guide, specifically #6.

I hope this helps and let me know if you have any further questions or concerns.

(Lucio Falco) #5

Hi Tina, thanks for your reply.
How can I increase the Octopus SQL connection Pool size?

(Tina) #6

Hey Lucio,
Sorry for the delayed response.

In our Troubleshooting Guide, https://octopus.com/docs/administration/managing-infrastructure/performance#troubleshooting, at the bottom of tip #6 describes how to increase the SQL Connection Pool Size.

Give this try and let me know how it goes.

Best Regards,
Tina

(system) closed #7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.