Octopus service stuck

Legacy Scale
1

Hi Team,

We are facing one issue, Octopus service was stuck and all build got failed. Once we restart the octopus service everything working fine.In octopus logs we are getting bellow error.

ERROR An unexpected error occurred while attempting to retrieve and execute a task: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource1 retry) at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Octopus.Shared.TransientFaultHandling.RetryPolicy.<>c__DisplayClass26_0.b__0() in Y:\work\refs\tags\3.3.2\source\Octopus.Shared\TransientFaultHandling\RetryPolicy.cs:line 172
at Octopus.Shared.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) in Y:\work\refs\tags\3.3.2\source\Octopus.Shared\TransientFaultHandling\RetryPolicy.cs:line 213
at Octopus.Core.RelationalStorage.RelationalTransaction…ctor(String connectionString, RetriableOperation retriableOperation, IsolationLevel isolationLevel, ISqlCommandFactory sqlCommandFactory, JsonSerializerSettings jsonSerializerSettings, RelationalMappings mappings, IKeyAllocator keyAllocator) in Y:\work\refs\tags\3.3.2\source\Octopus.Core\RelationalStorage\RelationalTransaction.cs:line 48
at Octopus.Core.RelationalStorage.RelationalStore.BeginTransaction(RetriableOperation retriableOperation) in Y:\work\refs\tags\3.3.2\source\Octopus.Core\RelationalStorage\RelationalStore.cs:line 58
at Octopus.Server.Orchestration.TaskQueueLimiter.HasTooManyRunningTasks() in Y:\work\refs\tags\3.3.2\source\Octopus.Server\Orchestration\TaskQueueLimiter.cs:line 28
at Octopus.Server.Orchestration.TaskQueue.PollForMoreWorkOnBackgroundThread(Object state) in Y:\work\refs\tags\3.3.2\source\Octopus.Server\Orchestration\TaskQueue.cs:line 57
2017-07-27 07:23:16.4416 73 INFO Shutting down the embedded web server

Kindly help me to resolved this issue.

Thanks,
Sandeep Verma

2

Hi Team,

Any update on this issue.

Thanks,
Sandeep Verma

3

Hi Sandeep,

Thanks for contacting us. I’m so sorry it took a while for someone to get back to you. Currently we have a few reports of Octopus Server hanging when there’s too many tasks, and this looks like it might be related. For now the only thing we can suggest is to restart the server.

We are working on the underlying issue. Hopefully we can fix that issue soon.

Thanks
Cam

4

Hi Sandeep,

We’ve recently shipped https://github.com/OctopusDeploy/Issues/issues/3702 and https://github.com/OctopusDeploy/Issues/issues/3701 which should help alleviate pressure on your Octopus Server, and give us a clearer understanding of what else could be going wrong.

Please upgrade to Octopus Server 3.16.1 or newer and let us know if this change helps.

At this point we aren’t able to reproduce the problem in our lab, so we are going to wait and see if this surgical change improves the situation before investigating further.

We’ll continue to track progress on this investigation here: https://github.com/OctopusDeploy/Issues/issues/3592

Hope that helps!
Cam

5

Hi Cam,

Thanks for your reply,

Can you please answer our below queries :

  • After upgrading the Octopus Server from 3.3.2 to Octopus 3.16.1 or newer will resolve our issue?

  • Is upgradation free? if not what is the costing for upgrade.

  • Advantage and disadvantage if we upgrade the octopus server?

Thanks & Regards,
Sandeep Verma | Software Engineer Analyst
Avanade Technology Group
Accenture | Delivery Center for Technology in India
OC:- Sandeep.kuma.verma@accenture.commailto:Sandeep.kuma.verma@accenture.com
Mobile:- +91-8600304896
www.accenture.comhttp://www.accenture.com/ | www.avanade.comhttp://www.avanade.com/

6

Hi Sandeep,

We’re not 100% certain that upgrading will solve your issue. However, upgrading in general will help with performance and other issues as 3.3.2 is quite old now.

Upgrading is free, so there’s no cost there.

Thanks
Cam

7

Hi Sandeep,

Sorry for this but earlier when I said upgrading is free I was mistaken. Upgrading requires a valid maintenance period, if that has expired then you can only upgrade to the latest minor patch that was released while your maintenance was valid.

Here is a list of all the things that have been added/fixed since 3.3.2 in Octopus.

Thanks
Cam