Deployments failing with SQL Server exception A transport-level error has occurred when receiving results from the server

Octopus Server
1

We started seeing this error a few months ago and it continually gotten worse to where its happening multiple times per day. The exception detail are below. i have uploaded the diagnostics logs under our account for support. To get the deployment to complete we must restarted the Octopus Deploy service and it then works for a few more hours before failing again. Please advise as this is becoming more and more disruptive.

Currently running build: Download: 2021.3.12461 - Octopus Deploy

Error while executing SQL command in transaction 'DeploymentTaskController|ServerTasks-263739-4bd0e8c0-b553-48a2-ae24-73822cd63ba3|T57': A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The I/O operation has been aborted because of either a thread exit or an application request.) The command being executed was: SELECT TOP 1 Id,Name,Created,EnvironmentId,ProjectId,ReleaseId,TaskId,DeployedBy,TenantId,DeployedToMachineIds,ChannelId,SpaceId,JSON FROM [dbo].[Deployment] WHERE [Id] = @Id System.Exception at Nevermore.CommandExecutor.ExecuteReaderAsync(CancellationToken cancellationToken) at Nevermore.Advanced.ReadTransaction.ExecuteReaderAsync(PreparedCommand preparedCommand, CancellationToken cancellationToken) at Nevermore.Advanced.ReadTransaction.StreamAsync[TRecord](PreparedCommand command, CancellationToken cancellationToken)+MoveNext() at Nevermore.Advanced.ReadTransaction.LoadAsync[TDocument,TKey](TKey id, CancellationToken cancellationToken) at Nevermore.Advanced.ReadTransaction.LoadAsync[TDocument,TKey](TKey id, CancellationToken cancellationToken) at Octopus.Core.RelationalStorage.RawRelationalTransaction.LoadAsync[TDocument,TKey](TKey id, CancellationToken cancellationToken) in RawRelationalTransaction.cs:line 39 at Octopus.Server.Web.Infrastructure.OctopusQueryExecutor.LoadAsync[TDocument,TKey](TKey id, CancellationToken cancellationToken) in OctopusQueryExecutor.cs:line 274 at Octopus.Core.Persistence.Database.ProjectPathDecorators.ProjectPathQueryExecutorDecorator.LoadAsync[TDocument,TKey](TKey id, CancellationToken cancellationToken) in ProjectPathQueryExecutorDecorator.cs:line 46 at Octopus.Core.Persistence.Database.DatabaseDocumentStore`2.GetAsync(TKey id, CancellationToken cancellationToken) in DatabaseDocumentStore.cs:line 64 at Octopus.Core.Persistence.Database.FullTableCacheDocumentStoreDecorator`2.GetAsync(TKey id, CancellationToken cancellationToken) in FullTableCacheDocumentStoreDecorator.cs:line 65 at Octopus.Core.Persistence.Database.ProjectPathDecorators.ProjectPathDocumentStoreDecorator`2.GetAsync(TKey id, CancellationToken cancellationToken) in ProjectPathDocumentStoreDecorator.cs:line 36 at Octopus.Core.Persistence.EntityTracking.EntityTracker.GetOrTrackAsync[TDocument,TKey](TKey id, Func`3 getDocument, CancellationToken cancellationToken) in EntityTracker.cs:line 70 at Octopus.Core.Persistence.EntityTracking.EntityTrackingDocumentStoreDecorator`2.GetAsync(TKey id, CancellationToken cancellationToken) in EntityTrackingDocumentStoreDecorator.cs:line 50 at Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2.GetAsync(TKey id, CancellationToken cancellationToken) in AuditingDocumentStoreDecorator.cs:line 67 at Octopus.Core.Persistence.Database.Deletion.DeleteRelatedDocumentsDocumentStoreDecorator`2.GetAsync(TKey id, CancellationToken cancellationToken) in DeleteRelatedDocumentsDocumentStoreDecorator.cs:line 38 at Octopus.Core.Persistence.Database.Deletion.VetoDocumentStoreDecorator`2.GetAsync(TKey id, CancellationToken cancellationToken) in VetoDocumentStoreDecorator.cs:line 37 at Octopus.Core.Persistence.Database.AccessCheckingDocumentStoreDecorator`2.GetAsync(TKey id, CancellationToken cancellationToken) in AccessCheckingDocumentStoreDecorator.cs:line 42 at Octopus.Core.Persistence.DocumentStore`2.GetAsync(TKey id, CancellationToken cancellationToken) in DocumentStore.cs:line 78 at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentPlanService.LoadStateContainer(DeploymentPlan plan, CancellationToken cancellationToken) in DeploymentPlanService.cs:line 108 at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionPlanService`3.Persist(TExecutionPlan plan, CancellationToken cancellationToken) in ExecutionPlanService.cs:line 62 at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.<>c__DisplayClass6_0.<PersistPlan in DeploymentTaskController.cs:line 74 at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass3_0`1.<Execute in UnitOfWorkExecutor.cs:line 87 at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.DoAsync(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 75 at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.DoAsync(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 75 at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T](Func`3 action, CancellationToken cancellationToken, String name) in UnitOfWorkExecutor.cs:line 90 at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.PersistPlan(DeploymentPlan plan, CancellationToken cancellationToken) in DeploymentTaskController.cs:line 76 at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionTaskController`3.ExecuteBase(ITaskLog taskLogRoot, CancellationToken cancellationToken) in ExecutionTaskController.cs:line 99 at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.Execute(ITaskLog taskLog, CancellationToken cancellationToken) in DeploymentTaskController.cs:line 56 at Octopus.Server.Orchestration.ServerTasks.RunningTask.<>c__DisplayClass31_1.<WorkerTask in RunningTask.cs:line 171 at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.DoAsync(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 75 at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.DoAsync(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 75 at Octopus.Server.Orchestration.ServerTasks.RunningTask.WorkerTask(CancellationToken cancellationToken) in RunningTask.cs:line 204 --Inner Exception-- SQL Error 995 - A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The I/O operation has been aborted because of either a thread exit or an application request.) Microsoft.Data.SqlClient.SqlException at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__188_0(Task`1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) at Nevermore.Transient.DbCommandExtensions.<>c__DisplayClass3_0.<ExecuteReaderWithRetryAsync at Nevermore.Transient.DbCommandExtensions.<>c__DisplayClass3_0.<ExecuteReaderWithRetryAsync at Nevermore.Transient.RetryPolicy.ExecuteActionAsync[TResult](Func`1 func) at Nevermore.Transient.DbCommandExtensions.ExecuteReaderWithRetryAsync(DbCommand command, RetryPolicy commandRetryPolicy, CommandBehavior commandBehavior, CancellationToken cancellationToken, RetryPolicy connectionRetryPolicy, String operationName) at Nevermore.CommandExecutor.ExecuteReaderAsync(CancellationToken cancellationToken) --Inner Exception-- The I/O operation has been aborted because of either a thread exit or an application request. System.ComponentModel.Win32Exception
3

Hi @OctopusDeployUser,

Thank you for contacting Octopus Support.

It looks like you are experiencing a bug related to a SQL 995 error. We have a GitHub issue created for this here.

We don’t have a patch for this yet. However, one possible workaround that has alleviated this for many customers is adding Pooling=false to your Octopus db connection string. Would it be possible for you to give that a try to see if the issue goes away?

Let me know at your earliest convenience.

Best Regards,
Donny

4

That issue has been open for close to a year - are you saying we have to continue to wait for a fix when our deployments are failing daily? Is there an ETA on when this will be resolved? Our QA team is doing manual deployments everyday because of this bug.

I will try adding the pool parameter and report back.

5

Hi @OctopusDeployUser,

Thank you for getting back to me. I apologize for the frustration this issue has caused you. I can assure you this issue is being treated with the highest priority.

We did get an update from our engineering team on the GitHub issue a few hours ago. Here’s what they said:

Thanks for your continued patience while we investigate this issue, and apologies for the delay in messaging here.

We have still not managed to reproduce this issue in a test environment. Our suspicion is that there is some environmental factor we have not yet accounted for that is key. Given this, we have been working closely with one of our affected customers to diagnose the issue. So far we have determined the following:

  • Octopus is not doing anything clearly wrong at or around the time the error occurs (e.g. closing connections, triggering cancellation tokens, etc.)
  • Only database commands performed in an async context are affected
  • Usage patterns within Octopus Deploy don’t seem to be a factor, as this error can occur immediately after start-up
  • In a PerfView trace we can see that the thread executing the SQL command is terminating unexpectedly, as the error message suggests, though there is no discernable cause for this

We currently have a support case open with Microsoft so they can advise us on what the cause of the thread termination might be. This is our primary focus going forward.

We’re also interested in eliminating .NET 5 as a factor. Please let us know if you are still experiencing this issue on any of the below versions or later, which run .NET 6:

  • 2022.1.2925
  • 2022.2.6978
  • 2022.3.3529

Let us know once you get a chance to try Pooling=False in the connection string if that alleviates the issue for you.

Best Regards,
Donny

6

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.