Service crash. Physical connection is not usable

Legacy Execution
1

Frequently the Octopus server service crashes with this error in the event log. What can I do to fix this? I am running on windows server 2012 on a EXSI host.

Version 4.0.10

2018-06-04 01:11:18.1305 5848 11 FATAL Unhandled AppDomain exception occurred: SQL Error -1 - A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
System.Data.SqlClient.SqlException
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error) at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync() at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest) at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest) at System.Data.SqlClient.SqlInternalConnection.BeginSqlTransaction(IsolationLevel iso, String transactionName, Boolean shouldReconnect) at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName) at System.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel) at Nevermore.RelationalTransaction..ctor(RelationalTransactionRegistry registry, RetriableOperation retriableOperation, IsolationLevel isolationLevel, ISqlCommandFactory sqlCommandFactory, JsonSerializerSettings jsonSerializerSettings, RelationalMappings mappings, IKeyAllocator keyAllocator, IRelatedDocumentStore relatedDocumentStore, String name) at Nevermore.RelationalStore.BeginTransaction(RetriableOperation retriableOperation, String name) at Octopus.Core.Model.Clustering.OctopusClusterService.GetCurrentNode() at Octopus.Core.Model.Clustering.OctopusClusterService.Heartbeat() at Octopus.Server.Schedules.Scheduler.HeartbeatAndUpdateRankIfRequired() at Octopus.Server.Schedules.Scheduler.RunAndRescheduleTasksAsRequired() at Octopus.Server.Schedules.Scheduler.RunScheduleThread() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart() System.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable) at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
at System.Data.SqlClient.SqlInternalConnection.BeginSqlTransaction(IsolationLevel iso, String transactionName, Boolean shouldReconnect)
at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)
at System.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)
at Nevermore.RelationalTransaction…ctor(RelationalTransactionRegistry registry, RetriableOperation retriableOperation, IsolationLevel isolationLevel, ISqlCommandFactory sqlCommandFactory, JsonSerializerSettings jsonSerializerSettings, RelationalMappings mappings, IKeyAllocator keyAllocator, IRelatedDocumentStore relatedDocumentStore, String name)
at Nevermore.RelationalStore.BeginTransaction(RetriableOperation retriableOperation, String name)
at Octopus.Core.Model.Clustering.OctopusClusterService.GetCurrentNode()
at Octopus.Core.Model.Clustering.OctopusClusterService.Heartbeat()
at Octopus.Server.Schedules.Scheduler.HeartbeatAndUpdateRankIfRequired()
at Octopus.Server.Schedules.Scheduler.RunAndRescheduleTasksAsRequired()
at Octopus.Server.Schedules.Scheduler.RunScheduleThread()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
ClientConnectionId:5988be81-c5d8-4809-9316-97d426a78ffa
Error Number:-1,State:0,Class:20

5

Hi,

Thanks for reaching out. Sorry yo hear you are having this issue with Octopus, would you be able to send through all the Octopus Server logs so we can investigate this issue further? usually the logs are located in C:\Octopus\Logs, the best thing to do would be to zip up that folder and send it through, so we can look through the logs to see what might be going wrong.\

Apologies for any inconvenience this has cause, hopefully we can get to the bottom of it quickly.

Kind Regads,
Tom W

6

OctopusTentacle.0.txt (32.1 KB)
OctopusTentacle.2.txt (22.1 KB)
OctopusTentacle.1.txt (32.1 KB)
OctopusTentacle.6.txt (16.1 KB)
OctopusTentacle.4.txt (32.1 KB)
OctopusTentacle.5.txt (34.6 KB)
OctopusTentacle.txt (16.1 KB)
OctopusTentacle.3.txt (51.3 KB)

7

I’d like to add that we use Octopus at the company I work for and we see the same issue there so I can get logs from that system as well if this doesn’t tell you anything.

8

Whoops! I accidentally sent tentacle logs. These are from the server. OctopusServer.0.txt (39.3 KB)
OctopusServer.1.txt (40.0 KB)
OctopusServer.txt (19.7 KB)
OctopusServer.5.txt (27.9 KB)
OctopusServer.2.txt (43.5 KB)
OctopusServer.4.txt (39.5 KB)
OctopusServer.3.txt (42.0 KB)
OctopusServer.6.txt (45.7 KB)

9

Hi,
Thanks for sending those logs through, Is the SQL Server located on a separate machine/VM to the Octopus Server?
From the logs it looks as those the Server loses connection to both the tentacle and SQL server within seconds of each other.

It might also be helpful if you are able to send through the logs from the company you work for. It might be worth sending both Server and Tentacle logs so we can get a better picture of what is going on.

Kind Regards,
Tom W

10

The SQL server is on a different VM on the same host. It’s very possible that it’s losing connection to the SQL server but surely that doesn’t cause the service to crash and never recover? I’ll see about gathering the other logs.

11

Hi,
You are absolutely correct, Octopus should be handling the connection loss properly as to not crash the service. After looking into this issue further, It looks like it was fixed in a Octopus v4.1.2.

If you upgrade to any version >= 4.1.2 you should no longer see this issue.

I hope that helps, please let me know if you have any further questions.

Kinds Regards,
Tom W

12

I’ve upgraded both instances. If you don’t hear back from me you can assume it’s fixed.

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.