Octopus Deploy - A transport-level error

Hi, We are currently running Octopus Deploy 3.11.15.
After a mainteance weekend we started intermittently seeing the following error in the OD logs.

2018-07-24 02:20:45.2182 333 ERROR Unhandled error on request: http://octopusdeploy.intranet.net/api/releases?ignoreChannelRules=False 0014637a58884eedae33e7299672072b by TeamCity : A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
System.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error) at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync() at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() at System.Data.SqlClient.TdsParserStateObject.TryReadUInt32(UInt32& value) at System.Data.SqlClient.TdsParserStateObject.TryReadPlpLength(Boolean returnPlpNullIfNull, UInt64& lengthLeft) at System.Data.SqlClient.TdsParser.TryReadPlpUnicodeChars(Char[]& buff, Int32 offst, Int32 len, TdsParserStateObject stateObj, Int32& totalCharsRead) at System.Data.SqlClient.TdsParser.TryReadSqlStringValue(SqlBuffer value, Byte type, Int32 length, Encoding encoding, Boolean isPlp, TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParser.TryReadSqlValue(SqlBuffer value, SqlMetaDataPriv md, Int32 length, TdsParserStateObject stateObj, SqlCommandColumnEncryptionSetting columnEncryptionOverride, String columnName) at System.Data.SqlClient.SqlDataReader.TryReadColumnInternal(Int32 i, Boolean readHeaderOnly) at System.Data.SqlClient.SqlDataReader.TryReadColumn(Int32 i, Boolean setTimeout, Boolean allowPartiallyReadColumn) at System.Data.SqlClient.SqlDataReader.GetValueInternal(Int32 i) at System.Data.SqlClient.SqlDataReader.GetValue(Int32 i) at Nevermore.RelationalTransaction.<Stream>d__381.MoveNext()
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable1 source) at Nevermore.QueryBuilder1.First()
at Nevermore.RelationalTransaction.LoadRequired[T](String id)
at Octopus.Server.Web.Api.Actions.VariableScopesProvider.GetProjectScopeValues(IRelationalTransaction transaction, String projectId) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Server\Web\Api\Actions\VariableScopesProvider.cs:line 45
at Octopus.Server.Web.Api.Actions.VariableScopesProvider.GetAvailableScopeValues(IRelationalTransaction transaction, VariableSet setModel) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Server\Web\Api\Actions\VariableScopesProvider.cs:line 20
at Octopus.Server.Web.Api.Rules.ReleaseSnapshotter.EmbedScopeValuesIfRequired(IRelationalTransaction transaction, VariableSet snapshot) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Server\Web\Api\Rules\ReleaseSnapshotter.cs:line 75
at Octopus.Server.Web.Api.Rules.ReleaseSnapshotter.SnapshotVariableSets(IRelationalTransaction transaction, Release release, Project project) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Server\Web\Api\Rules\ReleaseSnapshotter.cs:line 20
at Octopus.Server.Web.Api.Rules.ProjectSnapshotWillBeCreatedRule.AfterMapBeforeStore(Release release, ReleaseResource resource, ISpecialRuleContext context) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Server\Web\Api\Rules\ProjectSnapshotWillBeCreatedRule.cs:line 29
at Octopus.Server.Web.Infrastructure.Api.Responder1.ExecuteRules[TRule](Action2 ruleCallback) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Server\Web\Infrastructure\Api\Responder.cs:line 171
at Octopus.Server.Web.Infrastructure.Api.CreateResponseDescriptor2.Responder.Execute() in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Server\Web\Infrastructure\Api\CreateResponseDescriptor.cs:line 56 at Octopus.Server.Web.Infrastructure.Api.Responder1.Respond(TDescriptor options, NancyContext context) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Server\Web\Infrastructure\Api\Responder.cs:line 145
at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
at Octopus.Server.Web.Api.OctopusRestApiModule.<>c__DisplayClass0_0.<.ctor>b__0(Object o) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Server\Web\Api\OctopusRestApiModule.cs:line 46
at Nancy.Routing.Route.<>c__DisplayClass4.b__3(Object parameters, CancellationToken context)
ClientConnectionId:45e93098-2e4d-43b5-aaee-30c4759b75cb
Error Number:-1,State:0,Class:20
2018-07-24 02:21:04.6405 10 WARN

We have rebooted the servers a couple of time and uninstalled the MS Updates on the server incase one of the patches caused issues , however we are still intermittently seeing the above error messages and issues with OD. We have checked the database layer and there are no errors on that side of things . At the moment we are at a loss as to what may be causing this. If anyone has seen this befoe or has any suggestions or ideas that would be much appreciated. (note: we only have 1 OD server and 1 database server so it is not a loadlanacing issue or the case of a dodgy server in a cluster)

Thank you
Daisy

After removing the MS patches we did experience a few more issues. However it must have been just a could of adhoc errors . The issue above was cause by one of these MS patches

• KB4338815 - https://support.microsoft.com/en-gb/help/4338815/windows-81-update-kb4338815 - Security Update
• KB4338424 - https://support.microsoft.com/en-gb/help/4338424/description-of-the-security-and-quality-rollup-for-net-framework-3-5-s
• KB4338419 - https://support.microsoft.com/en-gb/help/4338419/description-of-the-security-and-quality-rollup-updates-for-net-framewo
• KB4345424 - https://support.microsoft.com/en-gb/help/4338824/windows-81-update-kb4338824
• KB4284815 - https://support.microsoft.com/en-gb/help/4284815

Since they have been uninstalled we have only seen 2 occurrences of the above error instead of 2000.

Hi @daisy.tilston-gould,

Thanks for getting in touch, and thanks for the information regarding the patches. We have had a few cases of this previously and haven’t reliably been able to track them down.

Out of interest, would you mind letting me know what infrastructure your VM is hosted on (is it internal, or AWS/Azure?).

Please let us know if there is anything else that I can assist with,

Regards,
Alex

Out of interest, as I suspect that the update that caused the issue was KB4338815, did you also install the patches from the known issues section?

Symptom Workaround
After installing this update, some devices running network monitoring workloads may receive the 0xD1 Stop error because of a race condition.
This issue is resolved in KB4338831.

Restarting the SQL Server service may fail occasionally with the error, “Tcp port is already in use”.
This issue is resolved in KB4338831.

When an administrator tries to stop the World Wide Web Publishing Service (W3SVC), the W3SVC remains in a “stopping” state, but cannot fully stop or be restarted.
This issue is resolved in KB4338831.

If you can let me know that would be greatly appreciated, it may assist if this issue appears in future.

Regards.
Alex

Hi @Alex.Rolley,
No we had not installed KB4338831.
But it could be worth trying it, as we will need to apply the MS patches at some stage.

The VM is hosted internally on a Win2012R2 VM.
We have noticed that we get the above error in the logs a couple of times a day . We havent been able to prove it , but we suspect it may be linked to the deadlocks that happen in OD. But this should be resolved when we upgrade as later versions fix the deadlock issue.

Regards
Daisy

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.