Octo.exe An existing connection was forcibly closed

Integration
, , ,
1

We are seeing intermittent failures in the last couple of weeks while deploying releases using octo.exe from our build server (Jenkins). The build server and octopus server are both in Azure and on the same subnet, so I do not suspect it is actually network related (No traffic is crossing our firewall). When the error occurs, Jenkins fails, however Octopus continues the deploy and actually ends up with a successful deploy… which in theory is good, but none of our CI tests end up running.

The error in the Jenkins console is:

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.CopyToAsyncCore(Stream destination, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Octopus.Client.OctopusAsyncClient.DispatchRequest[TResponseResource](OctopusRequest request, Boolean readResponse)
   at Octopus.Client.OctopusAsyncClient.Get[TResource](String path, Object pathParameters)
   at Octopus.Client.Repositories.Async.TaskRepository.GetDetails(TaskResource resource, Nullable`1 includeVerboseOutput, Nullable`1 tail)
   at Octopus.Cli.Commands.Deployment.TaskOutputProgressPrinter.Render(IOctopusAsyncRepository repository, ICommandOutputProvider commandOutputProvider, TaskResource resource)
   at Octopus.Client.Repositories.Async.TaskRepository.WaitForCompletion(TaskResource[] tasks, Int32 pollIntervalSeconds, Nullable`1 timeoutAfter, Func`2 interval)
   at Octopus.Cli.Commands.Deployment.DeploymentCommandBase.WaitForDeploymentToComplete(IReadOnlyList`1 deployments, ProjectResource project, ReleaseResource release)
   at Octopus.Cli.Commands.Deployment.DeploymentCommandBase.DeployRelease(ProjectResource project, ReleaseResource release)
   at Octopus.Cli.Commands.Deployment.DeployReleaseCommand.Request()
   at Octopus.Cli.Commands.ApiCommand.Execute()
   at Octopus.Cli.Commands.ApiCommand.Execute(String[] commandLineArguments)
   at Octopus.Cli.CliProgram.Run(String[] args)
Exit code: -3
Exec: Error executing command F:\octo\Octo.exe deploy-release --project=XXXX_DB --Deployto=QA --Tenant=$env:Tenant 
--version=$BuildVer --server=$OctoURL --apiKey=$OctoAPIKey --progress --deploymenttimeout=05:00:00 --timeout=1200.

We are running:
Octopus 2020.02.13
Octo.exe 7.3.7 (this was also happening on 6.15.1 so I tried upgrading as a fix)

3

Hi Chris,

Thanks for reaching out. Would you be able to DM me a full log from a build(one failure, one successful), as well as an Octopus Server log from the same timeframe? The default location for these is C:\Octopus\Logs.

Are there any local firewalls or antivirus on the VMs themselves that may be possibly closing the connection?

Have you ensured the settings in your Virtual network security groups are okay?

Please let me know if you have any questions.

Thanks,
Jeremy

4

Hi Jeremy,
Thanks for the response. I am gathering logs and will send them in a minute.

The Windows firewall on the Octopus server is configured to log all DENY events and I am not seeing anything denied from our build server IP at all.

Both servers are running eset for A/V, I am not seeing anything in either log

NSGs are disabled on both these hosts

8

Hi Jeremy,
We have had the issue happen a few more times recently and I am not seeing any metrics appear out of place. Is there anything else that we can look into?

9

Hey Chris,

Let me dig in a bit more and see if I can think of anything else. Please feel free to reach out in the mean time with any questions or concerns.

Thanks,
Jeremy

11

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.