Acquire Packages recently started failing

I have multiple issues as of lately with deploying nuget packages. I see no network issues and firewall is turned off on both machines. I have to retry the deploy multiple times before it actually gets past the “Acquiring Packages” stage. The retry “sometimes” works, but more often than not it fails. How can this be fixed long term?

Acquire packages
October 19th 2016 11:41:54Info
Acquiring packages
October 19th 2016 11:41:54Info
Making a list of packages to download
October 19th 2016 11:41:55Info
Delta compression is disabled for package transfers from the Octopus Server to deployment targets
October 19th 2016 11:41:55Info
Looking up the package location from the built-in package repository…
October 19th 2016 11:44:35Fatal
The step failed: Activity failed with error ‘An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.’.
QA
October 19th 2016 11:44:35Fatal
An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
Uploading: QA.3.10.7.46.nupkg (26%) Upload package QA version 3.10.7.46
October 19th 2016 11:41:55Info
Uploading package QA
October 19th 2016 11:41:55Info
Beginning streaming transfer of QA.3.10.7.46.nupkg-da6fc38b-8ceb-4d39-8083-9f025e612ef9
October 19th 2016 11:42:15Info
An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
Halibut.HalibutClientException: An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult)
— End of inner exception stack trace —
at System.Net.Security._SslStream.EndWrite(IAsyncResult asyncResult)
at Halibut.DataStream.StreamingDataStream.CopyAndReportProgress(Stream destination) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\DataStream.cs:line 138
at Halibut.Transport.Protocol.MessageExchangeStream.WriteEachStream(IEnumerable1 streams) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 274 at Halibut.Transport.Protocol.MessageExchangeStream.Send[T](T message) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 143 at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsClient(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 31 at Halibut.HalibutRuntime.<>c__DisplayClass25_0.<SendOutgoingHttpsRequest>b__0(MessageExchangeProtocol protocol) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 115 at Halibut.Transport.SecureClient.ExecuteTransaction(Action1 protocolHandler) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 62
— End of inner exception stack trace —

Server stack trace:
at Halibut.Transport.SecureClient.HandleError(Exception lastError, Boolean retryAllowed) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 181
at Halibut.HalibutRuntime.SendOutgoingHttpsRequest(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 117
at Halibut.ServiceModel.HalibutProxy.Invoke(IMessage msg) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\ServiceModel\HalibutProxy.cs:line 33

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Octopus.Shared.Contracts.IFileTransferService.UploadFile(String remotePath, DataStream upload)
at Octopus.Worker.Tentacles.TentacleRemoteEndpointFacade.UploadFile(String fileName, DataStream package) in Y:\Work\refs\tags\3.4.13\source\Octopus.Worker\Tentacles\TentacleRemoteEndpointFacade.cs:line 78
Octopus.Server version 3.4.13 (3.4.13+Branch.master.Sha.875581637fa0b39fc45c2ac494f01db8453a2b8c)
October 19th 2016 11:42:17Info
File upload failed. Retry attempt 1 of 5…
October 19th 2016 11:42:17Info
Beginning streaming transfer of QA.3.10.7.46.nupkg-da6fc38b-8ceb-4d39-8083-9f025e612ef9
October 19th 2016 11:42:38Info
An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
Halibut.HalibutClientException: An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult)
— End of inner exception stack trace —
at System.Net.Security._SslStream.EndWrite(IAsyncResult asyncResult)
at Halibut.DataStream.StreamingDataStream.CopyAndReportProgress(Stream destination) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\DataStream.cs:line 138
at Halibut.Transport.Protocol.MessageExchangeStream.WriteEachStream(IEnumerable1 streams) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 274 at Halibut.Transport.Protocol.MessageExchangeStream.Send[T](T message) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 143 at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsClient(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 31 at Halibut.HalibutRuntime.<>c__DisplayClass25_0.<SendOutgoingHttpsRequest>b__0(MessageExchangeProtocol protocol) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 115 at Halibut.Transport.SecureClient.ExecuteTransaction(Action1 protocolHandler) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 62
— End of inner exception stack trace —

Server stack trace:
at Halibut.Transport.SecureClient.HandleError(Exception lastError, Boolean retryAllowed) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 181
at Halibut.HalibutRuntime.SendOutgoingHttpsRequest(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 117
at Halibut.ServiceModel.HalibutProxy.Invoke(IMessage msg) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\ServiceModel\HalibutProxy.cs:line 33

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Octopus.Shared.Contracts.IFileTransferService.UploadFile(String remotePath, DataStream upload)
at Octopus.Worker.Tentacles.TentacleRemoteEndpointFacade.UploadFile(String fileName, DataStream package) in Y:\Work\refs\tags\3.4.13\source\Octopus.Worker\Tentacles\TentacleRemoteEndpointFacade.cs:line 78
Octopus.Server version 3.4.13 (3.4.13+Branch.master.Sha.875581637fa0b39fc45c2ac494f01db8453a2b8c)
October 19th 2016 11:42:40Info
File upload failed. Retry attempt 2 of 5…
October 19th 2016 11:42:40Info
Beginning streaming transfer of QA.3.10.7.46.nupkg-da6fc38b-8ceb-4d39-8083-9f025e612ef9
October 19th 2016 11:43:02Info
An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
Halibut.HalibutClientException: An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult)
— End of inner exception stack trace —
at System.Net.Security._SslStream.EndWrite(IAsyncResult asyncResult)
at Halibut.DataStream.StreamingDataStream.CopyAndReportProgress(Stream destination) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\DataStream.cs:line 138
at Halibut.Transport.Protocol.MessageExchangeStream.WriteEachStream(IEnumerable1 streams) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 274 at Halibut.Transport.Protocol.MessageExchangeStream.Send[T](T message) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 143 at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsClient(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 31 at Halibut.HalibutRuntime.<>c__DisplayClass25_0.<SendOutgoingHttpsRequest>b__0(MessageExchangeProtocol protocol) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 115 at Halibut.Transport.SecureClient.ExecuteTransaction(Action1 protocolHandler) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 62
— End of inner exception stack trace —

Server stack trace:
at Halibut.Transport.SecureClient.HandleError(Exception lastError, Boolean retryAllowed) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 181
at Halibut.HalibutRuntime.SendOutgoingHttpsRequest(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 117
at Halibut.ServiceModel.HalibutProxy.Invoke(IMessage msg) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\ServiceModel\HalibutProxy.cs:line 33

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Octopus.Shared.Contracts.IFileTransferService.UploadFile(String remotePath, DataStream upload)
at Octopus.Worker.Tentacles.TentacleRemoteEndpointFacade.UploadFile(String fileName, DataStream package) in Y:\Work\refs\tags\3.4.13\source\Octopus.Worker\Tentacles\TentacleRemoteEndpointFacade.cs:line 78
Octopus.Server version 3.4.13 (3.4.13+Branch.master.Sha.875581637fa0b39fc45c2ac494f01db8453a2b8c)
October 19th 2016 11:43:04Info
File upload failed. Retry attempt 3 of 5…
October 19th 2016 11:43:04Info
Beginning streaming transfer of QA.3.10.7.46.nupkg-da6fc38b-8ceb-4d39-8083-9f025e612ef9
October 19th 2016 11:43:24Info
An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
Halibut.HalibutClientException: An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult)
— End of inner exception stack trace —
at System.Net.Security._SslStream.EndWrite(IAsyncResult asyncResult)
at Halibut.DataStream.StreamingDataStream.CopyAndReportProgress(Stream destination) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\DataStream.cs:line 138
at Halibut.Transport.Protocol.MessageExchangeStream.WriteEachStream(IEnumerable1 streams) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 274 at Halibut.Transport.Protocol.MessageExchangeStream.Send[T](T message) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 143 at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsClient(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 31 at Halibut.HalibutRuntime.<>c__DisplayClass25_0.<SendOutgoingHttpsRequest>b__0(MessageExchangeProtocol protocol) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 115 at Halibut.Transport.SecureClient.ExecuteTransaction(Action1 protocolHandler) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 62
— End of inner exception stack trace —

Server stack trace:
at Halibut.Transport.SecureClient.HandleError(Exception lastError, Boolean retryAllowed) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 181
at Halibut.HalibutRuntime.SendOutgoingHttpsRequest(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 117
at Halibut.ServiceModel.HalibutProxy.Invoke(IMessage msg) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\ServiceModel\HalibutProxy.cs:line 33

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Octopus.Shared.Contracts.IFileTransferService.UploadFile(String remotePath, DataStream upload)
at Octopus.Worker.Tentacles.TentacleRemoteEndpointFacade.UploadFile(String fileName, DataStream package) in Y:\Work\refs\tags\3.4.13\source\Octopus.Worker\Tentacles\TentacleRemoteEndpointFacade.cs:line 78
Octopus.Server version 3.4.13 (3.4.13+Branch.master.Sha.875581637fa0b39fc45c2ac494f01db8453a2b8c)
October 19th 2016 11:43:26Info
File upload failed. Retry attempt 4 of 5…
October 19th 2016 11:43:26Info
Beginning streaming transfer of QA.3.10.7.46.nupkg-da6fc38b-8ceb-4d39-8083-9f025e612ef9
October 19th 2016 11:43:47Info
An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
Halibut.HalibutClientException: An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult)
— End of inner exception stack trace —
at System.Net.Security._SslStream.EndWrite(IAsyncResult asyncResult)
at Halibut.DataStream.StreamingDataStream.CopyAndReportProgress(Stream destination) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\DataStream.cs:line 138
at Halibut.Transport.Protocol.MessageExchangeStream.WriteEachStream(IEnumerable1 streams) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 274 at Halibut.Transport.Protocol.MessageExchangeStream.Send[T](T message) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 143 at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsClient(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 31 at Halibut.HalibutRuntime.<>c__DisplayClass25_0.<SendOutgoingHttpsRequest>b__0(MessageExchangeProtocol protocol) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 115 at Halibut.Transport.SecureClient.ExecuteTransaction(Action1 protocolHandler) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 62
— End of inner exception stack trace —

Server stack trace:
at Halibut.Transport.SecureClient.HandleError(Exception lastError, Boolean retryAllowed) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 181
at Halibut.HalibutRuntime.SendOutgoingHttpsRequest(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 117
at Halibut.ServiceModel.HalibutProxy.Invoke(IMessage msg) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\ServiceModel\HalibutProxy.cs:line 33

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Octopus.Shared.Contracts.IFileTransferService.UploadFile(String remotePath, DataStream upload)
at Octopus.Worker.Tentacles.TentacleRemoteEndpointFacade.UploadFile(String fileName, DataStream package) in Y:\Work\refs\tags\3.4.13\source\Octopus.Worker\Tentacles\TentacleRemoteEndpointFacade.cs:line 78
Octopus.Server version 3.4.13 (3.4.13+Branch.master.Sha.875581637fa0b39fc45c2ac494f01db8453a2b8c)
October 19th 2016 11:43:49Info
File upload failed. Retry attempt 5 of 5…
October 19th 2016 11:43:49Info
Beginning streaming transfer of QA.3.10.7.46.nupkg-da6fc38b-8ceb-4d39-8083-9f025e612ef9
October 19th 2016 11:44:12Info
An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
Halibut.HalibutClientException: An error occurred when sending a request to ‘https://qa-web:10933/’, after the request began: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult)
— End of inner exception stack trace —
at System.Net.Security._SslStream.EndWrite(IAsyncResult asyncResult)
at Halibut.DataStream.StreamingDataStream.CopyAndReportProgress(Stream destination) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\DataStream.cs:line 138
at Halibut.Transport.Protocol.MessageExchangeStream.WriteEachStream(IEnumerable1 streams) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 274 at Halibut.Transport.Protocol.MessageExchangeStream.Send[T](T message) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 143 at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsClient(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 31 at Halibut.HalibutRuntime.<>c__DisplayClass25_0.<SendOutgoingHttpsRequest>b__0(MessageExchangeProtocol protocol) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 115 at Halibut.Transport.SecureClient.ExecuteTransaction(Action1 protocolHandler) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 62
— End of inner exception stack trace —

Server stack trace:
at Halibut.Transport.SecureClient.HandleError(Exception lastError, Boolean retryAllowed) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\SecureClient.cs:line 181
at Halibut.HalibutRuntime.SendOutgoingHttpsRequest(RequestMessage request) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\HalibutRuntime.cs:line 117
at Halibut.ServiceModel.HalibutProxy.Invoke(IMessage msg) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\ServiceModel\HalibutProxy.cs:line 33

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Octopus.Shared.Contracts.IFileTransferService.UploadFile(String remotePath, DataStream upload)
at Octopus.Worker.Tentacles.TentacleRemoteEndpointFacade.UploadFile(String fileName, DataStream package) in Y:\Work\refs\tags\3.4.13\source\Octopus.Worker\Tentacles\TentacleRemoteEndpointFacade.cs:line 78
Octopus.Server version 3.4.13 (3.4.13+Branch.master.Sha.875581637fa0b39fc45c2ac494f01db8453a2b8c)
October 19th 2016 11:44:14Info
File upload failed. Retry attempt 6 of 5…
October 19th 2016 11:44:14

qa-web is pingable and I get no dropped connections while this is going on.

Hi Michael,

Thanks for getting in touch. That is frustrating.

Would you mind sending Tentacle logs for the Tentacle that is failing during the same time period as the deployment log? ie for https://qa-web:10933 on Octopus 19th.

The logs can be found in C:\Octopus\Logs on your Tentacle machine by default.

Cheers,
Shane

Sure! Here is the log file. It looks like it is trying to bind the port to TCP/IPV6. I have disabled that on both the server and the tentacle side.

OctopusTentacle.0.txt (127 KB)

Hey Shane,

I have even tried reinstalling the tentacle and redeploying and it still won’t work. If I reboot the system completely, then I am able to do a deploy. Nothing has changed on our network as we are hosted in Rackspace cloud. No network/system changes at all. I am the admin over these systems. Just wanted you to understand the background.

Yes, the tentacle is failing during the same period as the console says it is failing. 5-10 seconds roughly. After 3 unsuccessful deploys, I hit the reinstall on the tentacle manager and restarted the OD service and after 2-3 retries, it finally succeeds. I have no idea why I have to do these steps to get it to work.

This reminds me of the old song by Seven Mary Three, “Cumbersome”

OctopusTentacle.txt (75 KB)

It seems like all the ports are getting use. Getting this error now:

The step failed: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)

Hi Michael,

Are you getting that new error on the Server or Tentacle?

This might be a long shot: do you have any anti virus software on your Server or Tentacles that might be monitoring ports?

It may be worth doing a wireshark trace to capture the traffic on your Tentacle port.

Do you have any other Tentacles? Is there just one having this issue?

Cheers,
Shane

Hey Shane,

I see these errors on the server side as it is going through the process. I
do have other tentacles, but we do not deploy to those unfortunately. This
is a Rackspace hosted system, so I assume there is anti-virus loaded up on
those machines and will check there first to see if I can disable it and
then push a deploy. This seems to be on the tentacle side as I can do a
"reinstall" of the tentacle and then bounce the OD service to clear out any
old connections or open ports. Once I do that, the deploy will finally go
through.

Can I get an update to this please? Still seeing this issue.

Hi Michael,

Sorry for not getting back to you, for some reason I thought I was waiting on more information and removed this ticket from my queue.

Is your Tentacle in polling or listening mode? Would you mind trying the opposite mode (if it’s in listening, set it up as polling) to see if this has any effect?

We could also set up trace logging on your Octopus Deploy Server and Tentacle which would start logging all of the communications between them. You can do so by following the instructions here: http://docs.octopusdeploy.com/display/OD/Log+files

For the OD Server you would need to edit C:\Program Files\Octopus Deploy\Octopus\octopus.server.exe.nlog and for Tentacle C:\Program Files\Octopus Deploy\Tentacle\tentacle.exe.nlog. Capture the logs until you start running in to the Tentacle connection issue and then switch them back to Info, the log files will get quite large.

Cheers,
Shane