Package push from Octo.exe fails but package upload via web succeeds

Octopus Server
, , , ,
1

Hi,

I’m trying to use the TeamCity plugin to upload packages created by OctoPack to my Octopus server (same domain, different hosts). The upload fails so I use Octo.exe from the command line to investigate the fault without having to rebuild each time. Octo.exe successfully contacts the server and queries the existence of the package but the POST fails with the network connection being forcibly closed with error ‘System.Net.Http.HttpRequestException: Error while copying content to a stream.’

PS>.\octo.exe push --server http://gbrcam-fpsdev1:8000/ --apikey ****************** --package E:\TeamCity\buildAgent\work\50fc4d458a9475a6\Sepura.FPS\Sepura.FPS.Web\bin\Sepura.FPS.Web.3.0.3.0.nupkg  --logLevel verbose
Octopus Deploy Command Line Tool, version 6.17.0

Detected automation environment: "NoneOrUnknown"
DispatchRequest: GET http://gbrcam-fpsdev1:8000/api
DispatchRequest: GET http://gbrcam-fpsdev1:8000/api
DispatchRequest: GET http://gbrcam-fpsdev1:8000/api
DispatchRequest: GET http://gbrcam-fpsdev1:8000/api/spaces
Space name unspecified, process will run in the default space context
Handshaking with Octopus Server: http://gbrcam-fpsdev1:8000/
DispatchRequest: GET http://gbrcam-fpsdev1:8000/api
Handshake successful. Octopus version: 2020.1.15; API version: 3.0.0
DispatchRequest: GET http://gbrcam-fpsdev1:8000/api/users/me
DispatchRequest: GET http://gbrcam-fpsdev1:8000/api/users/Users-1/spaces
DispatchRequest: GET http://gbrcam-fpsdev1:8000/api/Spaces-1
DispatchRequest: GET http://gbrcam-fpsdev1:8000/api/users/me
Authenticated as: *************** <***************>
Pushing package: E:\TeamCity\buildAgent\work\50fc4d458a9475a6\Sepura.FPS\Sepura.FPS.Web\bin\Sepura.FPS.Web.3.0.3.0.nupkg...
Requesting signature for delta compression from the server for upload of a package with id 'Sepura.FPS.Web' and version '3.0.3.0'
DispatchRequest: GET http://gbrcam-fpsdev1:8000/api/Spaces-1/packages/Sepura.FPS.Web/3.0.3.0/delta-signature
No package with the same ID exists on the server
Falling back to pushing the complete package to the server
DispatchRequest: POST http://gbrcam-fpsdev1:8000/api/Spaces-1/packages/raw?overwriteMode=FailIfExists

System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.ConnectStream.EndWrite(IAsyncResult asyncResult)
   at System.Net.Http.StreamToStreamCopy.BufferWrittenCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Octopus.Client.OctopusAsyncClient.<DispatchRequest>d__56`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

(The working directory is a temporary directory on the TeamCity agent that contains Octo.exe deployed from the TeamCity build server)

The log from the Octopus server is here (C:\Octopus\Logs\OctopusServer.txt):
OctopusServer.log (7.6 KB)

(Note there is no entry in the log for the successful package upload via the web interface)

System diagnostics report:
OctopusDeploy-637238447608179169.zip (348.5 KB)

2

This popped up a few minutes after the package was uploaded by the web interface:

Part URI is not valid per rules defined in the Open Packaging Conventions specification. System.ArgumentException: Part URI is not valid per rules defined in the Open Packaging Conventions specification.
   at System.IO.Packaging.PackUriHelper.ValidatePartUri(Uri partUri)
   at System.IO.Packaging.Package.CreatePart(Uri partUri, String contentType, CompressionOption compressionOption)
   at System.IO.Packaging.Package.CreatePart(Uri partUri, String contentType)
   at Octopus.Server.Web.Api.Actions.Status.SystemReportResponder.IncludeSystemLogs(Package package) in C:\buildAgent\work\47b5b27d3625d6ca\source\Octopus.Server\Web\Api\Actions\Status\SystemReportResponder.cs:line 309
5

Hi David,

Are you still having this issue? I was awaiting a response from our other thread because my theory is that we broke this when creating URL reservations. Please let me know if you got it resolved or if you need further help.

Thanks,
Jeremy

6

Hi Jeremy, I have never succeeded in pushing packages to this server, only uploading packages via the web interface. I had a look in the OctopusServer.txt log but there are no errors reported in there. Even when I set the NLog level to debug I only see the GET:

2020-05-05 17:27:49.1292   6116     14 DEBUG  Request took 77ms: GET http://gbrcam-fpsdev1:8000/api/Spaces-1/packages/Logic.ServiceHub.WinSvc/3.0.5.0/delta-signature 9086e3225106427da28f39d3958533b4 David Robinson
2020-05-05 17:27:51.6115   6116      6  INFO  Stopping the Windows Service

(I stopped the service immediately after the failure)

The service was running under my domain account which is a member of the local adminstrators group for the server PC.

7

Hey David,

Could you please run a test for me and copy the package to the server itself and try to run the push command from the server itself? I want take the network out of the equation and see what kind of results we get.

Thanks,
Jeremy

8

Hi Jeremy,

Running this locally succeeded:

PS C:\Users\robinsond\Documents\Temp\3.0> .\octo.exe push --server http://gbr*******dev1:8000/ --apikey API-AN***********A4HIVCDSBYLC --package .\Logic.ServiceHub.WinSvc.3.0.5.0.nupkg
Octopus Deploy Command Line Tool, version 6.17.0

Detected automation environment: "NoneOrUnknown"
Space name unspecified, process will run in the default space context
Handshaking with Octopus Server: http://gbr*******dev1:8000/
Handshake successful. Octopus version: 2020.1.15; API version: 3.0.0
Authenticated as: David Robinson <***********>
Pushing package: C:\Users\robinsond\Documents\Temp\3.0\Logic.ServiceHub.WinSvc.3.0.5.0.nupkg...
Requesting signature for delta compression from the server for upload of a package with id 'Logic.ServiceHub.WinSvc' and version '3.0.5.0'
Calculating delta
The delta file (130,488 bytes) is 7.17% the size of the orginal file (1,819,517 bytes), uploading...
Delta transfer completed
Push successful
9

Hi David,

Since our previous log showed the development PC is able to correctly communicate with the server, but the connection gets closed, I am thinking this is network related. Our successful test locally on the server shows that the octopus server is set up and operational, but the network is interfering with communications. Is it possible for you to monitor the network traffic to see if there is any reason why the connection is being closed between the two machines on your network, maybe a firewall or router?

Thanks,
Jeremy

10

Hi Jeremy,

I’ve asked our IT people to check the firewall logs and they’ve flagged up this problem:

Hi David,
OK, I’ve found the error on our firewall. I think the reason you are seeing this now is that you are connecting over the VPN and that has to traverse the firewall, whereas in the office it is all LAN based. Anyhow, the problem is with Octopus generating bad HTML.
I’ve highlighted to errors in the log entry below.
Cheers
Cliff

Time: 2020-05-11T09:35:02Z
Id: 0a7bfefd-0100-00c0-5eb9-1c4600000017
Sequencenum: 67
Protection Name: Non Compliant HTTP
Severity: Critical
Confidence Level: Medium
Protection ID: BlockHttpNonProtocolCompliant
Performance Impact: Low
Protection Type: Protocol Anomaly HTTP
Policy Rule UID: 7cb49916-d373-45ba-a29c-f6829c207c9b
Sub Policy Name: 20131024-firewall01 Security
Sub Policy Uid: a357a75b-a1b5-409b-9d0b-b4bb286def70
Reason: illegal header format detected: Missing quotation mark
Client Type: Other: OctopusClient-dotnet/6.17.0 (Microsoft Windows NT 6.2.9200.0; x64) NoneOrUnknown octo
Information: illegal header format detected
Name: Block HTTP Non Compliant
…
Resource: http://gbr********dev1:8000/api/Spaces-1/packages/raw?overwriteMode=FailIfExists
…

Description: credant-client-gatekeeper Traffic Rejected from robinsond(10.123.2.199) to 10.123.130.19 due to illegal header format detected

RE Request ID ##50081## TeamCity package transfer to GBRCAM-FPSDEV1.msg (39 KB)

11

Hey David,

Thanks for the detailed response. I’m gonna get some extra eyes on the firewall information and I’ll keep you updated.

Thanks,
Jeremy

12

Hi Jeremy,

I can supply Wireshark logs from my PC if that would help.

David

13

Hey David,

Extra logs can never hurt! If you want to send me a private message with them attached I will pass them along.

Thanks,
Jeremy

15

Hi David,

Thanks for the Wireshark file. I’m not sure I’ve found the cause yet, but I can see a plausible explanation relating to how octo composes the Content-Disposition header. I’m hopeful we can provide some more information soon.

Regards,

David Young.

16

Hi David,

I have copied a sample octo-packed package from my home PC (where the push failed) onto a PC on the LAN to bypass the VPN and the push command works successfully.

Regards,

David

18

Hey David,

I wanted to reach out and let you know that they are still investigating the issue. Are you able to remote into a PC on-premise to do pushes in the mean time as a workaround?

Sincerely,
Jeremy

19

Hi Jeremy,

I’m moving my TeamCity build onto a machine within the local network so that I avoid the issues with the VPN firewall.

Thanks,

David

20

Hi David,

Glad you got a solution going that works for you. We will keep you posted of any updates.

Thanks,
Jeremy

22

Hey David,

I wanted to update you that the engineering team did a deep dive on your logs and issue. The problem stems from a .net issue. You can see more information on that issue here: https://github.com/dotnet/aspnetcore/issues/2677. Unfortunately, Microsoft chose not to resolve this issue, which is disappointing. If this does change, we’ll be sure to import the necessary changes into Octopus itself ASAP.

In the meantime, It looks like you’ve resolved your issue by moving your build server internally. Thank you for the information on the issue, I’m sure this information will help others in the future.

Please feel free to reach out with any other questions or concerns.

Thanks,
Jeremy

23

Hey David,

I wanted to give you one more update. Our engineers dug a little deeper and found that the issue lies in that your firewall isnt respecting RFC6266. The current Microsoft ASP.NET Core / Octopus behavior is consistent with the most current RFC, RFC6266. I wanted to update you as your firewall may cause other issues in your environment irrelevant to Octopus if it isnt updated to respect the latest RFC. Please feel free to reach out with any questions or concerns.

Thanks,
Jeremy

24

Hi Jeremy,
Thanks for the update, I’ll pass this on to our IT people for them to kick the supplier (Check Point, the product is Endpoint Security VPN).
David

25

Hi David,

You’re welcome! They should be able to fix it for you pretty easily I would assume. Have a great rest of your week.

Thanks,
Jeremy