Hi
We have a fairly urgent issue, using Octopus Deploy 2.3.6.1385 and the built-in Nuget server. We are getting timeouts after 5 mins, uploading large packages over a slow connection (long distance).
We have increased the timeout on the Nuget.exe, but it’s still timing out. Could you let me know if Octopus Nuget implementation has a timeout, and if so if it can be increased?
Thanks,
Mike
Hi Mike - we don’t set a timeout explicitly on the Octopus side, though there are a few components involved so it isn’t unthinkable that something may time out.
Can you let us know what the output/response from NuGet.exe is, and if there are any errors in the Octopus Configuration > Diagnostics log after the failed push?
Regards,
Nick
Hi Nick,
That’s for the response. The error from the Nuget client is.
Pushing MyPackage 1.0.0 to https://ocotpus-server/nuget/packages…
The request was aborted: The request was canceled.
I cannot find anything else in the error logs or diagnostics. We are at the point we are going to have to back out of the embedded Nuget server unless this can be resolved ASAP. Any help you can offer would be great.
Could you also clarify if nuget packages are automatically purged based on time. Are active packages whitelisted from the process? This would break our ability to support self-healing in AWS.
Thanks,
Mike
Hi Mike,
It’s possible we’re seeing HTTP.sys timeouts here - the only one I can find reference to is a 2:00 timeout on receiving entitiy bodies, so may not be the exact one. Given the difficulty of tracking it down I’m afraid reverting to your external feed is probably the best course of action right now.
I think that if this turns out to be widespread it will be a frustrating limitation; I’ve raised https://github.com/OctopusDeploy/Issues/issues/931 and will make sure it’s addressed as early as we can. Would you be so kind as to add some details of the server OS/.NET versions to the ticket?
Regarding package retention, purging needs to be enabled manually, and will only apply to packages not used in a release. Once retention policies clean up the release we also clean up the package. Unreleased packages can be removed after a fixed time. You need to edit the retention policy in order for this to take effect.
Does this package retention model match what you need? Still looking for feedback on any improvements we can make.
Hope this helps,
Nick
Hi Nick,
That’s for your help. I tried installing an instance of NugetGallery with IIS, but discovered I still get the timeout of 5mins during the push! I’m perplexed where the time out is occurring. I tried tweaking various IIS/web.config settings to no avail. It does appear to a timeout on the server-side, so maybe you are right about HTTP.SYS? I’m using Windows 2012.
I’m familiar with retention policies, but the dialog only shows Server Policy and Tentacle Policy. It wasn’t clear that this also applies to Nuget feed, but is good news. As long as it’s not purging an active releases then that is exactly what we need. We build and push packages on every checkin, so disk space becomes an issue after a while.
I did notice there seem to be 2 copies of the packages on disk?
Thanks,
Mike
Ah - sorry Mike, retention policies for packages are in 2.4 (see the right-hand panel in the retention policy tab in that version). In 2.3 they’ll be left alone indefinitely.
The two copies observation is interesting, though IIRC we also did some work in 2.4 to ensure we bypass NuGet’s machine cache, so that might eliminate the second copy you’re finding.
Regards,
Nick
Hi Nick,
Thanks for the clarification. Is 2.4 due out soon?
Thanks,
Mike
We’re working through the last items: https://github.com/OctopusDeploy/Issues/issues?milestone=30&page=1&state=open
Three to go (was ~8 this morning) so shouldn’t be far away.
Regards,
Nick
Are you aware of this bug in nuget ?
to sum up, do not use a timeout value which round by 60, or your custom timeout value will be ignored by nuget.exe.
I have encoutered this bug months ago when deploying large packages for octopus.
Wow thanks Thomas, was not aware of this!
I spent several hours on a convoluted work around, after failing to work out what was going on. Will try this out tomorrow!
Thomas, thanks! No doubt saved us some pain. Mike, did you have a chance to test it out?
Cheers!
Hi Nicholas,
Yes tested and working ok now with a timeout value not divisible by 60! An amazing bug in the Nuget client!
Thanks,
Mike
Wow! Thanks for the follow-up, glad to hear you’re up and running.
We have experienced the same timeout when our Octopus package went from 900Mb to 970Mb. After downloading and compiled the pre-release of the Nuget 3.0 exe respecting the timeout value of the push command, we manage to resolve the problem.
However, we observe that there are network traffic for uploading the package for about a minute. The following 5 minutes we observe a lot of CPU and memory usage on the octopus server. The memory usage on Octopus.Server.exe was almost 9 GB at the maximum during this “post processing”. CPU Usage was up to apx 70%. Our Octopus server is configured with 16GB memory and two cores of 2.4 GHz and we use an Enterprise licensed Octopus Server with version 2.5.9.554