Large package push to built-in feed failing

One of the things we use Octopus for is packing up routine database backups and pushing/transforming them for other environments. The largest database has been steadily growing and its backup surpassed ~2 gigs in size a couple months back, at which time our job that uses nuget push to the built-in repository began reporting this:

Pushing Our.Db.Backup.2022.1.18-01-Main.nupkg to 'https://our.selfhosted.octopus/nuget/packages'...
  PUT https://our.selfhosted.octopus/nuget/packages/
  InternalServerError https://our.selfhosted.octopus/nuget/packages/ 72569ms
  PUT https://our.selfhosted.octopus/nuget/packages/
  InternalServerError https://our.selfhosted.octopus/nuget/packages/ 70279ms
  PUT https://our.selfhosted.octopus/nuget/packages/
  InternalServerError https://our.selfhosted.octopus/nuget/packages/ 68907ms

Octopus diagnostics has this exception:

Unhandled error on request: PUT https://our.selfhosted.octopus/nuget/packages/ (guid) by OurPackagePushUser : Stream was too long. 

I realize we are pushing the limits of the package feed here, but I also see this issue awhile back implying some level of large-package support: Failure deploying large tar.gz packages

Am I right in thinking weā€™ve hit a fundamental limit of the built-in repository?

Hi Andrew,

Thanks for getting in touch and for your report! From what I can see, reports of this error message resulting from an attempt to push a large package (2GB+) should have been fixed quite a while back (v3.3.6, link to issue). I donā€™t see any issue in regards to limits here. Could you let me know which version of Octopus and Nuget CLI youā€™re running? If youā€™re already beyond v3.3.6, then Iā€™d like to give this a test to see if we can repro locally.

I look forward to hearing back!

Best regards,

Kenny

Octopus server: 2021.3 (8275)
Nuget CLI: NuGet Version: 6.0.0.280

Let me know if I should try newer versions of anything.

If it helps your repro, hereā€™s a snippet of our Powershell that generates the nuget push command.

## Nuget push
$packageObject = gci -Filter ($packageBase + "*.nupkg") | Sort-Object -Property LastWriteTime -Descending | Select-Object -First 1

$nugetPushArgs = @(
    "push",
    "$packageObject",
    "-ApiKey", $apiKey,
    "-Source", "$baseUrl/nuget/packages",
    "-Timeout", "3000")

$nugetPushTiming = [Diagnostics.Stopwatch]::StartNew()
$nugetPushOutput = & $nugetExecutable $nugetPushArgs | Out-String
$nugetPushTiming.Stop()

Thanks for the quick response! And itā€™s good to hear that this isnā€™t a known limitation.

Hi Andrew,

Thanks for following up with those details. Iā€™m attempting a repro at the moment and will let you know how it goes. :slight_smile:

Best regards,

Kenny

Hi Andrew,

Thanks for your patience. Sadly in regards to our troubleshooting efforts I was unsuccessful in my attempt to reproduce this error. I havenā€™t been able to find anything too helpful while digging either, unfortunately, so Iā€™m scratching my head.

Just to help narrow it down a bit more as I keep at it, could you let me know if youā€™re still hitting this issue when trying to push a much smaller package in the same way? You mentioned the exception in the diagnostics - do you see anything else relevant there?

I appreciate your help and patience so far, and I look forward to getting to the bottom of this one. :slight_smile:

Best regards,

Kenny

Thatā€™s all useful!

We have other (all smaller) pushes via the same method that have continued uninterrupted. Iā€™ll see if I can get a more detailed view of the precise size cutoff.

In the meantime, I pulled a full stack trace from the detailed Octopus logs. Sorry for not including this from the start.

Unhandled error on request: PUT https://our.selfhosted.octopus/nuget/packages/ (guid) by OurPackagePushUser  : Stream was too long. System.IO.IOException: Stream was too long.
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
   at Nancy.IO.RequestStream.MoveToWritableStream()
   at Nancy.IO.RequestStream..ctor(Stream stream, Int64 expectedLength, Int64 thresholdLength, Boolean disableStreamSwitching)
   at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in ./source/Octopus.Server/Owin/NancyMiddleware.cs:line 111
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.HttpOverrides.HttpMethodOverrideMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 38
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 48
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 55
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 67
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.Middleware.RequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/RequestLoggerMiddleware.cs:line 42
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 75
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 103

Hereā€™s an extraction from our logs around the time the problem started. I put a flat line for C# int max (2147483647). The jagged green line is the package size.

When the dots bunch up, above the int max line, that is our job trying to push the too-large backup every time it runs because it doesnā€™t see a prior release for the day in Octopus.

When the dots arenā€™t bunched up, that is the more normal daily push cadence, indicating that most of the time our jobs runs, it sees a prior push from the day already exists and doesnā€™t bother trying to push anything to Octopus.

Further, I can also query the log output for the push operation, and that output contains ā€˜errorā€™ only when package size exceeds int max.

(edit: better chart)

The actual limit looks to be a bit under int.max. Pushing garbage files, I can get a zip extraction error at 2147483147 bytes (int max - 500 bytes), the same error I get pushing much much smaller garbage files.

Moving up 250 bytes in size to 2147483397 (int max - 250), Octopus reports a slightly different exception:

Unhandled error on request: PUT https://our.selfhosted.octopus/nuget/packages/ (guid) by OurPackagePushUser   : Array dimensions exceeded supported range. System.OutOfMemoryException: Array dimensions exceeded supported range.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
   at Nancy.IO.RequestStream..ctor(Stream stream, Int64 expectedLength, Int64 thresholdLength, Boolean disableStreamSwitching)
   at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in ./source/Octopus.Server/Owin/NancyMiddleware.cs:line 111
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.HttpOverrides.HttpMethodOverrideMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 38
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 48
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 55
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 67
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.Middleware.RequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/RequestLoggerMiddleware.cs:line 42
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 75
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 103

Moving up 150 bytes to 2147483547 (int max - 100), we get the original ā€œStream was too longā€ exception.

If it matters, these pushes all originated from a different location than the normal job. Hopefully this helps.

Hi Andrew,

I greatly appreciate that awesome level of detail from your efforts. Just a quick update to let you know Iā€™m looking at this one at the moment, and Iā€™ll probably end up asking around internally with the engineers.

That int max is a very interesting thing to point out. I canā€™t seem to make sense as to how exactly that could be at play here, or if itā€™s just a coincidence, but the same bug that hit Microsoft Exchange as the clock ticked into 2022 might somehow be at play here as well. I saw this limit cause a failure just recently in Octo.exe creating a package with a number greater than 2147483647 in its version, e.g. 1.0.2147483647 which was also unexpected.

Iā€™ll let you know any and all updates, and let us know if you have any other questions in the meantime.

Best regards,

Kenny

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.