Large package push to built-in feed failing

One of the things we use Octopus for is packing up routine database backups and pushing/transforming them for other environments. The largest database has been steadily growing and its backup surpassed ~2 gigs in size a couple months back, at which time our job that uses nuget push to the built-in repository began reporting this:

Pushing Our.Db.Backup.2022.1.18-01-Main.nupkg to 'https://our.selfhosted.octopus/nuget/packages'...
  PUT https://our.selfhosted.octopus/nuget/packages/
  InternalServerError https://our.selfhosted.octopus/nuget/packages/ 72569ms
  PUT https://our.selfhosted.octopus/nuget/packages/
  InternalServerError https://our.selfhosted.octopus/nuget/packages/ 70279ms
  PUT https://our.selfhosted.octopus/nuget/packages/
  InternalServerError https://our.selfhosted.octopus/nuget/packages/ 68907ms

Octopus diagnostics has this exception:

Unhandled error on request: PUT https://our.selfhosted.octopus/nuget/packages/ (guid) by OurPackagePushUser : Stream was too long. 

I realize we are pushing the limits of the package feed here, but I also see this issue awhile back implying some level of large-package support: Failure deploying large tar.gz packages

Am I right in thinking we’ve hit a fundamental limit of the built-in repository?

Hi Andrew,

Thanks for getting in touch and for your report! From what I can see, reports of this error message resulting from an attempt to push a large package (2GB+) should have been fixed quite a while back (v3.3.6, link to issue). I don’t see any issue in regards to limits here. Could you let me know which version of Octopus and Nuget CLI you’re running? If you’re already beyond v3.3.6, then I’d like to give this a test to see if we can repro locally.

I look forward to hearing back!

Best regards,

Kenny

Octopus server: 2021.3 (8275)
Nuget CLI: NuGet Version: 6.0.0.280

Let me know if I should try newer versions of anything.

If it helps your repro, here’s a snippet of our Powershell that generates the nuget push command.

## Nuget push
$packageObject = gci -Filter ($packageBase + "*.nupkg") | Sort-Object -Property LastWriteTime -Descending | Select-Object -First 1

$nugetPushArgs = @(
    "push",
    "$packageObject",
    "-ApiKey", $apiKey,
    "-Source", "$baseUrl/nuget/packages",
    "-Timeout", "3000")

$nugetPushTiming = [Diagnostics.Stopwatch]::StartNew()
$nugetPushOutput = & $nugetExecutable $nugetPushArgs | Out-String
$nugetPushTiming.Stop()

Thanks for the quick response! And it’s good to hear that this isn’t a known limitation.

Hi Andrew,

Thanks for following up with those details. I’m attempting a repro at the moment and will let you know how it goes. :slight_smile:

Best regards,

Kenny

Hi Andrew,

Thanks for your patience. Sadly in regards to our troubleshooting efforts I was unsuccessful in my attempt to reproduce this error. I haven’t been able to find anything too helpful while digging either, unfortunately, so I’m scratching my head.

Just to help narrow it down a bit more as I keep at it, could you let me know if you’re still hitting this issue when trying to push a much smaller package in the same way? You mentioned the exception in the diagnostics - do you see anything else relevant there?

I appreciate your help and patience so far, and I look forward to getting to the bottom of this one. :slight_smile:

Best regards,

Kenny

That’s all useful!

We have other (all smaller) pushes via the same method that have continued uninterrupted. I’ll see if I can get a more detailed view of the precise size cutoff.

In the meantime, I pulled a full stack trace from the detailed Octopus logs. Sorry for not including this from the start.

Unhandled error on request: PUT https://our.selfhosted.octopus/nuget/packages/ (guid) by OurPackagePushUser  : Stream was too long. System.IO.IOException: Stream was too long.
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
   at Nancy.IO.RequestStream.MoveToWritableStream()
   at Nancy.IO.RequestStream..ctor(Stream stream, Int64 expectedLength, Int64 thresholdLength, Boolean disableStreamSwitching)
   at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in ./source/Octopus.Server/Owin/NancyMiddleware.cs:line 111
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.HttpOverrides.HttpMethodOverrideMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 38
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 48
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 55
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 67
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.Middleware.RequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/RequestLoggerMiddleware.cs:line 42
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 75
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 103

Here’s an extraction from our logs around the time the problem started. I put a flat line for C# int max (2147483647). The jagged green line is the package size.

When the dots bunch up, above the int max line, that is our job trying to push the too-large backup every time it runs because it doesn’t see a prior release for the day in Octopus.

When the dots aren’t bunched up, that is the more normal daily push cadence, indicating that most of the time our jobs runs, it sees a prior push from the day already exists and doesn’t bother trying to push anything to Octopus.

Further, I can also query the log output for the push operation, and that output contains ‘error’ only when package size exceeds int max.

(edit: better chart)

The actual limit looks to be a bit under int.max. Pushing garbage files, I can get a zip extraction error at 2147483147 bytes (int max - 500 bytes), the same error I get pushing much much smaller garbage files.

Moving up 250 bytes in size to 2147483397 (int max - 250), Octopus reports a slightly different exception:

Unhandled error on request: PUT https://our.selfhosted.octopus/nuget/packages/ (guid) by OurPackagePushUser   : Array dimensions exceeded supported range. System.OutOfMemoryException: Array dimensions exceeded supported range.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
   at Nancy.IO.RequestStream..ctor(Stream stream, Int64 expectedLength, Int64 thresholdLength, Boolean disableStreamSwitching)
   at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in ./source/Octopus.Server/Owin/NancyMiddleware.cs:line 111
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.HttpOverrides.HttpMethodOverrideMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 38
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 48
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 55
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 67
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.Middleware.RequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/RequestLoggerMiddleware.cs:line 42
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 75
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 103

Moving up 150 bytes to 2147483547 (int max - 100), we get the original “Stream was too long” exception.

If it matters, these pushes all originated from a different location than the normal job. Hopefully this helps.

Hi Andrew,

I greatly appreciate that awesome level of detail from your efforts. Just a quick update to let you know I’m looking at this one at the moment, and I’ll probably end up asking around internally with the engineers.

That int max is a very interesting thing to point out. I can’t seem to make sense as to how exactly that could be at play here, or if it’s just a coincidence, but the same bug that hit Microsoft Exchange as the clock ticked into 2022 might somehow be at play here as well. I saw this limit cause a failure just recently in Octo.exe creating a package with a number greater than 2147483647 in its version, e.g. 1.0.2147483647 which was also unexpected.

I’ll let you know any and all updates, and let us know if you have any other questions in the meantime.

Best regards,

Kenny

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.