One of the things we use Octopus for is packing up routine database backups and pushing/transforming them for other environments. The largest database has been steadily growing and its backup surpassed ~2 gigs in size a couple months back, at which time our job that uses nuget push to the built-in repository began reporting this:
Pushing Our.Db.Backup.2022.1.18-01-Main.nupkg to 'https://our.selfhosted.octopus/nuget/packages'...
PUT https://our.selfhosted.octopus/nuget/packages/
InternalServerError https://our.selfhosted.octopus/nuget/packages/ 72569ms
PUT https://our.selfhosted.octopus/nuget/packages/
InternalServerError https://our.selfhosted.octopus/nuget/packages/ 70279ms
PUT https://our.selfhosted.octopus/nuget/packages/
InternalServerError https://our.selfhosted.octopus/nuget/packages/ 68907ms
Octopus diagnostics has this exception:
Unhandled error on request: PUT https://our.selfhosted.octopus/nuget/packages/ (guid) by OurPackagePushUser : Stream was too long.
I realize we are pushing the limits of the package feed here, but I also see this issue awhile back implying some level of large-package support: Failure deploying large tar.gz packages
Am I right in thinking weāve hit a fundamental limit of the built-in repository?
Thanks for getting in touch and for your report! From what I can see, reports of this error message resulting from an attempt to push a large package (2GB+) should have been fixed quite a while back (v3.3.6, link to issue). I donāt see any issue in regards to limits here. Could you let me know which version of Octopus and Nuget CLI youāre running? If youāre already beyond v3.3.6, then Iād like to give this a test to see if we can repro locally.
Thanks for your patience. Sadly in regards to our troubleshooting efforts I was unsuccessful in my attempt to reproduce this error. I havenāt been able to find anything too helpful while digging either, unfortunately, so Iām scratching my head.
Just to help narrow it down a bit more as I keep at it, could you let me know if youāre still hitting this issue when trying to push a much smaller package in the same way? You mentioned the exception in the diagnostics - do you see anything else relevant there?
I appreciate your help and patience so far, and I look forward to getting to the bottom of this one.
We have other (all smaller) pushes via the same method that have continued uninterrupted. Iāll see if I can get a more detailed view of the precise size cutoff.
In the meantime, I pulled a full stack trace from the detailed Octopus logs. Sorry for not including this from the start.
Unhandled error on request: PUT https://our.selfhosted.octopus/nuget/packages/ (guid) by OurPackagePushUser : Stream was too long. System.IO.IOException: Stream was too long.
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
at Nancy.IO.RequestStream.MoveToWritableStream()
at Nancy.IO.RequestStream..ctor(Stream stream, Int64 expectedLength, Int64 thresholdLength, Boolean disableStreamSwitching)
at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in ./source/Octopus.Server/Owin/NancyMiddleware.cs:line 111
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.HttpOverrides.HttpMethodOverrideMiddleware.Invoke(HttpContext context)
at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 38
at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 48
at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 55
at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 67
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Octopus.Server.Web.Middleware.RequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/RequestLoggerMiddleware.cs:line 42
at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 75
at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 103
Hereās an extraction from our logs around the time the problem started. I put a flat line for C# int max (2147483647). The jagged green line is the package size.
When the dots bunch up, above the int max line, that is our job trying to push the too-large backup every time it runs because it doesnāt see a prior release for the day in Octopus.
When the dots arenāt bunched up, that is the more normal daily push cadence, indicating that most of the time our jobs runs, it sees a prior push from the day already exists and doesnāt bother trying to push anything to Octopus.
Further, I can also query the log output for the push operation, and that output contains āerrorā only when package size exceeds int max.
The actual limit looks to be a bit under int.max. Pushing garbage files, I can get a zip extraction error at 2147483147 bytes (int max - 500 bytes), the same error I get pushing much much smaller garbage files.
Moving up 250 bytes in size to 2147483397 (int max - 250), Octopus reports a slightly different exception:
Unhandled error on request: PUT https://our.selfhosted.octopus/nuget/packages/ (guid) by OurPackagePushUser : Array dimensions exceeded supported range. System.OutOfMemoryException: Array dimensions exceeded supported range.
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
at Nancy.IO.RequestStream..ctor(Stream stream, Int64 expectedLength, Int64 thresholdLength, Boolean disableStreamSwitching)
at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in ./source/Octopus.Server/Owin/NancyMiddleware.cs:line 111
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.HttpOverrides.HttpMethodOverrideMiddleware.Invoke(HttpContext context)
at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 38
at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 48
at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 55
at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 67
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Octopus.Server.Web.Middleware.RequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/RequestLoggerMiddleware.cs:line 42
at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 75
at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 103
Moving up 150 bytes to 2147483547 (int max - 100), we get the original āStream was too longā exception.
If it matters, these pushes all originated from a different location than the normal job. Hopefully this helps.
I greatly appreciate that awesome level of detail from your efforts. Just a quick update to let you know Iām looking at this one at the moment, and Iāll probably end up asking around internally with the engineers.
That int max is a very interesting thing to point out. I canāt seem to make sense as to how exactly that could be at play here, or if itās just a coincidence, but the same bug that hit Microsoft Exchange as the clock ticked into 2022 might somehow be at play here as well. I saw this limit cause a failure just recently in Octo.exe creating a package with a number greater than 2147483647 in its version, e.g. 1.0.2147483647 which was also unexpected.
Iāll let you know any and all updates, and let us know if you have any other questions in the meantime.