Our server suddenly started returning 500 errors for our custom icons for projects. The response is:
{
“ErrorMessage”: “Object reference not set to an instance of an object.”
}
Server version: 2020.6.4688
Log shows:
Unhandled error on request: GET http://octopus.xxxxxx.local/api/Spaces-1/projects/Projects-321/logo?cb=Projects-321-2019060709374007 80000a10-0004-f400-b63f-84710c7967bb by xxxxxx : Object reference not set to an instance of an object. System.NullReferenceException: Object reference not set to an instance of an object.
at Nancy.Responses.StreamResponse.<>c__DisplayClass2_0.b__0(Stream stream)
at Octopus.NancyOwin.NancyMiddleware.RequestComplete(NancyContext context, IDictionary2 environment) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Owin\NancyMiddleware.cs:line 165 at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Owin\NancyMiddleware.cs:line 108 --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.HttpOverrides.HttpMethodOverrideMiddleware.Invoke(HttpContext context) at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, Lazy1 unitOfWork, ILog log, IActionContextAccessor actionContextAccessor) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\UnitOfWorkMiddleware.cs:line 42
at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, Lazy`1 unitOfWork, ILog log, IActionContextAccessor actionContextAccessor) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\UnitOfWorkMiddleware.cs:line 42
at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\Middleware\OctopusClientOldVersionWarningMiddleware.cs:line 38
at Octopus.Server.Web.Middleware.RequestLoggerMiddleware.InvokeAsync(HttpContext context) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\Middleware\RequestLoggerMiddleware.cs:line 42
at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\Middleware\ErrorHandlingMiddleware.cs:line 70
Thanks for getting in touch, and welcome! I’m sorry to hear you’re hitting this issue. I haven’t been able to reproduce this, however, so I’d like to ask a few more questions about your setup if you don’t mind.
Did this issue start occurring immediately following something like a project cloning or migration? Or even a migration or update of Octopus Server itself or the database?
Maybe a shot in the dark, but doing a little digging in case it’s already a known issue, I came across this which seems very similar, and I’m wondering if any part of it lines up with your setup?
I’m hoping that helps us narrow down what’s going on so I can attempt to reproduce this issue!
I reviewed that post and it is similar as we moved servers recently - guess the images were cached for an amount of time as we moved a few weeks ago. I verified the artifacts directory and moved over the project artifacts, and restarted the service, but that didn’t solve the problem. I can see if I add an icon the artifact is updated and then works. Seems like a bug given it is a null reference exception vs something a bit more useful (i.e. the artifact is missing or the artifact is corrupted).
Thanks for the update. I agree, failing in this way, and not being helpful to troubleshoot, seems like something we ought to fix up. Also the original issue with the logo links during a migration seems possibly like a bug. However I’m unable to reproduce this locally. So I can attempt an exact repro, could you also let me know if there was any difference in versions between the source and destination servers during the move? Octopus needs exact matching versions to migrate from/to, so just want to rule out the possibility of some funny behavior if these differed in any way.
We upgraded the old instance to v2020.6.4688 and installed the same version on the new instance. We shutdown the old instance, disabled the services and pointed the new instance to the existing database. We then changed the configuration for the package cache setting as that apparently was not stored in the database. What’s strange is I didn’t notice the icon thing until a few days after the migration. Its possible I just missed it, but seems like something I would have noticed.
Thanks for following up! That is a bit strange, but you might be right that’s a caching thing that’s lasting a bit longer than expected. I’m wondering if perhaps during the migration the directory Octopus looks for the artifacts in was changed? So after moving the artifacts folder to the new server, Octopus might be configured to look for it in a different folder (since the file and reference to it shouldn’t have been changed during this process). Could you run the following command against Octopus.Server.exe CLI and let me know if this is pointing to the same directory in which you moved the artifacts to?
Octopus.Server.exe show-configuration
What is returned from this command will include a line similar to this: