Null ref when opening a project

Octopus Server
1

Hi we have a bunch of projects dating back to 2014 that recently stopped loading in the octopus deploy front end, we are currently running octo deploy version : 2023.1.10475.

Any idea how we fix this?

image
image743×213 6.95 KB

Unhandled error on request: "Object reference not set to an instance of an object." System.NullReferenceException: Object reference not set to an instance of an object.
   at Octopus.Core.Features.Projects.GetProjectLogoRequestHandler.AsBytes(Stream source) in ./source/Octopus.Core/Features/Projects/GetProjectLogoRequestHandler.cs:line 62
   at Octopus.Core.Features.Projects.GetProjectLogoRequestHandler.Handle(GetProjectLogoRequest request, CancellationToken cancellationToken) in ./source/Octopus.Core/Features/Projects/GetProjectLogoRequestHandler.cs:line 53
   at Octopus.Core.Infrastructure.Mediator.AutofacMediator.Request[TRequest,TResponse](IRequest`2 request, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/AutofacMediator.cs:line 37
   at Octopus.Core.Infrastructure.Mediator.Decorators.SystemComponentModelValidationDecorator.Request[TRequest,TResponse](IRequest`2 request, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/SystemComponentModelValidationDecorator.cs:line 60
   at Octopus.Core.Infrastructure.Mediator.Decorators.FluentValidationsDecorator.Request[TRequest,TResponse](IRequest`2 request, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/FluentValidationsDecorator.cs:line 78
   at Octopus.Core.Infrastructure.Mediator.Decorators.MessageBusSiphoningDecorator.Request[TRequest,TResponse](IRequest`2 request, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/MessageBusSiphoningDecorator.cs:line 37
   at Octopus.Server.Web.Controllers.Projects.GetProjectLogoController.GetProjectLogo(GetProjectLogoRequest request, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Controllers/Projects/GetProjectLogoController.cs:line 40
   at lambda_method7674366(Closure , Object )
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Octopus.Server.Web.Middleware.BoundaryTrailerRewriteMiddleware.Invoke(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/BoundaryTrailerRewriteMiddleware.cs:line 44
   at Octopus.Server.Web.Infrastructure.Authentication.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) in ./source/Octopus.Server/Web/Infrastructure/Authentication/AuthorizationMiddlewareResultHandler.cs:line 50
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 46
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 46
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 52
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 48
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 59
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, IUserAuthenticator userAuthenticator, IAuthCookieService authCookieService, IWebAuthCache authCache, ILogger logger) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 58
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.Middleware.LegacyRequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/LegacyRequestLoggerMiddleware.cs:line 41
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 76
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 117
3

Hi @Gavin_Toon1,

Thanks for posting your issue, and I’m sorry to hear you’re experiencing that behavior after upgrading.

Is this occurring for multiple users, or just one?

I think the first thing to eliminate would be anything being cached by your browser, as we have seen similar behavior from other users in the past post-upgrade. So, if you haven’t tried it yet, you might see if using a private session or a different browser allows these projects to load.

If that isn’t helpful, then if you could capture a HAR file while trying to load one of these projects and send that my way, I can review it for any additional clues. You can use the following link to send it to me securely:

Gavin_Toon1 | Octopus Support

Let me know once you’ve done that, and I will review the HAR to see if I can better understand what might be causing this.

Best regards,
Patrick

4

Thanks Patrick.

I can confirm its impacting all users, and attempting it in an incognito window had the same result.

I’ve uploaded a har file from an incognito test load of the deployments page.

6

Good morning @Gavin_Toon1,

Just jumping in for Patrick as he is currently offline as part of our US based team. I am sorry you are seeing this issue.

We have taken a good look at this and it seems to be an issue with the Project Logo and Octopus not being able to find it:

System.NullReferenceException: Object reference not set to an instance of an object.
   at Octopus.Core.Features.Projects.GetProjectLogoRequestHandler.AsBytes(Stream source) in ./source/Octopus.Core/Features/Projects/GetProjectLogoRequestHandler.cs:line 62

We cannot see where that project logo lives in the database so I have passed this over to our engineers who should be able to identify the exact issue and provide us with a script to sort this out for you.

We will be in touch once they have replied, they do work out of Australia so it will be later on tonight we hear anything if they are able to take a look later for you.

Kind Regards,
Clare

8

Good morning @Gavin_Toon1,

Our engineers got back to us last night and mentioned that the project logo setting lives directly inside the project JSON in the database, it only appears however if you have a different logo selected for that project, which is why I couldn’t find where it lived as my test instance only has the standard default logos for each project.

Here is a partial example of a projects JSON if you choose a different icon Octopus has built-in for your project:

AutoDeployReleaseOverrides":[],"ExtensionSettings":[],"Icon":{"Id":"map-signs","Color":"#3CA4F3"}}

And here is a partial example of a project where I have changed the logo to a custom image:

AutoDeployReleaseOverrides":[],"ExtensionSettings":[],"LogoAttachmentKey":"Projects-721-2023071908491661","LogoAttachmentMimeType":"image/jpeg"}

So it looks like the projects that are having this issue either have custom logos for that project or you have changed the icon to a different built-in Octopus one.

To confirm this is the case can you please run the below queries separately on your Octopus Database, if you have access to your Octopus DB. If you do not and you have no DBA to run the commands please let us know:

SELECT *
FROM dbo.Project
WHERE JSON LIKE '%"Icon"%'

And

SELECT *
FROM dbo.Project
WHERE JSON LIKE '%"LogoAttachmentKey"%'

Once you have done this are you able to post up the results of those queries to our support site link that Patrick sent over please and I will take a look for you and see if we can provide you with a query to clear those up.

Its odd that this has happened, did you just run a direct upgrade on the same server or did you move Octopus to a different server or change the Octopus Artifacts folder at all during the upgrade?

The project logos live in the default Artifacts folder - C:\Octopus\Artifacts so unless you changed that by using a command in our documentation here I am unsure why an upgrade would result in the logos missing.

Let me know once those SQL query results have been uploaded and I will take a look at them and get back to you with another SQL command to fix those projects.

Kind Regards,
Clare

9 
SELECT *
FROM dbo.Project
WHERE JSON LIKE '%"LogoAttachmentKey"%'

This query did create results, and I’ve uploaded them, However all the projects listed in that query work fine in our octo deploy instance, the project pages work perfectly (however some have missing logo images)

SELECT *
FROM dbo.Project
WHERE JSON LIKE '%"Icon"%'

this query returned no results

I’ve also uploaded one of the affected projects database project row(the one used in the HAR recording)

We aren’t sure when this started happening (likely the last year or so), as we have quite a few stale historic projects that are deployed very rarely. However I can confirm that this octopus deploy instance was originally in ravendB

10

Hey @Gavin_Toon1,

Very interesting, thank you for the DB query results and also one from an affected project, I will get that in front of our engineers as I cannot see anything wrong with the JSON in the affected project you sent over and if the others that have the Logo on are working fine its a bit puzzling as to the stack trace error.

Would you mind sending over the full diagnostic logs of your instance please so the engineers have it:

image
image3649×638 148 KB

Can we also confirm where you are seeing this error, is it on the projects overview page, so when you click on an affected project that error comes straight up?

When you are clicking on any of the projects in that query with the LogoAttachment in the JSON they all load fine?

I will collate all the information we have so far and get that to the engineers but they would want your Octopus server logs so they can take a closer look at pre and post calls when you have seen that error appear so if you can upload the full diagnostic logs that would be great please.

Kind Regards,
Clare

11

The error occurs when opening a project on the “Overview” and “Releases” page, the rest of the pages work. We have temporarily worked around the issue by deep linking into the releases page by using a known version number, and the deployments are working fine.

/deployments - breaks
/deployments/releases - breaks
/deployments/releases/2.1.118 - works

yes, all the pages for those projects work.

I’ve uploaded the full diagnostic logs

12

Hey @Gavin_Toon1,

Amazing, thank you for the logs and the breakdown of the page errors and also the fact you have a workaround!

I have given the engineers everything so far so I will add the diag logs and this extra information and let you know once they have responded.

Kind Regards,
Clare

13

Hey @Gavin_Toon1,

Our engineers got back to me last night and they also found it strange those projects load with the LogoAttachmentKey as null.

They seem to think the stack trace you posted up is a red herring and its not directly relating to the error you are seeing with the ‘obj’ issue.

Funnily enough I was testing something yesterday for this issue as there was another customer that had the same UI error. I managed to replicate the ‘obj’ issue on the same instance as you are using and also got the error in the HAR file on the ‘progression’ call but I then realised the stack trace was different so I changed tactics.

Now I am thinking you are seeing that issue I was testing yesterday as I also see it on the project overview page and the releases page but no other pages on that project. Here is what I tested yesterday so it looks to be the same issue you are seeing:

image
image2359×942 168 KB

Sorry to start from scratch here but can you please run the below query on your Octopus Database and post up the results in the same secure link. I will review them and if it is the same issue I replicated yesterday there is an easy fix for it luckily which we can get you to run and it should clean up all your projects in one go. I just need to ensure this is the same issue as the previous customer:

SELECT *
FROM dbo.Release
WHERE JSON Like '%"ActionName":null%'

Here we are looking for the JSON in the Release table, if you have a package reference in a project the action name for that step with the package reference in it may be set to ‘null’ which will result in the ‘Obj’ error as Octopus cannot find the step with the package reference in it.

JSON for a non-affected project:

"SelectedPackages":[{"ActionName":"Deploy a Package","PackageReferenceName":"","Version":"1.0.1"}]

JSON for an affected project:

"SelectedPackages":[{"ActionName":null,"PackageReferenceName":"","Version":"1.0.1"}]

So hopefully that SELECT query brings back a list of the projects that are actually broken and we can send you the commands to fix those up.

Again sorry for the initial confusion with the stack trace, hopefully we are on the right lines now. Let us know once you have uploaded the results of the SELECT query and also if you can correlate the results (or at least a few of them if there are a lot) with the broken projects in the UI that would be beneficial.

Kind Regards,
Clare

14

I’ve uploaded two files :

actionnamenull.zip

The full result of the action name null query

and

projectstatus.csv

which is a distinct list of all the projects (ids & slugs) and if their /deployments page is working or not.

We also have 77 projects that don’t show up in that actionname null query and they seem to all work.

15

Hi @Gavin_Toon1,

Thanks for providing us with the results of that query along with the additional information!

I’m not seeing anything obvious standing out in the results for the projects that are working vs the projects that aren’t working, but I’ve passed this information on to our developers to review further.

We’ll keep you posted on what we hear back from them, but I just wanted to give you the update that we’re still looking into this, and hoping to determine the root cause soon.

Best regards,

16

Hey @Gavin_Toon1,

Just wanted to pass on an update from the engineers last night, the US support team and then engineers did some testing based on what you have given us and none of us can workout why some of those projects are working and some are not. There does not seem to be a common denominator we can look at and write a script to get it fixed.

The engineer did mention would it be possible to clean up those releases manually, do you have a retention policy setup on your instance by any chance?

From what I have seen you have a fair few projects - 89 to be exact - which are broken so really the only way you will be able to clean those projects up is to have a retention policy set otherwise you will have to manually go into each of the 89 projects, through each release number (since the release page wont work) and then remove that release vi the UI - which is not feasible in my opinion.

If you do not have a policy set it would be worth reading our documentation on retention and setting that up as we do see a lot of complications with slow instances after a while of customer use if they do not have a policy set which cleans up old releases, you can customise the policy depending on your auditing requirements.

If you do want to try this out we always recommend going large first (ie keep the first 100 releases) and then wait a few days then reduce that down to say 80. If you start small it can overwhelm your Octopus server as it tries to remove thousands of releases at once so its best to start big and go smaller in chunks.

If you do have a policy set are you able to perhaps reduce it so it gets rid of some of those releases, that way you should hopefully be deleting the affected releases which will mean you will eventually be able to access the overview and release page for that project as it will no longer have corrupted releases its trying to load.

Let me know your thoughts on this but we are really struggling to pinpoint what the issue is with all of those releases. If you are unable to set a retention policy let me know and I can take that back to the engineers to see what other options we have. I am sorry you are still seeing this and we have not gotten to the bottom of it for you yet.

Kind Regards,
Clare

17

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.