Crash trying to replace certificate

mauricio · 13 September 2021 16:59

Hi, I have this certificate that Octopus complains about: “This certificate was unable to be parsed and may be in an invalid format. This certificate will not be able to be used in Octopus deployments and you may need to upload a new certificate which can be correctly loaded.”
So I try to upload it again but I’m getting a server crash.
Creating a new certificate works fine.

2021-09-13 16:54:27.9296   4336     79 ERROR  Unhandled error on request: POST http://build4.elevate.internal:8080/api/Spaces-1/certificates/certificates-prod-k8s-ca-certificate/replace 80000f56-0001-fa00-b63f-84710c7967bb by mauricio : Specified argument was out of the range of valid values.
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at Octopus.Core.Model.Certificate.GetValidMetadata() in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Core\Model\Certificates\Certificate.cs:line 251
   at lambda_method1113(Closure , Certificate )
   at Nevermore.Mapping.DelegatePropertyHandler`2.Read(Object target)
   at Nevermore.Util.DataModificationQueryBuilder.GetDocumentParameters(Func`2 allocateId, Object customAssignedId, Nullable`1 customIdAssignmentBehavior, Object document, DocumentMap mapping, DataModification dataModification, String prefix)
   at Nevermore.Util.DataModificationQueryBuilder.GetDocumentParameters(Func`2 allocateId, Object customAssignedId, IReadOnlyList`1 documents, DocumentMap mapping, DataModification dataModification)
   at Nevermore.Util.DataModificationQueryBuilder.PrepareInsert(IReadOnlyList`1 documents, InsertOptions options)
   at Nevermore.Advanced.WriteTransaction.Insert[TDocument](TDocument document, InsertOptions options)
   at Octopus.Server.Web.Api.Actions.Certificates.CertificateReplaceAction.ExecuteRegistered(String id) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Api\Actions\Certificates\CertificateReplaceAction.cs:line 66
   at Octopus.Server.Web.Infrastructure.Api.CustomActionWithIdResponder`1.ExecuteRegistered() in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Infrastructure\Api\CustomActionResponder.cs:line 46
   at Octopus.Server.Web.Infrastructure.Api.CustomResponder`1.Respond(TDescriptor options, NancyContext context) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Infrastructure\Api\CustomResponder.cs:line 288
   at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass14_0.<get_Routes>b__1(Object o, CancellationToken x) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Infrastructure\OctopusNancyModule.cs:line 82
   at Nancy.Routing.Route`1.Invoke(DynamicDictionary parameters, CancellationToken cancellationToken)
   at Nancy.Routing.DefaultRouteInvoker.Invoke(Route route, CancellationToken cancellationToken, DynamicDictionary parameters, NancyContext context)
   at Nancy.Routing.DefaultRequestDispatcher.Dispatch(NancyContext context, CancellationToken cancellationToken)
   at Nancy.NancyEngine.InvokeRequestLifeCycle(NancyContext context, CancellationToken cancellationToken, IPipelines pipelines)
   at Octopus.NancyOwin.NancyMiddleware.RequestComplete(NancyContext context, IDictionary`2 environment, CancellationToken cancellationToken) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Owin\NancyMiddleware.cs:line 171
   at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Owin\NancyMiddleware.cs:line 111
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.HttpOverrides.HttpMethodOverrideMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\OctopusClientOldVersionWarningMiddleware.cs:line 38
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\DynamicContentHeadersMiddleware.cs:line 48
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\MaintenanceModeMiddleware.cs:line 55
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\OctopusAuthenticationMiddleware.cs:line 67
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.Middleware.RequestLoggerMiddleware.InvokeAsync(HttpContext context) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\RequestLoggerMiddleware.cs:line 42
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\TelemetryMiddleware.cs:line 74
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\ErrorHandlingMiddleware.cs:line 100

Also because of this seemingly invalid certificate, the deployment releases page shows “Specified argument was out of the range of valid values.” without any context, so users will be left wondering what that is about.

garrett.dass · 13 September 2021 17:56

Hi @mauricio,

Thanks for contacting Octopus Deploy and I am sorry to see you’re having issues with your certificate.

Just a few clarifying questions if you don’t mind:

Which kind of certificate format are you uploading?
Am I correct that you are using the Replace option? Was the certificate that was there previously working properly? Are you able to revert back to the Archive without error?
Are you able to upload this as a new certificate instead of replacing it without issue?

Looking forward to hearing back.

Regards,
Garrett

mauricio · 13 September 2021 18:03

Hi Garrett,

Which kind of certificate format are you uploading?

PEM

Am I correct that you are using the Replace option?

Yes

Was the certificate that was there previously working properly? Are you able to revert back to the Archive without error?

Yes, though it was copied from a previous version of Octopus. I’ve just now compared the json payloads in the database against a new certificate and the CertificateMetadata property is missing, which is probably causing this. Probably a bit of an edge case but still I think a replace should work?

Are you able to upload this as a new certificate instead of replacing it without issue?

yes

Kind regards,
Mauricio

garrett.dass · 13 September 2021 19:37

Hi @mauricio,

Thanks for getting back to me so quickly.

I was speaking with the team and it does look like the CertificateMataData being missing would cause this. I was wondering which version you upgraded from?

Unfortunately, I’m not sure if the CertificateMetaData field was not required at some point and is now or if this might be some kind of data integrity error in the DB.

I am going to reach out to our engineers and see if the Replace behavior should work in this case, it does feel like it should.

For now, if possible, it would probably be best to upload into a New certificate and re-link to your Projects instead of replacing the certificate as the property field will be rebuilt with the correct values.

Please let me know if that works for you and I will keep you updated with anything I find.

Regards,
Garrett

mauricio · 13 September 2021 20:36

Hi Garrett,

I was wondering which version you upgraded from?

2018.something . Pretty old.

For now, if possible, it would probably be best to upload into a New certificate and re-link to your Projects instead of replacing the certificate as the property field will be rebuilt with the correct values.

Yep I’ve updated the json column directly in the database from a new entry and it works, so I have a workaround, cheers.

garrett.dass · 13 September 2021 21:05

Hi @mauricio,

I’m glad to see you got your workaround. I suspect upgrading from 2018 is likely the cause here, there are quite a few changes to how certificates are handled between 2018 and 2021.2, though I’m unable to narrow it down to anything in particular.

Please let us know if you have anymore questions or issues.

Regards,
Garrett

system · 14 October 2021 21:06

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.