System Exception "Network"

ouf.qatu · 26 April 2023 07:46

Hi,

We recently have this weird issue which is making our Octopus server useless. Whenever we try to create new process on any of the current projects we’ll get the following error :

We have Octopus server deployed to our EKS since months and this happened without doing any changes (as it was working flawlessly). We tried the following:

*Upgraded to the latest version
*Tried to access Octopus directly on it’s own IP (ELB and Ingress were taken out of the request way)
*Tried to login using local admin account will get us the same error on network

here’s the related logs from the pod:

Unhandled error on request: "network"

System.ArgumentException: network

at System.Net.IPNetwork.InternalParse(Boolean tryParse, String network, ICidrGuess cidrGuess, Boolean sanitanize, IPNetwork& ipnetwork)

at System.Net.IPNetwork.Parse(String network, ICidrGuess cidrGuess)

at Octopus.Server.Orchestration.Logging.IpNetworkChecker.<>c.<IsIpAddressInRangeOfAnyIpNetworks>b__0_0(String ip) in ./source/Octopus.Server/Orchestration/Logging/IpNetworkChecker.cs:line 10

at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext()

at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)

at Octopus.Server.Orchestration.Logging.IpNetworkChecker.IsIpAddressInRangeOfAnyIpNetworks(IPAddress ipAddress, IEnumerable`1 ipNetworks) in ./source/Octopus.Server/Orchestration/Logging/IpNetworkChecker.cs:line 10

at Octopus.Server.Orchestration.Logging.OriginalClientIpAddressRetriever.Get(HttpContext httpContext) in ./source/Octopus.Server/Orchestration/Logging/OriginalClientIpAddressRetriever.cs:line 26

at Octopus.Server.AutomationContexts.ApiAutomationContext.GetClientIpAddress() in ./source/Octopus.Server/AutomationContexts/ApiAutomationContext.cs:line 30

at Octopus.Core.Auditing.EventStore.CaptureEventAutomationContext(Event event) in ./source/Octopus.Core/Auditing/EventStore.cs:line 311

at Octopus.Core.Auditing.EventStore.Store[TDocument,TKey](IRawWriteQueryExecutor session, Event event, TDocument model, CancellationToken cancellationToken) in ./source/Octopus.Core/Auditing/EventStore.cs:line 125

at Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2.StoreEvent(Event event, TDocument document, IOctopusQueryExecutor queryExecutor, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Auditing/AuditingDocumentStoreDecorator.cs:line 310

at Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2.<>c__DisplayClass16_0.<<Update>b__1>d.MoveNext() in ./source/Octopus.Core/Persistence/Auditing/AuditingDocumentStoreDecorator.cs:line 107

--- End of stack trace from previous location ---

at Octopus.Core.Extensions.EnumerableExtensionMethods.Do[T](IAsyncEnumerable`1 items, Func`3 action, CancellationToken cancellationToken)+MoveNext() in ./source/Octopus.Core/Extensions/EnumerableExtensionMethods.cs:line 117

at Octopus.Core.Extensions.EnumerableExtensionMethods.Do[T](IAsyncEnumerable`1 items, Func`3 action, CancellationToken cancellationToken)+MoveNext() in ./source/Octopus.Core/Extensions/EnumerableExtensionMethods.cs:line 115

at Octopus.Core.Extensions.EnumerableExtensionMethods.Do[T](IAsyncEnumerable`1 items, Func`3 action, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()

at Octopus.Core.Extensions.EnumerableExtensionMethods.Done[T](IAsyncEnumerable`1 items, CancellationToken cancellationToken) in ./source/Octopus.Core/Extensions/EnumerableExtensionMethods.cs:line 226

at Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2.Update(TDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Auditing/AuditingDocumentStoreDecorator.cs:line 109

at Octopus.Core.Persistence.Database.Deletion.DeleteRelatedDocumentsDocumentStoreDecorator`2.Update(TDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Database/Deletion/DeleteRelatedDocumentsDocumentStoreDecorator.cs:line 42

at Octopus.Core.Persistence.Database.Deletion.VetoDocumentStoreDecorator`2.Update(TDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Database/Deletion/VetoDocumentStoreDecorator.cs:line 41

at Octopus.Core.Persistence.Database.AccessCheckingDocumentStoreDecorator`2.Update(TDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Database/AccessCheckingDocumentStoreDecorator.cs:line 96

at Octopus.Core.Persistence.DocumentStore`2.Update(TDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/DocumentStore.cs:line 94

at Octopus.Core.Model.Projects.Updaters.HealChannelVersionRulePackages.HealPackages(DeploymentAction newAction, DeploymentAction oldAction, ProjectId projectId, CancellationToken cancellationToken) in ./source/Octopus.Core/Model/Projects/Updaters/HealChannelVersionRulePackages.cs:line 69

at Octopus.Core.Model.Projects.Updaters.HealChannelVersionRulePackages.Handle(DeploymentProcessStepActionNameChangedV2Event e, CancellationToken cancellationToken) in ./source/Octopus.Core/Model/Projects/Updaters/HealChannelVersionRulePackages.cs:line 35

at Octopus.Core.Infrastructure.Mediator.AutofacMediator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/AutofacMediator.cs:line 59

at Octopus.Core.Infrastructure.Mediator.Decorators.SystemComponentModelValidationDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/SystemComponentModelValidationDecorator.cs:line 92

at Octopus.Core.Infrastructure.Mediator.Decorators.FluentValidationsDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/FluentValidationsDecorator.cs:line 120

at Octopus.Core.Infrastructure.Mediator.Decorators.MessageBusSiphoningDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/MessageBusSiphoningDecorator.cs:line 62

at Octopus.Core.Model.Projects.Updaters.RaiseDeploymentStepChangedEvents.Handle(DeploymentProcessChangedV2Event e, CancellationToken cancellationToken) in ./source/Octopus.Core/Model/Projects/Updaters/RaiseDeploymentStepChangedEvents.cs:line 37

at Octopus.Core.Infrastructure.Mediator.AutofacMediator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/AutofacMediator.cs:line 59

at Octopus.Core.Infrastructure.Mediator.Decorators.SystemComponentModelValidationDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/SystemComponentModelValidationDecorator.cs:line 92

at Octopus.Core.Infrastructure.Mediator.Decorators.FluentValidationsDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/FluentValidationsDecorator.cs:line 120

at Octopus.Core.Infrastructure.Mediator.Decorators.MessageBusSiphoningDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/MessageBusSiphoningDecorator.cs:line 62

at Octopus.Core.Model.Projects.Updaters.ProcessUpdater`2.UpdateProcess(TProcess process, TResource resource, Project project, ModifyEventCollection modifyEvents, CancellationToken cancellationToken) in ./source/Octopus.Core/Model/Projects/Updaters/ProcessUpdater.cs:line 55

at Octopus.Core.Model.Projects.Updaters.ProcessUpdaterAndValidator`2.UpdateAndValidate(TProcess process, TResource resource, Project project, CancellationToken cancellationToken) in ./source/Octopus.Core/Model/Projects/Updaters/ProcessUpdaterAndValidator.cs:line 42

at Octopus.Core.Features.DeploymentProcesses.Ocl.ModifyDeploymentProcessCommandHandler.Handle(ModifyDeploymentProcessCommand command, CancellationToken cancellationToken) in ./source/Octopus.Core/Features/DeploymentProcesses/Ocl/ModifyDeploymentProcessCommandHandler.cs:line 73

at Octopus.Core.Infrastructure.Mediator.AutofacMediator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/AutofacMediator.cs:line 26

at Octopus.Core.Infrastructure.Mediator.Decorators.SystemComponentModelValidationDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/SystemComponentModelValidationDecorator.cs:line 28

at Octopus.Core.Infrastructure.Mediator.Decorators.FluentValidationsDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/FluentValidationsDecorator.cs:line 35

at Octopus.Core.Infrastructure.Mediator.Decorators.MessageBusSiphoningDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/MessageBusSiphoningDecorator.cs:line 32

at Octopus.Server.Web.Controllers.DeploymentProcesses.ModifyDeploymentProcessController.ModifyDeploymentProcess(ModifyDeploymentProcessCommand command, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Controllers/DeploymentProcesses/ModifyDeploymentProcessController.cs:line 121

at lambda_method213561(Closure , Object )

at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)

at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)

at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)

at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)

at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)

at Octopus.Server.Web.Middleware.BoundaryTrailerRewriteMiddleware.Invoke(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/BoundaryTrailerRewriteMiddleware.cs:line 44

at Octopus.Server.Web.Infrastructure.Authentication.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) in ./source/Octopus.Server/Web/Infrastructure/Authentication/AuthorizationMiddlewareResultHandler.cs:line 50

at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)

at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 46

at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 46

at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 52

at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 48

at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 59

at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, IUserAuthenticator userAuthenticator, IAuthCookieService authCookieService, IWebAuthCache authCache, ILogger logger) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 58

at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)

at Octopus.Server.Web.Middleware.LegacyRequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/LegacyRequestLoggerMiddleware.cs:line 41

at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 76

at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 117

"HTTP" "PUT" to "xyz.xyz.com""/api/Spaces-1/projects/Projects-42/deploymentprocesses" "completed" with 400 in 00:00:00.1097854 (109ms) by "Super Man"

System.ArgumentException: network

at System.Net.IPNetwork.InternalParse(Boolean tryParse, String network, ICidrGuess cidrGuess, Boolean sanitanize, IPNetwork& ipnetwork)

at System.Net.IPNetwork.Parse(String network, ICidrGuess cidrGuess)

at Octopus.Server.Orchestration.Logging.IpNetworkChecker.<>c.<IsIpAddressInRangeOfAnyIpNetworks>b__0_0(String ip) in ./source/Octopus.Server/Orchestration/Logging/IpNetworkChecker.cs:line 10

at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext()

at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)

at Octopus.Server.Orchestration.Logging.IpNetworkChecker.IsIpAddressInRangeOfAnyIpNetworks(IPAddress ipAddress, IEnumerable`1 ipNetworks) in ./source/Octopus.Server/Orchestration/Logging/IpNetworkChecker.cs:line 10

at Octopus.Server.Orchestration.Logging.OriginalClientIpAddressRetriever.Get(HttpContext httpContext) in ./source/Octopus.Server/Orchestration/Logging/OriginalClientIpAddressRetriever.cs:line 26

at Octopus.Server.AutomationContexts.ApiAutomationContext.GetClientIpAddress() in ./source/Octopus.Server/AutomationContexts/ApiAutomationContext.cs:line 30

at Octopus.Core.Auditing.EventStore.CaptureEventAutomationContext(Event event) in ./source/Octopus.Core/Auditing/EventStore.cs:line 311

at Octopus.Core.Auditing.EventStore.Store[TDocument,TKey](IRawWriteQueryExecutor session, Event event, TDocument model, CancellationToken cancellationToken) in ./source/Octopus.Core/Auditing/EventStore.cs:line 125

at Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2.StoreEvent(Event event, TDocument document, IOctopusQueryExecutor queryExecutor, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Auditing/AuditingDocumentStoreDecorator.cs:line 310

at Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2.<>c__DisplayClass16_0.<<Update>b__1>d.MoveNext() in ./source/Octopus.Core/Persistence/Auditing/AuditingDocumentStoreDecorator.cs:line 107

--- End of stack trace from previous location ---

at Octopus.Core.Extensions.EnumerableExtensionMethods.Do[T](IAsyncEnumerable`1 items, Func`3 action, CancellationToken cancellationToken)+MoveNext() in ./source/Octopus.Core/Extensions/EnumerableExtensionMethods.cs:line 117

at Octopus.Core.Extensions.EnumerableExtensionMethods.Do[T](IAsyncEnumerable`1 items, Func`3 action, CancellationToken cancellationToken)+MoveNext() in ./source/Octopus.Core/Extensions/EnumerableExtensionMethods.cs:line 115

at Octopus.Core.Extensions.EnumerableExtensionMethods.Do[T](IAsyncEnumerable`1 items, Func`3 action, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()

at Octopus.Core.Extensions.EnumerableExtensionMethods.Done[T](IAsyncEnumerable`1 items, CancellationToken cancellationToken) in ./source/Octopus.Core/Extensions/EnumerableExtensionMethods.cs:line 226

at Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2.Update(TDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Auditing/AuditingDocumentStoreDecorator.cs:line 109

at Octopus.Core.Persistence.Database.Deletion.DeleteRelatedDocumentsDocumentStoreDecorator`2.Update(TDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Database/Deletion/DeleteRelatedDocumentsDocumentStoreDecorator.cs:line 42

at Octopus.Core.Persistence.Database.Deletion.VetoDocumentStoreDecorator`2.Update(TDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Database/Deletion/VetoDocumentStoreDecorator.cs:line 41

at Octopus.Core.Persistence.Database.AccessCheckingDocumentStoreDecorator`2.Update(TDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Database/AccessCheckingDocumentStoreDecorator.cs:line 96

at Octopus.Core.Persistence.DocumentStore`2.Update(TDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/DocumentStore.cs:line 94

at Octopus.Core.Model.Projects.Updaters.HealChannelVersionRulePackages.HealPackages(DeploymentAction newAction, DeploymentAction oldAction, ProjectId projectId, CancellationToken cancellationToken) in ./source/Octopus.Core/Model/Projects/Updaters/HealChannelVersionRulePackages.cs:line 69

at Octopus.Core.Model.Projects.Updaters.HealChannelVersionRulePackages.Handle(DeploymentProcessStepActionNameChangedV2Event e, CancellationToken cancellationToken) in ./source/Octopus.Core/Model/Projects/Updaters/HealChannelVersionRulePackages.cs:line 35

at Octopus.Core.Infrastructure.Mediator.AutofacMediator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/AutofacMediator.cs:line 59

at Octopus.Core.Infrastructure.Mediator.Decorators.SystemComponentModelValidationDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/SystemComponentModelValidationDecorator.cs:line 92

at Octopus.Core.Infrastructure.Mediator.Decorators.FluentValidationsDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/FluentValidationsDecorator.cs:line 120

at Octopus.Core.Infrastructure.Mediator.Decorators.MessageBusSiphoningDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/MessageBusSiphoningDecorator.cs:line 62

at Octopus.Core.Model.Projects.Updaters.RaiseDeploymentStepChangedEvents.Handle(DeploymentProcessChangedV2Event e, CancellationToken cancellationToken) in ./source/Octopus.Core/Model/Projects/Updaters/RaiseDeploymentStepChangedEvents.cs:line 37

at Octopus.Core.Infrastructure.Mediator.AutofacMediator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/AutofacMediator.cs:line 59

at Octopus.Core.Infrastructure.Mediator.Decorators.SystemComponentModelValidationDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/SystemComponentModelValidationDecorator.cs:line 92

at Octopus.Core.Infrastructure.Mediator.Decorators.FluentValidationsDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/FluentValidationsDecorator.cs:line 120

at Octopus.Core.Infrastructure.Mediator.Decorators.MessageBusSiphoningDecorator.Publish[TEvent](TEvent event, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/MessageBusSiphoningDecorator.cs:line 62

at Octopus.Core.Model.Projects.Updaters.ProcessUpdater`2.UpdateProcess(TProcess process, TResource resource, Project project, ModifyEventCollection modifyEvents, CancellationToken cancellationToken) in ./source/Octopus.Core/Model/Projects/Updaters/ProcessUpdater.cs:line 55

at Octopus.Core.Model.Projects.Updaters.ProcessUpdaterAndValidator`2.UpdateAndValidate(TProcess process, TResource resource, Project project, CancellationToken cancellationToken) in ./source/Octopus.Core/Model/Projects/Updaters/ProcessUpdaterAndValidator.cs:line 42

at Octopus.Core.Features.DeploymentProcesses.Ocl.ModifyDeploymentProcessCommandHandler.Handle(ModifyDeploymentProcessCommand command, CancellationToken cancellationToken) in ./source/Octopus.Core/Features/DeploymentProcesses/Ocl/ModifyDeploymentProcessCommandHandler.cs:line 73

at Octopus.Core.Infrastructure.Mediator.AutofacMediator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/AutofacMediator.cs:line 26

at Octopus.Core.Infrastructure.Mediator.Decorators.SystemComponentModelValidationDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/SystemComponentModelValidationDecorator.cs:line 28

at Octopus.Core.Infrastructure.Mediator.Decorators.FluentValidationsDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/FluentValidationsDecorator.cs:line 35

at Octopus.Core.Infrastructure.Mediator.Decorators.MessageBusSiphoningDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/MessageBusSiphoningDecorator.cs:line 32

at Octopus.Server.Web.Controllers.DeploymentProcesses.ModifyDeploymentProcessController.ModifyDeploymentProcess(ModifyDeploymentProcessCommand command, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Controllers/DeploymentProcesses/ModifyDeploymentProcessController.cs:line 121

at lambda_method213561(Closure , Object )

at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)

at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)

at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)

at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)

at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)

at Octopus.Server.Web.Middleware.BoundaryTrailerRewriteMiddleware.Invoke(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/BoundaryTrailerRewriteMiddleware.cs:line 44

at Octopus.Server.Web.Infrastructure.Authentication.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) in ./source/Octopus.Server/Web/Infrastructure/Authentication/AuthorizationMiddlewareResultHandler.cs:line 50

at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)

at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 46

at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 46

at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 52

at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 48

at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 59

at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, IUserAuthenticator userAuthenticator, IAuthCookieService authCookieService, IWebAuthCache authCache, ILogger logger) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 58

at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)

at Octopus.Server.Web.Middleware.LegacyRequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/LegacyRequestLoggerMiddleware.cs:line 41

at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 76

at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 117

"HTTP" "GET" to "xyz.xyz.com""/api/Spaces-1/projects/Projects-42/summary" "completed" with 200 in 00:00:00.0169091 (16ms) by "Super Man"

"HTTP" "POST" to "xyz.xyz.com""/api/telemetry/process" "completed" with 200 in 00:00:00.0278379 (27ms) by "Super Man"

"HTTP" "POST" to "xyz.xyz.com""/bff/telemetry/recordPortalOperation" "completed" with 200 in 00:00:00.0073732 (7ms) by "Super Man"

"HTTP" "POST" to "xyz.xyz.com""/api/telemetry/process" "completed" with 200 in 00:00:00.0345229 (34ms) by "Super Man"

donny.bell · 26 April 2023 10:27

Hi @ouf.qatu,

Sorry to see that you’ve run into this issue.

This doesn’t appear to be an error we’ve run into previously. However, in looking at the stack trace in the logs you provided, it looks like there may be an issue with either the Octopus Server container’s ability to reach the Octopus SQL DB and/or the persistent storage volume.

Can you confirm if the Octopus SQL DB and the persistent storage volume are reachable? It may also be worth restarting the Octopus Server instance if you haven’t already, as well.

Let me know regarding the above at your earliest convenience.

Best Regards,
Donny Bell

ouf.qatu · 26 April 2023 12:31

Hi @donny.bell ,

Thanks a lot for your reply.

I’ve restarted the underlysing SQL Server and Octopus server with no luck. And here’s the startup logs:

======== Writing Configuration File =======

mount: /sys/kernel/security: permission denied.
Could not mount /sys/kernel/security.
AppArmor detection and --privileged mode might break.
mount: /tmp: permission denied.
Creating empty configuration file: /root/.octopus/OctopusServer/Server/Server.linux.config
Setting home directory to: /root/.octopus/OctopusServer/Server
Saving instance: OctopusServer
Setting server node name to 'octopus-0'.

======== Initializing DB ========
Using supplied Master Key
Setting master key
Setting database connection string to: "server=octopod-database.ckblvtomekir.eu-central-1.rds.amazonaws.com,1433;database=Octopus;user id=admin;password=********"
========== SQL Database Connection Details ========
"SQL Server: octopod-database.ckblvtomekir.eu-central-1.rds.amazonaws.com,1433
SQL Database: Octopus
Authentication Type: SQL Server Credentials
User: admin
"
===================================================
Testing connection to the 'master' database...
Successfully connected to 'master' database!
Database "Octopus" already exists on SQL Server at "octopod-database.ckblvtomekir.eu-central-1.rds.amazonaws.com,1433", no need to create it.
Checking the Octopus Master Key has been configured.
Making sure it's safe to upgrade the database schema...
Ensuring pre-conditions for upgrading the database are satisfied...
Searching for indexes that might upset the database upgrade process...
- "PASS": "All columns use the default collation."
- "PASS": "Skipping the license compliance check since the database already has the expected schema. No changes are required."
- "PASS": "The version of your SQL Server satisfies Octopus Server installation requirements."
- "PASS": "There aren't any unexpected indexes in the database."
Executing always run pre scripts...
Executing TSQL Database Server script 'Script0000 - Set highest available compatibility level.sql'
Current COMPATIBILITY_LEVEL for Octopus is set to 150
Ensuring COMPATIBILITY_LEVEL for Octopus is set to 150
COMPATIBILITY_LEVEL for Octopus is already 150 or higher
Executing TSQL Database Server script 'Script0001 - Remove Namespace Prefix From Script Names.sql'
Preparing to remove namespace prefix from script names in the [SchemaVersions] table...
Table [SchemaVersions] exists, inspecting script names...
Script names are not prefixed, nothing to do.
Always-run pre scripts "succeeded" after 023ms.
Checking database schema upgrade history...
Checking to see if database schema upgrade is required...
Database already has the expected schema. No changes are required.
Executing always run post scripts...
Executing TSQL Database Server script 'Script0000 - Refresh Views.sql'
Refreshing view dbo.Dashboard
Refreshing view dbo.IdsInUse
Refreshing view dbo.MultiTenancyDashboard
Refreshing view dbo.Release_WithDeploymentProcess
Refreshing view dbo.RunbookSnapshot_WithRunbookProcess
Always-run post scripts "succeeded" after 094ms.
Database upgrade "succeeded" after 528ms.

These changes require a restart of the "OctopusServer".

donny.bell · 26 April 2023 13:02

Hi @ouf.qatu,

Thank you for getting back to me.

Just to confirm, this is only affecting saving changes to a Project’s process? Can you confirm if affected Projects are Git-enabled?

Let me know at your earliest convenience.

Best Regards,
Donny Bell

ouf.qatu · 26 April 2023 13:15

Hi @donny.bell ,

Actually it’s affecting the whole application.
Adding/editing projects, process steps even logging in using local account will produce the same issue.
All projects are not Git enabled.

I’m available till 6 PM CET.

sean.stanway · 26 April 2023 15:15

Hi @ouf.qatu

Can I ask if you have only IPv6 enabled on your server?
Can you also check if you have anything under Configuration > Settings > Web Portal > Trusted Proxies?

One of the changes that was recently added was to add the IP address of the calling task to the logs, which it looks like it the problem that you’re having. It’s attempting to compare the current IP address to a list of trusted proxies. It currently only looks to parse IPv4 addresses, so it may be that if IPv6 is enabled then this won’t work.

I’ll send off a message and we should have a reply tomorrow morning (our developers are Australian-based).

Kind Regards
Sean

ouf.qatu · 26 April 2023 16:03

Hi @sean.stanway ,

Thanks for your reply.

The issue was resolved after changing trusted proxies value from localhost:7007 to our subnet. Please note that we had to do that by updating the value directly on the SQL database as it was refusing any updates using the UI.

sean.stanway · 27 April 2023 07:19

Hi @ouf.qatu

I’m glad to hear that you managed to resolve this.

Just to update you; our engineering team was already aware of this particular exception and they are now backporting a fix to help resolve this issue for future on-premise upgrades. This should hopefully not happen again.

Have a good day!
Kind Regards
Sean

system · 28 May 2023 07:19

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.