Modification failed for 'ServerTask' document

alexander.rystrom · 27 June 2023 18:33

We have a “DeployAll” deployment that deploys 7 other deployments. It’s been working for months without issue. All of sudden today the error below is being return for one or more of the 7 deployment and showing the DeployAll deployment as failed, but looking into the failed deployment there aren’t any issues with it and it itself says it deployed fine and it deployed fine

                    |   == Failed: Step 5: Deploy MyApplication ==
18:13:15   Fatal    |     The step failed: Activity Deploy MyApplication on a Worker failed with error 'Modification failed for 'ServerTask' document with 'ServerTasks-9249' Id because submitted data was out of date. Refresh the document and try again.'.
18:13:15   Verbose  |     Deploy MyApplication completed
                    |   
                    |     == Failed: Worker ==
18:13:13   Verbose  |       Octopus Server version: 2023.3.4135
18:13:13   Verbose  |       Environment Information:
                    |       IsRunningInContainer: True
                    |       OperatingSystem: Linux 5.4.0-1105-azure #111~18.04.1-Ubuntu SMP Fri Mar 3 22:47:43 UTC 2023
                    |       OsBitVersion: x64
                    |       Is64BitProcess: True
                    |       CurrentUser: root
                    |       MachineName: octopus-i056623-5d889b5f48-bcmlf
                    |       ProcessorCount: 2
                    |       CurrentDirectory: /Octopus
                    |       TempDirectory: /tmp/
                    |       HostProcessName: Octopus.Server
                    |       PID: 1
18:13:14   Verbose  |       Creating deployment for release 2023.6.27-qa2+50849 of project 'MyApplication'
18:13:14   Info     |       Deploying release 2023.6.27-qa2+50849 of project 'MyApplication'. [View deployment](~/app#/Spaces-1/deployments/Deployments-2623)
18:13:15   Verbose  |       Updating manifest with output variables
18:13:15   Verbose  |       Updating manifest with action evaluated variables
18:13:15   Verbose  |       Modification failed for 'ServerTask' document with 'ServerTasks-9249' Id because submitted data was out of date. Refresh the document and try again.
                    |       Nevermore.StaleDataException
                    |       at Nevermore.Advanced.WriteTransaction.ApplyNewRowVersionIfRequired[TDocument](TDocument document, DocumentMap mapping, DataModificationOutput output)
                    |       at Nevermore.Advanced.WriteTransaction.UpdateAsync[TDocument](TDocument document, UpdateOptions options, CancellationToken cancellationToken)
                    |       at Octopus.Core.RelationalStorage.RawRelationalTransaction.Update[TKey](IId`1 document, CancellationToken cancellationToken, UpdateOptions options) in RawRelationalTransaction.cs:line 160
                    |       at Octopus.Server.Web.Infrastructure.OctopusQueryExecutor.Update[TKey](IId`1 instance, CancellationToken cancellationToken, UpdateOptions options) in OctopusQueryExecutor.cs:line 326
                    |       at Octopus.Core.Persistence.Database.ProjectPathDecorators.ProjectPathQueryExecutorDecorator.Update[TKey](IId`1 document, CancellationToken cancellationToken, UpdateOptions options) in ProjectPathQueryExecutorDecorator.cs:line 159
                    |       at Octopus.Core.Persistence.Database.SlugDecorators.SlugQueryExecutorDecorator.Update[TKey](IId`1 document, CancellationToken cancellationToken, UpdateOptions options) in SlugQueryExecutorDecorator.cs:line 156
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.<>c__DisplayClass22_0.<UpdateChildTaskCountWithRetry in DeployReleaseActionHandler.cs:line 403
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass4_0`1.<Execute in UnitOfWorkExecutor.cs:line 55
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 58
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 58
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T](Func`3 action, CancellationToken cancellationToken, String name) in UnitOfWorkExecutor.cs:line 58
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.<>c__DisplayClass22_0.<UpdateChildTaskCountWithRetry in DeployReleaseActionHandler.cs:line 405
                    |       at Polly.AsyncPolicy.<>c__DisplayClass40_0.<ImplementationAsync
                    |       at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
                    |       at Polly.AsyncPolicy.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.UpdateChildTaskCountWithRetry(String taskId, Int32 delta, ITaskLog taskLog, CancellationToken cancellationToken) in DeployReleaseActionHandler.cs:line 406
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.UpdateChildTaskCount(String taskId, Int32 delta, ITaskLog taskLog, CancellationToken cancellationToken) in DeployReleaseActionHandler.cs:line 389
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.PollTaskUntilComplete(ActionCommand command, DeploymentId deploymentId, String taskId, String releaseDescription, ITaskLog taskLog, CancellationToken cancellationToken) in DeployReleaseActionHandler.cs:line 321
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.PollTaskUntilComplete(ActionCommand command, DeploymentId deploymentId, String taskId, String releaseDescription, ITaskLog taskLog, CancellationToken cancellationToken) in DeployReleaseActionHandler.cs:line 375
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.Execute(ActionCommand command, DeploymentTarget machine, ITaskLog taskLog, CancellationToken cancellationToken) in DeployReleaseActionHandler.cs:line 264
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.Execute(ActionCommand command, DeploymentTarget machine, ITaskLog taskLog, CancellationToken cancellationToken) in DeployReleaseActionHandler.cs:line 274
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.Execute(ActionCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in DeployReleaseActionHandler.cs:line 105
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.OldActionDispatcher.<>c__DisplayClass4_0.<DispatchServerAction in OldActionDispatcher.cs:line 49
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken) in Guidance.cs:line 148
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Action callbackOnExclude, CancellationToken cancellationToken) in Guidance.cs:line 78
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.OldActionDispatcher.DispatchServerAction(ActionCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in OldActionDispatcher.cs:line 54
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.OldActionDispatcher.Dispatch(ActionCommand command, DeploymentTarget deploymentTarget, ITaskLog taskLog, CancellationToken cancellationToken) in OldActionDispatcher.cs:line 31
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.Dispatch(ActionCommand command, DeploymentTarget deploymentTarget, ITaskLog taskLogForTarget, ITaskLog taskLogRoot, IActionHandlerResolver actionHandlerResolver, Action guidanceExcludeCallback, CancellationToken cancellationToken) in NewActionDispatcher.cs:line 122
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.<>c__DisplayClass8_2.<TryExecuteActionAndInitLoggingContext in ProcessStepController.cs:line 295
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken) in Guidance.cs:line 148
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Action callbackOnExclude, CancellationToken cancellationToken) in Guidance.cs:line 78
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.<>c__DisplayClass8_1.<TryExecuteActionAndInitLoggingContext in ProcessStepController.cs:line 301
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in TransientErrorDetectionExecutor.cs:line 50
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in TransientErrorDetectionExecutor.cs:line 50
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.<>c__DisplayClass8_0.<TryExecuteActionAndInitLoggingContext in ProcessStepController.cs:line 306
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass7_0`4.<Execute in UnitOfWorkExecutor.cs:line 115
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 58
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 58
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3,T4](Func`6 action, CancellationToken cancellationToken, String name) in UnitOfWorkExecutor.cs:line 118
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.TryExecuteActionAndInitLoggingContext(ExecutionPlan plan, ExecutionPlanner planner, PlannedStep step, DeploymentTarget targetContext, PlannedAction action, ITaskLog taskLogForTarget, ITaskLog taskLogRoot, CancellationToken cancellationToken) in ProcessStepController.cs:line 327
18:13:15   Fatal    |       The action Deploy MyApplication on a Worker failed

britton.riggs · 27 June 2023 22:21

Hi @alexander.rystrom,

Thank you for reaching out, and welcome to the Octopus Deploy public community forum!

I’m sorry to hear you are having recent issues with your multi-deployment process in Octopus Deploy, but I would be happy to help take a closer look at things.

As an initial step, in doing some research on this, it seems like this issue might be related to running Octopus Deploy in an HA/multi-node setup - can you let me know if your Octopus Deploy instance is running in a similar setup? This will hopefully help me narrow in on the specifics of the problem, but let me know how you have things set up, and I can try to take a deeper dive into this.

Outside of this, would you be able to upload the complete diagnostics package from your node(s) for me to review (Configuration > Diagnostics via the Web UI on a particular node)? This package includes information about your Octopus Deploy server configuration and helpful diagnostic outputs that might expose more information about the issue you are experiencing.

Here is a secure link for uploading supporting files (such as the diagnostics package), so you don’t have to post this data to the public forum.

Looking forward to hearing back from you,

Britton

britton.riggs · 27 June 2023 22:52

Hi @alexander.rystrom,

Apologies for the double-post, but I noticed that you are using Octopus Deploy version 2023.3.4135 based on the log snippet you shared, so I’m assuming you are using a hosted version of Octopus Deploy (Octopus Deploy Cloud) - this version of Octopus Deploy does not include a diagnostics section (like I mentioned in my previous reply).

If that’s the case, would you mind if our team logged in to look at this issue directly? This access will be read-only and for investigative purposes only. If so, could you provide your instance URL (in the form *.octopus.app), a link to the central deployment project, and a link to the task log where this snippet came from? This will help our engineers in taking a deeper dive into this.

Sorry for the initial confusion, but let me know if it’s alright to log into your instance to take a look at this, and also feel free to let me know if you have any other questions regarding the information I’ve already provided.

Regards,

Britton

alexander.rystrom · 28 June 2023 11:52

Britton,

Go for it. Here’s the url: …

From the dashboard, it’s the “Deploy All” project. It deploys the other projects listed there. In each case the DeployAll can be displaying a different project or project(s) as the one(s) that failed, but when you go to look at that project it deployed successfully.

dom.richardson · 28 June 2023 13:38

Hi @alexander.rystrom

Our engineers have got back to use with a fix for this issue by disabling a recently added feature.
This will require restarting the Octopus Cloud instance to take effect.

We can either do this in your outage window or we can do this now.

Please let me know which you prefer.

Kind Regards,
Dom.

alexander.rystrom · 28 June 2023 13:54

You can restart it now. Thank you.

dom.richardson · 28 June 2023 14:30

Hi @alexander.rystrom

The instance has been restarted and the feature has been disabled successfully.

Please give this a try and let us know how that goes.

Kind Regards,
Dom.

alexander.rystrom · 28 June 2023 14:56

It appears to be working now. Thank you!

system · 29 July 2023 14:56

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.