Hey @mikepower79,
I took another look at your server logs, specifically for ServerTasks-3366253 as that is the task you are seeing as still showing as not completed and there are a few errors relating to what I think is the task log, this does point to permissions on the storage. I am unsure on what the first error means so I can get that in front of our engineers but it does look like you may need to take a look at your storage permissions as Octopus is struggling to write to task logs:
Access to the path '/taskLogs/servertasks-3366253_xfxfrqu8t9.txt' is denied.
System.ArgumentException: CorrelationId ServerTasks-3366253_XFXFRQU8T9 is not currently active
at Octopus.Core.Diagnostics.TaskLog.TaskLogFactory.Get(CorrelationId correlationId) in ./source/Octopus.Core/Diagnostics/TaskLog.cs:line 377
at Octopus.Server.Extensibility.ITSM.Core.Precondition.ChangeControlledDeploymentStartHandlerBase.Handle(DeploymentEvent e, CancellationToken cancellationToken) in ./source/Octopus.Server.Extensibility.ITSM.Core/Precondition/ChangeControlledDeploymentStartHandlerBase.cs:line 53
at Octopus.Core.Infrastructure.MessageBus.Middleware.UnitOfWorkMiddleware.Invoke(MessageContext context, Func`2 next, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/Middleware/UnitOfWorkMiddleware.cs:line 29
at Octopus.Core.Infrastructure.MessageBus.Middleware.UnitOfWorkMiddleware.Invoke(MessageContext context, Func`2 next, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/Middleware/UnitOfWorkMiddleware.cs:line 29
at Octopus.Core.Infrastructure.MessageBus.Middleware.LoggingMiddleware.Invoke(MessageContext context, Func`2 next, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/Middleware/LoggingMiddleware.cs:line 39
at Octopus.Core.Infrastructure.MessageBus.Middleware.CorrelationIdMiddleware.Invoke(MessageContext context, Func`2 next, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/Middleware/CorrelationIdMiddleware.cs:line 33
at Octopus.Core.Infrastructure.MessageBus.PipelineFactory.BuildAndRun[TEvent](Int64 sequenceNumber, TEvent e, IDictionary`2 metadata, String consumerGroupId, Type handlerType, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/PipelineFactory.cs:line 51
at Octopus.Core.Infrastructure.MessageBus.MessagePump.Dispatch[TEvent](Int64 sequenceNumber, TEvent e, IDictionary`2 metadata, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/MessagePump.cs:line 135
2023-05-18 10:26:47.6595 1 74 WARN An unhandled exception occurred while dispatching an event via the message bus: "CorrelationId ServerTasks-3366253_XFXFRQU8T9 is not currently active"
System.ArgumentException: CorrelationId ServerTasks-3366253_XFXFRQU8T9 is not currently active
"CorrelationId: "ServerTasks-3366253_XFXFRQU8T9/6ec38cf4a5854df198facc13655a1f3a""
"Message: "Enable Maintenance completed"
System.UnauthorizedAccessException: Access to the path '/taskLogs/servertasks-3366253_xfxfrqu8t9.txt' is denied.
---> System.IO.IOException: Permission denied
--- End of inner exception stack trace ---
"CorrelationId: "ServerTasks-3366253_XFXFRQU8T9/85fe2e16c7224a33aad3a76cf403d11f/60ec2f6113ff4f64bff995305632c6a8/b9ff3e936ae4446580144d9d6066aa37""
"Message: "Calamari Version: 25.3.3"
System.UnauthorizedAccessException: Access to the path '/taskLogs/servertasks-3366253_xfxfrqu8t9.txt' is denied.
---> System.IO.IOException: Permission denied
--- End of inner exception stack trace ---
at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing)
at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
at System.IO.StreamWriter.Dispose(Boolean disposing)
at Octopus.Core.Orchestration.Logging.Processing.AppendToLogFile.InitializeInternal() in ./source/Octopus.Core/Orchestration/Logging/Processing/AppendToLogFile.cs:line 49
at Octopus.Core.Orchestration.Logging.Processing.AppendToLogFile.Process(CorrelationId correlationId, ActivityLogEntry entry) in ./source/Octopus.Core/Orchestration/Logging/Processing/AppendToLogFile.cs:line 71
at Nevermore.Transient.RetryPolicy.<>c__DisplayClass26_0.<ExecuteAction>b__0()
at Nevermore.Transient.RetryPolicy.ExecuteAction[TResult](Func`1 func)
at Nevermore.Transient.RetryPolicy.ExecuteAction(Action action)
at Octopus.Server.Orchestration.Logging.ServerLogWriter.DoSafeAppend(ILogEntryProcessor processor, CorrelationId correlationId, ActivityLogEntry entry) in ./source/Octopus.Server/Orchestration/Logging/ServerLogWriter.cs:line 171
2023-05-18 11:18:19.9081 1 138 ERROR Failed to append the following log entry:"
"CorrelationId: "ServerTasks-3366253_XFXFRQU8T9/85fe2e16c7224a33aad3a76cf403d11f/60ec2f6113ff4f64bff995305632c6a8/b9ff3e936ae4446580144d9d6066aa37""
"Message: "Exit code: 0"
System.UnauthorizedAccessException: Access to the path '/taskLogs/servertasks-3366253_xfxfrqu8t9.txt' is denied.
---> System.IO.IOException: Permission denied
--- End of inner exception stack trace ---
You can see here too its failing to write to the logs:
2023-05-18 11:18:19.9081 1 138 ERROR Failed to append the following log entry:"
"CorrelationId: "ServerTasks-3366253_XFXFRQU8T9/85fe2e16c7224a33aad3a76cf403d11f/60ec2f6113ff4f64bff995305632c6a8/b9ff3e936ae4446580144d9d6066aa37""
Which would show as incomplete in the UI as the database has to get the information from the storage in order to update the steps in the UI. It will show as complete for the deployment as when that completes it updates directly from the database, but the logs themselves for the steps need to be taken from the storage and Octopus is having difficulty accessing them.
I will get the first error looked at by the engineers but in the meantime can you look at the permissions for that folder and ensure the service account Octopus is using for your HA nodes has access to that task logs folder.
I found at least 50 of those access denied errors so I would try and get those resolved and that might fix the issue.
I will update you on what the engineers say about the first error message.
Kind Regards,
Clare