Runtime Error on Deploy A Release process step

Octopus Server
, , ,
1

Hello we are regularly seeing this runtime error when using the “Deploy a Release” step for another Project.

This seems to be a new problem after a recent upgrade. We have other projects that are working with this feature. There seems to be no indication as to what is different or wrong with this Project.

Are there any suggestions to resolve this runtime error?

Octopus Server version: 2021.2.7580+Branch.release-2021.2.Sha.db69134412ffbf682ee48e345e2354b7834538ef 
October 18th 2021 16:09:09Verbose
Environment Information:
  IsRunningInContainer: False
  OperatingSystem: Microsoft Windows 10.0.17763
  OsBitVersion: x64
  Is64BitProcess: True
  CurrentUser: NT AUTHORITY\SYSTEM
  MachineName: EN-OCTOPUS
  ProcessorCount: 4
  CurrentDirectory: C:\windows\system32
  TempDirectory: C:\windows\TEMP\
  HostProcessName: Octopus.Server
  PID: 3196 
October 18th 2021 16:09:10Verbose
Creating deployment for release 506.0.40385-next of project 'VendorApiSupport' 
October 18th 2021 16:09:11Verbose
Stack empty.
System.InvalidOperationException
   at System.Collections.Generic.Stack`1.ThrowForEmptyStack()
   at System.Collections.Generic.Stack`1.Pop()
   at Octopus.Core.Features.Projects.ProjectScope.ProjectPathDisposable.Dispose() in ProjectScope.cs:line 105
   at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentFactory.CreateDeployment(IOctopusQueryExecutor queryExecutor, Deployment deployment, User user, DateTimeOffset queueTime, Nullable`1 queueTimeExpiry, ILifecycleProgressionEvaluator lifecycleProgressionEvaluator, IDeploymentSummaryService deploymentSummaryService, IEnumerable`1 additionalVariables, IPromptedVariableValueProvider promptedVariableValueProvider, Boolean isScheduledDeployment, ILog log) in DeploymentFactory.cs:line 184
   at Octopus.Core.Resources.Deployments.DeploymentCreator.CreateDeployment(DeploymentResource resource, DateTimeOffset queueTime, Boolean isScheduledDeployment, IOctopusQueryExecutor executorAsPrincipal, User user, IPromptedVariableValueProvider promptedVariableValueProvider, ILog log, CancellationToken cancellationToken, IEnumerable`1 additionalVariables) in DeploymentCreator.cs:line 68
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.<>c__DisplayClass18_1.<Execute in DeployReleaseActionHandler.cs:line 233
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass10_0`2.<Execute in UnitOfWorkExecutor.cs:line 166
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.DoAsync[T](IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 43
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.DoAsync[T](IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 43
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,TResult](Func`3 action, CancellationToken cancellationToken, String name) in UnitOfWorkExecutor.cs:line 167
   at Nito.AsyncEx.Synchronous.TaskExtensions.WaitAndUnwrapException[TResult](Task`1 task)
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.Execute(ActionCommand command, DeploymentTarget machine, ITaskLog taskLog) in DeployReleaseActionHandler.cs:line 220
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.DeployRelease.DeployReleaseActionHandler.Execute(ActionCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in DeployReleaseActionHandler.cs:line 102
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.OldActionDispatcher.<>c__DisplayClass4_0.<DispatchServerAction in OldActionDispatcher.cs:line 50
   at Octopus.Server.Infrastructure.EitherAsyncOrSync.Execute(CancellationToken cancellationToken) in EitherAsyncOrSync.cs:line 38
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(EitherAsyncOrSync callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken) in Guidance.cs:line 116
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(EitherAsyncOrSync callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Maybe`1 callbackOnExclude, CancellationToken cancellationToken) in Guidance.cs:line 67
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.OldActionDispatcher.DispatchServerAction(ActionCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in OldActionDispatcher.cs:line 51
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.OldActionDispatcher.Dispatch(ActionCommand command, DeploymentTarget deploymentTarget, ITaskLog taskLog, CancellationToken cancellationToken) in OldActionDispatcher.cs:line 33
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.Dispatch(ActionCommand command, DeploymentTarget deploymentTarget, ITaskLog taskLogForTarget, ITaskLog taskLogRoot, Maybe`1 guidanceExcludeCallback, CancellationToken cancellationToken) in NewActionDispatcher.cs:line 92
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionTaskController`1.<>c__DisplayClass38_2.<ExecuteActionAndInitLoggingContext in ExecutionTaskController.cs:line 601
   at Octopus.Server.Infrastructure.EitherAsyncOrSync.Execute(CancellationToken cancellationToken) in EitherAsyncOrSync.cs:line 38
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(EitherAsyncOrSync callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken) in Guidance.cs:line 116
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(EitherAsyncOrSync callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Maybe`1 callbackOnExclude, CancellationToken cancellationToken) in Guidance.cs:line 67
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionTaskController`1.<>c__DisplayClass38_1.<ExecuteActionAndInitLoggingContext in ExecutionTaskController.cs:line 607
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass4_0`2.<Execute in UnitOfWorkExecutor.cs:line 74
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.DoAsync(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 73
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.DoAsync(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 73
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2](Func`4 action, CancellationToken cancellationToken, String name) in UnitOfWorkExecutor.cs:line 75
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionTaskController`1.<>c__DisplayClass38_0.<ExecuteActionAndInitLoggingContext in ExecutionTaskController.cs:line 608
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionTaskController`1.ExecuteWithTransientErrorDetection(Func`2 action, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in ExecutionTaskController.cs:line 753
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionTaskController`1.ExecuteActionAndInitLoggingContext(PlannedStep step, DeploymentTarget targetContext, PlannedAction action, ITaskLog taskLogForTarget, ITaskLog taskLogRoot, CancellationToken cancellationToken) in ExecutionTaskController.cs:line 626 
October 18th 2021 16:09:11Fatal
The action Install Vendor API Support on the Octopus Server failed
3

Hi @mmargala,

First and foremost, welcome to the Octopus Forums!

I’m sorry to hear you’ve run into this.

My best guess based on that error is that there is something in the JSONs for this project with a null value or blank value where one is expected.

Are you able to DM me a project export so I can help look for the bad data? Exporting and Importing Projects - Octopus Deploy

Keep in mind if you send me an export I will need the password to import it to my instance. This means I will have access to your sensitive values. I may not need to import it as I should be able to see all the data I need without importing it, but importing may be handy to have, so if that makes you uncomfortable, we can try the export without the password to see if I can see any bad data in the JSON.

If you want to spelunk on your own, that’s cool with me too.

Please let me know how you’d like to proceed.

Best,
Jeremy

4

Hi @jeremy.miller , thanks for the reply.

We do have some sensitive information in the export so we would prefer not to send it if possible. Note this seems to be an intermittent issue, so I’m not really sure what could potentially be null/blank where one would be expected. The project this is failing in deploy’s contains about 18 “sub-projects”, and it doesn’t always seem to error out the same “sub-project”. All of the “sub-projects” follow the same project template.

If you have any other advice to trace this down, please let me know.

Thanks,

Mike

5

Hi @mmargala,

I’m stepping in for Jeremy while he is on leave.

Can you confirm if you have many steps/tasks running at once against HostProcessName: Octopus.Server?

I’m working on a similar issue with another customer with the same Stack empty error being generated by a project consisting of 12+ process steps that deploy releases of other projects in parallel.

Does this sound at all similar to the circumstances that are generating the error on your end?

Let me know at your earliest convenience.

Best Regards,
Donny

6

We have Octopus.Acquire.MaxParallelism set to 50 and OctopusBypassDeploymentMutex set to true for this project.

We do indeed have 18 steps(that deploy releases of other projects) attempting to run in parallel here.

Sounds very similar. Thanks,

Mike

7

Hi @mmargala,

Thank you for getting back to me and confirming.

Here is the link to the Github issue I created that you may subscribe to:

I think I have a workaround for this issue by setting the variable OctopusBypassDeploymentMutex to false​ for the orchestrator project that deploys the releases. I know this will likely impact how quickly the orchestrator task completes. However, in my limited testing, I was unable to generate the Stack empty error with the mutex disabled.

If you have any additional questions or if we can assist with anything else, please don’t hesitate to ask.

Best Regards,
Donny

8

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.