Cancelling Deployment

[dushan_perera]

Hi
We are currently running version 2022.3.10674, We are noticing below issues after we cancel on going deployment.

1. It will take long time to cancel
2. Even we cancel it will progress some random steps.
3. If we cancel the deployment mainly on Acquire packages step. Octopus will throw lots of manual intervention warnings and we have to action all the intervention to get deployment cancel.

[Daniel_Fischer]

Hi @dushan_perera,

Thanks for getting in touch! I’ll need to get some further information to better understand what could be going on here.

Would you be able to upload a copy of the raw deployment log where you’ve experienced this? Here is a link where you can securely upload the logs. Only Octopus staff can see the files uploaded here.

Looking forward to hearing from you and getting to the bottom of this.

Best regards,
Daniel

[dushan_perera]

Hi @Daniel_Fischer
I have uploaded the logs

[Daniel_Fischer]

Hi @dushan_perera,

Thanks for uploading that, though it’s going to take some time to digest with so many lines.

I’m wondering if you could also upload a copy of the deployment process to the same link as above?

Also, is the deployment you provided the logs for a child process? If the deployment was a part of a deploy a release step in another project, having some further information on the parent project would help.

Finally, just to clarify, you’re experiencing this after using orange Cancel button to attempt to stop the deployment?

Just a heads up that I might hand this over to our US team so they can continue working on it overnight, but this further information would be very helpful.

Best regards,
Daniel

[dane.falvo]

Hi @dushan_perera,

Daniel has finished for the day - but I’ve started digging through your log files.
I feel like there is a few things going on here, but at this point it may be a bit of trial and error to work out exactly what’s going on.

I believe one of the reasons Daniel asked for the Deployment Process was to see if issue number 2 - Random steps still running after cancelling the task, is possibly due to “Run Conditions” set on each of your steps.

Can you check to see if some of those steps that are still running after the deployment has been cancelled - have had their “Run Condition:” set to something other than “Success”

I’m also seeing a lot of errors like this:
System.IO.IOException: The process cannot access the file 'C:\Octopus\Tools\Calamari.xxx' because it is being used by another process.

Have you set the “BypassDeploymentMutex” variable on your project?

Are you possibly running the Tentacle service as a user other than a System account that doesn’t have the correct access rights on the folder path “C:\Octopus\Tools”?

Looking forward to hearing back from you.

Regards,

Regards,

[dushan_perera]

Hi @dane.falvo yes OctopusBypassDeploymentMutex is enabled.

And run condition is set to “Success”

Hi @Daniel_Fischer I attached the deployment process export.

[Daniel_Fischer]

Hi @dushan_perera,

Thanks for getting back and uploading the deployment process. We’re still looking into what could be causing the unresponsive cancellation of your deployments, but there are some things you can consider to reduce the amount of errors in your deployment. Resolving these errors could potentially address the unreliable cancelling, but I can’t guarantee that.

I will need some confirmation on a few questions asked above for us to narrow down possible causes.

1. Are you able to provide some details on your requirements for using the OctopusBypassDeploymentMutex variable? This variable is only suggested in some rare cases where it’s necessarry for the deployment, and we generally recommend using it where possible as it opens to door for a lot of the following errors we are seeing in your logs.

13:06:18   Verbose  |       Failed to extract package to 'C:\Octopus\Tools\Calamari.win-x64\24.0.18'
13:06:18   Verbose  |       System.IO.IOException: The process cannot access the file 'C:\Octopus\Tools\Calamari.win-x64\24.0.18\Microsoft.Azure.Management.Fluent.dll' because it is being used by another process.

2. Are you able to disable the OctopusBypassDeploymentMutex variable, run the deployment, and upload the deployment logs?

3. Can you provide confirmation on the method you’re using to cancel your deployments? We’re not certain whether you’re attempting to abort the deployment during an intervention or whether you’re trying to stop it with the Cancel button shown here.

4. Is the deployment which you provided the logs for a child process which has been started with a Deploy a Release step in another project?

The above information will help us continue troubleshooting this problem.

Looking forward to hearing from you and getting to the bottom of this.

Best regards,
Daniel

[dushan_perera]

Hi @Daniel_Fischer

Please find below

1. The error on point 1, occurring because our antivirus software blocking the Calamari installation. That scenario we are handling through separate runbook by temporary allowing calamari. Unfortunately logs that I provide had the same issue.

2. Sure I will try that and provide you the logs soon

3. I am cancelling deployment form the top right-hand corner cancel button.
   

   

4. Yes first step is calling a runbook on another project. But the issue is coming after that step as well.

[dane.falvo]

Thanks for the update @dushan_perera,

I feel like the OctopusBypassDeploymentMutex variable is the main cause of these issues, so I’ll hold off further investigation until you get the logs over to us. I see you’ve uploaded the deployment process, however the task logs are what we are chasing.

Please let us know when those logs are uploaded and we will get straight onto them.

Regards,