Cancelling a Deploy a Release step: release deployed, but marked as cancelled

Legacy Execution
1

I have a project A with a Deploy a Release step which deploys a release from project B. I start a deployment in A and cancel it. The end result is that the Deploy a Release step is marked as cancelled, but actually project B is deployed.

Problems I can see with this:

1. The cancellation request doesn’t cause child deployment to be cancelled.

Log fragment from project A:

Task ID:        ServerTasks-50293
Related IDs:    Deployments-25245, Channels-117, Releases-21482, Projects-84, Spaces-1, Environments-1
Task status:    Canceled
Task queued:    Wednesday, 25 September 2019 7:00:16 AM +00:00
Task started:   Wednesday, 25 September 2019 7:00:19 AM +00:00
Task completed: Wednesday, 25 September 2019 7:33:32 AM +00:00
Task duration:  33 minutes
Server version: 2019.8.3+Branch.tags-2019.8.3.Sha.8a69207dba03d0d0032ece992e1a60a07157ba5b
Server node:    EC2AMAZ-V9BN36K


                    | == Failed: Deploy AAAAAAAAAAAAAA release 2019.09.25-07.00 to UAT ==
07:00:21   Verbose  |   Guided failure is not enabled for this task
07:24:13   Info     |   Requesting cancellation...

                    |   == Failed: Step 17: BBBBBBBBBBBBBBBBBBBBB ==
07:32:52   Fatal    |     The step failed: This task has been canceled.
07:32:52   Info     |     The operation was canceled.
                    |   
                    |     Success: Worker
07:01:19   Verbose  |       Octopus Server version: 2019.8.3+Branch.tags-2019.8.3.Sha.8a69207dba03d0d0032ece992e1a60a07157ba5b
07:01:19   Verbose  |       Environment Information:
                    |       OperatingSystem: Microsoft Windows 10.0.14393
                    |       OsBitVersion: x64
                    |       Is64BitProcess: True
                    |       CurrentUser: EC2AMAZ-V9BN36K\svcOctopusServer
                    |       MachineName: EC2AMAZ-V9BN36K
                    |       ProcessorCount: 2
                    |       CurrentDirectory: C:\Windows\system32
                    |       TempDirectory: C:\Users\svcOctopusServer\AppData\Local\Temp\
                    |       HostProcessName: Octopus.Server
                    |       PID: 2332
07:01:19   Verbose  |       Creating deployment for release 5.0.1.6124+master of project 'BBBBBBBBBBBBBBBBBBBBB'
07:01:23   Info     |       Deploying release 5.0.1.6124+master of project 'BBBBBBBBBBBBBBBBBBBBB'. [View deployment](~/app#/Spaces-1/deployments/Deployments-25254)
07:32:52   Info     |       Deployment of release 5.0.1.6124+master of project 'BBBBBBBBBBBBBBBBBBBBB' completed successfully
07:32:52   Verbose  |       Updating manifest with output variables
07:32:52   Verbose  |       Updating manifest with action evaluated variables
07:32:52   Verbose  |       Successfully finished BBBBBBBBBBBBBBBBBBBBB on a Worker

Log fragment from project B:

Task ID:        ServerTasks-50301
Related IDs:    Deployments-25254, Channels-246, Releases-21295, Projects-165, Spaces-1, Environments-1
Task status:    Success
Task queued:    Wednesday, 25 September 2019 7:01:19 AM +00:00
Task started:   Wednesday, 25 September 2019 7:31:25 AM +00:00
Task completed: Wednesday, 25 September 2019 7:32:51 AM +00:00
Task duration:  1 minute
Server version: 2019.8.3+Branch.tags-2019.8.3.Sha.8a69207dba03d0d0032ece992e1a60a07157ba5b
Server node:    EC2AMAZ-V9BN36K

                    | == Warning: Deploy BBBBBBBBBBBBBBBBBBBBB  release 5.0.1.6124+master to UAT ==
07:31:26   Verbose  |   Guided failure is not enabled for this task
07:32:51   Info     |   The deployment completed successfully.
                    | 
                    |   == Success: Acquire packages ==
07:31:26   Info     |     Acquiring packages
07:31:26   Info     |     Making a list of packages to acquire
07:31:26   Verbose  |     No packages are required on the Octopus Server
07:31:26   Verbose  |     Delta compression is enabled for package transfers from the Octopus Server to deployment targets
07:32:32   Info     |     All packages have been acquired
07:32:32   Verbose  |     Acquire Packages completed
                    |

At the time when cancellation was requested (07:24:13), the child deployment was still not started, so it should have got cancelled.
Event if it was started, but still in package acquisition phase, the cancellation should be propagated from parent to child and the child deployment should be cancelled.

2. The status in parent project is incorrect

Task summary shows:

image

This clearly suggests that this step was cancelled. But Task Log shows:

image
image.png962×283 15.1 KB

Either the child task should really be cancelled (preferred), or at least the status in parent project should correctly show what really happened (that the child project was deployed).

1 Like
3

Hi Jakub,

Thanks for getting in touch! I can certainly see the confusion with this behavior, both with the child project actually deploying when cancelling the parent, and the misleading text in the error message. This is actually “by design” currently, but we certainly agree that this behavior and/or error text should be improved on.

Cancelling the parent project shows the step as cancelled, but the child project’s task will run through to completion. There’s debate on if the child task should be cancelled here (I’m leaning this way myself), but we should at least update the wording here. After a long discussion with my team, these changes will be harder to implement than it first appeared. The following enhancement issue will be used for updating this behavior, and you can track its progress at the following link.

If you’re wanting to cancel the child task, the only workaround is to go into the child task and cancel it directly (which you can follow with the View deployment link in the log). If you have many deploy a release steps, this should still only have to be done once each time, as cancelling the task will prevent the next from starting.

My apologies that this has caused confusion here. Please don’t hesitate to reach out if you have any questions or concerns moving forward. :slight_smile:

Best regards,

Kenny

1 Like
4

Thanks for the reply, @Kenneth_Bates.
I’ll track the GitHub issue.

Regarding your comment that “If you have many deploy a release steps, this should still only have to be done once each time, as cancelling the task will prevent the next from starting.” - it is true only for sequential deployment steps. In our case, we have up to ~70 parallel Deploy a Release steps in various projects, so the problem we face is that we cancel the master project, but still all those 70 projects continue deploying. We’d need to cancel each of them individually, which I’m sure you understand is an extremely tedious and unreliable way to cancel a broken production deployment.

5

Hi Jakub,

That is a great (and unfortunate) point, and I didn’t think about that. That certainly sounds extremely tedious. Would something like a manual intervention step in your parent project immediately before the Deploy a Release steps in order to pause the deployment which you could cancel at that point help out at all here? You could scope this step to run only in production as well.

I hope that helps. Let me know what you think or if you have any further questions. :slight_smile:

Best regards,

Kenny

6

Thanks for the suggestion, but a manual intervention step wouldn’t really help here - it would help only if I wanted to cancel the deployment immediately after starting it (before any “real” deployments - the parallel Deploy a Release steps - are started), while what I want is to be able to cancel an ongoing deployment (i.e. when the parallel phase has already started). For example, when one of the Deploy a Release steps completes and we immediately discover that it has broken the environment, we want to cancel the rest of the deployment to be able to roll back quickly - currently this is impossible, because all the remaining ~70 projects continue deploying unless stopped manually one by one.

7

Hi Jakub,

Thanks for following up, and I can see exactly what you mean here. After a good long think about this, I’m not sure how close to the ideal solution we can get. I think the closest would be to have a custom script to grab all of these separate child deployments to cancel, which you could then manually trigger right after you click the Cancel button on the parent project. The following is a sample script on cancelling a deployment which might help get you started if this sounds like it would help.

I was trying to find a way to be able to trigger this script to be run based on the cancellation, but I don’t think that’d be possible out of the box. A possible future option would be to subscribe to an event and trigger a script like this based on that event. The following UserVoice suggestion might be worth throwing some votes to. :slight_smile:

I’m sorry it’s not better news! Let me know what you think or if you have any other questions going forward. :slight_smile:

Best regards,

Kenny

8

Thanks for the tip. My team will discuss whether we want to invest in building a cancellation automation. Can you share what the decision regarding https://github.com/OctopusDeploy/Issues/issues/5873 is going to be - will the child deployments cancel, or will only the log message be corrected? We really hope cancellation of child steps will be supported out of the box.

As for UserVoice - all of my votes are on way more important items, mostly related to improvements related to variables. Unfortunately your team ignores them for years already (the oldest one is from February 2014).




2 Likes
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.