What does Abort actually do?

[tfbryan (Todd Bryan)]

I’ve seen multiple posts about manual intervention and the Abort option not stopping the project. What exactly does choosing Abort actually do if it does not stop all remaining deployment steps from executing?

Manual Intervention

Your only two options are to Proceed or Abort, but depending on your Run Conditions, Abort actually allows the project to proceed a well?

Error Intervention

Now we have more options as to how we can proceed, and these options seem to do exactly what they state they do, unlike Abort which is very misleading.

How are we supposed to have a multi-package deployment project with infrastructure orchestration if we cannot actually orchestrate?

(Scenario 1) Consider the situation where all steps are left as default.

Now we have one step fail and all remaining steps will be skipped including important infrastructure orchestration which leaves our deployment half-baked and potentially offline.

(Scenario 2) If we switch our steps to Always Run

Now we’ve lost all control over the actual approval process and cannot Abort a deployment short of hitting the Cancel button on the entire project.

Another thing to consider is what happens when the deployment steps are all Deploy a Release and you come across 1 child project that has to be canceled. That particular Deploy A Release step is now marked as failed which kicks off scenario 1 if you are using the default run condition settings. Ask me how I know this…

I’m trying to figure out how to deal with a potential failure during a massive train deployment without risking my entire stack going offline, while still allowing the change implementer the ability to call off a deployment if things don’t look ready. I’m not seeing much in the middle-ground here.

[finnian.dempsey]

Hi Todd,

Thanks for reaching out, I appreciate the detailed notes and screenshots! Hopefully I’ll be able to clear up some confusion around what ‘Abort’ does and provide some useful advice for setting up your deployment process.

How the abort option in the Manual Intervention step works is to throw a fatal error for that Manual Intervention Step. As you’ve mentioned, this doesn’t ‘abort’ the entire project, and it depends entirely on how the Run Conditions for each of the other steps have been configured. Being able to configure the Run Condition per step allows for greater control over the project as having a step run on ‘Failure’ or ‘Always’ is sometimes required for the Project, e.g. notifying a team via slack.

I’ve attached one of our blog posts that I think would be really helpful for you here, as it discusses common scenarios for deploying entire application stacks and some typical approval processes. It’s quite a long post and covers some complex scenarios so feel free to reach out if you have any questions. Also worth checking out our Samples instance to see the examples in Octopus for inspiration.

If I’ve missed any questions or you’d like me to clarify anything further, please let me know.

Best Regards,

[tfbryan (Todd Bryan)]

Thanks for sharing that article. Unfortunately it doesn’t really address Run Conditions or how to avoid any of the scenarios I described. The only thing I can think of is stacking multiple Run Conditions that evaluate against the state of the manual intervention steps but that seems like a bit of a high maintenance hack. All it would take is for someone to rename one of the manual intervention steps and everything is essentially broken.

[finnian.dempsey]

Hi Todd,

Avoiding any hacky solutions is definitely the best approach. I believe in this case guided failures, documentation here, could be the right feature to use.

Configuring a parent project for Guided Failure will also force all the child projects into that mode also, allowing for control over specific components. e.g. Choosing ‘Ignore’ for a child project will cause that child project to error but the rest of the deployment to continue, whereas selecting ‘Fail’ will cause the parent project to also fail.

If this doesn’t help or you have any questions feel free to let me know. I’ve also linked our Youtube channel which has some great videos from our Solutions Team about typical issues and solutions, definitely worth checking out!

Best Regards,

[tfbryan (Todd Bryan)]

Yes I’m aware of guided failure and we do utilize that feature, however I had a situation where one of the child projects had to be canceled do to an error on a script that was not allowed to be ignored. That created a chain reaction of all remaining steps in the parent project getting skipped, including my load-balancer “undrain” steps which left my entire web farm offline. As you can imagine, this was not an acceptable deployment result. This might be somewhat of a rare edge case, however, I need to make these deployments as usable as possible so that our change implementers can safely deal with situations as the may arise.

[finnian.dempsey]

Hi Todd,

I’d like to help figure out the best approach for this and if it is an edge case we haven’t thought of, raise a request with the engineers to make sure it’s handled appropriately. After investigating this in depth today, I have to admit there does seem to be something strange going on that I will definitely dive into further.

The issue seems to be that while it is possible to ‘Ignore’ a child step to continue the Parent project, the rest of the Child’s steps will also continue as ‘Ignore’ counts as a success. I’ve raised an internal discussion around this and will keep you posted once I have some helpful advice or workaround.

In the meantime, I found this Youtube video which is specifically focused on tips for deploying without taking down an entire Webfarm, which is useful but it doesn’t seem to cover the case mentioned above, which you appear to be facing.

Apologies for not understanding your predicament sooner, hopefully we can get to the bottom of this shortly.

Best Regards,

[finnian.dempsey]

Hi Todd,

Apologies for the delay in getting back to you, but I believe I finally have a solution!

I have managed to reproduce an environment with a Parent and Child project, which allows for the Child project to “Fail” and still allow for a choice on whether the remaining Parent steps are to proceed or not.

It is fairly confusing so I will do my best to explain the process and provide as much info as possible.

With Guided Failure configured for the Parent Project, the Child project will inherit it also, and so this means that if there is an error in the Child Project, Intervention will be required and the user can choose to ‘Fail’, ‘Ignore’ or ‘Retry’. If ‘Ignore’ is selected, then the deployment is to proceed as usual, with the remaining Child Steps to run along with the remaining Parent Steps. However if ‘Fail’ is selected, the remaining Child Steps aren’t to run, and the Parent Steps are to be decided if they are to be run or not.

The required components are all configured from the Parent Project. After the ‘Deploy a Child Project’ Step, I have configured a Manual Intervention Step with a Run Condition set to only run if the previous step fails, e.g. when the Child Project’s intervention was selected as ‘Fail’. Once this Step is reached, a user can select to either ‘Abort’ which will not continue with the Parent Steps, or to ‘Proceed’, which will continue with the Parent Steps. The catch here is that the remaining Parent Steps are to have a Run Condition based on the variable, #{Octopus.Action[Manual Intervention Required].Output.Manual.Approved} which will check if the Manual Intervention was ‘Aborted’.

Unfortunately I did run into a bug when testing this method, where if there are multiple interventions for the Child Project, the “Ignore” selection will incorrectly return as if the Project failed. This bug won’t effect the logic of the deployment, it will however incorrectly trigger the Manual Intervention step in the Parent Project, but if ‘Proceed’ is selected then everything will function as if the Child Project correctly Ignored the error.

In summary:
Configure Guided Failure for the Parent Project (Child Project will inherit this setting)
Configure a Manual Approval Step immediately proceeding the “Deploy a Child” step, which has the Run Condition to only run on previous steps Failure.
Configure the remaining Parent Steps with a run condition based of the result of the Manual Intervention Step. i.e: #{if Octopus.Action[Manual Intervention Required].Output.Manual.Approved != “False”}true#{/if} (This expression evaluates to False only if the Manual Intervention Step value was “Aborted”, so the steps will always run unless Abort is selected. This logic helps make the crux of the solution.)

Deploy Child Project:

Manual Intervention:

Remaining Steps:

I’ve also attached the Json for my Parent Project below. Feel free to reach out if you would like some further explanation around the process or you have any questions!

Best Regards,

parent-process.json (4.5 KB)