Error handling- Process RunCondtion is dependent on overall steps rather than previous step

Hello Team,

We have a complex Implementation, were we deploy 3 Murex modules. Currently we have a Default channel, where we can deploy all 3 modules sequentially one after the other.

We are also making use of guided failure mode to allow retry in case of any exception.

Problem: if there is any failure in one of the fiery module, release is result in failure and we can’t deploy second and third.

Example:

Process

  1. Manual intervention for module A
  2. Prep workspace
  3. Deploy Module A
  4. Manual intervention for module B
  5. Prep workspace
  6. Deploy Module B
  7. Manual intervention for module C
  8. Prep workspace
  9. Deploy Module C

When Run condition is now set to success of previous step — on failure of step 3, steps 4-6 gets skipped

When Run condition of step 4 to 6 is set to always, the steps are being invoked but there is a problem.

In case if user decides to Abort the Manual intervention take, then release should fail or abort but bec of Always condition, step 5 to 9 is invoked.

As per official documentation, the run condition should check the previous step state and act on.

Based on this we set step 5,6 and 8,9 as only when previous step is succeeded, and step 4,7 as always, but Octopus evaluates or check if all previous step is successful or not and failed the build by skipping 4,5 and 8,9 steps.

Is there any way we can achieve this? Please help.

Thanks,
Naga
9738120425

Hi @nagaharshithm,

Thank you for reaching out to us with your question and providing me with some great detail here.

Having a look through your scenario, I would like to go off and do some further investigation with this in my local lab environment to see if I can help come up with some ideas for you here.

Hopefully I can get back to you over the next couple of business days with some ideas here (based on what you have already told me).

Some initial thoughts that spring to mind here (and what I will also take a look at) would be to explore the following (if you have not done so already).

Can I also just check with you about the deployment process itself:

  • Do the modules need to be released sequentially in a specific order?
  • Can they be deployed together in parrallel?
  • Are they dependent on one another?

Can I just check to see what version of Octopus you are running too (in case there are any limitations with your version).

I look forward to hearing from you,

All the best,
Doug

Hello Doug,

Thanks for your response in short time.

Will update by tomorrow on the official version being used in Bank.

Murex is the application, we are trying to deploy using Octopus Deploy.

Have written separate process/steps to deploy these 3 components separately and also added a channels as an option where, users can deploy each components individually.

However on the Monthly or Major releases, we need to deploy all three components and its sequentially deployed one after the other.

Each of these components have a manual intervention task, where user can approve or abort the release Rishiyur before deployment of each component. Also guided failure mode is enabled which allows re try.

There are cases where some times in case of any failure in first or second component, due to some env specific or dynamic confit error, which needs to be fixed manually.

In such cases, we need to deploy other components, and report deployment as partial succeeds as some manual fix will be done.

Have not yet tried conditions based on variables, will be helpful. Also open for other options

Please let me know if you need more details.
Happy to share.

Thanks,
Naga

Hi @nagaharshithm,

Thank you for the update. I have taken a look at some possible ideas for you which may be of some interest or things you may want to consider:

Idea 1: Using Variables for Run Conditions.

Setting each step with the variable run condition:

#{unless Octopus.Deployment.Error}true#{/unless}

This will have the effect of:

If a step fails, guided failure intervenes.

  • If you choose FAIL on the step, all subsequent steps don’t run
  • If you choose RETRY on the step, a retry will occur, and any further failure will be handled by guided-failure.
  • If you choose IGNORE on the step, a failure (as far as the variable Octopus.Deployment.Error will not have occurred, as you’ve chosen to ignore it), and therefore the next set of steps will continue to run.

My deployment task looks like this when I go through using guided failure:

If you wanted more control e.g. you didn’t want step 3 to run if step 2 failed, then you could update the last steps (prior to the manual intervention) to a run condition like so:

#{unless Octopus.Deployment.Error}#{if Octopus.Action[Step Name].Output.Octopus.Action.Script.ExitCode =="0"}true#{/if}#{/unless}

You wound need to replace Step Name here with the step name of the first step after each manual intervention step

This would skip the step if the previous one failed. The other steps e.g step 3 could just keep the condition like #{unless Octopus.Deployment.Error} .

Idea 2: Separating modules into individual projects and using the “Deploy A Release Step

I also wanted to float the idea about potentially splitting out your deployment process and coordinating this using a project to deploy your modules independently (while keeping the order).

From the image below I have 4 projects, 3 are the individual modules and a project to Orchestrate these “Module Orchestration Project”.

Within my orchestration project I am using the “Deploy A Release Step” within my process to deploy these (with guided failure enabled):

From my testing, when I set the step to fail it will only fail the module that is not deploying correctly and will move onto the next deployment.

Doing it this way as they projects have been separated out they are actually independent of one another and will act as 3 separate deployments within Octopus itself.

As you can see from my deployment task log, module 2 failed but I could still successfully deploy module 1 and 3:

Idea 3: Using Child steps

Another idea I tried was to use child steps within my deployment process (with guided failure enabled). This was to try and keep this all within the same project and process:

In my test, I configured both module 1 and 2 to fail here. If I abort the Manual intervention step, guided failure will trigger and I can then also fail the step(s), but my deployment process will continue through the steps (ignoring the child steps from the failed step).

From my deployment log here I have a failed deployment for steps 1 and 2 but can see that Octopus still continues to deploy step 3 here:

Just as a heads up, when you are modifying your deployment process please ensure you are doing this in a controlled way. So you could either consider using our Configuration as Code feature to branch of changes to your deployment process.

An important note her is that once you make a project version controlled you can no longer switch this back as it is a one way option, so please make sure you understand this before making these decisions.

Or another option is to Clone your project and test against this in a safe way before modifying your main deployment process.

I hope this has helped to give you some ideas or things to try here. Personally I feel the best approach would be looking at using variables within the run condition to help build out your logic for your deployment process.

I appreciate there is quite a bit of information here, so, If you have any follow up questions for some of the ideas, please feel free to reach back out to us. If not, then good luck.

All the best,
Doug

Thanks a lot Doug.

These steps will help us and give more options. We have enabled confit as code and will give a try on these options.

Thanks again for all the help and support.
Have a great day.

Regards,
Naga

1 Like