Runbooks and Default Failure Mode

Hello everyone, I have a peculiar issue. I will try my best the explain the setup and the problem.

I have a project which is set to use “Use guided failure mode” as Default Failure Mode, which is what I want it to do during a deployment.

Within the same project I also have a Runbook with a trigger to run it every day at X AM.

The problem I have is, when the runbook project fails (e.g., time outs getting a web) - it does not really fail, it gets “stuck” on guided failure mode. Because of this, it does not send the email alert I have set up (RunbookRun failed).

The question I have is - does Runbook have its own Default Failure Mode setting, if not - is it planned as a possible feature in future?

The workaround I have in place is to add additional event category filter “Guided failure interruption raised”, however - I have few manual intervention steps, which means I will get email for each of them every time.

Thanks

Hi Rob,

I’m sorry that you are having this trouble with getting emailed notifications when a RunbookRun fails. If you don’t mind, I have a few questions:

Are you on cloud or self-hosted? If you are self-hosted, what version?
Could you provide a raw log from a runbook task that failed?
Are you using Subscriptions to send the email notifications? If so, would you mind taking a screenshot of the “RunbookRun Failed” subscription, with the “Event Filters” pane expanded?

I look forward to helping you get to the bottom of this soon.

Regards,
Donny

Hi Donny,

Thanks for your reply.

Correct, our installation is a self-hosted and we are running 2020.1.11 version currently.

Here is the requested raw log from the runbook that failed (anonymized): ServerTasks-25100.log.txt (25.4 KB)

Correct, we are using subscriptions to send the email notifications, here are the requested screenshots:


Hi Rob,

I see in the logs where you have several guided failure steps. I’m waiting for confirmation from my team as to whether we have a way to only email on the first one or not.

If not, this may be something I can put in a feature request for depending on what my team comes back with.

I appreciate your patience and I’ll get back to you as soon as I have more information.

Regards,
Donny

Hi @donny.bell ,

I would not worry about the email thing too much as I can add additional filter to the filter - “Guided failure interruption raised” and get email out, even if its not brilliant, but still usable.

My main question is - can I fail the runbook and bypass the guided failure mode, if a step fails, fail the whole task?

My runbook is running on a trigger - Run every day at 4 AM.

If the runbook step fails at Saturday 4am - it gets “stuck” on guided failure mode as it inherits it from the project settings, so the next day at 4 AM a new task is created and put in a queue as the previous task still is “running”, and then on Monday same thing happens, creating a queue of two tasks (what if I go on holidays and nobody notices this problem? There would be queue of 10+ tasks waiting to be ran).

Once I fail the guided failure mode, the queue gets going, but it is not something I want it to do if the task does exactly the same process over and over again.

Tl;Dr
I want for Runbook to completely fail on a step error or not inherit failure mode from project.

Hi Rob,

I did a bit of digging. I wasn’t able to find any workarounds to prevent a runbook from inheriting Guided Failure Mode being turned on.

We are looking at providing more granularity in runbooks as a feature update in future builds. I don’t have an ETA or anything at this time, unfortunately. Check our release notes for new builds from time to time to catch when it drops. Otherwise, feel free to send me a PM check in on the status from time to time. :slightly_smiling_face:

Regards,
Donny

+1 on this it would be great to be able to set the guided failure mode preference for triggers (for deployments and runbooks)