Runbook trigger issue after upgrade

[mikepower79]

Hi,
We just upgraded to version 2022.2.6832.
Today I have noticed that my runbooks are not running which are scheduled to run every 3 minutes with scheduled triggers.

When I check the audit trail of the trigger I see:

And when I select the Trigger from the audit page:

I get an error:

Is this a bug?

Kind Regards,
Micheál Power

[dan_close]

Hi @mikepower79,

Thanks for reaching out to Octopus support and I’m sorry to see that you are running into this issue with your Runbook triggers. This certainly looks like a possible bug and I would like to get a reproduction setup in-house. Can you let me know which version of Octopus you upgraded from?

I look forward to hearing back and please let me know if you have any other questions for us.

Thanks!
Dan

[mikepower79]

Hi @dan_close,
We upgraded from 2021.3.12372

Kind Regards,
Micheál Power

[dan_close]

Hi @mikepower79,

Thanks for getting back to me. I was able to replicate some of the behavior you are seeing, although it appears my scheduled trigger is still running as expected.

I do get the UI error when I click the trigger link from the audit like you are seeing. Do you get any errors when you attempt to access the trigger directly from the project screen?

Do you see anything noteworthy if you check your scheduled trigger logs? If possible could you download the results of the scheduled trigger logs and send them to us for review? I’ve created a secure upload link for you here.

I look forward to hearing back and please let me know if you have any other questions.

Thanks!
Dan

[mikepower79]

Hi @dan_close,
No I dont get an error when accessing the trigger from the project screen.
I will upload the Scheduled Trigger Logs.

Kind Regards,
Micheál Power

[dan_close]

Hi @mikepower79,

Thanks for sending that over. Would you mind also uploading your Octopus server log for us? I’m hoping there may be an error logged that could point us in the right direction. Is the Octopus Node Task Count Trigger the only trigger you’ve noticed that isn’t firing after the upgrade? I can see clearly in the logs that it stopped yesterday afternoon, however, it does appear other triggers are working as expected.

I look forward to hearing back and please let me know if you have any questions.

Thanks!
Dan

[mikepower79]

Hi @dan_close,
No I have a number of triggers that are not working.
Sure I will upload the Octopus log.

Kind Regards,
Micheál Power

[Daniel_Fischer]

Hi Micheál,

I’m just jumping in for Dan as I’m in Australia and it’s daytime.

I have managed to reproduce the TypeError you’re seeing when attempting to view scheduled triggers through the Audit page. Here is the GitHub issue I created for our developers to further investigate. It looks like manually browsing to the trigger appears to be working normally still. (As you already noted.)

Regarding the triggers not firing when they should. I will need to collect some further information from you to better understand what could be causing this, since I’m not certain that it’s related to the above bug.

I think the best information would be some further details about how your scheduled trigger is configured and the task logs from recent deployments it triggered. Furthermore, there could be some information hidden in the server logs which weren’t shown in the scheduled trigger logs you have already provided.

So if you could provide the following, it would really aid us in troubleshooting your remaining problem.

• System Diagnostics Report (From the same Diagnostics page you downloaded the Scheduled Trigger Logs. If you could upload the entire zip file, it would be greatly appreciated.)
• Screenshot of the expanded (full) configuration for scheduled trigger: Octopus89 Node Task Count Trigger (I’m assuming this one is failing to trigger? Otherwise a screenshot of a broken trigger will be required.)
• Full raw task logs for /app#/Spaces-167/tasks/ServerTasks-3399520 (This task is one of the last mentioned in your previously attached logs and might provide some missing context.

Ideally the logs and screenshots will all be for the same task and/or trigger, for further context when we dig into the logs.

You can use the same upload link as before to securely share these files.

Looking forward to hearing from you and getting to the bottom of this. If you have any questions or thought at all, please don’t hesitate to let me know.

Best regards,
Daniel

[mikepower79]

Hi @Daniel_Fischer,
Just seen this error in the Audit section:

What does this mean?

Kind Regards,
Micheál Power

[mikepower79]

Hi @Daniel_Fischer,
Screenshot of the Runbook Trigger:

[dan_close]

Hi @mikepower79,

Thanks for sending the screenshot of your trigger settings. As for what you saw in the Audit log, that’s just the Runbook retention policy running. By default, Runbooks are set to keep 100 runs per environment so any runs beyond that will be deleted.

Let us know once you have a chance to upload the other files and we will continue digging into this issue. Please let me know if you have any other questions as well.

Thanks,
Dan

[mikepower79]

Hi @dan_close,
The issue was with the version we just upgraded to. 2022.2.6832. We were getting serious performance issues so we upgraded to version 2022.2.6971
The runbook and performance issues have now gone.

Kind Regards,
Micheál Power