Subscription notification not sent, can't tell why

We have a subscription set up to send a notification to a webhook, for deployment events in the UAT or Production environments for specific project groups.

We had a deploy today for a project in one of those groups, to UAT, but the subscription did not fire:

The deployment in question was AutoId 1186995. We did get a notification for 1186996.

image

(I cut down the columns for size, but can provide more of the data if needed.)

Redeploying the release did send a notification as expected.

The only thing I can come up with is if there was some sort of race condition - the Occurred stamp in the Event table for that event was 2020-09-02 16:30:36.6260747 +00:00, or the exact same time (once you take time zones into consideration - the log screenshot is in Eastern Daylight Time) as it decided that “no notifications need to be sent”. If the Event is not written to the table in a single step, and the subscription engine read the record partway through, that could explain why the subscription rules didn’t pick it up…

We are still on Octopus v2019.12.8 - waiting for a new SQL server to be built so we can upgrade. However, I don’t see anything in the release notes for later versions that seems to be related.

Hi Andrew,

Thank you for pointing this out and getting in touch.

There were multiple issues and some major rework done on these triggers in 2019.13.0 which would fix this issue.

Some details regarding the issue you noticed and the fix are here: https://github.com/OctopusDeploy/Issues/issues/5911

Hopefully you are not too far away from the SQL server so you can go ahead with an upgrade.

I’m sorry I can’t help further with this issue, but please reach out if I can help with anything else.

Regards,

Dane.

Thanks. Just to confirm, those changes apply to subscriptions too, not just deployment triggers? (i.e., the deploy happened, it was the resulting DeploySucceeded subscription that was missed)

Hi Andrew,

This was a significant change with the way that triggers work. I have escalated the question to the engineering team, however haven’t received a definitive answer just yet.

I will let you know if someone has the time to dig into the code and look at the exact changes but until then, I am unable to give you an answer.

When your SQL server does get built, I would recommend upgrading to the latest versions as there is quite a few improvements since 2019.12.8. (https://octopus.com/downloads/compare?from=2019.12.8&to=2020.3.4)

On that note, you should be able to upgrade to 2019.13.0 without waiting for the SQL server rebuild - but I understand that a single upgrade is much better than doing two smaller upgrades.

Hopefully I will get a response later today.

Regards,

Dane

I’d considered that, but since 2019.13.x didn’t have a patch with the Chrome 83 fix wanted to avoid that if possible.

We got the build done late last week and upgraded to 2020.3.4 over the weekend. So at this point we’re just waiting for the response to confirm whether the fixes are expected to apply to the problem at hand too, or if we need to keep an eye out for the problem in the future.

Thanks for the follow-up!

Hi Andrew,

To answer your question directly, there has been no indication that the way subscriptions are handled, is different between the version you were on and the latest version of Octopus.

I’ve had a response from the engineering team. They have said that this screenshot alone, will not be enough to determine if there is a bug or not.

Can you please send through the verbose logs from that date/time from the endpoint:
https://<endpoint>/app#/Spaces-x/tasks?name=Subscription&spaces=Spaces-x`

( for example a localhost would be: http://localhost/app#/Spaces-1/tasks?name=Subscription&spaces=Spaces-1 )

If you can find the date/time of the subscription getting triggered (there can potentially be multiple entries at a single time), you will be able to select “Task Logs” followed by changing “Log Level” to Verbose. This will give you an idea of whether the Task fired for that particular machine.

Click on Download to download the log files and attach those files in the next reply to this ticket please.

This should give us enough information to go off, and start the reproduction of the issue if required.

Thanks for your patience.

Regards,

Dane.