Subscriptions Stopped Working

Legacy Execution
1

Hello, We a subscription setup to post a message to slack when a deployment has started and completed and for some reason it stopped working last week. I’m seeing this error in the logs.

2017-06-26 08:27:13.3931 11 INFO Reader took 712ms (298ms until the first record) in transaction ‘NotificationCreator’: SELECT * FROM dbo.[Event] WHERE ([Category] in (@eventgroupseventcategories_0, @eventgroupseventcategories_1, @eventgroupseventcategories_2, @eventgroupseventcategories_3, @eventgroupseventcategories_4, @eventgroupseventcategories_5)) AND ([Category] in (@eventcategories_0, @eventcategories_1, @eventcategories_2, @eventcategories_3, @eventcategories_4)) AND ([AutoId] > @fromautoid) AND ([AutoId] <= @toautoid) ORDER BY [Occurred] DESC
2017-06-26 08:27:23.4151 11 WARN Append log entry to ScheduledTasks_ProcessSubscriptions attempt #1 faulted, retrying in 00:00:00.1000000: The task was canceled.
2017-06-26 08:27:33.4301 11 WARN Append log entry to ScheduledTasks_ProcessSubscriptions attempt #2 faulted, retrying in 00:00:04.8540000: The task was canceled.
2017-06-26 08:27:38.5795 266 INFO https://jx2nutl01:10933/ 266 Opening a new connection
2017-06-26 08:27:38.5955 266 INFO https://jx2nutl01:10933/ 266 Performing TLS handshake
2017-06-26 08:27:38.6136 266 INFO https://jx2nutl01:10933/ 266 Secure connection established. Server at [::ffff:10.6.196.123]:10933 identified by thumbprint: A5B6EA1655427415BE717E32BAF4C0981002DE45, using protocol Tls11
2017-06-26 08:27:48.3195 11 WARN Append log entry to ScheduledTasks_ProcessSubscriptions attempt #3 faulted, retrying in 00:00:16.9750000: The task was canceled.
2017-06-26 08:28:15.3061 11 WARN Append log entry to ScheduledTasks_ProcessSubscriptions attempt #4 faulted, retrying in 00:00:30: The task was canceled.
2017-06-26 08:28:55.3369 11 INFO The scheduled task Octopus.Server.Schedules.ProcessSubscriptions started at 6/26/2017 12:27:12 PM +00:00 and took 00:01:42.6878171 which can make this Octopus Server node ‘JX2PDVM02’ appear to be offline. If this happens frequently please report this to the Octopus Support team.
2017-06-26 08:29:35.4187 11 WARN Append log entry to ScheduledTasks_ProcessSubscriptions attempt #1 faulted, retrying in 00:00:00.1000000: The task was canceled.
2017-06-26 08:29:45.4397 11 WARN Append log entry to ScheduledTasks_ProcessSubscriptions attempt #2 faulted, retrying in 00:00:05.1370000: The task was canceled.
2017-06-26 08:30:00.5991 11 WARN Append log entry to ScheduledTasks_ProcessSubscriptions attempt #3 faulted, retrying in 00:00:17.6140000: The task was canceled.

So far I haven’t been able to figure out what the issue is.

Thanks,
Matt

2

Hi Matt,

Thanks for getting in touch.

Could you please confirm which version of Octopus you are running?

It appears the subscriptions scheduler is failing to write/append to the subscriptions log file. The subscriptions log file it is trying to write to is typically found in this folder C:\Octopus\[your_instance]\TaskLogs, named scheduledtasks_processsubscriptions.txt.

Do you have any virus scanners running that may be putting any sort of lock on the log file(s)?

It would be worth logging onto the server and seeing if there’s something putting a lock on that file.

It may also be worth trying to restart your Octopus service and seeing if that releases the lock. Worst case, you could schedule a restart of the actual server (during a maintenance period so it won’t disrupt operations) and see if that helps. But our advice at this stage would be to try and find what is stopping Octopus from being able to write to that file.

Cheers
Mark