Hi, since yesterday night all tasks are being queued, but not executing. we only notice after few hours, when trying to deploy. Canceled all the pending task and tried to execute any task (health check), it is same queued.
In the log last successful task was at 22:37:33, after that for any task there is warning, Tasks are keep being queues still and there is no log written by any task. Please suggest.
Octopus version: 2018.8.2
2018-10-18 22:33:34.6766 2248 10 INFO Cleaning up completed SystemIntegrityCheck tasks. Deleting 1 finished task…
2018-10-18 22:33:34.7235 2248 10 INFO Deleted: ServerTasks-1765
2018-10-18 22:37:33.1197 2248 10 INFO Cleaning up completed Retention tasks. Deleting 1 finished task…
2018-10-18 22:37:33.1577 2248 10 INFO Deleted: ServerTasks-1538
2018-10-18 23:32:34.7143 2248 10 WARN Outstanding SynchronizeExternalSecurityGroupsForUsers tasks were not completed before the next task was due to be scheduled. If this error persists, check the Tasks tab for any running SynchronizeExternalSecurityGroupsForUsers tasks, and cancel them manually.
2018-10-18 23:33:10.1933 2248 10 WARN Outstanding health checks for the machine policy Default Machine Policy were not completed before the next task was due to be scheduled. If this error persists, check the Tasks tab for any running health check tasks, and cancel them manually.
2018-10-18 23:33:40.3792 2248 10 WARN Outstanding health checks for the machine policy Default Machine Policy were not completed before the next task was due to be scheduled. If this error persists, check the Tasks tab for any running health check tasks, and cancel them manually.
Thanks for getting in touch! I’m terribly sorry to hear you’re hitting this issue. We’ve seen this kind of stuck task issue in the recent past. Are you able/willing to try restarting the service and see if that resolves the issue for you?
Secondly, would you be able to check in your database for running or cancelling tasks? You can find these tasks with the following SQL query.
SELECT * FROM [OctoDbName].[dbo].[ServerTask]
WHERE [State] = 'Cancelling';
And replacing Cancelling with Running for the second query.
I hope this helps, and I look forward to hearing back!
Thanks Kenny. We are able to resolve issue with service restart. Was just wondering if any information available why it happens. Will keep watch, if it happens again.
I do have one more question on similar topic. While checking the logs for this I found event for subscription was executing every minute. Is this expected behavior? Can we change the schedule for subscription? Currently we don’t even have any subscription set up.
Thanks for following up! That’s great to hear that was able to resolve the issue. I suspected initially that it was due to a recent issue we’ve seen recently with tasks getting stuck while cancelling. Which version of Octopus are you currently running? An upgrade to latest (if applicable) may help prevent this, and we’d be very interested in hearing back if you encounter it again.
Would you be able to provide some more details on the subscription event you’re referring to? What’s the full entry in the log?
Hi Kenny, we are using 2018.8.2 version, just started using octopus
In the Diagnostics=>Subscription logs=> can see continues subscription events called, though no notification, as we haven’t set any.
Below info almost every second.
No notifications need to be sent at this time (upToEventAutoId:660).
In the task log “scheduledtasks_processsubscriptions.txt” is getting created every day around 450 KB. Please refer attached.
