How to clear a blocking task

matthew_gill · 25 November 2022 06:03

Hi

I have a step that is refusing to run due to another task :

Waiting for the script in task ServerTasks-468674 to finish as that script requires that no other Octopus scripts are executing on this target at the same time

This ‘ServerTasks-468674’ was originally cancelled. I have since tried to delete it to make the step progress. But it still refuses even after cancelling and re-launching the step.

How can I ‘kill’ ServerTasks-468674?

paraic.oceallaigh · 25 November 2022 06:11

Hi @matthew_gill ,
Thanks for reaching out and sorry to see you run into this issue with cancelling a task.

I’ll be honest and say we have recently had to dig through our code around cancelling tasks and fix several issues which are strongly affecting our customers. We have identified many issues which are centred around what happens when customers try to cancel a task.

In your case this Github issue Server Tasks not canceling when a the task is canceled · Issue #7903 · OctopusDeploy/Issues · GitHub seems to fit your symptoms. This is almost ready to be released and you may subscribe to it to see which version it gets fixed in.

The workarounds in that GH issue do suggest that restarting the Tentacle or worker may help but it may also need a restart of Octopus server itself.

This GH issue also mentions and details some of the other issues this piece of code has surfaced for us and that we are fixing right now

github.com/OctopusDeploy/Issues

A Cancelling Server Task can be Deleted

opened 06:33AM - 24 Nov 22 UTC

nathanwoctopusdeploy

kind/bug state/triage team/core-platform

### Team - [X] I've assigned a team label to this issue ### Severity _No resp…onse_ ### Version Earliest tried was 2022.3 ### Latest Version The problem does not exist in the latest build ### What happened? When a Server Task is cancelled it transitions into the `Cancelling` status while any active actions on targets or workers are being stopped and the thread running the server task is stopped. When in the `Cancelling` status the server task can be deleted. The delete action removes database entries and log files but doesn't do anything to the thread that is running the server task (which should be being cancelled) or actions on targets or workers. Recent issues in `2022.3` have exposed several bugs when trying to Cancel server tasks resulting in the cancellation being ignored or failing and the server task remaining in the `Cancelling` status until the task completes or in the case of a hung task e.g. a script that never finishes, octopus server or tentacle being restarted. By being able to delete the server task stuck in the `Cancelling` status we make it appear like everything is gone and stopped which it isn't and it masks other issues ### Reproduction - Run a deployment - Click cancel - Click delete Timing is essential above and we want a server task that takes a while to cancel to correctly repro the issue e.g. deploy to many concurrent targets ### Error and Stacktrace _No response_ ### More Information The behavior will be changed so that a `Cancelling` server task cannot be deleted. If customers were working around server tasks stuck in the `Cancelling` status by deleting them, the behavior change would stop them from doing this but is more correct. Customers should report any server tasks stuck in the `Cancelling` status for further investigation. A server task that is stuck in the `Cancelling` status can be killed by restarting the target or worker if that is the source of the hung server task or restarting octopus server, which will force the server task to stop. ### Workaround _No response_

If you like, you can send us the Octopus server logs and/or the Tentacle logs in this instance and we can dig a bit further to confirm this issue has been fully addressed already in one of the GH issues above.

I expect a fix will be release any day now as the fixed code has already been merged.

Let us know if you need anything further.

Kind regards,
Paraic

matthew_gill · 27 November 2022 23:42

Hi
We ended up resorting to a reboot of the octopus servers to resolve.
Looking forward to the future fix. Thanks.

system · 28 December 2022 23:43

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.