Deployments hanging, even when cancelled

We’re seeing some issues on Octopus 3.13 (currently 3.13.7, but also on a previous version, I think 3.13.3) where a task does not progress and will not fail properly when cancelled. This has happened twice in the last few days and did not resolve until I restarted the Octopus service. One of the tasks was a deployment, and the other was a tentacle upgrade. They seem to be waiting indefinitely on one or more hosts to complete a given step where the octopus server is not getting the proper status from the tentacle. During the tentacle upgrade, the log text was:

This server does not have version 3.7.67 of the Calamari package. It will be pushed automatically.
May 30th 2017 12:01:27Info
Calamari upgrade successful
May 30th 2017 12:01:31Info
Running Tentacle version 3.13.3
May 30th 2017 12:01:31Info
Beginning upgrade
May 30th 2017 12:01:41Info
Waiting for the upgrade to complete
May 30th 2017 12:01:52Info
Tentacle has stopped. Waiting for it to start…

The last message remained for the next 2+ hours, and the target machine had the tentacle service running. Restarting or stopping the service did nothing to allow this to progress.

Hi Robert,

Thanks for getting in touch and apologies for the issues you are seeing.

We have had a few reports of similar issues. It would be a huge help to get a process dump of the Octopus Server and Tentacle while this is happening to see why they are stuck. There are instructions here for creating a process dump: https://octopus.com/docs/reference/process-dumps#creating-a-process-dump

You can upload them securely here: https://file.ac/zpe20-SKgwk/

Cheers,
Shane