Octopus stuck on canceling release after full disk

Legacy Uncategorised
1

Our Octoups 2.0.11.1080 server is on a server that happened to fill its disk overnight (due in part to our NuGet packages in the Raven DB), and now is stuck on a user-initiated canceling of the release deployment. I shut the VM down and expanded the disk so it has plenty of extra space now, but the same release is still “Canceling”. This is the error I see on the dashboard:

The drive containing the directory ‘C:\Octopus\PackageCache\feeds-1’ on machine ‘VMSKY-DEPLOY’ does not have enough free disk space available for this operation to proceed. The disk only has 676 KB available; please free up at least 500 MB.
System.IO.IOException: The drive containing the directory ‘C:\Octopus\PackageCache\feeds-1’ on machine ‘VMSKY-DEPLOY’ does not have enough free disk space available for this operation to proceed. The disk only has 676 KB available; please free up at least 500 MB.
at Octopus.Platform.Util.OctopusPhysicalFileSystem.EnsureDiskHasEnoughFreeSpace(String directoryPath, Int64 requiredSpaceInBytes) in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Platform\Util\OctopusPhysicalFileSystem.cs:line 350
at Octopus.Shared.Packages.PackageDownloader.AttemptToDownload(PackageMetadata metadata, IFeed feed, IActivity log) in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Shared\Packages\PackageDownloader.cs:line 81
at Octopus.Shared.Packages.PackageDownloader.Download(PackageMetadata package, IFeed feed, PackageCachePolicy cachePolicy, IActivity log) in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Shared\Packages\PackageDownloader.cs:line 53
at Octopus.Server.Orchestration.Deploy.Acquire.DownloadAndPushToTentaclesOrchestrator.d__7.MoveNext() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\Orchestration\Deploy\Acquire\DownloadAndPushToTentaclesOrchestrator.cs:line 60

2

Upon further investigation, it turns out that this may have failed due to a hung Check Tentacle Health task, which I’ve attempted to cancel but seems to be hung. Based on what I read here (http://help.octopusdeploy.com/discussions/problems/15530-everything-got-queued), I attempted to change the document ServerTasks state from Cancelling to Cancelled, but got the error “Could not convert document ServerTasks-12630 to entity of type Octopus.Platform.Documents.Model.ServerTasks.ServerTask”. The task is waiting on two polling tentacles, however they are offline. I changed them to the disabled state and am hoping the task will time out at some point.

3

Annnd…ignore this problem! Changing the ServerTasks documents in the Raven DB to Failed instead of Cancelled seems to have made the problem go away.