I have some examples where the retention policy is logging that a folder is being deleted but only a partial recursive delete is occurring. An quick validation, within the retention policy cleanup, confirming that the file not longer exists, then reporting in the logs accurately would improve this.
Image showing the log reporting a folder was deleted and also the folder not deleted on the target:
Thanks for reaching out and providing a copy of the raw log. I’m sorry that you are running into this issue.
After sifting through the logs, I found the following:
16:34:49 Verbose | Could not delete directory ‘C:\Octopus\Applications\Sprint.Pilot1\Web.wwwroot\3.64.0.3496’ because some files could not be deleted: (5) Access is denied: [\?\C:\Octopus\Applications\Sprint.Pilot1\Web.wwwroot\3.64.0.3496\WSCalcA\bin*]
Lines 4416 through 4890 show all of the deletion requests and many instances of:
Directory ‘C:\Octopus\Applications\Sprint.Pilot1\Web.wwwroot\3.64.0.3496\XXXX’ still exists, despite requested deletion
It looks like something was using files within the directories at the time the retention policy was running. You might try having a look at Process Monitor to see what has those files locked. I would also check usual suspects such as antivirus, backup utilities, hung processes, etc.
If nothing turns up, let me know and we’ll dive deeper.
Thanks for the quick reply. What’s puzzling is that each of the remaining folders in the ~\3.64.0.3496 application directory are empty, else they only contain other empty folders, there are no actual “files” in there at all.
I also reviewed each subsequent Web.wwwroot package deployment, to this same server, after the 3.64.0.3500 release (the log you looked at), and none of them even tried again to delete the 3.64.0.3496 folder. I understand passively letting the deploy succeed when the retention policy fails, as to not fail the whole deployment, but I would expect it to minimally succeed with a warning then try again to delete on the next deployment.
As to your point of a process locking a file, the C:\Octopus\Applications~ directory is part of Octopus’ architecture. It doesn’t follow that anything outside of Octopus would be accessing that directory. Also as I mentioned above the files did actually get removed so using Process Monitor is moot for me at this point.
Being that there is awareness that a deletion failure occurred, as we can see in the logs, and Octopus eternally ignores it afterwards, is a major gap. This issue is causing servers to run out of space due to an issue that the Octopus software is touted and designed to mitigate. I have seen several support cases out on the help portal reporting the same and similar issues.
I see room for improvement here!
Ideas to consider:
• Attempting to delete any packages that are outside of the retention window on subsequent deployments
• Logging on the actual deploy log, in the UI, (not just the raw log) that the deletion failed
o Alternatively give users some clue that the clean-up plainly failed.
• Provide a configurable option to fail the deployment, or succeed with warning, when the retention policy fails.
• A standalone step type to execute a retention policy clean-up (i.e. to use in a Run book, etc).
Hope to continue this dialog for a productive outcome!
This is certainly unusual behavior. I would like to present this to my team at our next meeting on Monday.
Do you, by chance, have any more examples of this occurring? If so, could you provide those raw logs? It also might be helpful to see the tentacle logs. Judging by the raw log, I doubt there is any new information, but I would like to see them just in case.
Thanks for your patience while we get this sorted out.
I wanted to update you to let you know that I have submitted your suggestions to our internal product team.
The support team actually discussed these suggestions in length in one of our meetings this week. We recognize that there is room for improvement when it comes to control over tentacle retention policies and clean-up.
We really appreciate the feedback as this helps us to continue to improve Octopus.
