We are running regularly into an issue with deployment, where the process gets stuck when deploying a package. Deploying package: /home/docker/.octopus/OctopusServer/Files/Package@S1.0.431@3785128AA7D4204A948FEEF4DF9BF550.nupkg June 17th 2020 01:07:35 Warning Lock file existed but was not readable, and has existed for longer than lock timeout. Taking lock. June 17th 2020 01:09:35 Warning Lock file existed but was not readable, and has existed for longer than lock timeout. Taking lock.
This process seems to go on indefinitely until we kill the process.
What is Octopus doing at this step and what causes this issue?
Is there a way to setup a max time? We use a burstable instance on AWS EC2 and the process is CPU heavy and consumes all the credits if we don’t spot its running.
00:05:38 Verbose | Another process is using the deployment journal
We’ve seen this issue before where Octopus was running into a permissions issue in the /tmp directory. I suggest double-checking that a different process is not interfering in this case.
If that doesn’t turn anything up, let me know and we’ll dig deeper.
It looks like it is Octopus tasks that are getting orphaned and keeping the file locked. When a deploy got stick in the loop
Cancelled the task in the UI, so that there were no running processes on that deployment target
SSH into the machine
I could see the .lck file existed in the /temp directory. Trying to forcibly remove it failed, so I checked the running processes. There was a mono task running deploy-package. Which was surprising as the UI said all tasks were cancelled. I manually killed that and I could then remove the lock file and allow deployments to resume.
However I don’t know how I can automate this set of actions, the taking a lock seems to have no timeout and just goes on forever.