Terminated deployment leads to further failures due to bootstrap file permisssions

Hi ,
Sorry for the late response . The steps that was cancelled was for deploying a windows Service . Installation directory of windows service in APP server was something like WindowsService\XXXService . This folder “XXXservice” contains all dlls and confiigs .Deploy.ps1 file also getting created inside this folder . Deployment staff deleted the folder “XXXService” and recreated it .While recreating the folder windows automatically inherited the access permissions from parent folder which is “WindowsService” here

I hit this issue just now in Octopus 3.4.11

I have a project that deploys prerequisites and one of the pre-reqs had hung so I cancelled the deployment.

Every subsequent retry yielded the ‘file in use’ issue. There is no anti-virus to speak of on the target machine.

Resolution: I deleted the root pre-reqs folder on the target machine and this cleared up the issue.

Good luck debugging this one–it’s odd.

We have also encountered this, however we have been able to track it down to the way octopus creates the file.

Octopus creates the file with the Hidden attribute, if you attempt to delete a hidden file visa .net (as octopus would on the redeploy) you get an System.UnauthorizedAccessException thrown.

See msdn: https://msdn.microsoft.com/en-us/library/system.io.filemode.aspx

“If the file already exists but is a hidden file, an UnauthorizedAccessException exception is thrown.”

What I’d like to know is why the file is created with the Hidden attribute in the first place. The post-deploy script file isn’t, only the bootstrap file.

We are also seeing this on Octopus 3.4.12. Here’s what I observed:

  1. This deployment step, when run again on the same release, results in the same folder being deployed to as the previous release (I have a custom installation directory)
  2. The step deploys to multiple machines and fails due to an issue on one machine
  3. At the point that the deployment is marked as failed (there is a red icon for failure on that deployment and the option to try again), I look at each of the machines’ status for that deployment step. Some machines are green (success), some are red (failure), and some are blue (in progress).
  4. If I redeploy, this step fails on the machines marked as blue still on the previous deployment.
  5. If I go to the machine(s) and delete the Bootstrap.PostDeploy.ps1 and PostDeploy.ps1 files, the prior deployment where the step was marked blue (in progress) is suddenly marked green (success)!
  6. If none of the machines have the Bootstrap.PostDeploy.ps1 file in this folder, the deployment succeeds.

My theory:

The previous deployment is somehow stuck on some of the machines at a moment before it removes the PostDeploy files (or on the step that verifies that they are removed, perhaps). When these files are removed, it releases some kind of hold that the previous deployment has on these files, causing that deployment process to succeed and allowing the next deployment to succeed.

Hi @Robert,

The previous deployment is somehow stuck on some of the machines at a moment before it removes the PostDeploy files (or on the step that verifies that they are removed, perhaps). When these files are removed, it releases some kind of hold that the previous deployment has on these files, causing that deployment process to succeed and allowing the next deployment to succeed.

Your theory is correct. The key here is finding out what process is holding this file. It can be a pretty big number of things, but the 2 usual culprits are:

  1. IIS beeing too slow to release the file. This is rather easy to workaround by adding a “stop site” step at the beggining, followed by a start-sleep -seconds 60 to make sure IIS has all the time it needs.

  2. Anti-virus software that might be analizsing the bootstrap file. AVs usually monitor files that were recently created, which is the case of the bootstrap files. The easiest way to workaround this one is to add the “custom install directory” to a white list on the AV so it doesn’t get scanned.

All that said, we did some work around this in 3.3.20 to make sure the process is more robust and the file doesn’t get locked up so often: https://github.com/OctopusDeploy/Issues/issues/2568

Hope that helps with the troubleshooting.

Best regards,
Dalmiro

Hi Dalmiro,

For option 2, unfortunately our custom install directory is still variable (changes per release), but if the same release is deployed, we encounter this problem.

Also, we are using Octopus 3.4.12, so perhaps there was a regression since 3.3.20?

Thanks,
Robert

Hi Robert,

The right way to troubleshoot this would be to use a tool like Sysinternal’s Handle to figure out what’s holding the file. That way we’ll know who to blame :slight_smile: (Octopus or the AV).

Thanks,
Dalmiro

There is a top-level “System” process which is holding onto the file. I’m wondering if this is related to Simon Laing’s comment above:

"Octopus creates the file with the Hidden attribute, if you attempt to delete a hidden file visa .net (as octopus would on the redeploy) you get an System.UnauthorizedAccessException thrown.

“See msdn: https://msdn.microsoft.com/en-us/library/system.io.filemode.aspx