Terminated deployment leads to further failures due to bootstrap file permisssions

Hello, we’ve already encountered several times the situation when deployment can’t proceed because it fail to do smth with the bootstrap file. In deployment logs it looks like this:

System.UnauthorizedAccessException: Access to the path ‘xxxxx\Bootstrap.PostDeploy.ps1’ is denied.

Tentacle is running under the local system account.

We encounter this error after terminating previous deployment, e.g. after deployment is hanging for a long time. I can’t say we face this each time we terminate deployment, but we faced it.

Hi Dmitry,

Thanks for reaching out. Could you send us that full deployment log and also tell us which version of Octopus are you running?

http://docs.octopusdeploy.com/display/OD/Get+the+raw+output+from+a+task

Thanks,
Dalmiro

Hi Dalmiro, I’ve sent one of log files (removed some info from it) to support@octopusdeploy.com. We’ll wait for your conclusion. Thanks!

Hi Dmitry,

Thanks for sending the logs via email.

The message Access to the path '[File]' is denied. usually has one main culprit: Antivirus software. They tend to be configured to scan new files, locking them up in the process. That bootstrap file gets created by Octopus on each deployment, and that might be triggering an unexpected scan.

Can you confirm if you have an anti-virus like that, and if that is the case, try to add the Octopus folder and your Deployment directories as exceptions for the scanning? That should help with this problem.

Best regards,
Dalmiro

Hi @Dalmiro, I can say that we are getting this lock on bootstrap file each time we cancel deployment. This happens each time if we cancel it. I believe we don’t have antivirus software on servers :slight_smile: And also this error doesn’t occur in other cases. Only after cancelation.

Hi Dmitry,

I consulted this with the team, and we couldn’t figure out something that Octopus might be doing when you cancel the deployment that might cause this behavior.

A few questions came up though:

  • It sounds like you are canceling deployments very often. Why is that? Is there something that could be changed in the process so you don’t have to do it that often? Usually people just let the deployment finish (and fail).

  • Bootstrap.ps1 is a file we create during the deployment, which is a bundle of Octopus scripts + your scripts. I’m wondering if in your scripts you are doing something that takes a while to be canceled, thus locking the file. For example: Lets say you are running a SQL script that does a lot updates in your database during your deployment. If you cancel the deployment, that script will still be running for a couple of seconds, locking up the bootstrap.ps1 file until SQL decides to finally kill the process and free the file. So the question would be: is there something you are doing during in your deploy script that might be persisting after the cancellation?

Regards,
Dalmiro

Hi ,

I also facing exactly the same issue . After cancelling the deployment then next deployment is getting failed exactly on same step . This is the exception that I am also getting

System.UnauthorizedAccessException: Access to the path “X\XXX\Bootstrap.Octopus.Action.CustomScripts.Deploy.ps1’ is denied” .

Thanks
Sanoop

Hi Sanoop,

Thanks for jumping in. I’m gonna ask you the same thing I asked Dmitry:

  • Bootstrap.ps1 is a file we create during the deployment, which is a bundle of Octopus scripts + your scripts. I’m wondering if in your scripts you are doing something that takes a while to be canceled, thus locking the file. For example: Lets say you are running a SQL script that does a lot updates in your database during your deployment. If you cancel the deployment, that script will still be running for a couple of seconds, locking up the bootstrap.ps1 file until SQL decides to finally kill the process and free the file. So the question would be: is there something you are doing during in your deploy script that might be persisting after the cancellation?

We are looking for ways to reproduce this to be able to nail down if its a bug or an enviromental thing. Anything you can tell use about your Custom deploy script will be helpful.

Thanks,
Dalmiro

Hi Dalmiro ,

Step that failed was for deploying a windows service . Script was supposed to do below

  1. Transform Config
  2. Deploying the files
    3.Start the windows service (Since it is an existing service .Otherwise it will install service)

Thanks
Sanoop

Sanoop - I forgot to also ask you which version of Octopus are you running. Apologies for that.

  1. The reason why we cancel deployments is some flaws in deployment scripts, e.g.they hang or execute too long. We cancel it and then we get this file lock. We’re fixing scripts after all, but anyway we have moments where we get file locked, which causes deploy staff use g2m or rdp to go to the server and manually delete the file. This wasn’t an issue (we’ve never seen it) for 2.x, but for 3.x it’s a problem.

  2. I guess reasons might be different.
    2.1. Once we had regedit opened as a window (our bug), which prevented deployment from stopping. But when we hit “cancel”, regedit was stiil opened somewhere and it held deployment process.
    2.2. The other time we had some long running script, which didn’t do anything with external requests (no SQL queries), but rather was a result of badly implemented search logic. Deployment took 20m instead of 1m. This made deployment staff nervous and they cancelled deployment, which caused .boostrap file to lock.

Perhaps there were more, I just don’t recall now. But the thing is, we’ve surely had deployment script flaws with 2.x as well, and we surely cancelled deployments before (here we use Octopus for 2 years), but we’re getting bootstrap file lock only when we migrated to 3.x.

Octopus version is 3.3.6

I’ve been trying to reproduce this for quite some time now, but I couldn’t get that file lock even once.

Dmitry - Do you think you can schedule some time for a call on the link below so you can show me how to reproduce this in your instance?

https://calendly.com/dalmirogranias

Sanoop - The reason why I’m asking Dmitry and not yourself its because he seems to be in timezone closer to mine. If we find anything useful about this issue, I’ll make sure to let you know on this thread.

Regards,
Dalmiro

Hi ,

Do you know a work around for this issue . Deploy staff deleted the locked file manually(.Ps1) . But It still throwing the same error

Thanks
Sanoop

Hi Sanoop,

That ps1 file gets created with every deployment. So even if you delete it, It’ll be recreated during the next deployment.

I’ve discussed this with the team and at this point the only thing we can think of trying is to use a software that checks the file’s handles (to see what is locking it up) during the deployment, and start from there.

Do you think you can check what is locking that file using http://lockhunter.com/ ?

Dmitry - If possible also try using Lockhunter before scheduling our call to see if you can find anything.

Dalmiro

Another thing definitely worth trying is to delete the Tentacle instance, unisntall the Tentacle software and re-install it.

Not the prettiest approach, but since we can’t reproduce it on our end to debug it, we gotta try everything we can.

I checked with lock hunter .I couldn’t see anything locking the script file .I was able to delete the file .So I don’t think any external process locks it

Hi @Sanoop - Given that Dmitry hasn’t replied yet, would you be willing to have a call to see what we can figure out about this together? If you are, let me know if any of the available times in the link below work for you.

https://calendly.com/dalmirogranias

Thanks,
Dalmiro

It seems like cancelling the deployment some how alter access permission of installation directories (though windows didn’t show any issue with access permission) .Deploy staff deleted the installation directories (only for windows service where deployment failed) and recreated it . It works!

Hi Sanoop,

That’s good to hear. Do you think you could post a more detailed explanation of what did your staff do to workaround this? Just a brief example that anyone (specially Dmitry) that runs into this thread can follow from beginning to end.

Thanks,
Dalmiro