I am trying to deploy a package using the default built-in “Deploy a package” step template, but it intermittently fails when it attempts to copy a temporary powershell script from the Octopus application directory that has already been cleaned up. This error occurs during the process of copying files to the installation directory:
Copying package contents to 'D:\<installation directory>'
System.IO.FileNotFoundException: Could not find file 'D:\Octopus\Applications\...\...\...\Bootstrap.Octopus.FunctionAppenderContext.ps1'.
File name: 'D:\Octopus\Applications\...\...\...\Bootstrap.Octopus.FunctionAppenderContext.ps1'
Thanks for getting in touch! We have some thoughts on what could be causing this. To confirm, are you using a custom installation directory for this project? If so, it looks like the custom install directory is perhaps configured to purge the installation directory. If this is the case, you should be able to work around it by changing the custom installation directory to another location.
The other possible cause we considered was Anti-Virus, which we often see causing file not found issues during deployment, though the intermittent failure doesn’t really support that.
Don’t hesitate to let me know if you have any questions or further thoughts here.
The project is configured to use a custom installation directory, but the location is different on every deployment since it’s configured with a variable that uses the release version as part of the path. As a result, every deployment is to a “new” location, but it still has this issue.
Just jumping in for Daniel as he’s finished for the day. You’re right - a unique location should mean that the deployment folder is still available.
Intermittent failures are notoriously tricky to identify but I’m going to re-suggest what Daniel said regarding AV and offer a couple of other suggestions.
So first of all - please check your Anti-virus logs to see if it is accessing those files around the same time as the deployment. A reliable method, (although I don’t like recommending it, for obvious reasons) if the AV log files aren’t granular enough, is to disable AV for a short period of time to see if the issue is rectified. The AV could be checking for new files every x seconds/minutes - and the reason you are seeing intermittent issues is that the deployment starts and finishes before the AV does it’s next check.
Next, you should see if there is a pattern to your intermittent issues. For example, I noticed that the files reporting in the error are located on the D: drive. If the D: drive is relatively slow storage, and the issue only appears when the storage hasn’t been accessed for awhile - perhaps your facing some timeout issues, as it takes the storage too long to spin up.
On the other side of the coin - you can also get a similar issue if the access to the D: drive is too slow due to congestion (Network or Disk I/O) due to too much activity.
It could simply be some packet loss between the worker and the storage as well. Perhaps a few ping tests will help uncover something more nefarious.
Please let me know if that puts you on the right track or if those avenues are a dead end.
Unfortunately I have no access or permission to do anything with the AV, not even look at the logs.
Changing the Octopus application directory to a different disk also did not resolve the issue.
The problem is only on the Bootstrap.Octopus.FunctionAppenderContext.ps1 file. This file is not part of the package, rather it is created by Octopus in the application directory during the deployment along with other temporary files, such as PreDeploy.ps1. Octopus then deletes these scripts around 2-3 seconds later after it finishes running them. The issue is that after cleaning up the file, it still attempts to copy it to the custom installation directory.
I previously had a very similar issue to this a few months ago, although the file in question that time was the PreDeploy.ps1 rather than the FunctionAppenderContext.ps1. Originally I solved this by compiling a “custom” version of Calamari based on the publicly available repository at GitHub - OctopusDeploy/Calamari: | Public | The core deployment scripts and tools used by Octopus Deploy. The “custom” change was to explicitly exclude any files named “PreDeploy.ps1” from the copy step. However, we have recently updated Octopus to 2022.3+, and I have not yet attempted a similar fix due to this:
Octopus Server 2020.3+: Using a custom version of Calamari may not work
Calamari is currently being filleted into Sashimi. Due to the architectural changes involved in this transformation, using a custom version of Calamari with Octopus Server version 2020.3+ may not work. Please get in touch with support@octopus.com if this affects you, to help us make decisions about how we can support custom implementations of deployment steps.
Thanks for following up, and sorry to hear you’re hitting all of these complications in trying to deploy your project.
I’m not sure if it’s been suggested in the past, but given all of the constraints you have with your environment and AV configuration, I was wondering if using our Offline Package Drop feature might work better for your purposes?
You could use this to run the deployment and create an artifact, and then download that artifact and deploy it from a fixed location. The script would run directly beneath the user account you’re logged in under. This would bypass Calamari and basically run the deployment as Tentacle would.
Another option would be to set up a share on the target and use that for the package drop folder destination, as that would keep at least part of your process automated in getting the deployment to the target.
Let me know if you think that might work for you or if you have any additional questions regarding this feature.