Hi @kennethgarza,
As @clare.martin and @britton.riggs have previously said, thankyou for your patience as we try to diagnose these problems.
I’m one of the engineers that looks after Calamari and have been working through different approaches we might be able to take to track down the cause of this problem. I’ve just done a bit of a deep-dive into Windows file permissions, as my best hypothesis on what’s going wrong.
I’ve managed to replicate the exact error that we’re seeing in your environment (Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'Autofac, Version=4.8.0.0, Culture=neutral, PublicKeyToken=17863af14b0044da'. Access is denied.), however I’ve done it by explicitly setting some “deny” permissions on autofac.dll for the account I’m running calamari.exe under.
Given that your failures are transient, I doubt that this is the exact case, but I strongly suspect that something in your environment is causing some kind of permissions failure in a transient manner. It could be, for example, something to do with domain connectivity and Windows being able to resolve the account permissions properly, or it could be to do with another process exclusively locking the file.
Through my exploration, I discovered the guidance we gave you around looking into the event logs was a little incomplete. By default, Windows will not log Security Audit events for file access. You need to do some extra configuration to opt-in to collecting that info (otherwise the Event Logs would be littered with audit logs for every single file access event that happens, which would be a lot). When I enabled this audit logging locally, and then reproduced the error you’re seeing via setting deny permissions directly, I was able to see a corresponding Audit Failure in the Security logs. I’m hopeful that if you can configure audit logging as well, we may finally be able to uncover exactly what is causing the Access Denied error you’re seeing.
To do that, you firstly need to enable auditing of object access events. The way you do this will vary slightly depending on your domain/machine configuration. On my machine, I was able to do this by updating Local Security Policies > Security Settings > Local Policies > Audit Policy > Audit object access from “No auditing” to “Failure”. It looks like this on Windows 11:
The second step is to “opt in” to auditing for a specific scope of files that you want to monitor and a specific account. We’ll want to do this for your Tentacle home directory (which from the logs looks to be c:\octopus) and all subfolders/files, against the account you run your Tentacle as (from the logs, FPK\svc-octo-deploy).
There’s a good visual run-through of how to configure both of those parts on this blog post (note, that’s a commercial company’s blog, so advertisement for their product follows 
)
Once that’s done, you should start seeing Audit Failure events posted to the Windows Logs > Security event log, for any file access failures under the Tentacle directory for your Octopus service account. You’ll need to wait until the next failure, and then have a look. You can see an example from my local here:
Those near-unintelligible strings in the Access Reasons section are basically saying that access was denied because of an explicit “Deny” permission set on the file trying to be accessed for the given account.
Remote debugging of these kinds of problems, with the many different variables and things that can occur in a complex environment like this, is always tricky. I’m hopeful that with these steps and extra audit logs, we might be able to uncover exactly what is preventing Calamari from loading up successfully, and fix the root cause of this problem. 
that we can figure it out together.
Cheers,
Matt.
