Latest Tentacle version - The process cannot access the file

matthewericford · 11 November 2021 16:03

Hi,

I noticed this issue yesterday, but forgot to post! Using Choco I got the latest Tentacle installed on a server, but it failed. We rolled back one at a time and found v6.1.1056 was the last working version. We realised that Choco is ahead of Octopus.com (bizarrely ), but today we’ve just done another install using the latest from Octopus.com, which is now the bugged version.

It’s 50:50 if it errors seemingly, but here is one of the failing health checks:

                    |   == Failed: Check worker: OctoWorker07 ==
15:46:41   Verbose  |     Performing health check on machine
15:46:41   Verbose  |     Script isolation level: NoIsolation
15:46:42   Verbose  |     Executable directory is C:\Windows\system32\WindowsPowershell\v1.0
15:46:42   Verbose  |     Executable name or full path: C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe
15:46:42   Verbose  |     No user context provided. Running as current user.
15:46:42   Verbose  |     Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory 'C:\Octopus\Work\20211111154641-526575-1' using 'OEM United States' encoding running as 'NT AUTHORITY\SYSTEM' with the same environment variables as the launching process
15:46:47   Verbose  |     Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20211111154641-526575-1 exited with code 0
15:46:47   Verbose  |     Exit code: 0
15:46:47   Verbose  |     Script isolation level: NoIsolation
15:46:48   Verbose  |     Executable directory is C:\Windows\system32\WindowsPowershell\v1.0
15:46:48   Verbose  |     Executable name or full path: C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe
15:46:48   Verbose  |     No user context provided. Running as current user.
15:46:48   Verbose  |     Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory 'C:\Octopus\Work\20211111154647-526575-2' using 'OEM United States' encoding running as 'NT AUTHORITY\SYSTEM' with the same environment variables as the launching process
15:46:48   Info     |     Host Name: OctoWorker07
15:46:48   Info     |     Running As: WORKGROUP\SYSTEM (Local Administrator: True)
15:46:48   Info     |     Running Tentacle version 6.1.1132
15:46:48   Info     |     Tentacle communication uses a 'sha256RSA' certificate
15:46:49   Info     |     Drive C: has 115 GB available
15:46:49   Info     |     Drive D: has 7 GB available
15:46:58   Info     |     azure-cli                         2.30.0
15:46:58   Info     |     core                              2.30.0
15:46:58   Info     |     telemetry                          1.0.6
15:46:58   Info     |     Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
15:46:58   Info     |     Extensions directory 'C:\Windows\system32\config\systemprofile\.azure\cliextensions'
15:46:58   Info     |     Python (Windows) 3.8.9 (tags/v3.8.9:a743f81, Apr  6 2021, 13:22:56) [MSC v.1928 32 bit (Intel)]
15:46:58   Info     |     Legal docs and information: aka.ms/AzureCliLegal
15:46:58   Info     |     Your CLI is up-to-date.
15:46:58   Error    |     Please let us know how we are doing: https://aka.ms/azureclihats
15:46:58   Error    |     and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy
15:46:59   Info     |     DotNet assemblies loaded and working
15:46:59   Verbose  |     Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20211111154647-526575-2 exited with code 0
15:46:59   Fatal    |     The process cannot access the file 'C:\Octopus\Work\20211111154647-526575-2' because it is being used by another process.
                    |     Server exception:
                    |     System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The process cannot access the file 'C:\Octopus\Work\20211111154647-526575-2' because it is being used by another process.
                    |     at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
                    |     at System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound, WIN32_FIND_DATA& data)
                    |     at System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive, Boolean checkHost)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.<>c__DisplayClass43_0.<PurgeDirectoryAsync>b__1()
                    |     at System.Threading.Tasks.Task.Execute()
                    |     --- End of stack trace from previous location where exception was thrown ---
                    |     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    |     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.<PurgeDirectoryAsync>d__43.MoveNext()
                    |     --- End of stack trace from previous location where exception was thrown ---
                    |     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    |     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.<DeleteDirectory>d__14.MoveNext()
                    |     --- End of inner exception stack trace ---
                    |     at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
                    |     at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.DeleteDirectory(String path, DeletionOptions options)
                    |     at Octopus.Tentacle.Services.Scripts.ScriptService.CompleteScript(CompleteScriptCommand command)
                    |     --- End of inner exception stack trace ---
                    |     at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
                    |     at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
                    |     at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
                    |     at Halibut.ServiceModel.ServiceInvoker.Invoke(RequestMessage requestMessage)
                    |     at Halibut.Transport.Protocol.MessageExchangeProtocol.InvokeAndWrapAnyExceptions(RequestMessage request, Func`2 incomingRequestProcessor)
15:46:59   Verbose  |     The process cannot access the file 'C:\Octopus\Work\20211111154647-526575-2' because it is being used by another process.
                    |     Server exception:
                    |     System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The process cannot access the file 'C:\Octopus\Work\20211111154647-526575-2' because it is being used by another process.
                    |     at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
                    |     at System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound, WIN32_FIND_DATA& data)
                    |     at System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive, Boolean checkHost)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.<>c__DisplayClass43_0.<PurgeDirectoryAsync>b__1()
                    |     at System.Threading.Tasks.Task.Execute()
                    |     --- End of stack trace from previous location where exception was thrown ---
                    |     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    |     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.<PurgeDirectoryAsync>d__43.MoveNext()
                    |     --- End of stack trace from previous location where exception was thrown ---
                    |     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    |     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.<DeleteDirectory>d__14.MoveNext()
                    |     --- End of inner exception stack trace ---
                    |     at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
                    |     at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.DeleteDirectory(String path, DeletionOptions options)
                    |     at Octopus.Tentacle.Services.Scripts.ScriptService.CompleteScript(CompleteScriptCommand command)
                    |     --- End of inner exception stack trace ---
                    |     at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
                    |     at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
                    |     at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
                    |     at Halibut.ServiceModel.ServiceInvoker.Invoke(RequestMessage requestMessage)
                    |     at Halibut.Transport.Protocol.MessageExchangeProtocol.InvokeAndWrapAnyExceptions(RequestMessage request, Func`2 incomingRequestProcessor)
                    |     Halibut.HalibutClientException: The process cannot access the file 'C:\Octopus\Work\20211111154647-526575-2' because it is being used by another process.
                    |     Server exception:
                    |     System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The process cannot access the file 'C:\Octopus\Work\20211111154647-526575-2' because it is being used by another process.
                    |     at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
                    |     at System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound, WIN32_FIND_DATA& data)
                    |     at System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive, Boolean checkHost)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.<>c__DisplayClass43_0.<PurgeDirectoryAsync>b__1()
                    |     at System.Threading.Tasks.Task.Execute()
                    |     --- End of stack trace from previous location where exception was thrown ---
                    |     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    |     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.<PurgeDirectoryAsync>d__43.MoveNext()
                    |     --- End of stack trace from previous location where exception was thrown ---
                    |     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    |     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.<DeleteDirectory>d__14.MoveNext()
                    |     --- End of inner exception stack trace ---
                    |     at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
                    |     at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
                    |     at Octopus.Shared.Util.OctopusPhysicalFileSystem.DeleteDirectory(String path, DeletionOptions options)
                    |     at Octopus.Tentacle.Services.Scripts.ScriptService.CompleteScript(CompleteScriptCommand command)
                    |     --- End of inner exception stack trace ---
                    |     at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
                    |     at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
                    |     at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
                    |     at Halibut.ServiceModel.ServiceInvoker.Invoke(RequestMessage requestMessage)
                    |     at Halibut.Transport.Protocol.MessageExchangeProtocol.InvokeAndWrapAnyExceptions(RequestMessage request, Func`2 incomingRequestProcessor)
                    |     at Halibut.ServiceModel.HalibutProxy.EnsureNotError(ResponseMessage responseMessage)
                    |     at Halibut.ServiceModel.HalibutProxy.Invoke(MethodInfo targetMethod, Object[] args)
                    |     --- End of stack trace from previous location ---
                    |     at System.Reflection.DispatchProxyGenerator.Invoke(Object[] args)
                    |     at generatedProxy_1.CompleteScript(CompleteScriptCommand )
                    |     at Octopus.Server.Orchestration.Targets.Tentacles.ScriptObserver.Finish(ScriptTicket ticket, Int64 next, ScriptOutputFilter output) in ./source/Octopus.Server/Orchestration/Targets/Tentacles/ScriptObserver.cs:line 70
                    |     at Octopus.Server.Orchestration.Targets.Tentacles.ScriptObserver.ObserveUntilComplete(ScriptTicket ticket, ITaskLog taskLog, Action`1 collectArtifacts) in ./source/Octopus.Server/Orchestration/Targets/Tentacles/ScriptObserver.cs:line 30
                    |     at Octopus.Server.Orchestration.Targets.Tentacles.TentacleRemoteEndpointFacade.ExecuteCommand(StartScriptCommand command, ITaskLog taskLog) in ./source/Octopus.Server/Orchestration/Targets/Tentacles/TentacleRemoteEndpointFacade.cs:line 59
                    |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ExecutionTargets.TentacleExecutionTarget.Execute(ScriptCollection bootstrapperScripts, IReadOnlyList`1 boostrapperArguments, IReadOnlyList`1 files, Nullable`1 forceIsolationLevel, Boolean raw, ITaskLog taskLog, Nullable`1 isolationMutexTimeout) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ExecutionTargets/TentacleExecutionTarget.cs:line 67
                    |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.ExecuteRawScript(ScriptCollection scripts, IReadOnlyList`1 files, Boolean isRaw, ITaskLog taskLog, Nullable`1 isolationMutexTimeout, ExecutionIsolation isolationLevel) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ImmediateExecutor.cs:line 88
                    |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.CommandBuilders.ScriptCommandBuilder.Execute(ITaskLog taskLog) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/CommandBuilders/ScriptCommandBuilder.cs:line 67
                    |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.InvokeActionHandler(Machine target, ActionHandlerInvocation actionHandler, ActionAndTargetScopedVariables actionAndTargetScopedVariables, IExecutor executor, TargetManifest targetManifest, ITaskLog taskLog) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 171
                    |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation actionHandler, ITaskLog taskLog, VariableCollection variables) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 57
                    |     at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.TentacleMachineHealthController.CheckMachineHealth(Machine machine, ITaskLog taskLog)
                    |     at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.MachineHealthController.CheckHealth(Machine machine, ITaskLog taskLog) in ./source/Octopus.Server/Orchestration/ServerTasks/HealthCheck/Controllers/MachineHealthController.cs:line 48
                    |     at Octopus.Server.Orchestration.ServerTasks.HealthCheck.HealthCheckService.PerformHealthCheck(Machine machine, ITaskLog machineTaskLog, ITaskLog serverTaskLog, IHealthResultCollator healthResultCollator, CancellationToken cancellationToken, ExceptionHandling exceptionHandling, Action`2 customAction) in ./source/Octopus.Server/Orchestration/ServerTasks/HealthCheck/HealthCheckService.cs:line 86
                    |     Octopus.Server version 2021.3.8030 (2021.3.8030)
15:46:59   Verbose  |     Recording health check results

matthewericford · 11 November 2021 16:05

Also if it helps, this box is a brand new W2022 DC edition virtual machine hosted in Azure with just Az and the Tentacle installed.

finnian.dempsey · 12 November 2021 01:20

Hi @matthewericford,

Thanks for getting in touch!

Unfortunately I’m having trouble replicating this behaviour on my end, so I’d just like to confirm your process to make sure we are on the same page.

Using the Windows 2022 Image in Azure:

and installing Octopus Tentacle v3.1.1132 via choco:

You receive the above error message when performing a health check against the newly added tentacle, approximately 50% of the time. When you downgrade the tentacle version to 6.1.1056 then you are able to pass health checks without issue. Was there an error when you installed the Tentacle or just when performing a health check?

The error message suggests that there is a potential file lock occurring. If you haven’t rebooted the target instance recently I would definitely recommend trying that first.

Following that it would be worth checking if there are any running tasks against that instance. Navigate to Tasks → Show Advanced Filters and search for any ‘Executing’ or ‘Running’ tasks.

Feel free to let me know if I’ve misunderstood anything or you have any questions!

Best Regards,

matthewericford · 13 November 2021 09:23

Hi Finnian,

The error was on health check and doing some tasks too. It wasn’t every time, but I spooled up 5 identical setup machines and 2 had the issue. I also had set up Windows Defender to ignore both Octopus folders, so it wasn’t that locking it. The only thing I didn’t disable was drive indexing.

I rolled all back to the older version and not had a problem since.

finnian.dempsey · 15 November 2021 01:56

Hi @matthewericford,

Thanks for confirming that, it is strange indeed!

Drive indexing does seem like it could potentially be the culprit however I’m not completely convinced. We had a similar report of an Access Denied issue being caused by Watchdog. If you aren’t using Watchdog then it’s unlikely to be the same cause for you.

Just confirming when you were rolling back one version at a time, you had the issue with v6.1.1062?

I’ve reached out internally about the changes between those versions that could be causing this and will let you know if we require any further info as we investigate.

Feel free to let me know if you have any questions, thanks again for raising this!

Best Regards,

matthewericford · 15 November 2021 09:02

Hi Finnian,

Looking at the code compare, the Hardener is definitely in the right area as it’s around setting access control to directories, but it’s the actual deletion that’s failing.

Coincidentally, I’ve just upgraded my Tentacle by mistake (thanks Choco) and it’s borked again on the first install! It’s on 6.1.1119 now and it failed to do the cleanup after (successfully) running the script step.

It’s not the Watchdog either, as the VM’s we created didn’t have that installed at all.

Thanks

Matt

Justin_Walsh · 15 November 2021 14:46

Hey Matt,

Just jumping in for Finnian here, as he’s currently offline as a member of our Australian-based team.

The issue we were seeing on the other report of this was stemming from a Start-Process being called in the deployment, which does start the child process asynchronously. This returns control back to the Octopus deployment immediately, and then Octopus will think it’s done, and try to clean up the working directory. If the child process is still running, it will still be locking those files (as that’s where the command was run from), causing the Access Denied errors. We fixed it in the other report by appending the -Wait arg to the Start-Process so that it will wait for the process to complete before returning control back to the deployment.

I wonder if this could be what’s happening with your deployments too? The intermittent nature could just be based on system load or capability at execution time. We will continue to investigate this on our end, but hopefully, this gets you unstuck in the meantime.

system · 16 December 2021 14:46

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.