Tasks not finishing

Octopus Server
1

OS Version: Microsoft Windows 10.0.17763
CLR Version: .NET 6.0.14
Octo Version: 2022.4 (Build 8471)
Tentacles: 6.2.50

Since upgrading to the latest build we have had tasks (Health Checks and Deployments) frequently not finish. The actual steps that it hangs on seem random; IIS deployment tasks, applying retention policies, dacpac deployments, etc all seem to hang. I turned on TRACE logging, including with halibut, but nothing interesting seems to show up; it just shows no further log entries written until we cancel (see below for example). This occurs both with our polling and listening tentacles. All health and integrity checks seem to pass without any issues. The only other change I can think of is our IT team is evaluating CrowdStrike but they tell me they added exemptions for the relevant Octo executables.

Any advice on how to diagnose this issue would be greatly appreciated!

An example of one of the task logs, where I canceled it ~16 hours after it started:

                    |   == Failed: Apply retention policy on Tentacles ==
14:16:19   Info     |     Apply retention policies...
05:58:47   Fatal    |     The step failed: The operation was canceled.
05:58:47   Info     |     The operation was canceled.
                    |   
                    |     == Warning: DevWeb10 ==
                    |     
                    |       Warning: Applying retention policy using 2 Items to set Environments-21/Projects-323/Step-Deploy API to IIS/Machines-401/<default>
14:16:19   Verbose  |         Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-104997
14:16:19   Verbose  |         Executable directory is C:\Windows\system32\WindowsPowershell\v1.0
14:16:19   Verbose  |         Executable name or full path: C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe
05:58:46   Verbose  |         Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory 'E:\Octopus\Work\20230228191619-104997-1097' using 'OEM United States' encoding running as 'OURDOMAIN\gMSA_OctoDev$'
05:58:46   Verbose  |         Swallowing OperationCanceledException while waiting for last of the process output.
05:58:46   Verbose  |         Swallowing OperationCanceledException while waiting for last of the process output.
05:58:46   Verbose  |         Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in E:\Octopus\Work\20230228191619-104997-1097 exited with code -1
05:58:46   Warning  |         Bootstrapper did not return the bootstrapper service message
                    |       
                    |       Warning: Applying retention policy using 2 Items to set Environments-21/Projects-323/Step-Deploy WebUI to IIS/Machines-401/<default>
05:58:46   Verbose  |         Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-104997
05:58:46   Info     |         This task was canceled before it could start. Task (error - task not found) is still running.
05:58:47   Info     |         Script execution canceled.
05:58:47   Warning  |         Bootstrapper did not return the bootstrapper service message
2

Forgot to mention: I saw elsewhere something about making sure all tasks execute in the same “Space” - our setup only has one default space used.

4

Hi @jgarner,

Thank you for contacting Octopus Support.

Typically for Bootstrapper did not return the bootstrapper service message errors, this is a result of antivirus/endpoint protection.

Here is our documentation showing an example set of folders to whitelist. Depending on your installation configuration, you may need to adjust the target directories.

If you have any additional questions or if we can assist with anything else, please let us know.

Best Regards,
Donny

5

This resolved the issue, thanks!

1 Like
6

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.