IIS Deployment Hangs Indefinitely

I’m experiencing an issue where occasionally the IIS deploy step in the process will hang indefinitely.
Looking at the raw logs did not provide much help.

How can I start tracking down the cause of why this step hangs?

09:17:31   Verbose  |         Acquired mutex Global\Octopus-IIS-Metabase-Mutex
09:17:31   Info     |         The required certificate binding is already in place
09:17:31   Verbose  |         Acquired mutex Global\Octopus-IIS-Metabase-Mutex
09:17:31   Verbose  |         Loading Application pool
09:17:31   Info     |         Application pool "AppPool" already exists
09:17:31   Verbose  |         Acquired mutex Global\Octopus-IIS-Metabase-Mutex
09:17:31   Info     |         Set application pool identity: ApplicationPoolIdentity
09:17:31   Verbose  |         Acquired mutex Global\Octopus-IIS-Metabase-Mutex
09:17:33   Info     |         Set .NET framework version: v4.0
09:17:33   Verbose  |         Acquired mutex Global\Octopus-IIS-Metabase-Mutex
09:17:33   Verbose  |         Loading Site
09:17:33   Info     |         Site "AppPool" already exists
09:17:33   Verbose  |         Acquired mutex Global\Octopus-IIS-Metabase-Mutex
09:17:33   Verbose  |         Loading Site
09:17:33   Info     |         Application pool "AppPool" already assigned to "IIS:\Sites\AppPool"
09:17:33   Verbose  |         Acquired mutex Global\Octopus-IIS-Metabase-Mutex
09:17:33   Info     |         Setting physical path of IIS:\Sites\AppPool to X:\Applications\1.0.0
09:17:33   Verbose  |         Acquired mutex Global\Octopus-IIS-Metabase-Mutex
09:17:33   Info     |         Comparing existing IIS bindings with configured bindings...
09:17:33   Info     |         Looks OK
09:17:33   Info     |         Bindings are as configured. No changes required.
09:17:33   Verbose  |         Acquired mutex Global\Octopus-IIS-Metabase-Mutex
09:17:33   Info     |         Anonymous authentication enabled: True
09:17:33   Info     |         Applied configuration changes to section "system.webServer/security/authentication/anonymousAuthentication" for "MACHINE/WEBROOT/APPHOST/AppPool" at configuration commit path "MACHINE/WEBROOT/APPHOST"
09:17:33   Verbose  |         Acquired mutex Global\Octopus-IIS-Metabase-Mutex
09:17:33   Info     |         Basic authentication enabled: False
09:17:33   Info     |         Applied configuration changes to section "system.webServer/security/authentication/basicAuthentication" for "MACHINE/WEBROOT/APPHOST/AppPool" at configuration commit path "MACHINE/WEBROOT/APPHOST"
09:17:33   Verbose  |         Acquired mutex Global\Octopus-IIS-Metabase-Mutex
09:17:33   Info     |         Windows authentication enabled: False
09:17:33   Info     |         Applied configuration changes to section "system.webServer/security/authentication/windowsAuthentication" for "MACHINE/WEBROOT/APPHOST/AppPool" at configuration commit path "MACHINE/WEBROOT/APPHOST"
09:17:36   Info     |         IIS configuration complete
14:12:10   Verbose  |         Swallowing OperationCanceledException while waiting for last of the process output.
14:12:10   Verbose  |         Swallowing OperationCanceledException while waiting for last of the process output.
14:12:10   Verbose  |         Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in X:\Octopus\Work\20201112141712-202701-1250 exited with code -1
14:12:10   Verbose  |         Updating manifest with output variables
14:12:10   Verbose  |         Updating manifest with action evaluated variables
14:12:10   Fatal    |         The remote script failed with exit code -1

Hi @wbagdon1,

First of all, welcome to the Octopus forums!

Thanks for reaching out.

It looks like it finished all of the IIS stuff. Do you have any post-deploy custom scripts that are running in that step?

Please let me know.

Thanks,
Jeremy

Yes, I do have a post deploy powershell script.
However, I would expect to see it being called in the raw logs like I do when we have a successful deployment, for example:

14:13:02   Verbose  |         Deleting 'X:\Applications\1.0.0\Octopus.Features.IISWebSite_BeforePostDeploy.ps1'
14:13:02   Verbose  |         Executing 'X:\Applications\1.0.0\Octopus.Action.CustomScripts.PostDeploy.ps1'

Hi,

What’s the current version of Octopus you’re running?

Thanks,
Jeremy

I’m on 2020.4.6

Hi,

That’s pretty recent.

Since it’s intermittent, would you be able to check the tentacle and server logs at the corresponding time to see if there are any issues? It might also be helpful to check event viewer on the machine for the same timeframe.

Is there any antivirus running on the machine that may be interfering? That can also be intermittent.

Please let me know.

Thanks,
Jeremy

Yeah, I’ve checked those logs and event viewer with no luck.
We do have AV running, so that’s a possibility.

Although if it was the anti-virus I wouldn’t expect Octopus to be hung for hours

Hi,

Would you be able to whitelist powershell and all octopus folders and executables and see if the problem resolves?

Please let me know what you think.

Thanks,
Jeremy