Deployment hangs in Deploy to IIS step

Legacy Execution
1

Hi, today we experienced an issue where the deployment on 1 of our production servers was hanging during a deployment.
after 1.5 hour we canceled the deployment and ran it again.
What could be the cause of this hanging deployment?

RAW LOG:
11:57:31 Verbose | Name Value
11:57:31 Verbose | ---- -----
11:57:31 Verbose | PSVersion 4.0
11:57:31 Verbose | WSManStackVersion 3.0
11:57:31 Verbose | SerializationVersion 1.1.0.1
11:57:31 Verbose | CLRVersion 4.0.30319.42000
11:57:31 Verbose | BuildVersion 6.3.9600.19170
11:57:31 Verbose | PSCompatibleVersions {1.0, 2.0, 3.0, 4.0}
11:57:31 Verbose | PSRemotingProtocolVersion 2.2
11:57:31 Verbose | PowerShell Environment Information:
11:57:31 Verbose | OperatingSystem: Microsoft Windows NT 6.3.9600.0
11:57:31 Verbose | OsBitVersion: x64
11:57:31 Verbose | Is64BitProcess: True
11:57:31 Verbose | CurrentUser: NT AUTHORITY\SYSTEM
11:57:31 Verbose | MachineName: SERVER02
11:57:31 Verbose | ProcessorCount: 6
11:57:31 Verbose | CurrentDirectory: E:\DATA\Octopus\Applications\Production\AppName\201925.2.1-20190618-1
11:57:31 Verbose | CurrentLocation: E:\DATA\Octopus\Applications\Production\AppName\201925.2.1-20190618-1
11:57:31 Verbose | TempDirectory: C:\Windows\TEMP
11:57:31 Verbose | HostProcessName: powershell
11:57:31 Verbose | TotalPhysicalMemory: 16776692 KB
11:57:31 Verbose | AvailablePhysicalMemory: 8762788 KB
11:57:33 Info | Making sure a Web Application “/AppName201925b1” is configured as a child of “Portal” at “E:\DATA\Octopus\Applications\Production\AppName\201925.2.1-20190618-1”…
11:57:33 Verbose | Acquired mutex Global\Octopus-IIS-Metabase-Mutex
11:57:33 Verbose | Looking for the parent Site “Portal” at “IIS:\Sites\Portal”…
11:57:33 Verbose | Acquired mutex Global\Octopus-IIS-Metabase-Mutex
11:57:33 Verbose | Loading Application pool
11:57:33 Info | Application pool “AppName201925b1” already exists
11:57:33 Verbose | Acquired mutex Global\Octopus-IIS-Metabase-Mutex
11:57:33 Info | Set application pool identity: SpecificUser
11:57:33 Verbose | Acquired mutex Global\Octopus-IIS-Metabase-Mutex
11:57:36 Info | Set .NET framework version: v4.0
11:57:36 Info | “/AppName201925b1” does not exist. Creating Web Application pointing to IIS:\Sites\Portal\AppName201925b1 …
11:57:36 Verbose | Acquired mutex Global\Octopus-IIS-Metabase-Mutex
11:57:36 Info | Name Application pool Protocols Physical Path
11:57:36 Info | ---- ---------------- --------- -------------
11:57:36 Info | AppName201925b1 DefaultAppPool http E:\DATA\Octopus\Applications\P
11:57:36 Info | roduction\AppName\201925.2.1-201
11:57:36 Info | 90618-1
11:57:36 Verbose | Acquired mutex Global\Octopus-IIS-Metabase-Mutex
11:57:36 Verbose | Loading Site
11:57:36 Info | Assigning “IIS:\Sites\Portal\AppName201925b1” to application pool “AppName201925b1”…
13:29:20 Verbose | Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in E:\DATA\Octopus\Work\20190619095718-43427-453 exited with code -1

3

Hi @TomNL

Thanks for getting in touch, sorry to see that you had an issue with a deployment.

Firstly, are you able to confirm that on second run the deployment completed successfully?

As for researching the problem the first step would be if you can include the full raw task log from the failed deployment. This will give us some extra information that may help pinpoint if this is a known issue. If not I highly doubt we are going to find anything useful after the fact.

If this issue happens again what would be useful is if you can obtain a process dump of both the Tentacle process and any Powershell processes.

Thanks Tom, I look forward to hearing from you.

Regards,
Alex

4

Hi Alex,

Find the RAW log below.

I can confirm that the second deployment run was succesful. We did not restart anything on the tentacle server. we just canceled the deployment and started the deployment again.

The same task was also running on 3 other simular servers at the same time. these server where succesfull in both runs.

ServerTasks-43427.log.txt (80.2 KB)

5

Hi Tom,

Thanks for that. I’ve reviewed the logs and as I suspected there wasn’t much of value there (apart from confirming you were using a built in step). As the issue cleared itself there isn’t much I can do at this point in time, if it does occur again please grab the process dumps so that we can look into it more.

Thanks Tom,

Regards,
Alex

6

Hi Alex,

I’m Nick, Tom’s colleague. We’re facing the same issue again and I’ve made process dumps of both server and tentacle. Where can I drop them?

The deployment log only gives:

Task Progress

This task was queued 2 hours ago.

Task has not yet started

This task has not started yet, but is next in the queue and will be executed as soon as possible.

Task ID:        ServerTasks-43931
Related IDs:    Deployments-3661, Channels-62, Releases-1652, Projects-62, Spaces-1, Environments-4
Task status:    Queued
Task queued:    Wednesday, 26 June 2019 9:52:14 AM +02:00

                    | Pending: 
                    |

I see no pending tasks for that tentacle.

Regards,
Nick

7

Hi Alex,

In fact I did now find a 2-month old (!) task for the same project and tentacle, waiting at our step for manual intervention (proceed/abort).

[Edit] Cancelling that old task did the job: the deployment of the new release to that tentacle proceeded.

It would be very helpful if Octopus actually mentioned on the project release screen, where it says that the task is next in queue behind another task, exactly which other task is in front and holding up this task.

The odd thing is that the previous release of that project on that tentacle two weeks ago did not have this issue, while the same 2-month old (!) task waiting for manual intervention was also there at that time.

The difference is that in between we upgraded Octopus server from 2019.4.2 to 2019.5.10. It would seem that is the cause. Can you confirm this is by design?

Regards,
Nick

8

Hi Nick,

You are 100% right that it would be useful if we identified what you were blocked behind. I will raise that with the engineering team tomorrow morning and see if I can get some buy in.

As for the behavior change, that is the result of this issue which was included as part of Octopus 2019.5.8 and is definitely by design. Sorry that this caught you out, we are working on making these subtle but important changes more visible.

If there is anything else I can help you with please let me know,

Regards,
Alex

9

Hi Alex,

As my colleague Tom correctly pointed out to me, these are two separate issues that are unrelated: Tom’s RUNNING tentacle deployments which hang at Deploy to IIS step, and my QUEUED and not starting deployments. The latter has been at least explained by the (for us breaking) change on Manual Interventions in Octopus 2019.5.8, while the former still remains an unresolved issue.

So please keep this post open.

Apologies for the confusion in the thread.

Regards,
Nick

10

Hi Nick,

No problems, I did understand that these where two different issues that were unrelated. Please let me know if you encounter another hung deployment.

Regards,
Alex

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.