Octopus Tentacle Calamari execution on 1 of 9 servers has become exponentially slow

I have uploaded logs from 2 different processes that both deploy to a specific Dev server. Every deployment to this server has become exponentially slower. Where a deploy to any other environment takes maybe 2-3 minutes to this server it may take over an hour. This was our first server used with Octopus, but we clean up old files in 2 days if not deployed and have optimized our database, and tentacle software versions & PowerShell are the same version. The greatest delay occur on each execution of Calamari where it appears to hang for several minutes before continuing.

I have done PowerShell Measure-Commands against both Tentacle.exe & Calamari.exe, both are in the milliseconds. I have also check for thread contention, but found normal behavior each time. Please help me provide any information you may need to research this issue. I have scoured the blog here and the internet.

Thank you,
Doug

Hi Doug,

Thanks for getting in touch and providing those task logs! I’m sorry to hear you’re hitting these massive delays while deploying to this Tentacle. I’m wondering if there’s any significant difference in the server, like OS, this Tentacle is installed on as compared with others that are performing properly? Would you be willing to also send us a Tentacle log that covers the time period you’re deploying to it?

I know we’ve also shipped some patches to Tentacle, some performance related, over the last year so it might also be worth upgrading Tentacle to latest (we work to ensure backwards compatibility so it should work with your server version) and see if that improves this in any way.

We might also want to have a look at a performance trace at some point in the future if those details don’t help narrow this down.

I look forward to hearing back and getting to the bottom of this one.

Best regards,

Kenny

Hi Kenny, This server actually has double the resources of our other servers and the same OS and Calamari version.
Win Server 2012 R2
4 core 2Ghz intel Xeon CPU
8 GB RAM
Calamari ver. 12.1.1
The 2 logs I attached to my profile were from 2 different applications being deployed to this Web Dev server. So you want more of these logs? This provided to most verbosity of the processing. What can I provide you to enable you to dig deeper into this? Thank you, Doug

Kenny,
I also did a PowerShell “Measure-Command” on both Tentacle.exe and Calamari.exe and they ran 48 and 946 milliseconds respectively.

Hi Doug,

Thanks for keeping in touch and for that additional information. This is certainly a bit of a mystery. I think a worthwhile simple first step would be to upgrade Tentacle (latest is 6.0.576) and Calamari (latest being 18.1.1) at least on this one affected machine to see if the improvements and bugfixes implemented within that significant gap help, and by how much.

From there I think we’d need a performance trace to have a close enough look into this one to thoroughly troubleshoot. The following doc page should help get started recording the trace.

I look forward to hearing back!

Best regards,

Kenny

Hi Kenny,

I installed Tentacle 6.0.576. However, Calamari 12.1.1 is the latest version I could find, which is already installed. I even removed 12.1.1 and let the tentacle reinstall Calamari when the deployment began, and it installed 12.1.1. This Tentacle upgrade did not improve performance. I installed dotTrace and traced the execution of the Tentacle from start thru 20 minutes of processing. This captured 2 instances of the performance issue, which occurs when the Tentacle executes PowerShell to call Calamari. The version of PowerShell on this and all the servers shown below.

I uploaded a zip file of the dotTrace recording in addition to the Tentacle log file for the application being deployed, and 3 screen shots of Tentacle CPU, Memory, and Network during the execution of this deployment. The resource utilization is tiny for the Tentacle, so I just took screen shots. I saw no spikes on these throughout the deployment. You will notice in the log that the PowerShell execution of Calamari takes several minutes each time it executes. From monitoring it, it appears to be dormant like it is not even doing anything but waiting like on an interrupt.

Anyway, thank you for the help. You will find the upload on my profile.

Cheers,

Doug

image001.png

Hi Doug,

I appreciate the follow up and the additional information you uploaded for us to look into. Just confirming I’ve received it and starting the investigation. Please let me know if you have any questions or concerns in the meantime. :slight_smile:

Best regards,

Kenny

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.