Slow response times when deploying website

(rbrink) #1

Hi,

We’re experiencing very slow response times when depoying a website to an internal (DEV) server. Over time the process has become slower and slower. Much like this post we run around a hundred websites on one machine. It’s a physical machine with plenty CPU power and memory. Unfortunately we can’t remove website so we’re hoping for another solution.

Recently we added another DEV-machine as a second deployment target. That’s when it really started to show. The old machine needs about 2 minutes and 11 seconds to deploy a website, the new machine only needs around 40 seconds to deploy the same website!

This is the step from the deployment script that slows the process down: "Making sure a Website “xxx” is configured in IIS… ". In the timed deployment this step took just over 2 minutes!

The second machine is also a physical machine with plenty of CPU and memory available, so no real differences there. The major difference is that the old machine runs around 100 websites and the new machine around 60 at the moment.

Is there anything I can do to speed up the deployment process? Or get some debugging info for you to investigate?

Hope you can help. Thanks in advance!

Best regards,
Raymond

(Michael Richardson) #3

Hey Raymond,

That does indeed seem slow. We’re going to attempt to replicate this.
To help us diagnose, could you possibly supply your raw deployment log? You can email it you’d prefer.

Regards,
Michael

(rbrink) #4

Please find attached a raw log file.

raw_log.txt (57.5 KB)

(Michael Richardson) #5

Hi Raymond,

We’ve spent some time investigating your issue. We set up a test which deploys to a machine with hundreds of websites, but we were unable to replicate your performance.

You can see the powershell script that is being executed to perform the deployment to IIS here: https://github.com/OctopusDeploy/Calamari/blob/6a6927ff73880692f0aecfe50d0e59fce08baaae/source/Calamari/Scripts/Octopus.Features.IISWebSite_BeforePostDeploy.ps1#L408

The highlighted line shows where the Making sure a Website … log message is written.
A little further down the Finding SSL certificate with thumbprint … log message can be seen.

The interesting thing is that not much happens between those two lines. Basically all it does it deserialize the bindings and iterate over them.

One potential culprit that one of my colleagues suggested was Get-Command ConvertFrom-Json. There are scenarios where the Get-Command cmdlet can be slow, for example if some modules are being loaded from a network share. It may be worth executing this on the machine to see how long it takes?

If that fails to reveal a clue, another option is remotely debugging the Powershell that is being executed. We have documentation that describes how to do this. In your case, you would want to use the BreakAtStartOfBootstrapScript value.

I hope that proves useful. Please keep us in the loop? We’re keen to get to the bottom of it.