Slowness issue in many environments when we deploy in Parallel

Good Morning, Thanks a lot for your quick reply, Uploaded the Dumb files in the shared link.
Kindly check and let us know.

Hi @manikantanmca,

Thanks for those files, I will send them to our engineers to check over and will give you a shout once I have something from them.

Reach out in the meantime if you need anything else,

Kind Regards,

Clare Martin

Hi @manikantanmca,

I have an update from our engineers:

We took a look at the dumps. They seem consistent with what was said before - both full PowerShell dumps they provided were waiting on a mutex. That mutex is being held by another PowerShell process that is doing an IIS deployment.

The engineers have said that If you want to know the exact reason then we would need a dump of all of your PowerShell processes since only one would contain the IIS deployment that is being waited on. They said they couldn’t extract anything useful from the 3 other PowerShell dumps.

So the issue here is what was discussed before, IIS doesn’t like doing deployments in parallel, it waits for each deployment before it does the next one, this is why deploying 30 odd applications to one server fails because one of those deployments gets stuck and IIS just waits then as it will not run deployments in parallel it will wait for the ‘stuck’ deployment before it continues, this is why when you restart the server it then works, sometimes a deployment won’t get stuck and you will be able to deploy.

If you do provide us with all of your PowerShell processes we can figure out which one is getting stuck but it might be different for each time you deploy those 30 applications, we can’t really know.

The only thing I can suggest here I am afraid is to do your deployments in batches instead as you alluded to the fact doing 15 at a time does work, so you might have to split your deployments into two projects and do one, then run the other at a different time later on in the day.

I am really sorry we could not be more help here but unfortunately, we are limited to what IIS can handle and it seems like doing 30 deployments at a time is overloading it, resulting in a stuck deployment.

Let me know your thoughts on this, we are happy to go over all of your PowerShell dumps but as suggested, if we find one that gets stuck it might be different for each time you send those 30 deployments out.

The only way to fully ‘fix’ this issue for you is to deploy only 15 at a time.

Please let me know if you are happy to send us all of those PowerShell processes over,

Kind Regards,

Clare Martin

Hi,

The screenshot is for a step that uses a script module. The first line in that script prints “Started” and the last line prints “Completed”. The total script execution completes in a second as per the deployment logs. But the overall time taken for the step to complete is 1 minute and 4 seconds.
Why the deployment step takes 1 min and 4 sec when the script within step finish in less than a second.

Hi @Falcon_Francis,

Thanks for contacting us today.

This is most likely Octopus setting everything up and loading required dependencies for the deployment step, could I ask which template are you using for that step?

To gain a better understanding on why it has a delay/takes a long time would you be able to upload the raw task log this secure link and we can have a look further if you want?

Kind Regards,

Dom

Hi @dom.richardson
Thank you for the answer. Octopus was taking time to load the dependecies.

1 Like

We can see that the deployment is drastically slowing down on this line many times.

Do you see any reason why the script “Octopus.Features.IISWebSite_BeforePostDeploy.ps1” takes too much of a time? Can we use any parameters to skip the validation of IIS version?

What are the actions that happens between
12:49:51 Verbose | Invoking target script D:\inetpub\wwwroot------\1.0\Octopus.Features.IISWebSite_BeforePostDeploy.ps1 with parameters.
13:30:08 Verbose | Detected IIS Version 10.0

@dom.richardson - Can you please help here

Hi @Falcon_Francis,

Sorry about the delay in my response. We have that PowerShell script publicly available located here.

The version check should complete fairly quickly as this is just a registry check when looking through the script I do not see any ability to disable the IIS version check.

Is this an intermittent issue or is this every deployment?

Kind Regards,

Dom.

This is an intermittent issue and not very frequent. (May be once in two days I think). But when happens the deployment will be blocked for 30-40 mins.

Hi @Falcon_Francis,

Thanks for your reply.

It looks like it may be an environmental issue between your Tentacle and Octopus Server.
There is a few troubleshooting stuff we can do to wee where the issue is. I recommend using Tentacle ping tool and run it overnight to see if there are any connection issues.

We have some documentation on how to set this up here.

Hopefully we can get to the bottom of this.

Kind Regards,

Dom.

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.