Deploy gets stuck when updating IIS Website

Legacy Uncategorised
1

When deploying a release to multiple environments at the same time, some times the deploy is stuck when updating a IIS Website.

When it fails, it occurs to at least two environments, which may suggest it is a race condition. I cannot reproduce the error reliably, but if I deploy to 200+ environments it will happen at least a couple of times.

Attached screenshot of a stuck deploy.

Screen_Shot_2016-03-07_at_10.35.30.png
Screen_Shot_2016-03-07_at_10.35.30.png1514×622 81.1 KB

2

Hi,

Thanks for reaching out. A couple of questions about your scenario:

  • Are each of these environments a different set of customers to which you have to push the exact same app all at once?

  • Are you deploying the same app to all these environments?

  • If you deploy them in smaller batches (50x4), do you get this same error?

  • Which exact version of Octopus are you running?

Best regards,
Dalmiro

3

Hi,

  • Yes, each environment is a different customer. I use this setup to emulate multi tenancy support

  • Yes, same app, same release

  • Yes, it also happens if I deploy to 50 environments at the same time. It can also happen if I deploy to as few as 5 but the chance is a lot lower.

  • I am running 3.3.1, but it has happened since at least 3.0.15

Thanks for looking into this

4

Hi,

Thanks for all that info. As you might know already, we will be adding multi-tenancy support in 3.4, which will make this scenario a lot easier. At the time being (3.3 and below) the best approach is the one you are taking already (Env per customer).

  • Are the Targets (tentacles) on each Environment unique, or are some of them machines that are running multiple Tentacle instances? I’m interested in knowing if two tasks(deployment steps) are being executed against IIS on the same machine at the same time

  • Were you able to find a pattern on which set of Environments/Targes get “stuck”?

  • Could you share us a 2 deployment logs please: One of a successful deployment and another of one that got stuck. If possible try to send logs of 2 environments that were triggered at the same time.

Regards,
Dalmiro

5

Hi,

Thank you for your patience. Sorry I’m late on getting back to you.

  • The tentacles are not unique on each Environment. As we have a lot of environments of each machine they also share the tentacle. However, this also means, as your asking that two deployments step in fact are executed against the same IIS at the same time.

  • No

  • Yes. I have attached two logs.

Success.txt (14 KB)

Stuck.txt (12 KB)

6

Hi,

Dalmiro is on leave for a few more days so I am assisting with this ticket.
Would you mind also providing the Tentacle logs from around the time of the stuck deployment.
They can be found in C:\Octopus\Logs on standard installs.

Vanessa

7

Hi, thanks for helping.

I have attached the log from the tentacle running the two previously mentioned deploys.

OctopusTentacle.2.txt (74 KB)

8

Thanks for the logs and for waiting for my return!

Is this “kaai” site being deployed at the same time on the same machine with the same name on IIS? Because if that is the case, I don’t think IIS will be able to handle it.