Octopus waiting for already completed scripts


#1

Hey everybody

We’re having some issues with a deployment that is running way to slow and we feel like the behavior is really weird.

For the Setup. We first run some preparation steps before we deploy multiple web applications to the same server/tentacle in parallel. This is our DEV stage, later environments deploy the applications to different servers.

We run PowerShell-scripts here for many use cases, but every time we deploy we see multiple steps that are just hanging with the comment “Waiting for the script in task ServerTask-99999 to finish as this script requires that no other Octopus scripts are executing on this target at the same time”.

When click through the ServerTasks at some point we reach a Task that has been completed, but there’s still another ServerTask waiting for completion of this script.

This waiting can take ages. Sometimes it’s around 20 minutes until the waiting script finally realizes it can continue… Why do our scripts wait so long? Do we need to configure a polling interval somewhere? What could cause this behavior?

Thanks in advance


(Michael Noonan) #3

Hi!

Thanks for getting in touch! Which version of Octopus are you running? We have seen some cases in the past where blocking could be too aggressive. Upgrading your Octopus installation would be a good place to start. That way if we discover there is a problem we can patch it quickly.

That particular message you see is because by default each Tentacle will only run a single script at a time. This is a safe-by-default approach. This page usually proves helpful when people are looking to make their deployments run faster or with higher concurrency: https://octopus.com/docs/deployment-process/performance

If you are still seeing this behaviour after upgrading and you think something is wrong, please grab the relevant task logs and the Octopus Server log and send them to support@octopus.com. Having all this information should help us correlate what is going wrong.

Hope that helps!
Mike


#4

Hi Mike

Thanks for the timely reply.

We are actually using v2018.9.16 which is not the newest but fairly new. The versions affected after my googling yesterday were arround v3 or v4?.

I took a look at the performance page, but I can’t find much about running scripts in parallel, which would probably be the way to go for us. But I want to stress again, I feel like steps are waiting way longer for a script to finish, than it actually took for the script to run, so we are still waiting even if everything is complete…

Regards,
Max


(Michael Noonan) #5

Hi Max,

Thanks for keeping in touch. Unfortunately this gets quite tricky to analyse in detail, and I’ll need your help to gather all the logs we need.

  1. Switch the log level of the Tentacles involved in this reproduction to Trace which will record when all the locks are acquired/released. I’m not sure how many Tentacles are involved in your deployment. This could be easy or heartbreaking!
  2. Run the deployment to reproduce the issue.
  3. Go to Configuration -> Diagnostics -> Diagnostics Report and grab the zip file.
  4. Grab all the Tentacle logs involved in the deployment.

Send all of that to support@octopus.com with a link to this topic.

Hope that helps!
Mike