We’re having some issues with a deployment that is running way to slow and we feel like the behavior is really weird.
For the Setup. We first run some preparation steps before we deploy multiple web applications to the same server/tentacle in parallel. This is our DEV stage, later environments deploy the applications to different servers.
We run PowerShell-scripts here for many use cases, but every time we deploy we see multiple steps that are just hanging with the comment “Waiting for the script in task ServerTask-99999 to finish as this script requires that no other Octopus scripts are executing on this target at the same time”.
When click through the ServerTasks at some point we reach a Task that has been completed, but there’s still another ServerTask waiting for completion of this script.
This waiting can take ages. Sometimes it’s around 20 minutes until the waiting script finally realizes it can continue… Why do our scripts wait so long? Do we need to configure a polling interval somewhere? What could cause this behavior?
Thanks for getting in touch! Which version of Octopus are you running? We have seen some cases in the past where blocking could be too aggressive. Upgrading your Octopus installation would be a good place to start. That way if we discover there is a problem we can patch it quickly.
That particular message you see is because by default each Tentacle will only run a single script at a time. This is a safe-by-default approach. This page usually proves helpful when people are looking to make their deployments run faster or with higher concurrency: https://octopus.com/docs/deployment-process/performance
If you are still seeing this behaviour after upgrading and you think something is wrong, please grab the relevant task logs and the Octopus Server log and send them to support@octopus.com. Having all this information should help us correlate what is going wrong.
We are actually using v2018.9.16 which is not the newest but fairly new. The versions affected after my googling yesterday were arround v3 or v4?.
I took a look at the performance page, but I can’t find much about running scripts in parallel, which would probably be the way to go for us. But I want to stress again, I feel like steps are waiting way longer for a script to finish, than it actually took for the script to run, so we are still waiting even if everything is complete…
Thanks for keeping in touch. Unfortunately this gets quite tricky to analyse in detail, and I’ll need your help to gather all the logs we need.
Switch the log level of the Tentacles involved in this reproduction to Trace which will record when all the locks are acquired/released. I’m not sure how many Tentacles are involved in your deployment. This could be easy or heartbreaking!
Run the deployment to reproduce the issue.
Go to Configuration -> Diagnostics -> Diagnostics Report and grab the zip file.
Grab all the Tentacle logs involved in the deployment.
Still have very similar issues in 2020 with Octopus version 2019.13.7. Not the latest version, but as I can see on different forums, some previous versions already have some fixes in same area.
When I’m deploying on 20-25 instances simultaneously in parallel (20-25 different targets) and deployment process has step with same target (NLB instance in my case), all deployments stuck on this step with same target and waiting for something…
I see the same “Waiting for the script in task ServerTask-99999 to finish as this script requires that no other Octopus scripts are executing on this target at the same time” message and most interesting that this ServerTask-99999 is already finished 10-15 minutes ago.
So, each deploy has delay with step execution for 20-25 minutes and total deployment time increasing dramatically. Internal investigations gives nothing. Seems that something with Octopus queue management routines.
Maybe I have missed solution for such cases already?
Welcome, and thanks for getting in touch! I’m very sorry to hear you’re also hitting this issue. Like mentioned in the latest reply by Mike, this does get tricky to analyse in detail, and the information requested in it will help us to understand the issue you’re facing in your specific situation, and how to proceed in resolving it. Would you be willing to send us an email to support@octopus.com with the following details attached in it?
Switch the log level of the Tentacles involved in this reproduction to Trace which will record when all the locks are acquired/released. I’m not sure how many Tentacles are involved in your deployment. This could be easy or heartbreaking!
Run the deployment to reproduce the issue.
Go to Configuration -> Diagnostics -> Diagnostics Report and grab the zip file.
Grab all the Tentacle logs involved in the deployment.
Additionally a link in the email pointing back to your post here would be helpful for reference.
