Our multi-tenanted deployments are sporadically taking a long time to complete, and upon investigating the logs i’ve found periods of several minutes where tasks are waiting even though the locking task has already moved on to another step. I’m unable to provide full logs due to sensitive information but I have attached some excerpts with proprietary info removed. I can provide further context for logs if needed.
To summarise the log examples - one deployment comprised of three tenants, with two steps referencing packages. The tasks are 126056, 126057, and 126058. Task 126058 began ‘Acquiring Packages’ at 20:51:18 and reported it was waiting on 126057 until 20:58:38, yet the logs for 126057 reported that it had completed acquiring packages and moved on to the next step at 20:51:56.
Please can you help me understand what accounts for the seemingly unexplained wait period of several minutes in the logs for task 126058?
We’re running Server v2018.12.1 and Tentacle version at the time of the attached logs was 3.19.0 but yesterday I updated to 3.25.0 (no conclusions as to whether this has helped yet).
Thanks for the information. Is this an intermittent issue? Does it only occur for one tentacle/worker, or many? Please let me know if it continues after the upgrade of tentacle version.
It does seem intermittent, and not related to particular targets, projects, tenants, environments etc - still looking for some kind of pattern but it seems completely random at the moment. Deployments even of large projects are generally taking 2 or 3 minutes, but will occasionally get stuck as per my original post.
We do make quite a lot of improvements regarding stuff like this and you are a fair bit behind in versioning. Is there any possibility of upgrading your server and tentacle versions?
Unfortunately, without there being a pattern and it being intermittent there’s not much I can do to help. It does appear to point to either versioning or environmental, though.
We are starting to plan an upgrade but have no schedule for it yet. We’re also going to expand the server to spread the load of deployments a bit more. Once the upgrade is done I’ll post again if the issue is still ongoing.