In the middle of our deployment window we suddenly had Octopus spawning what I presume are internal cleanup tasks – i.e. 12k tasks named "Delete Release "
We were forced to cancel the tasks via the API to reduce the incredible SQL pressure it was causing and backlog of "real deployments’. It also seemed like it was popping the tasks off at a much lower rate than the 50/node (4x nodes) we have set. Not sure if this is an internal throttling for consistency. If it was, it was probably the right choice b/c the backend couldn’t take much more than what it was doing.
Q: What caused this?
Q How can we control WHEN this happens? Is there a way to force this in the API during an acceptable time period? I understand this kind of cleanup is necessary.
Sorry for the delay in getting back you on this one. What we believe happened in your case is that someone enabled release retention policies which is what caused the large spike in SQL usage alongside 12k + running tasks.
We don’t have a way of controlling when this setting is enabled, as the retention tasks run on a schedule (every 4hrs IIRC). What we have done in newer versions of Octopus (2018.5.1 and newer) is bundle all of the delete release tasks into one task, leaving slots free for other tasks/deployments to continue. Since its only one task it is possible to turn off retention and cancel the task during peak periods, then re-enabling retention at a more suitable time (e.g. weekends). S
We also made the task(s) much faster in 2018.6.7, so if at all possible I would recommend upgrading to latest or at least 2018.6.7.
Hopefully that answers your question, please let me know if there is anything else that I can assist with.