Apply retention policies task causing slowness in the UI

Hi,
We are currently running version 2022.2.6971, are we are seeing an issue when the Apply retention policy tasks are running.
While the tasks are running the UI is very slow and I have noticed that a high number of tasks will appear that are waiting in the queue. Then when I select the waiting in queue the UI does not respond. Then the high number of tasks in the queue will suddenly drop to a single figure.
The Apply retention policy tasks run every 4 hours and is impacting the UI response.

Also the DTUs are spiking to 100% while this is happening.

Kind Regards,
Micheál Power

Hi @mikepower79,

Thank you for contacting Octopus Support. I’m sorry you are running into this issue.

I hope you don’t mind, I have a few clarifying questions:

  • Approximately how many “Apply retention policies” tasks are being created at once?
  • Is this only occurring once every 4 hours?
  • Do you have any automation in place that re-runs tasks?
  • Are any other tasks exhibiting similar behavior?

You may get additional information by clicking on one of the “Apply retention policies” tasks and checking the pane on the right side. Here is an example that shows a user re-ran the task:

I look forward to hearing back from you.

Best Regards,
Donny

Hi @donny.bell,
There are 24 Apply Retention Policies task that ran at 2:15PM today.
Yes every 4 hours
No automation in place that re-runs tasks

As you can see the Apply Retention Policies tasks take about 20 minutes to complete

Also the Task history is empty on each task , so the task was no re-run

Kind Regards,
Micheál Power

Hi @mikepower79,

Thank you for the quick response.

Could you upload a copy of your Server Logs via this secure upload link?

If we don’t find any hints in the Server Logs, I may need to lean on our Development Team to get their eyes on this issue.

Let me know once you are able to upload the logs and I’ll have a look.

Best Regards,
Donny

Hi @donny.bell,

I will send on logs to you, I have to get logs from each node as Octopus is running on AKS.
I have seen in the Diagnostics that spaces = 1185, and we only have about 30 spaces.
Why is it showing such a high number of spaces?

Would this have anything to do with the slowness when Apply Retention Policies is running?

Kind Regards,
Micheál Power

Hey @mikepower79,

Donny is away today so we will await the logs you send over. I just wanted to alleviate your concerns on the amount of spaces your Octopus instance is showing.

I imagine you have quite a few users on your instance? We introduced a feature called ‘Private Spaces’ which was rolled out to our cloud customers. The idea was to give each user their own private space so they could test out their own projects away from production spaces, each users private space was automatically added when an account was created.

Our engineers passed this feature onto our On-Prem customers a little too early and there were a few issues surrounding their introduction, they disabled private spaces for on-prem customers in the DB and in some bits of Octopus (each user still got a space in the DB but they couldnt access it in the Octopus UI). Unfortuantely this still created issues so the engineers have completely disabled this feature so it doesnt get created in the DB for each user.

This disabling has yet to reach the current On-Prem installs though, so, long story short, those spaces I imagine are just your users private spaces, hence why the number is so high. I hope that helps alleviate any concerns you have here.

We have not seen private spaces influence retention policies but will keep that in mind when looking at your logs. Since they are private spaces, retention policies wont apply to them anyway so that should not be the issue here.

Let us know when you are able to upload the logs and we can take a look at them.

Kind Regards,

Clare

Hi @clare.martin,
Thanks alot for the feedback. That’s good to know.

Kind Regards,
Micheál Power

1 Like

Hi @donny.bell,
Can you share a link to upload the server logs please.

Kind Regards,
Micheál Power

Hey @mikepower79,

Here is your secure link to send us the logs.

Can you let us know once you have uploaded them and I will take a look.

Kind Regards,

Clare

Hi @clare.martin,
Octopus container logs uploaded.

Kind Regards,
Micheál Power

1 Like

Hey @mikepower79,

Thanks for sending those over it seems like sometimes the retention policy task can take over 5 seconds to load which is quite a long time. I did find one for 15 and 19 seconds:

I am going to get this in front of our engineers as I don’t think that query should be taking that long to execute. This is what is causing the slowness for the page loading / crashing out I imagine.

I will let you know any outcomes or suggestions our engineers have, please reach out in the meantime if you have any other questions.

Kind Regards,

Clare

1 Like