Hi guys,
Since we installed Octopus 3.4 we have noticed that it reads all Nuget packages on start. Now we run 3.7.4 and Octopus still does so.
It wouldn’t be a great issue if we had a couple of thousand packages in our library. But we have 52000 packages, so this read process lasts for 8 hours and utilize disk for 100%.
I have attached screenshots, so you could verify the veracity of my words.
I would be very appreciated if you help us solve this issue.
Thank you in advance!
Kind regards,
Denis Titusov
Hi Denis,
Thanks for reaching out!
We have some customers that (for various reasons) write packages directly to disk and expect Octopus to find them there. So, on start up, we need to fully read the packages and make sure the index is up to date. It runs this as a background task, so it shouldn’t be having too much of an impact on your Octopus server.
That said, 8 hours of 100% utilisation doesn’t sound great. It appears that our background process is not as background as it should be.
Is this causing noticable performance issues, or is it more of a “this is making our monitoring go a bit crazy” issue?
Do you have a need to keep all 52000 packages? If that is not the case, you should probably checkout retention policies to cut down the packages stored.
Also, do you write packages direct to the disk? Or do you push direct to the inbuilt package store/use an external package store?
We’ve got a few ideas how we can tackle this, but once we’ve got your feedback on the above, we can figure out the best way forward.
Hope this helps!
Regards,
Matt
Hi Matt,
Thank you for your response!
Let me answer your questions.
Is this causing noticable performance issues, or is it more of a “this is making our monitoring go a bit crazy” issue?
This background process utillize disk up to 100%, and it causes performance issue with any operation, which reads/writes packages from/to disk. For example, while Octopus indexes packages on its start we face performance degradation with creating and deploying releases.
Do you have a need to keep all 52000 packages?
I’m afraid, yes. We have about 250 projects in Octopus and each project have about 40 steps in average. If we store only last 5 versions of every package it is 50000 in total.
Also, do you write packages direct to the disk? Or do you push direct to the inbuilt package store/use an external package store?
We push all packages via builtin NuGet feed. I should also point out, that we store all packages on the external network storage, so it’s utilization doesn’t affect OS performance and vise versa.
I hope, this information helps you. If you need an extra detail, please feel free to request it.
Kind regards,
Denis Titusov
Hi Denis,
Thanks for sharing those details. I’ve logged an issue with some thoughts about how we can tackle this and improve the experience in this scenario. I’m hopeful we’ll be able to tackle it soon.
Regards,
Matt
Hi Matt,
Thank you for information! I’ll stay tuned.
Kind regards,
Denis Titusov
Hi Denis
Happy to share that we’ve just released 3.7.15, which adds a toggle to disable the built in package store re-indexing at startup.
Hope that helps!
Regards,
Matt
