Within one of our projects, we need to mass deploy a nuget package and expand the same nuget package (using Octopus’ built-in feature) numerous times asynchronously. We have started to notice that this step sometimes takes 10 times longer than expected. Example: on host A, the nuget package will be expanded 5 times at once. The 4th or 5th nuget package, that’s getting expanded alongside of 1,2,3, will take longer to expand.
We are unsure if we have breached a threshold/best practice in Octopus (running 2018.7.12) or on our hosts (IO read slowdown). The issue is not with copying the nuget package to the hosts from the nuget feed, but the actual unpacking of the nuget package. Can you assist? How does this Octopus feature unpackage nuget files? We don’t see any errors in the Octopus logs either. Thank you!
Thanks for reaching out - this is an interesting question. It feels like it might be an issue with hitting IOPS limits, at first impression. I would look into your system metrics to see how your Disk IO performance and utilization is doing during your deployment process.
Obviously, without knowing the exact workflow here, I can only speculate, but you may notice considerable improvements by just extracting the package once, and then using the extracted folder for your subsequent copies?
I hope this helps, and please don’t hesitate to let us know if you have any further questions!
Thank you Justin for your update. Does your team have any best practices for performing the action that we are performing above? Does the NupkgExtractor.cs lock the file while expanding it? We added additional disk IO monitoring, in our environment, and we will review the results of those monitors after a deployment. Thanks again!
I don’t believe it does lock, from inspection, so it does point to perhaps just disk IO limits - how does your monitoring look?
Regarding best practices, we’ll be happy to help! Can you explain a little about what you’re doing in your deployment process? Feel free to post here, or you can contact our Advisory team via advice@octopus.com, if you would prefer to do it privately. From there, we can work with you to try and optimize your process.