OctoPack/NuGet performance

I have a web project in which i need to copy the files of a large proprietary cms along everytime i deploy.

28000 files and around 600MB

I have TeamCity set up to build the solution and zip it, and then i do a manual deploy.
I am however looking into using OctopusDeploy for deployments.

When i run my current build (builds the solution copies all the files to a folder and zips it and copies it to a network drive)

It all takes around 2-3 minutes.

Now i have tried doing the same with OctoPack and the whole process takes around 20-30 minutes. (The build is super fast, but creating the package takes 99% of the time)

So my question is: Is this the expected performance?
Or am i doing something wrong?

Below i include the command that is run when creating the package.

[12:00:56][Exec] "D:\TeamCity\BuildAgent\work\95a042d0632396bd\Website\..\packages\OctoPack.1.0.109\tools\NuGet.exe" pack "publish\Website-LRI.nuspec" -OutputDirectory "bin" -basePath "publish" -Version "1.0.0.44" -NoPackageAnalysis [12:00:57][Exec] Attempting to build package from 'Website-LRI.nuspec'. [12:22:48][Exec] Successfully created package 'bin\Website-LRI.1.0.0.44.nupkg'.

Hi,

Yes, I think you have discovered a problem with NuGet, which is very inefficient when it comes to large files (the whole package is loaded in memory).

I plan to contribute some changes to NuGet to fix this, but in the mean time, it might be best to try and find ways to reduce the number of files in the archive. One option might be to only package up files you’ve changed, since I’m guessing most of the CMS doesn’t change much. Your Deploy.ps1 script could then copy the base CMS files from a network share or existing archive, and then deploy your custom DLL’s/controls over the top. This would probably also speed up the deployment since you could use a tool like Robocopy to perform the copy, so that only modified files are copied.

I hope this helps,

Paul

Okay that seems like a reasonable explanation.

That is definately a way to implement it, the problem being though, that we have many projects with different versions of the cms so basically gonna be alot of work.

Thank you for your quick reply.

Is there any work around for the issue. Adding 20 minutes to our build just for the octopack step is killing us.

Well, i ended up zipping some of the folders that had a large number files in a separate teamcity buildstep and then including that artifact as a dependency to the root directory of the build that had the octopack step. That took the build time down by 15 minutes… then i updated the deploy.ps1 to extract the zipped files.

Any update on this? Still seems to take a long time to copy projects with lots of resources.

Hi Tom,

Thanks for getting in touch! This thread is quite old now, and was for pretty old versions. Are you able to provide a build log or let me know what problems you are experiencing?
We haven’t been notified of anything similar to this recently.

Vanessa

I too have this problem on a large web application with about 19k files. build is 1 minute, msdeploy is about 2 minutes, octopack is about 10 minutes!

[09:33:24] : [OctoPack] Using package version: 2016.413.22
[09:33:24] : [OctoPack] CreateOctoPackPackage (12m)
[09:33:24] : [CreateOctoPackPackage] OctoPack: Written files: 366
[09:33:24] : [CreateOctoPackPackage] OctoPack: A NuSpec file named ‘m2.nuspec’ was not found in the project root, so the file will be generated automatically. However, you should consider creating your own NuSpec file so that you can customize the description properly.
[09:33:24] : [CreateOctoPackPackage] OctoPack: Packaging an ASP.NET web application (Web.config detected)
[09:33:24] : [CreateOctoPackPackage] OctoPack: Add content files
[09:33:24] : [CreateOctoPackPackage] OctoPack: Added file: apple-touch-icon.png
…snip… 18k more later (adding the files to nuspec only takes 2 seconds)
[09:33:26] : [CreateOctoPackPackage] OctoPack: Attempting to build package from ‘m2.nuspec’.
…as you can see this line took 12 minutes! Not sure if octopack supports more verbose logs, this was teamcity verbose output.
[09:45:25] : [Step 5/5] Publishing artifacts (5s)
[0

Hi Gabe,

Octopack adding files to NuSpec is very fast because its just updating a text file. NuGet then has to read the NuSpec and add the files to the package which is the slow part.
You can simulate this behaviour outside of Octopack by calling NuGet with that NuSpec and comparing times. NuGet is slow for larger amounts of files.

Octopack just calls NuGet pack so you shouldn’t find any difference.

Thanks,
Daniel

Okay, that makes sense that it’s a nuget implementation thing. Thank you for the response!

Hi Gabe,

I’m Glad I could help! Please feel free to contact us if you have any more questions/issues in the future.

Regards,
Daniel