The problem: we have many web sites to deploy. Each of them is big (think hundreds of megabytes), but most of the sites have a lot in common and substantial number of binaries are repeating from site to site. Some binaries we build, some come from external NuGet packages.
Naive application of Octopus Deploy would put each web site to separate NuGet package. As the result Octopus would have to move gigabytes of data over network to each web server for each deployment. Performance of such deployment is going to be dismal.
One thing we do (before we looked at Octopus) was to use TAR instead of ZIP and make it pack identical files as a links rather than actual bits. This reduces the size of the package 20 times and makes everything manageable again. Unfortunately as I remember from the previous discussion with Octopus support - Octopus uses some open source implementation of TAR which does not understand linking (for unpacking at least) and we cannot use this workaround with Octopus.
Please let me know if you can come out with any creative solution for that problem - it is crucial for our decision to accommodate Octopus or not.
My own best idea so far is make single site deployment package not a single NuGet, but basically NuGet per DLL with explicit inter-package dependencies. This way identical DLLs would not be copied over again and again as Octopus hopefully caches the downloads and take the package it already downloaded to the target. Please let me know how feasible this plan is. Primary question, of course is - would Octopus even understand deployment NuGet package with dependencies?
Also if you can come out with anything better than that - please let me know.
Thank you!
Konstantin
Hi Konstantin,
Octopus Deploy uses delta-compression when transferring packages.
Because each of your sites is a separate package, the entire package would be transferred the first time for each site, but subsequent deployments will only transfer a delta.
I would certainly recommend relying on this initially, as it doesn’t require any fussing with splitting\linking packages etc. And it is very effective in most situations.
Is there any reason this wouldn’t work for you? We can investigate other options if not.
Regards,
Michael
That’s very good news! I was not aware of it.
Let me understand, delta-compression is applied to each individual file within NuGet package or to the zipped package itself?
Delta-compression is applied to the package.
The library we use to perform the compression can be found here, and here is a blog post we published on the feature.
Note that delta-compression in not effective for tar.gz or tar.bz files. Because they are archived then compressed, small changes in contents can result in very different files. It is recommended to use NuGet packages to take the best advantage of delta-compression.
Regards,
Michael
Are you saying that delta compression makes sense for ZIP because it compresses each file individually and then “tar” them together?
But this feature is at the same time great ZIP weakness when we expect a lot of identical files. In this case RAR (or tar.gz with symlinks) may work much better. Ideally best compression method should be autodetected based on content. It would also be good to let specify particular compression method explicitly.
But for now I understand the answer and if my understanding is right please feel free to close this topic.
Konstantin