Dealing with large files

We are currently working on a software package which we will need to deploy to some remote servers. These packages have very large data files (around 5GB) which will be updated and deployed from time to time. The connection to the production servers is not terribly slow, but not extremely speedy either. Has anyone used Octopus to deploy/test in such a scenario? Is there anything we should be aware of, or any ideas how to optimize the rollout process?

Hi Ryan,

Thanks for getting in touch! I’ve seen deployments with large files from a few different customers in different scenarios.
A couple of things that help speed these processes up that might work for you are:

  1. Have the files directly downloaded by the Tentacles - this helps with bandwidth
  2. Setup a pre-task in the project (or another project) to send the files before a major deployment to production. For example, have a package step that doesn’t really do anything to the files, but uses them, so you can run those steps and skip all others, to upload the files before a deployment is scheduled. Then when the deployment runs it will see the files are already there and not re-upload them taking that out of the deployment itself. Takes coordination and pre-planning, but if those files are only changed so often, I think will help in the long run.
  3. You will want to experiment with delta compression being on or off. I am reading this as a single very large file, not a NuGet package that is large containing lots of files.

This might come down to experimentation. What you find works best on the network and with the files. Combinations or parts of above will help.

Let me know what you think :slight_smile:
Vanessa

Just to update on this, we have implemented a few custom Powershell scripts which runs on the Tentacle installed on the Octopus server, and pulls the files (over SSH), then pushes them (again over SSH) to the target server.

Meanwhile, I have seen that the upcoming features in 3.3 will fit perfectly with this. We will be able to remove the Tentacle from the server, as well as place the tar.gz files directly into the Octopus package server via the API. Can’t wait!