Is there any way to send a file from the Octopus server to the deployment target?
Let me give you an example of why we need this functionality, maybe it can also help to clarify the question:
In our Octopus package repository we have an application that we need to deploy to our customers
We configure the application using tenant variables (each customer is a tenant)
After configuring the application we sign it and turn it into an MSIX that can be installed on the server of the customer
All of these steps are executed on our own Octopus server as they require a lot of software and access to our signing certificate.
With all of these steps done we now have an MSIX file that can be installed on the deployment target we selected, however the MSIX is stuck on our Octopus server as we created it there.
Is there a task in Octopus that we can use to transfer a file from our Octopus server to our chosen deployment target?
If not do you have any idea how we could do something like this?
Here’s some of the things we’ve experimented with:
Zipping the MSIX file and uploading it back to our Octopus package repository, the issue here is that we need to specify the version when we create the release, so we can’t use the package we create in the deployment process.
As a solution to that we thought about splitting it up into two projects, one to create and upload the MSIX and one to install it, we think this could work but would very much prefer to have a single deployment process instead of two.
Thanks for getting in touch! You can achieve this by using the Transfer a Package step in your deployment.
This step allows you to input a file name and path to be transferred to your destination target. You can add this step to your process and have it transfer the file to as many targets as you need. A script step can be used afterwards to unpack or complete your deployment however you require by using the output variables produced during the Transfer a Package step.
Octopus.Action[StepName].Output.Package.DirectoryPath- The directory the package was transferred to
Octopus.Action[StepName].Output.Package.FileName - The name of the package
Octopus.Action[StepName].Output.Package.FilePath- The full path to the package
Let me know if this helps or if you have any further questions at all.
We still have to push the package to the feed, ideally we’d just transfer the zip file from the hard drive of the Octopus server to our target.
If we use the Transfer a Package step we can’t select the correct version to deploy as it doesn’t exist yet when creating the release.
Here’s an example for the second issue, I push the package to the feed in step 5, you can see the package is version 1.0.0-Lab-Environment-Main. When I now try to pull it to the deployment target in step 6 it pulls the original 1.0.0 package as that is the one I had selected while creating the release.
Like I said we could probably get around this issue by creating a second project that solely handles the creation of the customer specific packages but that adds even more complexity to our deploy process.
What we’re basically looking for is the opposite of Artifacts. Where Artifacts can be used to send something from the target to the server we need a feature that can send something from the server to the target.
Thanks for the additional info. I think the deploy a release step will solve both of your problems.
The feature acts somewhat like a parent-child processes. You can create a new release for the child project during your deployment after the package has been pushed to the feed. This new release will have your updated package version and the parent process will pause until it’s finished. You will have to maintain two deployment processes, but this feature makes deploying them a bit easier.
Let me know if you have any questions or further thoughts here.
I’m not sure that I understand how this helps me here, let me rephrase our question using a different example situation.
Imagine we have a package MyPackage, it contains a single file Tenant.txt which is completely empty.
When we create a release we want two things to happen:
Tenant.txt should be updated so it contains the value of the tenant variable “Name”, for security reasons we decide this has to run on our Octopus server instead of our target. (it’s fiction okay )
This new MyPackage which contains an updated Tenant.txt now needs to be transferred to our target
So that means that the MyPackage that exists on our Octopus feed is not the MyPackage we want to send to our target, instead we want to send the updated one that we’ve just created on the hard drive of our Octopus server. We also absolutely can’t modify Tenant.txt on the target machine, this has to happen on the Octopus server.
In my mind there are two solutions to this issue:
We can transfer it straight from the hard drive of the Octopus server to the hard drive of our target, for this I’d imagine a step “Transfer from hard drive” where we can fill a field for “Path to file” and “Destination”. (this is just an example as far as I know no task like this exists)
We can upload the MyPackage from the Octopus server back to our feed and use it in a Transfer Package task
Like I said as far as I know option 1 doesn’t exist so we’d have to use option 2 (or an alternate option that I don’t know about).
The issue with option 2 is that we need to select the version of the package we want to deploy to our target when creating the release, we can’t do that as that this version gets created in the release.
An extremely oversimplified version of the process could look like this.
When I try to create a release for this process it prompts me for a version of MyPackage to use in step 1 (as it should) but it also prompts me to select a version to use in step 2 (which I can’t supply because the version I want to use gets created in step 1)
I also tried using the Deploy a Release step as you suggested but I don’t see how this fixes the issue.
Do you have any ideas on how to overwrite the version mid deploy or set it to a variable we can modify in the script? Or maybe another option that bypasses the feed entirely?
Thanks for the further clarification. I can see where you’re stuck and have some helpful info, but I should also note that Octopus was not entirely designed to function like a build-server with the handling of new packages in the feed during deployment. Octopus generally assumes the package is ready and complete deploying and configuring changes on the target side of things.
That said, there are usually always multiple ways to do what you need in Octopus. The suggestion I made last message to use the deploy a release step should still accomplish your goals, but isn’t really built to handle new releases being created during its deployment. If you know the release version which will be created for the child project, you can manually enter it as a specific version. You will see an error on the next page warning you that the package with that release version doesn’t exist. However, if you have a script to create the new release you have specified, it will be picked up correctly.
This method makes the process a bit less automated when using the UI, as you will need to know which version of release that will be created during the deployment.
Another method would be to avoid using the deploy a release step and keep the two projects separated entirely. You can create the package(s) and have them modified for each tenant, then you can use Automatic Release Creation alongside an Automatic Deployment Phase in your lifecycle.
This second method is a bit messier and doesn’t give you as much fine control. The Automatic Release Creation feature is somewhat legacy and not out preferred way of automating release creation. (We favour the use of octo.exe from the build server.)
I think I’ll hand this conversation over to our solutions team who might have some further tricks or cleaner suggestions to help you out here. Furthermore, they’re in US/UK time-zone whereas I’m all the way down in Australia. Their hours will better match yours and they will be able to provide faster responses moving forward.
In the meantime, feel free to let me know if the above info helps you at all.
Thanks for the help, we’ve managed to get a working PoC by using the first method you suggested. In our PoC we push the updated packages as dynamic packages so we can reuse the release for each tenant/environment combination.
However I’m still left with two questions:
First off I’m still not a fan of using two projects for this, there’s a general agreement that our PoC is hard to understand among the people who I’ve shown it to. We could probably cut the entire need for a second project if we had an option to only download the packages when they’re effectively needed instead of downloading them right at the start of the release. I doubt someting like that is built into Octopus though as our very specific scenario is probably the only time when this feature would be helpful.
I’m probably overlooking this option somewhere but I can’t seem to find a way to choose the version of the child project we want to deploy using octo.exe. This is extremely important for us as we usually create releases from within an Azure DevOps release pipeline. I’m basically looking for the equivalent of --package but for specifying the version of the child project to deploy when creating the release of the parent.
If there really is no way to merge these 2 projects into 1 project then so be it, we’ll just have to learn to live with the complexity but we absolutely need a way to choose the version of the child project using octo.exe otherwise our implementation is blocked again.
Thanks for getting back in touch with us, I understand the concerns and the questions that you have.
I’ve reached out to our solutions team who should be able to provide either a pathway for you to better perform the actions you’re looking to perform or confirm whether the suggest “Deploy a Release” step is the best option for your use-case.
We’ll be in touch as soon as we can with any additional information, don’t hesitate to reach out if you have any further information or any questions.
Hey @yenti.verle , my name is Cory and I work with the customer solutions team here at Octopus. I wanted to reach out and let you know that I’m taking a look at your use case, and I’m hoping to have some more information and resources for you a little later today.
In the meantime though, I want to confirm a couple of details about your end goals to make sure I’m on the right track -
At the end of the day, your ultimate goal is to create a release (we’ll call it 1.0.0), select a tenant (CompanyA) and environment to deploy to, click deploy, and the following happens:
MyPackage v1.0.0 is pulled from the Octopus built-in repository and deployed to a folder on the Octopus Server.
During deployment, your package takes advantage of variable substitution to add in your specified variable values, including tenant specific variables
Once variable replacement is done, the package is signed and repackaged as something like MyPackage-CompanyA-1.0.0.msix.
This new tenanted package is then uploaded back to your Octopus Server’s built in feed
The tenanted deployment then continues, using the newly created package to deploy to the customer’s infrastructure
Are the steps above correct for your intended goal/use case? Please let me know if there’s any specific content missing, and I’ll try to find some recommendations to help you model your goal state within Octopus.
Some other general notes and questions -
Like Daniel mentioned previously, “Octopus as a build server” tends to be an anti pattern. One of the main advantages of Octopus is that a deployment process represents a consistent way that packages move through your system - when your process pulls a package, modifies it, then repackages and redeploys it, it can be difficult to reason about what happened to your project and where in the pipeline.
To the above - can you explain a bit more about the process and tooling required for signing your packages? I know you mentioned that it requires access to a certificate as well as specific software - there may be a possibility of utilizing something like Execution Containers to manage some of that heavy lifting, which could allow you to manage the .msix creation process on the customer hardware (If this worked, it would only require them to have Docker instead of ALL of the singing tooling)
Beyond the two prior points, I wanted to put another +1 for treating this as two separate projects. In an ideal world, your deployment process is just the deployment itself - managing too many pieces can make it difficult to keep track of what’s happening and where. Have you considered using a runbook to create and push your msix package to Octopus? That would still be connected to your project specifically, and your deployment process could be focused specifically on deploying the customized package, and would allow you to better set your release versioning.
The ultimate goal for us is to be able to create a MSIX on our Octopus Server and then download that MSIX to the target of our current deployment. The MSIX creation and deployment process doesn’t matter much here which is why I later simplified it to the MyPackage example.
The process you’ve linked is exactly what we want our end result to be, our PoC currently does the following (using two projects):
MyPackage v1.0.0 is pulled and stored to a folder on the Octopus Server
We substitute some variables in a configuration file and zip the modified MyPackage as MyPackage.CustomerA.EnvironmentA.1.0.0.zip (in the real version we would first turn it into an MSIX and sign it before zipping it)
This zip file is pushed back to our Octopus feed using octo.exe
We use octo.exe to create a new release of the child project which is configured to use version 1.0.0 of MyPackage.#{Tenant}.#{Environment}, the version of the child project is the same as the parent project
Finally we deploy the release of the child project we created earlier which does the actual deployment of the MSIX file
To answer some of your questions:
I absolutely understand we’re using Octopus for something it isn’t supposed to be doing; the issue is that our application requires some customer specific information which we have stored as tenant specific variables. If we didn’t need those tenant specific variables we would absolutely offload the MSIX creation to an Azure DevOps pipeline but it’s impossible for us to do so right now.
Signing the MSIX requires quite a lot of tools which our customers would prefer not to install on production servers, for example we’d need to install some of the Windows 10 SDK just to get it signed. We can’t rely on Execution Containers as most of our clients don’t use Docker. Another one of the reasons we need it to run on our servers (which I didn’t specify originally) is that our certificate is stored in an Azure KeyVault (it has to be stored in a HSM), for security reasons we only allow our own servers to access this KeyVault. Finally, some of our customers don’t have internet on their production machines so even if all of the previous issues weren’t existent we wouldn’t even be able to connect to our KeyVault.
I see what you’re saying, the complexity may just be because we’ve never used the Deploy a Release task before, instead we have always relied on long deploy processes which we can only maintain because a specific team knows how they work. It’s definitely an interesting task and as we use it we’ll probably feel a lot more comfortable with the multi-project approach we have to use here. I haven’t considered a runbook yet, we currently don’t use that feature at all and instead create a project for everything (which again, is probably a bit of misuse of Octopus).
I hope that information can already help you understand our situation a bit better.
Finally I’d like to poll for the second bit of my question from today again . In the real world we’ll be creating the release of the “parent” from an Azure DevOps pipeline using octo.exe, however we run into an issue here because the version of our “child” to deploy has to be set manually (as it gets created by the parent). Using the GUI we can simply use this field:
But as far as I can tell we can’t set this using octo.exe. Again this is probably something I’ve overlooked in the documentation but it is an absolute necessity for us to be able to set it this way.
Thanks for sharing that additional context, that’s very helpful! We’re definitely biased, but ultimately we just want to make sure you’re getting the best possible experience within the product for your deployment and automation projects!
To answer your specific request - check out these docs on the create-release command for some information on the various ways to specify package versions for your projects using the CLI. If you have any questions or run into any errors, let me know and we can work through debugging the output.
Once you get your process stable and running, please feel free to reach back out either here or through the advice@octopus.com mailbox - we’d love to help as you continue iterating on and modernizing your pipeline!
)
. In the real world we’ll be creating the release of the “parent” from an Azure DevOps pipeline using octo.exe, however we run into an issue here because the version of our “child” to deploy has to be set manually (as it gets created by the parent). Using the GUI we can simply use this field: