Hi, We are in the process of moving to worker pools, and to my surprise I notice that it’s not possible to use a simple “Deploy Package” step targeting a worker pool.
We use a process where we “deploy” a nuget containing various built artifacts to a temp folder, and then have custom step templates do various deployment types using the unzipped (“deployed”) artifact in that folder.
For some reason it seems that this is not possible using workers.
What would be the best way to achieve similar functionality using Workers?
I see other questions about the same, and referencing packages in step templates wont work for us - that would create a link that counteract how we use step templates in the first place. In our current setup (which is just using a regular tentacle) we’ll have 50-100 different projects producing (for example) an artifact containing serverless framework deployment, and then the one step template that knows all about how those should be deployed.
So I’m a bit stumped, I really hoped that workers were our ticket forward to get better scalability on our deployment processes, but it seems that they only have a very small subset of the features available on a Tentacle.
Thank you for reaching out. It is possible to do that. I did something very similar to get an older step template to run on workers. We will be using the Reference Package feature added in 2018.8.x.
To begin with, when using workers, I work under the assumption each step is going to be done on different workers (unless you have one worker in the pool). You might see in the logs it use the same worker during the deployment. But when you work under that assumption of different workers per step, that makes you write your process so it is resilient enough to handle the rare cases when that happens.
In my case I have a step which downloads a package and generates a Redgate Database Release. It is not doing a “deployment” It only needs to extract the package and run some scripts against a database.
I have a step template which runs a PowerShell script. First I am going to add a variable to reference the package name.
You can then pass in the package name to the step template. For this particular step template I also made it so the feed is a variable. If you use the same feed that isn’t necessary.
Please note: once the step is finished, that directory is deleted. If you have multiple steps attempting to access the contents of the same package, I’d look into having a script copy that package to a shared location.
The problem with this approach is that it won’t allow is to gradually transition from regular Tentacles to Workers with the same step template, as we’d be changing how the step template works. How come you don’t offer a simple “extract a nuget at this location” type option for workers?
btw, don’t get me wrong, it will probably work and I’m thankful for the help. It’s just a more complex/difficult approach than what we envisioned (which is something we very often conclude with regarding Octopus things).
Yeah, to gradually transition you would need to create a copy of the step template, or step templates, to leverage this new functionality. It is not the worlds best solution, but it would be less of a big bang.
The reasoning behind limiting the deploy a package step to targets only is because of all the extra functionality provided by the deploy a package step. You can configure it to have any of the following features, several of them related specifically to deployments.
Workers weren’t designed for deployments like that. One worker would get the package and have custom functionality applied while the others wouldn’t. Steps weren’t really designed to disable/enable functionality based on the type of target. The package reference functionality in run a script step was what seemed the most logical to us.
Looking at the package reference solution now, one definitive drawback is the loss of dropdown to easily select package/feed id. Developers would have to copy/paste that in and hope they got it right.
It’s also not clear to me how versions would be handled, as the “deploy package” would have a different version from the Step Template package.
For the most part Octopus will do its best to run on the same worker during a deployment. But, it will switch workers when needed. For example, if a worker were to be deleted or go offline. At first my deployment was using db-worker-01.
This doesn’t looks like its working with step templates that have package references like the guide posted above shows how to set up. Looking at the rest api call when I create a release manually, it looks like this:
This Cake code again just calls octo.exe, which has the following documentation:
--package=VALUE [Optional] Version number to use for a package
in the release. Format: StepName:Version or
PackageID:Version or
StepName:PackageName:Version. StepName,
PackageID, and PackageName can be replaced with
an asterisk.
I don’t see any way to specify ActionName etc in octo create-release.
So my question: Given a step template referencing a package, I need to construct the create-release command that allows me to specify individual versions for those packages. Could you help me with what that command would look like? I can’t figure it out - I’m unable to map the octo cli documentation to what I’m seeing in the rest api.
Without including the packagename in the command I kept getting the latest version which was 2019.1.2. That…was annoying. It also makes it much more brittle.
We have a couple of features in the pike we plan on working on. To do some of them, we will need to add a package variable type, which should solve some of these problems.
Workers, in their current state, is looking like more and more of a rabbit hole for your use case. With workers v1 being stateless, it feels like we are trying to fight against the grain. Believe me, the email you sent and this forum thread has triggered a lot of internal discussion around what the next iteration of workers should look like. We would really like to reach out to you for feedback when we start on that. Hopefully, we can come up with a good solution for you.
awesome, I’ll try that. Cake expects a KV pair, but there’s nothing stopping us from making the key a colon-separated concat of multiple strings. We’ll make it work.
Regarding workers in general: Yeah. Definetely not as easy as we’d hoped it would be. I guess if I understood the reasoning for some of the descisions it would be easier to deal with but some of the limitations just seem…random I guess :-).
Glad to hear there are discussions ongoing. Nothing would please me more than if Octopus would cater better for cross-platform shop just trying to run their own custom bash scripts
sorry, just looked at your suggestion now.
Given that I have a step template with version 2.0.0. Its referring to another package (the deploy package), of which I want to deploy 3.0.0.
What would be the correct octo.exe syntax? In your example you only specify the version of the deploy package not the step template package version. We need to specify both.
? (havent tried it, just thinking out loud). octo.exe documentation lacks information on how to specify multiple packages, I don’t even know if using the --package parameter twice is allowed.
EDIT: Just tested this now, and it seems to work: The double :: is used to indicate the step template version itself:
This means that we’ll excplicitly set the step template version to 2.0.0, while other packages in the deploy will refer to defaultpackageversion to get their version.
I must say I have seen cleaner cli’s
To answer your question, you can specify the --package parameter multiple times. It looks like the doco needs to be cleaned up a bit and some additional examples included. Other parameters which can be included multiple times have a blurb stating that.
Thank you again for the feedback! We always appreciate honest feedback (even brutally honest), that is the only we as a company can improve.
