We have a third party company pushing nuget packages to their nuget repostiories which we then use as external feeds. In order to achieve some level automated of CI/CD I have been looking into how we can poll the external end points and check that against whats already been deployed. I know it can be done through the API (although that comes with it’s own complications for multi package projects) but with the ability to reference package metadata as a parameter in a process step, it seems like a good way to do this would be to say “get the package from the feed, check it against the package deployed in x step in the last successful release, if they aren’t the same version start a deployment”. I tried Octopus.Tentacle.PreviousSuccessfulInstallation.PackageVersion but i think because at least one of the projects isn’t actually deploying a package (the package contains a dacpac and the dacpac step doesn’t install the package anywhere as far as i know) this variable won’t be the answer. Is there a way to just reference a process step in a previous deployment and get the package metadata from that?
Hello Adam,
I’m interested in hearing more about your deployment set up and what you’re wanting to accomplish.
When you talk about comparing versions, is the goal to avoid deploying certain packages from the external feeds? Or just ensuring that you’re only deploying new versions and not redeploying existing packages?
Generally speaking, your points above are correct - using the API is likely the easiest way to collect the data you need, although that doesn’t help when there aren’t clear installation versions (like your DACPAC scenario you mentioned). It may be possible to use a script to set an output variable with the deployed version of the package and compare against that data.
If you can provide some more detail about how you’re wanting to utilize the information, we can hopefully come up with a better solution to help make that possible.
Hi Cory,
Thanks for getting back to me. The purpose of what i’m trying to do is simply to create an automated release creation process, much like you’d set up from a build server, but as we don’t have access to the source control and the third party don’t have access to our octopus deploy server, I have to base any solutions on whether there is a new package in the external NuGet feed. As i understand it, the built in octopus feed allows for release creation when a package is added, but this isn’t possible for external feeds? I’m just wondering if there is a solution that is recommended to achieve this using external feeds.
Thanks,
Adam
Thanks for the added info Adam!
A couple of additional questions if you don’t mind answering - firstly, what sort of NuGet provider are you using with your third party? Wondering if there are webhooks or other mechanisms you can respond to from that feed that would help you get consistent release triggers.
If there aren’t any native triggers or API capabilities, the next option would be to set up polling of the feed so you can create releases when new packages show up. This can be done a couple of different ways - if you create runbooks, you can continue controlling all of this through Octopus.
Let me know what makes the most sense and I can get you some more details around setting those solutions up to make sure they work the way you’re wanting.
Thanks again for getting back to me, i fear i may have not been clear enough in what the aim of my question is…
To answer your question, they are using the azure dev ops built in git repository for their package hosting.
I have already written a powershell script that can call their end point, get the latest package and then i can use that info to interrogate some info somewhere about whether or not the latest package is new and from there I could use the Octopus API to create a release. What I really want to know boils down to a couple of questions:
- is there a way to find out “is this a new package” without keeping that information physically somewhere? At the moment my best effort has resulted in a text file stored on the disk where the script runs, and this can be interrogated and updated if the versions don’t match.
- is there a less clunky way of doing this, similar to how it is possible to just set up a trigger on octopus deploy’s built in package repository, that you know of? Ideally it would be built in to Octopus Deploy but i get that i can just run the powershell in a runbook. This brings me back to the first point, and the title of this post, can i get the value of the last released package version?
2b) Part of what makes the powershell so clunky is getting the package info to provide to the octopus api for the release, is there a more simple way to just say to octopus “create a release with the latest package”?
Thanks,
Adam
Hey Adam, sorry for backtracking a bit, Monday has my head all backwards! With your specific questions:
- Is there a way to find out “is this a new package” without keeping that information physically somewhere? At the moment my best effort has resulted in a text file stored on the disk where the script runs, and this can be interrogated and updated if the versions don’t match.
Your best bet is to compare it to your most recently deployed version. There are a couple of sample scripts in the Rest API example repo like GetLastSuccessfulForProjectAndEnvironment that you can use to get the most recently deployed version. This should help you compare most recent deployed version against available versions coming into the feed.
- Is there a less clunky way of doing this, similar to how it is possible to just set up a trigger on octopus deploy’s built in package repository, that you know of? Ideally it would be built in to Octopus Deploy but i get that i can just run the powershell in a runbook. This brings me back to the first point, and the title of this post, can i get the value of the last released package version? 2b. Part of what makes the powershell so clunky is getting the package info to provide to the octopus api for the release, is there a more simple way to just say to octopus “create a release with the latest package”?
Unfortunately, the Automatic Release Creation in Octopus only supports the built in feed, which makes automating around external feeds more difficult. As far as selecting versions - the default behavior of the Octopus CLI tool’s create-release command is to use the latest version of the release (documented here).
I know PowerShell can be clunky , especially when you’re just trying to fill in gaps around existing features, but hopefully the above can help get you started in the right direction. Hopefully that at least pushed the conversation forward, let me know your thoughts and we can keep digging in.
Hi Cory,
So i’ve sort of settled on a hybrid of the various methods we’ve discussed in order to get this to work and have created multiple runbooks with a step template that has a package control type as a parameter. I’ve definitely not ironed out all the kinks yet but the workflow basically uses the external feed from within Octopus Deploy, uses a file on disk as i did previously to compare versions, and then creates a release using the octo cli with just a project name.
We’re moving away from my initial question now but I do have an issue off the back of this: ideally i want 1 project with multiple runbooks that can run for each package that needs checking. When I have no scope on my variables it works as expected but when i scope the package feed to a process (runbook) it throws an error at run creation, i have attached a screenshot of the error.
As stated, this runs fine with a parameterised package feed as long as there is no scope, but seeing as i want to use multiple run books in the one project and the packages use different feeds, i will need to use a scoped variable for the package name and package feed. Is this expected behaviour?
Finally and slightly less importantly, is there a way to group run books? One of the projects for instance uses 3 packages, it would be useful if there was a way to say “if this release is created for package A then we dont need a release for package B and C even if they’re new so stop processing” as the intention will be to create the release with latest for all packages.
Thanks again, you’ve been very helpful thus far.
Glad you’re getting closer!
For the error, have you tried updating the variable snapshot on that specific runbook snapshot?
From the runbook overview, you can click the three dot menu in the top right, followed by View Snapshots, then click on the published snapshot and Update Variables under the Variable Snapshot heading. That should update your variable associations for the snapshot. Alternatively, you can publish a new snapshot using the latest changes. If that doesn’t solve it, may need to look a little closer at how it’s being scoped to determine what’s going on.
As far as your final question, there are a couple of ways you can approach it. You could use an orchestration project that calls the Run Octopus Deploy Runbook step template, and use a combination of run conditions and output variables to control the flow of your process. It all depends on how exactly you want it to behave, happy to talk more specifics/provide examples if you’d like!
I’ve tried updating the snapshot yes, I just published and tried that too and it’s the same error. I’ve attached a screenshot of the variable and the scope on it, as you can see, the 2 values are scoped to the 2 processes:
As far as I can tell this is working for other variables with the same scope. This includes the “run” screen where it asks me to select the package (also attached): but when i actually click “Run” in the top right I am presented with the error.
I’ll take a look at orchestration and let you know how i get on!
After a little testing I have hit my first hurdle. Can i set and access output variables from a runbook? So i set the deployment step to run Runbook-A and then have a second deployment step to run Runbook-B. In the powershell for the runbook i add a line “Set-OctopusVariable -name “ReleaseCreated” -value “True”” if a release is created and then in the process step for “Run runbook-B” i set a run condition:
#{if #{Octopus.Action[Run Runbook-A].Output.ReleaseCreated} == “True”}True#{/if}
This value keeps evaluating to false. I assume this is because the runbook is running as a separate task to the process step, is there a way to reference the runbook? I can’t find this info in the documentation.
Unfortunately, it looks like I may have misled you here. Because the runbook step template uses the API, it loses access to the output variables.
I talked to my team about your use case, and another team member pointed to another possible solution that may help you here. It involves using two endpoints to check the most recently deployed version and the most recently added version to the feed.
The first endpoint to know is OctopusURL/Api/[SPACE-ID]/DeploymentProcess/[Deployment-Process-Id]/template . This endpoint returns the base information for the deployment process, including a Packages object with a key called VersionSelectedLastRelease. This can get you the most recently selected version for your previous project deployment.
The second endpoint is OctopusURL/Api/[SPACE-ID]/feeds/[FEED-ID]/packages/versions?packageId=[PACKAGE-ID]&take=1 . This endpoint returns the package versioning metadata for the given package Id. The take 1 parameter ensures you only get the most recent result back.
With those two API calls, you should be able to compare the data between your most recently deployed version of a package as well as the most recently available package version in the feed. That should enable you to iterate in a single script and create releases as needed, which would keep you from having to track output variables across various individual runbook runs.
Let me know your thoughts on the above, and my apologies again for leading you down the wrong path with runbook output variables.
Hi Cory,
I think we’ve just about got there!
I haven’t actually started running it in octopus yet but for anyone interested going forward i’ll put my script below. It should only need one variable per project and will just run on a schedule. I’ve added a call to get the project by it’s slug so that variables are more friendly for anyone who may have to work on it in the future but you can just use the project id and skip that API call entirely if you want.
I ended up going with purely API calls and not using the octo CLI as this should make it slighly more portable if/when we need to migrate our octopus server removing 1 more dependency.
Script below:
$octopusURL = $env:OctopusURL
$apiKey = $env:OctopusAPIKey$ProjectName = [ProjectSlug]
$header = @{ “X-Octopus-ApiKey” = $apiKey }
#Get project info
$getprojectURL = $OctopusURL + “/API/projects/” + $ProjectName
$project = (Invoke-WebRequest $getprojectURL -Method Get -Headers $header).content | ConvertFrom-Json
#Get process info including packages
$processUrl = $OctopusURL + “/API/deploymentprocesses/deploymentprocess-” + $project.Id + “/template”
$process = (Invoke-WebRequest $processUrl -Method Get -Headers $header).content | ConvertFrom-Json
$releaseURL = $octopusURL + “/API/releases”$needsRelease = $false
#foreach package in the process, get that package from the feed and compare versions
foreach($package in $process.Packages)
{
$packageURL = $OctopusURL + “/API/feeds/” + $package.FeedId + “/packages/versions?packageId=” + $package.PackageId + “&take=1”
$mostRecent = (Invoke-WebRequest $packageURL -Method Get -Headers $header).content | ConvertFrom-JsonWrite-Host "Previous release version for" $package.PackageId "was:" $package.VersionSelectedLastRelease "and current version is:" $mostRecent.Items[0].Version if($package.VersionSelectedLastRelease -ne $mostRecent.Items[0].Version) { $needsRelease = $true } #for all packages regardless of whether it's new, add a new member to the object declaring the most recent version $package | Add-Member -NotePropertyName Version -NotePropertyValue $mostRecent.Items[0].Version}
#if a release is needed, “packages” will now contain our latest version data
if($needsRelease -eq $true)
{
Write-Host “Creating Release”
$body =
@{
ProjectId = $project.Id
ReleaseNotes = “put something here”
Version = $process.NextVersionIncrement
SelectedPackages = $process.Packages
} | ConvertTo-Json
Invoke-WebRequest $releaseURL -Method POST -Body $body -Headers $header
}
Thanks for your ongoing help with this! It was invaluable.
Glad to hear that’s got you moving in the right direction! And thanks for sharing your script, always interested in seeing unique/interesting solutions to use cases like this.
Don’t hesitate to reach out if you have any other questions or need a hand!