Best approach for deploying related and unrelated projects

Advice
#1

We have components that might be made of database, static scripts and a service.
We have projects which will be made up of a number of components
We have products that are made up of a number of projects
We would like to be able to deploy at any of these levels - a service/database, a component, a product or all products

We have auto-deploy to Dev on changes being committed to source control
We would like to have an overnight deploy to QA - taking the version that it is on Dev (database, scripts and service)

I have set up ‘component wrappers’ that will deploy a single component - but I haven’t used these yet as I’m not sure how the version numbers will relate - the component wrapper version and the parts of the component (DB, Static data, Service) that have been.

I’m also not sure if it’s worth creating a project-level wrapper that includes all the components for that project - again, what will happen with the version numbers - project, component and individual items (DB, scripts, Service)?

Some changes mean we need to deploy the whole project or all products (core components changed).

Can you give some advice on the best way to achieve the above?

#2

Thank you for reaching out. I am very curious about this particular statement:

We would like to have an overnight deploy to QA - taking the version that it is on Dev (database, scripts and service)

Does this mean your desire is whatever is in Dev will automatically get promoted to QA at night? For example, you have the following:

  • Dev: 2020.6.1
  • QA: 2020.6.0

Would you want it set to automatically push 2020.6.1 to QA? However, in a scenario where both versions are the same, would you want the process to do nothing?

Next question, is this to solve the push from Dev to QA only? Or are you looking for something to help promote from QA to Staging or Staging to Prod? Follow up question to that, will the push from QA to Staging or Staging to Prod need to be automated in a similar fashion as Dev to QA, on a nightly schedule? Or will that be a once a week/fortnight/month scenario where you manually push once QA is finished testing?

#3

Thanks for the quick response Bob.
Yes. If Dev is 2020.6.1 and QA is 2020.6.0, upgrade QA to the Dev version
If they are the same, doing nothing would be nice

We don’t currently intend automating deployments past QA (to staging or Prod/Live). This would be a manual step once QA approved the release they have tested.

As I see it, we need to be able to deploy ‘components’ (DB, Static data, Service), Projects (a number of components) and products (a number of projects) without the need for manually deploying every item.

Obviously, if there is a better way, I’d be happy to hear alternative approaches.

It seems the Scheduled Trigger does what we want for the QA deployment, but rather than go through and add this to every project (over 100 projects) I was hoping I could create projects to deploy the other projects - but then these wrapping projects would have their own release numbers and wouldn’t be used in the auto-deploy to dev.

Thanks
Richard

#4

The scheduled triggers was going to be my first recommended approach, but it might not meet your exact needs. What I would’ve done is write an API script to rip through all the projects and create a new trigger. The body of the trigger doesn’t require you to specify a specific step or role, which makes it much easier. Here is an example of the body.

image
image750×584 17.8 KB

The concern I have with the wrapper project (and wrapper of wrapper projects) to auto promote from dev to QA is depending on your approach, you could end up creating a lot of releases that don’t go anywhere.

That being said, there is some value in having a wrapper project. You can control the order of what is being pushed up, rather than trying to “guess”. With the deploy a release step you can set the deployment condition to be “if the selected release is not the current” or “if the selected release has a higher version” to skip over any projects that have already been deployed.

image
image1897×1200 231 KB

The downside to the deploy a release step is the release number of the child project is part of the snapshot. It isn’t smart enough to say “deploy the latest release of 2020.6.x to QA/Stage/Production.” Instead, it says “the version of the child project is 2020.6.1 and that is what it will be for this release.”

My recommendation would be to set up that wrapper project. With a few caveats

  1. Create two lifecycles that will only be used by this wrapper project. The first one has QA only in it. That lifecycle will be used for your auto-promotion from Dev to QA (you don’t need Dev in this lifecycle, only QA). That lifecycle’s retention policy is very aggressive. The second lifecycle is for your environments beyond QA (Stage / Production). You’d create a manual release for that when you’re ready to promote.
  2. Set the deployment conditions to be either if the selected release is not the current release or if the selected release has a higher version than the current release. That way you only deploy when something has changed.

I hope that helps!

#5

Many thanks Bob, a very detailed reply.
I’ll try the wrapper-projects and see what that does for us.

Thanks again
Richard

#6

I have few questions about the wrapper approach:

  1. I’m not sure what you mean by “The downside to the deploy a release step is the release number of the child project is part of the snapshot. It isn’t smart enough to say “deploy the latest release of 2020.6.x to QA/Stage/Production.” Instead, it says “the version of the child project is 2020.6.1 and that is what it will be for this release.”” Will this still promote the latest version on Dev to QA?

  2. How do I use the 2 lifecycles in a single wrapper project? Change the lifecycle when I want to go to live and back for the next QA deployment?

  3. Will I need to create a new release of the wrapper, then deploy that?

  4. Is there a relationship between the wrapper version number and the deployed ‘step’ versions numbers?

  5. Do all the contained parts (database, static script, service) and the wrapper need to have the same lifecycle? A colleague who was trying this said they did have to have the same lifecycle

  6. Is it possible to have a hierarchy - so a product wrapper deploys a number of project wrappers as separate steps. The project wrappers deploy component wrappers as separate steps and the component wrappers deploy the database, static script and service as separate steps? Would there be any drawbacks to that - version numbers problems, not being able to have a hierarchy etc?

Thanks
Richard

#7

Hi Richard,

Let’s dive into your questions by walking you through setting this up on a sample project. First up, you’ll want to create the new lifecycles. In my case, it is “test” instead of QA, but the idea is the same.

image
image617×769 18.3 KB

Please note: You really don’t need two lifecycles, I recommend having one so you can enforce version rules in the staging/prod lifecycle.

In your orchestrator project, you’ll have two channels, one for stage/prod (the default), the other for Test.

image
image1276×715 43 KB

Next up is a scheduled trigger. This is where the secret sauce is. You’ll be creating a new release, to deploy to the test channel, and deploy to the test environment.

image
image1882×1116 176 KB

When that release is created by the trigger it will grab whatever version is in Dev and promote it to Test/QA. In my case:

image
image946×937 65.2 KB

image
image1525×1252 116 KB

When it is time to promote from QA to Staging / Production a new release will need to be created. When you create that release you select the channel and the project versions to run. The one downside of the deploy a release step is it snapshots the version number. By default it picks the very latest it can find in Dev, not QA. You can create channel version rules to help manage some of that.

image
image1873×1434 273 KB

Let’s see if this answered your questions.

  1. Yes
  2. The trigger will be configured to run the test (or QA) only channel. When you need to promote to staging you’ll create a new release with the specific versions from QA you wish to deploy. If you only have a test -> staging -> prod lifecycle you won’t need to create a new release.
  3. Yes, in this scenario. If you only have a test -> staging -> prod lifecycle you won’t need to create a new release.
  4. There can be. In the settings screen you can choose the package to base the version number off of. I wouldn’t recommend it for the trigger as you’ll end up trying to create a release with the same version number over and over.
    image
    image1901×974 97.8 KB
  5. No, they don’t need to have the same lifecycle. The rule is the child project’s lifecycle’s conditions must be met for promotion.
  6. It is possible, but you are increasing complexity. There is only so much a human brain can hold.

I hope that helps!

#9

Thanks again for an amazingly detailed description.
If I create a new lifecycle with all 3 environments in, I assume I don’t need to have 2 channels set up.
As a matter of interest, how did you get your lifecycle without phases in? If I try that I get an error about only the first one being allowed without phases…

#10

Hi Richard,

The default lifecycle is the only one without phases specified. All the other lifecycles have phases.

image
image1578×920 45.6 KB

#11

Hi Bob. I have the lifecycle setup (QA, Staging, Live/Prod), created 2 channels and have tried to create the Trigger. However, when setting-up the Trigger, I can’t select the Dev environment as the source environment. What am I missing?

If I try to use it without a trigger, it doesn’t deploy the version from Dev, but from another environment (I don’t know which yet). What controls which version is promoted?

Thanks
Richard

#12

Hi Richard,

You’ll want to configure the trigger action to be Deploy new release in the parent project with the destination environment of QA. When that new release is created it will pick the latest changes it can find.

image
image1885×1115 182 KB

#14

Many thanks for the detailed response Bob. I’ve set things up how I think they should be, and it’s mostly working. However, some of the projects aren’t deploying the latest version from Dev, but is deploying the highest version from any of the other environments (staging, temporary test environments, other customer environments etc) - e.g. Dev is at 1.10.0.21, QA is at 1.10.0.20 and a customer-specific Staging is at 1.10.10 - and the customer-specific staging version is the one that gets deployed.
Does Octopus pick the latest from any environment? Is there a way to specify which version or which environment should be promoted?
Thanks
Richard

#15

Via a trigger there isn’t a good way, you’d be required to create channel rules which add to the complexity.

It is interesting it is picking 1.10.10, but you mentioned that is customer specific. Is this a multi-tenant project, and if do dev and QA being deployed untenanted or?

#16

Thanks Bob. That’s not good news. We’re not using tenants, this is the same code deployed to different servers for each ‘client’. So we have our generic product which has - Generic QA, Generic Staging and Generic Live. Then we have a client product, built from the same code, Client1 QA, Client1 Staging and Client1 Live.
We also have a number of other environments falling (generally) in the Dev environment - Dev.Laptops, Generic Staging, Client1 Staging (mmm, that could be my problem).
Is there a way to perform an overnight deploy that promotes the current version on Dev to QA?

Thanks
Richard

#17

What is tripping me up is why it didn’t pick 1.10.0.21 for the child projects. Isn’t that the most recent release created?

#18

it’s because the 1.10.10 has only 3 fields (major.minor.buildnumber) while the real ones are 4 fields (major.minor.patch.buildnumber) and 1.10.0.21 is lower (patch number) than 1.10.10…
This was due to an error on someone else’s part while testing other deployments. What I’m concerned about is not knowing which environment it’s promoting from. It should always be Dev, not one based on rules I don’t know about…
Thanks again for your time
Richard

#19

Hrm, another option will be to use the chain deployment step template instead of the deploy a release. https://library.octopus.com/step-templates/18392835-d50e-4ce9-9065-8e15a3c30954/actiontemplate-chain-deployment

The key difference with this one is you can either specify a release or ask for the latest release in the subproject’s channel.

#20

Thanks again Bob.
I got to the bottom of the issue last night. The scheduled deployment was promoting the latest build to QA - regardless of whether that had been deployed or which environment it had been deployed to. So, if a project hadn’t changed for a month and was at 2.15.6.50, but a test version had been created later at 2.15.7, the automated deployment would promote 2.15.7.
I worked around the problem by deleting all the later releases which had 3-field version numbers (major.minor.patch). Obviously we need to be aware of this if we do manual releases again.
So it seems we can’t use this to promote what is on Dev, only the latest release.

Is there something we can change to promote what is on Dev, rather than the latest build?

I looked at the chain deployment step, but couldn’t see a way to use Dev as the source. Am I missing something?

Richard

#21

Hi Richard,

Hrm, it shouldn’t be this hard. In my mind, this is a very common use case. I am working on a step template that will hopefully solve this use case. Timeline TBD, but I hope to have it out early 2021.

#22

Thanks Bob. That was my feeling too. Octopus is so good, I’d think this should be bread and butter for it.
I was also wondering if we’d have the same problem with the next deploy - QA to Live/Prod

Richard