Variable Scoping and Dynamic Feeds

Octopus Server
, , , ,
1

My team recently experienced an issue with the way we had variables scoped for Kubernetes deploys, but I believe that it worked at one point and has now changed with a more recent upgrade of our Octopus server. I’m hoping someone can shed some light on what you expect so that our assumptions match yours.

Example Process:

  • External Feeds
    • NonProd ECR
    • ProdNA ECR
    • ProdEU ECR
  • Environments
    • NonProd
      • Targets
        • NonProd K8s Cluster: Roles = Kubernetes, NonProd
        • NonProd WebServers: Roles = WebServer, NonProd
    • Prod
      • Targets
        • ProdNA K8s Cluster: Roles = Kubernetes, Prod-NA
        • ProdEU K8s Cluster: Roles = Kubernetes, Prod-EU
        • ProdNA Web Servers: Roles = WebServer, Prod-NA
        • ProdEU Web Servers: Roles = WebServer, Prod-EU
  • Process
    1. Deploy Config: All variables resolve based on NonProd, Prod-NA or Prod-EU
    2. Deploy Backend Apps: On Behalf of Kubernetes
    3. Deploy Web Apps: On Behalf of WebServer
    4. Run System Tests

In the above pattern, we use a variable named ContainerRegistry with values scoped to NonProd, Prod-NA and Prod-EU to take advantage of dynamic feeds for K8s deploys. This process pattern has been working for awhile now, but we’ve observed that when it comes to the package feed referenced by “Deploy a Kubernetes Container” step template our variables are not resolving and therefore all production pods are referencing the NonProd ECR in our K8s cluster. The issue is, the pods are deployed the first time just fine so we don’t have any issues until the first time K8s needs to reschedule the pod.

I do know that we’ve updated Octopus a handful of times in the last 1-2 months and are within a version of the latest release now; I am not sure which version we were on when this was working the way our process is built.

So, question is, can a variables be expected to evaluate based on target roles of a target included in a deploy even though the steps in the deploy are evaluating based on different roles associated with those targets? If so, there may be a bug with the step template we are using. If not, do you have any suggestions for our process that accomplish the same outcome?

3

Hi @OfficeSpacePirate,

Sorry to hear you’re having issues with this.

Which specific version of Octopus Server are you currently on?

Would you be able to send me a screenshot (via private message if you prefer) of your variables page, as well as the step that is giving you issues? Could you also send me the deployment process JSON? To get this go to the Project, then the process section, then click the 3 dots to the right of add step and click download JSON.

Please let me know.

Thanks,
Jeremy

4

I can, but my example above is far simpler than our actual process and variables. Are you sure you’d rather parse through a giant JSON? Also, does this mean that what I’ve outlined should work?

5

Is step 2 the one giving you issue?

You have it deploying on behalf of Kubernetes, but how is the feed referenced within that step?

I would really only need the information from the problem step to try and reproduce the behavior you’re seeing on my end.

Please let me know what you think.

Thanks,
Jeremy

6

Ok, then I would suggest we avoid the noise :slight_smile:

Yes, we are deploying on behalf of Kubernetes, but the feed is referenced based on NonProd, Prod-NA or Prod-EU.

The reason we were trying to do this is because some things change based on environment, some based on AWS region, some based on target type (K8s vs standalone servers), etc. Target roles seemed like a really convenient way to achieve that.

7

Would you be able to send me screenshots of how you’re referencing the feed within the step?

And just to be sure I understand, you have a Deploy container step that’s scoped to Kubernetes role, and you want it to resolve based on the second role of the target, but it’s always resolving to the NonProd version of the variable (even if the second role is ProdNA/ProdEU)? That variable only has those three values, and all 3 are explicitly scoped, correct?

Thanks,
Jeremy

8

Almost totally correct:

  1. We use the Deploy Kubernetes Containers step on behalf of the target role Kubernetes
  2. The instances of that step deploy {Id of image here; not a variable} from #{ContainerRegistry}
  3. The variable ContainerRegistry has three values
    • {Id of NonProd ECR feed} - unscoped
    • {Id of ProdNA ECR feed} - scoped to target role Prod-NA
    • {Id of ProdEU ECR feed} - scoped to target role Prod-EU

Also, I did try scoping the NonProd value, updating the variables on the release and redeploying; it ended up responding with an error saying that it #{ContainerRegistry} no longer resolved with a value.

9

Let me do some tests to try and reproduce the behavior.

Which version are you currently on exhibiting the behavior? You can find your version in any task log or in the upper right when you click your profile.

Thanks,
Jeremy

10

We are on 2020.3.6

11

Thanks for that. I’m gonna do some testing and I’ll get back to you with more information.

Please feel free to reach out in the meantime with any questions or concerns.

Thanks,
Jeremy

1 Like
12

Hi,

Could you please send me a screenshot of the container part of your step so I can see exactly how you have this setup? (Where you’re defining the package ID and feed)

Thanks,
Jeremy

13

Like this?

image
image2521×1457 316 KB

14

Hi Clayton,

I’ve reproduced your error. I’m going to reach out to engineering and discuss this with them and get back to you.

Please let me know if you have any questions or concerns in the meantime.

Thanks,
Jeremy

1 Like
16

Hey Clayton,

I did some more digging, and unfortunately this one looks to be an old bug. One that can’t be fixed until the entire deployment process gets redesigned. https://github.com/OctopusDeploy/Issues/issues/2920

As a workaround, would you be able to have a meta project that has you deploying a release of 3 child projects within it that have the correct feeds for each role?

Please let me know what you think.

Thanks,
Jeremy

17

I will just need to change how we use roles/variable scoping. I really appreciate your time looking into this.

Thanks,
Clayton

18

Hey Clayton,

You’re very welcome. I’m sorry there wasn’t a better outcome.

Hopefully your redesign goes smoothly!

Best,
Jeremy

19

Hey Jeremy,

I just realized that this doesn’t have to do with the two-layer target role scoping I was utilizing, but target roles and feeds period. Because our deploy is pushing to two targets at the same time, I don’t know how to scope this variable to make it work at all. In the example you gave, I could have two child steps of one that run in parallel, but then the change from nonprod to prod is still not possible while also differentiating between NA (North America) and EU (Europe). Ideas?

Thanks,
Clayton

20

Hey Clayton,

Sorry about that. You’re right, unfortunately, it’s all roles and feeds.

I think you have two ways you could do this.

First way: Have 3 environments: NonProd, ProdNA, ProdEU, and scope the feed to the environments.
Second way: Have 2 environments: NonProd, Prod, and 3 tenants (NonProd, ProdNA, ProdEU), and make the feed a tenant variable. (you could also work tenants into your current setup potentially.)

Please let me know what you think.

Thanks,
Jeremy

21

After looking through the documentation for 5 minutes, it looks like maybe Tenanted Deployments are what we should have been doing in the first place. It will take me some time to get everything moved over, but I will let you know after I’ve tried it out.

Thanks again for your help,
Clayton

22

Hey Clayton,

You’re very welcome! Please let me know how the migration to using Tenants goes.

Best,
Jeremy