Approaches to integrating tools with Octopus

Legacy Modelling
1

I am in the process of expanding the use of HashiCorp’s Terraform tool in Octopus Deploy, in particular for AWS and some Azure deployments. Terraform is basically a desired state configuration based system where you describe the layout of the infrastructure and the tool figures out how to make it happen. It fits really well with Octopus, as it handles getting the environment ready to deploy, and Octopus does the deployment (our first use case is terraform to get ECS configured, Octopus to deploy the container). The tool is ultimately an exe and produces/consumes a variety of artifacts during usage. The primary interactions with Octopus are:

  1. variables (same basic usage as Octopus variables)
  2. remote state (the unique location of the deployment metadata/state in S3 (or some other store))
  3. state files (the input/output of terraform, as derived from AWS/Azure before/after deployment)

I have a PowerShell step template that drives the tool, but I’m looking to do something a little bit deeper. Some of things I would like to do:

  1. Have the ability for a user to define the set of step variables
    Right now, we are using the OctopusParameters dictionary to pull in all parameters, and have code that finds all the matching terraform variables, and then set the terraform variables to the values from Octopus. This works correctly, but it isn’t very self documenting, as there is no place to see/understand what Octopus variables are being used in a given run of Terraform. In some ways, I want a template for a step template, with a base set of parameters, but the ability to add more.

  2. Have the remote state being tied to the environment/project/deployment
    Right now, I’m setting the remote state location (S3 blob) based on step variables. This works, but any errors during the step execution may result in the state file not being updated (e.g., error in get/push to S3). What I would really like is the ability to provide/consume state files that are associated with the Octopus deployment/project/release. I can add deployment artifacts, but they are really just for reporting. For example, Octopus tracks the state of the files that it sends to the tentacles, and I would love to have it also track the state files, which would then be available in the next release/deployment. I don’t know enough about Octopus internals to know if these are the correct terms.

  3. Stop a deployment for human intervention.
    One of the features of Terraform is that you can generate a plan. You can then inspect the plan, and decide if you want to continue (e.g., check to make sure that your database isn’t able to be dropped). I would love to plug this into the manual intervention mechanism, so the user can be given the choice to continue the deployment in light what what will be changed.

There are likely other interactions that I’ll need. I can continue to hack my way through them, but it doesn’t hang together well. What I want to do is extend Octopus (e.g., Docker or Azure ARM integration) that goes beyond Step Templates. Is this possible? If so, is there any documentation?

Thanks,
Erick

2

Hi Erik,

Thanks for getting in touch! Good to hear you got Terraform working. We wouldn’t mind a PR if you were happy to share your work with a wider community :).

Going back to your main issue, as you noticed there is nothing in Octopus that is built specifically for Terraform. That being said there are a few generic features that you can use to make your life easier.

  1. To make things a bit more explicit you could pass required parameters using Script Parameters. If you think we should provide a better experience in this case then please go to User Voice and describe your ideal solution.
  2. I’m not sure I understand the problem correctly, an example would help here, but would Output Variables be of any use?
  3. You can create an Output Variable in the step that generates Terraform plan and then in your Manual Intervention step you can use Step Run Condition that is based on that variable.

Unfortunately at the moment there is no way to have a custom step template that has the same set of features as one of our built-in ones. This is something we are thinking about but we don’t have any specific plans yet.

In your particular case custom scripts executed via a Run a Script step seem to be the best way to implement integration between Terraform and Octopus.

Please, let me know how you go.

Regards,

Pawel

3

Hey Pawel,

Thanks for your reply. I’m happy to contribute to the community library once I get it working as I want.

  1. Nice addition with the Script Parameter! That does provide a bridge, even if it isn’t perfect. I will think on the solution that I would really like. Are there any existing proposals on User Voice in this area?

  2. It is a tough situation to describe. The basic gist is that terraform is driven by a file (a JSON file called the state file). In a majority of cases, the state file should be scoped to the project/environment. Imagine if instead of defining a variable, you could define a scope, and the “state file” (or really just a value) is a read/write value. A scenario might better describe it:

Project A in Environment E1 and E2 (E1 promotes to E2)
When a release of Project A is deployed to E1, it needs to provision infrastructure for Project A in E1. When it is promoted to E2, it needs to provision for E2. A new release is created that includes a change to the terraform. When a release of Project A is deployed to E1, it should load the state file that was last used when Project A was deployed to E1, modify it, and save it. The same thing happens during a promotion to E2. These types of resources would include things like DNS entries, load balancers, instance counts, VM sizes, etc. This scenario could be handled by storing the state file in S3, under a path similar to //.

The problem is that some resources are only scope to the project. For example, a shared redis cluster, where the environment is part of the key scheme. In this case, the state file should be shared for all environments/releases/deployments for the project.

Another example is artifacts related to a specific deployment. For example, a S3 bucket that stores a lambda or code file, or a set of docker environment variables.

In a perfect world, the user would be able to specify what the state file should vary by. So they could select Environment and Role, and then there would be a variable/slot/location that varies by Environment and Role that I can read/write from. Or they could select Project and Deployment, and so forth. In the step template, I would be able to load the value that was last used for the given scope condition (project, environment, role, deployment, etc). It’s almost the inverse of scoping a variable, where instead of assigning a value to a scope, you assign a pointer to a resolved scope.

I may be able to build a poor man’s version of this by adding a set of check boxes to the step template, one for each “vary by” and then building a path in S3. The problem is that this is data that is critical to the deployment, but is stored outside octopus. So if the project is renamed, the storage is now out of sync. I can use the ID of various things, but that is fragile in other ways.

  1. I am hoping that I can get the terraform work into a single step template, as otherwise it gets hard to maintain (e.g., having to keep the versions of both steps in sync across all projects).

What would be good is if I could “raise” a manual intervention request within the step template. For example, a lot of times a change to terraform will result in the (correct) destruction of some resources. This is correct, but in such a case we want positive confirmation, so it’s not a silent delete. A case with this is when you have linked resources in AWS, where changing the parent results in child resources being deleted (the children of the “old” parent), and the creation of new child resources (i.e.g, children of the “new” parent). This is often the resource of resources in AWS/Azure not being able to be changed in some way, and so they need to be deleted/created. As I mentioned, the delete/create is the correct behavior, but it is sometimes serious enough that we want to make sure that the user intended to take this action, and that it wasn’t the result of a bad merge or typo in the terraform file.

This is a lot of detail, and I’d be happy to provide any additional scenarios. Most of this I can probably hack around by creative use of step templates (e.g., the scoping issue). The one that I’m not sure I can make work is #3, as that is not something that I can find. I would think that some of these scenarios apply to a Cloud Formation or Azure ARM deployment as well, but I’m not as familiar with the behavior of those to say for certain how it would work.

Any feedback is greatly appreciated. The world of deployment keeps getting more complex :slight_smile:

Thanks,
Erick

4

Hi Erick,

Thanks for such extensive feedback. I appreciate it.

  1. There isn’t an existing User Voice suggestion. I guess Terraform just started getting momentum among our customers.
  2. Now I understand your scenario much better. Unfortunately there isn’t a anything in Octopus that would help you here and you would have to script all of that yourself. Happy to help with this part.
  3. There is no way to raise a manual intervention request from code. I also understand your concerns regarding having to maintain two steps. That being said we have recently (in Octopus 3.12) added ability to automatically update all projects to the latest version of a step template . Would that solve your problem? What version of Octopus are you on?

Please let me know if I can help in any other way.

Regards,

Pawel

5

Hey Pawel,

  1. I’ll give some thought and make a user voice suggestion. I think the gist will be that I can define parameters in the script (e.g., powershell arguments), and the UI would show all those in the step.

  2. Yeah, I didn’t think there would be. I’m going to go with an attempt to use a set of scoping checkboxes and see how far that takes us.

  3. I understand that feature, but I don’t think I would use it. It is really nice to have projects have snapshots of the step, as I’m usually improving the step template, and this often involves breaking existing usages. I will likely just fail the deploy, and force the user to set an “ok to delete” variable on the next deploy. I may come up with something else, but it’s a start.

Thanks for your help!

Erick