Manage a Kubernetes cluster through the "Apply a Terraform template" step

Octopus Cloud
#1

I have a Terraform template that needs to update a Kubernetes config map. To do this, I’m attempting to use the Apply a Terraform template step and a Kubernetes deployment target. However, the Apply a Terraform template step is unable to connect to the Kubernetes cluster.

Given that there are no configurable features for the Apply a Terraform template step, and thus I can’t add a pre-deploymet script to set ~/.kube/config, how can I use Terraform to manage a Kubernetes cluster?

Many thanks,
David

More information

Step 1

First, Kubernetes cluster (AWS EKS) and associated resources are deployed using the Apply a Terraform template step. This step is working successfully.

Step 2

Next, a deployment target is dynamically created by using the AWS CLI script step to describe the cluster that was created by Terraform, and then create the target via the Create Kubernetes Target command. This step is also working.

Step 3

Finally, a Kubernetes config map needs to be updated through Terraform, and it’s this step that is proving problematic.

The step also uses the Apply a Terraform template step, but this time the step is run on behalf of the deployment target that was dynamically created by the previous step. The hope was that the worker could use the kubectl CLI with ~/.kube/config as required (because I assume the Kubernetes deployment target sets it somewhere), but that doesn’t seem to be the case judging by the error message -

Error: Post “http://localhost/api/v1/namespaces/kube-system/configmaps”: dial tcp 127.0.0.1:80: connect: connection refused
  on …/…/modules/user-management/user-management.tf line 3, in resource “kubernetes_config_map” “aws_auth”:
    3: resource “kubernetes_config_map” “aws_auth” {

#3

Hey David,

First of all, welcome to the Octopus Forums!

Thanks for reaching out.

I’m currently digging into this to find the best solution for your scenario. I will update you as I progress.

Please feel free to reach out in the meantime with any other questions or concerns.

Best,
Jeremy

#5

Hey David,

I think the answer here is to use the Terraform provider directly to configure the connection. The docs here have the details: Manage Kubernetes Resources via Terraform | Terraform - HashiCorp Learn

Please let me know if that works or if you have any other questions or concerns.

Best,
Jeremy

#6

Hi Jeremy,

Sorry for the delay on reply with this one.

I had already looked in to the Kubernetes provider through Terraform, but due to the fact that the Terraform deployment first creates the EKS cluster, the provider can never work within the template (as far as I can see - but I am not a Terraform expert).

Instead, I’ve had to add a few additional tools to our Docker image (now has aws, kubectl and terraform) that we use to deploy and instead use an AWS CLI script to first set ~/.kube/config. It’s not ideal as it’s harder to maintain several slightly different scripts as opposed to to the Terraform steps, but it works.

Here’s a basic example…

#!/bin/bash

### Change to the directory containing the Terraform template to apply.
cd ./path/to/dir

### Initialise Terraform deployment.
terraform init -no-color -get-plugins=true

### Plan Terraform deployment.
terraform plan -no-color -out=octopus.tfplan

### Apply Terraform deployment from plan.
terraform apply -no-color -auto-approve octopus.tfplan

Thanks,
David

#7

Hey David,

Thanks for the information.

I’m going to pass this along to our Kubernetes SME to see if there’s a more maintainable way to handle it.

Please feel free to reach out in the meantime.

Best,
Jeremy

#8

Hey David,

I heard back from my colleague.

He said if your terraform template has Octopus template placeholders, you can inject values before terraform sees the template.

like this:

provider "kubernetes" {
  host = "https://cluster_endpoint:port"
  client_certificate     = #{CertFile}
  client_key             = #{ClientKey}
  cluster_ca_certificate = #{CACert}
}

Please let me know if that helps.

Best,
Jeremy

#9

Thanks for the reply Jeremy,

The issue I’m facing here is that the Terraform template will first create the Kubernetes cluster (EKS in AWS), so it’s impossible to know those values beforehand.

Thanks,
David

#11

Hey David,

These variables should already be available to you within your deployment as output variables.

Can you please turn on Variable Logging and find the variable names with the required information for the above template injection?

Please let me know if that works or if you have any questions about the above.

Best,
Jeremy

#12

Hi Jeremy,

I think we’ve got our wires crossed somewhere. A (simplified) Terraform template will -

  1. Create VPC, EKS cluster and node group.
  2. Import the AWS auth ConfigMap that was generated when the EKS cluster was created.
  3. Update the AWS auth ConfigMap.

If I were to declare the Terraform provider in the template, the EKS cluster wouldn’t yet exist, so it would be impossible for me or Octopus to know the required values.

From looking at the documentation it doesn’t seem that you can make a Terraform provider depend on a resource, and instead it’s configured when you run terraform init. This also means that I can’t use Terraform output variables, as the provider is configured before the template is applied and thus those output values are not known.

If I’m misunderstanding you then any further explanation would be great, but unfortunately I don’t think Terraform is able to do these mildly complex things.

Thanks,
David

#13

Hey David,

Sorry for the confusion and thanks for the clarification. I’ve been looking into this with a colleague today but I’m going to reach out to our SME again and get his take.

Please let me know if you have any questions in the meantime.

Best,
Jeremy

#14

Hey David,

Would you be able to send me a deployment process JSON?
To get this go to the Project, then the process section, then click the 3 dots to the right of add step and click download JSON.

You can DM it to me for privacy.

Please let me know.

Best,
Jeremy

#15

Sure, happy to share via a DM.

Note that the project has turned into a bit of a beast though, so not sure if it will be helpful.

Thanks,
David

#16

Hey David,

Thanks for uploading it.

We’re going to take a look and I will update you as soon as I can.

Best,
Jeremy