I have a Terraform template that needs to update a Kubernetes config map. To do this, I’m attempting to use the Apply a Terraform template step and a Kubernetes deployment target. However, the Apply a Terraform template step is unable to connect to the Kubernetes cluster.
Given that there are no configurable features for the Apply a Terraform template step, and thus I can’t add a pre-deploymet script to set ~/.kube/config, how can I use Terraform to manage a Kubernetes cluster?
Many thanks,
David
More information
Step 1
First, Kubernetes cluster (AWS EKS) and associated resources are deployed using the Apply a Terraform template step. This step is working successfully.
Step 2
Next, a deployment target is dynamically created by using the AWS CLI script step to describe the cluster that was created by Terraform, and then create the target via the Create Kubernetes Target command. This step is also working.
Step 3
Finally, a Kubernetes config map needs to be updated through Terraform, and it’s this step that is proving problematic.
The step also uses the Apply a Terraform template step, but this time the step is run on behalf of the deployment target that was dynamically created by the previous step. The hope was that the worker could use the kubectl CLI with ~/.kube/config as required (because I assume the Kubernetes deployment target sets it somewhere), but that doesn’t seem to be the case judging by the error message -
Error: Post “http://localhost/api/v1/namespaces/kube-system/configmaps”: dial tcp 127.0.0.1:80: connect: connection refused
on …/…/modules/user-management/user-management.tf line 3, in resource “kubernetes_config_map” “aws_auth”:
3: resource “kubernetes_config_map” “aws_auth” {
I had already looked in to the Kubernetes provider through Terraform, but due to the fact that the Terraform deployment first creates the EKS cluster, the provider can never work within the template (as far as I can see - but I am not a Terraform expert).
Instead, I’ve had to add a few additional tools to our Docker image (now has aws, kubectl and terraform) that we use to deploy and instead use an AWS CLI script to first set ~/.kube/config. It’s not ideal as it’s harder to maintain several slightly different scripts as opposed to to the Terraform steps, but it works.
Here’s a basic example…
#!/bin/bash
### Change to the directory containing the Terraform template to apply.
cd ./path/to/dir
### Initialise Terraform deployment.
terraform init -no-color -get-plugins=true
### Plan Terraform deployment.
terraform plan -no-color -out=octopus.tfplan
### Apply Terraform deployment from plan.
terraform apply -no-color -auto-approve octopus.tfplan
The issue I’m facing here is that the Terraform template will first create the Kubernetes cluster (EKS in AWS), so it’s impossible to know those values beforehand.
I think we’ve got our wires crossed somewhere. A (simplified) Terraform template will -
Create VPC, EKS cluster and node group.
Import the AWS auth ConfigMap that was generated when the EKS cluster was created.
Update the AWS auth ConfigMap.
If I were to declare the Terraform provider in the template, the EKS cluster wouldn’t yet exist, so it would be impossible for me or Octopus to know the required values.
From looking at the documentation it doesn’t seem that you can make a Terraform provider depend on a resource, and instead it’s configured when you run terraform init. This also means that I can’t use Terraform output variables, as the provider is configured before the template is applied and thus those output values are not known.
If I’m misunderstanding you then any further explanation would be great, but unfortunately I don’t think Terraform is able to do these mildly complex things.
Sorry for the confusion and thanks for the clarification. I’ve been looking into this with a colleague today but I’m going to reach out to our SME again and get his take.
Please let me know if you have any questions in the meantime.
Would you be able to send me a deployment process JSON?
To get this go to the Project, then the process section, then click the 3 dots to the right of add step and click download JSON.