Terraform Apply Step | Manage AWS provider and NOT to refactor code for deployment

(Martin Woods) #1

Hi, I’ve Terraform OD Q:

I want to be able to run the same terraform syntax I’ve written locally in Visual Studio and test via the PowerShell Console to then run this same code in Octopus without changing/refactoring to fit Octopus. So that it’s generic to both ways of deploying it, one from the command line (PS) and the other OD.

So the only thing that should be different between the two types of deployment is that tfvars file that’s supplied via -var-file="?.tfvars"

My two files are terraform.tfvars for my local machine in VS and octopus.tfvars for use in Octopus Deploy

My main issue here is with the variables I’ve set in my local main.tf file for the provider being AWS, so I have the following vars

#VARIABLES
variable "aws_access_key" {}
variable "aws_secret_key" {}
variable "aws_region" {}

# PROVIDERS
provider "aws" {
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
  region     = "${var.aws_region}"
}

If I keep these in my tf file I get the following errors:

Error: Unassigned variable 
August 13th 2019 17:06:49Error
The input variable "aws_access_key" has not been assigned a value. This is a 
August 13th 2019 17:06:49Error
bug in Terraform; please report it in a GitHub issue. 

Is there a way I can actually set these as project variables and pull the values from the AWS account that was setup under Infrastructure > Accounts? So I can set my project var equal to the location of where account.AccessKey & account.SecretKey are stored

Or another method might be to set a value within my octopus.tfvars file which is the octopus specific vars file in my package, so for example I might have something like:

# octopus.tfvars

aws_access_key = "#{account.AccessKey}"
aws_secret_key = "#{account.SecretKey}"
aws_region = "#{?.region}"

NOTE: I have already setup an Account for AWS under Infra, so this is NOT my problem. As this part is working, I just don’t want to REFACTOR my code to use one deployment process in PowerShell to use a different one in Octopus Deploy

Or I’m I missing something from the documentation? From what I read that the values in the tf & vars files override values set at the step level

As you understand I DON’T want to delete this values as it’ll fail terraform validation on my client

Hopefully I’ve explained this all clearly and you get what I’m trying to achieve, fingers crossed :slight_smile:

I can give you and example of a similar problem I was having earlier with using two types of values for my PEM file, as my client needed a path to a pem on my file system and in Octopus the same PEM value was set to a secret string in a project var. My way to solve this was to use the Ternary Operator condition ? value_if_true : value_if_false

My code looked like this:

#variable
variable "deploy_env" {
  default = "local"
}

#Resource
resource "aws_instance" "nginx" {
  ami           = "${var.aws_ami}"
  instance_type = "t2.nano"
  key_name      = "${var.key_name}"

  connection {
    host        = self.public_ip
    type        = "ssh"
    user        = "ec2-user"
    private_key = "${var.deploy_env == "octopus" ? "${var.aws_private_key}" : "${file(var.aws_private_key)}"}"
  }

octopus.tfvars:

deploy_env = "octopus"

And the above actually works so I can have local client side way to manage my private_key and an Octopus Deploy way, which I was pretty happy with

But I’m still trying to come up with a way to solve my accessKey & secretKey, any advice you have would be great thanks?

Cheers,
Martin

(Martin Woods) #2

When I used the code in my octopus.tfvars file

aws_access_key = "#{aws.AccessKey}"
aws_secret_key = "#{aws.SecretKey}"

I got the following error:

Terraform has been successfully initialized! 
August 13th 2019 23:13:16Info
"C:\Program Files\Octopus Deploy\Octopus\bin\terraform.exe" apply -no-color -auto-approve -var-file="terraform_tfvars/octopus.tfvars" 
August 13th 2019 23:13:24Error
Error: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid. 
August 13th 2019 23:13:24Error
	status code: 403, request id: 
August 13th 2019 23:13:24Error
  on 02_terraform.tf line 21, in provider "aws": 
August 13th 2019 23:13:24Error
  21: provider "aws" { 
August 13th 2019 23:13:24Error
Calamari.Integration.Processes.CommandLineException: The following command: "C:\Program Files\Octopus Deploy\Octopus\bin\terraform.exe" apply -no-color -auto-approve -var-file="terraform_tfvars/octopus.tfvars" 
August 13th 2019 23:13:24Error
With the working directory of: D:\Octopus\Work\20190813221301-48726-140\staging 
August 13th 2019 23:13:24Error
Failed with exit code: 1 
August 13th 2019 23:13:24Error
Error: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid. 
August 13th 2019 23:13:24Error
	status code: 403, request id:  
August 13th 2019 23:13:24Error
  on 02_terraform.tf line 21, in provider "aws": 
August 13th 2019 23:13:24Error
  21: provider "aws" { 
August 13th 2019 23:13:24Error
   at Calamari.Integration.Processes.CommandResult.VerifySuccess() 
August 13th 2019 23:13:24Error
   at Calamari.Terraform.TerraformCLIExecutor.ExecuteCommand(String[] arguments) 
August 13th 2019 23:13:24Error
   at Calamari.Terraform.ApplyTerraformConvention.Execute(RunningDeployment deployment, Dictionary`2 environmentVariables) 
August 13th 2019 23:13:24Error
   at Calamari.Terraform.TerraformConvention.<InstallAsync>d__4.MoveNext() 
August 13th 2019 23:13:24Error
--- End of stack trace from previous location where exception was thrown --- 
August 13th 2019 23:13:24Error
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 
August 13th 2019 23:13:24Error
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
August 13th 2019 23:13:24Error
   at Calamari.Terraform.TerraformConvention.Install(RunningDeployment deployment) 
August 13th 2019 23:13:24Error
   at Calamari.Deployment.ConventionProcessor.RunInstallConventions() 
August 13th 2019 23:13:24Error
   at Calamari.Deployment.ConventionProcessor.RunConventions() 
August 13th 2019 23:13:24Error
Running rollback conventions... 
August 13th 2019 23:13:24Error
The following command: "C:\Program Files\Octopus Deploy\Octopus\bin\terraform.exe" apply -no-color -auto-approve -var-file="terraform_tfvars/octopus.tfvars" 
August 13th 2019 23:13:24Error
With the working directory of: D:\Octopus\Work\20190813221301-48726-140\staging 
August 13th 2019 23:13:24Error
Failed with exit code: 1 
August 13th 2019 23:13:24Error
Error: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid. 
August 13th 2019 23:13:24Error
	status code: 403, request id: 
August 13th 2019 23:13:24Error
  on 02_terraform.tf line 21, in provider "aws": 
August 13th 2019 23:13:24Error
  21: provider "aws" { 
August 13th 2019 23:13:24Error
Calamari.Integration.Processes.CommandLineException 
August 13th 2019 23:13:24Error
   at Calamari.Integration.Processes.CommandResult.VerifySuccess() 
August 13th 2019 23:13:24Error
   at Calamari.Terraform.TerraformCLIExecutor.ExecuteCommand(String[] arguments) 
August 13th 2019 23:13:24Error
   at Calamari.Terraform.ApplyTerraformConvention.Execute(RunningDeployment deployment, Dictionary`2 environmentVariables) 
August 13th 2019 23:13:24Error
   at Calamari.Terraform.TerraformConvention.<InstallAsync>d__4.MoveNext() 
August 13th 2019 23:13:24Error
--- End of stack trace from previous location where exception was thrown --- 
August 13th 2019 23:13:24Error
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 
August 13th 2019 23:13:24Error
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
August 13th 2019 23:13:24Error
   at Calamari.Terraform.TerraformConvention.Install(RunningDeployment deployment) 
August 13th 2019 23:13:24Error
   at Calamari.Deployment.ConventionProcessor.RunInstallConventions() 
August 13th 2019 23:13:24Error
   at Calamari.Deployment.ConventionProcessor.RunConventions() 
August 13th 2019 23:13:24Error
   at Calamari.Terraform.TerraformCommand.Execute(String[] commandLineArguments) 
August 13th 2019 23:13:24Error
   at Calamari.Program.Execute(String[] args) 
August 13th 2019 23:13:25Fatal
The remote script failed with exit code 100 
August 13th 2019 23:13:25Fatal
The action Terraform Apply New AWS Test Instance on the Octopus Server failed
(Martin Woods) #3

You can ignore the above error, I had a typo in my octopus.tfvars file and my step is now working

So this ticket can be resolved, sorry for the hassle! :slight_smile:

Thanks

(Michael Noonan) #5

Hi @martin.woods,

I’m glad you figured it out while we were waking up here in Australia. Are you happy with the solution and can have a single file working in both Visual Studio and Octopus? This is an important aspect to me as well.

Hope that helps!
Mike

1 Like
(Martin Woods) #6

Good Morning @Michael_Noonan,

Thanks for the reply, and yes the goal is basically to have a single file

I’ve been doing my homework on trying to use conditional code in HCL and it’s a wee bit cumbersome and apparently there are issues from using a lot of conditional logic in Terraform, this is a good blog on it

This is how I managed to address my PEM file/var issue mentioned as an example in the main description above

My only other question now is that one of the menu items within the the Terraform Apply Step is used for Region, how do I reference that value in my tfvars file without hard coding the region variable, I was guessing it was something like #{awsAccName.Region}

Thanks,
Martin

(Michael Noonan) #7

Hi @martin.woods,

My only other question now is that one of the menu items within the the Terraform Apply Step is used for Region, how do I reference that value in my tfvars file without hard coding the region variable, I was guessing it was something like #{awsAccName.Region}

Probably the best approach is to create a project variable called something like Aws.Region and set the value to eu-west-1.

Now in your Terraform Apply step you can bind the Region setting to #{Aws.Region} and you can use that same binding expression inside your HCL.

Hope that helps!
Mike

1 Like
(Martin Woods) #8

Hi @Michael_Noonan,

Thanks again for the reply, just as and FYI we located this data under

Project > Variables > Preview then check the box Show system variables and this expanded the var list to reveal the system vars, where you can see Octopus.Action.Aws.Region

Thanks,
Martin