Terraform Apply Step | PEM Variable set to 'Unix(LF) UCS-2 LE BOM'

(Martin Woods) #1

Hi,

GOAL

Using Terraform I want to use a connection to run an remote-exec on a Amazon Linux EC2 instance in AWS

As part of the connection block we need to connect using an SSH key pair using a Private Key PEM to auth with the Public Key stored on AWS, which is a standard action

Here is a simple example:

    resource "aws_instance" "nginx" {
      ami           = "${var.aws_ami}"
      instance_type = "t2.nano"
      key_name      = "${var.key_name}"

      connection {
        type        = "ssh"
        user        = "ec2-user"
        private_key = "${var.aws_key_path}"
      }

      provisioner "remote-exec" {
        inline = [
          "sudo amazon-linux-extras install epel -y",
          "sudo yum update -y",
          "sudo amazon-linux-extras install nginx1.12 -y",
          "sudo systemctl enable nginx.service",
          "sudo systemctl start nginx.service",
          "sudo systemctl status nginx.service"
        ]
      }
    }

And my tfvars file looks like:

key_name = "#{awsPublicKey}"
aws_key_path = "#{martinTestPrivateKey}"

My variable martinTestPrivateKey is saved as type sensitive under my project

I copy and paste my Private Key PEM which is encoded as UTF-8 with Windows Line Endings CR LF

ISSUE

When I run my Terraform Apply step with this setup my script falls over with the error:
invalid value "octopus_vars.tfvars" for flag -var-file: Error parsing octopus_vars.tfvars: At 9:20: expected '/' for comment

We wrote a Powershell script to determine what happens to the PEM data after it’s stored in Octopus Deploy Project Variable:

if (! (Test-Path $temp_pem_folder)) {
	New-Item -Path $temp_pem_folder -ItemType "directory"
}
$file_name = "martinPrivateKey.pem"

$long_path = Join-Path $temp_pem_folder $file_name

$martinPrivateKey | Out-File $long_path
write-highlight $long_path

When we view this data in Notepad++ it tells us that the data of the file is saved as:
Unix(LF) UCS-2 LE BOM

When I test this with a less complex string in my var martinTestPrivateKey, so a one liner rather than multiple lines and I check my test file in Notepad++ the data is saved as Windows(CR LF) UCS-2 LE BOM which is still incorrect for Terraform to understand.

WORKAROUND
For me to workaround this problem I have to create a step before and a step after my Terraform Apply Step

The first step takes the contents of my variable and fixes it in a temp file so that the file is saved as UTF-8 with Windows File endings and NOT Unix file endings or UTF-8 BOM

Then I run my Apply which works and after that I run my third step which is to remove the file with the temp sensitive data

BUG/FUNCTIONALITY

Ok here is the thing I really don’t want to add unnecessary steps to my Project when in my mind they’re not needed. I hope and like to think that the data I add to my variable is not changed by Octopus Deploy to a encoding/format that breaks my Terraform Script, and also why does what I see as a Windows tech decide to change my file to add Unix style line endings

Can you let us know if this is by design for some reason that is not known to us yet? Otherwise I’d see this as a bug that needs to be fixed?

NOTE:
I do understand that the standard way to read data from a PEM file in Terraform is to use the following syntax using the var function where the var.aws_key_path is the path to your PEM

  connection {
    host        = self.public_ip
    user        = "ec2-user"
    private_key = "${file(var.aws_key_path)}"
  }

But the file functions job is to convert the file data to a string with UTF-8 encoding

Just in case someone tries to point that out :slight_smile:

1 Like
(Matt Richardson) #5

Hi Martin

Thanks for getting in touch, and double thanks for such a detailed post - it’s made my life much easier to get to the bottom of it.

While I think the encoding is a little odd, I’m not sure it’s the cause of this issue - I think it’s due to the the multi-line nature of of the pem file. I’ve found that if you use HereStrings in the tfvars file, it works.

Here’s what I’ve used and managed to get a successful deployment:

nginx.tf:

variable "aws_ami" {
  type = "string"
}
variable "key_name" {
  type = "string"
}
variable "aws_key_path" {
  type = "string"
}
provider "aws" {
  region = "ap-southeast-2"
}

resource "aws_instance" "nginx" {
  ami           = "${var.aws_ami}"
  instance_type = "t2.nano"
  key_name      = "${var.key_name}"
  vpc_security_group_ids = ["sg-12345678"]
  subnet_id = "subnet-12345678"
  associate_public_ip_address = true

  connection {
    type        = "ssh"
    user        = "ec2-user"
    private_key = "${var.aws_key_path}"
  }

  provisioner "remote-exec" {
    inline = [
      "sudo amazon-linux-extras install epel -y",
      "sudo yum update -y",
      "sudo amazon-linux-extras install nginx1.12 -y",
      "sudo systemctl enable nginx.service",
      "sudo systemctl start nginx.service",
      "sudo systemctl status nginx.service"
    ]
  }
}

nginx.tfvars:

aws_ami = "ami-0dc96254d5535925f"
key_name = "#{awsPublicKey}"
aws_key_path = <<-EOF
#{martinTestPrivateKey}
-EOF

The martinTestPrivateKey variable was set as sensitive, and was a multiline value:

-----BEGIN RSA PRIVATE KEY-----
20 or so lines of hex
-----END RSA PRIVATE KEY-----

(I’d copied it from a UTF-8 file with windows line endings).

Have a go with that, and let me know how you get along.

Happy deployments!

Regards,
Matt

PS - thanks - this was a fun thing to investigate :slight_smile:

(Martin Woods) #6

Hi @Matt.Richardson - thanks for the quick reply and I don’t like to log anything without a wee bit of detail. I’d prefer not to keep my audience guessing as raising a ticket is not a script for a suspense movie! :slight_smile:

And great job on your suggestion as it works :+1: so thanks for that

Just on the encoding, do you still think this is a bug, and also what is the history on the changes you suggested, as in what drew you to the conclusion that the code did not like the multiple file endings, is this something you’ve seen before with Terraform HCL(2)?

Cheers

(Martin Woods) #7

I posted a comment here but now I deleted as it’s no longer relevant

(Matt Richardson) #8

Hi Martin

Thanks for letting me know it all worked.

Regarding the encoding - as it’s not causing any issues, I’m inclined to consider it not worth fixing. I concur it’s not ideal - it should keep the same encoding, but as it’s not breaking anything, I dont think it’ll make the grade for getting onto the priority list to get fixed. If you do find the the encoding does break a scenario, please let us know - that’ll definitely increase the priority.

The thing that caught my eye on the first post that made me think multiline was the At 9:20: expected '/' part… This was interesting as the tfvars file you showed only had 2 lines in it. I’ve also done a reasonable amount with certs, so I knew the newlines could often be a pain.

Happy deployments!

Regards,
Matt

2 Likes
(Martin Woods) #9

Hi @Matt.Richardson,

Again thanks for the quick and informative reply

Just as an FYI for you and to share the knowledge if you didn’t know already

I used the special var Octopus.Action.Terraform.CustomTerraformExecutable to set my project to use the Terraform binary v0.12.6

So using version 12 syntax, I then reran my job as above and it fell over with the error:

Error: Unterminated template string
No closing marker was found for the string. 
August 6th 2019 14:54:07 Error
Calamari.Integration.Processes.CommandLineException: The following command: "C:\Program Files\Octopus Deploy\Octopus\bin\terraform.exe" apply -no-color -auto-approve -var-file="octopus_vars.tfvars" 
August 6th 2019 14:54:07 Error
With the working directory of: C:\Octopus\Work\20190806135350-47862-353\staging 
August 6th 2019 14:54:07 Error
Failed with exit code: 1 
August 6th 2019 14:54:07 Error
Error: Unterminated template string 
August 6th 2019 14:54:07 Error
  on octopus_vars.tfvars line 34:

I raised this on stackoverflow as per Terraform support guidelines in ref:

A guy called Martin Atkins who I think is a dev at Terraform got back to state

“The correct syntax for a “flush heredoc” does not include a dash on the final marker”

So I changed my code again after viewing the doc link he sent me in his reply to the following

aws_key_path = <<EOT
#{terraformPrivateKey}
EOT

And this ran and worked as expected using the v0.12 binary, so just in case you’re looking to move away from 11 syntax to 12 syntax as I know from looking at searches online they’ve had issues with heredoc since the change to HLC2 and this was a recent fix

Thanks again,
Martin

(Matt Richardson) #10

Hi Martin

Thanks for the info - in my explorations I had run across something similar to that. Good to have more public info on it though to help in google searches.

As a side note, I’ve just merged a change to log the terraform version that Octopus executes - that should help customers in future by making it more obvious and help narrow down issues due to v0.11/v0.12 differences.

Happy deployments!

Cheers,
Matt

1 Like