Issue with EKS Ingress creation

[Conor_Boyd]

Hi,

Octopus Server version: 2022.1.2133
EKS version: 1.22

I am trying to use the Ingress section of the Deploy a Container step to have my EKS cluster successfully create and manage an AWS ALB & associated Target Group(s) on my behalf.

I have achieved this successfully in a number of other Octopus projects.

I am using a number of Octopus variables in my Ingress definition:

I did not have one of the variables (#{ALB.Tags}) correctly declared when I performed my first deployment.

My EKS Ingress got created okay, but the AWS Load Balancer Controller failed to parse the Ingress attributes successfully. So no ALB or Target Group got created.

When I described the Ingress, I got a message that says:

Type Reason Age From Message

---

Warning FailedBuildModel 2m55s (x349 over 3d18h) ingress Failed build model due to failed to parse stringMap annotation, alb.ingress.kubernetes.io/tags: #{ALB.Tags}

So I then made sure I create the Octopus variable, and performed a fresh deployment.

Now, when I describe the Ingress, I can see that the attribute seems to be successfully set, but I am still getting the same FailedBuildModel error and no ALB created.

Name: vw-ad-gph-ingr
Labels: Octopus.Action.Id=7b36b120-5fec-444f-96a5-68111c33f14e
Octopus.Deployment.Id=deployments-55943
Octopus.Deployment.Tenant.Id=untenanted
Octopus.Environment.Id=environments-61
Octopus.Kubernetes.DeploymentName=vw-admin-graph
Octopus.Kubernetes.SelectionStrategyVersion=SelectionStrategyVersion2
Octopus.Project.Id=projects-1801
Octopus.RunbookRun.Id=
Octopus.Step.Id=e28d98c2-d61c-4897-a8f5-0632f0083dea
Namespace: bussys-dev
Address:
Ingress Class:
Default backend:
Rules:
Host Path Backends

---

vw-admin-graph.dev.bussys.trimble.com
/* vw-ad-gph-svc:vw-ad-gph-port (192.168.59.191:8080)
Annotations: alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:623576528635:certificate/2c84584f-ccd5-4698-b732-c5f77e792b56
alb.ingress.kubernetes.io/group.name: geois-dev-bussys
alb.ingress.kubernetes.io/healthcheck-path: /actuator/health/liveness
alb.ingress.kubernetes.io/listen-ports: [{“HTTP”: 80}, {“HTTPS”: 443}]
alb.ingress.kubernetes.io/load-balancer-attributes:
access_logs.s3.enabled=true,access_logs.s3.bucket=geois-dev-logging,access_logs.s3.prefix=ALB,idle_timeout.timeout_seconds=125
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/security-groups: sg-0590be6a93e14929c,sg-087ca52513047cab9,sg-0c4d4a993606ba2b2,sg-0ffe752efbc0fa8c2
alb.ingress.kubernetes.io/shield-advanced-protection: true
alb.ingress.kubernetes.io/ssl-redirect: 443
alb.ingress.kubernetes.io/tags: env=dev
alb.ingress.kubernetes.io/wafv2-acl-arn:
arn:aws:wafv2:us-west-2:623576528635:regional/webacl/DefaultALBWebACL_us-west-2/22bb9c1f-5a3a-4a85-bbae-702bd67ad6bd
kubernetes.io/ingress.class: alb

The alb.ingress.kubernetes.io/tags attribute seems set appropriately now (with a value of env=dev above), but it feels like there’s some leftover EKS config somewhere that is still referring to the original non-replaced Octopus variable #{ALB.Tags}.

Even if I delete the Ingress manually and perform a fresh deployment, I still get the same error.

I’d really appreciate some help as I’m stumped.

Cheers,

Conor

[finnian.dempsey]

Hi @Conor_Boyd,

Thanks for reaching out!

I’ll dig into this today and see if I can reproduce it on my end, I really appreciate all the info you’ve provided!

Would you be willing to also share the Task logs for the original deployment where it set http://alb.ingress.kubernetes.io/tags:#{ALB.Tags} as well as the most recent one that used the correct value but still failed?

You should be able to upload any files securely via this link, feel free to let me know if there are any issues with it!

Best Regards,

[Conor_Boyd]

Thanks for following up.

I think I might have identified the problem.

It appears that a separate deployment in a separate namespace on the same cluster which also was attempting to create an Ingress still had an unreplaced Octopus token.

Once I deleted that broken Ingress, my latest deployment simply worked, and the Ingress (and underlying ALB, etc) was successfully created.

Bizarrely the Ingress from my latest deployment (and the log messages in the AWS Load Balancer Controller container) seemed confused as to which was the problematic ingress.

ie, when describing the “good” Ingress, the Events section listed the FailedBuildModel warning, even though this applied to a completely separate Ingress.

I’m surprised that K8s wouldn’t process my “good” Ingress specification when it was choking on the “bad” one, but once I removed the “bad” Ingress spec, the “good” one completed.

I’m not sure I can identify the Octo task logs for the earlier deployments sorry.

I’m reasonably comfortable that I’ve identified what the underlying issue was, even if I’m a little unhappy with K8s for lack of clarity.

LMK if you still want more info to investigate further, but I think now that the issue was with K8s, not Octopus. I just didn’t initially know how to investigate further.

Cheers,

Conor

[finnian.dempsey]

Hi @Conor_Boyd,

That’s great to hear you managed to get it working!

I’ll keep exploring this on our end to see if we can improve how we are handling this, from my understanding deployments to one namespace shouldn’t be impacting deployments to another so there could be an issue to investigate here.

I’ll update this thread if I find any bugs or have any updates!

Best Regards,