"client.authentication.k8s.io/v1alpha1" error: exec plugin: invalid apiVersion

tirelibirefe · 3 June 2022 11:57

v2022.1 Build 2744
The problem can be reproduced in the latest build

What happened?

Hello,
The problem is already addresses here and here.

The problem is that
The workarounds are Octopus Server / Windows based; I use Ubuntu worker for deployment.

Reproduction

In order to reproduce the problem, use Octopus latest release; use Ubuntu worker note and attempt to deploy helm chart to a Kubernetes (EKS).

Error and Stacktrace

Task ID:        ServerTasks-1157809
Related IDs:    Deployments-1103, Channels-282, Releases-922, Projects-281, Spaces-1, Environments-81
Task status:    Failed
Task queued:    Friday, 03 June 2022 2:39:26 PM +03:00
Task started:   Friday, 03 June 2022 2:39:26 PM +03:00
Task completed: Friday, 03 June 2022 2:39:31 PM +03:00
Task duration:  4 seconds
Server version: 2022.1.2744
Server node:    MY-OCTOPUS

                    | == Failed: Deploy myproject-campus-web1 release 1.0.1-test to AWS EKS TEST Kubernetes ==
14:39:27   Verbose  |   Guided failure is not enabled for this task
14:39:30   Verbose  |   Releasing workers that required packages.
14:39:30   Verbose  |   Released worker wn1 from lease WorkerTaskLeases-102
14:39:31   Fatal    |   The deployment failed because one or more steps failed. Please see the deployment log for details.
                    | 
                    |   == Success: Acquire packages ==
14:39:27   Info     |     Acquiring packages
14:39:27   Info     |     Making a list of packages to acquire
14:39:27   Verbose  |     Leased worker wn1 from pool Ubuntu (lease WorkerTaskLeases-102).
14:39:27   Verbose  |     Package myproject-campus-web v1.0.1-test is required by action 'Install or Upgrade a Helm Chart'
14:39:27   Verbose  |     No packages are required on the Octopus Server
14:39:27   Verbose  |     Delta compression is enabled for package transfers from the Octopus Server to deployment targets
14:39:27   Verbose  |     Machine wn1 still needs packages myproject-campus-web v1.0.1-test for action ('Install or Upgrade a Helm Chart')
14:39:27   Verbose  |     Checking package cache for package myproject-campus-web v1.0.1-test
14:39:27   Info     |     Package myproject-campus-web v1.0.1-test was found in cache. No need to download from feed.
14:39:27   Verbose  |     Using file: C:\Octopus\Packages\Spaces-1\feeds-builtin\myproject-campus-web\myproject-campus-web.1.0.1-test.tgz
14:39:28   Info     |     All packages have been acquired
14:39:28   Verbose  |     Acquire Packages completed
                    |   
                    |     Success: wn1
                    |     
                    |       Success: Upload package myproject-campus-web v1.0.1-test
14:39:27   Verbose  |         Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-1157809
14:39:27   Verbose  |         Executable directory is /bin
14:39:27   Verbose  |         Executable name or full path: /bin/bash
14:39:27   Verbose  |         No user context provided. Running as current user.
14:39:27   Verbose  |         Starting /bin/bash in working directory '/etc/octopus/wn1/Work/20220603113927-1157809-75' using 'Unicode (UTF-8)' encoding running as 'root' with the same environment variables as the launching process
14:39:27   Verbose  |         Process /bin/bash in /etc/octopus/wn1/Work/20220603113927-1157809-75 exited with code 0
14:39:27   Verbose  |         Using Calamari.linux-x64 21.2.0
14:39:27   Verbose  |         Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-1157809
14:39:27   Verbose  |         Executable directory is /bin
14:39:27   Verbose  |         Executable name or full path: /bin/bash
14:39:27   Verbose  |         No user context provided. Running as current user.
14:39:27   Verbose  |         Starting /bin/bash in working directory '/etc/octopus/wn1/Work/20220603113927-1157809-76' using 'Unicode (UTF-8)' encoding running as 'root' with the same environment variables as the launching process
14:39:27   Verbose  |         Calamari Version: 21.2.0
14:39:27   Verbose  |         Environment Information:
14:39:27   Verbose  |         OperatingSystem: Unix 5.4.0.113
14:39:27   Verbose  |         OsBitVersion: x64
14:39:27   Verbose  |         Is64BitProcess: True
14:39:27   Verbose  |         Running on Mono: False
14:39:27   Verbose  |         CurrentUser: root
14:39:27   Verbose  |         MachineName: octopus-wn1
14:39:27   Verbose  |         ProcessorCount: 2
14:39:27   Verbose  |         CurrentDirectory: /etc/octopus/wn1/Work/20220603113927-1157809-76
14:39:27   Verbose  |         TempDirectory: /tmp/
14:39:28   Verbose  |         HostProcess: Calamari (37252)
14:39:28   Verbose  |         Package retention is disabled.
14:39:28   Verbose  |         Package myproject-campus-web 1.0.1-test hash a212564bdc6b6e41d2fae14b3dcdd68fe6a8d2c8 has already been uploaded
14:39:28   Verbose  |         Process /bin/bash in /etc/octopus/wn1/Work/20220603113927-1157809-76 exited with code 0
14:39:28   Verbose  |         Exit code: 0
14:39:28   Info     |         Package myproject-campus-web version 1.0.1-test found in cache. No need to upload this 3.875 KB package. Using /etc/octopus/wn1/Files/myproject-campus-web@S1.0.1-test@000991BABE582D4BA17B31CF871C93EA.tgz
                    |       
                    |   == Failed: Step 1: Install or Upgrade a Helm Chart ==
14:39:30   Fatal    |     The step failed: Activity Install or Upgrade a Helm Chart on eks-dev1 failed with error 'The remote script failed with exit code 1'.
14:39:30   Verbose  |     Install or Upgrade a Helm Chart completed
                    |   
                    |     == Failed: Worker on behalf of eks-dev1 ==
14:39:28   Verbose  |       Octopus Server version: 2022.1.2744
14:39:28   Verbose  |       Environment Information:
                    |       IsRunningInContainer: False
                    |       OperatingSystem: Microsoft Windows 10.0.17763
                    |       OsBitVersion: x64
                    |       Is64BitProcess: True
                    |       CurrentUser: NT AUTHORITY\SYSTEM
                    |       MachineName: KU-OCTOPUS
                    |       ProcessorCount: 2
                    |       CurrentDirectory: C:\Windows\system32
                    |       TempDirectory: C:\Windows\TEMP\
                    |       HostProcessName: Octopus.Server
                    |       PID: 2896
14:39:28   Verbose  |       Executing Install or Upgrade a Helm Chart (type Upgrade a Helm Chart) on wn1
14:39:28   Verbose  |       Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-1157809
14:39:28   Verbose  |       Executable directory is /bin
14:39:28   Verbose  |       Executable name or full path: /bin/bash
14:39:28   Verbose  |       No user context provided. Running as current user.
14:39:28   Verbose  |       Starting /bin/bash in working directory '/etc/octopus/wn1/Work/20220603113928-1157809-77' using 'Unicode (UTF-8)' encoding running as 'root' with the same environment variables as the launching process
14:39:28   Verbose  |       Process /bin/bash in /etc/octopus/wn1/Work/20220603113928-1157809-77 exited with code 0
14:39:28   Verbose  |       Using Calamari.linux-x64 21.2.0
14:39:28   Verbose  |       Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-1157809
14:39:28   Verbose  |       Executable directory is /bin
14:39:28   Verbose  |       Executable name or full path: /bin/bash
14:39:28   Verbose  |       No user context provided. Running as current user.
14:39:28   Verbose  |       Starting /bin/bash in working directory '/etc/octopus/wn1/Work/20220603113928-1157809-78' using 'Unicode (UTF-8)' encoding running as 'root' with the same environment variables as the launching process
14:39:28   Verbose  |       Calamari Version: 21.2.0
14:39:28   Verbose  |       Environment Information:
14:39:28   Verbose  |       OperatingSystem: Unix 5.4.0.113
14:39:28   Verbose  |       OsBitVersion: x64
14:39:28   Verbose  |       Is64BitProcess: True
14:39:28   Verbose  |       Running on Mono: False
14:39:28   Verbose  |       CurrentUser: root
14:39:28   Verbose  |       MachineName: octopus-wn1
14:39:28   Verbose  |       ProcessorCount: 2
14:39:28   Verbose  |       CurrentDirectory: /etc/octopus/wn1/Work/20220603113928-1157809-78
14:39:28   Verbose  |       TempDirectory: /tmp/
14:39:29   Verbose  |       HostProcess: Calamari (37278)
14:39:29   Verbose  |       Extracting package to: /etc/octopus/wn1/Work/20220603113928-1157809-78/staging
14:39:29   Verbose  |       Extracted 12 files
14:39:30   Verbose  |       Executing '/etc/octopus/wn1/Work/20220603113928-1157809-78/staging/Octopus.Action.CustomScripts.PreDeploy.sh'
14:39:30   Verbose  |       Setting Proxy Environment Variables
14:39:30   Verbose  |       "chmod" u=rw,g=,o= "/etc/octopus/wn1/Work/20220603113928-1157809-78/staging/kubectl-octo.yml"
14:39:30   Verbose  |       Temporary kubectl config set to /etc/octopus/wn1/Work/20220603113928-1157809-78/staging/kubectl-octo.yml
14:39:30   Verbose  |       "/usr/bin/kubectl" version --client --short --request-timeout=1m
14:39:30   Verbose  |       Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
14:39:30   Verbose  |       Client Version: v1.24.1
14:39:30   Verbose  |       Kustomize Version: v4.5.4
14:39:30   Verbose  |       No namespace provided. Using default
14:39:30   Verbose  |       "/usr/bin/kubectl" config set-cluster octocluster --server=https://4860E06EC0663C0AC0496CA8B8236770.gr7.eu-central-1.eks.amazonaws.com --request-timeout=1m
14:39:30   Verbose  |       Cluster "octocluster" set.
14:39:30   Verbose  |       "/usr/bin/kubectl" config set-context octocontext --user=octouser --cluster=octocluster --namespace=default --request-timeout=1m
14:39:30   Verbose  |       Context "octocontext" created.
14:39:30   Verbose  |       "/usr/bin/kubectl" config use-context octocontext --request-timeout=1m
14:39:30   Verbose  |       Switched to context "octocontext".
14:39:30   Verbose  |       "/usr/bin/kubectl" config set clusters.octocluster.certificate-authority-data <data> --request-timeout=1m
14:39:30   Verbose  |       Property "clusters.octocluster.certificate-authority-data" set.
14:39:30   Info     |       Creating kubectl context to https://4860E06EC0663C0AC0496CA8B8236770.gr7.eu-central-1.eks.amazonaws.com (namespace default) using EKS cluster name eks-dev1
14:39:30   Verbose  |       "/usr/bin/kubectl" config set-credentials octouser --exec-command=aws-iam-authenticator --exec-api-version=client.authentication.k8s.io/v1alpha1 --exec-arg=token --exec-arg=-i --exec-arg=eks-dev1 --request-timeout=1m
14:39:30   Verbose  |       User "octouser" set.
14:39:30   Verbose  |       "/usr/bin/kubectl" get namespace default --request-timeout=1m
14:39:30   Error    |       error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"
14:39:30   Verbose  |       "/usr/bin/kubectl" create namespace default --request-timeout=1m
14:39:30   Error    |       error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"
14:39:30   Verbose  |       Calamari.Common.Commands.CommandException: PreDeploy script returned non-zero exit code: 1
14:39:30   Verbose  |       at Calamari.Common.Features.Behaviours.ConfiguredScriptBehaviour.Execute(RunningDeployment context) in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari.Common\Features\Behaviours\ConfiguredScriptBehaviour.cs:line 70
14:39:30   Verbose  |       at Calamari.Deployment.Conventions.ConfiguredScriptConvention.Install(RunningDeployment deployment) in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari.Shared\Deployment\Conventions\ConfiguredScriptConvention.cs:line 22
14:39:30   Verbose  |       at Calamari.Deployment.ConventionProcessor.RunInstallConventions() in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari.Shared\Deployment\ConventionProcessor.cs:line 75
14:39:30   Verbose  |       at Calamari.Deployment.ConventionProcessor.RunConventions() in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari.Shared\Deployment\ConventionProcessor.cs:line 32
14:39:30   Error    |       Running rollback conventions...
14:39:30   Verbose  |       Adding journal entry:
14:39:30   Verbose  |       <Deployment Id="13e50219-af93-42e1-aa25-4747f757fc36" EnvironmentId="Environments-81" TenantId="" ProjectId="Projects-281" InstalledOn="2022-06-03 11:39:30" RetentionPolicySet="Environments-81/Projects-281/Step-Install or Upgrade a Helm Chart/Machines-161/&lt;default&gt;" ExtractedTo="/etc/octopus/wn1/Work/20220603113928-1157809-78/staging" CustomInstallationDirectory="" WasSuccessful="False">
14:39:30   Verbose  |       <Package PackageId="myproject-campus-web" PackageVersion="1.0.1-test" DeployedFrom="/etc/octopus/wn1/Files/myproject-campus-web@S1.0.1-test@000991BABE582D4BA17B31CF871C93EA.tgz" />
14:39:30   Verbose  |       </Deployment>
14:39:30   Error    |       PreDeploy script returned non-zero exit code: 1
14:39:30   Verbose  |       Process /bin/bash in /etc/octopus/wn1/Work/20220603113928-1157809-78 exited with code 1
14:39:30   Verbose  |       Updating manifest with output variables
14:39:30   Verbose  |       Updating manifest with action evaluated variables
14:39:30   Fatal    |       The remote script failed with exit code 1
14:39:30   Fatal    |       The action Install or Upgrade a Helm Chart on eks-dev1 failed

More Information

it looks like aws-iam-authenticator sets the wrong api version to the context.

Workaround

I have no a workaround yet; please advise a workaround.

Actually I define the k8s context to worker node myself and not use aws-iam-authenticator but the workaround advised for Octopus is to install “aws-iam-authenticator --version 0.5.3”.

If I install aws-iam-authenticator --version 0.5.3 to Ubuntu worker node, does it fix the problem?Regarding the the documentation of AWS, I don’t need to install aws-iam-authenticator because I use aws-cli/2.7.5.

here

If you’re running the AWS CLI version 1.16.156 or later, then you don’t need to install the authenticator.

dan_close · 3 June 2022 14:41

Hi @tirelibirefe,

Thanks for reaching out to Octopus support! After reading through the discussions on this issue as well as the AWS documentation you linked it looks like you should have a few different workarounds available.

As mentioned in the ticket and Slack thread installing v0.5.3 of aws-iam-authenticator on your Linux worker should resolve the error. From all previous reports, this workaround has been successful. I’m not sure if you build your own workers or not, but if you were to use our worker-tools image it should have the correct version already installed.

Based on the AWS docs it looks like you could also go the route of using the aws eks get-token command as well. I don’t have personal experience with that command so I’m not positive what would be involved in making that change on your end.

Let me know if you get a chance to try either of these and if you run into any issues or have any other questions for us.

Thanks!
Dan

tirelibirefe · 3 June 2022 15:31

Hello @dan_close
Thanks for answer and attention.

I think “aws-iam-authenticator v0.5.3” is only for Windows; I couldn’t have located it anywhere for Ubuntu.

aws eks get-token --cluster-name eks-dev1
Actually I didn’t expect it would fix the problem. Because I tried aws eks update-kubeconfig --region eu-central-1 --name eks-dev1 at multiple different levels of deployment, it didn’t work. Because Octopus doesn’t use root user’s existing context, it creates its own context by substituting the variables with octouser and octocontext; I mean “at Ubuntu” ; I don’t know about Windows. Yes, I tried it and didn’t work.

I use Ubuntu virtual machine as worker-node, not a container. So I don’t use worker-tools.

Thanks & Regards

dan_close · 3 June 2022 17:49

Hi @tirelibirefe,

Here is a link to the Dockerfile we use to build our worker image which has the command to download the exact version you need for aws_iam_authenticator. We just grab the code right from the GitHub repo so you should be able to do the same on your Ubuntu worker.

I did also see one other user who reported they were able to fix this issue but updating to kubectl 1.21.9. That may be worth exploring depending on the version you’re running now.

Please let me know if you make any progress or have any other questions.

Thanks!
Dan

tirelibirefe · 3 June 2022 19:09

the downloaded version which is 0.5.3 doesn’t aware of its version:

$ aws-iam-authenticator version
{"Version":"unversioned"}

kubectl

$ kubectl version --client -o yaml
clientVersion:
  buildDate: "2022-01-19T17:51:12Z"
  compiler: gc
  gitCommit: b631974d68ac5045e076c86a5c66fba6f128dc72
  gitTreeState: clean
  gitVersion: v1.21.9
  goVersion: go1.16.12
  major: "1"
  minor: "21"
  platform: linux/amd64

…and here is the result:

@dan_close
I think Octopus tried something else using existing context here:

...
helm upgrade --install --namespace "mynamespace" --reset-values --install --set image.tag=1.0.0-test.6 "campus-web" "/etc/octopus/wn1/Work/20220603190211-1158300-138/staging/mynamespace-campus-web" 
June 3rd 2022 22:02:12Verbose
Setting Proxy Environment Variables 
June 3rd 2022 22:02:12Verbose
"chmod" u=rw,g=,o= "/etc/octopus/wn1/Work/20220603190211-1158300-138/staging/kubectl-octo.yml" 
June 3rd 2022 22:02:12Verbose
Temporary kubectl config set to /etc/octopus/wn1/Work/20220603190211-1158300-138/staging/kubectl-octo.yml 
June 3rd 2022 22:02:12Verbose
"/usr/bin/kubectl" version --client --short --request-timeout=1m 
June 3rd 2022 22:02:12Verbose
Client Version: v1.21.9 
June 3rd 2022 22:02:12Verbose
No namespace provided. Using default 
June 3rd 2022 22:02:12Verbose
"/usr/bin/kubectl" config set-cluster octocluster --server=https://4860E06EC0663C0AC0496CA8B8236770.gr7.eu-central-1.eks.amazonaws.com --request-timeout=1m 
June 3rd 2022 22:02:12Verbose
Cluster "octocluster" set. 
June 3rd 2022 22:02:12Verbose
"/usr/bin/kubectl" config set-context octocontext --user=octouser --cluster=octocluster --namespace=default --request-timeout=1m 
June 3rd 2022 22:02:12Verbose
Context "octocontext" created. 
June 3rd 2022 22:02:12Verbose
"/usr/bin/kubectl" config use-context octocontext --request-timeout=1m 
June 3rd 2022 22:02:12Verbose
Switched to context "octocontext". 
June 3rd 2022 22:02:13Verbose
"/usr/bin/kubectl" config set clusters.octocluster.certificate-authority-data <data> --request-timeout=1m 
June 3rd 2022 22:02:13Verbose
Property "clusters.octocluster.certificate-authority-data" set. 
June 3rd 2022 22:02:13Info
Creating kubectl context to https://4860E06EC1111111118236770.gr7.eu-central-1.eks.amazonaws.com (namespace default) using EKS cluster name eks-dev1 
June 3rd 2022 22:02:13Verbose
"/usr/bin/kubectl" config set-credentials octouser --exec-command=aws-iam-authenticator --exec-api-version=client.authentication.k8s.io/v1alpha1 --exec-arg=token --exec-arg=-i --exec-arg=eks-dev1 --request-timeout=1m 
June 3rd 2022 22:02:13Verbose
User "octouser" set. 
June 3rd 2022 22:02:13Verbose
"/usr/bin/kubectl" get namespace default --request-timeout=1m 
June 3rd 2022 22:02:13Verbose
NAME      STATUS   AGE 
June 3rd 2022 22:02:13Verbose
default   Active   97d 
June 3rd 2022 22:02:13Verbose
Setting Proxy Environment Variables 
June 3rd 2022 22:02:13Verbose
Bash Environment Information: 
June 3rd 2022 22:02:13Verbose
  OperatingSystem: Linux octopus-wn1 5.4.0-113-generic #127-Ubuntu SMP Wed May 18 14:30:56 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux 
June 3rd 2022 22:02:13Verbose
  CurrentUser: root 
June 3rd 2022 22:02:13Verbose
  HostName: octopus-wn1 
June 3rd 2022 22:02:13Verbose
  ProcessorCount: 2 
June 3rd 2022 22:02:13Verbose
  CurrentDirectory: /etc/octopus/wn1/Work/20220603190211-1158300-138/staging 
June 3rd 2022 22:02:13Verbose
  TempDirectory: /tmp 
June 3rd 2022 22:02:13Verbose
  HostProcessID: 51369 
June 3rd 2022 22:02:13Error
Error: Kubernetes cluster unreachable: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"
...

is there a way adding following command before Octopus run "/usr/bin/kubectl" get namespace default --request-timeout=1m command?

aws eks update-kubeconfig --region eu-central-1 --name eks-dev1

dan_close · 3 June 2022 21:12

Hi @tirelibirefe,

Unfortunately with the built-in Helm step you’re using there is no way to add those commands. You could try adding a pre-deployment or deployment script into your step but I do not believe that will have an impact on this issue.

Out of curiosity, are you able to perform a Health Check on your Kubernetes cluster as mentioned in the bug ticket? I would like to try to get engineering input on this issue however they are based in Australia and not back in the office until next Tuesday. In the meantime please let me know if you have any other questions.

Thanks,
Dan

tirelibirefe · 4 June 2022 19:08

Hello @dan_close
You’re right, I added pre-deployment script and it didn’t fix the problem. Becuase as I mentioned above, Octopus creates its own context, doesn’t care what we change out of its context via scripts.

Sorry, I didn’t understand what you meant; “which ticket”, “who is Australia based”?

I found a workaround a few hours ago. I switched my deployment target from Linux worker node to Windows Octopus Server (I’ve organized there aws credentials, kubectl, awscli etc.) and Octopus Server (Windows) can deploy Helm chart to Kubernetes. Actually using Octopus server as a worker node is not something I prefer.

Let’s wait couple of days; maybe we get a number of updates.

Thank you very much again for your attention. Have a good weekend.

finnian.dempsey · 6 June 2022 02:49

Hi @tirelibirefe,

Just stepping in for Dan, I see you’ve hit that issue that I raised a while back regarding the aws-iam-authenticator!

I’ll check in with the devs about what happened regarding the move to use the AWS CLIv2 to authenticate instead of the aws-iam-authenticator library, but in the meantime the AWS docs outline how to install it for linux (under the linux tab) however if that doesn’t work then having docker installed on the worker will allow for a container to be used (which our worker-tools image has the required version installed):

Let me know how you get on or if you have any questions at all, I’ll post here once I have any further info from the devs.

Best Regards,

tirelibirefe · 6 June 2022 06:51

Hello @finnian.dempsey Thanks for your contribution.

Regarding to AWS documentation aws-iam-authenticator is no longer needed.

If you’re running the AWS CLI version 1.16.156 or later, then you don’t need to install the authenticator.

I am not sure aws-iam-authenticaticator is still needed. The current release of awscli is aws-cli/2.7.5, I mean 1.16.156 is so old release. It looks like running aws eks update-kubeconfig... would be enough.

Thx& Regards

adam.hollow · 6 June 2022 08:15

Hi @tirelibirefe,

Thanks for your response!

It could be the case that aws-iam-authenticator is no longer needed, however, it appears that Octopus is built to leverage that technology (which may be changed in the near future).

I wonder if you’re able to try following the documentation that Finnian linked in regards to installing the aws-iam-authenticator library, then creating a new release inside Octopus and attempting the deployment again?

It may be that the version of eks/k8s and the version of the aws-iam-authenticator isn’t compatible.
Installing via the instructions on Amazon’s documentation may be a solution whilst we look at modifying our usage internally.

I hope this helps! Please let us know if you have any questions or concerns.

Kind Regards,
Adam

tirelibirefe · 6 June 2022 11:22

Hello @adam.hollow
thanks for your advise.

Regarding to @dan_close ‘s advise above I followed the steps Octopus’ Dockerfile which is the same with AWS documentation; it doesn’t fix the problem. Because all these commands create the context for user root . Octopus substitues root with octouser and creates its own context octocontext and uses octocontext in order to connect kubernetes. The problem is in the octocontext.

The thing what I don’t understand is that:

Octopus uses the same logic which is used in Linux worker node but things work on Windows not on Linux woker node. Maybe aws-iam-authenticator package is different in Windows and it’s not incompatible with Kubernetes.

I don’t know. Maybe incompatibility is not at aws-iam-authenticator. Becuase when I install aws-iam-authenticator regarding to @dan_close 's advise, I would be able to access to Kubernetes as root user but Octopus created wrong context and couldn’t access to Kubernetes.

dan_close · 6 June 2022 14:37

Hi @tirelibirefe,

Would you mind sharing your deployment process JSON with us so that we can attempt to reproduce this issue in-house? If so you can upload it using the link from last week.

I look forward to hearing back and please let me know if you have any other questions for us.

Thanks,
Dan

tirelibirefe · 7 June 2022 03:00

Hi @dan_close

{
  "Id": "deploymentprocess-Projects-281",
  "SpaceId": "Spaces-1",
  "ProjectId": "Projects-281",
  "Version": 63,
  "Steps": [
    {
      "Id": "c8295724-1374-4b48-8864-d7f9d1955639",
      "Name": "Install or Upgrade a Helm Chart",
      "PackageRequirement": "LetOctopusDecide",
      "Properties": {
        "Octopus.Action.TargetRoles": "dev"
      },
      "Condition": "Success",
      "StartTrigger": "StartAfterPrevious",
      "Actions": [
        {
          "Id": "8cc52545-7782-4e54-91bc-89536e104c5a",
          "Name": "Install or Upgrade a Helm Chart",
          "ActionType": "Octopus.HelmChartUpgrade",
          "Notes": null,
          "IsDisabled": false,
          "CanBeUsedForProjectVersioning": true,
          "IsRequired": false,
          "WorkerPoolId": "WorkerPools-1",
          "Container": {
            "Image": null,
            "FeedId": null
          },
          "WorkerPoolVariable": null,
          "Environments": [],
          "ExcludedEnvironments": [],
          "Channels": [],
          "TenantTags": [],
          "Packages": [
            {
              "Id": "69e17563-96ed-45bb-9aec-6f07b7d6f1f1",
              "Name": "",
              "PackageId": "enterprise-commerce-web",
              "FeedId": "feeds-builtin",
              "AcquisitionLocation": "Server",
              "Properties": {
                "SelectionMode": "immediate"
              }
            },
            {
              "Id": "0c830d84-d6a2-4b2f-83e8-ebaabdc87992",
              "Name": "enterprisecommerceweb",
              "PackageId": "docker/enterprisecommerceweb",
              "FeedId": "Feeds-1061",
              "AcquisitionLocation": "NotAcquired",
              "Properties": {
                "Extract": "False",
                "SelectionMode": "immediate",
                "Purpose": "DockerImageReference"
              }
            }
          ],
          "Condition": "Success",
          "Properties": {
            "Octopus.Action.Helm.ResetValues": "True",
            "Octopus.Action.Helm.ClientVersion": "V3",
            "Octopus.Action.RunOnServer": "true",
            "Octopus.Action.Helm.Namespace": "enterprise",
            "Octopus.Action.Helm.AdditionalArgs": "--install --set image.tag=#{Octopus.Action.Package[enterprisecommerceweb].PackageVersion}",
            "Octopus.Action.Package.PackageId": "enterprise-commerce-web",
            "Octopus.Action.Package.FeedId": "feeds-builtin",
            "Octopus.Action.Package.DownloadOnTentacle": "False",
            "Octopus.Action.Helm.ReleaseName": "commerce-web"
          },
          "Links": {}
        }
      ]
    }
  ],
  "LastSnapshotId": null,
  "Links": {
    "Self": "/api/Spaces-1/projects/Projects-281/deploymentprocesses",
    "Project": "/api/Spaces-1/projects/Projects-281",
    "Template": "/api/Spaces-1/projects/Projects-281/deploymentprocesses/template{?channel,releaseId}",
    "Validation": "/api/Spaces-1/projects/Projects-281/deploymentprocesses/validate"
  }
}

If you run this on Windows, it works, if you run this on Linux worker node, it doesn’t…

finnian.dempsey · 7 June 2022 07:02

Hi @tirelibirefe,

Thanks for sending that through!

I agree that we should be using the latest method of authentication and that relying on this older tool isn’t ideal. Hopefully the discussion gains some traction to get this implemented but unfortunately it is hard coded in Calamari at the moment.

I’ll keep digging into this to make sure that Linux deployments using the aws-iam-authenticator are possible, and will keep you posted with any updates/suggestions to get it working until we make the switch over to use the aws cli instead.

Best Regards,

system · 4 August 2022 02:31

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.

"client.authentication.k8s.io/v1alpha1" error: exec plugin: invalid apiVersion

What happened?

Reproduction

Error and Stacktrace

More Information

Workaround

Hello @dan_close You’re right, I added pre-deployment script and it didn’t fix the problem. Becuase as I mentioned above, Octopus creates its own context, doesn’t care what we change out of its context via scripts.

Sorry, I didn’t understand what you meant; “which ticket”, “who is Australia based”?

Hello @dan_close
You’re right, I added pre-deployment script and it didn’t fix the problem. Becuase as I mentioned above, Octopus creates its own context, doesn’t care what we change out of its context via scripts.