v2022.1 Build 2744
The problem can be reproduced in the latest build
What happened?
Hello,
The problem is already addresses here and here.
The problem is that
The workarounds are Octopus Server / Windows based; I use Ubuntu worker for deployment.
Reproduction
In order to reproduce the problem, use Octopus latest release; use Ubuntu worker note and attempt to deploy helm chart to a Kubernetes (EKS).
Error and Stacktrace
Task ID: ServerTasks-1157809
Related IDs: Deployments-1103, Channels-282, Releases-922, Projects-281, Spaces-1, Environments-81
Task status: Failed
Task queued: Friday, 03 June 2022 2:39:26 PM +03:00
Task started: Friday, 03 June 2022 2:39:26 PM +03:00
Task completed: Friday, 03 June 2022 2:39:31 PM +03:00
Task duration: 4 seconds
Server version: 2022.1.2744
Server node: MY-OCTOPUS
| == Failed: Deploy myproject-campus-web1 release 1.0.1-test to AWS EKS TEST Kubernetes ==
14:39:27 Verbose | Guided failure is not enabled for this task
14:39:30 Verbose | Releasing workers that required packages.
14:39:30 Verbose | Released worker wn1 from lease WorkerTaskLeases-102
14:39:31 Fatal | The deployment failed because one or more steps failed. Please see the deployment log for details.
|
| == Success: Acquire packages ==
14:39:27 Info | Acquiring packages
14:39:27 Info | Making a list of packages to acquire
14:39:27 Verbose | Leased worker wn1 from pool Ubuntu (lease WorkerTaskLeases-102).
14:39:27 Verbose | Package myproject-campus-web v1.0.1-test is required by action 'Install or Upgrade a Helm Chart'
14:39:27 Verbose | No packages are required on the Octopus Server
14:39:27 Verbose | Delta compression is enabled for package transfers from the Octopus Server to deployment targets
14:39:27 Verbose | Machine wn1 still needs packages myproject-campus-web v1.0.1-test for action ('Install or Upgrade a Helm Chart')
14:39:27 Verbose | Checking package cache for package myproject-campus-web v1.0.1-test
14:39:27 Info | Package myproject-campus-web v1.0.1-test was found in cache. No need to download from feed.
14:39:27 Verbose | Using file: C:\Octopus\Packages\Spaces-1\feeds-builtin\myproject-campus-web\myproject-campus-web.1.0.1-test.tgz
14:39:28 Info | All packages have been acquired
14:39:28 Verbose | Acquire Packages completed
|
| Success: wn1
|
| Success: Upload package myproject-campus-web v1.0.1-test
14:39:27 Verbose | Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-1157809
14:39:27 Verbose | Executable directory is /bin
14:39:27 Verbose | Executable name or full path: /bin/bash
14:39:27 Verbose | No user context provided. Running as current user.
14:39:27 Verbose | Starting /bin/bash in working directory '/etc/octopus/wn1/Work/20220603113927-1157809-75' using 'Unicode (UTF-8)' encoding running as 'root' with the same environment variables as the launching process
14:39:27 Verbose | Process /bin/bash in /etc/octopus/wn1/Work/20220603113927-1157809-75 exited with code 0
14:39:27 Verbose | Using Calamari.linux-x64 21.2.0
14:39:27 Verbose | Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-1157809
14:39:27 Verbose | Executable directory is /bin
14:39:27 Verbose | Executable name or full path: /bin/bash
14:39:27 Verbose | No user context provided. Running as current user.
14:39:27 Verbose | Starting /bin/bash in working directory '/etc/octopus/wn1/Work/20220603113927-1157809-76' using 'Unicode (UTF-8)' encoding running as 'root' with the same environment variables as the launching process
14:39:27 Verbose | Calamari Version: 21.2.0
14:39:27 Verbose | Environment Information:
14:39:27 Verbose | OperatingSystem: Unix 5.4.0.113
14:39:27 Verbose | OsBitVersion: x64
14:39:27 Verbose | Is64BitProcess: True
14:39:27 Verbose | Running on Mono: False
14:39:27 Verbose | CurrentUser: root
14:39:27 Verbose | MachineName: octopus-wn1
14:39:27 Verbose | ProcessorCount: 2
14:39:27 Verbose | CurrentDirectory: /etc/octopus/wn1/Work/20220603113927-1157809-76
14:39:27 Verbose | TempDirectory: /tmp/
14:39:28 Verbose | HostProcess: Calamari (37252)
14:39:28 Verbose | Package retention is disabled.
14:39:28 Verbose | Package myproject-campus-web 1.0.1-test hash a212564bdc6b6e41d2fae14b3dcdd68fe6a8d2c8 has already been uploaded
14:39:28 Verbose | Process /bin/bash in /etc/octopus/wn1/Work/20220603113927-1157809-76 exited with code 0
14:39:28 Verbose | Exit code: 0
14:39:28 Info | Package myproject-campus-web version 1.0.1-test found in cache. No need to upload this 3.875 KB package. Using /etc/octopus/wn1/Files/myproject-campus-web@S1.0.1-test@000991BABE582D4BA17B31CF871C93EA.tgz
|
| == Failed: Step 1: Install or Upgrade a Helm Chart ==
14:39:30 Fatal | The step failed: Activity Install or Upgrade a Helm Chart on eks-dev1 failed with error 'The remote script failed with exit code 1'.
14:39:30 Verbose | Install or Upgrade a Helm Chart completed
|
| == Failed: Worker on behalf of eks-dev1 ==
14:39:28 Verbose | Octopus Server version: 2022.1.2744
14:39:28 Verbose | Environment Information:
| IsRunningInContainer: False
| OperatingSystem: Microsoft Windows 10.0.17763
| OsBitVersion: x64
| Is64BitProcess: True
| CurrentUser: NT AUTHORITY\SYSTEM
| MachineName: KU-OCTOPUS
| ProcessorCount: 2
| CurrentDirectory: C:\Windows\system32
| TempDirectory: C:\Windows\TEMP\
| HostProcessName: Octopus.Server
| PID: 2896
14:39:28 Verbose | Executing Install or Upgrade a Helm Chart (type Upgrade a Helm Chart) on wn1
14:39:28 Verbose | Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-1157809
14:39:28 Verbose | Executable directory is /bin
14:39:28 Verbose | Executable name or full path: /bin/bash
14:39:28 Verbose | No user context provided. Running as current user.
14:39:28 Verbose | Starting /bin/bash in working directory '/etc/octopus/wn1/Work/20220603113928-1157809-77' using 'Unicode (UTF-8)' encoding running as 'root' with the same environment variables as the launching process
14:39:28 Verbose | Process /bin/bash in /etc/octopus/wn1/Work/20220603113928-1157809-77 exited with code 0
14:39:28 Verbose | Using Calamari.linux-x64 21.2.0
14:39:28 Verbose | Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-1157809
14:39:28 Verbose | Executable directory is /bin
14:39:28 Verbose | Executable name or full path: /bin/bash
14:39:28 Verbose | No user context provided. Running as current user.
14:39:28 Verbose | Starting /bin/bash in working directory '/etc/octopus/wn1/Work/20220603113928-1157809-78' using 'Unicode (UTF-8)' encoding running as 'root' with the same environment variables as the launching process
14:39:28 Verbose | Calamari Version: 21.2.0
14:39:28 Verbose | Environment Information:
14:39:28 Verbose | OperatingSystem: Unix 5.4.0.113
14:39:28 Verbose | OsBitVersion: x64
14:39:28 Verbose | Is64BitProcess: True
14:39:28 Verbose | Running on Mono: False
14:39:28 Verbose | CurrentUser: root
14:39:28 Verbose | MachineName: octopus-wn1
14:39:28 Verbose | ProcessorCount: 2
14:39:28 Verbose | CurrentDirectory: /etc/octopus/wn1/Work/20220603113928-1157809-78
14:39:28 Verbose | TempDirectory: /tmp/
14:39:29 Verbose | HostProcess: Calamari (37278)
14:39:29 Verbose | Extracting package to: /etc/octopus/wn1/Work/20220603113928-1157809-78/staging
14:39:29 Verbose | Extracted 12 files
14:39:30 Verbose | Executing '/etc/octopus/wn1/Work/20220603113928-1157809-78/staging/Octopus.Action.CustomScripts.PreDeploy.sh'
14:39:30 Verbose | Setting Proxy Environment Variables
14:39:30 Verbose | "chmod" u=rw,g=,o= "/etc/octopus/wn1/Work/20220603113928-1157809-78/staging/kubectl-octo.yml"
14:39:30 Verbose | Temporary kubectl config set to /etc/octopus/wn1/Work/20220603113928-1157809-78/staging/kubectl-octo.yml
14:39:30 Verbose | "/usr/bin/kubectl" version --client --short --request-timeout=1m
14:39:30 Verbose | Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
14:39:30 Verbose | Client Version: v1.24.1
14:39:30 Verbose | Kustomize Version: v4.5.4
14:39:30 Verbose | No namespace provided. Using default
14:39:30 Verbose | "/usr/bin/kubectl" config set-cluster octocluster --server=https://4860E06EC0663C0AC0496CA8B8236770.gr7.eu-central-1.eks.amazonaws.com --request-timeout=1m
14:39:30 Verbose | Cluster "octocluster" set.
14:39:30 Verbose | "/usr/bin/kubectl" config set-context octocontext --user=octouser --cluster=octocluster --namespace=default --request-timeout=1m
14:39:30 Verbose | Context "octocontext" created.
14:39:30 Verbose | "/usr/bin/kubectl" config use-context octocontext --request-timeout=1m
14:39:30 Verbose | Switched to context "octocontext".
14:39:30 Verbose | "/usr/bin/kubectl" config set clusters.octocluster.certificate-authority-data <data> --request-timeout=1m
14:39:30 Verbose | Property "clusters.octocluster.certificate-authority-data" set.
14:39:30 Info | Creating kubectl context to https://4860E06EC0663C0AC0496CA8B8236770.gr7.eu-central-1.eks.amazonaws.com (namespace default) using EKS cluster name eks-dev1
14:39:30 Verbose | "/usr/bin/kubectl" config set-credentials octouser --exec-command=aws-iam-authenticator --exec-api-version=client.authentication.k8s.io/v1alpha1 --exec-arg=token --exec-arg=-i --exec-arg=eks-dev1 --request-timeout=1m
14:39:30 Verbose | User "octouser" set.
14:39:30 Verbose | "/usr/bin/kubectl" get namespace default --request-timeout=1m
14:39:30 Error | error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"
14:39:30 Verbose | "/usr/bin/kubectl" create namespace default --request-timeout=1m
14:39:30 Error | error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"
14:39:30 Verbose | Calamari.Common.Commands.CommandException: PreDeploy script returned non-zero exit code: 1
14:39:30 Verbose | at Calamari.Common.Features.Behaviours.ConfiguredScriptBehaviour.Execute(RunningDeployment context) in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari.Common\Features\Behaviours\ConfiguredScriptBehaviour.cs:line 70
14:39:30 Verbose | at Calamari.Deployment.Conventions.ConfiguredScriptConvention.Install(RunningDeployment deployment) in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari.Shared\Deployment\Conventions\ConfiguredScriptConvention.cs:line 22
14:39:30 Verbose | at Calamari.Deployment.ConventionProcessor.RunInstallConventions() in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari.Shared\Deployment\ConventionProcessor.cs:line 75
14:39:30 Verbose | at Calamari.Deployment.ConventionProcessor.RunConventions() in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari.Shared\Deployment\ConventionProcessor.cs:line 32
14:39:30 Error | Running rollback conventions...
14:39:30 Verbose | Adding journal entry:
14:39:30 Verbose | <Deployment Id="13e50219-af93-42e1-aa25-4747f757fc36" EnvironmentId="Environments-81" TenantId="" ProjectId="Projects-281" InstalledOn="2022-06-03 11:39:30" RetentionPolicySet="Environments-81/Projects-281/Step-Install or Upgrade a Helm Chart/Machines-161/<default>" ExtractedTo="/etc/octopus/wn1/Work/20220603113928-1157809-78/staging" CustomInstallationDirectory="" WasSuccessful="False">
14:39:30 Verbose | <Package PackageId="myproject-campus-web" PackageVersion="1.0.1-test" DeployedFrom="/etc/octopus/wn1/Files/myproject-campus-web@S1.0.1-test@000991BABE582D4BA17B31CF871C93EA.tgz" />
14:39:30 Verbose | </Deployment>
14:39:30 Error | PreDeploy script returned non-zero exit code: 1
14:39:30 Verbose | Process /bin/bash in /etc/octopus/wn1/Work/20220603113928-1157809-78 exited with code 1
14:39:30 Verbose | Updating manifest with output variables
14:39:30 Verbose | Updating manifest with action evaluated variables
14:39:30 Fatal | The remote script failed with exit code 1
14:39:30 Fatal | The action Install or Upgrade a Helm Chart on eks-dev1 failed
More Information
it looks like aws-iam-authenticator sets the wrong api version to the context.
Workaround
I have no a workaround yet; please advise a workaround.
Actually I define the k8s context to worker node myself and not use aws-iam-authenticator but the workaround advised for Octopus is to install “aws-iam-authenticator --version 0.5.3”.
If I install aws-iam-authenticator --version 0.5.3 to Ubuntu worker node, does it fix the problem?Regarding the the documentation of AWS, I don’t need to install aws-iam-authenticator because I use aws-cli/2.7.5.
If you’re running the AWS CLI version 1.16.156 or later, then you don’t need to install the authenticator.