Issue with calamari upgrade

Octopus Server
1

Hi Team,

Good Day!!

I am new to octopus deploy ,I am trying to connect and deploy application from octopus deploy server to GCP Kubernetes cluster and as apart of this I had created the deployment target and when I was trying to run the upgrade calamary tab under connectivity for health check I am running into error mentioned below.

But the strange thing is that I can able to run the same error command in the command line successfully.

Error Info from task log
++++++++++++++
16:43:18 Verbose | Successfully authenticated with gcloud
16:43:19 Info | Creating kubectl context to GKE Cluster called gke-demo-cluster (namespace default) using a Google Cloud Account
16:43:19 Verbose | “C:\tools\gcloud\version\google-cloud-sdk\bin\gcloud.cmd” container clusters get-credentials gke-demo-cluster --zone=us-central1-c
16:43:21 Error | Fetching cluster endpoint and auth data.
16:43:21 Error | ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=403, message=Required “container.clusters.get” permission(s) for “projects/mykubernetesproject/zones/us-central1-c/clusters/gke-demo-cluster”.
16:43:21 Verbose | Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20230714144316-181-2 exited with code 1
16:43:21 Verbose | Exit code: 1
16:43:21 Fatal | The remote script failed with exit code 1
16:43:21 Verbose | The remote script failed with exit code 1
| Octopus.Server.Orchestration.Targets.Tasks.ActionHandlerFailedException: The remote script failed with exit code 1
| at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.SuccessArbitrator.ThrowIfNotSuccessful(IActionHandlerResult result) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/SuccessArbitrator.cs:line 22
| at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation actionHandler, ITaskLog taskLog, CancellationToken cancellationToken, VariableCollection variables) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 78
| at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.VirtualTargetHealthController.CheckHealth(Machine machine, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/HealthCheck/Controllers/VirtualTargetHealthController.cs:line 118
| at Octopus.Server.Orchestration.ServerTasks.HealthCheck.HealthCheckService.PerformHealthCheck(Machine machine, IHealthResultCollator healthResultCollator, CancellationToken cancellationToken, ExceptionHandling exceptionHandling, Func`3 customAction) in ./source/Octopus.Server/Orchestration/ServerTasks/HealthCheck/HealthCheckService.cs:line 86
| Octopus.Server version 2023.2.12998 (2023.2.12998)
16:43:21 Verbose | Recording health check results
|

Command line info
+++++
C:\Users\NareshUmapathy>“C:\tools\gcloud\version\google-cloud-sdk\bin\gcloud.cmd” container clusters get-credentials gke-demo-cluster --zone=us-central1-c
Fetching cluster endpoint and auth data.
kubeconfig entry generated for gke-demo-cluster.

Any help or suggestions can he helpful.

BR,
Naresh

3

Hi Naresh,

Thanks for reaching out, and sorry to hear you are having trouble fully connecting to your Kubernetes cluster in GCP, but I’m happy to help take a closer look at things.

As an initial step, this line from the log you uploaded stuck out to me:

16:43:21 Error | ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=403, message=Required “container.clusters.get” permission(s) for “projects/mykubernetesproject/zones/us-central1-c/clusters/gke-demo-cluste

From this error, I found the following StackOverflow post that has a few suggestions that may help in resolving this, and this answer in particular seemed helpful (but I recommend reviewing the complete thread, as there are other suggestions in there as well):

I hope this information helps, but if not, feel free to upload the complete task log for this failing health check for me to review at the following secure link:

Octopus Deploy Support Files | Secure Upload Link

Regards,

Britton

4

Hi Britton,

Thanks for quick turnaround.

I had followed the steps from stack overflow link mentioned and added the grants accordingly, but finally landed with the same error.
Uploaded the full task log in the support portal link mentioned.

Grants info
++++++
C:\Users\NareshUmapathy>gcloud projects add-iam-policy-binding mykubernetesproject-392809 --member=serviceAccount:462044468244-compute@developer.gserviceaccount.com --role=roles/container.developer
Updated IAM policy for project [mykubernetesproject-392809].
bindings:

C:\Users\NareshUmapathy>“C:\tools\gcloud\version\google-cloud-sdk\bin\gcloud.cmd” container clusters get-credentials gke-demo-cluster --zone=us-central1-c
Fetching cluster endpoint and auth data.
kubeconfig entry generated for gke-demo-cluster.

BR,
Naresh.

5

Hi Naresh,

Thank you for the update, and sorry to hear you are still having trouble, but I’d be happy to continue helping!

As a next step, in taking a deeper dive into that error message, I found the following StackExchange post that may help further here:

This post also links out to Google’s documentation on service accounts in GCP, which may also help shed some light on the issue here:

Hopefully adjusting the roles for the GCP service account you’re using in Octopus Deploy helps, but from the error, it does look like your process is still being blocked by a permissions constraint in GCP.

Regards,

Britton

6

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.