Update Calamari on rancher k8 cluster error

(Erwinjohn Staana) #1

I have a kubernetes cluster that I am managing via Rancher, when I tried to do an update calamari task I am getting this error

"Calamari.exe : kubectl.exe : W1104 03:44:19.539441 3296 loader.go:223] Config not found:
At C:\Octopus\Work\20191104034417-2045-675\Bootstrap.ps1:27 char:1

  • & “C:\Octopus\OctopusServer\Tools\Calamari\7.1.9\Calamari.exe” run-s …
  • CategoryInfo : NotSpecified: (kubectl.exe : W…fig not found: :String) [], RemoteException
    …"

Do I need to manually download the kubeconfig file and set it to the server/worker?

(Kenneth Bates) #5

Hi @erwinjohn.staana,

Thanks for getting in touch! I’m sorry to hear you’re hitting this confusing behavior. We talked about it as a team and we’re currently unsure what could be causing this. Would you be willing to expand a bit on your configuration to give us a clearer picture on your scenario? Could you also send us a copy of your task log for further context?

I look forward to hearing back and getting to the bottom of this one!

Best regards,

Kenny

(Erwinjohn Staana) #6

Hi Kenneth,

Below is the raw task log and my Kubernetes target configuration:

Task ID: ServerTasks-2119

Related IDs: Machines-41, Spaces-1

Task status: Failed

Task queued: Monday, 04 November 2019 11:16:32 AM +00:00

Task started: Monday, 04 November 2019 11:16:32 AM +00:00

Task completed: Monday, 04 November 2019 11:16:38 AM +00:00

Task duration: 6 seconds

Server version: 2019.8.6+Branch.tags-2019.8.6.Sha.f755ea9a7328cae14d6f281cc344f6934864974b

Server node: octopus-server

== Failed: Update Calamari on plasma-aks ==

11:16:32 Info | 1 machine will have Calamari updated to the latest version.

11:16:38 Verbose | Checking for Tentacles to update

11:16:38 Fatal | At least one machines was not successfully updated with the latest version of Calamari.

== Failed: plasma-aks ==

11:16:32 Verbose | Updating Calamari

11:16:32 Verbose | Performing health check on machine

11:16:32 Verbose | Using Calamari 7.1.9

11:16:32 Verbose | Running this script in the Octopus Server security context (octopus-server\sldcadmin)

11:16:32 Verbose | Starting C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory ‘C:\Octopus\Work\20191104111632-2119-744’
using ‘OEM United States’ encoding running as ‘octopus-server\sldcadmin’ with the same environment variables as the launching process

11:16:33 Verbose | Octopus Deploy: Calamari version 7.1.9

11:16:33 Verbose | Environment Information:

11:16:33 Verbose | OperatingSystem: Microsoft Windows NT 10.0.17763.0

11:16:33 Verbose | OsBitVersion: x64

11:16:33 Verbose | Is64BitProcess: True

11:16:33 Verbose | CurrentUser: octopus-server\sldcadmin

11:16:33 Verbose | MachineName: octopus-server

11:16:33 Verbose | ProcessorCount: 2

11:16:33 Verbose | CurrentDirectory: C:\Octopus\Work\20191104111632-2119-744

11:16:33 Verbose | TempDirectory: C:\Users\sldcadmin\AppData\Local\Temp\

11:16:33 Verbose | HostProcessName: Calamari

11:16:33 Verbose | Performing variable substitution on ‘C:\Octopus\Work\20191104111632-2119-744\Script.ps1’

11:16:33 Verbose | Executing ‘C:\Octopus\Work\20191104111632-2119-744\Script.ps1’

11:16:33 Verbose | Setting Proxy Environment Variables

11:16:34 Verbose | Name Value

11:16:34 Verbose | ---- -----

11:16:34 Verbose | PSVersion 5.1.17763.771

11:16:34 Verbose | PSEdition Desktop

11:16:34 Verbose | PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}

11:16:34 Verbose | BuildVersion 10.0.17763.771

11:16:34 Verbose | CLRVersion 4.0.30319.42000

11:16:34 Verbose | WSManStackVersion 3.0

11:16:34 Verbose | PSRemotingProtocolVersion 2.3

11:16:34 Verbose | SerializationVersion 1.1.0.1

11:16:34 Verbose | PowerShell Environment Information:

11:16:34 Verbose | OperatingSystem: Microsoft Windows NT 10.0.17763.0

11:16:34 Verbose | OsBitVersion: x64

11:16:34 Verbose | Is64BitProcess: True

11:16:34 Verbose | CurrentUser: octopus-server\sldcadmin

11:16:34 Verbose | MachineName: octopus-server

11:16:34 Verbose | ProcessorCount: 2

11:16:34 Verbose | CurrentDirectory: C:\Octopus\Work\20191104111632-2119-744

11:16:34 Verbose | CurrentLocation: C:\Octopus\Work\20191104111632-2119-744

11:16:34 Verbose | TempDirectory: C:\Users\sldcadmin\AppData\Local\Temp\

11:16:34 Verbose | HostProcessName: powershell

11:16:34 Verbose | TotalPhysicalMemory: 8388148 KB

11:16:34 Verbose | AvailablePhysicalMemory: 5158280 KB

11:16:34 Verbose | Temporary kubectl config set to C:\Octopus\Work\20191104111632-2119-744\kubectl-octo.yml

11:16:34 Verbose | No namespace provded. Using default

11:16:34 Error | Calamari.exe : kubectl.exe : W1104 11:16:34.540317 6920 loader.go:223] Config not found:

11:16:34 Error | At C:\Octopus\Work\20191104111632-2119-744\Bootstrap.ps1:27 char:1

11:16:34 Error | + & “C:\Octopus\OctopusServer\Tools\Calamari\7.1.9\Calamari.exe” run-s …

11:16:34 Error | + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

11:16:34 Error | + CategoryInfo : NotSpecified: (kubectl.exe : W…fig not found: :String) [], RemoteException

11:16:34 Error | + FullyQualifiedErrorId : NativeCommandError

11:16:34 Verbose | Process C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20191104111632-2119-744 exited with code
1

11:16:37 Verbose | Exit code: 1

11:16:37 Fatal | The remote script failed with exit code 1

11:16:37 Verbose | The remote script failed with exit code 1

Octopus.Server.Orchestration.Targets.Tasks.ActionHandlerFailedException: The remote script failed with exit code 1
at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.SuccessArbitrator.ThrowIfNotSuccessfull(IActionHandlerResult

result) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\SuccessArbitrator.cs:line 23

at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation

actionHandler, VariableCollection variables) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\AdHocActionDispatcher.cs:line 65

at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.VirtualTargetHealthController.CheckHealth(Machine machine)

in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\HealthCheck\Controllers\VirtualTargetHealthController.cs:line 64

at Octopus.Server.Orchestration.ServerTasks.HealthCheck.HealthCheckService.PerformHealthCheck(Machine machine, ExceptionHandling

exceptionHandling, Action`2 customAction) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\HealthCheck\HealthCheckService.cs:line 78

Octopus.Server version 2019.8.6 (2019.8.6+Branch.tags-2019.8.6.Sha.f755ea9a7328cae14d6f281cc344f6934864974b)

11:16:37 Verbose | Recording health check results

== Failed: Summary ==

11:16:38 Info | Unhealthy:

11:16:38 Info | - plasma-aks at Octopus.Core.Model.Machines.Endpoints.KubernetesEndpoint,
error: The remote script failed with exit code 1

11:16:38 Fatal | One or more machines could not have Calamari updated. Please see the output Log for details.

Please note that I also have an AKS cluster imported and managed via Rancher when I use the Rancher provided kubeconfig, I also encounter the same error
but when using Azure Service Principal as user, I was able to do Upgrade Calamari and deploy to the cluster.

I also did some fiddling in the server/node where the Upgrade Calamari is being ran to check if the config really exists. At first I had a hard time seeing
since after the task the working folder is being deleted so I uncheck the delete file and delete folder permission of the user so that the work folder will not be deleted. When I checked the folder only kubectl-octo.yml.lock is present there is no kubectl-octo.yml.

The Authentication “plasma-aks” is configured as token and I just copy-paste the token as shown from my rancher kube config. Below is the Rancher config
with the user and token info stripped out.

apiVersion: v1

kind: Config

clusters:

  • name: “plasma-aks”

cluster:

server: “https://rancher.sldcmnl.online/k8s/clusters/c-fs7fv

users:

  • name: “user-xxxxxx”

user:

token: “kubeconfig-user-xxxxxx:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”

contexts:

  • name: “plasma-aks”

context:

user: “user-xxxxxx”

cluster: “plasma-aks”

current-context: “plasma-aks”

Best regards,

Erwin John Sta Ana

Techinical Lead

SYNLAB Services S. L.

Global Delivery Centre

Verge de Guadalupe 18

08950 Esplugues de Llobregat

Barcelona / Spain

E-Mail:
erwin.john.sta.ana@synlab.com

Web:
www.synlab.com

Sede social: / Corporate Seat: Barcelona / España

Dirección comercial: / Business Address: Verge de Guadalupe 18, 08950 Esplugues de Llobregat / Barcelona / España

Sociedad inscrita en el Registro Mercantil con el n.º: / Commercial Register No.: B67433573
Tribunal del registro: / Register Court: Barcelona
CIF: / VAT Reg. No.: B67433573
Directores ejecutivos: / Managing Directors: Don Albert Sumarroca Claverol.

Este mensaje, y sus posibles documentos adjuntos, son confidenciales, pueden contener información privilegiada y se dirigen exclusivamente a su destinatario.
Únicamente el destinatario indicado puede leer, copiar y hacer uso de este mensaje. Si ha recibido el presente correo electrónico por error, le pedimos que se ponga en contacto con nosotros a la mayor brevedad posible (bien contestando por correo electrónico
o llamándonos al +34 93 363 6000 y que elimine este mensaje. Se prohíbe cualquier uso o difusión no autorizados del contenido parcial o total de este mensaje. Puesto que no puede descartarse que el contenido original del correo electrónico haya sido distorsionado
durante el proceso de transferencia de los datos, SYNLAB Services S.L. no asume ninguna responsabilidad por el contenido del presente correo.

  • This message (including any attachments) is confidential and may be privileged and is intended for the recipient only. Only
    the intended recipient may read, copy and use this message. If you have received this e-mail in error please contact us immediately (by return e-mail or via phone on +34 93 363 6000) and delete this message. Any unauthorised use or dissemination of this message
    in whole or in part is strictly prohibited. Since it cannot be ruled out that the original content of the e-mail has been distorted during the data transfer process,*SYNLAB Services S.L.
    cannot be held responsible for the contents of the e-mail.
(Erwinjohn Staana) #7

Hi Kenneth,

Below is the raw task log and my Kubernetes target configuration:

Task ID: ServerTasks-2119

Related IDs: Machines-41, Spaces-1

Task status: Failed

Task queued: Monday, 04 November 2019 11:16:32 AM +00:00

Task started: Monday, 04 November 2019 11:16:32 AM +00:00

Task completed: Monday, 04 November 2019 11:16:38 AM +00:00

Task duration: 6 seconds

Server version: 2019.8.6+Branch.tags-2019.8.6.Sha.f755ea9a7328cae14d6f281cc344f6934864974b

Server node: octopus-server

== Failed: Update Calamari on plasma-aks ==

11:16:32 Info | 1 machine will have Calamari updated to the latest version.

11:16:38 Verbose | Checking for Tentacles to update

11:16:38 Fatal | At least one machines was not successfully updated with the latest version of Calamari.

== Failed: plasma-aks ==

11:16:32 Verbose | Updating Calamari

11:16:32 Verbose | Performing health check on machine

11:16:32 Verbose | Using Calamari 7.1.9

11:16:32 Verbose | Running this script in the Octopus Server security context (octopus-server\sldcadmin)

11:16:32 Verbose | Starting C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory ‘C:\Octopus\Work\20191104111632-2119-744’
using ‘OEM United States’ encoding running as ‘octopus-server\sldcadmin’ with the same environment variables as the launching process

11:16:33 Verbose | Octopus Deploy: Calamari version 7.1.9

11:16:33 Verbose | Environment Information:

11:16:33 Verbose | OperatingSystem: Microsoft Windows NT 10.0.17763.0

11:16:33 Verbose | OsBitVersion: x64

11:16:33 Verbose | Is64BitProcess: True

11:16:33 Verbose | CurrentUser: octopus-server\sldcadmin

11:16:33 Verbose | MachineName: octopus-server

11:16:33 Verbose | ProcessorCount: 2

11:16:33 Verbose | CurrentDirectory: C:\Octopus\Work\20191104111632-2119-744

11:16:33 Verbose | TempDirectory: C:\Users\sldcadmin\AppData\Local\Temp\

11:16:33 Verbose | HostProcessName: Calamari

11:16:33 Verbose | Performing variable substitution on ‘C:\Octopus\Work\20191104111632-2119-744\Script.ps1’

11:16:33 Verbose | Executing ‘C:\Octopus\Work\20191104111632-2119-744\Script.ps1’

11:16:33 Verbose | Setting Proxy Environment Variables

11:16:34 Verbose | Name Value

11:16:34 Verbose | ---- -----

11:16:34 Verbose | PSVersion 5.1.17763.771

11:16:34 Verbose | PSEdition Desktop

11:16:34 Verbose | PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}

11:16:34 Verbose | BuildVersion 10.0.17763.771

11:16:34 Verbose | CLRVersion 4.0.30319.42000

11:16:34 Verbose | WSManStackVersion 3.0

11:16:34 Verbose | PSRemotingProtocolVersion 2.3

11:16:34 Verbose | SerializationVersion 1.1.0.1

11:16:34 Verbose | PowerShell Environment Information:

11:16:34 Verbose | OperatingSystem: Microsoft Windows NT 10.0.17763.0

11:16:34 Verbose | OsBitVersion: x64

11:16:34 Verbose | Is64BitProcess: True

11:16:34 Verbose | CurrentUser: octopus-server\sldcadmin

11:16:34 Verbose | MachineName: octopus-server

11:16:34 Verbose | ProcessorCount: 2

11:16:34 Verbose | CurrentDirectory: C:\Octopus\Work\20191104111632-2119-744

11:16:34 Verbose | CurrentLocation: C:\Octopus\Work\20191104111632-2119-744

11:16:34 Verbose | TempDirectory: C:\Users\sldcadmin\AppData\Local\Temp\

11:16:34 Verbose | HostProcessName: powershell

11:16:34 Verbose | TotalPhysicalMemory: 8388148 KB

11:16:34 Verbose | AvailablePhysicalMemory: 5158280 KB

11:16:34 Verbose | Temporary kubectl config set to C:\Octopus\Work\20191104111632-2119-744\kubectl-octo.yml

11:16:34 Verbose | No namespace provded. Using default

11:16:34 Error | Calamari.exe : kubectl.exe : W1104 11:16:34.540317 6920 loader.go:223] Config not found:

11:16:34 Error | At C:\Octopus\Work\20191104111632-2119-744\Bootstrap.ps1:27 char:1

11:16:34 Error | + & “C:\Octopus\OctopusServer\Tools\Calamari\7.1.9\Calamari.exe” run-s …

11:16:34 Error | + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

11:16:34 Error | + CategoryInfo : NotSpecified: (kubectl.exe : W…fig not found: :String) [], RemoteException

11:16:34 Error | + FullyQualifiedErrorId : NativeCommandError

11:16:34 Verbose | Process C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20191104111632-2119-744 exited with code
1

11:16:37 Verbose | Exit code: 1

11:16:37 Fatal | The remote script failed with exit code 1

11:16:37 Verbose | The remote script failed with exit code 1

Octopus.Server.Orchestration.Targets.Tasks.ActionHandlerFailedException: The remote script failed with exit code 1
at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.SuccessArbitrator.ThrowIfNotSuccessfull(IActionHandlerResult

result) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\SuccessArbitrator.cs:line 23

at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation

actionHandler, VariableCollection variables) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\AdHocActionDispatcher.cs:line 65

at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.VirtualTargetHealthController.CheckHealth(Machine machine)

in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\HealthCheck\Controllers\VirtualTargetHealthController.cs:line 64

at Octopus.Server.Orchestration.ServerTasks.HealthCheck.HealthCheckService.PerformHealthCheck(Machine machine, ExceptionHandling

exceptionHandling, Action`2 customAction) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\HealthCheck\HealthCheckService.cs:line 78

Octopus.Server version 2019.8.6 (2019.8.6+Branch.tags-2019.8.6.Sha.f755ea9a7328cae14d6f281cc344f6934864974b)

11:16:37 Verbose | Recording health check results

== Failed: Summary ==

11:16:38 Info | Unhealthy:

11:16:38 Info | - plasma-aks at Octopus.Core.Model.Machines.Endpoints.KubernetesEndpoint,
error: The remote script failed with exit code 1

11:16:38 Fatal | One or more machines could not have Calamari updated. Please see the output Log for details.

Please note that I also have an AKS cluster imported and managed via Rancher when I use the Rancher provided kubeconfig, I also encounter the same error
but when using Azure Service Principal as user, I was able to do Upgrade Calamari and deploy to the cluster.

I also did some fiddling in the server/node where the Upgrade Calamari is being ran to check if the config really exists. At first I had a hard time seeing
since after the task the working folder is being deleted so I uncheck the delete file and delete folder permission of the user so that the work folder will not be deleted. When I checked the folder only kubectl-octo.yml.lock is present there is no kubectl-octo.yml.

cid:image001.png@01D593D2.727C9480

The Authentication “plasma-aks” is configured as token and I just copy-paste the token as shown from my rancher kube config

Best regards,

Erwin John Sta Ana

Techinical Lead

SYNLAB Services S. L.

Global Delivery Centre

Verge de Guadalupe 18

08950 Esplugues de Llobregat

Barcelona / Spain

E-Mail:
erwin.john.sta.ana@synlab.com

Web:
www.synlab.com

Sede social: / Corporate Seat: Barcelona / España

Dirección comercial: / Business Address: Verge de Guadalupe 18, 08950 Esplugues de Llobregat / Barcelona / España

Sociedad inscrita en el Registro Mercantil con el n.º: / Commercial Register No.: B67433573
Tribunal del registro: / Register Court: Barcelona
CIF: / VAT Reg. No.: B67433573
Directores ejecutivos: / Managing Directors: Don Albert Sumarroca Claverol.

Este mensaje, y sus posibles documentos adjuntos, son confidenciales, pueden contener información privilegiada y se dirigen exclusivamente a su destinatario.
Únicamente el destinatario indicado puede leer, copiar y hacer uso de este mensaje. Si ha recibido el presente correo electrónico por error, le pedimos que se ponga en contacto con nosotros a la mayor brevedad posible (bien contestando por correo electrónico
o llamándonos al +34 93 363 6000 y que elimine este mensaje. Se prohíbe cualquier uso o difusión no autorizados del contenido parcial o total de este mensaje. Puesto que no puede descartarse que el contenido original del correo electrónico haya sido distorsionado
durante el proceso de transferencia de los datos, SYNLAB Services S.L. no asume ninguna responsabilidad por el contenido del presente correo.

  • This message (including any attachments) is confidential and may be privileged and is intended for the recipient only. Only
    the intended recipient may read, copy and use this message. If you have received this e-mail in error please contact us immediately (by return e-mail or via phone on +34 93 363 6000) and delete this message. Any unauthorised use or dissemination of this message
    in whole or in part is strictly prohibited. Since it cannot be ruled out that the original content of the e-mail has been distorted during the data transfer process,*SYNLAB Services S.L.
    cannot be held responsible for the contents of the e-mail.
(Tom Peters) #8

Hi,

Thanks for sharing that information. Unfortunately, we’re still not quite sure what’s going on here. There are a few things you can do to help us further narrow the problem down:

  1. Are you using any anti-virus software on the machine that is running kubectl? If so, can you disable your anti-virus software and try the health check again?
  2. What version of kubectl is installed on the machine? You can discover by running the kubectl version command.
  3. The script that is failing is open source and available here. This line of code shows the last message that we see in your attached log before the error occurs. I suspect that one of these lines of code are failing. You could try to run these lines of code manually on the target machine to see what is happening - this might give us a bit more information about how it is failing and help us narrow down the problem.

Let me know how you go!

Regards,
Tom

(Erwinjohn Staana) #9

Hi Tom,

(Tom Peters) #10

This is a really confusing issue. I’m trying to see if there are any ways we can get more information to diagnose this issue.

In the meantime,

  1. Can you try to run the health check again but with Windows Defender disabled? It is possible that it is quarantining some critical files. You could also check its history to see if it has quarantined anything important.
  2. When you tried to manually run the script, did you run it as the same user that runs the Tentacle service? If not, could you try to run the script manually again, but run it as that same user?

Regards,
Tom

(Erwinjohn Staana) #11

Hi Tom,

Apologies for the late reply since I am currently on client site. As for the questions/tasks that you sent last time,