Please advise as to how to resolve this problem.
Thanks! Bob
Hello, we have a Linux worker and Octopus is flagging that worker with “Not running latest version of Calamari.” I have attempted the upgrade via the Octopus UI but it is still being flagged. I have upgraded the tentacle on the worker to the latest version (as best I can tell) and its still being flagged the same.
Hi Bob,
Thank you for reaching out to us with your query about Calamari.
It sounds like you’ve successfully updated the Tentacle software but not Calamari. Calamari should be updated automatically when you run a deployment to the target. Can you please confirm that you’ve ran a deployment to this target and, if it didn’t upgrade during the deployment, can you please provide the relevant task log? It’d also be useful if you could confirm what happens when you try to upgrade via the UI (for example, error messages).
Best Regards,
Charles
Hi Charles, thanks for getting back with me so quickly! The machine is a worker so we don’t deploy to it. But I created a runbook with a “deploy a package” step and deployed the package to the worker. I see in the log below the Calamari version, but nothing about updating it. When I check the connectivity of the worker, its still flagged with “Not running latest version of Calamari” same as the image I posted.
TASK LOG of PACKAGE DEPLOYMENT TO WORKER
Task ID: ServerTasks-3951192
Related IDs: RunbookRuns-3680, RunbookSnapshots-284, Projects-482, Runbooks-81, Spaces-1, Environments-1
Task status: Success
Task queued: Friday, 23 October 2020 1:14:16 PM +00:00
Task started: Friday, 23 October 2020 1:14:16 PM +00:00
Task completed: Friday, 23 October 2020 1:14:29 PM +00:00
Task duration: 13 seconds
Server version: 2020.4.2+Branch.tags-2020.4.2.Sha.98abeed4da0a5925c8ad8dea7f91bda088a5858d
Server node: EC2AMAZ-V9LR1F2
| == Success: Run Deploy to Worker to Upgrade Calamari on Stage ==
13:14:16 Info | Deploying only to these specifically included deployment targets: octopus-kube-bastion
13:14:16 Verbose | Guided failure is not enabled for this task
13:14:29 Info | The run completed successfully.
|
| == Success: Acquire packages ==
13:14:16 Info | Acquiring packages
13:14:16 Info | Making a list of packages to acquire
13:14:16 Verbose | Package r2deploy v20296.1 is required by action ‘Deploy a Package’
13:14:16 Verbose | No packages are required on the Octopus Server
13:14:16 Verbose | Delta compression is enabled for package transfers from the Octopus Server to deployment targets
13:14:16 Verbose | Machine octopus-kube-bastion still needs packages r2deploy v20296.1 for action (‘Deploy a Package’)
13:14:20 Info | All packages have been acquired
13:14:20 Verbose | Acquire Packages completed
|
| Success: octopus-kube-bastion
|
| Success: Upload package r2deploy v20296.1
13:14:16 Verbose | Checking package cache for package r2deploy v20296.1
13:14:16 Info | Package r2deploy v20296.1 was found in cache. No need to download from feed.
13:14:16 Verbose | Using file: D:\Octopus\Packages\Spaces-1\feeds-builtin\r2deploy\r2deploy.20296.1.zip
13:14:16 Verbose | Using Calamari.linux-x64 14.11.1
13:14:16 Verbose | Requesting upload…
13:14:16 Verbose | Establishing SSH connection…
13:14:16 Verbose | SSH connection established
13:14:16 Verbose | Beginning streaming transfer of command.sh to $HOME.octopus\OctopusServer\Work\20201023131416-3951192-23871
13:14:16 Verbose | Establishing SFTP connection…
13:14:17 Verbose | SSFTP connection established
13:14:17 Verbose | Stream transfer complete
13:14:17 Verbose | Requesting upload…
13:14:18 Verbose | Beginning streaming transfer of Variables.secret to $HOME.octopus\OctopusServer\Work\20201023131416-3951192-23871
13:14:18 Verbose | Stream transfer complete
13:14:18 Verbose | Requesting upload…
13:14:18 Verbose | Beginning streaming transfer of Variables.Bash.secret to $HOME.octopus\OctopusServer\Work\20201023131416-3951192-23871
13:14:19 Verbose | Stream transfer complete
13:14:20 Verbose | Calamari Version: 14.11.1
13:14:20 Verbose | Environment Information:
13:14:20 Verbose | OperatingSystem: Unix 4.4.0.1117
13:14:20 Verbose | OsBitVersion: x64
13:14:20 Verbose | Is64BitProcess: True
13:14:20 Verbose | Running on Mono: False
13:14:20 Verbose | CurrentUser: octopus
13:14:20 Verbose | MachineName: ip-10-0-194-243
13:14:20 Verbose | ProcessorCount: 1
13:14:20 Verbose | CurrentDirectory: /home/octopus/.octopus/OctopusServer/Work/20201023131416-3951192-23871
13:14:20 Verbose | TempDirectory: /tmp/
13:14:20 Verbose | HostProcess: Calamari (7156)
13:14:20 Verbose | Unable to get the system proxy settings due to not running under .Net Framework. Calamari will not use any proxy settings.
13:14:20 Verbose | Package r2deploy 20296.1 hash 9548361bea0976c4959429c2a717f5934f948c72 has already been uploaded
13:14:20 Verbose | SFTP connection disposed.
13:14:20 Verbose | SSH connection disposed.
13:14:20 Verbose | Exit code: 0
13:14:20 Info | Package r2deploy version 20296.1 found in cache. No need to upload this 1.166 KB package. Using /home/octopus/.octopus/OctopusServer/Files/r2deploy@S20296.1@C1154A355183B8489249DF4A6F628109.zip
|
| == Success: Step 1: Deploy a Package ==
13:14:25 Verbose | Deploy a Package completed
|
| == Success: octopus-kube-bastion ==
13:14:21 Verbose | Octopus Server version: 2020.4.2+Branch.tags-2020.4.2.Sha.98abeed4da0a5925c8ad8dea7f91bda088a5858d
13:14:21 Verbose | Environment Information:
| IsRunningInContainer: False
| OperatingSystem: Microsoft Windows 10.0.14393
| OsBitVersion: x64
| Is64BitProcess: True
| CurrentUser: NT AUTHORITY\SYSTEM
| MachineName: EC2AMAZ-V9LR1F2
| ProcessorCount: 16
| CurrentDirectory: C:\Windows\system32
| TempDirectory: C:\Windows\TEMP
| HostProcessName: Octopus.Server
| PID: 13856
13:14:21 Verbose | Executing Deploy a Package (type Deploy a Package) on octopus-kube-bastion
13:14:21 Verbose | Using Calamari.linux-x64 14.11.1
13:14:21 Verbose | Requesting upload…
13:14:21 Verbose | Establishing SSH connection…
13:14:21 Verbose | SSH connection established
13:14:21 Verbose | Beginning streaming transfer of command.sh to $HOME.octopus\OctopusServer\Work\20201023131421-3951192-23872
13:14:21 Verbose | Establishing SFTP connection…
13:14:21 Verbose | SSFTP connection established
13:14:21 Verbose | Stream transfer complete
13:14:22 Verbose | Requesting upload…
13:14:22 Verbose | Beginning streaming transfer of Variables.secret to $HOME.octopus\OctopusServer\Work\20201023131421-3951192-23872
13:14:22 Verbose | Stream transfer complete
13:14:23 Verbose | Requesting upload…
13:14:23 Verbose | Beginning streaming transfer of Variables.Bash.secret to $HOME.octopus\OctopusServer\Work\20201023131421-3951192-23872
13:14:23 Verbose | Stream transfer complete
13:14:24 Verbose | Calamari Version: 14.11.1
13:14:24 Verbose | Environment Information:
13:14:24 Verbose | OperatingSystem: Unix 4.4.0.1117
13:14:24 Verbose | OsBitVersion: x64
13:14:24 Verbose | Is64BitProcess: True
13:14:24 Verbose | Running on Mono: False
13:14:24 Verbose | CurrentUser: octopus
13:14:24 Verbose | MachineName: ip-10-0-194-243
13:14:24 Verbose | ProcessorCount: 1
13:14:24 Verbose | CurrentDirectory: /home/octopus/.octopus/OctopusServer/Work/20201023131421-3951192-23872
13:14:24 Verbose | TempDirectory: /tmp/
13:14:24 Verbose | HostProcess: Calamari (7277)
13:14:24 Verbose | Unable to get the system proxy settings due to not running under .Net Framework. Calamari will not use any proxy settings.
13:14:24 Info | Deploying package: /home/octopus/.octopus/OctopusServer/Files/r2deploy@S20296.1@C1154A355183B8489249DF4A6F628109.zip
13:14:24 Verbose | Extracting package to: /home/octopus/.octopus/Applications/OctopusServer/Stage/r2deploy/20296.1
13:14:24 Verbose | Extracted 4 files
13:14:24 Verbose | The package has been installed to: /home/octopus/.octopus/Applications/OctopusServer/Stage/r2deploy/20296.1
13:14:24 Verbose | If you would like the package to be installed to an alternative location, please use the ‘Custom installation directory’ feature
13:14:24 Verbose | Adding journal entry:
13:14:24 Verbose |
13:14:24 Verbose |
13:14:24 Verbose |
13:14:25 Verbose | SFTP connection disposed.
13:14:25 Verbose | SSH connection disposed.
13:14:25 Verbose | Updating manifest with output variables
13:14:25 Verbose | Updating manifest with action evaluated variables
13:14:25 Verbose | Planning retention policy
13:14:25 Verbose | Successfully finished Deploy a Package on octopus-kube-bastion
|
| == Success: Apply retention policy on Tentacles ==
13:14:25 Info | Apply retention policies…
13:14:29 Verbose | Apply Tentacle Retention Policy completed
|
| == Success: octopus-kube-bastion ==
|
| Success: Applying retention policy using 90 Days to set Environments-1/Projects-482/Step-Deploy a Package/Machines-171/
13:14:25 Verbose | Using Calamari.linux-x64 14.11.1
13:14:25 Verbose | Requesting upload…
13:14:25 Verbose | Establishing SSH connection…
13:14:25 Verbose | SSH connection established
13:14:25 Verbose | Beginning streaming transfer of command.sh to $HOME.octopus\OctopusServer\Work\20201023131425-3951192-23873
13:14:25 Verbose | Establishing SFTP connection…
13:14:26 Verbose | SSFTP connection established
13:14:26 Verbose | Stream transfer complete
13:14:26 Verbose | Requesting upload…
13:14:26 Verbose | Beginning streaming transfer of Variables.secret to $HOME.octopus\OctopusServer\Work\20201023131425-3951192-23873
13:14:27 Verbose | Stream transfer complete
13:14:27 Verbose | Requesting upload…
13:14:27 Verbose | Beginning streaming transfer of Variables.Bash.secret to $HOME.octopus\OctopusServer\Work\20201023131425-3951192-23873
13:14:27 Verbose | Stream transfer complete
13:14:28 Verbose | Calamari Version: 14.11.1
13:14:28 Verbose | Environment Information:
13:14:28 Verbose | OperatingSystem: Unix 4.4.0.1117
13:14:28 Verbose | OsBitVersion: x64
13:14:28 Verbose | Is64BitProcess: True
13:14:28 Verbose | Running on Mono: False
13:14:28 Verbose | CurrentUser: octopus
13:14:28 Verbose | MachineName: ip-10-0-194-243
13:14:28 Verbose | ProcessorCount: 1
13:14:28 Verbose | CurrentDirectory: /home/octopus/.octopus/OctopusServer/Work/20201023131425-3951192-23873
13:14:28 Verbose | TempDirectory: /tmp/
13:14:28 Verbose | HostProcess: Calamari (7399)
13:14:28 Verbose | Unable to get the system proxy settings due to not running under .Net Framework. Calamari will not use any proxy settings.
13:14:29 Info | Keeping deployments from the last 90 days
13:14:29 Verbose | Keeping /home/octopus/.octopus/Applications/OctopusServer/Stage/r2deploy/20296.1 and /home/octopus/.octopus/OctopusServer/Files/r2deploy@S20296.1@C1154A355183B8489249DF4A6F628109.zip as it was installed 0 days and 0 hours ago
13:14:29 Info | Did not find any deployments to clean up
13:14:29 Verbose | SFTP connection disposed.
13:14:29 Verbose | SSH connection disposed.
13:14:29 Verbose | Exit code: 0
Charles, here’s the raw task log of the Octopus servers attempt to update Calamari.
Task ID: ServerTasks-3951234
Related IDs: Workers-21, Spaces-1
Task status: Success
Task queued: Friday, 23 October 2020 1:26:50 PM +00:00
Task started: Friday, 23 October 2020 1:26:50 PM +00:00
Task completed: Friday, 23 October 2020 1:26:57 PM +00:00
Task duration: 7 seconds
Server version: 2020.4.2+Branch.tags-2020.4.2.Sha.98abeed4da0a5925c8ad8dea7f91bda088a5858d
Server node: EC2AMAZ-V9LR1F2
| == Warning: Update Calamari on octopus-kube-bastion ==
13:26:50 Verbose | Cleaning up completed UpdateCalamari tasks. Deleting 0 finished tasks…
13:26:50 Info | 1 machine will have Calamari updated to the latest version.
13:26:57 Verbose | Checking for Tentacles to update
|
| == Warning: octopus-kube-bastion ==
13:26:50 Verbose | Updating Calamari
13:26:50 Verbose | Using Calamari.netfx 14.11.1
13:26:50 Verbose | Requesting upload…
13:26:50 Verbose | Establishing SSH connection…
13:26:51 Verbose | SSH connection established
13:26:51 Verbose | Beginning streaming transfer of command.sh to $HOME.octopus\OctopusServer\Work\20201023132650-3951234-23879
13:26:51 Verbose | Establishing SFTP connection…
13:26:51 Verbose | SSFTP connection established
13:26:51 Verbose | Stream transfer complete
13:26:52 Verbose | Requesting upload…
13:26:52 Verbose | Beginning streaming transfer of Variables.secret to $HOME.octopus\OctopusServer\Work\20201023132650-3951234-23879
13:26:52 Verbose | Stream transfer complete
13:26:53 Verbose | Requesting upload…
13:26:53 Verbose | Beginning streaming transfer of Variables.Bash.secret to $HOME.octopus\OctopusServer\Work\20201023132650-3951234-23879
13:26:53 Verbose | Stream transfer complete
13:26:54 Verbose | Calamari Version: 14.11.1
13:26:54 Verbose | SFTP connection disposed.
13:26:54 Verbose | SSH connection disposed.
13:26:54 Verbose | Exit code: 0
13:26:54 Verbose | Performing health check on machine
13:26:54 Verbose | Establishing SSH connection…
13:26:54 Verbose | SSH connection established
13:26:54 Verbose | SSH connection disposed.
13:26:54 Verbose | Exit code: 0
13:26:54 Verbose | Requesting upload…
13:26:54 Verbose | Establishing SSH connection…
13:26:54 Verbose | SSH connection established
13:26:55 Verbose | Beginning streaming transfer of command.sh to $HOME.octopus\OctopusServer\Work\20201023132654-3951234-23881
13:26:55 Verbose | Establishing SFTP connection…
13:26:55 Verbose | SSFTP connection established
13:26:55 Verbose | Stream transfer complete
13:26:56 Verbose | Requesting upload…
13:26:56 Verbose | Beginning streaming transfer of health-check.sh to $HOME.octopus\OctopusServer\Work\20201023132654-3951234-23881
13:26:56 Verbose | Stream transfer complete
13:26:57 Info | Host Name: ip-10-0-194-243
13:26:57 Info | Running As: \octopus
13:26:57 Info | Bash version: 4.3.48(1)-release
13:26:57 Info | OS: Linux 4.4.0-1117-aws x86_64
13:26:57 Info | Distro: Ubuntu 16.04.4 LTS (xenial)
13:26:57 Info | Filesystem Size Used Avail Use% Mounted on
13:26:57 Info | udev 488M 0 488M 0% /dev
13:26:57 Info | tmpfs 100M 4.3M 95M 5% /run
13:26:57 Info | /dev/xvda1 146G 13G 133G 9% /
13:26:57 Info | tmpfs 496M 0 496M 0% /dev/shm
13:26:57 Info | tmpfs 5.0M 0 5.0M 0% /run/lock
13:26:57 Info | tmpfs 496M 0 496M 0% /sys/fs/cgroup
13:26:57 Info | tmpfs 100M 0 100M 0% /run/user/1001
13:26:57 Info | Time Zone: UTC +0000
13:26:57 Info | Mono version: Mono JIT compiler version 5.12.0.226 (tarball Thu May 3 09:48:32 UTC 2018)
13:26:57 Warning | Not running latest version of Calamari
13:26:57 Verbose | SFTP connection disposed.
13:26:57 Verbose | SSH connection disposed.
13:26:57 Verbose | Exit code: 0
13:26:57 Verbose | Checking if Calamari should be installed or updated
13:26:57 Info | Calamari is missing or out of date and will be installed when the machine is the target of a deployment.
13:26:57 Verbose | Recording health check results
|
| == Success: Summary ==
13:26:57 Info | Online, has warnings:
13:26:57 Info | - octopus-kube-bastion at ssh://ip-10-0-194-243.ec2.internal:22/
Hi Bob,
Thank you for the response and for the logs.
It’s possible that the version of Calamari on the worker is ‘stuck’ and can’t be upgraded. You may be able to resolve this simply by moving the Calamari folder and allowing a deploy to run (moving is less destructive than deleting so we’d recommend it).
You might also want to try setting the update policy for Calamari to “Always” and then running the health check step again or a deploy again. This could potentially nudge the upgrade into starting.
Could you please try the above and let me know how it goes?
Best Regards,
Charles
Hi Charles,
This is a Linux (ubuntu) machine setup as a worker. A find for Calamari on the entire machine turns up the following directories:
root@ip-10-0-194-243:/home/octopus/.octopus/OctopusServer/Tools# find / -type d | grep “Calamari”
/home/octopus/.octopus/OctopusServer/Tools/Calamari.netfx /home/octopus/.octopus/OctopusServer/Tools/Calamari.netfx/14.11.1 /home/octopus/.octopus/OctopusServer/Tools/Calamari.netfx/14.11.1/ScriptCS /home/octopus/.octopus/OctopusServer/Tools/Calamari.netfx/14.11.1/FSharp /home/octopus/.octopus/OctopusServer/Tools/Calamari.linux-x64 /home/octopus/.octopus/OctopusServer/Tools/Calamari.linux-x64/14.11.1 /home/octopus/.octopus/OctopusServer/Tools/Calamari.linux-x64/14.11.1/ScriptCS /home/octopus/.octopus/OctopusServer/Tools/Calamari.linux-x64/14.11.1/FSharp
I moved /home/octopus/.octopus/OctopusServer/Tools/Calamari.netfx & /home/octopus/.octopus/OctopusServer/Tools/Calamari.linux-x64 and then redeployed. Octopus installed /home/octopus/.octopus/OctopusServer/Tools/Calamari.linux-x64. I then ran a connectivity health check and Octopus installed /home/octopus/.octopus/OctopusServer/Tools/Calamari.netfx
However, this is the same as what was there before and when I run the connectivity health check again, it says that the Calamari version is out of date!
Task ID: ServerTasks-3962212
Related IDs: Workers-21, Spaces-1
Task status: Success
Task queued: Monday, 26 October 2020 3:28:08 AM +00:00
Task started: Monday, 26 October 2020 3:28:08 AM +00:00
Task completed: Monday, 26 October 2020 3:28:12 AM +00:00
Task duration: 4 seconds
Server version: 2020.4.2+Branch.tags-2020.4.2.Sha.98abeed4da0a5925c8ad8dea7f91bda088a5858d
Server node: EC2AMAZ-V9LR1F2
| == Warning: Check octopus-kube-bastion health ==
03:28:08 Info | Starting health check for a limited set of machines.
03:28:08 Verbose | Health check was requested for 1 machine
03:28:08 Verbose | Found 1 matching machine
03:28:08 Info | Performing health check on 1 machine.
03:28:12 Verbose | Checking for Tentacles to update
|
| == Warning: Check worker: octopus-kube-bastion ==
03:28:08 Verbose | Performing health check on machine
03:28:08 Verbose | Establishing SSH connection…
03:28:09 Verbose | SSH connection established
03:28:09 Verbose | SSH connection disposed.
03:28:09 Verbose | Exit code: 0
03:28:09 Verbose | Requesting upload…
03:28:09 Verbose | Establishing SSH connection…
03:28:09 Verbose | SSH connection established
03:28:10 Verbose | Beginning streaming transfer of command.sh to $HOME.octopus\OctopusServer\Work\20201026032809-3962212-25765
03:28:10 Verbose | Establishing SFTP connection…
03:28:10 Verbose | SSFTP connection established
03:28:10 Verbose | Stream transfer complete
03:28:10 Verbose | Requesting upload…
03:28:11 Verbose | Beginning streaming transfer of health-check.sh to $HOME.octopus\OctopusServer\Work\20201026032809-3962212-25765
03:28:11 Verbose | Stream transfer complete
03:28:11 Info | Host Name: ip-10-0-194-243
03:28:11 Info | Running As: \octopus
03:28:11 Info | Bash version: 4.3.48(1)-release
03:28:11 Info | OS: Linux 4.4.0-1117-aws x86_64
03:28:11 Info | Distro: Ubuntu 16.04.4 LTS (xenial)
03:28:11 Info | Filesystem Size Used Avail Use% Mounted on
03:28:11 Info | udev 488M 0 488M 0% /dev
03:28:11 Info | tmpfs 100M 12M 88M 12% /run
03:28:11 Info | /dev/xvda1 146G 9.6G 136G 7% /
03:28:11 Info | tmpfs 496M 0 496M 0% /dev/shm
03:28:11 Info | tmpfs 5.0M 0 5.0M 0% /run/lock
03:28:11 Info | tmpfs 496M 0 496M 0% /sys/fs/cgroup
03:28:11 Info | tmpfs 100M 0 100M 0% /run/user/1000
03:28:11 Info | tmpfs 100M 0 100M 0% /run/user/1001
03:28:11 Info | Time Zone: UTC +0000
03:28:11 Info | Mono version: Mono JIT compiler version 5.12.0.226 (tarball Thu May 3 09:48:32 UTC 2018)
03:28:11 Warning | Not running latest version of Calamari
03:28:12 Verbose | SFTP connection disposed.
03:28:12 Verbose | SSH connection disposed.
03:28:12 Verbose | Exit code: 0
03:28:12 Verbose | Checking if Calamari should be installed or updated
03:28:12 Info | Calamari is missing or out of date and will be installed when the machine is the target of a deployment.
03:28:12 Verbose | Recording health check results
|
| == Success: Summary ==
03:28:12 Info | Online, has warnings:
03:28:12 Info | - octopus-kube-bastion at ssh://ip-10-0-194-243.ec2.internal:22/
|
Hi Bob,
Thank you for getting back to me with the detailed logs.
We have attempted to reproduce this by installing the same version of Octopus Deploy server that you are running but it seems to work correctly. The appropriate version of Calamari (14.11.1) is installed onto the Linux machine and the warning message stops being displayed.
In your case it looks like the correct version is getting installed but for some reason the warning is still being shown. This isn’t likely to interfere with your deployments but it’d still be good to find out what it is happening.
Could you please advise which target runtime you are using - self-contained Calamari or Calamari on Mono?
Best Regards,
Charles
Hi Charles,
I may have found the problem. I see now that this machine, octopus-kube-bastion is setup as a worker and a deployment target. On the worker the Target Runtime is: Calamari on Mono. As a deployment target, this same machine’s Target Runtime is: Self-Contained Calamari.
What should I do: is it valid for the same machine to be a worker and a deployment target at the same time? I think its always used as a worker but the project’s that use it as a deployment target didn’t know about workers maybe.
Please advise. For now, I changed the worker to Self-Contained Calamari (Default).
Thanks,
Bob
Hi Bob,
Thank you for the update - that’s a good catch.
I’m not aware of any reason you couldn’t use one machine for both, although if it turns out you don’t need the deployment target functionality then it might be best to change it to be a worker only for the sake of simplicity.
It’s worth noting that workers and deployment targets have separate health checks. Are you able to confirm if both the worker health check and the deployment target health check were giving the same upgrade message?
Best Regards,
Charles
Yeah, both health checks are passing now. I plan to switch over any projects using it as a DT to a worker. Thanks for your help and support! Bob
Hi Bob,
That’s great, thank you for the update!
Please get back in touch with us if you have any questions.
Best Regards,
Charles
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.