Calamari.exe: cannot execute binary file: Exec format error

Octopus server in docker on linux.
trying to run powershell script to deploy package to S3:

   Task ID:        ServerTasks-168
Related IDs:    Deployments-2, Channels-1, Releases-1, Projects-1, Spaces-2, Environments-21
Task status:    Failed
Task queued:    Sunday, 01 March 2020 4:47:52 PM +00:00
Task started:   Sunday, 01 March 2020 4:47:52 PM +00:00
Task completed: Sunday, 01 March 2020 4:47:53 PM +00:00
Task duration:  less than a second
Server version: 2019.13.7+Branch.tags-2019.13.7.Sha.53ca25f175bb58dd2c3f307258e26699353050f3
Server node:    Octopus

                    | == Failed: Deploy Brand Gateway release 2020.02.29 to Staging ==
16:47:52   Verbose  |   Guided failure is not enabled for this task
16:47:53   Fatal    |   The deployment failed because one or more steps failed. Please see the deployment log for details.
                    | 
                    |   == Success: Acquire packages ==
16:47:52   Info     |     Acquiring packages
16:47:52   Info     |     Making a list of packages to acquire
16:47:52   Verbose  |     No packages are required on any deployment targets
16:47:52   Info     |     All packages have been acquired
16:47:52   Verbose  |     Acquire Packages completed
                    |   
                    |     Success: Octopus Server
                    |     
                    |       Success: Download package staging v2020.02.29 from Octopus Server (built-in)
16:47:52   Verbose  |         Checking package cache for package staging v2020.02.29
16:47:52   Info     |         Package staging v2020.02.29 was found in cache. No need to download from feed.
16:47:52   Verbose  |         Using file: /repository/Spaces-2/feeds-builtin/staging/staging.2020.02.29.zip
16:47:52   Info     |         Successfully downloaded staging 2020.02.29 (4.098 MB)
16:47:52   Verbose  |         Stored package in /repository/Spaces-2/feeds-builtin/staging/staging.2020.02.29.zip
                    |       
                    |   == Failed: Step 1: Deploy to Elastic Beanstalk ==
16:47:52   Fatal    |     The step failed: Activity Deploy to Elastic Beanstalk on the Octopus Server failed with error 'The remote script failed with exit code 126'.
16:47:52   Verbose  |     Deploy to Elastic Beanstalk completed
                    |   
                    |     == Failed: Octopus Server ==
16:47:52   Verbose  |       Octopus Server version: 2019.13.7+Branch.tags-2019.13.7.Sha.53ca25f175bb58dd2c3f307258e26699353050f3
16:47:52   Verbose  |       Environment Information:
                    |       OperatingSystem: Linux 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020
                    |       OsBitVersion: x64
                    |       Is64BitProcess: True
                    |       CurrentUser: root
                    |       MachineName: deb625b8aa13
                    |       ProcessorCount: 12
                    |       CurrentDirectory: /Octopus
                    |       TempDirectory: /tmp/
                    |       HostProcessName: dotnet
                    |       PID: 1
16:47:52   Verbose  |       Executing Deploy to Elastic Beanstalk (type Run an AWS CLI Script) on Octopus Server
16:47:52   Verbose  |       Using Calamari.Cloud 9.0.5
16:47:52   Verbose  |       Using Octopus.Dependencies.AWSPS 3.3.390
16:47:52   Verbose  |       Using Octopus.Dependencies.AWSCLI32 1.16.178
16:47:52   Verbose  |       Using Calamari.Cloud 9.0.5
16:47:52   Verbose  |       Using Octopus.Dependencies.AWSPS 3.3.390
16:47:52   Verbose  |       Using Octopus.Dependencies.AWSCLI32 1.16.178
16:47:52   Verbose  |       Running this script in the Octopus Server security context (root)
16:47:52   Verbose  |       Starting /bin/bash in working directory '/home/octopus/.octopus/OctopusServer/Server/Work/20200301164752-168-6' using 'Unicode (UTF-8)' encoding running as 'root' with the same environment variables as the launching process
16:47:52   Error    |       /home/octopus/.octopus/OctopusServer/Server/Work/20200301164752-168-6/Bootstrap.sh: line 36: /home/octopus/.octopus/OctopusServer/Server/Tools/Calamari.Cloud/9.0.5/Calamari.exe: cannot execute binary file: Exec format error
16:47:52   Verbose  |       Process /bin/bash in /home/octopus/.octopus/OctopusServer/Server/Work/20200301164752-168-6 exited with code 126
16:47:52   Verbose  |       Updating manifest with output variables
16:47:52   Verbose  |       Updating manifest with action evaluated variables
16:47:52   Fatal    |       The remote script failed with exit code 126
16:47:52   Fatal    |       The action Deploy to Elastic Beanstalk on the Octopus Server failed

I have the same problem. My repro is just adding a kubernetes deployment target - you will see the health check fail over and over with the same error. Trying to use the deployment target also fails with the same calamari error.

I’m using 2019.13.7-linux

Task ID:        ServerTasks-33
Related IDs:    Machines-1, Spaces-1
Task status:    Failed
Task queued:    Tuesday, 10 March 2020 11:51:14 PM +00:00
Task started:   Tuesday, 10 March 2020 11:51:14 PM +00:00
Task completed: Tuesday, 10 March 2020 11:51:14 PM +00:00
Task duration:  less than a second
Server version: 2019.13.7+Branch.tags-2019.13.7.Sha.53ca25f175bb58dd2c3f307258e26699353050f3
Server node:    octopus-bb8f6d4f8-bx8nl

                    | == Failed: Check Drawboard US health ==
23:51:14   Info     |   Starting health check for a limited set of machines.
23:51:14   Verbose  |   Health check was requested for 1 machine
23:51:14   Verbose  |   Found 1 matching machine
23:51:14   Info     |   Performing health check on 1 machine.
23:51:14   Verbose  |   Checking for Tentacles to update
23:51:14   Fatal    |   The health check failed. One or more machines were not available.
                    | 
                    |   == Failed: Check deployment target: Drawboard US ==
23:51:14   Verbose  |     Performing health check on machine
23:51:14   Verbose  |     Using Calamari.Cloud 9.0.5
23:51:14   Verbose  |     Using Octopus.Dependencies.AzureCLI 2.0.50
23:51:14   Verbose  |     Using Octopus.Dependencies.AzureCmdlets 6.13.1
23:51:14   Verbose  |     Using Calamari.Cloud 9.0.5
23:51:14   Verbose  |     Using Octopus.Dependencies.AzureCLI 2.0.50
23:51:14   Verbose  |     Using Octopus.Dependencies.AzureCmdlets 6.13.1
23:51:14   Verbose  |     Running this script in the Octopus Server security context (root)
23:51:14   Verbose  |     Starting /bin/bash in working directory '/home/octopus/.octopus/OctopusServer/Server/Work/20200310235114-33-10' using 'Unicode (UTF-8)' encoding running as 'root' with the same environment variables as the launching process
23:51:14   Error    |     /home/octopus/.octopus/OctopusServer/Server/Work/20200310235114-33-10/Bootstrap.sh: line 36: /home/octopus/.octopus/OctopusServer/Server/Tools/Calamari.Cloud/9.0.5/Calamari.exe: cannot execute binary file: Exec format error
23:51:14   Verbose  |     Process /bin/bash in /home/octopus/.octopus/OctopusServer/Server/Work/20200310235114-33-10 exited with code 126
23:51:14   Verbose  |     Exit code: 126
23:51:14   Fatal    |     The remote script failed with exit code 126
23:51:14   Verbose  |     The remote script failed with exit code 126
                    |     Octopus.Server.Orchestration.Targets.Tasks.ActionHandlerFailedException: The remote script failed with exit code 126
                    |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.SuccessArbitrator.ThrowIfNotSuccessfull(IActionHandlerResult result) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\SuccessArbitrator.cs:line 22
                    |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation actionHandler, VariableCollection variables) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\AdHocActionDispatcher.cs:line 67
                    |     at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.VirtualTargetHealthController.CheckHealth(Machine machine) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\HealthCheck\Controllers\VirtualTargetHealthController.cs:line 64
                    |     at Octopus.Server.Orchestration.ServerTasks.HealthCheck.HealthCheckService.PerformHealthCheck(Machine machine, ExceptionHandling exceptionHandling, Action`2 customAction) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\HealthCheck\HealthCheckService.cs:line 76
                    |     Octopus.Server version 2019.13.7 (2019.13.7+Branch.tags-2019.13.7.Sha.53ca25f175bb58dd2c3f307258e26699353050f3)
23:51:14   Verbose  |     Recording health check results
                    |   
                    |   == Failed: Summary ==
23:51:14   Info     |     Unhealthy:
23:51:14   Info     |     - [Drawboard US](~/app#/Spaces-1/infrastructure/machines/Machines-1/settings) at Octopus.Core.Model.Machines.Endpoints.KubernetesEndpoint, error: The remote script failed with exit code 126
23:51:14   Fatal    |     One or more machines were not available. Please see the output Log for details.

Just hop on a pod and give it a go

kubectl.exe exec -it octopus-bb8f6d4f8-bx8nl -n us-dev /bin/bash
root@octopus-bb8f6d4f8-bx8nl:/Octopus# /home/octopus/.octopus/OctopusServer/Server/Tools/Calamari.Cloud/9.0.5/Calamari.exe
bash: /home/octopus/.octopus/OctopusServer/Server/Tools/Calamari.Cloud/9.0.5/Calamari.exe: cannot execute binary file: Exec format error

and…

root@octopus-bb8f6d4f8-bx8nl:/Octopus# dotnet /home/octopus/.octopus/OctopusServer/Server/Tools/Calamari.Cloud/9.0.5/Calamari.exe
A fatal error was encountered. The library 'libhostpolicy.so' required to execute the application was not found in '/home/octopus/.octopus/OctopusServer/Server/Tools/Calamari.Cloud/9.0.5/'.
Failed to run as a self-contained app. If this should be a framework-dependent app, add the /home/octopus/.octopus/OctopusServer/Server/Tools/Calamari.Cloud/9.0.5/Calamari.runtimeconfig.json file specifying the appropriate framework.

If you try and create a runtimeconfig for the exe (targeting netcoreapp2.2) you get this

root@octopus-74df4d9cbb-tjbfm:/Octopus# dotnet /home/octopus/.octopus/OctopusServer/Server/Tools/Calamari.Cloud/9.0.5/Calamari.exe

Unhandled Exception: System.NotSupportedException: The requested security protocol is not supported.
   at System.Net.ServicePointManager.ValidateSecurityProtocol(SecurityProtocolType value)
   at Calamari.Program.EnableAllSecurityProtocols() in C:\buildAgent\work\e0cefbed4ad11812\source\Calamari\Program.cs:line 142
   at Calamari.Program.Main(String[] args) in C:\buildAgent\work\e0cefbed4ad11812\source\Calamari\Program.cs:line 34
Writing minidump with heap to file /tmp/coredump.1126
Written 72847360 bytes (17785 pages) to core file
Aborted (core dumped)

I’m just returning to octopus after a few years. From what I have read Calamari seems to be a part of most if not all deployment operations now… So… have you tested this image at all? ever?

Octopus seems to unzip /Octopus/Calamari.Cloud.nupkg into the tools folder everytime and uses that. I can see in the same folder a linux nupkg variant is there but its not used.

I tried replacing the .Cloud with the linux version but the “Bootstrap.sh” that is running insists on running Calamari.exe

I managed to fool octopus into running the included linux package but it immediately complained it had no Calamari.Azure. I swapped out v9 with v11 which does seem to have the azure package.

It then complained the azure cli wasn’t installed so I installed it.
Then… whatever this is:

Successfully authenticated with the Azure CLI
05:00:39   Info     |     Invoking target script "/home/octopus/.octopus/OctopusServer/Server/Work/20200311050034-98-25/Octopus.KubectlBashContext.sh" with  parameters
05:00:39   Error    |     /home/octopus/.octopus/OctopusServer/Server/Work/20200311050034-98-25/Octopus.KubectlBashContext.sh: line 2: $'\r': command not found
05:00:39   Error    |     /home/octopus/.octopus/OctopusServer/Server/Work/20200311050034-98-25/Octopus.KubectlBashContext.sh: line 14: $'\r': command not found
05:00:39   Error    |     /home/octopus/.octopus/OctopusServer/Server/Work/20200311050034-98-25/Octopus.KubectlBashContext.sh: line 15: syntax error near unexpected token `$'{\r''
05:00:39   Error    |     /home/octopus/.octopus/OctopusServer/Server/Work/20200311050034-98-25/Octopus.KubectlBashContext.sh: line 15: `function check_app_exists {
05:00:39   Error    |     '
05:00:39   Error    |     /home/octopus/.octopus/OctopusServer/Server/Work/20200311050034-98-25/Octopus.AzureContext.sh: line 55: exit: too many arguments
05:00:39   Verbose  |     Process /bin/bash in /home/octopus/.octopus/OctopusServer/Server/Work/20200311050034-98-25 exited with code 1
05:00:39   Verbose  |     Exit code: 1

I am defeated.

Hi Alexander and Sam,

I’m sorry you’ve both hit this issue. We are still in the process of improving Linux support.

The problems you have been experiencing have been addressed in the upcoming 2020.1 release. We will be making it public in the next week or so. Please reach out if you’d like a pre-release build before then.

The GitHub issues are included for your reference:


Thank you
Frank

Hi Gentlemen,

Just wanted to follow up on this a little more. Our Octopus Server docker images where released to the public before they were fully baked and we were ready, which is 100% our fault.

As Frank mentioned 2020.1 is the first official publicly supported release of Octopus Server on Linux, and in turn on containers running on Linux.

Apologies for the confusion and wasted effort that this caused, however you can be assured from 2020.1 and forward this will be a fully tested and supported scenario for running Octopus.

Please let me know if you have any further questions.

Regards,
Alex

Thanks frank.lin and Alex.Rolley. Looking forward to it.