Deployments failing on a package acquire step for linux targets

Octopus Server
, , , ,
1 
================================================================= 
June 12th 2023 18:46:25Error
	External Debugger Dump: 
June 12th 2023 18:46:25Error
================================================================= 
June 12th 2023 18:46:25Error
mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb 
June 12th 2023 18:46:25Error
/home/javaman/.octopus/10-80-16-120-core-cvent-org/Work/20230612131612-2281719-305201/command.sh: line 60: 18684 Aborted                 setsid mono "$HOME/.octopus/10-80-16-120-core-cvent-org/Tools/Calamari.netfx/25.3.3/Calamari.exe" register-package -packageId "exchange-rates" -packageVersion "3.0.3-J11-LIBR-20220525" -packageVersionFormat "Semver" -packagePath "/home/javaman/.octopus/10-80-16-120-core-cvent-org/Files/exchange-rates@S3.0.3-J11-LIBR-20220525@80EBBABC2ECB5243BFC06C6E8DF8B9BF.zip" -taskId "ServerTasks-2281719" -sensitiveVariables "$HOME/.octopus/10-80-16-120-core-cvent-org/Work/20230612131612-2281719-305201/Variables.secret" -sensitiveVariables "$HOME/.octopus/10-80-16-120-core-cvent-org/Work/20230612131612-2281719-305201/Variables.Bash.secret" -sensitiveVariablesPassword=$variablePassword 
June 12th 2023 18:46:25Verbose
SFTP connection disposed. 
June 12th 2023 18:46:25Verbose
SSH connection disposed. 
June 12th 2023 18:46:25Verbose
Exit code: 134 
June 12th 2023 18:46:25Fatal
The remote script failed with exit code 134

Octo Server Version: 2023.1.9788
Deployment target is an Amazon Linux 2 machine running - Calamari on Mono setup

3

Good afternoon @Naman.Kumar,

Thank you for contacting Octopus Support and sorry to hear you are struggling with a deployment on a Linux target.

Is this the first time you have deployed to that target or have you deployed successfully before and now this is not working?

If its the first time you have deployed to that target we have some documentation about setting up Calamari with mono as you need to be specific in which version you are setting up for the calamari install when adding an SSH target.

  1. Specify whether Mono is installed on the SSH target or not to determine which version of Calamari will be installed.

We have a guide on the supported versions here so make sure you are running a supported mono version on your Linux box.

Version 5.14.0 or greater of Mono is supported. Version 3.10 or greater may work, however we strongly recommend using a supported version.

We are actually looking to depreciate mono so I would suggest you see if you can use the self contained version instead of mono to future proof your SSH targets.

Let me know if you have mono 5.14.0 or greater on that machine or if you can switch to using the self-contained calamari without mono as that would be your best bet for when we depreciate mono.

I look forward to hearing from you,
Kind Regards,
Clare

4

Yes we have created a fresh instance to deploy on and the deployment has been failing since. We have tried creating multiple nodes now just in case the initial one had a node specific issue.

We register all linux targets the same way via chef, Facing this issue for this certain deployment project only.
This deployment does fail on this specific reason at some or the other package acquire step, if you can create a secure link, I can share the verbose logs with you.

5

Hey @Naman.Kumar,

Thank you for getting back to me and letting me know this is only failing for this specific project, I have created you a secure link here you can use to send the logs over.

It may say in the logs, but can you also confirm what version of Mono you are using please, have you tried using our self hosted calamari to see if that fixes this issue?

Since you will have to use the self hosted calamari in the future as I mentioned we will no longer support Mono, it might be worth spinning up a Linux box with the self hosted calamari version on it to see if that fixes this issue for you.

Let me know when you have sent the logs over and we can take a look.

Kind Regards,
Clare

6

mono --version
Mono JIT compiler version 6.12.0.107 (tarball Wed Dec 9 21:42:51 UTC 2020)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
Interpreter: yes
LLVM: yes(610)
Suspend: hybrid
GC: sgen (concurrent by default)

7

Hi @Naman.Kumar,

Thanks for uploading that task log for our review - I’m stepping in for Clare as she’s gone offline for the day, but I’m happy to help!

In doing some further digging on this, it looks like our documentation might not be in alignment as far as supported versions of Mono go, but I’ve reached out internally to get some clarification on this.

From my initial read of things, it looks like Octopus Deploy supports up to version 5.14.0 of Mono (rather than greater than), which could explain why you are running into trouble here.

I will let you know what I find out from the team on this, and let me know if you come across anything new on your side in the meantime as well.

Regards,

Britton

9

Hi @Naman.Kumar,

I wanted to follow up on this, as I found a recent blog post that discusses the deprecation of Mono in Octopus Deploy in greater detail.

Within this documentation, our team highlights the process of moving away from Mono to .NET Core, and I’m wondering if this might be a way forward with your process as well - I’ll link to this specific section below, but let me know if you have any questions or concerns:

I also got that thread created with our engineering team on the specific requirements for using Mono in Octopus Deploy, and I will let you know when I hear back regarding this.

I hope this additional information helps.

Regards,

Britton

10

Hi @Naman.Kumar,

Thanks for your patience. I’ll jump in with an update for Clare and Britton as they’re both offline. :slight_smile:

Along with Britton’s comments on moving to .NET Core (self-contained Calamari) as a result of our support for Mono soon to be deprecated (his linked blog post offers a lot of comprehensive information on this upcoming change), another important note is ScriptCS doesn’t work on Mono version 5.16 and higher (noted here), so we generally recommend using Mono version 5.14.

Let us know if you have any further questions!

Best regards,

Kenny

11

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.