Facing issues with mono, when using linux workers for deployment

Octopus Server
1

The deployments are failing to acquire packages, this step runs on Linux worker nodes

12:39:11   Error    |       *** Error in `mono': double free or corruption (fasttop): 0x0000000001fe4d90 ***
12:39:11   Error    |       ======= Backtrace: =========
12:39:11   Error    |       /lib64/libc.so.6(+0x81329)[0x7f19dd89b329]
12:39:11   Error    |       mono[0x63e34a]
12:39:11   Error    |       mono[0x5d7f43]
12:39:11   Error    |       [0x4186d42c]
12:39:11   Error    |       ======= Memory map: ========
12:39:11   Error    |       00400000-00868000 r-xp 00000000 103:01 13553421                          /usr/bin/mono-sgen
12:39:11   Error    |       00a68000-00a71000 r--p 00468000 103:01 13553421                          /usr/bin/mono-sgen
12:39:11   Error    |       00a71000-00a74000 rw-p 00471000 103:01 13553421                          /usr/bin/mono-sgen
12:39:11   Error    |       00a74000-00a90000 rw-p 00000000 00:00 0
12:39:11   Error    |       015a4000-0208f000 rw-p 00000000 00:00 0                                  [heap]
12:39:11   Error    |       4177b000-4187b000 rwxp 00000000 00:00 0
12:39:11   Error    |       41b00000-41b10000 rwxp 00000000 00:00 0
12:39:11   Error    |       7f19c8000000-7f19c8021000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19c8021000-7f19cc000000 ---p 00000000 00:00 0
12:39:11   Error    |       7f19cdc18000-7f19cdc28000 r--p 00000000 103:01 171966581                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/Sprache.dll
12:39:11   Error    |       7f19cdc28000-7f19cde2a000 r--p 00000000 103:01 318767243                 /usr/lib/mono/gac/System.Data/4.0.0.0__b77a5c561934e089/System.Data.dll
12:39:11   Error    |       7f19cde2a000-7f19cdf10000 r--p 00000000 103:01 352321676                 /usr/lib/mono/gac/System.Runtime.Serialization/4.0.0.0__b77a5c561934e089/System.Runtime.Serialization.dll
12:39:11   Error    |       7f19cdf10000-7f19cdf2e000 r--p 00000000 103:01 369098918                 /usr/lib/mono/gac/System.Numerics/4.0.0.0__b77a5c561934e089/System.Numerics.dll
12:39:11   Error    |       7f19cdf2e000-7f19cdf3e000 rwxp 00000000 00:00 0
12:39:11   Error    |       7f19cdf3e000-7f19ce148000 r--p 00000000 103:01 171966579                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/BouncyCastle.Crypto.dll
12:39:11   Error    |       7f19ce148000-7f19ce2c1000 r--p 00000000 103:01 171967889                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/AWSSDK.Core.dll
12:39:11   Error    |       7f19ce2c1000-7f19ce375000 r--p 00000000 103:01 171967878                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/AWSSDK.S3.dll
12:39:11   Error    |       7f19ce375000-7f19ce3a4000 r--p 00000000 103:01 171966575                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/System.Net.Http.dll
12:39:11   Error    |       7f19ce3a4000-7f19ce3b9000 r--p 00000000 103:01 377487524                 /usr/lib/mono/gac/System.ComponentModel.DataAnnotations/4.0.0.0__31bf3856ad364e35/System.ComponentModel.DataAnnotations.dll
12:39:11   Error    |       7f19ce3b9000-7f19ce45c000 r--p 00000000 103:01 171967881                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/NuGet.Core.dll
12:39:11   Error    |       7f19ce45c000-7f19ce47e000 r--p 00000000 103:01 33554599                  /usr/lib/mono/gac/System.Xml.Linq/4.0.0.0__b77a5c561934e089/System.Xml.Linq.dll
12:39:11   Error    |       7f19ce47e000-7f19ce500000 r--p 00000000 103:01 171966574                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/SharpCompress.dll
12:39:11   Error    |       7f19ce500000-7f19ce600000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19ce60b000-7f19ce614000 r--p 00000000 103:01 171966591                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/System.Threading.Tasks.dll
12:39:11   Error    |       7f19ce614000-7f19ce674000 r-xp 00000000 103:01 98390                     /usr/lib64/libpcre.so.1.2.0
12:39:11   Error    |       7f19ce674000-7f19ce874000 ---p 00060000 103:01 98390                     /usr/lib64/libpcre.so.1.2.0
12:39:11   Error    |       7f19ce874000-7f19ce875000 r--p 00060000 103:01 98390                     /usr/lib64/libpcre.so.1.2.0
12:39:11   Error    |       7f19ce875000-7f19ce876000 rw-p 00061000 103:01 98390                     /usr/lib64/libpcre.so.1.2.0
12:39:11   Error    |       7f19ce876000-7f19ce89a000 r-xp 00000000 103:01 36766                     /usr/lib64/libselinux.so.1
12:39:11   Error    |       7f19ce89a000-7f19cea99000 ---p 00024000 103:01 36766                     /usr/lib64/libselinux.so.1
12:39:11   Error    |       7f19cea99000-7f19cea9a000 r--p 00023000 103:01 36766                     /usr/lib64/libselinux.so.1
12:39:11   Error    |       7f19cea9a000-7f19cea9b000 rw-p 00024000 103:01 36766                     /usr/lib64/libselinux.so.1
12:39:11   Error    |       7f19cea9b000-7f19cea9d000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19cea9d000-7f19ceab3000 r-xp 00000000 103:01 81292                     /usr/lib64/libresolv-2.17.so
12:39:11   Error    |       7f19ceab3000-7f19cecb3000 ---p 00016000 103:01 81292                     /usr/lib64/libresolv-2.17.so
12:39:11   Error    |       7f19cecb3000-7f19cecb4000 r--p 00016000 103:01 81292                     /usr/lib64/libresolv-2.17.so
12:39:11   Error    |       7f19cecb4000-7f19cecb5000 rw-p 00017000 103:01 81292                     /usr/lib64/libresolv-2.17.so
12:39:11   Error    |       7f19cecb5000-7f19cecb7000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19cecb7000-7f19cecba000 r-xp 00000000 103:01 318447                    /usr/lib64/libkeyutils.so.1.5
12:39:11   Error    |       7f19cecba000-7f19ceeb9000 ---p 00003000 103:01 318447                    /usr/lib64/libkeyutils.so.1.5
12:39:11   Error    |       7f19ceeb9000-7f19ceeba000 r--p 00002000 103:01 318447                    /usr/lib64/libkeyutils.so.1.5
12:39:11   Error    |       7f19ceeba000-7f19ceebb000 rw-p 00003000 103:01 318447                    /usr/lib64/libkeyutils.so.1.5
12:39:11   Error    |       7f19ceebb000-7f19ceec9000 r-xp 00000000 103:01 62879                     /usr/lib64/libkrb5support.so.0.1
12:39:11   Error    |       7f19ceec9000-7f19cf0c9000 ---p 0000e000 103:01 62879                     /usr/lib64/libkrb5support.so.0.1
12:39:11   Error    |       7f19cf0c9000-7f19cf0ca000 r--p 0000e000 103:01 62879                     /usr/lib64/libkrb5support.so.0.1
12:39:11   Error    |       7f19cf0ca000-7f19cf0cb000 rw-p 0000f000 103:01 62879                     /usr/lib64/libkrb5support.so.0.1
12:39:11   Error    |       7f19cf0cb000-7f19cf0ce000 r-xp 00000000 103:01 26340                     /usr/lib64/libcom_err.so.2.1
12:39:11   Error    |       7f19cf0ce000-7f19cf2cd000 ---p 00003000 103:01 26340                     /usr/lib64/libcom_err.so.2.1
12:39:11   Error    |       7f19cf2cd000-7f19cf2ce000 r--p 00002000 103:01 26340                     /usr/lib64/libcom_err.so.2.1
12:39:11   Error    |       7f19cf2ce000-7f19cf2cf000 rw-p 00003000 103:01 26340                     /usr/lib64/libcom_err.so.2.1
12:39:11   Error    |       7f19cf2cf000-7f19cf300000 r-xp 00000000 103:01 62870                     /usr/lib64/libk5crypto.so.3.1
12:39:11   Error    |       7f19cf300000-7f19cf4ff000 ---p 00031000 103:01 62870                     /usr/lib64/libk5crypto.so.3.1
12:39:11   Error    |       7f19cf4ff000-7f19cf501000 r--p 00030000 103:01 62870                     /usr/lib64/libk5crypto.so.3.1
12:39:11   Error    |       7f19cf501000-7f19cf502000 rw-p 00032000 103:01 62870                     /usr/lib64/libk5crypto.so.3.1
12:39:11   Error    |       7f19cf502000-7f19cf5db000 r-xp 00000000 103:01 62877                     /usr/lib64/libkrb5.so.3.3
12:39:11   Error    |       7f19cf5db000-7f19cf7da000 ---p 000d9000 103:01 62877                     /usr/lib64/libkrb5.so.3.3
12:39:11   Error    |       7f19cf7da000-7f19cf7e8000 r--p 000d8000 103:01 62877                     /usr/lib64/libkrb5.so.3.3
12:39:11   Error    |       7f19cf7e8000-7f19cf7eb000 rw-p 000e6000 103:01 62877                     /usr/lib64/libkrb5.so.3.3
12:39:11   Error    |       7f19cf7eb000-7f19cf835000 r-xp 00000000 103:01 26376                     /usr/lib64/libgssapi_krb5.so.2.2
12:39:11   Error    |       7f19cf835000-7f19cfa35000 ---p 0004a000 103:01 26376                     /usr/lib64/libgssapi_krb5.so.2.2
12:39:11   Error    |       7f19cfa35000-7f19cfa36000 r--p 0004a000 103:01 26376                     /usr/lib64/libgssapi_krb5.so.2.2
12:39:11   Error    |       7f19cfa36000-7f19cfa38000 rw-p 0004b000 103:01 26376                     /usr/lib64/libgssapi_krb5.so.2.2
12:39:11   Error    |       7f19cfa38000-7f19cfaf8000 r-xp 00000000 103:01 647318                    /usr/lib64/libmono-native.so.0.0.0
12:39:11   Error    |       7f19cfaf8000-7f19cfcf7000 ---p 000c0000 103:01 647318                    /usr/lib64/libmono-native.so.0.0.0
12:39:11   Error    |       7f19cfcf7000-7f19cfcf8000 r--p 000bf000 103:01 647318                    /usr/lib64/libmono-native.so.0.0.0
12:39:11   Error    |       7f19cfcf8000-7f19cfcf9000 rw-p 000c0000 103:01 647318                    /usr/lib64/libmono-native.so.0.0.0
12:39:11   Error    |       7f19cfcf9000-7f19d0000000 r--p 00000000 103:01 46137826                  /usr/lib/mono/gac/System.Xml/4.0.0.0__b77a5c561934e089/System.Xml.dll
12:39:11   Error    |       7f19d0000000-7f19d0021000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19d0021000-7f19d4000000 ---p 00000000 00:00 0
12:39:11   Error    |       7f19d4003000-7f19d4017000 r--p 00000000 103:01 171966571                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/Microsoft.Web.XmlTransform.dll
12:39:11   Error    |       7f19d4017000-7f19d404a000 r--p 00000000 103:01 171966570                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/YamlDotNet.dll
12:39:11   Error    |       7f19d404a000-7f19d4067000 r--p 00000000 103:01 171966586                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/Octostache.dll
12:39:11   Error    |       7f19d4067000-7f19d4087000 r--p 00000000 103:01 171966585                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/Microsoft.Web.Administration.dll
12:39:11   Error    |       7f19d4087000-7f19d4114000 r--p 00000000 103:01 171966578                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/Newtonsoft.Json.dll
12:39:11   Error    |       7f19d4114000-7f19d4393000 r--p 00000000 103:01 50331730                  /usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll
12:39:11   Error    |       7f19d4393000-7f19d449d000 r--p 00000000 103:01 260047275                 /usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll
12:39:11   Error    |       7f19d449d000-7f19d4d49000 r-xp 00000000 103:01 146801354                 /usr/lib/mono/4.5/mscorlib.dll.so
12:39:11   Error    |       7f19d4d49000-7f19d4f49000 ---p 008ac000 103:01 146801354                 /usr/lib/mono/4.5/mscorlib.dll.so
12:39:11   Error    |       7f19d4f49000-7f19d4f6b000 r--p 008ac000 103:01 146801354                 /usr/lib/mono/4.5/mscorlib.dll.so
12:39:11   Error    |       7f19d4f6b000-7f19d4f6c000 rw-p 008ce000 103:01 146801354                 /usr/lib/mono/4.5/mscorlib.dll.so
12:39:11   Error    |       7f19d4f6c000-7f19d4f94000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19d4f94000-7f19d53ff000 r--p 00000000 103:01 146801353                 /usr/lib/mono/4.5/mscorlib.dll
12:39:11   Error    |       7f19d53ff000-7f19d63ff000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19d63ff000-7f19d6400000 ---p 00000000 00:00 0
12:39:11   Error    |       7f19d6400000-7f19d7000000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19d7003000-7f19d7005000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19d7005000-7f19d7019000 r--p 00000000 103:01 171967877                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/System.ValueTuple.dll
12:39:11   Error    |       7f19d7019000-7f19d7029000 r--p 00000000 103:01 171967886                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/Octopus.Versioning.dll
12:39:11   Error    |       7f19d7029000-7f19d702a000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19d702a000-7f19d703c000 r--p 00000000 103:01 104857720                 /usr/lib/mono/gac/I18N.West/4.0.0.0__0738eb9f132ed756/I18N.West.dll
12:39:11   Error    |       7f19d703c000-7f19d7046000 r--p 00000000 103:01 109052007                 /usr/lib/mono/gac/I18N/4.0.0.0__0738eb9f132ed756/I18N.dll
12:39:11   Error    |       7f19d7046000-7f19d706e000 r--p 00000000 103:01 171966572                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/Calamari.Shared.dll
12:39:11   Error    |       7f19d706e000-7f19d70a0000 r--p 00000000 103:01 171966584                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/Autofac.dll
12:39:11   Error    |       7f19d70a0000-7f19d70d6000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19d70d6000-7f19d70d7000 ---p 00000000 00:00 0
12:39:11   Error    |       7f19d70d7000-7f19d70d8000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19d70d8000-7f19d70e0000 ---p 00000000 00:00 0
12:39:11   Error    |       7f19d70e0000-7f19d72d7000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19d72d7000-7f19dd81a000 r--p 00000000 103:01 81264                     /usr/lib/locale/locale-archive
12:39:11   Error    |       7f19dd81a000-7f19dd9de000 r-xp 00000000 103:01 1087                      /usr/lib64/libc-2.17.so
12:39:11   Error    |       7f19dd9de000-7f19ddbdd000 ---p 001c4000 103:01 1087                      /usr/lib64/libc-2.17.so
12:39:11   Error    |       7f19ddbdd000-7f19ddbe1000 r--p 001c3000 103:01 1087                      /usr/lib64/libc-2.17.so
12:39:11   Error    |       7f19ddbe1000-7f19ddbe3000 rw-p 001c7000 103:01 1087                      /usr/lib64/libc-2.17.so
12:39:11   Error    |       7f19ddbe3000-7f19ddbe8000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19ddbe8000-7f19ddbfd000 r-xp 00000000 103:01 585                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
12:39:11   Error    |       7f19ddbfd000-7f19dddfc000 ---p 00015000 103:01 585                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
12:39:11   Error    |       7f19dddfc000-7f19dddfd000 r--p 00014000 103:01 585                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
12:39:11   Error    |       7f19dddfd000-7f19dddfe000 rw-p 00015000 103:01 585                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
12:39:11   Error    |       7f19dddfe000-7f19ddeff000 r-xp 00000000 103:01 81272                     /usr/lib64/libm-2.17.so
12:39:11   Error    |       7f19ddeff000-7f19de0fe000 ---p 00101000 103:01 81272                     /usr/lib64/libm-2.17.so
12:39:11   Error    |       7f19de0fe000-7f19de0ff000 r--p 00100000 103:01 81272                     /usr/lib64/libm-2.17.so
12:39:11   Error    |       7f19de0ff000-7f19de100000 rw-p 00101000 103:01 81272                     /usr/lib64/libm-2.17.so
12:39:11   Error    |       7f19de100000-7f19de1e9000 r-xp 00000000 103:01 26351                     /usr/lib64/libstdc++.so.6.0.19
12:39:11   Error    |       7f19de1e9000-7f19de3e9000 ---p 000e9000 103:01 26351                     /usr/lib64/libstdc++.so.6.0.19
12:39:11   Error    |       7f19de3e9000-7f19de3f1000 r--p 000e9000 103:01 26351                     /usr/lib64/libstdc++.so.6.0.19
12:39:11   Error    |       7f19de3f1000-7f19de3f3000 rw-p 000f1000 103:01 26351                     /usr/lib64/libstdc++.so.6.0.19
12:39:11   Error    |       7f19de3f3000-7f19de408000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19de408000-7f19de41f000 r-xp 00000000 103:01 81290                     /usr/lib64/libpthread-2.17.so
12:39:11   Error    |       7f19de41f000-7f19de61e000 ---p 00017000 103:01 81290                     /usr/lib64/libpthread-2.17.so
12:39:11   Error    |       7f19de61e000-7f19de61f000 r--p 00016000 103:01 81290                     /usr/lib64/libpthread-2.17.so
12:39:11   Error    |       7f19de61f000-7f19de620000 rw-p 00017000 103:01 81290                     /usr/lib64/libpthread-2.17.so
12:39:11   Error    |       7f19de620000-7f19de624000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19de624000-7f19de626000 r-xp 00000000 103:01 81270                     /usr/lib64/libdl-2.17.so
12:39:11   Error    |       7f19de626000-7f19de826000 ---p 00002000 103:01 81270                     /usr/lib64/libdl-2.17.so
12:39:11   Error    |       7f19de826000-7f19de827000 r--p 00002000 103:01 81270                     /usr/lib64/libdl-2.17.so
12:39:11   Error    |       7f19de827000-7f19de828000 rw-p 00003000 103:01 81270                     /usr/lib64/libdl-2.17.so
12:39:11   Error    |       7f19de828000-7f19de82f000 r-xp 00000000 103:01 81320                     /usr/lib64/librt-2.17.so
12:39:11   Error    |       7f19de82f000-7f19dea2e000 ---p 00007000 103:01 81320                     /usr/lib64/librt-2.17.so
12:39:11   Error    |       7f19dea2e000-7f19dea2f000 r--p 00006000 103:01 81320                     /usr/lib64/librt-2.17.so
12:39:11   Error    |       7f19dea2f000-7f19dea30000 rw-p 00007000 103:01 81320                     /usr/lib64/librt-2.17.so
12:39:11   Error    |       7f19dea30000-7f19dea45000 r-xp 00000000 103:01 62850                     /usr/lib64/libz.so.1.2.7
12:39:11   Error    |       7f19dea45000-7f19dec44000 ---p 00015000 103:01 62850                     /usr/lib64/libz.so.1.2.7
12:39:11   Error    |       7f19dec44000-7f19dec45000 r--p 00014000 103:01 62850                     /usr/lib64/libz.so.1.2.7
12:39:11   Error    |       7f19dec45000-7f19dec46000 rw-p 00015000 103:01 62850                     /usr/lib64/libz.so.1.2.7
12:39:11   Error    |       7f19dec46000-7f19dec68000 r-xp 00000000 103:01 1074                      /usr/lib64/ld-2.17.so
12:39:11   Error    |       7f19dec68000-7f19dec6f000 r--p 00000000 103:01 171966588                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/Octopus.CoreUtilities.dll
12:39:11   Error    |       7f19dec6f000-7f19dec8d000 r--p 00000000 103:01 251658702                 /usr/lib/mono/gac/System.Configuration/4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
12:39:11   Error    |       7f19dec8d000-7f19decef000 r--p 00000000 103:01 171966589                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/Calamari.Common.dll
12:39:11   Error    |       7f19decef000-7f19ded30000 r--p 00000000 103:01 171966583                 /home/javaman/.octopus/OctopusServer/Tools/Calamari.netfx/24.0.8/Calamari.exe
12:39:11   Error    |       7f19ded30000-7f19dedf7000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19dedf7000-7f19dee56000 ---p 00000000 00:00 0
12:39:11   Error    |       7f19dee56000-7f19dee5e000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19dee5e000-7f19dee65000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19dee65000-7f19dee66000 rw-s 00000000 00:13 6115433                    /dev/shm/mono.7765
12:39:11   Error    |       7f19dee66000-7f19dee67000 rw-p 00000000 00:00 0
12:39:11   Error    |       7f19dee67000-7f19dee68000 r--p 00021000 103:01 1074                      /usr/lib64/ld-2.17.so
12:39:11   Error    |       7f19dee68000-7f19dee69000 rw-p 00022000 103:01 1074                      /usr/lib64/ld-2.17.so
12:39:11   Error    |       7f19dee69000-7f19dee6a000 rw-p 00000000 00:00 0
12:39:11   Error    |       7fff0eb81000-7fff0eb89000 ---p 00000000 00:00 0
12:39:11   Error    |       7fff0f35f000-7fff0f380000 rw-p 00000000 00:00 0                          [stack]
12:39:11   Error    |       7fff0f3e2000-7fff0f3e4000 r-xp 00000000 00:00 0                          [vdso]
12:39:12   Error    |       ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
3

Thanks for reaching out @Naman.Kumar,

Sorry to see you are having issues with Mono when using Linux Workers.

I’ll need a bit more information to help me get to the bottom of this.

Does this happen immediately or are you receiving this error midway through a deployment? Possibly while a custom script is running?

From a quick Google, it looks like these errors can be caused by a “Double Deletion”, or more specifically, trying to move, delete or copy a memory location which isn’t available anymore.

Sometimes it can be caused by using a command that will move/delete the specific location of a pointers memory address. So if you have two pointers in a directory pointing to the same location, and the command you use, operates on all of the target locations of the pointers in the folder, the first command on a specific memory location could work but would leave the memory location empty, which will cause the error.

If you could provide a bit more information on when this error occurs, what type of linux worker you are using, what version of mono you have installed and anything else that you consider relevant, I will definitely be happy to help.

Regards,

4

Hey @dane.falvo ,
so this happens during the “Acquire Package” step in deployment and is happening intermittently. I can share the deployment logs if you can generate a secure link for the same

5

Hi @Naman.Kumar!

Just jumping in for Dane here, as he’s wrapped up for the day, but you can upload your logs here: Support - Octopus Deploy

If it’s not clear in the logs, can you also let us know some more data about your target environment (Distribution/Version/Architecture/Mono version etc.)

Thanks!

6

I uploaded the log file in the above link.
The process comprises 3 steps

image
image1500Ă—498 69.2 KB

before the second step, it acquires the package to be used in the step and that is where the error is thrown

PS: I have tried running it on a different worker node, but it is still failing 10-15% of deployments with the same error.

7

Thanks for the log, @Naman.Kumar!

Digging through it, I can see you’re running RHEL7 - what version of Mono is installed on these machines? We support and test against mono 5.14.0 and greater - how does this line up with what’s installed on the machine?

8 
Mono JIT compiler version 6.12.0.107 (tarball Wed Dec  9 21:42:51 UTC 2020)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	Interpreter:   yes
	LLVM:          yes(610)
	Suspend:       hybrid
	GC:            sgen (concurrent by default)
9

Great - thanks for the update. I’ll spin up a local reproduction to see if I can replicate it with those exact versions. Is anything else noted in the system logs regarding mono crashing?

10

Hi @Naman.Kumar

I spun up a reproduction on this one (RHEL 7.9/Mono 6.12.0.107), then called a script with a referenced package, as listed above, and it all went through as expected:

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)

$ uname -a
Linux ip-172-31-2-174.us-west-2.compute.internal 3.10.0-1160.81.1.el7.x86_64 #1 SMP Thu Nov 24 12:21:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ mono --version
Mono JIT compiler version 6.12.0.107 (tarball Wed Dec  9 21:42:51 UTC 2020)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com

Tasklog:

                   |       Success: Upload package blahsh v1.1.1
16:56:18   Verbose  |         Acquiring isolation mutex SshRemoteEndpointFacade-28:ea:a2:4e:a5:23:59:be:15:d9:9a:f2:21:53:ae:f9-RunningScript with NoIsolation in ServerTasks-1024266
16:56:18   Verbose  |         Establishing SSH connection...
16:56:18   Verbose  |         Using ssh-ed25519 to authenticate SSH Endpoint fingerprint
<snip for brevity>
16:56:28   Verbose  |         Started Calamari PID 8558
16:56:28   Verbose  |         Calamari Version: 24.0.25
16:56:28   Verbose  |         Environment Information:
16:56:28   Verbose  |         OperatingSystem: Unix 3.10.0.1160
16:56:28   Verbose  |         OsBitVersion: x64
16:56:28   Verbose  |         Is64BitProcess: True
16:56:28   Verbose  |         Running on Mono: True
16:56:28   Verbose  |         CurrentUser: ec2-user
16:56:28   Verbose  |         MachineName: ip-172-31-2-174
16:56:28   Verbose  |         ProcessorCount: 2
16:56:28   Verbose  |         CurrentDirectory: /home/ec2-user/.octopus/OctopusServer/Work/20221213165624-1024266-18
16:56:28   Verbose  |         TempDirectory: /tmp/
16:56:29   Verbose  |         HostProcess: Calamari (8558)
16:56:29   Verbose  |         Package blahsh version 1.1.1 hash c1059f27454fda73adfbbc0678b2ff96f3fc1335 has not been uploaded.
16:56:29   Verbose  |         Finding earlier packages that have been uploaded to this Tentacle.
16:56:29   Verbose  |         No earlier packages for blahsh has been uploaded
16:56:29   Verbose  |         Found 0 earlier versions of blahsh on this Tentacle
16:56:29   Verbose  |         SFTP connection disposed.
16:56:29   Verbose  |         SSH connection disposed.
16:56:29   Verbose  |         Exit code: 0
16:56:29   Info     |         No earlier versions of blahsh were found on the target so we can't perform a delta upload.
16:56:29   Info     |         Uploading package blahsh (150 B)...
16:56:29   Verbose  |         Requesting upload...
16:56:29   Verbose  |         Establishing SSH connection...
16:56:29   Verbose  |         Using ssh-ed25519 to authenticate SSH Endpoint fingerprint
16:56:29   Verbose  |         SSH connection established
16:56:30   Verbose  |         Beginning streaming transfer of blahsh@S1.1.1@C1541F250772EE409A5C5074723BBD91.zip to $HOME/.octopus/OctopusServer/Files
16:56:30   Verbose  |         Establishing SFTP connection...
16:56:30   Verbose  |         Using ssh-ed25519 to authenticate SSH Endpoint fingerprint
16:56:30   Verbose  |         SFTP connection established
16:56:30   Verbose  |         Stream transfer complete
16:56:31   Verbose  |         SFTP connection disposed.
16:56:31   Verbose  |         SSH connection disposed.

Looking at the log and what it’s doing when it crashes, I wonder if disabling the new version of the package retention feature will alleviate this for you.

Could you try setting an environmental variable on your Octopus server, with the name OCTOPUS__FeatureToggles__PackageRetentionFeatureToggle and a value of False. From here, if you can restart the Octopus service, and then try the deployment again?

Other than that, this could potentially be an environmental issue with these servers - I wonder if there’s more data in the system logs, as mentioned above?

1 Like
12

Hey, we are running Octopus on Linux containers.
I didn’t find anything “mono” related in the Octo logs.

Are you asking to check system logs from Octo Server (Container) or the worker node?
Additionally, Can you help me with the server command to set this variable on Octopus Server?

14

Hi @Naman.Kumar!

Sorry for the confusion - if you could check the logs on the worker node to see if there’s anything there that might provide any further information about the mono crash there. Is there a low memory event? Or any other services indicating any memory-related problems?

In regards to the environment variable, since you’re running on a docker container, you will want to pass this in on your docker run as an env variable -e OCTOPUS__FeatureToggles__PackageRetentionFeatureToggle=False or in your deployment if you’re running on Kubernetes.

I hope this helps!

15

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.