SSL Errors When Switching From Fedora 35 to Fedora 36

tompurl · 22 August 2022 22:12

Some of our onpremise TeamCity agents run in Kubernetes as long-lived Docker containers. I am in the process of updating these agents to use a new version of the base image.

Old (Working) Base Image: quay.io/buildah/stable:v1.23.3
New (Broken) Base Image: quay.io/buildah/stable:v1.26

The problem is that the updated containers can’t connect to Octopus using octo.dll. Here’s an example of an error that we get:

[root@teamcity-agent-linux-02 Core]# dotnet octo.dll list-machines --server=https://octopusdeploy.somecompany.com --apikey='API-CLEVERAPIKEY'
Octopus CLI, version 7.4.3383

Detected automation environment: "NoneOrUnknown"

System.Exception: Unable to connect to the Octopus Deploy server. See the inner exception for details.
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.TypeInitializationException: The type initializer for 'SslMethods' threw an exception.
 ---> System.TypeInitializationException: The type initializer for 'Ssl' threw an exception.
 ---> System.TypeInitializationException: The type initializer for 'SslInitializer' threw an exception.
 ---> Interop+Crypto+OpenSslCryptographicException: error:0E076071:configuration file routines:module_run:unknown module name
   at Interop.SslInitializer..cctor() in /_/src/Common/src/Interop/Unix/System.Net.Security.Native/Interop.Initialization.cs:line 33
   --- End of inner exception stack trace ---
   at Interop.Ssl..cctor() in /_/src/Common/src/Interop/Unix/System.Net.Security.Native/Interop.Initialization.cs:line 16
   --- End of inner exception stack trace ---
   at Interop.Ssl.SslV2_3Method()
   at Interop.Ssl.SslMethods..cctor() in /_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Ssl.cs:line 194
   --- End of inner exception stack trace ---
   at Interop.OpenSsl.AllocateSslContext(SslProtocols protocols, SafeX509Handle certHandle, SafeEvpPKeyHandle certKeyHandle, EncryptionPolicy policy, SslAuthenticationOptions sslAuthenticationOptions) in /_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 253
   at System.Net.Security.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, SslAuthenticationOptions sslAuthenticationOptions) in /_/src/Common/src/System/Net/Security/Unix/SafeDeleteSslContext.cs:line 47
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, ArraySegment`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /_/src/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 148
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 723
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 557
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 528
   at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 422
   at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 340
   at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.cs:line 234
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.cs:line 403
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/FutureFactory.cs:line 1040
   at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.cs:line 402
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs:line 172
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs:line 172
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 662
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 675
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 345
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 554
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.cs:line 326
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 84
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in /_/src/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 546
   at Octopus.Client.OctopusAsyncClient.DispatchRequest[TResponseResource](OctopusRequest request, Boolean readResponse)
   at Octopus.Client.OctopusAsyncClient.Get[TResource](String path, Object pathParameters)
   at Octopus.Client.OctopusAsyncRepository.LoadRootDocumentInner()
   --- End of inner exception stack trace ---
   at Octopus.Client.OctopusAsyncRepository.LoadRootDocumentInner()
   at Octopus.Client.OctopusAsyncClient.Create(OctopusServerEndpoint serverEndpoint, OctopusClientOptions options, Boolean addHandler, String requestingTool)
   at Octopus.Client.OctopusAsyncClient.Create(OctopusServerEndpoint serverEndpoint, OctopusClientOptions options, String requestingTool)
   at Octopus.Cli.Commands.ApiCommand.Execute(String[] commandLineArguments) in C:\BuildAgent\work\289bf0fca31007af\source\Octopus.Cli\Commands\ApiCommand.cs:line 177
   at Octopus.Cli.CliProgram.Run(String[] args) in C:\BuildAgent\work\289bf0fca31007af\source\Octopus.Cli\CliProgram.cs:line 54
Exit code: -3

Here’s what I get on one of the older, working containers:

[root@teamcity-agent-linux-00]# dotnet octo.dll list-machines --server=https://octopusdeploy.somecompany.com --apikey='API-CLEVERAPIKEY'
Octopus CLI, version 7.4.3383

Detected automation environment: "NoneOrUnknown"
Space name unspecified, process will run in the default space context
Handshaking with Octopus Server: https://octopusdeploy.somecompany.com
Handshake successful. Octopus version: 2021.3.8275; API version: 3.0.0
Authenticated as: Tom Purl <tom.purl@some-company.com> 
Loading environments...
Loading machines...
Loading machines from all environments...
Machines: 22
...lots of hosts...

The issue seems to be that our custom CA cert for the octopusdeploy.somecompany.com host isn’t properly installed. However, the process for adding that cert to that container hasn’t changed. Also, I can run this on the “broken” agent and it works:

curl -L https://octopusdeploy.somecompany.com

Here is how we’re installing the custom CA cert:

cp -rL /config/ca_*.crt /etc/pki/ca-trust/source/anchors/ -R
update-ca-trust

I then thought that it might be an issue with our dotnet runtimes, but it looks like things are mostly identical. Here are the SDK’s installed on the working nodes:

[root@teamcity-agent-linux-00 certs]# dotnet --list-sdks
3.1.419 [/usr/lib64/dotnet/sdk]
6.0.106 [/usr/lib64/dotnet/sdk]

…and here’s the SDK’s on the broken node:

[root@teamcity-agent-linux-02 Core]# dotnet --list-sdks
3.1.421 [/usr/lib64/dotnet/sdk]
6.0.107 [/usr/lib64/dotnet/sdk]

The only other difference is that the underlying OS version in the container’s base imge. Here’s the OS info for the working container:

[root@teamcity-agent-linux-00-848946954b-58pkg Core]# cat /etc/os-release 
NAME="Fedora Linux"
VERSION="35 (Container Image)"
ID=fedora
VERSION_ID=35
VERSION_CODENAME=""
PLATFORM_ID="platform:f35"
PRETTY_NAME="Fedora Linux 35 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:35"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f35/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=35
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=35
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container

…and here’s the OS info for the broken container:

[root@teamcity-agent-linux-02-688cb4b6dc-pvz8l Core]# cat /etc/os-release 
NAME="Fedora Linux"
VERSION="36 (Container Image)"
ID=fedora
VERSION_ID=36
VERSION_CODENAME=""
PLATFORM_ID="platform:f36"
PRETTY_NAME="Fedora Linux 36 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:36"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=36
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=36
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container

So we have switched from Fedora 35 to Fedora 36. I checked the release notes for the new OS version and didn’t see anything that would cause this type of issue though.

britton.riggs · 22 August 2022 22:49

Hi @tompurl,

Thanks for reaching out, and welcome to the Octopus Deploy community!

I’m sorry to hear you’re having trouble with your latest container image and the Octopus Deploy .dll, but I’d be happy to help take a look at this.

In reviewing the error message you provided, I did pick out the following line:

Interop+Crypto+OpenSslCryptographicException: error:0E076071:configuration file routines:module_run:unknown module name

In doing some initial research, I came across the following GitHub issue that references an identical error, with a recommended solution of adding the environment variable CLR_OPENSSL_VERSION_OVERRIDE with a value of 1.1.

Alternatively, it looks like another user had success in using the environment variable DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER with a value of 0.

After setting either of these values I would recommend restarting your local session so the environment variable is pulled in as expected.

It looks like this error has to do with the package libssl, and we have seen some compatibility issues around this package when using newer versions of Linux with some of the Octopus Deploy command-line tooling.

I hope this initial information helps in resolving the issue, but feel free to reach back out if I can be of any more help.

Best regards,

Britton

tompurl · 23 August 2022 15:30

Thank you for your quick reply @britton.riggs !

I tried both of these environment variables and the second one did the trick.

britton.riggs · 23 August 2022 16:02

Hi @tompurl,

You’re welcome, I’m happy to hear the second environment variable worked for you!

Hope you have a good rest of your week and happy deployments!

Britton

system · 23 September 2022 16:03

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.