SMTP ssl certificate error on v2019.9.10

We have an ssl cert error where we’re trying to send an smtp request from an octo server in a secure env that doesn’t have direct internet access. We created an aws vpc endpoint for SES(email-smtp) for the server to connect to but we are getting an ssl cert error when trying to test our smtp request in octopus. We were able to get the root CAs in the octopus server’s system trust store but the error persists. We’ve verified that we were able to connect to the vpc endpoint from the server and sent a test email successfully(https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp-client-command-line.html). We believe the problem lies somewhere in the email configuration in octopus rather than the server. Can you give us additional guidance on how to proceed?

Error Response:

    The SSL certificate presented by the server is not trusted by the system for one or more of the 
    following reasons:
    1. The server is using a self-signed certificate which cannot be verified.
    2. The local system is missing a Root or Intermediate certificate needed to verify the server's 
    certificate.
    3. The certificate presented by the server is expired or invalid.

Hi @t.xu,

First of all, welcome to the Octopus Forums!

Thanks for reaching out, sorry you’re having issues with your certs.

Which store have you placed the certificates in?

Are you running Octopus as a service account? Does it have access to the certificate store you have the certificates in?

Thanks,
Jeremy

Hey @jeremy.miller

We’ve added the Amazon Root CAs to the local machine’s trusted store.

And to answer your second question Octopus is ran by local admin so it should have access to the certificate store.

Best,
Thomas

Hi Thomas,

Is your entire certificate chain in the store, or just the root CA?

Thanks,
Jeremy

We just pinned the root CA certs. We also successfully ran aws tests to make sure we’d be able to send an email from the service to the vpc endpoint we have set up(ref: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp-client-command-line.html). But it won’t work from Octo.

Hi Thomas,

I spoke with one of our engineers regarding this and he did some investigation to get us some next steps.

Octopus uses MailKit to do its mail portions.

Could you please run Install-Package MailKit in Powershell outside of Octopus.

Then run the following c sharp code (filling in your info):

using (var client = new MailKit.Net.Smtp.SmtpClient())
{
    await client.ConnectAsync("smtp.example.com", 465, MailKit.Security.SecureSocketOptions.SslOnConnect, CancellationToken.None);
    // or
    await client.ConnectAsync("smtp.example.com", 465, MailKit.Security.SecureSocketOptions.StartTls, CancellationToken.None);
}

Octopus tries both methods, so if either works, Octopus should also work. If it doesn’t work this may give us an indication on what’s happening.

Here is a FAQ for MailKit that may also help you troubleshoot the issue: https://github.com/jstedfast/MailKit/blob/master/FAQ.md

Please let me know how it goes and if you have any other questions or concerns.

Thanks,
Jeremy

Hey Jeremy,

Currently the machine is in a secure environment and installing packages like that is carefully controlled. Do you know if the full chain needs to be trust for MailKit to function?

Hey Thomas,

I’m sorry but I don’t know that. I’ll get back with the engineers and update you. They are on holiday tomorrow so it will likely be Monday when you hear from me.

Please let me know if you have any questions in the meantime.

Thanks,
Jeremy

Any word?

Hey Thomas,

I’m really sorry about the delay. Let me reach out again and see if I can get you an answer tonight.

Please feel free to reach out in the meantime.

Thanks,
Jeremy

Hey Thomas,

I heard back. Sorry about the misinformation on the first post.

Install-Package MailKit

using (var client = new MailKit.Net.Smtp.SmtpClient())
{
    await client.ConnectAsync("smtp.example.com", 465, MailKit.Security.SecureSocketOptions.SslOnConnect, CancellationToken.None);
    // or
    await client.ConnectAsync("smtp.example.com", 465, MailKit.Security.SecureSocketOptions.StartTls, CancellationToken.None);
}

Is all C#, no PowerShell is involved.

To answer the chain question, as far as we’re aware we would expect that the full certificate chain would need to be trusted in order for the certificate to be considered valid. We can’t say this with 100% certainty that this is how Mailkit behaves, but is what is expected.

As a test, you should be able to run the above C# code on a machine outside of the secure environment once you get the cert in place on that machine and you should get the same error that you would have received in the secure environment.

Please let me know what you think.

Thanks,
Jeremy

Hey Jeremy,

Sorry it took so long to respond.

Add-Type -Path "C:\Program Files\PackageManagement\NuGet\Packages\MailKit.2.8.0\lib\net46\MailKit.dll"
Add-Type -Path "C:\Program Files\PackageManagement\NuGet\Packages\MimeKit.2.9.1\lib\net46\MimeKit.dll"

$SMTP     = New-Object MailKit.Net.Smtp.SmtpClient
$Message  = New-Object MimeKit.MimeMessage
$TextPart = [MimeKit.TextPart]::new("plain")
$TextPart.Text = "This is a test via MailKit"
$Message.From.Add("")
$Message.To.Add("")
$Message.Subject = 'Test Message'
$Message.Body    = $TextPart 
 
$SMTP.Connect('smtp.example.com', 465, [MailKit.Security.SecureSocketOptions]::SslOnConnect, $False) 
#$SMTP.Connect('smtp.example.com', 465, [MailKit.Security.SecureSocketOptions]::StartTls, $False) 

$SMTP.Send($Message)
$SMTP.Disconnect($true)
$SMTP.Dispose() 

So as you see we’re running MailKit in Powershell. And using Port 465 the [MailKit.Security.SecureSocketOptions]::SslOnConnect option is able to send a message to the receiver, but not when I use [MailKit.Security.SecureSocketOptions]::StartTls option. I get the error

"The SMTP server has unexpectedly disconnected." 

Interestingly enough the [MailKit.Security.SecureSocketOptions]::StartTls option works when I use port 587. And the error I get from [MailKit.Security.SecureSocketOptions]::SslOnConnect option looks similar to the original error response we’re getting

Exception calling "An error occurred while attempting to establish an SSL or TLS connection.
This usually means that the SSL certificate presented by the server is not trusted by the system for one or more of the following reasons: 
1. The server is using a self-signed certificate which cannot be verified.
2. The local system is missing a Root or Intermediate certificate needed to verify the server's certificate.
3. A Certificate Authority CRL server for one or more of the certificates in the chain is temporarily unavailable.
4. The certificate presented by the server is expired or invalid.
Another possibility is that you are trying to connect to a port which does not support SSL/TLS.
It is also possible that the set of SSL/TLS protocols supported by the client and server do not match.
See https://github.com/jstedfast/MailKit/blob/master/FAQ.md#SslHandshakeException for possible solutions.

Let me know if there’s anything else you need from me.

Hi,

It looks like that last log may have been cutoff. Would you be able to check if there was more to it?

"Exception calling “An error occurred while attempting to establish an SSL or TLS connection.
This usually means that the SSL certificate presented by the server is not trusted by the system for one or more of the following reasons:”

Thanks,
Jeremy

1 Like

Oh it’s the same as the original error from Octo earlier. But I’ll update previous comment

Hi,

I’m not sure if you’ve tried it yet, but have you tried toggling this setting and attempting it and see if it works?

Thanks,
Jeremy

My heart sank for a moment. But yeah it was toggled to use SSL/TLS

Could you please try unchecking it just to see if it works?

Thanks,
Jeremy

Didn’t work. Got this error though:

530: Must issue a STARTTLS command first MailKit.Security.AuthenticationException at MailKit.Net.Smtp.SmtpClient.AuthenticateAsync at MailKit.Net.Smtp.SmtpClient.Authenticate(Encoding encoding, ICredentials credentials, CancellationToken cancellationToken) at Octopus.Core.Util.SmtpClient.SmtpClientWrapper.CreateSmtpClient(SmtpConfiguration configuration, CancellationToken token) in SmtpClientWrapper.cs:line 76 at Octopus.Core.Util.SmtpClient.SmtpClientWrapper.Send(SmtpConfiguration configuration, MailMessage message, CancellationToken token) in SmtpClientWrapper.cs:line 33 at Octopus.Server.Orchestration.ServerTasks.TestEmail.TestEmailTaskController.Execute() in TestEmailTaskController.cs:line 54 at Octopus.Server.Orchestration.ServerTasks.RunningTask.RunMainThread() in RunningTask.cs:line 104 --Inner Exception-- Must issue a STARTTLS command first MailKit.Net.Smtp.SmtpCommandException

Hi,

Would you please be able to privately message a screenshot of your smtp configuration in Octopus to me?

Thanks,
Jeremy

Hi,

We’ve managed to replicate the issue within our AWS environment.

The problem is that to validate the AWS SES certificate, your server needs to be able to reach out and retrieve an appropriate Certificate Revocation List during the STARTTLS protocol execution: https://en.wikipedia.org/wiki/Certificate_revocation_list

You can see in the below wireshark capture the DNS lookup for CRL retrieval - this is issued when attempting to send mail.

The addresses that are resolved for this CRL (13.35.149.*) will not be able to be resolved from your server, as it does not have egress to the internet.

Without being able to retrieve this list, the server certificate validation fails.

Given SES requires all email sending to be encrypted via TLS https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-connect.html and both modes of TLS will require checking the CA’s CRL for the provided certificate, you will need to provide outbound access for these CRL requests to complete from your server.

Note that this isn’t the only outbound communication Octopus does - certain steps, like our Azure steps, make calls to external services. These will also fail if your Octopus server is air-gapped from the internet.

Cheers,

Andrew.

1 Like