We have an ssl cert error where we’re trying to send an smtp request from an octo server in a secure env that doesn’t have direct internet access. We created an aws vpc endpoint for SES(email-smtp) for the server to connect to but we are getting an ssl cert error when trying to test our smtp request in octopus. We were able to get the root CAs in the octopus server’s system trust store but the error persists. We’ve verified that we were able to connect to the vpc endpoint from the server and sent a test email successfully(https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp-client-command-line.html). We believe the problem lies somewhere in the email configuration in octopus rather than the server. Can you give us additional guidance on how to proceed?
Error Response:
The SSL certificate presented by the server is not trusted by the system for one or more of the
following reasons:
1. The server is using a self-signed certificate which cannot be verified.
2. The local system is missing a Root or Intermediate certificate needed to verify the server's
certificate.
3. The certificate presented by the server is expired or invalid.
Currently the machine is in a secure environment and installing packages like that is carefully controlled. Do you know if the full chain needs to be trust for MailKit to function?
I’m sorry but I don’t know that. I’ll get back with the engineers and update you. They are on holiday tomorrow so it will likely be Monday when you hear from me.
Please let me know if you have any questions in the meantime.
I heard back. Sorry about the misinformation on the first post.
Install-Package MailKit
using (var client = new MailKit.Net.Smtp.SmtpClient())
{
await client.ConnectAsync("smtp.example.com", 465, MailKit.Security.SecureSocketOptions.SslOnConnect, CancellationToken.None);
// or
await client.ConnectAsync("smtp.example.com", 465, MailKit.Security.SecureSocketOptions.StartTls, CancellationToken.None);
}
Is all C#, no PowerShell is involved.
To answer the chain question, as far as we’re aware we would expect that the full certificate chain would need to be trusted in order for the certificate to be considered valid. We can’t say this with 100% certainty that this is how Mailkit behaves, but is what is expected.
As a test, you should be able to run the above C# code on a machine outside of the secure environment once you get the cert in place on that machine and you should get the same error that you would have received in the secure environment.
So as you see we’re running MailKit in Powershell. And using Port 465 the [MailKit.Security.SecureSocketOptions]::SslOnConnect option is able to send a message to the receiver, but not when I use [MailKit.Security.SecureSocketOptions]::StartTls option. I get the error
"The SMTP server has unexpectedly disconnected."
Interestingly enough the [MailKit.Security.SecureSocketOptions]::StartTls option works when I use port 587. And the error I get from [MailKit.Security.SecureSocketOptions]::SslOnConnect option looks similar to the original error response we’re getting
Exception calling "An error occurred while attempting to establish an SSL or TLS connection.
This usually means that the SSL certificate presented by the server is not trusted by the system for one or more of the following reasons:
1. The server is using a self-signed certificate which cannot be verified.
2. The local system is missing a Root or Intermediate certificate needed to verify the server's certificate.
3. A Certificate Authority CRL server for one or more of the certificates in the chain is temporarily unavailable.
4. The certificate presented by the server is expired or invalid.
Another possibility is that you are trying to connect to a port which does not support SSL/TLS.
It is also possible that the set of SSL/TLS protocols supported by the client and server do not match.
See https://github.com/jstedfast/MailKit/blob/master/FAQ.md#SslHandshakeException for possible solutions.
Let me know if there’s anything else you need from me.
It looks like that last log may have been cutoff. Would you be able to check if there was more to it?
"Exception calling “An error occurred while attempting to establish an SSL or TLS connection.
This usually means that the SSL certificate presented by the server is not trusted by the system for one or more of the following reasons:”
530: Must issue a STARTTLS command first MailKit.Security.AuthenticationException at MailKit.Net.Smtp.SmtpClient.AuthenticateAsync at MailKit.Net.Smtp.SmtpClient.Authenticate(Encoding encoding, ICredentials credentials, CancellationToken cancellationToken) at Octopus.Core.Util.SmtpClient.SmtpClientWrapper.CreateSmtpClient(SmtpConfiguration configuration, CancellationToken token) in SmtpClientWrapper.cs:line 76 at Octopus.Core.Util.SmtpClient.SmtpClientWrapper.Send(SmtpConfiguration configuration, MailMessage message, CancellationToken token) in SmtpClientWrapper.cs:line 33 at Octopus.Server.Orchestration.ServerTasks.TestEmail.TestEmailTaskController.Execute() in TestEmailTaskController.cs:line 54 at Octopus.Server.Orchestration.ServerTasks.RunningTask.RunMainThread() in RunningTask.cs:line 104 --Inner Exception-- Must issue a STARTTLS command first MailKit.Net.Smtp.SmtpCommandException
We’ve managed to replicate the issue within our AWS environment.
The problem is that to validate the AWS SES certificate, your server needs to be able to reach out and retrieve an appropriate Certificate Revocation List during the STARTTLS protocol execution: https://en.wikipedia.org/wiki/Certificate_revocation_list
You can see in the below wireshark capture the DNS lookup for CRL retrieval - this is issued when attempting to send mail.
The addresses that are resolved for this CRL (13.35.149.*) will not be able to be resolved from your server, as it does not have egress to the internet.
Without being able to retrieve this list, the server certificate validation fails.
Given SES requires all email sending to be encrypted via TLS https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-connect.html and both modes of TLS will require checking the CA’s CRL for the provided certificate, you will need to provide outbound access for these CRL requests to complete from your server.
Note that this isn’t the only outbound communication Octopus does - certain steps, like our Azure steps, make calls to external services. These will also fail if your Octopus server is air-gapped from the internet.
