Troubleshooting TLS connection problems -- Sometimes the problem is not what you think it is
Daniel Nashed – 27 September 2024 20:13:31
I got a help request today about configuring a SMTP connection for a relay host.
The admin configured an outgoing relay host. The connection was supposed to be SMTP over TLS on port 587. But it didn't work.
He got the following error:
SMTP: SSL bad peer certificate. Connection refused.
Then the admin enabled SSL debugging and got the following error message, but had no idea what it could mean:
[032279:000013-00007FC3EDA63640] 09/26/2024 11:35:41.57 SSLReadRecord> Rejecting connection - record contentType not in range for SSLv3 or TLS
[032279:000013-00007FC3EDA63640] 09/26/2024 11:35:41.57 SSLReadRecord> First 4 bytes of SSL/TLS record: 0x34 0x32 0x31 0x20
[032279:000013-00007FC3EDA63640] 09/26/2024 11:35:41.57 SSL_Handshake> After handshake state = HandshakeServerHello (5); Status = -6974
The most common root cause for this problem is that the other side is not speaking SSL/TLS at all.
In this particular case the other side was speaking STARTTLS instead of SMTP over TLS.
The start of the session is unencrypted until the STARTTLS command is issued and the handshake is performed.
I had a previous customer issue with a LDAP connection, which was expecting secure LDAP but the remote side was unencrypted.
Solution
In this case the solution was to switch the Domino server to STARTTLS (Negotiated TLS) in the server document for outgoing SMTP.
The LDAP case was solved by choosing the right encrypted port.
Conclusion
On first glance this shows up like a SSL/TLS connectivity problem. But knowing details about the SSL handshake helps you to get an idea about the root cause.
It is very unlikely a TLS version out of range could occur. So it is very likely the other side is not a SSL/TLS connection at all.
Now this blog post might help others to google the solution and we can prevent some support calls.
- Comments [1]
![[HCL Domino]](../lotus-domino.gif){kind=small}
![[Domino on Linux]](../domino-linux.gif){kind=small}
![[Nash!Com]](../nashcom.gif){kind=small}
![[Daniel Nashed]](../daniel-nsh.gif){kind=small}