[Semibug] Self-resolved: answer to my technical question this month

[Josh Grosse]

The TL;DR - I figured out my error.  As expected, it was
in my test configuration files.

For anyone actually interested:

I'd asked if anyone had an inkling why my perfectly valid
self-signed x509 certificates were failing in a lab that I
had prepared for a migration to the new OpenSMTPd release.
It's an isolated test environment to prevent any test 
results from getting away from me and annoying the Internet.

I set up the test environment because much of the mail server
logical flow and the provisioning syntax has changed, and
I have two mail servers with complex flows between them, 
using PKI and authentication to eliminate several forms
of attack on the main internet-facing MTA, and vxlan(4)
to tunnel SMTP traffic between these servers.

But my test environment was not functioning properly, 
because of certificate failures.  My production 
environment runs fine, using a public CA and their
provided certificates.

For this test environment, I'd followed the starttls(8)
man page, studied and re-studied the smtpd.conf(5) man
page, and was at a loss.  So I took the opportunity this
week to ask at the meeting if anyone had advice.

My own thought was to reach out to the lead developer
Gilles Chehade, who has been kind enough to offer
support to the user community on provisioning problems.
But I was hoping to avoid bothering him.  Again. :)

This evening, one of the *many* tests I ran was without
authentication.  And suddenly my certificates were 
acceptable between my two test MTAs. .

Authentication is a form of password protection, and can
be used -- when the session is encrypted -- of ensuring 
that the two ends of the session are from known and 
trusted mail systems.  The authentication in SMTP happens
in plain text, so OpenSMTPd rightly only permits it when
the SMTP session occurs in encrypted form.

With OpenSMTPd, this encryption requires TLS or SMTPS.
You can't configure it without encryption.  And encryption
requires an X.509 certificate.  And in my enclosed little
lab (virtual machines on a laptop), self-signed seemed
like it should have been a perfect solution.

What I discovered was that if one is using TLS or SMTPS
without authentication ... then a self-signed certificate
is perfectly OK.  But, if you enable authentication, then
you must have a cert signed by a valid CA.

This restriction is hidden in plain sight, in smtpd.conf(5),
but I missed it, as it just happened to be the last line
in a paragraph on authentication tables:

        The label corresponds to an entry in a credentials table,
        as documented in table(5).  It is used with the
        "smtp+tls" and "smtps" protocols for authentication.
        Server certificates for those protocols are verified by
        default.

My next tests will be to override the default with "tls no-verify"
and if succesful, I can complete my testing without having to
deal with the hassle of obtaining new certs just for my lab
machines, and then openg up my lab's network to the Internet
for verifying the certs.  I want to make sure my mail servers
are operating properly and not spewing traffic they shouldn't 
be.