Linux

Postfix SNI: "map lookup problem" and TLS Alert 80 — the texthash Fix

Per-domain mail hostnames via Postfix SNI: why smtpd hit TLS alert 80 though postmap worked, the ACL fix for Let's Encrypt keys, and test commands.

Lukas

Lukas

— 6 min read
Share
Postfix SNI: "map lookup problem" and TLS Alert 80 — the texthash Fix

I wanted every mail domain on a server to have its own mail.<domain> hostname with a proper Let's Encrypt certificate, so a client's Outlook connects to mail.example.com and not to some server name it has never heard of. Postfix and Dovecot both support that via SNI. Dovecot just worked. Postfix threw map lookup problem and killed every TLS handshake with alert 80, while every manual test I ran said the map was fine.

TL;DR: Two separate problems. First, Postfix couldn't read the Let's Encrypt keys (root-only directory) — fixed with an ACL plus a default ACL so renewals inherit it. Second, the SNI map was configured as hash:, and smtpd failed on it even though postmap -q as the postfix user returned the right result. Switching to texthash: fixed it:

postconf -e 'tls_server_sni_maps=texthash:/etc/postfix/postfix-sni.conf'
postfix reload

The setup here is a KeyHelp panel (see KeyHelp as a Plesk alternative), but the Postfix part applies to any box that does SNI with Postfix.

Step 0: getting the SNI config generated at all

In KeyHelp the mail hostname isn't a checkbox on the main domain. What worked for me:

  1. Create mail.example.com as its own domain in the panel (owned by the same client).
  2. On that domain enable email domain and Let's Encrypt.
  3. Trigger the panel's SSL maintenance run from its application directory:
php bin/keyhelp.php run ssl-maintenance

Only after that run did the panel write the two SNI files:

/etc/postfix/postfix-sni.conf
/etc/dovecot/dovecot-sni.conf

Before it, there's nothing to debug — Postfix just presents the server's default certificate. Don't try to call the panel's cron job scripts for SSL directly; they aren't built to run standalone and the one I tried crashed.

The Postfix file follows the usual SNI map format — roughly: hostname on the left, key and certificate chain files on the right (paths shortened):

mail.example.com /etc/ssl/keyhelp/letsencrypt/.../privkey.pem /etc/ssl/keyhelp/letsencrypt/.../fullchain.pem

Trap 1: Postfix can't read the keys

The Let's Encrypt files under /etc/ssl/keyhelp/letsencrypt/ are in directories with mode 700, owned by root. Dovecot reads its certificates as root before dropping privileges, so it doesn't care. Postfix's smtpd runs as the postfix user and does care.

Fix with ACLs instead of loosening the modes for everyone:

# existing files and dirs: let postfix read files and traverse dirs
setfacl -R -m u:postfix:rX /etc/ssl/keyhelp/letsencrypt

# default ACL on the dirs, so files created by future renewals inherit it
setfacl -R -d -m u:postfix:rX /etc/ssl/keyhelp/letsencrypt

The second line is the important one. Without the default ACL everything works today and breaks at the first renewal, when new files appear without your ACL — roughly 60 days later, when nobody remembers this post.

Check it:

getfacl /etc/ssl/keyhelp/letsencrypt | grep postfix
sudo -u postfix cat /etc/ssl/keyhelp/letsencrypt/<dir>/fullchain.pem > /dev/null && echo ok

Be aware what you've done: the postfix user can now read all private keys in there, not just mail ones. For a box where Postfix is already the most exposed daemon, I can live with it. If you can't, give Postfix copies of only the mail certs via a renewal hook instead.

Trap 2: hash: fails in smtpd, works everywhere else

With permissions fixed, I expected it to work. It didn't. Connecting with SNI got me a handshake failure, and the log said:

grep -i "sni\|map lookup" /var/log/mail.log | tail

... map lookup problem ...

On the client side, OpenSSL reported TLS alert 80 — internal_error. That's the server saying "something broke on my end", not a certificate validation problem on yours.

The panel had configured:

tls_server_sni_maps = hash:/etc/postfix/postfix-sni.conf

So I did the obvious thing and queried the map exactly as Postfix would, as the right user:

sudo -u postfix postmap -q mail.example.com hash:/etc/postfix/postfix-sni.conf

Correct answer. Readable, right key, right files. And smtpd still failed.

I'll be honest: I didn't fully nail down why smtpd chokes on the indexed map. One thing worth knowing if you hit this: for tls_server_sni_maps with an indexed type like hash:, Postfix expects the table to have been built with postmap -F, which loads the actual key and certificate contents into the database instead of just the file names. A table built with a plain postmap will still answer postmap -q happily — with file names — and that would explain exactly this "works by hand, fails in smtpd" pattern. I didn't verify that's what the panel did, so treat it as a strong suspect, not the diagnosis.

What I did verify is the fix.

The fix: texthash

postconf -e 'tls_server_sni_maps=texthash:/etc/postfix/postfix-sni.conf'
postfix reload

texthash: reads the plain text file directly, no postmap step, no .db file. That has a second advantage on a panel-managed box: when the panel regenerates postfix-sni.conf (new domain, renewed cert), there's no separate database that can go stale. Postfix reads the text file when a process starts, so a postfix reload after changes makes it immediate.

I'd have tried lmdb: as well, but it wasn't available in the Postfix build on that box:

postconf -m        # lists supported map types

After the switch, SNI handshakes went through immediately.

Gotcha: panel updates can undo this

The panel owns main.cf. A panel update may well rewrite tls_server_sni_maps back to hash:, and you'll be back to alert 80 with no idea why mail clients suddenly complain. After every panel update:

postconf tls_server_sni_maps

If it says hash:, set it back. I'd love a cleaner way to pin it, but I haven't found a supported override for this setting yet.

Dovecot needs none of this

Worth saying explicitly so you don't go fixing things that aren't broken: Dovecot reads dovecot-sni.conf and the certs as root at startup. IMAPS with the right per-domain certificate was working before I touched anything on the Postfix side. If IMAP is fine and SMTP isn't, the problem is Postfix-specific — permissions or the map type.

Testing it properly

Always test with an explicit -servername. Without it, you're testing the default certificate and learning nothing about SNI.

SMTP with STARTTLS on port 25 (server-to-server) and 587 (submission):

openssl s_client -connect mail.example.com:25 -starttls smtp \
  -servername mail.example.com -brief </dev/null

openssl s_client -connect mail.example.com:587 -starttls smtp \
  -servername mail.example.com -brief </dev/null

Implicit TLS on 465 if you offer it:

openssl s_client -connect mail.example.com:465 \
  -servername mail.example.com -brief </dev/null

IMAPS for comparison:

openssl s_client -connect mail.example.com:993 \
  -servername mail.example.com -brief </dev/null

What you want in the output: the peer certificate's subject/SAN matching mail.example.com and a valid chain. To see the certificate details explicitly:

openssl s_client -connect mail.example.com:587 -starttls smtp \
  -servername mail.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -ext subjectAltName -enddate

If you get the server's hostname certificate instead of the mail hostname: SNI isn't being used — check tls_server_sni_maps is set and the domain is in the file. If you get alert 80: look in the mail log, it's the server side. If the connection works but the client still warns: DNS — make sure mail.example.com actually points at this server. If you're moving domains around, lower the TTL first.

Checklist

  • mail.<domain> exists as its own domain, mail + Let's Encrypt on
  • SSL maintenance has run, both SNI files exist
  • postfix has rX ACL on the LE directory, including a default ACL
  • tls_server_sni_maps is texthash:, re-checked after every panel update
  • tested with openssl s_client -servername on every port you offer

If you came here from an expired-panel situation where certificates stopped renewing entirely, this post on Let's Encrypt after Plesk is the other half of the story. And once the mail hostnames work, do the boring part too: SPF, DKIM and DMARC, or the nicest certificate won't keep you out of spam folders.

Related posts

Read more