Linux

Let's Encrypt Kept Failing After a Server Move. The Culprit Was a Second A Record.

Two A records after a migration meant round robin, and Let's Encrypt's multi-perspective validation kept hitting the old server. Diagnosis and fix.

I moved my blog to a new server. New box, fresh panel, the app container running, reverse proxy configured. All that was missing was the DNS switch and a Let's Encrypt certificate. I asked the DNS provider to point the A record at the new IP.

They added the new A record. They did not remove the old one.

So the name now resolved to two IPs: the old server and the new one. Classic DNS round robin, completely unintended. And Let's Encrypt validation failed. Again. And again. Four attempts, before I stopped hammering it.

TL;DR: If ACME validation fails right after a migration, check for duplicate A/AAAA records at the authoritative nameserver. Let's Encrypt validates from multiple network perspectives; with two A records, some of them hit the old server, which doesn't have the challenge token, and the validation fails. Fix DNS, flush resolver caches, retry once. Stop retrying blindly, there's a failed-validation rate limit.

The diagnosis commands

Example values throughout: blog.example.com, old server 198.51.100.20, new server 203.0.113.10.

What does the authoritative nameserver say? Not your laptop's resolver, not the server's cache, the source of truth:

dig +short NS example.com
dig +short A blog.example.com @ns1.dns-provider.example

Two lines of output where you expect one is your answer:

203.0.113.10
198.51.100.20

Then check what the public resolvers have cached, because that's what the rest of the world sees:

for r in 1.1.1.1 8.8.8.8 9.9.9.9; do echo "== $r"; dig +short A blog.example.com @$r; done

Next, talk to each server directly, bypassing DNS entirely. curl --resolve pins a name to an IP for one request, with the correct SNI and Host header:

curl -sI --resolve blog.example.com:443:203.0.113.10 https://blog.example.com/ | head -5
curl -sI --resolve blog.example.com:443:198.51.100.20 https://blog.example.com/ | head -5
curl -sI --resolve blog.example.com:80:203.0.113.10  http://blog.example.com/.well-known/acme-challenge/test

And look at what certificate each one actually delivers:

openssl s_client -connect 203.0.113.10:443 -servername blog.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

-servername sends SNI. Leave it out and you're testing the server's default certificate, which is not what a browser would get.

What was actually going on

Two separate problems that made each other worse.

Problem 1: validation hits the wrong box. For HTTP-01, Let's Encrypt fetches http://blog.example.com/.well-known/acme-challenge/<token>. It does this not from one place but from several network perspectives, and enough of them have to succeed. Each perspective resolves the name itself. With two A records in the answer, some of those requests go to the old server. The old server has never heard of that token, answers with a 404 or its own app, and the validation fails. With round robin, the odds that every perspective lands on the right IP are bad, and you can't influence them.

Multi-perspective validation exists to stop attackers who can hijack routing from one network location. It also makes "half-migrated DNS" fail reliably instead of randomly, which, honestly, is the better outcome.

Problem 2: no certificate, no HTTPS vhost. The new server runs KeyHelp as the panel. The panel only writes the HTTPS vhost (including my reverse proxy directives to the app container) once a certificate exists. No cert, no HTTPS vhost. So the new box answered for the blog with a 302 or a 404 from the default host, and on HTTPS it presented the server's own host name certificate instead of one for the blog.

From a Windows machine, curl against the new IP then fails with:

schannel: SEC_E_WRONG_PRINCIPAL (0x80090322) - The target principal name is incorrect.

and openssl s_client shows subject=CN = <the server's hostname>. That looks like a broken TLS config. It's not. It's the panel's fallback, because there's no domain cert yet. Which there can't be, because of problem 1.

Meanwhile, depending on which A record a visitor's resolver handed out, they got the old server with the old version of the blog, or the new one with a 302 and the wrong cert.

Gotcha: stop hammering Let's Encrypt

I made four attempts before I stopped. Don't push it further than that. Let's Encrypt has a rate limit on failed validations, in the region of 5 per hour per account and hostname (check their rate-limit docs for the current numbers, they've been reworked over time). Hit it and you're locked out for that name for a while, even after you fix the real problem.

Rule of thumb: one failure, look at the error. Two failures with the same error, stop and diagnose with the commands above. Retrying is not debugging.

If you need to test, use the staging environment (--staging with certbot, or the panel's equivalent); it has much higher limits.

The fix

1. Remove the old A record. In my case that meant getting the DNS provider's support to delete the record they'd left behind. Then verify at the authoritative server, not anywhere else:

dig +short A blog.example.com @ns1.dns-provider.example
# exactly one line: 203.0.113.10

Check AAAA too. A stale IPv6 record pointing at the old box causes exactly the same mess, and Let's Encrypt prefers IPv6 when it's there.

2. Flush caches on the server. The panel and the ACME client on the server resolve the name too, and systemd-resolved may still hold the old answer:

resolvectl flush-caches
resolvectl query blog.example.com

3. Temporary /etc/hosts entry for the panel's local check. KeyHelp checks locally whether the domain resolves to the server before it goes ahead. While public caches were still catching up, I pinned the name on the server itself:

echo "203.0.113.10 blog.example.com" >> /etc/hosts

This only affects lookups on that server. It does nothing for Let's Encrypt, whose validators resolve via public DNS. That's why step 1 is non-negotiable.

4. Re-trigger the certificate. I re-saved the domain's SSL/security settings in the KeyHelp UI; KeyHelp also has a CLI maintenance run for SSL (php bin/keyhelp.php run ssl-maintenance from the panel's install directory). With DNS clean, the certificate was there within about a minute, and with it the HTTPS vhost and the proxy rules. https:// returned 200 from the app, http:// redirected to HTTPS.

5. Remove the /etc/hosts entry.

sed -i '/blog\.example\.com/d' /etc/hosts

Do this right away. A forgotten hosts entry is a time bomb: the day you move again, this server will keep talking to itself and you'll spend an evening wondering why.

6. Stop the old instance properly. The app on the old server ran in Docker with a restart policy. docker stop alone isn't enough; on the next reboot or daemon restart it comes back, and if anything still points at that IP, it's serving stale content:

docker update --restart=no ghost
docker compose stop

Stopped, not deleted.

How to avoid this next time

Lower the TTL before a migration, a day or so ahead. Mine was sitting at 86400; lowering the TTL first turns a day of mixed caches into minutes.
Change the record, don't add one. When asking a provider's support to do it, say explicitly "replace", and verify with dig @authoritative afterwards. Don't trust the ticket reply.
Test the new server before the switch with curl --resolve. You can check the whole app, minus the real cert, without touching DNS.
Pre-issue the cert via DNS-01 if your DNS provider has an API. DNS-01 doesn't care which server the A record points to, so you can have the certificate ready before the cutover. With a provider that needs a support ticket per change, that's not an option, which is how I ended up here.

If you're coming from a dead Plesk renewal, I wrote about moving to standalone certbot, and the panel I moved to is covered in KeyHelp as a free Plesk alternative. For the reverse proxy part, see Apache in front of Docker apps or the nginx reverse proxy setup. If you'd rather have a web server that does ACME itself, Caddy vs nginx is the comparison.