My Hosting Panel's Firewall Kept Wiping Docker's iptables Chains
KeyHelp loads its firewall with iptables-restore and flushes Docker's chains. How to spot it, the quick fix, and the options to make it stick.
I put Docker CE on a server that also runs the KeyHelp hosting panel. A handful of app stacks run as containers, bound to 127.0.0.1, with the panel's Apache in front as reverse proxy. Worked fine — until I changed something in the panel's firewall settings, and container networking was simply gone.
TL;DR: The panel applies its firewall with iptables-restore, feeding it a complete *filter table. Without --noflush, iptables-restore replaces the whole table — including the chains Docker created (DOCKER, DOCKER-USER and friends). Docker doesn't notice. Quick fix:
systemctl restart docker
That makes dockerd recreate all of its rules. The rest of this post is how to confirm that's your problem and what your options are to stop it happening again.
The setup
- Ubuntu with the KeyHelp panel (see KeyHelp as a free Plesk alternative) — Apache, mail, the panel's own iptables firewall with an INPUT policy of DROP
- Docker CE + compose installed afterwards (standard install)
- every stack in
/opt/<name>, ports published only on localhost:
services:
app:
image: example/app
ports:
- "127.0.0.1:8091:80"
- Apache in the panel proxies the domain to
http://127.0.0.1:8091/(details in Apache as reverse proxy for Docker apps)
Binding to 127.0.0.1 is non-negotiable here. Docker's published ports bypass a lot of host firewall logic, and on a box where a panel manages the firewall you really don't want a container accidentally listening on the public interface. Localhost-only means the only way in is through Apache.
Why it breaks
Docker doesn't have its own firewall. It writes iptables rules into the host's tables at startup and whenever containers or networks change: its own chains in the filter table (DOCKER, DOCKER-USER, isolation chains; exact names vary a bit between Docker versions), jump rules from FORWARD into them, and NAT rules for port publishing and outbound masquerading.
iptables-restore works on whole tables. When you feed it something like:
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT
...
COMMIT
…then for the filter table it flushes every chain and deletes every user-defined chain that isn't in the input. Docker's chains aren't in the panel's ruleset, so they're gone. The FORWARD jumps to them are gone. Docker still thinks everything is set up, because nothing told it otherwise.
Whether the nat table gets hit too depends on whether the panel's ruleset contains a *nat section — iptables-restore only replaces the tables it's given. Check both rather than assume.
The --noflush (-n) flag would change this: rules are added to the existing tables instead of replacing them. The panel doesn't use it, and that's a defensible choice from its side — it wants to own the firewall state completely. It just doesn't know Docker exists.
Diagnosing it
Do this right after a panel firewall change, or whenever proxied apps stop answering for no obvious reason.
Are Docker's chains still there?
iptables -n -L DOCKER
iptables -n -L DOCKER-USER
iptables -t nat -n -L DOCKER
On a healthy box those list chains (possibly with few rules). After the wipe you get:
iptables: No chain/target/match by that name.
Does FORWARD still jump into Docker's chains?
iptables -n -L FORWARD --line-numbers | head -20
Docker's network config itself is fine — that's the confusing part:
docker network ls
docker network inspect bridge | grep -A3 IPAM
docker ps
Containers show as running, networks exist, IPs are assigned. Everything inside Docker looks healthy, because the damage is outside it.
Then test the actual path Apache uses:
curl -sI http://127.0.0.1:8091/
and from inside a container, outbound:
docker exec -it <container> sh -c 'wget -qO- -T5 https://example.com >/dev/null && echo out-ok'
If the chains are missing, you've found it. A full dump for comparison before/after is worth keeping:
iptables-save > /root/iptables-$(date +%F-%H%M).rules
The quick fix
systemctl restart docker
dockerd rebuilds its chains, jump rules and NAT rules on start. Re-run the checks above; the chains are back and the apps answer again.
Cost: restarting the daemon restarts your containers (unless you've configured live-restore), so every stack has a short outage. On my box that's fine. If you have something stateful that hates being bounced, plan for it.
The real problem: this only fixes it until the next panel firewall change. And the person making that change in the panel may not know Docker is on the box at all.
Making it stick: the options
I haven't settled on a permanent fix yet. Here's what's on the table, with the trade-offs as I see them.
1. Process: restart Docker after every firewall change
Zero engineering. Write it down next to the panel login: "changed firewall → systemctl restart docker". Works as long as there's one admin and that admin reads notes. That's the current state.
2. A watchdog timer
A systemd timer that runs every minute or so, checks whether DOCKER exists, and restarts Docker if not:
#!/bin/sh
# /usr/local/sbin/docker-chain-check.sh (sketch, NOT deployed)
iptables -n -L DOCKER >/dev/null 2>&1 || systemctl restart docker
# docker-chain-check.timer (sketch)
[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
I considered this and didn't set it up. Trade-offs:
- Up to one interval of downtime after every firewall change, plus the container restart.
- It hides the problem. In six months nobody remembers why containers occasionally restart by themselves.
- Restart loops. If something else breaks the chain check — say a future Docker version renames chains, or Docker runs with
iptables: false— you've built a machine that restarts Docker every minute forever. Any watchdog like this needs a rate limit and an alert, not just a restart. - It's one more piece of persistent self-healing on a production box, which deserves a deliberate decision rather than a quick hack.
If you go this way, see systemd timers instead of cron for the unit pattern, and make the script log loudly when it acts.
3. Hook the firewall reload directly
Cleaner than polling: react to the actual event. If you can find which file the panel writes before calling iptables-restore, a systemd .path unit with PathChanged= on that file can trigger a oneshot service that restarts Docker. Instant, no polling, no blind restarts. The catch is that it depends on panel internals that can change with any update. I haven't pinned down that file on my box, so I can't give you a tested path.
4. --noflush, if the panel lets you
If your panel has a setting or hook to load rules with iptables-restore --noflush, that removes the root cause. KeyHelp doesn't expose that as far as I've found. Some other panels and firewall tools do, so check yours.
5. Take iptables away from Docker
Set "iptables": false in /etc/docker/daemon.json and manage NAT/forwarding yourself in the panel's ruleset. Then nothing gets wiped because Docker never wrote anything. In practice this means you now maintain Docker's networking by hand — outbound masquerading, inter-container traffic, every network you add. I wouldn't do this on a box that's supposed to stay low-maintenance.
What I'd recommend
For a single-admin box: option 1 plus a monitor that actually tells you when a proxied app stops answering. An HTTP check against each app domain from outside catches this and every other way the stack can fail — Uptime Kuma does that in ten minutes.
If more people touch the panel, I'd move to option 3 if the panel internals cooperate, or option 2 with a rate limit and an alert.
And as a general rule: when two tools both think they own iptables, one of them will lose, silently. Find out which before production finds out for you. If you run plain ufw instead of a panel, the same question applies — ufw firewall rules covers the basics.
