Docker perimeter hardening — the DOCKER-USER chain
Read this before publishing any container port. Docker's DNAT in the NAT table runs before
ufw's INPUT chain, so a container published as 0.0.0.0:<port> is reachable straight from the
WAN even when ufw has no matching allow rule. DOCKER-USER is the only hook that can plug it,
and to plug reliably it must end in -j DROP.
★ Incident — 2026-11-17: the safeguard was found GONE
State on the host: the chain had been reduced to a bare
```
-N DOCKER-USER
-A DOCKER-USER -j RETURN # i.e. "do nothing"
```
The DROP, the conntrack accept and the wg0 accept were all missing. Practical consequence:
with the DROP gone everything falls through to DOCKER-FORWARD, which ends in its own
per-bridge accepts, so containers stayed reachable — including from the WAN on any port
published 0.0.0.0. It was not blowing up, so nothing alarmed.
Suspected cause: Docker 29.1.3 (build 29.1.3-0ubuntu3~22.04.2). Docker 29 replaced the
legacy DOCKER-ISOLATION-STAGE-1/2 chains with nft-native rules of the shape
```
table ip raw { chain PREROUTING {
-A PREROUTING -d 172.19.0.6/32 ! -i br-b3b3e833d73c -j DROP # one per container IP
...
} }
```
— and in the process rewrote DOCKER-USER back to a bare RETURN. DOCKER-ISOLATION-STAGE-1
and -STAGE-2 are empty/absent on this host; the raw PREROUTING DROPs are what isolates
bridges now. Not reproduced from scratch, hence "suspected": Docker was never observed mid-rewrite.
The other half — raw PREROUTING breaks cross-bridge container traffic
Those same Docker 29 raw rules carry a real cost that took a while to find:
- They are ipv4-raw-PREROUTING rules.
raw is processed before conntrack, so they see
every packet including ones belonging to established flows — and packets arriving on the wrong
-i interface match ! -i br-X and are dropped, so the flow never establishes.
- The
-i restriction means it only bites for a direct container→container IP hop across
different bridges. Same-bridge traffic never traverses PREROUTING. Anything via the host IP
(SNAT to the host, arrives on the right bridge) is unaffected.
- Symptom:
abs (172.21.0.2) -> postgres (172.19.0.6:5432) = TCP SYN retransmit, TIMEOUT,
0 packets on the destination bridge in tcpdump. Live counter 1620 hits on the postgres rule.
proxy_arp is 0 on every bridge, so cross-bridge packets terminate at the host either way —
you always pay a host round-trip, but it works with the raw rule present.
Diagnosis shortcut that avoids the whole detour: DOCKER-USER was showing pkts=0 on every
rule while it sat above DOCKER-FORWARD in the FORWARD chain — the counters were simply being
read wrong (see below). Tracing the four FORWARD hops in order is what localised it to raw.
Applying / re-applying the safeguard
/usr/local/sbin/docker-user-hardening.sh — idempotent, flushes and rebuilds from the live
interface list. Run by docker-user-hardening.service (oneshot, After/Requires=docker.service,
PartOf=docker.service, enabled) so it re-applies after every Docker restart, and persisted into
/etc/iptables/rules.v4 for boot via netfilter-persistent (enabled).
Rule order matters: conntrack accept → wg0/10.88.0.0/24 accept → per-bridge -i/-o accept
(for both docker0 and every br-*) → final -j DROP.
```bash
sudo /usr/local/sbin/docker-user-hardening.sh
sudo iptables -S DOCKER-USER # must end in -j DROP
sudo systemctl restart docker && sudo iptables -S DOCKER-USER | wc -l # must still be ~19
```
Verification (do all four — a chain that merely looks right is the failure mode)
```bash
1. chain shape — last rule MUST be DROP
sudo iptables -S DOCKER-USER | tail -1
2. WAN reachability on the PUBLIC IP — all of these must print "closed"
for p in 3333 5678 9443 5432 6379 8888 9120 8096 8098 13378 11434; do
timeout 2 bash -c "</dev/tcp/45.61.176.215/$p" 2>/dev/null && echo "OPEN $p (bad)" || echo "closed $p"
done
exactly four may be open on the WAN: 22, 80, 443, 51820/udp
3. container egress must survive
docker exec hashicorp-vault sh -c 'wget -qO- -T3 http://1.1.1.1 >/dev/null && echo EGRESS_OK'
4. WG-only endpoints must answer
curl -s -o /dev/null -m3 -w '%{http_code}\n' http://10.88.0.1:8091/
```
Two traps that cost real time here
nc and /dev/tcp are absent in most containers. hashicorp-vault and yantrikdb have
neither, so every probe there returns a false FAIL. hermes-n8n has node — use it as the
canary instead:
```bash
docker exec hermes-n8n node -e "const n=require('net');const s=n.connect(5432,'172.19.0.6');
s.on('connect',()=>{console.log('OK');process.exit(0)});
s.on('error',()=>{console.log('ERR');process.exit(0)});
setTimeout(()=>{console.log('TIMEOUT');process.exit(0)},4000)"
```
Reading chain counters. iptables -L -v puts the packet count in column 1 as pkts. An
awk '$1>0' filter over it silently matches nothing (the column is literally the string pkts),
which manufactures a convincing "the chain is never traversed" conclusion. Use
awk 'NR>2 && $2>0', or read sudo nft list chain ip filter DOCKER-USER.
Note on br_netfilter
br_netfilter is not loaded (/proc/sys/net/bridge/ absent; every bridge reports
nf_call_iptables=0), so forwarded frames are filtered only in ip filter, never in `bridge
filter`. Loading it adds a second filter point and changes jump timings — do not load it while
debugging this, or you will chase a moving target.
★ 2026-11-17 (later) — redesigned after a Claude Code review
Claude Code attacked the original br-* allow-list design and broke it:
1. The old loop was a one-shot snapshot of /sys/class/net. Any Docker network created
after the unit ran had no matching ACCEPT, fell through to the final DROP, and blackholed —
both its egress and intra-network traffic. Reproduced live: created a throwaway network
after the unit ran, the new container had working egress and DNS only because the redesign
landed first.
2. -s 10.88.0.0/24 -j ACCEPT was redundant and spoofable — rp_filter=2 (loose) makes
10.88.0.0/24 routable via wg0, so a spoofed source arriving on the WAN interface passed.
Dropped; -i wg0 alone is correct.
3. iptables -F ... || exit 0 was a silent no-op if the chain was missing — exactly the
failure mode being guarded against. Now the chain is checked and the script exit 1s.
4. Flush-then-append was non-atomic (a ms window with an empty chain). Now iptables-restore --noflush.
The current design is minimal and future-proof — block only what needs blocking, so new bridges
work with no maintenance:
```
-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-USER -i wg0 -j ACCEPT
-A DOCKER-USER -i $WAN -j DROP # $WAN resolved from the default route, not hardcoded
-A DOCKER-USER -j RETURN
```
Anything arriving on docker0/a bridge/wg0/lo is already covered by Docker's own DOCKER-FORWARD
accepts, so no interface enumeration is needed. The script asserts the DROP is present and
exits non-zero otherwise.
Recurrence was also silent — healthcheck.sh checked ufw but nothing about iptables. It now
asserts the DOCKER-USER DROP and alerts otherwise. Proved by negative test: forcing the chain back
to a bare RETURN fires the alert; restoring clears it.
Rollback for both files: *.bak-20261117. The unit was restarted and re-verified after the swap.
Related: Network and access, Nginx vhosts, Security hardening