The default advice for monitoring a remote box is "open 9100 and firewall it to your Prometheus IP". That works right up until you re-image the Prometheus box, the IP changes, and you're either locked out of your own metrics or — worse — you left it open. node_exporter on the public internet is a free inventory of your kernel version, mounted disks and uptime, handed to anyone who asks.

So I don't expose it. My game server and my monitoring box sit in a WireGuard mesh, node_exporter binds to the tunnel address only, and Prometheus scrapes it over 10.77.0.x. Nothing listens on the public interface. If you're not on the VPN, the metrics endpoint does not exist for you.

TL;DR:

# on the game server — bind the exporter to the WG address, nothing else
ExecStart=/usr/local/bin/node_exporter --web.listen-address=10.77.0.2:9100

# on the monitoring box — scrape the tunnel IP
static_configs:
  - targets: ['10.77.0.2:9100']

No firewall rule needed for 9100 at all, because it was never reachable in the first place.

The tunnel

Two peers: monitoring box (10.77.0.1) and game server (10.77.0.2). If you've never set up WireGuard, start at /wireguard-vpn-server-ubuntu/ — this is that, with a tiny subnet and no default-route trickery.

Game server side, /etc/wireguard/wg0.conf:

[Interface]
Address = 10.77.0.2/24
PrivateKey = <gameserver-private>
# no ListenPort needed — it dials out

[Peer]
PublicKey = <monitoring-public>
Endpoint = monitor.example.com:51820
AllowedIPs = 10.77.0.0/24
PersistentKeepalive = 25

PersistentKeepalive = 25 is not optional here. The game server is very likely behind some NAT or stateful firewall, and without keepalive the tunnel goes quiet, the mapping expires, and Prometheus starts scraping a black hole. Scrapes fail, you get a page, you SSH in and everything's fine — it's just the tunnel that fell asleep. 25 seconds keeps it awake.

AllowedIPs = 10.77.0.0/24 and nothing else — this is a management tunnel, not a VPN. You do not want game traffic routing through it.

Bring it up and make it stick:

systemctl enable --now wg-quick@wg0
wg show    # handshake should be seconds old, not minutes

Binding the exporter to the tunnel

Install node_exporter, then override the listen address. This is the whole security model, so get it right:

# /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus node_exporter
After=network-online.target wg-quick@wg0.service
Wants=network-online.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=10.77.0.2:9100 \
  --collector.systemd \
  --collector.processes
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Two things worth calling out.

After=wg-quick@wg0.service — bind to an address that doesn't exist yet and the process exits with "cannot assign requested address". Then it restarts, fails again, and after a few tries systemd gives up entirely. Order it after the tunnel.

--collector.systemd gives you node_systemd_unit_state, which is how you alert on the game server process itself being down, not just the box being up. That distinction matters — a machine at 2% CPU with a dead game server looks perfectly healthy on a CPU graph.

Verify it's actually private:

ss -tlnp | grep 9100
# LISTEN 0 4096 10.77.0.2:9100 ...   ← good
# LISTEN 0 4096 *:9100 ...           ← bad, fix the flag

Prometheus side

Standard scrape config, tunnel IPs as targets:

scrape_configs:
  - job_name: 'gameserver'
    scrape_interval: 15s
    static_configs:
      - targets: ['10.77.0.2:9100']
        labels:
          host: ash
          role: gameserver

If Prometheus runs in Docker on the monitoring box (mine does — see /prometheus-grafana-monitoring-stack-docker/ for that stack), the container needs to reach 10.77.0.0/24. With the default bridge network it can, because the host routes it — but only if the container isn't in its own netns with a restrictive route. If scrapes fail from inside the container while curl 10.77.0.2:9100/metrics works fine on the host, that's your problem: run the Prometheus container with network_mode: host, or add a route. Host networking is the boring, working answer for a single-box monitoring stack.

The alerts that actually matter

A game server has two failure modes that a generic "CPU > 80%" rule will never catch.

The process is gone but the box is fine. With the systemd collector:

- alert: GameServerDown
  expr: node_systemd_unit_state{name="hytale.service",state="active"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Game server unit is not active on {{ $labels.host }}"

for: 1m so a restart doesn't page you. A crash-loop will still trip it, because it never stays active for a full minute.

Traffic that looks like an attack. Game servers get UDP-flooded, and the tell is packets-per-second, not bandwidth — a flood of tiny packets can wreck a box while barely moving the bytes graph.

- alert: PacketFlood
  expr: rate(node_network_receive_packets_total{device="eth0"}[2m]) > 50000
  for: 2m
  annotations:
    summary: "{{ $value | humanize }} pps inbound on {{ $labels.host }}"

- alert: BandwidthSpike
  expr: rate(node_network_receive_bytes_total{device="eth0"}[2m]) * 8 > 500e6
  for: 2m
  annotations:
    summary: "{{ $value | humanize }}bps inbound on {{ $labels.host }}"

Alert on both, separately. Numbers are per-box: watch your own graphs for a week, take the busiest legitimate hour, and set the threshold well above it. My first attempt paged me every time a popular streamer's audience joined at once. That's not an incident, that's a good day.

Route them somewhere you'll actually see — Discord webhook via Alertmanager, since that's where the players are already yelling. The alert reaching you two minutes before the first "server lagging???" message is the whole point.

The Grafana panels I keep

I've deleted more panels than I've kept. The four that earn their space:

Panel Query Why
Game server up node_systemd_unit_state{name="hytale.service",state="active"} The only binary that matters
Inbound pps rate(node_network_receive_packets_total[2m]) Attack tell, before latency shows it
CPU per core 1 - rate(node_cpu_seconds_total{mode="idle"}[2m]) Game loops are single-threaded; the average hides a pegged core
Memory + swap node_memory_MemAvailable_bytes Swapping = stutter, long before OOM

Per-core CPU, not aggregate. A game server pinning one core at 100% while the other seven idle shows up as "12% CPU" on the average — and your players are the ones telling you it's lagging, not your dashboard.

What this doesn't cover

node_exporter sees the machine, not the game. It doesn't know player counts, tick rate, or that the world save is corrupt. For that you need something reading the game's own logs or query port — I run a small log-watching bot alongside this for exactly that reason. The two layers are complementary: Prometheus tells you the box is sick, log-watching tells you the game is.

And the tunnel itself is now a dependency. If WireGuard dies, scrapes fail and you get a page that says "gameserver down" when the game server is perfectly fine. Alert on it explicitly so you can tell the two apart:

- alert: TunnelDown
  expr: up{job="gameserver"} == 0
  for: 3m
  annotations:
    summary: "Cannot scrape {{ $labels.host }} — WireGuard tunnel or host down"

Small price. Nothing is listening on a public port, and that's a trade I'll take every time.

Related posts