Prometheus tells me the box is healthy. It does not tell me that the game server accepted a player, kicked them with a cryptic interaction error, and is now sitting there at 3% CPU, "up", serving nobody. Metrics see the machine. The log sees the game.

So there's a second thing running next to the metrics stack: a small Python process that tails the log, matches a handful of patterns, and posts to a Discord webhook. It's about a hundred lines. It has caught more real incidents than every dashboard I've built.

TL;DR — the whole idea:

for line in tail(LOG):
    for rule in RULES:
        if rule.pattern.search(line):
            post_to_discord(rule.format(line))

Everything below is the part that makes it survive contact with reality: not spamming, not dying, and not lying to you.

Why not just use the monitoring stack

Because the interesting failures aren't numeric. "Server is up but every join fails" has no metric. "World save threw an exception" has no metric. "Player X got kicked 40 times in a minute" has no metric. They all have a log line, and the log line usually names the problem in plain English.

You could ship the logs to Loki and alert on them there, and if you already run Loki, do that. If you don't, standing up a log-aggregation stack to notice a crash is a lot of moving parts for one job.

The watcher

#!/usr/bin/env python3
import re, time, json, urllib.request, os

WEBHOOK = os.environ["DISCORD_WEBHOOK"]
LOG     = "/opt/gameserver/logs/latest.log"
ROLE    = "<@&123456789012345678>"   # role to ping, or "" for none

RULES = [
    dict(name="crash",  level="fatal",
         pattern=re.compile(r"(FATAL|Exception in server tick|Watchdog)", re.I),
         ping=True),
    dict(name="freeze", level="fatal",
         pattern=re.compile(r"Can't keep up|Server thread stalled", re.I),
         ping=True),
    dict(name="kick",   level="warn",
         pattern=re.compile(r"kicked .* for (?!leaving)", re.I),
         ping=False),
]

def tail(path):
    while not os.path.exists(path):
        time.sleep(2)
    f = open(path, "r", errors="replace")
    f.seek(0, 2)                       # start at EOF — don't replay history
    inode = os.fstat(f.fileno()).st_ino
    while True:
        line = f.readline()
        if line:
            yield line.rstrip()
            continue
        time.sleep(0.5)
        try:                            # did logrotate swap the file under us?
            if os.stat(path).st_ino != inode:
                f.close()
                f = open(path, "r", errors="replace")
                inode = os.fstat(f.fileno()).st_ino
        except FileNotFoundError:
            pass

def post(rule, line):
    content = f"{ROLE} **{rule['name']}** on `ash`\n```{line[:1800]}```" \
              if rule["ping"] else f"**{rule['name']}**\n```{line[:1800]}```"
    req = urllib.request.Request(
        WEBHOOK,
        data=json.dumps({"content": content}).encode(),
        headers={"Content-Type": "application/json"},
    )
    try:
        urllib.request.urlopen(req, timeout=10)
    except Exception as e:
        print(f"webhook failed: {e}", flush=True)   # never crash on a failed post

last = {}
COOLDOWN = 300

for line in tail(LOG):
    for rule in RULES:
        if not rule["pattern"].search(line):
            continue
        now = time.time()
        if now - last.get(rule["name"], 0) < COOLDOWN:
            break                       # rate-limited, stay quiet
        last[rule["name"]] = now
        post(rule, line)
        break

That's it. Standard library only — no deps, no venv, no pip install breaking on a Debian upgrade.

The four things that make it usable

1. seek(0, 2) before the first read. Open a log and read from the top and you'll replay every crash from the last three months into Discord in about two seconds. Ask me how I know. Start at the end; you only care about what happens from now on.

2. Inode check for logrotate. Game servers rotate latest.log on every restart. Your file handle happily keeps pointing at the old, now-unlinked inode and you never see another line — the bot looks alive and tells you nothing, which is worse than crashing. Compare the inode, reopen when it changes. See /logrotate-linux-log-rotation/ if rotation itself is new to you.

3. Cooldown per rule. A crash loop writes the same FATAL a hundred times a minute. Without the 5-minute cooldown, Discord rate-limits you, the channel becomes unreadable, and people mute it — at which point the bot has negative value. One message per rule per five minutes is plenty; you don't need a hundred to know it's broken.

4. @-ping only on fatal. This is the rule I most want to hand to past-me. If everything pings, nothing pings. Kicks and warnings post silently; only crash and freeze ping the role. The moment you ping for something people can't or won't act on, they mute the channel and the fatal ping goes unseen too.

Run it under systemd

# /etc/systemd/system/hytale-watch.service
[Unit]
Description=Game server log watcher → Discord
After=network-online.target

[Service]
User=gamewatch
Environment=DISCORD_WEBHOOK=https://discord.com/api/webhooks/xxx/yyy
ExecStart=/usr/bin/python3 /opt/hytale-watch/watch.py
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

systemctl enable --now hytale-watch
journalctl -u hytale-watch -f

Restart=always because a bot that dies silently is worse than no bot — you'll believe the silence means "no incidents". If you want the full anatomy of a unit file, /systemd-service-unit-file/ covers it.

Keep the webhook URL out of the unit file if the repo is shared — anyone with that URL can post as your bot. EnvironmentFile=/etc/hytale-watch.env with chmod 600 is the better shape.

Tuning the rules is the actual work

The code is done in an hour. The rules take a week, because you have to watch what your server actually logs. My process:

# what does this thing scream about, ranked?
journalctl -u gameserver --since "7 days ago" \
  | grep -iE "error|warn|fatal|exception" \
  | sed 's/[0-9]//g' \
  | sort | uniq -c | sort -rn | head -20

Strip the numbers so timestamps and player IDs collapse together, then count. The top of that list is your noise — things that happen constantly and mean nothing. Do not alert on them. The bottom of the list, the things that happened once or twice, is where the real incidents hide.

I started with a rule on every ERROR. Two hundred messages the first day, most of them a harmless plugin whining about a missing optional config. Now I match FATAL and two specific strings, and when it fires, something is genuinely wrong.

Match Ping Why
FATAL, Watchdog, Exception in server tick yes Server is dying or dead
Can't keep up, stalled yes Freeze — players feel this before you do
kicked ... for <reason> no Useful signal, not an emergency
Generic ERROR never Noise. This is a trap.

Where it fits

Two layers, and they answer different questions.

Prometheus + Grafana answers "is the machine healthy?" — CPU, memory, network, is the unit active. See /monitor-gameserver-wireguard-prometheus/ for the setup I run.

The log bot answers "is the game working?" — which is the question your players are actually asking when they DM you. It doesn't replace metrics and it doesn't want to. It just reads the one file that already contains the answer, and tells you before someone else does.

That's a good hundred lines of code.

Related posts