Uptime Kuma tells me if a box is down. Netdata tells me what's happening right now on one host. Neither answers "what did CPU look like last Tuesday at 3am when the job OOM'd?" For that I want time-series with retention, and that's Prometheus scraping metrics into a database with Grafana drawing the graphs. It's more moving parts, so I only run it where I actually need history — but where I do, it's the thing I open first.

TL;DR — three containers, one compose file:

# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports: ["9090:9090"]
    restart: unless-stopped
  node-exporter:
    image: prom/node-exporter:latest
    pid: host
    volumes: ["/:/host:ro,rslave"]
    command: ["--path.rootfs=/host"]
    restart: unless-stopped
  grafana:
    image: grafana/grafana:latest
    volumes: [grafana-data:/var/lib/grafana]
    ports: ["3000:3000"]
    restart: unless-stopped
volumes: { prom-data: {}, grafana-data: {} }

Details below. New to compose? Start with /docker-install-ubuntu/.

What each piece does

  • Prometheus is the brain. It pulls metrics from targets over HTTP every N seconds and stores them as a time series. Pull, not push — you tell Prometheus where to scrape, not the other way round.
  • node_exporter is a tiny agent that exposes host metrics (CPU, RAM, disk, network) at :9100/metrics. One per machine you want to watch.
  • Grafana is the dashboards. It queries Prometheus with PromQL and draws the graphs. It stores nothing about your metrics itself, just how to display them.

The mental model that took me too long to get: Grafana is not the database. Prometheus is. Grafana is a nice window onto it.

The scrape config

Prometheus does nothing until you tell it what to scrape. This is the whole config for a single host plus itself:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alerts.yml"

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]
        labels:
          instance: web01

Because everything's on the same compose network, node-exporter:9100 resolves by service name — no IPs. Adding a second server later is one more target line pointing at its host:9100 (open 9100 only to the Prometheus box, never the world).

docker compose up -d
curl -s localhost:9090/-/ready     # Prometheus healthy

Open http://server:9090/targets — both jobs should be green UP. If node is red, it's almost always the exporter not reachable on the network, not Prometheus.

Wire up Grafana

Log in at :3000 (admin/admin, it forces a change). Add the data source once:

  • Connections → Data sources → Prometheus
  • URL: http://prometheus:9090 (service name again, not localhost — Grafana is in a container)
  • Save & test → green.

Then don't build a node dashboard by hand. Import the community one: Dashboards → Import → ID 1860 (Node Exporter Full). Pick your Prometheus source, and you've got CPU, memory, disk I/O, network and filesystem panels in about ten seconds. I've never found a reason to redraw that one from scratch.

Alerts that actually fire

Graphs you have to look at are useless at 3am. Prometheus evaluates alert rules and flips them to firing; a separate Alertmanager routes them to email/Slack/Telegram. Start with the rules — routing can come later:

# alerts.yml
groups:
  - name: host
    rules:
      - alert: DiskWillFill
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "Root disk under 10% on {{ $labels.instance }}"

      - alert: HostDown
        expr: up{job="node"} == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "node_exporter unreachable on {{ $labels.instance }}"

The for: 10m matters — it means "only fire if this stays true for 10 minutes", which kills the flapping alerts that train you to ignore the channel. Reload with curl -X POST localhost:9090/-/reload (needs --web.enable-lifecycle in the command) or just restart the container. Watch them at :9090/alerts.

Retention and disk

Prometheus keeps 15 days by default and it will grow. Set retention explicitly on the command so you know what you signed up for:

  prometheus:
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"

A single node at 15s scrape is a few hundred MB a month — trivial. Scraping twenty hosts with high-cardinality app metrics is a different conversation, and that's when you learn what "cardinality" means the hard way.

Gotchas

  • localhost inside a container is the container. The single most common mistake: pointing Grafana at http://localhost:9090. That's Grafana's own localhost. Use the service name.
  • node_exporter needs the host, not the container. Without pid: host and the rootfs mount it happily reports the container's metrics — near-empty and useless. The command/volume in the TL;DR is what makes it see the real host.
  • Exposing 9090 and 3000 to the internet. Prometheus has no auth at all. Put both behind a reverse proxy with a login, or bind them to localhost and tunnel. See /nginx-reverse-proxy-setup/.
  • Scrape interval vs. rule interval. A for: 2m on a 15s scrape is fine. A for: 30s with a 60s scrape can never satisfy the window — the rule sees one sample and resets. Keep for a comfortable multiple of the interval.
  • latest tags. Fine for a homelab, a footgun in prod. Pin versions once it matters so an unattended pull doesn't change your storage format under you.

Where this fits

This is the heavyweight of my monitoring setup. For a plain "is it up, ping me if not" I still reach for /uptime-kuma-self-hosted-monitoring/, and for a live per-host glance without any storage story /netdata-monitoring-ubuntu/ is one command. Prometheus is what you add when you need to answer questions about the past. Run the containers under systemd if you want them treated as real services: /systemd-service-unit-file/.

Related posts