A disk will die. The only question is whether your VMs notice. On Proxmox I run ZFS for exactly one reason: I can yank a failing drive, slot a new one, and resilver while everything stays online.
TL;DR:
# create a mirror from raw disks (by-id, ALWAYS)
zpool create -o ashift=12 tank mirror \
/dev/disk/by-id/ata-WDC_WD40EFZX_AAA \
/dev/disk/by-id/ata-WDC_WD40EFZX_BBB
zfs set compression=lz4 atime=off tank
pvesm add zfspool tank-vms -pool tank -content images,rootdir -sparse 1
Details below.
Use by-id, not /dev/sdX
/dev/sda is not stable. Reboot, add a USB stick, swap a cable — the kernel reassigns letters and now your pool references the wrong disk. ZFS handles this at runtime, but you do not want to be reading zpool status at 2am trying to figure out which physical drive sdc is today.
ls -l /dev/disk/by-id/ | grep -v part
Use the ata-* or wwn-* paths. They're printed on the drive label and survive everything. If you already built a pool with /dev/sdX, fix it:
zpool export tank
zpool import -d /dev/disk/by-id tank
ashift=12 forces 4K sectors. Every modern drive is 4Kn or 512e-pretending-to-be-512. Get this wrong at create time and you cannot change it later — you'd have to destroy and rebuild the pool. Always ashift=12.
mirror vs raidz1 vs raidz2 — pick the right one
This is the decision that matters. You cannot change a vdev's layout after creation.
- mirror (2 disks): one can die. Best IOPS, simplest to grow (add more mirror vdevs), fastest resilver. Lose 50% capacity. My default for VM storage.
- raidz1 (3+ disks): one can die. ~67%+ usable. Slower, and resilver on big disks is scary — a second drive often fails during the multi-hour rebuild. Fine for 4TB-class disks, sketchy above ~8TB.
- raidz2 (4+ disks): two can die. The safe choice for large arrays. Survives a second failure mid-resilver.
# 4-disk raidz2 for a bulk/backup pool
zpool create -o ashift=12 tank raidz2 \
/dev/disk/by-id/ata-DISK1 /dev/disk/by-id/ata-DISK2 \
/dev/disk/by-id/ata-DISK3 /dev/disk/by-id/ata-DISK4
For VM workloads I run mirrors. For a backup target where IOPS don't matter, raidz2. Never raidz1 on disks bigger than 8TB.
Dataset properties
Set these on the pool root so they inherit down:
zfs set compression=lz4 tank # free; almost always a net win
zfs set atime=off tank # stop writing on every read
lz4 compression is nearly free on CPU and skips incompressible data automatically. Turn it on everywhere.
recordsize matters for VM zvols. The default 128k is right for general files but wrong for a VM doing 4K-16K random IO inside its guest filesystem — you get read/write amplification. For VM-heavy pools, set the zvol volblocksize to 16k–64k:
# Proxmox controls this per-storage; set the default block size
pvesm set tank-vms -blocksize 32k
For a dataset holding large media, push the other way with recordsize=1M.
Add it to Proxmox
GUI: Datacenter > Storage > Add > ZFS, pick tank, content = Disk image + Container, tick Thin provision. Or CLI:
pvesm add zfspool tank-vms -pool tank -content images,rootdir -sparse 1
-sparse 1 is thin provisioning — a 100G disk only consumes what's actually written. Combined with snapshots this is the whole reason to bother. See my Ubuntu cloud-init template post for spinning up VMs that land straight on this pool.
Cap the ARC before it eats your RAM
ZFS uses a memory cache (ARC) that by default grabs up to half your RAM. On a Proxmox box that RAM is for VMs, and the ARC counts as "used" — you'll see memory pressure and wonder why. Cap it.
Rule of thumb: give the ARC what you can spare, often 4–16 GB on a homelab node. Here, 8 GB:
# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=8589934592
echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf
update-initramfs -u
reboot
That number is bytes (8 * 1024^3). Check it took: cat /sys/module/zfs/parameters/zfs_arc_max.
THE big gotcha: never put ZFS on hardware RAID
ZFS wants raw disks. It does its own checksumming, redundancy, and error recovery, and it assumes it can see and trust the physical drives directly.
Put it on top of a hardware RAID controller and you break all of that: the controller hides drives behind its own RAID, lies about write completion via its cache, and remaps sectors silently. When ZFS detects corruption it can no longer find a good copy to heal from, because the layer underneath it scrambled the topology. You get the overhead of ZFS with none of the safety.
If your server has a RAID card, flash it to IT mode (HBA / passthrough) so it presents disks raw. LSI 9211-8i and friends do this fine. No HBA, no IT mode — use the onboard SATA ports directly. Do not configure a single-disk RAID0 per drive as a workaround; the cache and remapping problems remain.
The exception: a true HBA in IT mode is exactly what you want — it's a dumb pass-through, which is the point.
ECC RAM — nice, not mandatory
ZFS doesn't require ECC RAM, despite what forum threads claim. ECC protects in-flight data in memory from bit flips, which is good hygiene for any storage server. But non-ECC ZFS is still far safer than non-ECC anything-else, because ZFS at least checksums data on disk. If your hardware has ECC, great. If it doesn't, run ZFS anyway — just don't lose sleep over it.
Scrub monthly
A scrub reads every block and verifies checksums, healing silent corruption from the redundant copy. On Debian/Proxmox the package ships a systemd timer:
systemctl enable --now zfs-scrub-monthly@tank.timer
systemctl list-timers | grep zfs-scrub
Prefer cron? One line:
0 3 1 * * /sbin/zpool scrub tank
Read the result with zpool status. state: ONLINE and errors: No known data errors is what you want. Any non-zero in the READ/WRITE/CKSUM columns means a disk is going.
Replace a dead disk
The whole point. Say zpool status shows a drive DEGRADED or FAULTED:
zpool offline tank /dev/disk/by-id/ata-WDC_WD40EFZX_BBB
Physically swap the drive. Find the new disk's by-id path, then:
zpool replace tank \
ata-WDC_WD40EFZX_BBB \
/dev/disk/by-id/ata-WDC_WD40EFZX_CCC
ZFS resilvers — rebuilds the new disk from the surviving copy. Watch it:
zpool status -v tank
You'll see resilver in progress with a percentage and ETA. VMs keep running the whole time. When it hits errors: No known data errors and the pool is ONLINE, you're done. On a mirror this is fast; on raidz with large disks it can take a day — which is exactly why raidz1 + huge disks is a bad bet.
Snapshots are not backups
Snapshots are instant and nearly free, and they live on the same pool:
zfs snapshot tank/vm-101-disk-0@before-upgrade
zfs list -t snapshot
zfs rollback tank/vm-101-disk-0@before-upgrade
Take one before every risky change. But a snapshot dies with the pool — if the chassis burns or the controller cooks every drive, your snapshots burn with it. They are not a backup.
For actual backups, push VMs to a separate box with Proxmox Backup Server, and ship an offsite copy with restic to S3/MinIO. One pool of redundant disks plus snapshots covers a disk death; it does not cover a host death.
Verify
zpool status -v # ONLINE, no errors, all disks present
zfs list # datasets + space, thin usage
zpool list # pool capacity and health
zfs get compression,atime,recordsize tank
If zpool status is ONLINE with zero errors and your properties stuck, the pool is doing its job — and the next dead disk is a swap, not an outage.
