My Server Ran Out and My Backups Were Truncated: What I Learned
A Proxmox host went away for good and the offsite backups turned out to be cut off. What I could recover, what was gone, and the checks I run now.
A Proxmox host I ran went away. Not a hardware failure, not a hack — the contract ran out and the machine was gone for good. I had backups on offsite storage, and I still lost data. Not everything, but enough to make me rewrite how I think about "I have backups".
This post is the honest version: what was on the box, what the backups actually contained when I needed them, and the checks I do now. None of it is new wisdom. All of it is stuff I had "on the list".
TL;DR: Three VM/container backups sat on offsite storage. One was complete. One was missing a few percent at the end of the upload. The biggest one — a Nextcloud VM — was truncated at 54 of roughly 65 GB because the transfer had aborted and left a partial file with the right name. A separate database dump job that should have been my second line of defense had never run once. I recovered most of the Nextcloud files, but the live database was gone and I had to fall back to a dump that was about three months old. Lesson: a backup you haven't restored is a hope, not a backup.
What was on the host
Three guests mattered:
- a container running a web panel — backed up with vzdump, archive complete
- a VM running a self-hosted PaaS with a few small apps
- a Nextcloud VM with several users' files
The backups were vzdump archives (.tar.zst for the container, .vma.zst for the VMs), copied over SSH to a Storage-Box-style offsite location. I'd looked at the folder. The files were there. Names looked right. Sizes looked "big".
That's the whole problem in one sentence: I checked that the files existed, not that they were complete.
What I found when I needed them
The container: fine. Full archive, extracted cleanly.
The PaaS VM: vzdump itself had run through, but the copy on the offsite storage was missing roughly 3% at the end. I could extract almost the whole disk image. The one thing I actually needed from it — an SQL dump of an app's database lying in a home directory — happened to be inside the part that survived. Lucky, not good.
The Nextcloud VM: this is the one that hurt. The archive on the storage was 54 GB of about 65 GB. The upload had been done with rsync --partial, it had aborted, and --partial did exactly what it says: kept the partial file. Under the final filename. Nothing about the directory listing told me it was incomplete.
# what I should have been running all along
zstd -t vzdump-qemu-*.vma.zst # integrity test, fails on a truncated stream
ls -l --block-size=M vzdump-* # and compare against the source size
zstd -t on a truncated file fails at the end. It would have told me in seconds. I never ran it.
The database dump job: I had a job that was supposed to dump the PaaS VM's databases to a private Git repo as an extra copy. It had never run — it got blocked during setup, I moved on to something else, and then the host was gone. So the "second copy" I mentally counted on never existed.
What I could recover
The vma tool that unpacks Proxmox VM backups lives on Proxmox. I didn't have a Proxmox box any more, and I had a truncated file that the normal tooling isn't built to handle gracefully. I ended up writing my own small VMA extractor in Python — that's its own post: extracting a Proxmox VMA backup without Proxmox.
The rough path for the Nextcloud VM:
# decompress as far as it goes; zstd errors at the end of a truncated stream, that's expected
zstd -dc --long=31 vzdump-qemu-NNN.vma.zst | python3 vma_extract.py restore/
# attach the raw disk, let fsck put the pieces somewhere
losetup -fP --show restore/disk-1.raw
fsck -y /dev/loop0p1
mount -o ro /dev/loop0p1 /mnt/rescue
About 83% of the disk was intact. After fsck -y, a large part of the filesystem ended up in lost+found — including, thankfully, the complete Nextcloud webroot with its data/ directory. Around 21 GB of files came back. Whatever lived in the missing last 17% of the disk is gone.
Then the second problem: the live Nextcloud database was not recoverable. The only database dump I had was about three months old. So the files were current (where they survived), but shares, metadata and everything else in the DB reflected the older state.
Gotcha: the restore environment bites too
Even with the files in hand, bringing Nextcloud back up wasn't a straight copy. The newer distro I restored onto shipped a PHP version that the Nextcloud release from the backup didn't run on. Fighting that on the host would have been a rabbit hole, so I ran the matching official Nextcloud image in Docker (nextcloud:33-apache, which ships PHP 8.4) with host networking, pointed at a database on the host. Two config edits were needed: datadirectory had to point at the container path, and dbhost had to change from localhost to 127.0.0.1 because inside the container localhost means the socket, not the host's TCP port.
Point being: "restore" means getting a working service, not just files. Test that too.
The lessons, concretely
1. Verify integrity, not existence
After every offsite copy:
zstd -t "$FILE" || alert "backup $FILE is corrupt or truncated"
For .tar.zst you can go further with zstd -dc "$FILE" | tar -t > /dev/null. It's slow on big archives, but it reads the whole thing. Compare the remote size with the local size before deleting anything locally.
2. Don't let --partial lie to you
rsync --partial is great for resuming. It's terrible if nothing checks whether the resume ever happened. Either upload to a temporary name and rename on success, or check rsync's exit code and alert on anything non-zero. A partial file with the final name is worse than no file — no file at least looks wrong.
3. Prove that scheduled jobs actually run
My dump job didn't fail. It never started. Nothing alerts on "nothing happened". The fix is cheap: every backup job writes a timestamp on success, and a separate check alerts if that timestamp is older than expected. A dead-man's switch, not an error handler. I've written about the same idea for idempotent cron jobs, and systemd timers at least give you systemctl list-timers to see when something last fired.
4. Database dumps are not optional
A VM image catches the database files mid-flight. A logical dump (mysqldump, pg_dump) is consistent and tiny — the app DB I needed was a few megabytes. Dump the DB separately, on its own schedule, and ship it separately. If I'd had a fresh Nextcloud dump, the result would have been "some files missing" instead of "some files missing plus three months of metadata".
5. Do a restore drill before you need it
Once a quarter, pick a backup, restore it somewhere disposable, and check that the service comes up. Not "the archive extracts" — the service. That's how you find out about the PHP mismatch and the localhost vs 127.0.0.1 thing on a quiet afternoon instead of at 2 a.m.
6. Treat contract end dates like a disaster drill
If a machine has an end date — a contract, a trial, a provider shutting down a product — put it in the calendar with a buffer and do the final backup and its verification well before it. A backup that's verified early can be redone; one that's still uploading when the machine disappears is whatever you get.
What I'd set up differently
- vzdump to Proxmox Backup Server instead of shipping single huge files — PBS verifies chunks and transfers incrementally, so there's no 65 GB single-file upload to fail halfway
- for file-level stuff, something like restic with encryption to S3-compatible storage, which has
restic checkbuilt in - separate DB dumps, shipped separately
- a freshness check on every job that alerts somewhere I actually look
FAQ
Isn't the real problem that the host went away? Hosts go away — contracts end, providers fold, hardware dies. That part is normal. The failure was trusting backups I'd never verified.
Is a truncated vzdump archive worthless? No. Everything before the cut is readable if you parse the format yourself, and filesystems are surprisingly forgiving with fsck. But "surprisingly forgiving" is not a backup strategy.
Was server-side encryption a concern? It would have been. With Nextcloud's server-side encryption enabled, files in data/ are encrypted and you need the keys and secrets from the instance to decrypt them. In my case it was off, so the rescued files were readable as-is. Check that before you need it.
