> ## Content Index > Fetch the complete content index at: https://blog.diengdoh.com/llms.txt > Use this file to discover other available public pages before exploring further. # My Server Ran Out and My Backups Were Truncated: What I Learned - URL: https://blog.diengdoh.com/server-provider-outage-backup-lessons/ - Published: 2026-07-27T10:00:00.000Z - Updated: 2026-10-11T10:16:27.000Z - Description: A Proxmox host went away for good and the offsite backups turned out to be cut off. What I could recover, what was gone, and the checks I run now. - Author: Lukas - Tags: Linux, Security A Proxmox host I ran went away. Not a hardware failure, not a hack — the contract ran out and the machine was gone for good. I had backups on offsite storage, and I still lost data. Not everything, but enough to make me rewrite how I think about "I have backups". This post is the honest version: what was on the box, what the backups actually contained when I needed them, and the checks I do now. None of it is new wisdom. All of it is stuff I had "on the list". **TL;DR:** Three VM/container backups sat on offsite storage. One was complete. One was missing a few percent at the end of the upload. The biggest one — a Nextcloud VM — was **truncated at 54 of roughly 65 GB** because the transfer had aborted and left a partial file with the right name. A separate database dump job that should have been my second line of defense **had never run once**. I recovered most of the Nextcloud files, but the live database was gone and I had to fall back to a dump that was about three months old. Lesson: a backup you haven't restored is a hope, not a backup. ## What was on the host Three guests mattered: - a **container running a web panel** — backed up with vzdump, archive complete - a **VM running a self-hosted PaaS** with a few small apps - a **Nextcloud VM** with several users' files The backups were vzdump archives (`.tar.zst` for the container, `.vma.zst` for the VMs), copied over SSH to a Storage-Box-style offsite location. I'd looked at the folder. The files were there. Names looked right. Sizes looked "big". That's the whole problem in one sentence: I checked that the files *existed*, not that they were *complete*. ## What I found when I needed them **The container:** fine. Full archive, extracted cleanly. **The PaaS VM:** vzdump itself had run through, but the copy on the offsite storage was missing roughly 3% at the end. I could extract almost the whole disk image. The one thing I actually needed from it — an SQL dump of an app's database lying in a home directory — happened to be inside the part that survived. Lucky, not good. **The Nextcloud VM:** this is the one that hurt. The archive on the storage was **54 GB of about 65 GB**. The upload had been done with `rsync --partial`, it had aborted, and `--partial` did exactly what it says: kept the partial file. Under the final filename. Nothing about the directory listing told me it was incomplete. ```bash # what I should have been running all along zstd -t vzdump-qemu-*.vma.zst # integrity test, fails on a truncated stream ls -l --block-size=M vzdump-* # and compare against the source size ``` `zstd -t` on a truncated file fails at the end. It would have told me in seconds. I never ran it. **The database dump job:** I had a job that was supposed to dump the PaaS VM's databases to a private Git repo as an extra copy. It had never run — it got blocked during setup, I moved on to something else, and then the host was gone. So the "second copy" I mentally counted on never existed. ## What I could recover The `vma` tool that unpacks Proxmox VM backups lives on Proxmox. I didn't have a Proxmox box any more, and I had a truncated file that the normal tooling isn't built to handle gracefully. I ended up writing my own small VMA extractor in Python — that's its own post: [extracting a Proxmox VMA backup without Proxmox](/extract-proxmox-vma-backup-without-proxmox/). The rough path for the Nextcloud VM: ```bash # decompress as far as it goes; zstd errors at the end of a truncated stream, that's expected zstd -dc --long=31 vzdump-qemu-NNN.vma.zst | python3 vma_extract.py restore/ # attach the raw disk, let fsck put the pieces somewhere losetup -fP --show restore/disk-1.raw fsck -y /dev/loop0p1 mount -o ro /dev/loop0p1 /mnt/rescue ``` About 83% of the disk was intact. After `fsck -y`, a large part of the filesystem ended up in `lost+found` — including, thankfully, the complete Nextcloud webroot with its `data/` directory. Around 21 GB of files came back. Whatever lived in the missing last 17% of the disk is gone. Then the second problem: the live Nextcloud database was not recoverable. The only database dump I had was about three months old. So the **files** were current (where they survived), but shares, metadata and everything else in the DB reflected the older state. ## Gotcha: the restore environment bites too Even with the files in hand, bringing Nextcloud back up wasn't a straight copy. The newer distro I restored onto shipped a PHP version that the Nextcloud release from the backup didn't run on. Fighting that on the host would have been a rabbit hole, so I ran the matching official Nextcloud image in Docker (`nextcloud:33-apache`, which ships PHP 8.4) with host networking, pointed at a database on the host. Two config edits were needed: `datadirectory` had to point at the container path, and `dbhost` had to change from `localhost` to `127.0.0.1` because inside the container `localhost` means the socket, not the host's TCP port. Point being: "restore" means getting a working service, not just files. Test that too. ## The lessons, concretely ### 1\. Verify integrity, not existence After every offsite copy: ```bash zstd -t "$FILE" || alert "backup $FILE is corrupt or truncated" ``` For `.tar.zst` you can go further with `zstd -dc "$FILE" | tar -t > /dev/null`. It's slow on big archives, but it reads the whole thing. Compare the remote size with the local size before deleting anything locally. ### 2\. Don't let `--partial` lie to you `rsync --partial` is great for resuming. It's terrible if nothing checks whether the resume ever happened. Either upload to a temporary name and rename on success, or check rsync's exit code and alert on anything non-zero. A partial file with the final name is worse than no file — no file at least looks wrong. ### 3\. Prove that scheduled jobs actually run My dump job didn't fail. It never started. Nothing alerts on "nothing happened". The fix is cheap: every backup job writes a timestamp on success, and a separate check alerts if that timestamp is older than expected. A dead-man's switch, not an error handler. I've written about the same idea for [idempotent cron jobs](/idempotency-cron-automation-duplicate-uploads/), and [systemd timers](/systemd-timers-instead-of-cron/) at least give you `systemctl list-timers` to see when something last fired. ### 4\. Database dumps are not optional A VM image catches the database files mid-flight. A logical dump (`mysqldump`, `pg_dump`) is consistent and tiny — the app DB I needed was a few megabytes. Dump the DB separately, on its own schedule, and ship it separately. If I'd had a fresh Nextcloud dump, the result would have been "some files missing" instead of "some files missing plus three months of metadata". ### 5\. Do a restore drill before you need it Once a quarter, pick a backup, restore it somewhere disposable, and check that the service comes up. Not "the archive extracts" — the service. That's how you find out about the PHP mismatch and the `localhost` vs `127.0.0.1` thing on a quiet afternoon instead of at 2 a.m. ### 6\. Treat contract end dates like a disaster drill If a machine has an end date — a contract, a trial, a provider shutting down a product — put it in the calendar with a buffer and do the final backup *and its verification* well before it. A backup that's verified early can be redone; one that's still uploading when the machine disappears is whatever you get. ## What I'd set up differently - vzdump to [Proxmox Backup Server](/proxmox-backup-server-setup/) instead of shipping single huge files — PBS verifies chunks and transfers incrementally, so there's no 65 GB single-file upload to fail halfway - for file-level stuff, something like [restic with encryption to S3-compatible storage](/restic-backup-s3-minio-encrypted/), which has `restic check` built in - separate DB dumps, shipped separately - a freshness check on every job that alerts somewhere I actually look ## FAQ **Isn't the real problem that the host went away?** Hosts go away — contracts end, providers fold, hardware dies. That part is normal. The failure was trusting backups I'd never verified. **Is a truncated vzdump archive worthless?** No. Everything before the cut is readable if you parse the format yourself, and filesystems are surprisingly forgiving with `fsck`. But "surprisingly forgiving" is not a backup strategy. **Was server-side encryption a concern?** It would have been. With Nextcloud's server-side encryption enabled, files in `data/` are encrypted and you need the keys and secrets from the instance to decrypt them. In my case it was off, so the rescued files were readable as-is. Check that before you need it. --- ### Related posts - [Tailscale vs WireGuard — Which VPN Should You Actually Run?](https://blog.diengdoh.com/tailscale-vs-wireguard/) - [One Public IP, Many VMs — Port-Splitting a Proxmox Host with nftables](https://blog.diengdoh.com/proxmox-one-public-ip-multiple-vms/) - [CrowdSec — A Modern Take on fail2ban](https://blog.diengdoh.com/crowdsec-modern-fail2ban-alternative/)