↓ Skip to main content
  1. Posts/

VPS backups that still work on the day you need them

Table of Contents

Most people find out how good their backups are on the worst possible day. A disk dies, someone runs the wrong rm, a plugin gets exploited and the attacker encrypts /var/www. At that point the only question that matters is whether you can get yesterday’s data back, and how long it takes.

This post covers a setup for one VPS, or a handful of them, that answers that question with a yes. It uses restic, but the ideas carry over to borg or anything similar.

A copy is not a backup
#

Three things often get called a backup, and only one of them is.

An rsync mirror to another server is a copy. If you run it with --delete, a file you deleted by accident disappears from the mirror on the next run. If ransomware encrypts your files, the mirror happily syncs the encrypted versions. You only ever have the latest state, and the latest state is exactly what’s broken.

A provider snapshot is useful, and you should take one before risky upgrades. But it usually lives on the same platform as the VPS, you often can’t pick individual files out of it, and if you lose access to the account you lose the snapshots with it. Treat it as a fast undo button, not as your backup.

A backup has history (you can go back to last Tuesday), lives somewhere the server can’t destroy, and has been restored at least once to prove it works.

The 3-2-1 rule, scaled down
#

The classic rule is three copies of your data, on two different kinds of storage, with one copy off-site. For a VPS that translates into something practical:

  1. The live data on the VPS.
  2. A restic repository on a separate machine or object storage, with a different provider if you can manage it.
  3. Optionally, a second repository somewhere else, or a periodic copy of the first one.

The important part is the second point. A backup stored on the same disk, or on another VPS in the same account, doesn’t protect you from the account or the platform having a bad day.

Setting up restic
#

Install it from your distribution (apt install restic on Debian and Ubuntu) and keep the repository password in a file only root can read:

install -m 600 /dev/null /root/.restic-pass
openssl rand -base64 32 > /root/.restic-pass

export RESTIC_REPOSITORY="sftp:backup@backup.example.com:/srv/restic/web01"
export RESTIC_PASSWORD_FILE="/root/.restic-pass"

restic init

Store a copy of that password somewhere outside the server, like a password manager. Without it the repository is encrypted noise, and you’ll need it most when the server itself is gone.

Then back up what you can’t rebuild. That’s usually configuration, application data and home directories, not the whole operating system:

restic backup /etc /var/www /home /root \
  --exclude-caches \
  --exclude '/var/www/*/cache'

Databases need a dump
#

Copying the files of a running MySQL or PostgreSQL server gives you a backup that may not start. Dump the database and feed the dump straight into restic:

mysqldump --single-transaction --all-databases \
  | restic backup --stdin --stdin-filename all-databases.sql

sudo -u postgres pg_dumpall \
  | restic backup --stdin --stdin-filename postgres.sql

--single-transaction gives a consistent dump of InnoDB tables without locking them for the whole run.

Keep history, but not forever
#

Restic keeps every snapshot until you tell it otherwise. A retention policy keeps the repository from growing without limit while still letting you go back in time:

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

That gives you a week of daily points, a month of weekly ones and half a year of monthly ones. Adjust it to how far back you realistically need to go. Problems are often noticed late, so a few months of history is worth the storage.

Protect the backups from the server
#

This is the part most setups skip. If the VPS can delete its own backups, so can anyone who takes over the VPS. An attacker with root will find the backup credentials in a few minutes.

The fix is to make the repository append-only from the server’s point of view. The server can add new snapshots but can’t delete or overwrite old ones. With restic, run the repository behind rest-server in append-only mode:

rest-server --path /srv/restic --append-only --private-repos

On the VPS, the repository URL then changes from sftp: to something like rest:https://web01:CHANGE_ME@backup.example.com:8000/web01/, where the user and password come from the .htpasswd file of rest-server. With --private-repos, each user can only reach its own repository.

With borg, the same idea goes into authorized_keys on the backup host:

command="borg serve --append-only --restrict-to-path /srv/borg/web01",restrict ssh-ed25519 AAAA... web01

Pruning old snapshots then happens from the backup host or another trusted machine, never from the VPS itself.

Schedule it and watch it
#

Put the pieces from above into one script, with the repository settings in an environment file:

# /root/.restic-env
RESTIC_REPOSITORY=rest:https://web01:CHANGE_ME@backup.example.com:8000/web01/
RESTIC_PASSWORD_FILE=/root/.restic-pass
#!/bin/bash
# /usr/local/sbin/backup.sh
set -euo pipefail

mysqldump --single-transaction --all-databases \
  | restic backup --stdin --stdin-filename all-databases.sql

restic backup /etc /var/www /home /root \
  --exclude-caches \
  --exclude '/var/www/*/cache'

The forget --prune step isn’t in the script on purpose. With an append-only repository the server isn’t allowed to delete anything, so pruning runs on the backup host, as described in the previous section.

A systemd timer is easier to monitor than a crontab line, because failures show up in the journal and in systemctl --failed:

# /etc/systemd/system/restic-backup.service
[Unit]
Description=restic backup

[Service]
Type=oneshot
EnvironmentFile=/root/.restic-env
ExecStart=/usr/local/sbin/backup.sh
# /etc/systemd/system/restic-backup.timer
[Unit]
Description=Nightly restic backup

[Timer]
OnCalendar=*-*-* 03:15
RandomizedDelaySec=30m
Persistent=true

[Install]
WantedBy=timers.target
systemctl daemon-reload
systemctl enable --now restic-backup.timer
systemctl list-timers restic-backup.timer

Persistent=true runs a missed backup after the machine comes back up. Add some way of hearing about failures, whether that’s mail from a wrapper script or a ping to a monitoring service when the job succeeds. A backup job that has been failing silently for three months looks exactly like one that works, until you need it.

Test restores, on a schedule
#

An untested backup is a hope. Restic can check the repository structure and read back a sample of the actual data:

restic check --read-data-subset=5%

That catches corruption, but it doesn’t tell you whether you backed up the right things. For that you have to restore something and look at it:

restic snapshots
restic restore latest --target /tmp/restore-test --include /etc/nginx
diff -r /etc/nginx /tmp/restore-test/etc/nginx

Once in a while, do the full exercise on a throwaway VPS. Install a clean system, restore the files and the database dump, and see if the application comes up. You will almost always find something missing, like a cron job, a TLS certificate or a file outside the paths you back up. It’s much better to find that on a quiet afternoon.

A short checklist
#

  • History: you can restore a file as it was a week or a month ago.
  • Off the box: the repository lives on different hardware, ideally with a different provider.
  • Append-only: the server can write backups but not delete them.
  • Databases are dumped, not copied as live files.
  • The repository password is stored somewhere other than the server.
  • Failures alert someone.
  • Restores have been tested, recently.

If you can tick all of those, a dead disk or a compromised server becomes a bad afternoon instead of a lost business.

Related