필사 모드: The complete backup and restore guide: designing backwards from the recovery scenario

• Introduction
• 1. RPO and RTO — the starting point for every decision
• 2. The 3-2-1 rule and its modern amendments
• 3. rsync — the tool most used and most often got wrong
• 4. tar — archive fundamentals
• 5. Consistency — data that changes mid-backup
• 6. Verification — without a restore rehearsal it is not a backup
• 7. Five ways a backup quietly rots
• Quiz: check your understanding
• Closing
• References
• Further reading

Introduction

The most common misconception about backups is "we are taking backups, so we are safe". Reality is different. A backup you have never restored from is not a backup, it is a pile of files you believe is a backup. In real incidents, backups turn out to be useless not usually because nobody took them, but because nobody ever ran the restore procedure even once.

This blog already has an open source backup tool comparison. That post compares features and trade-offs tool by tool. This one deals with the problem that comes before tool selection. The subject is the design order: decide first which recovery scenarios you have to survive, then derive the backup method backwards from there.

The tools covered are mostly the ones present in a stock Linux environment: rsync, tar, LVM snapshots, and database dumps. The distribution baseline is RHEL-family 8 and later, Debian 12, and Ubuntu 22.04 and later, and the explanations assume GNU tools. The tar, sed, and date on BSD or macOS take different options, so be careful when you move scripts across.

1. RPO and RTO — the starting point for every decision

Two numbers decide all the rest.

• RPO (Recovery Point Objective): how many minutes or hours of data you are allowed to lose. It decides the backup interval.
• RTO (Recovery Time Objective): how many hours are acceptable between the failure and service resumption. It decides the backup method and storage location.

Pick a tool without settling these two values and it will be mismatched, guaranteed. An RPO of 5 minutes while you take one dump a day, or an RTO of 1 hour while you have to run an 8-hour restore from tape — that is what it looks like.

The last row is the important one. Backup and high availability are different tools solving different problems. Replication instantly deletes the data you deleted by mistake right along with it. Ransomware replicates too. That is why you still need backups even when replication is in place.

2. The 3-2-1 rule and its modern amendments

The classic 3-2-1 rule is 3 copies, 2 media types, 1 offsite. Two more clauses have been appended to it recently.

• 1 copy offline or immutable — because of incidents where ransomware encrypted the backups as well.
• 0 unverified copies — a copy that has not passed restore verification does not count.

What this means in practice is clear. The backup server must not hold credentials for the production server. If production is compromised, the backups get deleted along with it. This is exactly why a pull model, where the backup server connects to production and pulls, is safer than a push model where production pushes into the backup server.

If you use object storage, turning on versioning and object lock is a substantive defence. Check the exact setting names and procedures in the documentation for the storage product you use.

The "2 media types" condition also needs a modern reading. It used to mean disk and tape, but today it is better read as two stores with different failure modes. Two regions in the same cloud account are not two media types, because if the account itself is locked or billing stops you cannot reach either one. Judge by whether the failure causes overlap.

And the backup list has to contain not only the data but everything needed to bring that data back to life: configuration files, certificates and private keys, database schemas, deployment artefacts, and the restore procedure document. There really are plenty of real cases where recovery dragged on for days because the data was backed up but the way to stand the service up was not.

3. rsync — the tool most used and most often got wrong

rsync -aHAX --numeric-ids --delete --dry-run /srv/data/ backup@10.0.9.2:/backup/prod/data/
rsync -aHAX --numeric-ids --delete /srv/data/ backup@10.0.9.2:/backup/prod/data/

Here is what the options mean exactly.

• -a (--archive) is documented as equivalent to -rlptgoD. That covers recursion, symlink preservation, permissions, times, group, owner, and special files.
• There are things -a does not include. ACLs (-A), extended attributes (-X), and hard links (-H) have to be specified separately. Not knowing this leaves permissions subtly different after a restore.
• --numeric-ids uses numeric UID/GID instead of names. It stops owners from being mapped to the wrong accounts when you restore onto a different system.
• --delete removes files from the destination that do not exist at the source.
• -n (--dry-run) simulates without making any actual change.
• -i (--itemize-changes) shows line by line which changes are going to happen.

Destructive command warning: if --delete runs while the source is empty or its mount has fallen off, it wipes every bit of data at the destination. This is an accident that genuinely happens often in practice. Check with --dry-run first, and put a guard in the script that verifies the source is not empty.

#!/usr/bin/env bash
set -euo pipefail
SRC=/srv/data
DST=backup@10.0.9.2:/backup/prod/data/

mountpoint -q "$SRC" || { echo "source not mounted, abort" >&2; exit 1; }
[ "$(ls -A "$SRC" | wc -l)" -gt 0 ] || { echo "source empty, abort" >&2; exit 1; }

rsync -aHAX --numeric-ids --delete "$SRC/" "$DST"

The other thing people get wrong constantly is the trailing slash. As the documentation puts it, a slash at the end of the source path means "copy the contents of this directory", and no slash means "copy this directory itself".

rsync -a /srv/data  /backup/    # /backup/data/... gets created
rsync -a /srv/data/ /backup/    # the contents land directly in /backup/...

Generation management can be implemented cheaply with hard links.

rsync -aHAX --numeric-ids --delete \
  --link-dest=/backup/prod/daily.1 \
  /srv/data/ /backup/prod/daily.0/

--link-dest hard-links to the file in the directory you name instead of copying an unchanged file. The result is that every generation looks like a full snapshot while the disk only holds the deltas.

4. tar — archive fundamentals

tar -czf /backup/etc-2026-08-15.tar.gz -C / etc
tar -tzf /backup/etc-2026-08-15.tar.gz | head
tar -xzf /backup/etc-2026-08-15.tar.gz -C /restore/

Going by the documentation, the options are these.

• -c (--create), -x (--extract), -t (--list), and -f (--file) are the base combination.
• Compression is -z (gzip), -j (bzip2), -J (xz), and --zstd (zstd).
• -C (--directory) changes directory before the work begins. Always use it so absolute paths never end up inside the archive.
• -p (--preserve-permissions) and --numeric-owner bring permissions and ownership back exactly on restore.
• --acls, --xattrs, and --selinux include ACLs, extended attributes, and SELinux contexts respectively. On a system running SELinux, restoring without these options leaves the service unable to start.
• --one-file-system does not cross into another filesystem. It prevents the accident where mounted network storage gets dragged along into the backup.
• --exclude and --exclude-from specify exclusion patterns.
• --listed-incremental manages the snapshot file used for incremental backups.
• --strip-components removes leading path elements at restore time.

A practical shape for backing up system configuration looks like this.

sudo tar -czf /backup/etc-$(date +%F).tar.gz \
  --acls --xattrs --selinux --numeric-owner \
  --exclude='/etc/shadow-' \
  -C / etc

Always check the path when you restore. Extracting straight with -C / overwrites the current system configuration. The safe procedure is to extract into a temporary path, inspect the contents, and then move only the files you actually need.

mkdir -p /restore/etc-check
sudo tar -xzf /backup/etc-2026-08-15.tar.gz -C /restore/etc-check
diff -r /restore/etc-check/etc/nginx /etc/nginx | head -40

GNU tar and BSD tar (bsdtar) differ in option handling and in some of their behaviour. The classic example is the extra extended-attribute files that appear when you unpack an archive made on macOS on a Linux box. Verify backup scripts on the distribution you actually run in production.

5. Consistency — data that changes mid-backup

If the application is writing to a file while you copy it, the backup is not a picture of any point in time at all, it is a mixture. This is especially fatal for databases.

The solutions come in three layers.

Layer 1 — application-level dumps. The most reliable and the most portable.

pg_dump -Fc -d appdb -f /backup/appdb-2026-08-15.dump
mysqldump --single-transaction --routines --triggers appdb > /backup/appdb.sql

--single-transaction obtains a consistent snapshot through transaction isolation. Check the exact options and limitations for each database in that product documentation.

Layer 2 — filesystem snapshots. LVM, Btrfs, and ZFS support them. Take a snapshot, back that up, and the point in time is frozen.

sudo lvcreate --size 10G --snapshot --name snap_data /dev/vg0/data
sudo mount -o ro /dev/vg0/snap_data /mnt/snap
sudo tar -czf /backup/data.tar.gz -C /mnt/snap .
sudo umount /mnt/snap
sudo lvremove -y /dev/vg0/snap_data

There is an important limitation here. An LVM snapshot is only a block-level point-in-time freeze; it does not reflect data the application was still holding in memory. For a database you need a procedure that forces a checkpoint or briefly halts writes just before the snapshot. On top of that, a snapshot volume that fills up gets invalidated, so size it generously.

sudo lvs -o lv_name,lv_size,data_percent,snap_percent

Layer 3 — continuous archiving. Use it when RPO has to come down to minutes. You keep archiving the database WAL or binary log and recover to an arbitrary point in time. Configuration and recovery procedures differ per product, so follow that documentation.

6. Verification — without a restore rehearsal it is not a backup

Split verification into three stages and you can do it without gaps.

Stage 1 — does the backup exist. Check that the file size is not zero and that the timestamp is recent. Surprisingly many problems get caught right at this stage.

ls -lh /backup/prod/ | tail -5
find /backup/prod -type f -mtime -1 | wc -l

Stage 2 — can the backup be read. Check compression integrity and the archive listing.

gzip -t /backup/etc-2026-08-15.tar.gz && echo 'gzip OK'
tar -tzf /backup/etc-2026-08-15.tar.gz > /dev/null && echo 'archive OK'
sha256sum -c /backup/checksums.sha256

Stage 3 — does it actually restore. This alone is real verification. Restore onto a different host or a temporary environment, bring the service up, and confirm the data is from the point in time you expected.

A rehearsal checklist is worth keeping in this shape.

• Are the credentials for reaching the backup store kept outside the production server
• Is the restore procedure document somewhere other than the laptop of the person doing the restore
• Are the encryption keys needed for the restore stored separately
• Does the time the rehearsal actually took fit inside the RTO
• Does the application start after the restore and pass its consistency checks

Encryption key storage is left out particularly often. Encrypt the backup and keep the key only on the server being backed up, and the moment that server disappears the backup becomes a meaningless binary file.

Once a quarter is a realistic minimum for the rehearsal interval. And the rehearsal has to be performed by someone other than the person who built the backup, working from the documentation alone, or the gaps in the procedure never surface.

7. Five ways a backup quietly rots

Way 1 — it is missing from the backup set. You added a new volume but never put it in the backup script. Manage the target list alongside the infrastructure definition so it gets picked up automatically.

Way 2 — you only see the success logs and never see the failures. cron is silent by default. You need something that checks the exit code and reports failures.

#!/usr/bin/env bash
set -euo pipefail
trap 'echo "backup FAILED at line $LINENO" >&2; exit 1' ERR

rsync -aHAX --numeric-ids --delete /srv/data/ /backup/prod/data/
echo "backup OK $(date -Is)"

The combination of set -euo pipefail and trap ... ERR is the minimum line of defence. It is far better to record the success timestamp to a file or to monitoring and watch whether the most recent success is within 24 hours.

Way 3 — ransomware gets replicated into the backup target along with everything else. Generation management and immutable storage are the defence. A setup that keeps only the newest backup is helpless in this situation.

Way 4 — the environment on the restore target system is different. When UID/GID mapping, SELinux context, kernel version, or filesystem features differ, the restore succeeds but the service does not come up. --numeric-ids and --selinux are the preparation against this.

Way 5 — the backup hurts production performance, which shrinks the backup window. You end up lengthening the interval, and at some point you stop meeting the RPO. Run it at lower priority.

ionice -c 3 nice -n 19 rsync -aHAX --numeric-ids --bwlimit=50M /srv/data/ /backup/prod/data/

--bwlimit lowers network bandwidth and ionice -c 3 lowers disk priority. Once the backup stops getting in the way of production, you gain the headroom to shorten the interval.

Quiz: check your understanding

mountpoint -q /srv/data || exit 1
[ "$(ls -A /srv/data | wc -l)" -gt 0 ] || exit 1
rsync -aHAX --numeric-ids --delete --dry-run /srv/data/ /backup/prod/data/

rsync -aHAX --numeric-ids /srv/data/ /backup/prod/data/

pg_dump -Fc -d appdb -f /backup/appdb.dump
mysqldump --single-transaction appdb > /backup/appdb.sql

Closing

Backup design that starts from the tool is guaranteed to end up mismatched. It has to start from which incident you have to undo, how much of it, and how fast. The answer to that is RPO and RTO, and everything else is derived from there.

And there is only one line to remember at the end. A backup you have never restored from is not a backup. Put a restore rehearsal on the calendar within this quarter. It has to be a rehearsal run by someone other than the person who built the backup, working from the documentation alone, with a clock running. That one rehearsal saves you hours during a real incident.

References

• rsync(1) — man7.org (verified 2026-08-15)
• tar(1) — man7.org (verified 2026-08-15)
• The official GNU tar manual (verified 2026-08-15)
• The official rsync site (verified 2026-08-15)

Further reading

• Previous: The complete Linux logging operations guide
• Next: The complete TLS certificate guide
• Open source backup tool comparison — the characteristics of restic, Borg, Kopia, and the rest
• The complete file descriptors and inodes guide — how hard links and inodes behave
• Linux terminal — practise rsync and tar options
• crontab parser — check backup schedule expressions