restic for creating backups of my home directory
of my laptop. Now it has become my main solution for backing up my most valued data.
The old script
In the olden days, when I was still using CD-ROMs (or actually CD-RWs) to store backups, I had three snapshots (each a set of two CDs). When I created a new backup, I rotated them by overwriting the oldest disks.
My actual sets of CDs, last used in 2005: red was the most recent backup, yellow one version older and the set of CDs in the green case contains the oldest backup
Later on, when I was using an external hard disk, I had a script that made incremental backups using hard links so I could have multiple full backups, but not need x times the space. Unfortunately I lost that script.
After not creating backups for a while, I wrote a Bash script which handled the whole backup process:
- opening the (LUKS encrypted) external drive,
- mounting it,
- performing the backup, and finally
- unmounting and closing the device again.
The following line was handling the actual backing up:
rsync -av --delete --exclude-from=backup.excludes /home $MOUNT_DIR/$(hostname -s)/
Plain and simple.
This protected me from losing my data if my machine got stolen or if the hard drive failed. And it served me well for a number of years. But on multiple occasions I would have loved to have an older version of a file that got changed or even deleted before my last backup. This triggered me to look for another solution a few years back.
Restic to the rescue
Since I didn’t want to reinvent the wheel myself again, I was looking for backup
solutions and ran across restic. After a bit of reading
and experimenting, I basically replaced the rsync line from the previous
script with the following:
restic -r $MOUNT_DIR --verbose backup "/home/${USER}" --exclude-file="/home/${USER}/backup.excludes"
And then I added a few more commands to the script:
- “
restic -r $MOUNT_DIR check” to check the repository for errors - “
restic -r $MOUNT_DIR forget --keep-last 7 --prune” to only keep the last 7 snapshots
Benefits
What did this change bring me?
For starters, if we assume I make a backup each week, I can go back at least 7 weeks in time. In practice I’m not that consistent anymore so I can go back even further in time.
What’s more: according to restic each of my backups is about 105–100 GB of data. But due to the compression and deduplication these 7 (incremental) backups only use 69 GB combined.
Another benefit is that the backups are encrypted. So if I would want to store them in the cloud, I can be sure the cloud provider cannot access my data.
And speaking of which: while I use local storage (my external disk) and a remote REST Server (see below), restic supports a bunch of other backends: Amazon S3, Azure blob storage and Google Cloud Storage to name a few.
A backup is worth nothing if you cannot restore from it. Besides restoring a full snapshot, with restic I can also mount the snapshots to navigate the files and view them individualy. This allows me to browse around in my snapshot just like any other file system.
In case you had not noticed yet: I’ve become a fan of restic.
Next level
Initially I was only using restic to make backups of my laptop to an external hard drive. But recently I’ve taken it to a next level and use it for offsite backups as well.
This way any local machine (like my NAS) only needs to be able to run restic and I’m good to go. I just point restic to the internal VM and it handles the VPN for me.
restic -r rest:https://<haproxy_host>/<repository>
The reason I put Nginx in front of the Rest Server is so that it can handle the authentication and TLS termination. Sure, the Rest Server itself can also do this, but I feel Nginx is more battle tested.
This setup allows me to automatically run my backups each night since I do not have to attach a USB disk. It also gives me an offsite backup, without a monthly bill from a storage provider. What’s more: it gives me a sense of control. I know exactly how everything works and if disaser strikes and I need to restore all my data, I can bring the remote machine home and have local access to it.
Setting this up was something I have been putting off for a while now, but this project is finally finished.
Useful resources
Some of the resources that helped me:
To be honest, I do not recall why I chose restic over BorgBackup (Borg for short) at the time. But if you are in the market for a tool to create backups with deduplication, compression and encryption, you might want to check out both projects and see which fits your situation best.
]]>