Backups done properly

I have a lot of VPS (I have a bad habit of collecting them). One of my main fears is losing the data I have. While I could build like 90% of it again from various sources, I would rather not go through the trouble when the inevitable happens. Like you know, accidentally missing the . on rm -r ./. Or a flood. Or just the hard drive deciding it’s done its job and going kapoof. Or getting ransomwared. Or realizing you did a misconfiguration and need to go back. Backups are just a great safety net.

This safety net also helps me experiment. If I want to play with SQL queries for example, backups let me do those without fear. If I am going to test a configuration in my Docker containers, I can do those. If I accidentally mess up my code, I can restore it. RAID cannot help me with any of that.

Just recently, I upgraded my Home Server’s RAM from 4 GB to 32 GB. The backups actually helped me feel at ease because even if I messed it up somehow, my data is still safe. Even if I accidentally spilled water or dropped a component, I wouldn’t be too worried.

Golden Rule: Your Provider’s Backup Does Not Count

One thing I know not to do is to trust the provider they are doing backups. I have read far too many threads where people thought they were safe but the data is corrupted, or the provider was charging them but not actually backing up. Or you know, the data center going up in flames and it turns out the backups were stored in that same data center.

In the 3-2-1 backup rule, the provider’s backup is not a copy. Never consider it as one. I always do my own backups which is why I never pay extra for it.

Plus, you don’t really have control over the provider’s backup. Sometimes, I need to backup more often than needed (because I am doing a lot of experiments). By doing it myself, this is not a huge issue.

Note that this also applies to the cloud. No - if you uploaded your data on AWS or Google Drive or whatever, it’s one copy. AWS has just deleted this person’s data and Google has deleted this organization’s data so you cannot say that the big providers are doing this properly. Not to mention some clouds just being unreliable - saying they backed up your copy but they actually did not. I still have a lot of beef with OneDrive for this.

Backup Journey

Phase 1: No Backups

I did not always backup. Thankfully, not much has happened since then but it’s not always going to be like it. Plus, no backups meant I was one hard drive failure or one ransomware away from everything.

So I started backing up. Besides, the benefits of backups are enormous even if I did not ever get a house fire (and I hope I never do).

Phase 2: Manual Copies

This is the first step I did. Every day or whenever I feel like it (mostly the latter), I created a zip file and then copied it to a flash drive or something. That was my backup. But there’s a few problems:

1. If I have 50GB worth of files, in just 30 days I will have used up 1.5TB of space. This is incredibly inefficient as some files do not change every day.
2. Zipping is an incredibly expensive operation. This isn’t viable once I get more data.
3. It’s a manual process. It was normal for me to go months without backing up because it was not automated.

Clearly, there needed to be some improvements. Frankly, this was just dumb.

Phase 3: Version Control

Then I started doing version control backups. Yes - Git. But Git has a problem: It is not meant for backups and it was not designed for it.

For example, backups can and do expire. There are some items I want to keep forever (like my family photos) but some I want to just delete after their use has been deemed complete (e.g. that random high school project). Git will not allow me to delete them without rebuilding the repository.

Plus, if I want to somehow back them up to a remote, and then restore one backup, the whole Git repo needs to be downloaded. Even if I only need one file.

Git was not made for backups so this is not a slight against it. It is a version control tool first and foremost - and a great one at that. But as a backup tool, it is terrible - though I would still say better than manual copy pasting.

Phase 4: RClone

Next was RClone. RClone is great as it provides deduplication. If I do rclone copy source destination, if I only changed one file, that one file will be the only one changed.

The problem though are versions. RClone excels in transfers and managing multiple remotes. But if for some reason I need a file from three months ago, unless I did not change that specific file, it will most likely be gone.

I can do versioning myself but again, this is going to end up being a manual process. Thankfully, I found a great solution.

Phase 5: Restic

After one week of using RClone, a Redditor recommended Restic. I love it immediately. It provides deduplication, compression, version control, and other tools very useful for backups. Encryption is a big plus: As long as I keep the key safe, providers never get access to the backups.

Encryption is pretty great too: It means I don’t really need to mind whether the provider is doing proper encryption or not. If all they are getting is encrypted blobs, even a hacker will not find it useful.

Backing Up Databases

One thing to note when backing up databases is to NOT just back up the files. This is very tempting to do but it is a bad idea because a process could be writing while you are doing the copies. This could leave your database corrupted when you restore it.

Most databases these days handle this better but given that backups are meant to be reliable, it is better to use the tools given by the databases themselves and pipe them to Restic. I will show how I back up Postgres and SQLite databases but you can search for how to back up MariaDB, MySQL, or any other database.

PostgreSQl

For example, I generally like PostgreSQL so the process of backing up the db is simple:

# Backup
pg_dump -U $USER $DATABASE | gzip | restic backup --stdin --stdin-filename mydb.sql.gz

I generally like compressing the backup. You can also not do this but I find it doesn’t add a lot of complexity so I just add it. This will back up the database to a file named mydb.sql.gz. All you have to do when restoring is to decompress the file and run the SQL:

# Restore
gunzip mydb.sql.gz
psql -U $USER $DATABASE -f mydb.sql

SQLite

Another is SQLite. In general, you can just back up the file if nothing is writing to it. However, there’s still a possibility of data corruption so it’s best to just use the tools already available:

# Backup
sqlite3 source.db ".backup backup.db"

# Restore
sqlite3 destination.db ".read backup.db"

What not to Back Up

There are things I do NOT back up - and this is a deliberate decision.

Anything I can easily obtain again are NOT backed up. That means Spotify media, Steam games, etc. If ever in the future Steam tells me that they will stop hosting a game for whatever reason, then I back it up. But if I can download it from their servers, I do not back it up.

Another are temporary files. These are temporary for a reason so there’s no need to save them.

OS files are the last ones. I don’t really want a full blown MacOS or Ubuntu Server in my files. If I suddenly need to upgrade, it’s hard to determine which files can go and which ones can stay.

Automatic Backups

However, these are all for naught if it turned out when disaster strikes that my last backup was 5 months ago. Automating is hugely important for me. Restic itself does not have a way to automate this since it relies on the OS schedulers to do this. That’s fine though since I am okay with the command line.

Cron Jobs

The easiest way for Linux is through cron. For reference, here’s my backup cron job:

0 0 * * * /opt/backup.sh

This runs every day at 12:00 in the morning. Every morning, I receive a summary of the backup results so I know if something is wrong.

The /opt/backup.sh is also pretty simple:

#!/bin/bash
# This does NOT remove old backups automatically. That is left to a trusted computer

if [ "$EUID" -ne 0 ]; then
    echo "This script must be run as root (use sudo)"
    exit 1
fi

export RESTIC_REPOSITORY="SET THIS"
export RESTIC_PASSWORD="SET THIS"

### BACKUP DATABASES ###

# Immich
echo "Saving Immich Database..."
export POSTGRES_PASSWORD="POSTGRES_PASSWORD"
LINKWARDEN_DB=$(docker compose exec -t database pg_dump "immich" -U "POSTGRES_USER")
echo "$LINKWARDEN_DB" | gzip | restic backup --stdin --stdin-filename "immich.sql.gz" --tag "immich"

### Files ###
echo "Saving Files..."
restic backup \
    /opt/stacks \
    /opt/files \
    --tag files

You can customize it further if you want. In my actual bash script, my RESTIC_REPOSITORY and RESTIC_PASSWORD is handled by my secrets manager but you can modify the script for your own setup.

That being said, I found this cumbersome when trying to backup personal devices like my Macbook or my work laptop. In fact, adding a Restic tool to Windows Task Scheduler got me on IT Radar. That was fun. So I used another tool that’s related to Restic.

Backrest

When I can use cron, I do use it. But for personal devices, I prefer Backrest. Backrest is nice because it does the automation and also provides a GUI - which is actually hugely important for people who are scared of the CLI.

All I have to do is add the repositories, create a plan, and let it do its job. If I need to do a manual backup, I can do that too. Restoring is also pretty easy and I have done so many many times. It’s an absolutely great tool. I just don’t use it if I can since it’s unnecessary overhead compared to just using cron. However, I can recommend it if you’re not tech-savvy or know someone who isn’t since it itself is an amazing piece of software.

Backup Strategy

Currently, my backup strategy is two copies on different local storages and two copies on different cloud providers. However, I do recommend one more thing: Test restores. A backup isn’t a backup until you’ve restored from it.

I have tested restores already when I moved some parts in a VPS to my local server. Thankfully, restic performed wonderfully. I got this confidence though since I actually tested my backups in advance. While I could have also just rclone copy the move, I deliberately decided to use restic to test the backups again.

It’s critical to test a backup before you need it because you never know it but your backup is actually corrupted. Or that your files are incomplete since you forgot to backup a folder. Or that your cloud provider had a maintenance for two days without telling you in advance.

Backups went from being an afterthought to a critical part of my infrastructure. The peace of mind is worth every gigabyte of storage.