I Lost Everything So You Don't Have To

The Background

It all started one fateful August day. I tried connecting to my NAS and… error. Over and over the network storage device I was so proud of had failed. After some diagnostics, looks like it was just one SSD that had failed. No problem– we can just resilver. After attempting a resilver (which took far longer than I expected) a second drive failed. 🫨

This is a story about how I lost terabytes of data and how you can avoid the same fate by learning to backup your own.

If you want to skip the story, here’s the “How to Do Backups” section.

When the final drive failed, it was still okay, because I hadn’t fully moved to the NAS yet. I kept a final backup of everything on an external HDD… This includes almost everything I’ve worked on and collected over the last few years except for what’s on GitHub. Games I’ve worked on, graphic assets, fonts, music, 3D models, code and so on.

While I waited for some new SSDs, somehow I stumbled on Hyprland. It’s a minimal tiling compositor for Wayland which can run the latest Linux kernels - something Pop!_OS lags behind on. I was excited to give it a go when the new drives arrived.

The replacement drives came and I plugged them into the NAS and started from scratch. Again, nothing was lost quite yet.

Once it was running again, rather than uploading my files to the NAS over the network which would take a while, I decided to just copy what I need from the current Linux install. I copied them onto the drive containing my only remaining backup.

This was the mistake I should have avoided. Copying such a large amount of data onto an HDD already on its last leg was something I just hadn’t thought through. When the copy finished (it took a few tries) I unplugged it, plugged it into the same computer and everything appeared to be fine.

Feeling a bit bold, I was certain that everything was saved. I even have a checklist so I can know for sure that I’m not forgetting anything:

Computer EOL Checklist

• Save SSH keys
• Save any GPG keys
• Backup the entire keychain for at least 1 year, longer if feasible
• Check for Photos, Documents, Desktop folder
• Check Downloads folder
• Deactivate any licenses
  • Music software, video editing software, design software, etc.
• Check docker for databases to backup
• Thunderbird config

Tragedy Stikes

But even that couldn’t save me. Booting into Hyprland, the first thing I wanted to do was check my data. After opening “Dolphin”, the default file explorer and checking for my drive… nothing.

Usually I’m pretty good at basic data recoveries, so I remained hopeful– I installed ntfs-3g, gparted, and some other disk repair / data recovery tools and spent the better part of two days hunting for even a single file to recover… nothing.

I’ve never seen a drive this badly bruised, even macOS wouldn’t recognize it– a rare sight. This is when I started to accept it might be a lost cause.

It was quite bleak there for a while, but 2 months have passed and the only real loss was the 3 weeks worth of work that I hadn’t been pushing to GitHub…

Lessons Learned

From this loss, I’ve had to learn some things the hard way.

1. Slow down and verify your backups before deleting data.
   • For me this means using a second computer to verify the files.
   • If the copy operation is taking too long, don’t take shortcuts to try and speed things up, it might be a sign there’s an issue or possibly even a better way.
2. Use remote storage whenever possible (🔑)
   • Do whatever you can to make sure that your workflow encourages frequent pushing.
3. Have at least two copies before deleting the last known working copy. Having a third copy is even better.
4. NAS and RAID != Backup. Now I know this to be true.

How to Do Backups

What I needed was:

• A service that is priced based on what I use
• Not vendor-locked
• Backs up my files with (optional) encryption

Previously I had defaulted to Backblaze. It’s… not good in my experience.

While researching what other people use online, two options that kept coming up were: Borg and Restic.

After looking at them for a while, they look very similar. The TL;DR is that Restic is easier to use and supports more cloud storage options out-of-the-box, but Borg has more advanced features and is more efficient (citation needed.)

Using Borg and Restic looks pretty confusing when you’re first getting started. When I first used Borg, I chose to use Vorta (a Borg GUI client) before switching to borgmatic for good. Vorta can schedule backups, view backups, and more without having to remember commands, which is great for beginners.

1. Sign Up for BorgBase

BorgBase is a simple storage host for Borg and Restic “repositories”. You can think of a repository like a bucket on S3. I’ve got one for my home directory and one for my LXC backups on the free plan.

2. Download a Borg Client

Vorta, borgmatic, etc. there’s plenty of options for clients depending on what suits you.

3. Initialize Repository

You’ll need to initialize the repo. At this step you can provide which encryption method to use. From what I understand, the encryption method cannot be updated later.

$ borg init -e repokey-blake2 ssh://[email protected]/./repo

4. Configure & Schedule Your Backups

If you’re using borgmatic, your config is a YAML file. If you’re using a GUI, the config lives in the GUI.

Here’s my borgmatic config. I’ve set it to exclude certain files/directories that can be easily reproduced to save on space.

source_directories:
  - /home/me/repos

exclude_patterns:
  - "*.pyc"
  - "__pycache__"
  - "node_modules"
  - "target"
  - "build"
  - "*.gguf"
  - "*.safetensors"
  - "_build"
  - ".elixir_ls"
  - ".DS_Store"
  - ".next"
  - ".venv"
  - "venv"

# Set this to false if you want to backup external filesystems (USB, NAS, etc.)
one_file_system: true

repositories:
  - path: ssh://[email protected]/./repo

exclude_caches: true

compression: auto,zstd
encryption_passphrase: "hunter2" # Not my real password
archive_name_format: "{hostname}-{now}"

# Number of times to retry a failing backup
# Needs recent Borgmatic version
retries: 5
retry_wait: 5

# Keep 3 backups, one per day, for the most recent 3 distinct days
# Keep up to 4 backups, one per week, for the most recent 4 distinct weeks
# Keep up to 12 backups, one per month, for the most recent 12 distinct months
# Older archives will be deleted
keep_daily: 3
keep_weekly: 4
keep_monthly: 12

# Don't read the whole repo each time
skip_actions:
  - check
check_last: 3

ssh_command: ssh -i /home/me/.ssh/id_rsa_personal

Set it to sync once every week, every few days, hours, etc. it’s up to you. Each sync will only track changes, meaning it doesn’t duplicate the same data for each sync. I recommend turning on an alert in BorgBase for your important repos so that if your backup doesn’t run for X amount of time, you’ll get an email (you can also configure it to send a webhook).

With that, Borg should be setup to backup your data on a schedule.

How to Read Backups

Creating a backup is one thing– reading from that backup is another. I’m happy to tell you that it is extremely quick and easy with this solution. BorgBase has the full guide here, but in short:

$ mkdir borg-mnt
$ borg mount ssh://[email protected]/./repo borg-mnt

That’s it– without any wait time you’ll have access to all of the “archives” within the repo and be able to inspect each file as if it were already on your disk:

$ mkdir borg-mnt
$ borg mount --rsh ssh://[email protected]/./repo borg-mnt
Enter passphrase for key ssh://[email protected]/./repo:
$ ls -lah borg-mnt
total 4.0K
drwxr-xr-x 1 me users    0 Feb 21 16:24 .
drwxr-xr-x 8 me users 4.0K Feb 21 16:23 ..
drwxr-xr-x 1 me users    0 Oct  1  2024 archlinux-2024-10-01-221515
drwxr-xr-x 1 me users    0 Oct  4  2024 archlinux-2024-10-04-232307
drwxr-xr-x 1 me users    0 Oct  5  2024 archlinux-2024-10-05-221555
drwxr-xr-x 1 me users    0 Dec  8  2024 archlinux-2024-12-08-093441
drwxr-xr-x 1 me users    0 May 16  2025 fedora-me-2025-05-16-194048
drwxr-xr-x 1 me users    0 May 16  2025 fedora-me-2025-05-16-195215
drwxr-xr-x 1 me users    0 Aug  2  2025 fedora-me-2025-08-02-153706
drwxr-xr-x 1 me users    0 Oct  5 22:54 fedora-me-2025-10-05-225454
drwxr-xr-x 1 me users    0 Jan 11 02:09 nixos-2026-01-11T02:09:12
drwxr-xr-x 1 me users    0 Dec 11  2024 pop-os-2024-12-11-005732
drwxr-xr-x 1 me users    0 Dec 14  2024 pop-os-2024-12-14-005748
drwxr-xr-x 1 me users    0 Dec 30  2024 pop-os-2024-12-30-175135
$ cat borg-mnt/pop-os-2024-12-11-005732/home/me/repos/scrap/polipo/atom.c
/*
Copyright (c) 2003-2006 by Juliusz Chroboczek

Permission is hereby granted, free of charge, to any person obtaining a copy
[...]

To disconnect, just use:

$ fusermount -u borg-mnt

Conclusion

It’s a shame, but most of the things I really want backed up are already backed up elsewhere. The loss was more sentimental than anything. And what happened to the 3 weeks of code I lost?

Well, it’s frustrating, but there’s always a silver lining. Writing and re-writing the same codebase a few times gave me a good feel for the bottlenecks and tech debt that may arise. Even though that code is lost, I still know what should have been done differently. This latest iteration is production grade.

Another silver lining to the whole situation is that I finally spent the time to setup a proper backup solution– one that actually works (looking at you Backblaze…). If you’re like me, and have data that you’d like to keep for the foreseeable future, I highly recommend this setup, your future self might thank you.