I Lost Everything So You Don't Have To

The Background

It all started one fateful August day. I tried connecting to my NAS and… error. Over and over the network storage device I was so proud of had failed. After some diagnostics, looks like it was just one SSD that had failed. No problem - we can just resilver. After attempting a resilver (which took far longer than I expected) a second drive failed. 🫨

This is a story about how I lost terabytes of data and how you can avoid the same fate by learning to backup your own.

When the final drive failed, it was still okay, because I hadn’t fully moved to the NAS yet. I kept a final backup of everything on an external HDD… This includes almost everything I’ve worked on and collected over the last few years except for what’s on GitHub. Games I’ve worked on, graphic assets, fonts, music, 3D models, code and so on.

While I waited for some new SSDs, somehow I stumbled on Hyprland. It’s a minimal tiling compositor for Wayland which can run the latest Linux kernels - something Pop!_OS lags behind on. I was excited to give it a go when the new drives arrived.

The replacement drives came and I plugged them into the NAS and started from scratch. Again, nothing was lost quite yet.

Once it was running again, rather than uploading my files to the NAS over the network which would take a while, I decided to just copy what I need from the current Linux install. I copied them onto the drive containing my only remaining backup.

This was the mistake I should have avoided. Copying such a large amount of data onto an HDD already on its last leg was something I just hadn’t thought through. When the copy finished (it took a few tries) I unplugged it, plugged it into the same computer and everything appeared to be fine.

Feeling a bit bold, I was certain that everything was saved. I even have a checklist so I can know for sure that I’m not forgetting anything:

Computer EOL Checklist

• Save SSH keys
• Save any GPG keys
• Backup the entire keychain for at least 1 year, longer if feasible
• Check for Photos, Documents, Desktop folder
• Check Downloads folder
• Check for any crypto wallet private keys
• Deactivate any licenses
  • Music software, video editing software, design software, etc.
• Check docker for databases to backup
• Thunderbird config

Tragedy Stikes

But even that couldn’t save me. Booting into Hyprland, the first thing I wanted to do was check my data. After opening “Dolphin”, the default file explorer and checking for my drive… nothing.

Usually I’m pretty good at basic data recoveries, so I remained hopeful - I installed ntfs-3g, gparted, and some other disk repair / data recovery tools and spent the better part of two days hunting for even a single file to recover… nothing.

I’ve never seen a drive this badly bruised, even macOS wouldn’t recognize it - a rare sight. This is when I started to accept it might be a lost cause.

It was quite bleak there for a while, but 2 months have passed and the only real loss was the $400 in crypto I didn’t back up. Another big loss was the 3 weeks worth of work that I hadn’t been pushing to GitHub…

Lessons Learned

From this loss, I’ve had to learn some things the hard way.

1. Slow down and verify your backups before deleting data.
   • For me this means using a second computer to verify the files.
   • If the copy operation is taking too long, don’t take shortcuts to try and speed things up, it might be a sign there’s an issue or possibly even a better way.
2. Use remote storage whenever possible (🔑)
   • Do whatever you can to make sure that your workflow encourages frequent pushing.
3. Have at least two copies before deleting the last known working copy. Having a third copy is even better.
4. NAS and RAID != Backup. Now I know this to be true.

How to Do Backups

What I needed was:

• A service that is priced based on what I use
• Not vendor-locked
• Backs up my files with (optional) encryption

Previously I had defaulted to Backblaze. It’s… not good in my experience.

While looking what other people use online, two options that kept coming up were: Borg and Restic.

After looking at them for a while, they look very similar. I’ve heard of Borg in the past, and it seems like it has wide-spread support so I just picked it. Restic is (I think) the new kid on the block.

Using Borg and Restic looks pretty confusing when you look at the terminal commands, but I’ve been using Vorta (a Borg GUI client) with very little friction. It can schedule backups, view backups, and more without having to remember commands.

1. Sign Up for BorgBase

BorgBase is a simple storage host for Borg and Restic “repositories”. You can think of a repository like a bucket on S3. I’ve got one for my code and one for my VM snapshots on the free plan.

2. Download a Borg Client

Vorta, borgmatic, etc. there’s plenty of options for clients depending on what suits you. I’m happy with Vorta on my desktop as a GUI and borgmatic on headless servers.

3. Connect to BorgBase

Just like Git, the next step is to show the client how to connect to the remote repository. It’s as simple as pasting a URL that BorgBase provides. It looks something like: ssh://[email protected]/./repo

4. Configure & Schedule Your Backups

Set it to sync once every week, every few days, hours, etc. it’s up to you. Each sync will only track changes, meaning it doesn’t duplicate the same data on every sync.

With that, Borg should continue to backup data on a schedule.

It has a configuration for compression and I know Vorta has presets for excluding certain file types and folders (e.g. node_modules). This can tremendouly cut down on the amount of space the backups require.

Conclusion

It’s a shame, but most of the things I really want backed up are already backed up elsewhere. The loss was more sentimental than anything… aside from the $400. And the projects I was working on?

Well, it’s frustrating, but there’s always a silver lining. Writing and re-writing the same codebase a few times gave me a good feel for the bottlenecks and tech debt that may arrise. Even though that code is lost, I still know what should have been done differently. This latest iteration is production grade.