When I first started self-hosting the main website and instances for Lemmy, Matrix and PeerTube, I knew I was at risk of severe consequences if I ever lost this data.

Because of this, back in May I started making a backup script to back everything up which I planned to frequently run. I ran my first backup with that script but I couldn’t be bothered to do the rest of the work, which was automating this.

My plans were shifted to automate backups after the release of version 1.5.0, because why would I choose to do tedious unnecessary sysadmin work with no direct benefit when I could do the more exciting work of developing my game.

This plan was working fine, until 2 September when I opened my PeerTube instance and it failed to fetch notifications. I was curious why I couldn’t see my notifications, so I did some digging by opening the network section of Firefox/Librewolf’s developer tools and inspected the failed request. What I saw was alarming, an error that looked something like “invalid page in block xxx of relation base/xxx/xxx”. After a bit of research, I found out that things were as bad as I thought, the database was corrupted.

It wasn’t just that PeerTube database that got corrupted, as the database of the Lemmy instance inside my Docker container also got corrupted, but luckily other important files in my Matrix homeserver and the main website seemed to be untouched.

This resulted in me losing all Lemmy posts and PeerTube videos since May, but luckily I kept copies of the important content, that is, the blogs and the videos. Over the past two weeks I have reinstalled everything, restoring what I had from the previous filesystem if available, and automatically restoring Lemmy and PeerTube data from the May backup. This is the reason why some of my servers were temporarily down. From now on things should have high uptime.

Right now I am still in the process of manually restoring blogs and videos made after the May backup. Once I complete that I will finish my backup automation and make sure that this never again happens. Development will resume once I’ve finished automating this backup process.

Update: All blog posts and videos have been reposted! Note that these are new posts, so the date posted and other metadata like the post ids aren’t preserved.