Wongm's Rail Gallery » News ยป Shooting myself in the foot

Shooting myself in the foot

Thursday 22 July 2021 10:12 PM | Categories:

The other night I did my usual updates to the web server that drives this website, and in the process I managed to blow away all of my data! Thankfully it's all back to normal now, and it only took me a few hours of wasted effort!

The backstory

My attempt to upgrade my webserver nuked the symlink to the data directory, then my attempt to recreate the symlink to get stuff working again, I deleted the actual data!

I have automated backups of the web server and the MySQL database, but I don't have automated backups of the data directory.

I eventually worked out how to recreate a working symlink to the data directory - I never documented it in my server build plan.

The next step was to reupload the 78900+ deleted files from my local backups, then check to see how out of sync that backup is!

On the upside, I had the database side of the restore working rather quickly and all of the metadata was available - I just need the files they describe to hurry up and get onto to the cloud - only 27.3 GB of data to upload! 

By monkeying around with my SFTP settings I was able to max out my upload bandwidth - have gone from 8 Mb/s to 9.5 Mb/s. Eventually it finished at 7.30am this morning.

The rest of the time has been syncing the out of date directory tree of files with what the database says was the "real" hierarchy. I did this by running SQL queries to generate bash scripts to move files to where they're supposed to go, and then create another set of scripts to delete duplicate files.

Viewed 695 times.