Database Disaster Recovery in Kubernetes

This is part two of the “MySQL Group Replication in Kubernetes” article. You can find the first article here.

So, database disaster recovery is used to describe the activities that need to be done to recover the database and the data from a disaster. The Disaster can be a cyber attack, a mistake you did, or anything. It’s better to know how to recover the database from a situation like that.

Let’s make a disaster 💣

We all know MySQL stores data in the default directory of “/var/lib/mysql”. In our statefulset, we mount a persistent volume to that path in the pod containers. So all the database data will be stored in that persistent volume. We don’t lose data when something happens to the pod. Data will remain in the persistent volume.

But we need to make a disaster to demonstrate the recovery process. So let’s delete everything in the master server's persistent volume.

1. First, access the bash shell of the “mysql-0” pod container.

kubectl exec -it mysql-0 -- /bin/bash

2. Delete everything inside the “/var/lib/mysql” directory.

rm -rf /var/lib/mysql/*

Now we have a disaster. 🤯

Recovery plan ✅

When spinning up a new slave database pod, it will automatically replicate all data from another database pod to the new pod’s persistent volume. This is the way we are going to use for the recovery process.

So first, we can convert the master database into a slave database and a slave database into the master database. Then we can delete the “mysql-0” pod and its persistent volume. So as I mentioned before, statefulset will spin up a new “mysql-0” pod again and replicate all data into its new persistent volume. Then we can revert back the master/slave database conversions as previous.

Let’s do the disaster recovery 🤠

1. First, replace the statefulset. This will convert the master database (mysql-0) into a slave database and a slave database (mysql-1) into the master database.

kubectl replace -f mysql-statefulset-dr.yml

This is the same statefulset we deployed before but with some modifications (You can find those modifications in lines 33, 55 and 57).

2. Delete the “mysql-0” pod and its persistent volume.

kubectl delete pod mysql-0 & kubectl delete pvc data-mysql-0 && fg

After deleting the pod and PV, the new statefulset will spin up the “mysql-0” pod again as a slave database. So the pod will automatically clone the data from another pod to the newly created empty persistent volume.

This step is not mandatory but it’s better to follow it because we can make sure everything will replicate as expected.

3. Finally, replace the statefulset with the original one. This will revert back the master/slave database conversions as previous.

kubectl replace -f mysql-statefulset.yml

Now database recovery process is done. Everything is good to go. You can recover from a disaster scenario like that within 1 hour.

Hope you found this article helpful! See you in another article 🫡

Thank you!

👉 Follow me on Twitter 🐦 for more updates 👈