Devsolved
Symptom
During a late-night database replication fix, an exhausted engineer attempted to clear a stuck PostgreSQL replication process on a staging node. Unfortunately, the engineer had multiple terminal windows open and ran a destructive command on the primary production node instead.
PostgreSQL replication cluster
The Commands Executed
The engineer intended to wipe the `db2.cluster.gitlab.com` (secondary) data directory to restart replication. Instead, they ran the following on `db1.cluster.gitlab.com` (primary):
Code
bash
sudo rm -rf /var/opt/gitlab/postgresql/data/
Within seconds, I realized my mistake and hit Ctrl+C, but 300GB of production data had already been deleted. Only 4.5GB remained.
Root Cause
The incident escalated into a catastrophic outage because 5 out of 5 backup mechanisms failed simultaneously.
- LVM Snapshots: Failed because the backup script was broken.
- Regular Backups: Failed due to a version mismatch between PostgreSQL pg_dump (9.2) and the server (9.6).
- Azure Disk Backups: Were enabled for the NFS server, but not the DB server.
- S3 Backups: Failed because the bucket was empty (the cron job was silently failing).
- Replication: The deletion instantly replicated to the remaining nodes.
Resolution
The team discovered an LVM snapshot taken 6 hours prior by a completely unrelated backup mechanism that was meant for staging environment refreshes. They spent 18 hours carefully copying the data over to a new production instance.
5 Whys Root Cause Drill-Down
1
Why was production data deleted?
An engineer ran rm -rf on the primary node.
2
Why did they run it on primary?
They thought they were connected to the staging node.
3
Why did the backups fail?
Cron jobs were silently failing without monitoring alerts.
Discussions 0