Free Fire
Symptom
🔍
Website Offline: Users attempting to access GitLab.com faced a 502 HTTP error page.Data Disappearance: Internal teams noticed web requests to the
database failing entirely as the underlying files were being actively erased.
Replication Drop: Monitoring charts showed the primary database node suddenly disconnecting from the cluster and dropping to zero disk usage.
Symptom
🔍
Website Offline: Users attempting to access GitLab.com faced a 502 HTTP error page.Data Disappearance: Internal teams noticed web requests to the
database failing entirely as the underlying files were being actively erased.
Replication Drop: Monitoring charts showed the primary database node suddenly disconnecting from the cluster and dropping to zero disk usage.
Investigation
🕵️
Once the alarm sounded, the team undertook a frantic diagnostic process:Step 1 (Scope the Damage): The engineer realized the mistake within seconds and cancelled the rm -rf command, but 300GB was already reduced to just ~4.5GB of fragmented files.
Step 2 (Backup Audit): The team systematically checked the automated backup cron jobs, uncovering that every single layer of redundancy was producing empty files or throwing unmonitored errors.
Step 3 (The Lifeline): Engineers combed through manual local drives and found a stroke of luck—a developer had manually run a test backup 6 hours before the incident for an unrelated staging deployment.
Root Cause
🔀
The incident was caused by a combination of human error and invisible systemic configuration failures:Human Error (The Trigger): The engineer had multiple terminal windows open. He intended to wipe a buggy secondary replica database (db2.cluster) to restart replication from scratch, but executed the command on the primary database (db1.cluster).
Silent Backup Failures (The Catastrophe):LVM Snapshots: Failed because the script was written to ignore errors and the database version mismatch caused it to output 0-byte files.
AWS S3 Buckets: Remained empty because they were configured to sync the (now non-existent) LVM snapshots.
Azure Backups: Failed due to mismatched database cross-region configurations.pg_dump: The standard Postgres dump script crashed silently every night because it ran out of memory, and no alerting mechanism was tied to its failure.
Resolution
🛠️
Instead of panicking behind closed doors, GitLab chose absolute transparency:The Restore: The team used the 6-hour-old manual database dump, transferred it to a fresh production environment, and meticulously re-indexed the database to bring GitLab back online.
Radical Transparency: GitLab created a public Google Doc tracking the live notes, opened a public issue tracker, and live-streamed their internal recovery war-room on YouTube so the entire world could watch them debug in real-time.Hardening the System: Post-incident, GitLab completely re-engineered their backup alerts, modified terminal prompts to clearly highlight production vs. staging environments, and banned raw destructive commands on live servers without multi-party authentication.
Discussions 1