DevSolved Logo DevSolved
Search... ⌘K
Write Log In Sign Up
Community Feed

Join DevSolved

Share war stories & investigate outages.

Sign Up for Free Log In to Account
ESC
↵ Search all
ESC Close
Pro tip: Type at least 2 characters
HIGH Resolved

Git Force-Push Catastrophe: The Vanishing Database Migration

Avatar Devsolved Oct 12, 2023 — 09:00 UTC MTTR: 8 hours 2 min read 195001 views
Executive Summary

A developer trying to clean up their commit history using an interactive rebase and a force push accidentally erased a critical database migration from the `main` branch, breaking staging deployments for two days.

The Symptom

Symptom

The automated staging deployment pipeline started failing with a fatal ORM error: `column "stripe_customer_id" does not exist`. Looking at the codebase, the migration file that added this column was completely missing, even though the pull request that supposedly added it was marked as "Merged" two days prior.

Timeline

  • 1
    Monday 10:00 update

  • 2
    Monday 11:30 update

  • 3
    Tuesday 09:00 update

  • 4
    Tuesday 09:15 update

  • 5
    Tuesday 09:30 update

  • 6
    Tuesday 10:00 update

Root Cause Analysis

Root Cause

This was a classic Git history rewrite collision. Developer B was working on a long-lived branch. When they executed `git rebase main`, they were confronted with a complex merge conflict regarding the database schema version file. In a panic to resolve the conflict, they accidentally accepted "their" changes (the outdated version of the schema) over the incoming changes, essentially erasing Developer A's commit from their local history. Because they had to use `git push --force` to update their PR branch after the rebase, the PR didn't show the deleted file clearly. When merged (using a Squash merge), the resulting commit on `main` effectively reverted Developer A's migration.

Git is a powerful weapon. An interactive rebase followed by a force-push is like juggling chainsaws; eventually, you'll drop one on your foot.

Resolution

Resolution

We had to utilize `git reflog` on the build server to identify the exact commit hash where the migration was lost. We then cherry-picked the lost migration commit back onto `main` and generated a new database schema lockfile. To prevent this permanently, we implemented branch protection rules on GitHub, absolutely forbidding force pushes to `main`, and requiring branches to be "Up to date" before merging (blocking merges if the base branch has advanced).
Code bash 
# Finding the lost commit
git log --all -S "stripe_customer_id"
# Found hash 8a9b2c. Cherry-picking it back.
git cherry-pick 8a9b2c

Preventive Action Items

critical Enable "Require branches to be up to date before merging" in GitHub settings @DevOps
high Host an internal workshop on safe rebasing and `git push --force-with-lease` @Engineering Management
|

Discussions 0

Most recent