Search Developer War Stories & Outage Solutions

Devsolved • Jul 26

Resolved

Knight Capital 2012: $440 Million Lost in 45 Minutes

DIAGNOSTIC TRACE

critical

A deployment of new trading software left a dead code path on one of eight servers. When activated, the server went rogue, buying high and selling low at algorithmic speed.

6200 0 180009

Devsolved • Jul 26

Resolved

Knight Capital 2012: $440 Million Lost in 45 Minutes

DIAGNOSTIC TRACE

critical

A deployment of new trading software left a dead code path on one of eight servers. When activated, the server went rogue, buying high and selling low at algorithmic speed.

6200 0 180000

Devsolved • Nov 28

Resolved

AWS Kinesis 2020: US-EAST-1 Cascading Failure

DIAGNOSTIC TRACE

48h spent critical

Adding new capacity to the Kinesis front-end fleet pushed the servers over the operating system maximum thread limit, breaking internal communication and taking down Cognito, CloudWatch, and Lambda.

#aws

5930 0 120000

Devsolved • Jul 26

Resolved

AWS S3 US-East-1 Outage: The Typo That Broke the Cloud

DIAGNOSTIC TRACE

critical

An authorized S3 team member executed a command intending to remove a small number of billing servers. A typo in the command parameters removed a massive set of storage subsystem servers instead.

#aws

5120 0 110006

Devsolved • Jul 26

Resolved

AWS S3 US-East-1 Outage: The Typo That Broke the Cloud

DIAGNOSTIC TRACE

critical

An authorized S3 team member executed a command intending to remove a small number of billing servers. A typo in the command parameters removed a massive set of storage subsystem servers instead.

#aws

5120 0 110003

Devsolved • Jul 21

Resolved

CrowdStrike Falcon 2024: The Global BSOD Outage

DIAGNOSTIC TRACE

72h spent critical

A parameter count mismatch within Channel File 291 triggered an out-of-bounds memory read in the Falcon sensor kernel driver, causing 8.5 million Windows devices to BSOD globally.

4521 0 89003

Devsolved • Jul 26

Resolved

GitLab 2017: Primary Database Deleted via Accidental `rm -rf`

DIAGNOSTIC TRACE

critical

An engineer accidentally deleted the primary PostgreSQL database directory on the production server instead of the staging server, resulting in a 300GB data loss and an 18-hour outage.

4210 1 125022

Devsolved • Jul 26

Resolved

GitLab 2017: Primary Database Deleted via Accidental `rm -rf`

DIAGNOSTIC TRACE

critical

An engineer accidentally deleted the primary PostgreSQL database directory on the production server instead of the staging server, resulting in a 300GB data loss and an 18-hour outage.

4210 0 125000

Devsolved • Jul 26

Resolved

Fastly Global Outage 2021: A Single Customer Regex Took Down the Internet

DIAGNOSTIC TRACE

critical

An undiscovered bug in the VCL (Varnish Configuration Language) compiler triggered a catastrophic global outage when a single customer updated their configuration with a specific regex pattern.

3890 0 95010

Devsolved • Jul 26

Resolved

Fastly Global Outage 2021: A Single Customer Regex Took Down the Internet

DIAGNOSTIC TRACE

critical

An undiscovered bug in the VCL (Varnish Configuration Language) compiler triggered a catastrophic global outage when a single customer updated their configuration with a specific regex pattern.

3890 0 95001

Devsolved • Apr 12

Resolved

AWS Cost Optimization: The $40,000 S3 Bandwidth Bill

DIAGNOSTIC TRACE

12h spent high

Serving static assets directly from a public Amazon S3 bucket instead of using a Content Delivery Network (CDN) resulted in astronomical bandwidth costs and high global latency.

#aws

3840 0 72006

Devsolved • Nov 20

Resolved

Cloudflare 2025: The Bot Management Feature File Crash

DIAGNOSTIC TRACE

12h spent critical

A database permission change caused a duplicate-row query, doubling the size of an ML feature file. The oversized file caused memory exhaustion and crashed edge nodes globally, taking down sites across the internet.

#database

3105 0 65400

Devsolved • Jul 26

Resolved

BGP Route Leak: How a Small ISP Took Down Google

DIAGNOSTIC TRACE

critical

A tiny internet service provider in Nigeria accidentally advertised routes for Google's IP space. Major telecom networks blindly accepted these routes, blackholing traffic to Google services globally.

2801 0 75014

Devsolved • Jul 26

Resolved

BGP Route Leak: How a Small ISP Took Down Google

DIAGNOSTIC TRACE

critical

A tiny internet service provider in Nigeria accidentally advertised routes for Google's IP space. Major telecom networks blindly accepted these routes, blackholing traffic to Google services globally.

2800 0 75000