AWS S3 US-East-1 Outage: The Typo That Broke the Cloud

Devsolved Jul 26, 2026 — 15:48 UTC MTTR: N/A 110003 views

Executive Summary

An authorized S3 team member executed a command intending to remove a small number of billing servers. A typo in the command parameters removed a massive set of storage subsystem servers instead.

Symptom

In 2017, the AWS US-East-1 region experienced massive error rates for Amazon S3. Because S3 is foundational to AWS, this caused cascading failures across EC2, EBS, Lambda, and thousands of internet businesses.

Server Racks offline

The Investigation

The S3 team was debugging an issue with the billing system causing it to operate slowly. To fix it, an engineer needed to take a few servers offline.

Root Cause

The engineer executed a playbook script. However, one of the inputs to the script was entered incorrectly. Rather than removing the intended 2 servers, it executed against a significantly larger set of servers.

Code bash

aws s3api remove-servers --subsystem index --count 200  // Intended: --count 2

The servers removed supported two critical S3 subsystems: the Index subsystem and the Placement subsystem. They had a "cold start" problem that took hours to boot.

Resolution

AWS restored the Index subsystem after 4 hours. Following the incident, AWS removed the ability for tooling to execute capacity removals exceeding a predefined safety threshold, regardless of inputs.

Preventive Action Items

P1 Implement hard limits on capacity removal tooling @S3 Platform

Join DevSolved