AWS Kinesis 2020: US-EAST-1 Cascading Failure

MTTR (Mean Time To Recovery): 14 hours

Symptom

At 5:15 AM PST, error rates spiked for Amazon Kinesis Data Streams in the US-EAST-1 region. Because Kinesis is a foundational service, this triggered a massive cascading failure across AWS. Cognito, CloudWatch, EventBridge, and Lambda all experienced severe degradation. Even the AWS Service Health Dashboard was unable to update because it relied on Cognito.

The Root Cause

Root Cause

The Kinesis front-end fleet uses a microservices mesh where every server must maintain communication with every other server to share "shard-map" data. The system was designed to use exactly one OS thread per connection.

To handle increased holiday traffic, AWS engineers added new servers to the fleet. As the new servers came online, they attempted to build connections to the existing fleet.

This addition pushed the total number of inter-node connections past the operating system's maximum allowed thread limit per process.

Unable to spawn new threads, the servers could no longer update their shard-maps, rendering them unable to route requests to the backend data clusters. The front-end fleet completely locked up.

Code java

java.lang.OutOfMemoryError: unable to create new native thread
  at java.lang.Thread.start0(Native Method)
  at java.lang.Thread.start(Thread.java:717)

Resolution Strategy

Resolution

Because the front-end servers were locked, AWS engineers had to slowly and manually remove the newly added capacity, and then carefully restart the front-end fleet in small batches to ensure the thread limits were not breached again during the cold-start phase.

To prevent this in the future, AWS migrated the Kinesis front-end to larger EC2 instance types. By using larger (but fewer) servers, the total node count in the fleet decreased, drastically reducing the number of OS threads required for the mesh communication.

Join DevSolved