DevSolved Logo DevSolved
Search... ⌘K
Write Log In Sign Up
Community Feed

Join DevSolved

Share war stories & investigate outages.

Sign Up for Free Log In to Account
ESC
↵ Search all
ESC Close
Pro tip: Type at least 2 characters
HIGH Resolved

AWS Lambda Cold Start Avalanche: The Serverless Timeout

aws
Avatar Devsolved Oct 30, 2023 — 09:30 UTC MTTR: 7 hours 1 min read 122000 views
Executive Summary

A sudden burst of marketing traffic triggered thousands of concurrent AWS Lambda executions. The 4-second cold start time caused API Gateway to timeout, resulting in a 100% error rate for new users.

The Symptom

Symptom

During a major marketing push, our serverless signup API dropped all traffic. API Gateway returned `504 Gateway Timeout`. CloudWatch logs showed thousands of new Lambda environments spinning up, but execution times were hitting 5.5 seconds.

Root Cause Analysis

Root Cause

Our backend was written in Java/Spring Boot and deployed as a monolithic AWS Lambda function. When a Lambda function experiences a "cold start" (initializing a brand new container), the JVM has to boot, Spring Boot does component scanning, and DB connections are established. This took ~5 seconds. Because traffic spiked instantly from 10 to 500 req/sec, AWS spun up 490 new Lambda containers simultaneously. All of them hit the 5-second cold start penalty. However, API Gateway had a hard timeout of 3 seconds.

Resolution

Resolution

We enabled "AWS Lambda Provisioned Concurrency" for the signup endpoint, keeping 100 containers permanently warm. Long-term, we migrated the Spring Boot monolith to a lightweight Node.js/Go function, reducing cold start times to 150ms.

Preventive Action Items

critical Configure Provisioned Concurrency for all critical user path Lambdas @Backend Team
high Rewrite Java monolith Lambdas into smaller Go functions @Architecture Team
|

Discussions 0

Most recent