Fastly Global Outage 2021: A Single Customer Regex Took Down the Internet

Devsolved Jul 26, 2026 — 12:41 UTC MTTR: N/A 95005 views

Executive Summary

An undiscovered bug in the VCL (Varnish Configuration Language) compiler triggered a catastrophic global outage when a single customer updated their configuration with a specific regex pattern.

Symptom

At 09:47 UTC, 85% of Fastly's global network returned 503 Service Unavailable errors. Major websites including Amazon, Reddit, GitHub, and the UK Government went offline instantly.

Global Traffic Drop

The Investigation

Fastly engineers noticed the outage correlated perfectly with a customer pushing a configuration change. The configuration change itself was entirely valid, but it triggered a dormant bug in the edge software.

Code c

if (req.http.Fastly-FF) {
  set req.http.Fastly-FF = regsub(req.http.Fastly-FF, "^[^,]+,?", "");
}

Root Cause

The edge servers compile customer VCL down to native C code for performance. A bug in the C compiler logic caused the resulting executable to segfault when parsing a specific regex.

Because Fastly's edge POPs share the same binary, they all crashed simultaneously when processing requests for that customer.

Resolution

Within 49 minutes, engineers identified the specific customer configuration and disabled it. The edge nodes immediately recovered. Fastly then spent 36 hours rolling out a permanent patch to the VCL compiler across their entire fleet.

Timeline

1

09:47 UTC critical

Global 503 errors spike across 85% of network.

— System
2

10:27 UTC investigating

Identified the customer configuration causing the compiler segfault.

— Engineers
3

10:36 UTC resolved

Disabled the specific customer configuration, edge POPs recovered.

— Engineers

Join DevSolved