DevSolved Logo DevSolved
Search... ⌘K
Write Log In Sign Up
Community Feed

Join DevSolved

Share war stories & investigate outages.

Sign Up for Free Log In to Account
ESC
↵ Search all
ESC Close
Pro tip: Type at least 2 characters
CRITICAL Resolved

Cloudflare 2025: The Bot Management Feature File Crash

database
Avatar Devsolved Nov 20, 2025 — 14:30 UTC MTTR: 12 hours 65401 views
Executive Summary

A database permission change caused a duplicate-row query, doubling the size of an ML feature file. The oversized file caused memory exhaustion and crashed edge nodes globally, taking down sites across the internet.

MTTR (Mean Time To Recovery): 3 hours 15 minutes

Symptom

At roughly 15:45 UTC, traffic passing through Cloudflare began dropping severely. Millions of users attempting to visit Cloudflare-protected websites encountered 502/503 HTTP errors. The system entered a global "fail-closed" state.

Architecture Context

Cloudflare utilizes a sophisticated Bot Management system powered by Machine Learning. Every 5 minutes, an internal pipeline queries a ClickHouse database to generate a "feature file" which is then propagated to every edge node globally.

The Root Cause

Root Cause

An infrastructure engineer performed a routine permission update on the ClickHouse database. This inadvertently triggered a bug in the query logic, causing it to return duplicate rows.

Because of the duplicate rows, the generated Machine Learning feature file doubled in size from 50MB to 100MB.

When this oversized file was pushed to the edge network, the service responsible for parsing it (which had strict memory allocation limits) crashed due to Out-Of-Memory (OOM) errors. Because the Bot Management system is designed to "fail-closed" (block traffic when unsure), it began blocking legitimate internet traffic globally.

5 Whys Analysis

5 Whys Root Cause Drill-Down

1
Why did internet traffic drop globally?
The edge proxy crashed and defaulted to a fail-closed state.
2
Why did the edge proxy crash?
It ran out of memory while parsing the Bot Management ML feature file.
3
Why did it run out of memory?
The feature file was exactly twice its normal size.
4
Why was the file twice its normal size?
The internal pipeline query returned duplicate rows.
5
Why did the query return duplicate rows?
A routine permission update on the ClickHouse database altered the query execution plan.

Resolution

Resolution

Engineers identified the oversized feature file and disabled its distribution. They reverted the ClickHouse database permission change and restarted the Bot Management services across the edge fleet.
|

Discussions 0

Most recent