Devsolved
Symptom
At 9:30 AM, the NYSE opened. Immediately, Knight Capital's trading algorithms began generating millions of erratic orders on 150 different stocks. By 10:15 AM, they had accumulated a multi-billion dollar position and realized a $440 million cash loss.
Stock Market Chart
Root Cause
The engineers deployed the new code manually to 8 servers, but forgot to deploy it to the 8th server. When the new system sent orders with the `Power Peg` flag to the 8th server, the server executed a dormant, 8-year-old piece of code designed for testing. This code bought stocks relentlessly, ignoring price and volume limits.
Instead of immediately shutting down the system, engineers attempted to debug it live in production. They actually removed the 7 good servers from the routing table, sending 100% of traffic to the 1 broken server, multiplying the losses.
Resolution
The system was finally hard-killed at 10:15 AM. The catastrophic loss forced Knight Capital to secure a massive emergency bailout the next day, effectively wiping out the company's independence.
Code
javascript
if (flag === "PowerPeg") {
// Intended: Run new smart routing
// Actual (Server 8): Run 2003 test code that spams market orders
executeTestLoop();
}
Discussions 0