Devsolved
The Symptom
Symptom
During peak load, about 1% of API requests randomly failed with `502 Bad Gateway`. The Node.js application logs showed absolutely zero errors. The Nginx Ingress logs in Kubernetes showed `upstream prematurely closed connection while reading response header from upstream`.
Root Cause Analysis
Root Cause
This is a classic TCP race condition. Node.js has a default `keepAliveTimeout` of 5 seconds. Nginx has a default `keepalive_timeout` of 60 seconds (but was configured to 75s in our cluster). Nginx would reuse an idle TCP connection that had been open for 4.9 seconds and send a new request. At that exact millisecond, Node.js decided the connection was idle for 5 seconds and abruptly closed the TCP socket. Nginx received a RST packet while trying to send data, resulting in a 502 error for the user.
5 Whys Root Cause Drill-Down
1
Why #1
2
Why #2
3
Why #3
4
Why #4
Resolution
Resolution
We explicitly configured the Node.js Express server to have a `keepAliveTimeout` that is *strictly greater* than the Nginx timeout. We set Node.js to 65000ms (65s), ensuring that Nginx is always the one to gracefully close idle connections, preventing the race condition.
Code
javascript
// The fix: Ensure upstream timeout > proxy timeout
const server = app.listen(8080);
server.keepAliveTimeout = 65000; // 65 seconds
server.headersTimeout = 66000; // Keep-alive + 1s
Discussions 0