Devsolved
The Symptom
Symptom
Timeline
-
1
-
2
-
3
-
4
-
5
Root Cause Analysis
Root Cause
Kafka relies on consumer heartbeats to know if a consumer is alive. However, Kafka also enforces a `max.poll.interval.ms` (default 5 minutes). This means a consumer *must* finish processing its batch of messages and call `.poll()` again within 5 minutes. Because our database write latency spiked, processing a single batch of 500 messages suddenly took 6 minutes. The Kafka broker assumed the consumer was dead (livelock), kicked it out of the group, and triggered a "Rebalance" to reassign the partitions to other consumers. During a rebalance, ALL consumers in the group pause processing. Once the rebalance finished, a new consumer picked up the exact same heavy batch, took 6 minutes, got kicked out, and triggered ANOTHER rebalance. This loop paralyzed the entire 50-node cluster.
The crux of the issue was a fundamental misunderstanding of how Kafka handles backpressure. We thought that as long as the background heartbeat thread was ticking, Kafka would be happy. We didn't realize the main processing thread had a strict deadline.
// The fatal configuration (using defaults)
const consumer = kafka.consumer({ groupId: 'analytics-group' })
await consumer.run({
eachBatch: async ({ batch }) => {
// If this takes > 5 minutes (300000ms), Kafka kills the consumer
await insertIntoDatabase(batch.messages);
}
})5 Whys Root Cause Drill-Down
Resolution
Resolution
// The fix: Tune batch sizes and timeouts
const consumer = kafka.consumer({
groupId: 'analytics-group',
maxWaitTimeInMs: 100,
})
await consumer.run({
eachBatchAutoResolve: true,
partitionsConsumedConcurrently: 1,
// Keep batch small so it processes fast
})
Discussions 0