Devsolved
The Symptom
Symptom
At the end of the month, the cron job responsible for generating invoice summaries failed repeatedly. The Node.js application logged a MongoError: `PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 104857600 bytes`.
Root Cause Analysis
Root Cause
The aggregation pipeline used a `$match`, followed by a `$sort`, and finally a `$group` stage across 5 million records. MongoDB enforces a strict 100MB memory limit per stage for aggregations. Because the `$sort` stage occurred *before* the data was grouped and reduced, it attempted to hold all 5 million full documents in RAM simultaneously, immediately blowing past the 100MB limit.
Code
javascript
db.invoices.aggregate([
{ $match: { status: "paid" } },
{ $sort: { createdAt: -1 } }, // FAILS HERE: Trying to sort 5M docs in RAM
{ $group: { _id: "$customerId", total: { $sum: "$amount" } } }
])Resolution
Resolution
We added `{ allowDiskUse: true }` to the aggregation options, which tells MongoDB to spill the excess sorting data to temporary files on the disk instead of failing. We also optimized the pipeline by moving the `$sort` stage *after* the `$group` stage, significantly reducing the amount of data that needed to be sorted.
Discussions 0