Devsolved
Financial Impact: 15% of inbound Stripe webhooks were dropped and had to be manually replayed.
The Symptom
Symptom
Our Datadog dashboards started alerting heavily for `PodRestartRate`. The payment microservice was constantly restarting. `kubectl describe pod` revealed the Reason as `OOMKilled` with an exit code of 137. However, our APM (Application Performance Monitoring) inside the JVM showed memory usage was perfectly stable.
Root Cause Analysis
Root Cause
We allocated 2GB of RAM to the Kubernetes pod limits (`limits: memory: "2Gi"`). However, the Java application was started without explicit `-Xmx` heap size flags. By default, older Java 8 runtimes look at the *host node's* total physical RAM (which was 64GB on our AWS instances) and dynamically set the heap size to 1/4th of that (16GB). The JVM thought it had 16GB of RAM to use, but the Linux cgroup enforced by Kubernetes brutally killed the process the millisecond it crossed the 2GB limit.
Code
yaml
resources:
requests:
memory: "1Gi"
limits:
memory: "2Gi"
# The fatal flaw:
# command: ["java", "-jar", "app.jar"]Resolution
Resolution
We upgraded the container base image to a modern Java 17 runtime that respects Linux cgroups (`-XX:+UseContainerSupport`), and explicitly configured the JVM to use a percentage of the container's memory using `-XX:MaxRAMPercentage=75.0`.
Discussions 0