Kubernetes Exit Code 137 (OOMKilled): Root Cause, Cgroup v2 & Production Triage

1. Linux Cgroup v2 Mechanics & OOM Score Architecture

Modern Linux distributions running Kubernetes (v1.25+) utilize cgroup v2 to enforce hierarchical resource isolation. When you define resources.limits.memory: 2Gi in your pod specification, the container runtime (containerd or CRI-O) configures the kernel cgroup file memory.max.

The Linux kernel tracks both active memory and page cache. As your application allocates buffers, the kernel attempts memory reclamation by evicting cached pages. When active anonymous memory exceeds memory.max and no reclaimable file cache remains, the kernel's memory controller fires the out-of-memory killer, issuing an unmaskable SIGKILL to the parent PID of the container.

2. Node.js, Python & JVM Runtime Memory Traps

Different application runtimes exhibit distinct memory allocation behaviors inside Docker containers:

Node.js & V8 Engine Heap Calibration

In Node.js versions prior to v20, the V8 garbage collector calculates its heap allocation ceiling based on the host OS rather than the container cgroup limit. If a container is restricted to 1GB RAM on an AWS EC2 instance with 32GB RAM, Node.js will not trigger aggressive garbage collection until memory reaches 2GB+, causing the Linux kernel to terminate the container prematurely.

The Fix: Always pass the --max-old-space-size flag to the Node.js binary, setting it to approximately 75% of your container memory limit:

# For a container with resources.limits.memory: 2048Mi
CMD ["node", "--max-old-space-size=1536", "server.js"]

Java Virtual Machine (JVM) Container Flags

Modern OpenJDK runtimes (Java 11+) are container-aware, but default heap settings can allocate up to 50% of available RAM, leaving insufficient memory for off-heap buffers, Metaspace, and native thread stacks. Configure the JVM with:

ENV JAVA_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=50.0"

3. Real-Time Kubectl Diagnostic Commands

When an alert fires for a crashing pod, execute these diagnostic commands in sequence to isolate the failure domain:

# 1. Check pod status and previous termination reason
kubectl get pod <pod-name> -n production -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

# 2. Inspect kernel OOM events in cluster events log
kubectl get events -n production --field-selector reason=OOMKilled

# 3. Identify top memory-consuming pods on the node
kubectl top pods -n production --sort-by=memory

# 4. View real-time node allocatable memory pressures
kubectl describe node <node-name> | grep -A 5 "Conditions:"

4. Production Kubernetes Manifest Optimization

To eliminate random evictions and establish predictable container scheduling, configure your deployment manifests using the **Guaranteed QoS Pattern**:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
  namespace: production
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: api-server
        image: production-registry.internal/api:v4.2
        resources:
          requests:
            cpu: "1000m"
            memory: "2Gi"  # Set equal to limits for Guaranteed QoS
          limits:
            cpu: "2000m"
            memory: "2Gi"
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 10
        env:
        - name: NODE_OPTIONS
          value: "--max-old-space-size=1536"

5. Prometheus Alerting Rules for Memory Starvation

Configure Prometheus and Alertmanager to warn your SRE team when container memory usage approaches 85% of its hard limit before the kernel OOM killer triggers:

groups:
- name: kubernetes-memory-alerts
  rules:
  - alert: ContainerMemoryNearLimit
    expr: sum(container_memory_working_set_bytes{container!=""}) by (namespace, pod, container) 
          / sum(container_spec_memory_limit_bytes{container!=""}) by (namespace, pod, container) * 100 > 85
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "Container {{ $labels.container }} in pod {{ $labels.pod }} at {{ $value | printf \"%.1f\" }}% of memory limit"
      runbook_url: "https://xseedinfo.com/articles/kubernetes-exit-code-137-oomkilled.html"

6. Vertical Pod Autoscaler (VPA) Recommendation Engine

To eliminate guesswork when sizing container memory limits, deploy the Kubernetes Vertical Pod Autoscaler in Off (recommendation) mode. VPA tracks actual 99th-percentile working set memory across peak production traffic and generates empirical memory sizing recommendations:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-gateway-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: api-gateway
  updatePolicy:
    updateMode: "Off"  # Generates recommendations without restarting pods
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      minAllowed:
        cpu: "250m"
        memory: "512Mi"
      maxAllowed:
        cpu: "4000m"
        memory: "8Gi"

7. SRE Incident Triage Checklist

1. Verify whether the crash was container-level (OOMKilled: true) or node-level eviction via kubectl describe pod.
2. Inspect application logs immediately preceding the crash for batch operations, unbounded database queries (e.g. SELECT * without pagination), or unhandled WebSocket buffers.
3. Adjust runtime heap flags (--max-old-space-size or -XX:MaxRAMPercentage) to ensure garbage collection fires before cgroup boundaries.
4. Set requests.memory equal to limits.memory to assign Guaranteed QoS class to critical workloads.
5. Deploy Vertical Pod Autoscaler (VPA) in recommendation mode to observe actual 99th-percentile working set memory across peak production traffic.

Aftab M.

Founder of Digixfly.com & Editor-in-Chief of XSeedInfo

Aftab M. is the Founder of Digixfly.com, an AI-powered technical marketing and digital growth agency. He directs technical investigations, infrastructure benchmarks, and unit economics teardowns for technology leaders across North America.

About the Author • Editorial Standards • Contact Desk