All Articles
Deep Dive·5 min read

Your GKE Pods Keep Getting OOMKilled — Here's Why

A pod that restarts with reason OOMKilled hit its memory limit. Here is what that really means and how to stop the crash loop.

Part of: Google Cloud Platform (GCP) Engineer

Your GKE pod keeps restarting, and kubectl describe shows the container's last state was OOMKilled. The kernel killed your process for exceeding its memory limit. It is one of the most common GKE production issues, and the fix is rarely "add more memory" blindly. Here is how to reason about it.

What OOMKilled Actually Means

Each container has a memory limit. When the process tries to use more than that limit, the Linux kernel's out-of-memory killer terminates it, and Kubernetes restarts the pod. It is not the node running out of memory — it is your container exceeding its own ceiling:

kubectl describe pod my-app | grep -A3 "Last State" # Last State: Terminated # Reason: OOMKilled

Cause 1: Limit Set Too Low

If you guessed the memory limit, you may simply have set it below what the app needs under real load. Look at actual usage and set the limit above the observed peak:

kubectl top pod my-app # see real memory usage
resources: requests: memory: "512Mi" limits: memory: "1Gi" # comfortably above the observed peak

Cause 2: A Real Memory Leak

If usage climbs steadily until it hits any limit you set, the app is leaking — raising the limit just delays the crash. Profile the process, look for unbounded caches or accumulating buffers, and fix the leak. A limit is a safety net, not a cure for growth without bound.

Cause 3: JVM/Runtime Not Aware of the Limit

Older runtimes ignore the container limit and size their heap to the node's memory, then get killed. Make the runtime container-aware — modern JVMs do this by default, but confirm:

-XX:MaxRAMPercentage=75.0   # size the heap relative to the container limit

Summary

OOMKilled means the container blew past its own memory limit and the kernel stopped it. Check real usage with kubectl top, raise the limit only if it was genuinely too low, hunt for a leak if usage grows without bound, and make sure your runtime respects the container limit rather than the node's. Match the limit to reality and the crash loop ends.

#GCP#GKE#Kubernetes#Reliability

Want to go deeper?

Google Cloud Platform (GCP) Engineer

View Course