Kubernetes Pod Stuck in Pending¶
Problem¶
A pod stays Pending: the scheduler hasn't assigned it to a node, so no container has been created.
Because nothing has started, there are no container logs. The answer is in the pod's events.
Symptoms¶
kubectl get pod -o wideshows<none>in theNODEcolumn.- Events show
FailedSchedulingwith a message like0/6 nodes are available: …. - New replicas from a scale-up or rollout never become ready.
- Sometimes there are no events at all — a clue in itself (see below).
Likely Causes¶
The FailedScheduling message lists why each group of nodes was rejected. Match the phrases:
| Phrase in the message | Cause |
|---|---|
Insufficient cpu, Insufficient memory |
No node has enough unrequested capacity for the pod's resources.requests |
node(s) had untolerated taint {…} |
Nodes are tainted and the pod has no matching toleration |
node(s) didn't match Pod's node affinity/selector |
nodeSelector or required node affinity excludes the nodes |
pod has unbound immediate PersistentVolumeClaims |
A PVC the pod mounts isn't bound |
node(s) had volume node affinity conflict |
The volume lives in a zone where no eligible node exists |
node(s) didn't match pod topology spread constraints |
A whenUnsatisfiable: DoNotSchedule spread rule can't be met |
node(s) didn't match pod anti-affinity rules |
Required anti-affinity leaves no valid node |
node(s) didn't have free ports for the requested pod ports |
hostPort already used on every candidate node |
node(s) were unschedulable |
Nodes are cordoned |
Too many pods |
Nodes hit their maximum pod count (or run out of pod IP addresses) |
Diagnostic Commands¶
# 1. The scheduler's explanation
kubectl describe pod orders-api-7c9d8b6f5-k2x4p | sed -n '/Events:/,$p'
kubectl get events -n orders --field-selector reason=FailedScheduling --sort-by=.lastTimestamp
# 2. What the pod asks for
kubectl get pod orders-api-7c9d8b6f5-k2x4p -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.requests}{"\n"}{end}'
kubectl get pod orders-api-7c9d8b6f5-k2x4p -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity}{"\n"}{.spec.tolerations}{"\n"}'
# 3. What nodes can offer — requests already allocated, not live usage
kubectl describe nodes | grep -A 8 "Allocated resources"
kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory,TAINTS:.spec.taints[*].key'
kubectl get nodes --show-labels
# 4. Storage the pod depends on
kubectl get pvc -n orders
Requests, not usage
The scheduler places pods by requests. A node at 15% real CPU usage can still be "full" if its pods request most of its allocatable CPU. kubectl top nodes shows usage and won't explain Insufficient cpu.
Step-by-Step Debugging¶
- Read the full
FailedSchedulingmessage. Each clause accounts for a group of nodes:0/6 nodes are available: 3 Insufficient memory, 3 node(s) had untolerated taint {dedicated: gpu}. - Handle the largest group first. In that example, fixing memory makes three nodes eligible; tolerating the GPU taint is probably wrong for a web service anyway.
- Compare requests with allocatable capacity. A single pod requesting more than any node's allocatable memory can never schedule, no matter how many nodes you add.
- Check constraints you may have forgotten: node selectors from a Helm values file, anti-affinity that requires one replica per node, topology spread across zones with only two zones available.
- Check the PVC if storage is mentioned — see Storage Problems.
- If there are no events, confirm the pod's
schedulerNamepoints to a scheduler that exists, and check whether a scheduling gate is set (.spec.schedulingGates). - If you run an autoscaler, check whether it tried to add a node and why it didn't — its events and logs explain constraints like instance types or max group size.
Solutions¶
Not enough capacity¶
# Right-size requests from observed usage
kubectl set resources deployment/orders-api -n orders --requests=cpu=250m,memory=384Mi
Or add capacity: scale the node group, or let the Cluster Autoscaler or Karpenter provision nodes. Check that a node type in the pool is actually large enough for the pod.
Taints and tolerations¶
Only add a toleration when the workload belongs on those nodes:
A toleration allows scheduling onto tainted nodes; it doesn't require it. Pair it with node affinity to target a dedicated pool. See Scheduling, Affinity, and Taints.
Node selector or affinity mismatch¶
Fix the selector, or label the intended nodes:
kubectl get nodes -l workload=api # does any node have the label?
kubectl label node ip-10-0-2-15 workload=api
In managed clusters, set labels on the node group or NodePool definition rather than on individual nodes, which are replaced.
Topology spread or anti-affinity can't be satisfied¶
Relax a hard rule when strict placement isn't essential:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway # was DoNotSchedule
labelSelector:
matchLabels:
app: orders-api
Or use preferredDuringSchedulingIgnoredDuringExecution for anti-affinity instead of required….
Volume in the wrong zone¶
Use a StorageClass with volumeBindingMode: WaitForFirstConsumer, so the volume is created in the zone where the pod is scheduled rather than before scheduling.
Cordoned nodes or pod limits¶
For Too many pods, check the node's .status.allocatable.pods and, on AWS VPC CNI, the IP addresses available to the node.
Prevention¶
- Set requests from measured usage with some headroom, and review them after load changes.
- Keep a
LimitRangewith sensible defaults so pods without requests don't get surprising values. - Make sure every node pool has at least one instance type large enough for the biggest pod it should run.
- Prefer soft spread and anti-affinity rules unless availability truly requires hard ones.
- Alert on pods pending longer than a few minutes:
kube_pod_status_phase{phase="Pending"} == 1held for 10 minutes, using kube-state-metrics.
Production Considerations¶
- Pending pods during a rollout stall it safely if
maxUnavailableis0— old pods keep serving. - Priority and preemption: with PriorityClasses, higher-priority pods can evict lower-priority ones to fit. Check events for
Preemptedbefore assuming the cluster is simply full. - ResourceQuota is different. A quota violation stops the pod from being created at all — you'll see
FailedCreateon the ReplicaSet, not aPendingpod. - Capacity planning: recurring
Insufficientevents are a signal to plan capacity, not only to add a node — see Capacity Planning.
Related Problems¶
- PVC stuck in Pending
- Node NotReady and cluster problems
- ImagePullBackOff — scheduled, but the image won't pull
- Resource requests, limits, and QoS
- All pod startup errors