- Published on
Fixing Kubernetes Pod Pending — How to Read the Rejection Reason the Scheduler Left Behind
- Authors
- Name
- Youngju Kim
- @fjvbn20031
Introduction — the Pod Has Been Sitting in Pending for 10 Minutes
The rollout reports success, but the Pod does not move.
kubectl get pod -n analytics
NAME READY STATUS RESTARTS AGE
clickhouse-0 0/1 Pending 0 11m
report-worker-7f4c8d9b6-hs2p 0/1 Pending 0 11m
report-worker-7f4c8d9b6-k9vt 1/1 Running 0 11m
Pending means the Pod object was created but no node has been chosen yet. This is not a container problem but a placement problem, so looking at the logs shows nothing. The container never started at all.
kubectl logs clickhouse-0 -n analytics
Error from server (BadRequest): container "clickhouse" in pod "clickhouse-0" is waiting to start: ContainerCreating
Fortunately the scheduler records why it could not place the Pod as an event, every single time. That one sentence is the entire diagnosis.
How to Decode the Scheduler Rejection Message
kubectl describe pod clickhouse-0 -n analytics | tail -6
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 11m default-scheduler 0/5 nodes are available: 2 Insufficient cpu, 1 node(s) had untolerated taint {dedicated: batch}, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
You need to know the structure of this sentence.
- 0/5 — 0 out of a total of 5 nodes are viable. The leading number is the count of nodes that passed
- What comes after the colon is a tally of the elimination reason per node. 2 + 1 + 2 = 5 matches the node count exactly
- The final
preemption:clause means that even priority-based preemption could not free up a slot
The second item carries a decisive implication. For each node, the scheduler filter chain stops at the first plugin that fails. So a node is tallied under only one reason. A node that is both short on CPU and carrying a taint shows only the earlier reason. In practice this means it is normal for the next reason to appear the moment you fix the one that was reported. You have to peel them off one at a time.
Events have a default retention of one hour, so they may already be gone on an old Pending Pod. In that case, recreate the Pod to produce fresh events.
kubectl get events -n analytics --field-selector reason=FailedScheduling --sort-by=.lastTimestamp | tail -3
LAST SEEN TYPE REASON OBJECT MESSAGE
3m41s Warning FailedScheduling pod/clickhouse-0 0/5 nodes are available: 2 Insufficient cpu, ...
Cause Branches and Diagnosis
| Cause | Scheduler message | Diagnostic command | Fix |
|---|---|---|---|
| CPU/memory requests exceeded | Insufficient cpu, Insufficient memory | Allocated resources in kubectl describe node | Lower requests or add nodes |
| No toleration | had untolerated taint | Taints in kubectl describe node | Add tolerations or remove the taint |
| Selector/affinity mismatch | didn't match Pod's node affinity/selector | kubectl get nodes --show-labels | Label the node or relax the condition |
| PVC not bound | pod has unbound immediate PersistentVolumeClaims | kubectl describe pvc | Check that the StorageClass exists and the provisioner runs |
| Volume zone mismatch | volume node affinity conflict | Check nodeAffinity on the PV | Secure a node in the same zone |
| Pod anti-affinity self-exclusion | didn't match pod anti-affinity rules | Compare the replica count with the topology count | Relax to preferred or add nodes |
| Spread constraint violation | didn't match pod topology spread constraints | Check the current distribution with kubectl get pods -o wide | Relax with ScheduleAnyway, adjust maxSkew |
| Per-node pod count ceiling | Too many pods | Query allocatable.pods with kubectl get node -o jsonpath | Change the CNI configuration or add nodes |
| Node NotReady/cordon | node(s) were unschedulable | kubectl get nodes | Recover the node or uncordon it |
Resource Shortage — the Yardstick Is requests, Not Usage
This is the most common branch and, at the same time, the one most frequently misdiagnosed.
kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ip-10-0-1-14 842m 21% 4118Mi 31%
ip-10-0-2-77 1105m 27% 5203Mi 39%
ip-10-0-3-91 667m 16% 3877Mi 29%
Every node sits in the 20 percent range for CPU and the 30 percent range for memory. If you jump from here to the conclusion "there is capacity to spare, so why will it not schedule", you will burn hours. The scheduler does not look at actual usage. It looks only at the sum of requests across the Pods already placed on the node.
kubectl describe node ip-10-0-1-14 | grep -A10 "Allocated resources"
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 3730m (93%) 6200m (155%)
memory 11544Mi (89%) 15Gi (118%)
ephemeral-storage 0 (0%) 0 (0%)
pods 24 24
Actual usage is 21 percent, reserved capacity is 93 percent. From the scheduler point of view this node is nearly full. The cause is usually a handful of Pods that claimed excessive requests.
kubectl get pods -A --field-selector spec.nodeName=ip-10-0-1-14 \
-o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,MEM:.spec.containers[*].resources.requests.memory' \
| sort -k3 -hr | head -6
NS POD CPU MEM
analytics spark-executor-3 1500m 4Gi
analytics spark-executor-1 1500m 4Gi
search indexer-6c4d9f7b8-h2klp 500m 2Gi
payments ledger-5f8b7c6d9-nm3xt 200m 1Gi
Two Spark executors reserve 3 CPU cores and barely use any of it in practice.
At this point you have to draw a precise line between the roles of requests and limits.
- requests — the sole yardstick for scheduling. The remaining room is computed by subtracting these values from the node allocatable. At container runtime it acts as a cgroup weight for CPU, and for memory it carries no enforcement at all
- limits — plays no part whatsoever in scheduling. It is nothing but a runtime ceiling. CPU limits cause throttling, memory limits cause OOM kills
So no matter how large you set limits, that alone never blocks scheduling; conversely, large requests occupy the node even when the Pod does not actually use the capacity.
A common wrong answer: deleting requests entirely so scheduling goes through. The Pod comes up, but its QoS class becomes BestEffort, so it is the first thing evicted the moment the node comes under pressure. On top of that the Pod counts as 0 in the scheduler arithmetic, which overpacks the node and puts the neighboring Pods at risk as well. The correct fix is to recompute requests from actual usage.
kubectl top pod -A --containers --sort-by=cpu | head -6
NAMESPACE POD NAME CPU(cores) MEMORY(bytes)
analytics spark-executor-3 executor 180m 1204Mi
analytics spark-executor-1 executor 166m 1189Mi
search indexer-6c4... indexer 410m 1802Mi
It reserves 1500m and uses 180m. Lowering it to roughly 400m frees up an entire node.
When the Node Will Not Take the Pod
Taints and Tolerations
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key,EFFECT:.spec.taints[*].effect'
NAME TAINTS EFFECT
ip-10-0-1-14 dedicated NoSchedule
ip-10-0-2-77 <none> <none>
ip-10-0-3-91 node.kubernetes.io/disk-pressure NoSchedule
ip-10-0-4-22 nvidia.com/gpu NoSchedule
Only the second node has no taint. The taint on the third node was not attached by a person — it is something the kubelet attached automatically. When you see taints in the disk-pressure, memory-pressure, pid-pressure, unreachable, or not-ready family, that is a node failure signal, not a placement policy. Rather than working around it by attaching a toleration, you should repair the node.
If the taint is intentional, attach a toleration.
spec:
tolerations:
- key: dedicated
operator: Equal
value: batch
effect: NoSchedule
Node Selector and Affinity Mismatch
When affinity/selector shows up in the message, start by checking whether a node matching the conditions actually exists.
kubectl get pod clickhouse-0 -n analytics -o jsonpath='{.spec.nodeSelector}'
{"disktype":"nvme","topology.kubernetes.io/zone":"ap-northeast-2c"}
kubectl get nodes -l disktype=nvme,topology.kubernetes.io/zone=ap-northeast-2c
No resources found
Removing the conditions one at a time reveals which side is at fault.
kubectl get nodes -l disktype=nvme
NAME STATUS ROLES AGE VERSION
ip-10-0-1-14 Ready <none> 9d v1.31.4
kubectl get nodes -l topology.kubernetes.io/zone=ap-northeast-2c
No resources found
The nvme node exists, but there is not a single node in that zone. Either the zone condition is wrong, or the node group in that zone was scaled down to 0.
The Per-Node Pod Count Ceiling and NotReady
If the message is Too many pods, the problem is a count, not resources.
kubectl get nodes -o custom-columns='NAME:.metadata.name,MAXPODS:.status.allocatable.pods'
NAME MAXPODS
ip-10-0-1-14 29
ip-10-0-2-77 29
ip-10-0-3-91 29
On EKS with the VPC CNI, the ceiling is determined by how many ENIs and IPs a node can attach, so with a small instance type the node cannot take more pods even when CPU and memory are left over. You have to enable prefix delegation or move to a larger instance type.
It is also common for the node itself to have dropped out.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-1-14 Ready <none> 9d v1.31.4
ip-10-0-2-77 Ready,SchedulingDisabled <none> 9d v1.31.4
ip-10-0-3-91 NotReady <none> 9d v1.31.4
SchedulingDisabled means someone cordoned the node. In most cases somebody forgot to uncordon it after maintenance.
Storage and Placement Constraints
An Unbound PVC, and Pending That Is Perfectly Normal
When the Pod is Pending and the PVC is Pending too, most people suspect storage, but there are two situations here that you absolutely must tell apart.
First, the case that is a genuine problem.
kubectl describe pvc clickhouse-data -n analytics | tail -5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 9m (x14 over 12m) persistentvolume-controller storageclass.storage.k8s.io "fast-ssd" not found
The StorageClass name is wrong. This is the classic case where the name changed during a cluster migration.
kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp3 (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 61d
io2 ebs.csi.aws.com Delete WaitForFirstConsumer true 61d
Now the case that is normal.
kubectl describe pvc report-cache -n analytics | tail -4
Events:
Type Reason Age From Message
---- ---- ---- ---- -------
Normal WaitForFirstConsumer 12s (x8 over 2m) persistentvolume-controller waiting for first consumer to be created before binding
This is not an error. A StorageClass whose VOLUMEBINDINGMODE is WaitForFirstConsumer creates the volume in the same zone as the node only after the Pod has been placed on a node. The volume comes into being only once the Pod is placed, so a Pending PVC is exactly as designed.
The trouble starts here. If the Pod cannot be placed for some other reason, the PVC stays Pending forever as well. People then look at the PVC and set out to fix storage, when what actually needs fixing is the scheduling failure of the Pod. If the PVC is waiting on WaitForFirstConsumer, ignore the PVC and look only at the FailedScheduling message on the Pod.
If nodes are replaced after the volume has already been created, the zones can fall out of alignment.
kubectl describe pod clickhouse-0 -n analytics | grep FailedScheduling
Warning FailedScheduling 4m default-scheduler 0/5 nodes are available: 5 node(s) had volume node affinity conflict.
It means the volume lives in ap-northeast-2a but there is no schedulable node in that zone.
kubectl get pv -o custom-columns='NAME:.metadata.name,ZONE:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values' | head -3
NAME ZONE
pvc-8f3c1d5b-2e7a-94c0-f13d-68b5a2e9c47f [ap-northeast-2a]
Self-Exclusion Caused by Anti-Affinity
If you apply required host-level anti-affinity to 3 replicas but have only 2 nodes, the third Pod stays Pending forever.
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: report-worker
topologyKey: kubernetes.io/hostname
kubectl describe pod report-worker-7f4c8d9b6-hs2p -n analytics | grep FailedScheduling
Warning FailedScheduling 9m default-scheduler 0/2 nodes are available: 2 node(s) didn't match pod anti-affinity rules.
There are clearly times when this constraint is needed, but the moment the replica count exceeds the topology count it silently blocks the rollout. If availability is the goal, preferred is usually enough.
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: report-worker
topologyKey: kubernetes.io/hostname
topologySpreadConstraints
Spread constraints are gentler than anti-affinity, but if you leave whenUnsatisfiable at DoNotSchedule they block placement just the same.
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: report-worker
With 3 zones, if every node in one zone fills up, placement is refused because a skew of 1 cannot be maintained even though there is room in another zone.
kubectl get pods -n analytics -l app=report-worker -o wide
NAME READY STATUS NODE ZONE
report-worker-7f4c8d9b6-k9vt 1/1 Running ip-10-0-1-14 2a
report-worker-7f4c8d9b6-m2xd 1/1 Running ip-10-0-2-77 2b
report-worker-7f4c8d9b6-hs2p 0/1 Pending <none> <none>
Unless availability is an absolute requirement, switching to ScheduleAnyway is the pragmatic move. The constraint then feeds only into scoring and does not block placement.
Why Cluster Autoscaler Does Not Add Nodes
You expect the autoscaler to add capacity when nodes run short, but there are plenty of cases where it does not. Here too the reason is left in an event.
kubectl describe pod clickhouse-0 -n analytics | grep -A2 NotTriggerScaleUp
Normal NotTriggerScaleUp 2m17s (x21 over 9m) cluster-autoscaler pod didn't trigger scale-up: 1 max node group size reached, 2 node(s) had untolerated taint {dedicated: batch}, 1 node(s) had volume node affinity conflict
It is telling you three things at once. One node group has reached its maximum size, two groups carry a taint this Pod cannot tolerate, and one group has a volume zone that does not match. The autoscaler simulates in advance whether this Pod could be placed assuming a new node were brought up, so once it decides placement is impossible, it does not add nodes.
Look at the controller-side logs and the status ConfigMap as well.
kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200 | grep -i "no expansion\|max size\|scale_up"
I0726 09:52:14.118 scale_up.go:300] Pod analytics/clickhouse-0 can't be scheduled on eks-general-2c. Predicate checking error: node(s) had volume node affinity conflict
I0726 09:52:14.119 scale_up.go:472] No expansion options
kubectl -n kube-system get configmap cluster-autoscaler-status \
-o jsonpath='{.data.status}' | head -12
Cluster-autoscaler status at 2026-07-26 09:52:14:
Cluster-wide:
Health: Healthy
ScaleUp: NoActivity (ready=5 registered=5)
ScaleDown: NoCandidates
NodeGroups:
Name: eks-general-2a
Health: Healthy (ready=3 cloudProviderTarget=3 minSize=1 maxSize=3)
ScaleUp: NoActivity
When maxSize and cloudProviderTarget are equal, you have hit the ceiling.
A common wrong answer: adding nodes unconditionally whenever Pending shows up. Anti-affinity, spread constraints, and volume zone mismatches either do not clear up when you add nodes, or the nodes land in the wrong zone and only raise the bill. Read the NotTriggerScaleUp message first and remove the reason it names.
Preventing Recurrence
Pending is usually the result of requests being sized with no evidence behind them. Three mechanisms cut it down.
First, enforce defaults and ceilings on the namespace. This keeps Pods that omitted requests from getting in.
apiVersion: v1
kind: LimitRange
metadata:
name: default-requests
namespace: analytics
spec:
limits:
- type: Container
default:
cpu: 500m
memory: 512Mi
defaultRequest:
cpu: 100m
memory: 256Mi
max:
cpu: 4
memory: 16Gi
Second, run VPA in recommendation mode to obtain suggested values grounded in actual usage. Even without going as far as automatic application, just seeing the numbers is worth it.
kubectl get vpa spark-executor -n analytics -o jsonpath='{.status.recommendation.containerRecommendations[0]}'
{"containerName":"executor","lowerBound":{"cpu":"180m","memory":"1000Mi"},"target":{"cpu":"320m","memory":"1400Mi"},"upperBound":{"cpu":"1200m","memory":"3Gi"}}
Third, catch the duration of Pending with an alert. Placement failures never show up in application metrics at all, so a separate rule is required.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: scheduling-alerts
namespace: monitoring
spec:
groups:
- name: scheduling
rules:
- alert: PodPendingTooLong
expr: kube_pod_status_phase{phase="Pending"} == 1
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} has been Pending for 15m"
- alert: NodeRequestsNearlyFull
expr: sum by (node) (kube_pod_container_resource_requests{resource="cpu"}) / on(node) kube_node_status_allocatable{resource="cpu"} > 0.9
for: 30m
labels:
severity: warning
Wrapping Up — the Scheduler Already Told You the Answer
There is almost never a moment in Pending diagnosis that calls for guessing. One sentence in the FailedScheduling event tallies the node count and the elimination reasons, and each reason maps to a single row in the table above. Once you remember that the filter chain stops at the first failure per node, so the reasons surface one after another, the job turns into peeling them off one at a time.
The one sentence to remember is this. Scheduling is decided by the arithmetic of requests, not by actual usage, and just because kubectl top looks idle does not mean there is room on the node.