Split View: Kubernetes Pod Pending 상태 해결 — 스케줄러가 남긴 거절 사유 읽는 법
Kubernetes Pod Pending 상태 해결 — 스케줄러가 남긴 거절 사유 읽는 법
들어가며 — 파드가 Pending에서 10분째 그대로입니다
배포는 성공했다고 나오는데 파드가 움직이지 않습니다.
kubectl get pod -n analytics
NAME READY STATUS RESTARTS AGE
clickhouse-0 0/1 Pending 0 11m
report-worker-7f4c8d9b6-hs2p 0/1 Pending 0 11m
report-worker-7f4c8d9b6-k9vt 1/1 Running 0 11m
Pending은 파드 오브젝트는 만들어졌지만 아직 노드가 정해지지 않았다는 뜻입니다. 컨테이너 문제가 아니라 배치 문제이므로, 로그를 봐도 아무것도 나오지 않습니다. 컨테이너가 아예 시작되지 않았기 때문입니다.
kubectl logs clickhouse-0 -n analytics
Error from server (BadRequest): container "clickhouse" in pod "clickhouse-0" is waiting to start: ContainerCreating
다행히 스케줄러는 왜 배치하지 못했는지를 매번 이벤트로 남깁니다. 그 한 문장이 진단의 전부입니다.
스케줄러의 거절 메시지 해독법
kubectl describe pod clickhouse-0 -n analytics | tail -6
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 11m default-scheduler 0/5 nodes are available: 2 Insufficient cpu, 1 node(s) had untolerated taint {dedicated: batch}, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
이 문장의 구조를 알아야 합니다.
- 0/5 — 전체 5개 노드 중 0개가 가능. 앞 숫자가 통과한 노드 수입니다
- 콜론 뒤는 노드별 탈락 사유의 집계입니다. 2 + 1 + 2 = 5로 노드 수와 정확히 맞습니다
- 마지막
preemption:절은 우선순위 기반 선점으로도 자리를 못 만들었다는 뜻입니다
두 번째 항목에 결정적인 함의가 있습니다. 스케줄러의 필터 체인은 노드마다 처음 실패한 플러그인에서 멈춥니다. 그래서 한 노드는 하나의 이유로만 집계됩니다. CPU도 부족하고 테인트도 있는 노드는 앞선 이유 하나만 보여 줍니다. 실무적으로 이 말은, 보고된 이유를 고쳤더니 곧바로 다음 이유가 나타나는 일이 정상이라는 뜻입니다. 한 번에 하나씩 벗겨 내야 합니다.
이벤트는 기본 보존 기간이 한 시간이라 오래된 Pending 파드에서는 사라져 있을 수 있습니다. 그럴 때는 파드를 다시 만들어 이벤트를 새로 뽑습니다.
kubectl get events -n analytics --field-selector reason=FailedScheduling --sort-by=.lastTimestamp | tail -3
LAST SEEN TYPE REASON OBJECT MESSAGE
3m41s Warning FailedScheduling pod/clickhouse-0 0/5 nodes are available: 2 Insufficient cpu, ...
원인 분기와 진단
| 원인 | 스케줄러 메시지 | 진단 명령 | 해결 |
|---|---|---|---|
| CPU·메모리 요청량 초과 | Insufficient cpu, Insufficient memory | kubectl describe node의 Allocated resources | requests 하향 또는 노드 증설 |
| 톨러레이션 없음 | had untolerated taint | kubectl describe node 의 Taints | tolerations 추가 또는 테인트 제거 |
| 셀렉터·어피니티 불일치 | didn't match Pod's node affinity/selector | kubectl get nodes --show-labels | 노드 라벨 부여 또는 조건 완화 |
| PVC 미바인딩 | pod has unbound immediate PersistentVolumeClaims | kubectl describe pvc | 스토리지클래스 존재와 프로비저너 확인 |
| 볼륨 존 불일치 | volume node affinity conflict | PV의 nodeAffinity 확인 | 같은 존에 노드 확보 |
| 파드 안티어피니티 자기 배제 | didn't match pod anti-affinity rules | 복제본 수와 토폴로지 수 비교 | preferred로 완화 또는 노드 추가 |
| 분산 제약 위반 | didn't match pod topology spread constraints | kubectl get pods -o wide로 현재 분포 확인 | ScheduleAnyway로 완화, maxSkew 조정 |
| 노드당 파드 수 상한 | Too many pods | kubectl get node -o jsonpath 로 allocatable.pods 조회 | CNI 설정 변경 또는 노드 증설 |
| 노드 NotReady·cordon | node(s) were unschedulable | kubectl get nodes | 노드 복구 또는 uncordon |
리소스 부족 — 사용량이 아니라 requests가 기준입니다
가장 흔하고, 동시에 가장 자주 오진되는 갈래입니다.
kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ip-10-0-1-14 842m 21% 4118Mi 31%
ip-10-0-2-77 1105m 27% 5203Mi 39%
ip-10-0-3-91 667m 16% 3877Mi 29%
노드마다 CPU가 20퍼센트대이고 메모리도 30퍼센트대입니다. 여기서 "자원은 남는데 왜 스케줄이 안 되지"라는 결론으로 가면 몇 시간을 낭비합니다. 스케줄러는 실제 사용량을 보지 않습니다. 노드에 이미 배치된 파드들의 requests 총합만 봅니다.
kubectl describe node ip-10-0-1-14 | grep -A10 "Allocated resources"
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 3730m (93%) 6200m (155%)
memory 11544Mi (89%) 15Gi (118%)
ephemeral-storage 0 (0%) 0 (0%)
pods 24 24
실제 사용량 21퍼센트, 예약량 93퍼센트입니다. 스케줄러 입장에서 이 노드는 거의 꽉 찬 노드입니다. 원인은 대개 requests를 과도하게 잡은 파드 몇 개입니다.
kubectl get pods -A --field-selector spec.nodeName=ip-10-0-1-14 \
-o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,MEM:.spec.containers[*].resources.requests.memory' \
| sort -k3 -hr | head -6
NS POD CPU MEM
analytics spark-executor-3 1500m 4Gi
analytics spark-executor-1 1500m 4Gi
search indexer-6c4d9f7b8-h2klp 500m 2Gi
payments ledger-5f8b7c6d9-nm3xt 200m 1Gi
Spark 실행기 두 개가 CPU 3코어를 예약하고 실제로는 거의 쓰지 않습니다.
여기서 requests와 limits의 역할을 정확히 구분해야 합니다.
- requests — 스케줄링의 유일한 기준입니다. 노드의 allocatable에서 이 값들을 빼서 남는 자리를 계산합니다. 컨테이너 실행 시점에는 CPU의 경우 cgroup 가중치로, 메모리의 경우 아무 강제력 없이 작용합니다
- limits — 스케줄링에 전혀 관여하지 않습니다. 실행 중 상한일 뿐입니다. CPU limits는 스로틀링을, 메모리 limits는 OOM 킬을 유발합니다
그래서 limits를 아무리 크게 잡아도 스케줄이 되지 않는 일은 없고, 반대로 requests를 크게 잡으면 실제로 쓰지 않아도 노드를 점유합니다.
흔한 오답: requests를 아예 지워서 스케줄을 통과시키는 것. 파드는 뜨지만 QoS 클래스가 BestEffort가 되어 노드가 압박을 받는 순간 가장 먼저 축출됩니다. 게다가 그 파드는 스케줄러 계산에서 0으로 취급되므로 노드를 과밀하게 만들어 이웃 파드까지 위험해집니다. 올바른 해법은 실제 사용량을 근거로 requests를 재산정하는 것입니다.
kubectl top pod -A --containers --sort-by=cpu | head -6
NAMESPACE POD NAME CPU(cores) MEMORY(bytes)
analytics spark-executor-3 executor 180m 1204Mi
analytics spark-executor-1 executor 166m 1189Mi
search indexer-6c4... indexer 410m 1802Mi
1500m을 예약하고 180m을 쓰고 있습니다. 400m 정도로 낮추면 노드 하나가 통째로 비워집니다.
노드가 받아 주지 않는 경우
테인트와 톨러레이션
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key,EFFECT:.spec.taints[*].effect'
NAME TAINTS EFFECT
ip-10-0-1-14 dedicated NoSchedule
ip-10-0-2-77 <none> <none>
ip-10-0-3-91 node.kubernetes.io/disk-pressure NoSchedule
ip-10-0-4-22 nvidia.com/gpu NoSchedule
두 번째 노드만 테인트가 없습니다. 세 번째 노드의 테인트는 사람이 붙인 것이 아니라 kubelet이 자동으로 붙인 것입니다. disk-pressure, memory-pressure, pid-pressure, unreachable, not-ready 계열 테인트가 보이면 그건 배치 정책이 아니라 노드 장애 신호입니다. 톨러레이션을 붙여 우회할 것이 아니라 노드를 고쳐야 합니다.
의도적인 테인트라면 톨러레이션을 붙입니다.
spec:
tolerations:
- key: dedicated
operator: Equal
value: batch
effect: NoSchedule
노드 셀렉터와 어피니티 불일치
메시지에 affinity/selector가 나오면 조건에 맞는 노드가 실제로 있는지부터 확인합니다.
kubectl get pod clickhouse-0 -n analytics -o jsonpath='{.spec.nodeSelector}'
{"disktype":"nvme","topology.kubernetes.io/zone":"ap-northeast-2c"}
kubectl get nodes -l disktype=nvme,topology.kubernetes.io/zone=ap-northeast-2c
No resources found
조건을 하나씩 빼 보면 어느 쪽이 문제인지 드러납니다.
kubectl get nodes -l disktype=nvme
NAME STATUS ROLES AGE VERSION
ip-10-0-1-14 Ready <none> 9d v1.31.4
kubectl get nodes -l topology.kubernetes.io/zone=ap-northeast-2c
No resources found
nvme 노드는 있지만 해당 존에 노드가 하나도 없습니다. 존 조건이 잘못됐거나 그 존의 노드 그룹이 0으로 축소된 것입니다.
노드당 파드 수 상한과 NotReady
메시지가 Too many pods이면 자원이 아니라 개수 문제입니다.
kubectl get nodes -o custom-columns='NAME:.metadata.name,MAXPODS:.status.allocatable.pods'
NAME MAXPODS
ip-10-0-1-14 29
ip-10-0-2-77 29
ip-10-0-3-91 29
EKS에서 VPC CNI를 쓰면 노드가 붙일 수 있는 ENI와 IP 수로 상한이 결정되므로, 인스턴스 타입이 작으면 CPU와 메모리가 남아도 파드를 더 못 받습니다. 접두사 위임을 켜거나 인스턴스 타입을 키워야 합니다.
노드 자체가 빠진 경우도 흔합니다.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-1-14 Ready <none> 9d v1.31.4
ip-10-0-2-77 Ready,SchedulingDisabled <none> 9d v1.31.4
ip-10-0-3-91 NotReady <none> 9d v1.31.4
SchedulingDisabled는 누군가 cordon한 상태입니다. 유지보수 후 uncordon을 잊은 경우가 대부분입니다.
스토리지와 배치 제약
PVC 미바인딩, 그리고 정상적인 Pending
파드가 Pending이고 PVC도 Pending이면 대부분 스토리지를 의심하지만, 여기에는 반드시 구분해야 할 두 상황이 있습니다.
먼저 진짜 문제인 경우입니다.
kubectl describe pvc clickhouse-data -n analytics | tail -5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 9m (x14 over 12m) persistentvolume-controller storageclass.storage.k8s.io "fast-ssd" not found
스토리지클래스 이름이 틀렸습니다. 클러스터를 옮기면서 이름이 달라진 전형적인 경우입니다.
kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp3 (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 61d
io2 ebs.csi.aws.com Delete WaitForFirstConsumer true 61d
이제 정상인 경우입니다.
kubectl describe pvc report-cache -n analytics | tail -4
Events:
Type Reason Age From Message
---- ---- ---- ---- -------
Normal WaitForFirstConsumer 12s (x8 over 2m) persistentvolume-controller waiting for first consumer to be created before binding
이건 오류가 아닙니다. VOLUMEBINDINGMODE가 WaitForFirstConsumer인 스토리지클래스는 파드가 노드에 배치된 다음에 그 노드와 같은 존에 볼륨을 만듭니다. 파드가 먼저 배치되어야 볼륨이 생기므로, PVC가 Pending인 것은 설계대로입니다.
문제는 여기서 생깁니다. 파드가 다른 이유로 배치되지 못하면 PVC도 영원히 Pending에 머뭅니다. 그러면 PVC를 보고 스토리지를 고치려 들지만, 실제로 고쳐야 하는 것은 파드의 스케줄링 실패입니다. PVC가 WaitForFirstConsumer로 대기 중이면 PVC는 무시하고 파드의 FailedScheduling 메시지만 보십시오.
이미 볼륨이 생성된 뒤에 노드가 교체되면 존이 어긋날 수 있습니다.
kubectl describe pod clickhouse-0 -n analytics | grep FailedScheduling
Warning FailedScheduling 4m default-scheduler 0/5 nodes are available: 5 node(s) had volume node affinity conflict.
볼륨은 ap-northeast-2a에 있는데 그 존에 스케줄 가능한 노드가 없다는 뜻입니다.
kubectl get pv -o custom-columns='NAME:.metadata.name,ZONE:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values' | head -3
NAME ZONE
pvc-8f3c1d5b-2e7a-94c0-f13d-68b5a2e9c47f [ap-northeast-2a]
안티어피니티에 의한 자기 배제
복제본 3개에 호스트 단위 필수 안티어피니티를 걸었는데 노드가 2개면, 세 번째 파드는 영원히 Pending입니다.
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: report-worker
topologyKey: kubernetes.io/hostname
kubectl describe pod report-worker-7f4c8d9b6-hs2p -n analytics | grep FailedScheduling
Warning FailedScheduling 9m default-scheduler 0/2 nodes are available: 2 node(s) didn't match pod anti-affinity rules.
이 제약은 필요할 때가 분명히 있지만, 복제본 수가 토폴로지 수를 넘는 순간 조용히 배포를 막습니다. 가용성이 목적이라면 대개 preferred로 충분합니다.
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: report-worker
topologyKey: kubernetes.io/hostname
topologySpreadConstraints
분산 제약은 안티어피니티보다 부드럽지만 whenUnsatisfiable을 DoNotSchedule로 두면 똑같이 막힙니다.
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: report-worker
존이 3개인데 한 존의 노드가 전부 꽉 차면, 다른 존에 자리가 있어도 스큐 1을 지킬 수 없어 배치가 거부됩니다.
kubectl get pods -n analytics -l app=report-worker -o wide
NAME READY STATUS NODE ZONE
report-worker-7f4c8d9b6-k9vt 1/1 Running ip-10-0-1-14 2a
report-worker-7f4c8d9b6-m2xd 1/1 Running ip-10-0-2-77 2b
report-worker-7f4c8d9b6-hs2p 0/1 Pending <none> <none>
가용성이 절대 조건이 아니라면 ScheduleAnyway로 바꾸는 것이 실용적입니다. 이때 제약은 점수 계산에만 반영되고 배치를 막지 않습니다.
Cluster Autoscaler가 노드를 안 늘리는 이유
노드가 부족하면 오토스케일러가 늘려 주리라 기대하지만, 늘지 않는 경우가 자주 있습니다. 이때도 이유는 이벤트에 남습니다.
kubectl describe pod clickhouse-0 -n analytics | grep -A2 NotTriggerScaleUp
Normal NotTriggerScaleUp 2m17s (x21 over 9m) cluster-autoscaler pod didn't trigger scale-up: 1 max node group size reached, 2 node(s) had untolerated taint {dedicated: batch}, 1 node(s) had volume node affinity conflict
세 가지를 동시에 말하고 있습니다. 한 노드 그룹은 최대 크기에 도달했고, 두 그룹은 이 파드가 견디지 못하는 테인트를 가지며, 한 그룹은 볼륨 존이 맞지 않습니다. 오토스케일러는 새 노드를 띄웠다고 가정했을 때 이 파드가 배치 가능한지를 미리 시뮬레이션하므로, 배치 불가로 판정되면 노드를 늘리지 않습니다.
컨트롤러 쪽 로그와 상태 컨피그맵도 함께 봅니다.
kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200 | grep -i "no expansion\|max size\|scale_up"
I0726 09:52:14.118 scale_up.go:300] Pod analytics/clickhouse-0 can't be scheduled on eks-general-2c. Predicate checking error: node(s) had volume node affinity conflict
I0726 09:52:14.119 scale_up.go:472] No expansion options
kubectl -n kube-system get configmap cluster-autoscaler-status \
-o jsonpath='{.data.status}' | head -12
Cluster-autoscaler status at 2026-07-26 09:52:14:
Cluster-wide:
Health: Healthy
ScaleUp: NoActivity (ready=5 registered=5)
ScaleDown: NoCandidates
NodeGroups:
Name: eks-general-2a
Health: Healthy (ready=3 cloudProviderTarget=3 minSize=1 maxSize=3)
ScaleUp: NoActivity
maxSize와 cloudProviderTarget이 같으면 상한에 걸린 것입니다.
흔한 오답: Pending이 보이면 무조건 노드를 추가하는 것. 안티어피니티, 분산 제약, 볼륨 존 불일치는 노드를 늘려도 풀리지 않거나, 엉뚱한 존에 노드가 늘어 비용만 증가합니다. NotTriggerScaleUp 메시지를 먼저 읽고 그 이유를 제거해야 합니다.
재발 방지
Pending은 대개 requests 산정이 근거 없이 이루어진 결과입니다. 세 가지 장치로 줄일 수 있습니다.
첫째, 네임스페이스에 기본값과 상한을 강제합니다. requests를 빼먹은 파드가 들어오는 것을 막습니다.
apiVersion: v1
kind: LimitRange
metadata:
name: default-requests
namespace: analytics
spec:
limits:
- type: Container
default:
cpu: 500m
memory: 512Mi
defaultRequest:
cpu: 100m
memory: 256Mi
max:
cpu: 4
memory: 16Gi
둘째, VPA를 권고 모드로 돌려 실제 사용량 기반 권장치를 확보합니다. 자동 적용까지 하지 않아도 숫자를 보는 것만으로 가치가 있습니다.
kubectl get vpa spark-executor -n analytics -o jsonpath='{.status.recommendation.containerRecommendations[0]}'
{"containerName":"executor","lowerBound":{"cpu":"180m","memory":"1000Mi"},"target":{"cpu":"320m","memory":"1400Mi"},"upperBound":{"cpu":"1200m","memory":"3Gi"}}
셋째, Pending 지속 시간을 알림으로 잡습니다. 배치 실패는 애플리케이션 지표에 전혀 나타나지 않으므로 별도 규칙이 필요합니다.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: scheduling-alerts
namespace: monitoring
spec:
groups:
- name: scheduling
rules:
- alert: PodPendingTooLong
expr: kube_pod_status_phase{phase="Pending"} == 1
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} has been Pending for 15m"
- alert: NodeRequestsNearlyFull
expr: sum by (node) (kube_pod_container_resource_requests{resource="cpu"}) / on(node) kube_node_status_allocatable{resource="cpu"} > 0.9
for: 30m
labels:
severity: warning
마치며 — 스케줄러는 이미 답을 말했습니다
Pending 진단에서 추측이 필요한 순간은 거의 없습니다. FailedScheduling 이벤트의 한 문장에 노드 수와 탈락 사유가 집계되어 있고, 각 사유는 위 표의 한 행에 대응합니다. 필터 체인이 노드당 첫 실패에서 멈추기 때문에 이유가 순차적으로 드러난다는 점만 기억하면, 한 번에 하나씩 벗겨 내는 작업이 됩니다.
기억할 한 문장은 이것입니다. 스케줄링은 실제 사용량이 아니라 requests의 산술로 결정되며, kubectl top이 한가해 보인다고 해서 노드에 자리가 있는 것은 아닙니다.
Fixing Kubernetes Pod Pending — How to Read the Rejection Reason the Scheduler Left Behind
Introduction — the Pod Has Been Sitting in Pending for 10 Minutes
The rollout reports success, but the Pod does not move.
kubectl get pod -n analytics
NAME READY STATUS RESTARTS AGE
clickhouse-0 0/1 Pending 0 11m
report-worker-7f4c8d9b6-hs2p 0/1 Pending 0 11m
report-worker-7f4c8d9b6-k9vt 1/1 Running 0 11m
Pending means the Pod object was created but no node has been chosen yet. This is not a container problem but a placement problem, so looking at the logs shows nothing. The container never started at all.
kubectl logs clickhouse-0 -n analytics
Error from server (BadRequest): container "clickhouse" in pod "clickhouse-0" is waiting to start: ContainerCreating
Fortunately the scheduler records why it could not place the Pod as an event, every single time. That one sentence is the entire diagnosis.
How to Decode the Scheduler Rejection Message
kubectl describe pod clickhouse-0 -n analytics | tail -6
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 11m default-scheduler 0/5 nodes are available: 2 Insufficient cpu, 1 node(s) had untolerated taint {dedicated: batch}, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
You need to know the structure of this sentence.
- 0/5 — 0 out of a total of 5 nodes are viable. The leading number is the count of nodes that passed
- What comes after the colon is a tally of the elimination reason per node. 2 + 1 + 2 = 5 matches the node count exactly
- The final
preemption:clause means that even priority-based preemption could not free up a slot
The second item carries a decisive implication. For each node, the scheduler filter chain stops at the first plugin that fails. So a node is tallied under only one reason. A node that is both short on CPU and carrying a taint shows only the earlier reason. In practice this means it is normal for the next reason to appear the moment you fix the one that was reported. You have to peel them off one at a time.
Events have a default retention of one hour, so they may already be gone on an old Pending Pod. In that case, recreate the Pod to produce fresh events.
kubectl get events -n analytics --field-selector reason=FailedScheduling --sort-by=.lastTimestamp | tail -3
LAST SEEN TYPE REASON OBJECT MESSAGE
3m41s Warning FailedScheduling pod/clickhouse-0 0/5 nodes are available: 2 Insufficient cpu, ...
Cause Branches and Diagnosis
| Cause | Scheduler message | Diagnostic command | Fix |
|---|---|---|---|
| CPU/memory requests exceeded | Insufficient cpu, Insufficient memory | Allocated resources in kubectl describe node | Lower requests or add nodes |
| No toleration | had untolerated taint | Taints in kubectl describe node | Add tolerations or remove the taint |
| Selector/affinity mismatch | didn't match Pod's node affinity/selector | kubectl get nodes --show-labels | Label the node or relax the condition |
| PVC not bound | pod has unbound immediate PersistentVolumeClaims | kubectl describe pvc | Check that the StorageClass exists and the provisioner runs |
| Volume zone mismatch | volume node affinity conflict | Check nodeAffinity on the PV | Secure a node in the same zone |
| Pod anti-affinity self-exclusion | didn't match pod anti-affinity rules | Compare the replica count with the topology count | Relax to preferred or add nodes |
| Spread constraint violation | didn't match pod topology spread constraints | Check the current distribution with kubectl get pods -o wide | Relax with ScheduleAnyway, adjust maxSkew |
| Per-node pod count ceiling | Too many pods | Query allocatable.pods with kubectl get node -o jsonpath | Change the CNI configuration or add nodes |
| Node NotReady/cordon | node(s) were unschedulable | kubectl get nodes | Recover the node or uncordon it |
Resource Shortage — the Yardstick Is requests, Not Usage
This is the most common branch and, at the same time, the one most frequently misdiagnosed.
kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ip-10-0-1-14 842m 21% 4118Mi 31%
ip-10-0-2-77 1105m 27% 5203Mi 39%
ip-10-0-3-91 667m 16% 3877Mi 29%
Every node sits in the 20 percent range for CPU and the 30 percent range for memory. If you jump from here to the conclusion "there is capacity to spare, so why will it not schedule", you will burn hours. The scheduler does not look at actual usage. It looks only at the sum of requests across the Pods already placed on the node.
kubectl describe node ip-10-0-1-14 | grep -A10 "Allocated resources"
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 3730m (93%) 6200m (155%)
memory 11544Mi (89%) 15Gi (118%)
ephemeral-storage 0 (0%) 0 (0%)
pods 24 24
Actual usage is 21 percent, reserved capacity is 93 percent. From the scheduler point of view this node is nearly full. The cause is usually a handful of Pods that claimed excessive requests.
kubectl get pods -A --field-selector spec.nodeName=ip-10-0-1-14 \
-o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,MEM:.spec.containers[*].resources.requests.memory' \
| sort -k3 -hr | head -6
NS POD CPU MEM
analytics spark-executor-3 1500m 4Gi
analytics spark-executor-1 1500m 4Gi
search indexer-6c4d9f7b8-h2klp 500m 2Gi
payments ledger-5f8b7c6d9-nm3xt 200m 1Gi
Two Spark executors reserve 3 CPU cores and barely use any of it in practice.
At this point you have to draw a precise line between the roles of requests and limits.
- requests — the sole yardstick for scheduling. The remaining room is computed by subtracting these values from the node allocatable. At container runtime it acts as a cgroup weight for CPU, and for memory it carries no enforcement at all
- limits — plays no part whatsoever in scheduling. It is nothing but a runtime ceiling. CPU limits cause throttling, memory limits cause OOM kills
So no matter how large you set limits, that alone never blocks scheduling; conversely, large requests occupy the node even when the Pod does not actually use the capacity.
A common wrong answer: deleting requests entirely so scheduling goes through. The Pod comes up, but its QoS class becomes BestEffort, so it is the first thing evicted the moment the node comes under pressure. On top of that the Pod counts as 0 in the scheduler arithmetic, which overpacks the node and puts the neighboring Pods at risk as well. The correct fix is to recompute requests from actual usage.
kubectl top pod -A --containers --sort-by=cpu | head -6
NAMESPACE POD NAME CPU(cores) MEMORY(bytes)
analytics spark-executor-3 executor 180m 1204Mi
analytics spark-executor-1 executor 166m 1189Mi
search indexer-6c4... indexer 410m 1802Mi
It reserves 1500m and uses 180m. Lowering it to roughly 400m frees up an entire node.
When the Node Will Not Take the Pod
Taints and Tolerations
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key,EFFECT:.spec.taints[*].effect'
NAME TAINTS EFFECT
ip-10-0-1-14 dedicated NoSchedule
ip-10-0-2-77 <none> <none>
ip-10-0-3-91 node.kubernetes.io/disk-pressure NoSchedule
ip-10-0-4-22 nvidia.com/gpu NoSchedule
Only the second node has no taint. The taint on the third node was not attached by a person — it is something the kubelet attached automatically. When you see taints in the disk-pressure, memory-pressure, pid-pressure, unreachable, or not-ready family, that is a node failure signal, not a placement policy. Rather than working around it by attaching a toleration, you should repair the node.
If the taint is intentional, attach a toleration.
spec:
tolerations:
- key: dedicated
operator: Equal
value: batch
effect: NoSchedule
Node Selector and Affinity Mismatch
When affinity/selector shows up in the message, start by checking whether a node matching the conditions actually exists.
kubectl get pod clickhouse-0 -n analytics -o jsonpath='{.spec.nodeSelector}'
{"disktype":"nvme","topology.kubernetes.io/zone":"ap-northeast-2c"}
kubectl get nodes -l disktype=nvme,topology.kubernetes.io/zone=ap-northeast-2c
No resources found
Removing the conditions one at a time reveals which side is at fault.
kubectl get nodes -l disktype=nvme
NAME STATUS ROLES AGE VERSION
ip-10-0-1-14 Ready <none> 9d v1.31.4
kubectl get nodes -l topology.kubernetes.io/zone=ap-northeast-2c
No resources found
The nvme node exists, but there is not a single node in that zone. Either the zone condition is wrong, or the node group in that zone was scaled down to 0.
The Per-Node Pod Count Ceiling and NotReady
If the message is Too many pods, the problem is a count, not resources.
kubectl get nodes -o custom-columns='NAME:.metadata.name,MAXPODS:.status.allocatable.pods'
NAME MAXPODS
ip-10-0-1-14 29
ip-10-0-2-77 29
ip-10-0-3-91 29
On EKS with the VPC CNI, the ceiling is determined by how many ENIs and IPs a node can attach, so with a small instance type the node cannot take more pods even when CPU and memory are left over. You have to enable prefix delegation or move to a larger instance type.
It is also common for the node itself to have dropped out.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-1-14 Ready <none> 9d v1.31.4
ip-10-0-2-77 Ready,SchedulingDisabled <none> 9d v1.31.4
ip-10-0-3-91 NotReady <none> 9d v1.31.4
SchedulingDisabled means someone cordoned the node. In most cases somebody forgot to uncordon it after maintenance.
Storage and Placement Constraints
An Unbound PVC, and Pending That Is Perfectly Normal
When the Pod is Pending and the PVC is Pending too, most people suspect storage, but there are two situations here that you absolutely must tell apart.
First, the case that is a genuine problem.
kubectl describe pvc clickhouse-data -n analytics | tail -5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 9m (x14 over 12m) persistentvolume-controller storageclass.storage.k8s.io "fast-ssd" not found
The StorageClass name is wrong. This is the classic case where the name changed during a cluster migration.
kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp3 (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 61d
io2 ebs.csi.aws.com Delete WaitForFirstConsumer true 61d
Now the case that is normal.
kubectl describe pvc report-cache -n analytics | tail -4
Events:
Type Reason Age From Message
---- ---- ---- ---- -------
Normal WaitForFirstConsumer 12s (x8 over 2m) persistentvolume-controller waiting for first consumer to be created before binding
This is not an error. A StorageClass whose VOLUMEBINDINGMODE is WaitForFirstConsumer creates the volume in the same zone as the node only after the Pod has been placed on a node. The volume comes into being only once the Pod is placed, so a Pending PVC is exactly as designed.
The trouble starts here. If the Pod cannot be placed for some other reason, the PVC stays Pending forever as well. People then look at the PVC and set out to fix storage, when what actually needs fixing is the scheduling failure of the Pod. If the PVC is waiting on WaitForFirstConsumer, ignore the PVC and look only at the FailedScheduling message on the Pod.
If nodes are replaced after the volume has already been created, the zones can fall out of alignment.
kubectl describe pod clickhouse-0 -n analytics | grep FailedScheduling
Warning FailedScheduling 4m default-scheduler 0/5 nodes are available: 5 node(s) had volume node affinity conflict.
It means the volume lives in ap-northeast-2a but there is no schedulable node in that zone.
kubectl get pv -o custom-columns='NAME:.metadata.name,ZONE:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values' | head -3
NAME ZONE
pvc-8f3c1d5b-2e7a-94c0-f13d-68b5a2e9c47f [ap-northeast-2a]
Self-Exclusion Caused by Anti-Affinity
If you apply required host-level anti-affinity to 3 replicas but have only 2 nodes, the third Pod stays Pending forever.
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: report-worker
topologyKey: kubernetes.io/hostname
kubectl describe pod report-worker-7f4c8d9b6-hs2p -n analytics | grep FailedScheduling
Warning FailedScheduling 9m default-scheduler 0/2 nodes are available: 2 node(s) didn't match pod anti-affinity rules.
There are clearly times when this constraint is needed, but the moment the replica count exceeds the topology count it silently blocks the rollout. If availability is the goal, preferred is usually enough.
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: report-worker
topologyKey: kubernetes.io/hostname
topologySpreadConstraints
Spread constraints are gentler than anti-affinity, but if you leave whenUnsatisfiable at DoNotSchedule they block placement just the same.
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: report-worker
With 3 zones, if every node in one zone fills up, placement is refused because a skew of 1 cannot be maintained even though there is room in another zone.
kubectl get pods -n analytics -l app=report-worker -o wide
NAME READY STATUS NODE ZONE
report-worker-7f4c8d9b6-k9vt 1/1 Running ip-10-0-1-14 2a
report-worker-7f4c8d9b6-m2xd 1/1 Running ip-10-0-2-77 2b
report-worker-7f4c8d9b6-hs2p 0/1 Pending <none> <none>
Unless availability is an absolute requirement, switching to ScheduleAnyway is the pragmatic move. The constraint then feeds only into scoring and does not block placement.
Why Cluster Autoscaler Does Not Add Nodes
You expect the autoscaler to add capacity when nodes run short, but there are plenty of cases where it does not. Here too the reason is left in an event.
kubectl describe pod clickhouse-0 -n analytics | grep -A2 NotTriggerScaleUp
Normal NotTriggerScaleUp 2m17s (x21 over 9m) cluster-autoscaler pod didn't trigger scale-up: 1 max node group size reached, 2 node(s) had untolerated taint {dedicated: batch}, 1 node(s) had volume node affinity conflict
It is telling you three things at once. One node group has reached its maximum size, two groups carry a taint this Pod cannot tolerate, and one group has a volume zone that does not match. The autoscaler simulates in advance whether this Pod could be placed assuming a new node were brought up, so once it decides placement is impossible, it does not add nodes.
Look at the controller-side logs and the status ConfigMap as well.
kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200 | grep -i "no expansion\|max size\|scale_up"
I0726 09:52:14.118 scale_up.go:300] Pod analytics/clickhouse-0 can't be scheduled on eks-general-2c. Predicate checking error: node(s) had volume node affinity conflict
I0726 09:52:14.119 scale_up.go:472] No expansion options
kubectl -n kube-system get configmap cluster-autoscaler-status \
-o jsonpath='{.data.status}' | head -12
Cluster-autoscaler status at 2026-07-26 09:52:14:
Cluster-wide:
Health: Healthy
ScaleUp: NoActivity (ready=5 registered=5)
ScaleDown: NoCandidates
NodeGroups:
Name: eks-general-2a
Health: Healthy (ready=3 cloudProviderTarget=3 minSize=1 maxSize=3)
ScaleUp: NoActivity
When maxSize and cloudProviderTarget are equal, you have hit the ceiling.
A common wrong answer: adding nodes unconditionally whenever Pending shows up. Anti-affinity, spread constraints, and volume zone mismatches either do not clear up when you add nodes, or the nodes land in the wrong zone and only raise the bill. Read the NotTriggerScaleUp message first and remove the reason it names.
Preventing Recurrence
Pending is usually the result of requests being sized with no evidence behind them. Three mechanisms cut it down.
First, enforce defaults and ceilings on the namespace. This keeps Pods that omitted requests from getting in.
apiVersion: v1
kind: LimitRange
metadata:
name: default-requests
namespace: analytics
spec:
limits:
- type: Container
default:
cpu: 500m
memory: 512Mi
defaultRequest:
cpu: 100m
memory: 256Mi
max:
cpu: 4
memory: 16Gi
Second, run VPA in recommendation mode to obtain suggested values grounded in actual usage. Even without going as far as automatic application, just seeing the numbers is worth it.
kubectl get vpa spark-executor -n analytics -o jsonpath='{.status.recommendation.containerRecommendations[0]}'
{"containerName":"executor","lowerBound":{"cpu":"180m","memory":"1000Mi"},"target":{"cpu":"320m","memory":"1400Mi"},"upperBound":{"cpu":"1200m","memory":"3Gi"}}
Third, catch the duration of Pending with an alert. Placement failures never show up in application metrics at all, so a separate rule is required.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: scheduling-alerts
namespace: monitoring
spec:
groups:
- name: scheduling
rules:
- alert: PodPendingTooLong
expr: kube_pod_status_phase{phase="Pending"} == 1
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} has been Pending for 15m"
- alert: NodeRequestsNearlyFull
expr: sum by (node) (kube_pod_container_resource_requests{resource="cpu"}) / on(node) kube_node_status_allocatable{resource="cpu"} > 0.9
for: 30m
labels:
severity: warning
Wrapping Up — the Scheduler Already Told You the Answer
There is almost never a moment in Pending diagnosis that calls for guessing. One sentence in the FailedScheduling event tallies the node count and the elimination reasons, and each reason maps to a single row in the table above. Once you remember that the filter chain stops at the first failure per node, so the reasons surface one after another, the job turns into peeling them off one at a time.
The one sentence to remember is this. Scheduling is decided by the arithmetic of requests, not by actual usage, and just because kubectl top looks idle does not mean there is room on the node.