Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — the Pod Has Been Sitting in Pending for 10 Minutes

The rollout reports success, but the Pod does not move.

kubectl get pod -n analytics

NAME                          READY   STATUS    RESTARTS   AGE
clickhouse-0                  0/1     Pending   0          11m
report-worker-7f4c8d9b6-hs2p  0/1     Pending   0          11m
report-worker-7f4c8d9b6-k9vt  1/1     Running   0          11m

Pending means the Pod object was created but no node has been chosen yet. This is not a container problem but a placement problem, so looking at the logs shows nothing. The container never started at all.

kubectl logs clickhouse-0 -n analytics

Error from server (BadRequest): container "clickhouse" in pod "clickhouse-0" is waiting to start: ContainerCreating

Fortunately the scheduler records why it could not place the Pod as an event, every single time. That one sentence is the entire diagnosis.

How to Decode the Scheduler Rejection Message

kubectl describe pod clickhouse-0 -n analytics | tail -6

Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  11m                   default-scheduler  0/5 nodes are available: 2 Insufficient cpu, 1 node(s) had untolerated taint {dedicated: batch}, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.

You need to know the structure of this sentence.

0/5 — 0 out of a total of 5 nodes are viable. The leading number is the count of nodes that passed
What comes after the colon is a tally of the elimination reason per node. 2 + 1 + 2 = 5 matches the node count exactly
The final preemption: clause means that even priority-based preemption could not free up a slot

The second item carries a decisive implication. For each node, the scheduler filter chain stops at the first plugin that fails. So a node is tallied under only one reason. A node that is both short on CPU and carrying a taint shows only the earlier reason. In practice this means it is normal for the next reason to appear the moment you fix the one that was reported. You have to peel them off one at a time.

Events have a default retention of one hour, so they may already be gone on an old Pending Pod. In that case, recreate the Pod to produce fresh events.

kubectl get events -n analytics --field-selector reason=FailedScheduling --sort-by=.lastTimestamp | tail -3

LAST SEEN   TYPE      REASON             OBJECT             MESSAGE
3m41s       Warning   FailedScheduling   pod/clickhouse-0   0/5 nodes are available: 2 Insufficient cpu, ...

Cause Branches and Diagnosis

Cause	Scheduler message	Diagnostic command	Fix
CPU/memory requests exceeded	Insufficient cpu, Insufficient memory	Allocated resources in `kubectl describe node`	Lower requests or add nodes
No toleration	had untolerated taint	Taints in `kubectl describe node`	Add tolerations or remove the taint
Selector/affinity mismatch	didn't match Pod's node affinity/selector	`kubectl get nodes --show-labels`	Label the node or relax the condition
PVC not bound	pod has unbound immediate PersistentVolumeClaims	`kubectl describe pvc`	Check that the StorageClass exists and the provisioner runs
Volume zone mismatch	volume node affinity conflict	Check nodeAffinity on the PV	Secure a node in the same zone
Pod anti-affinity self-exclusion	didn't match pod anti-affinity rules	Compare the replica count with the topology count	Relax to preferred or add nodes
Spread constraint violation	didn't match pod topology spread constraints	Check the current distribution with `kubectl get pods -o wide`	Relax with ScheduleAnyway, adjust maxSkew
Per-node pod count ceiling	Too many pods	Query allocatable.pods with `kubectl get node -o jsonpath`	Change the CNI configuration or add nodes
Node NotReady/cordon	node(s) were unschedulable	`kubectl get nodes`	Recover the node or uncordon it

Resource Shortage — the Yardstick Is requests, Not Usage

This is the most common branch and, at the same time, the one most frequently misdiagnosed.

kubectl top node

NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ip-10-0-1-14   842m         21%    4118Mi          31%
ip-10-0-2-77   1105m        27%    5203Mi          39%
ip-10-0-3-91   667m         16%    3877Mi          29%

Every node sits in the 20 percent range for CPU and the 30 percent range for memory. If you jump from here to the conclusion "there is capacity to spare, so why will it not schedule", you will burn hours. The scheduler does not look at actual usage. It looks only at the sum of requests across the Pods already placed on the node.

kubectl describe node ip-10-0-1-14 | grep -A10 "Allocated resources"

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests       Limits
  --------           --------       ------
  cpu                3730m (93%)    6200m (155%)
  memory             11544Mi (89%)  15Gi (118%)
  ephemeral-storage  0 (0%)         0 (0%)
  pods               24             24

Actual usage is 21 percent, reserved capacity is 93 percent. From the scheduler point of view this node is nearly full. The cause is usually a handful of Pods that claimed excessive requests.

kubectl get pods -A --field-selector spec.nodeName=ip-10-0-1-14 \
  -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,MEM:.spec.containers[*].resources.requests.memory' \
  | sort -k3 -hr | head -6

NS          POD                            CPU     MEM
analytics   spark-executor-3               1500m   4Gi
analytics   spark-executor-1               1500m   4Gi
search      indexer-6c4d9f7b8-h2klp        500m    2Gi
payments    ledger-5f8b7c6d9-nm3xt         200m    1Gi

Two Spark executors reserve 3 CPU cores and barely use any of it in practice.

At this point you have to draw a precise line between the roles of requests and limits.

requests — the sole yardstick for scheduling. The remaining room is computed by subtracting these values from the node allocatable. At container runtime it acts as a cgroup weight for CPU, and for memory it carries no enforcement at all
limits — plays no part whatsoever in scheduling. It is nothing but a runtime ceiling. CPU limits cause throttling, memory limits cause OOM kills

So no matter how large you set limits, that alone never blocks scheduling; conversely, large requests occupy the node even when the Pod does not actually use the capacity.

A common wrong answer: deleting requests entirely so scheduling goes through. The Pod comes up, but its QoS class becomes BestEffort, so it is the first thing evicted the moment the node comes under pressure. On top of that the Pod counts as 0 in the scheduler arithmetic, which overpacks the node and puts the neighboring Pods at risk as well. The correct fix is to recompute requests from actual usage.

kubectl top pod -A --containers --sort-by=cpu | head -6

NAMESPACE   POD                 NAME              CPU(cores)   MEMORY(bytes)
analytics   spark-executor-3    executor          180m         1204Mi
analytics   spark-executor-1    executor          166m         1189Mi
search      indexer-6c4...      indexer           410m         1802Mi

It reserves 1500m and uses 180m. Lowering it to roughly 400m frees up an entire node.

When the Node Will Not Take the Pod

Taints and Tolerations

kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key,EFFECT:.spec.taints[*].effect'

NAME           TAINTS                              EFFECT
ip-10-0-1-14   dedicated                           NoSchedule
ip-10-0-2-77   <none>                              <none>
ip-10-0-3-91   node.kubernetes.io/disk-pressure    NoSchedule
ip-10-0-4-22   nvidia.com/gpu                      NoSchedule

Only the second node has no taint. The taint on the third node was not attached by a person — it is something the kubelet attached automatically. When you see taints in the disk-pressure, memory-pressure, pid-pressure, unreachable, or not-ready family, that is a node failure signal, not a placement policy. Rather than working around it by attaching a toleration, you should repair the node.

If the taint is intentional, attach a toleration.

spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: batch
      effect: NoSchedule

Node Selector and Affinity Mismatch

When affinity/selector shows up in the message, start by checking whether a node matching the conditions actually exists.

kubectl get pod clickhouse-0 -n analytics -o jsonpath='{.spec.nodeSelector}'

{"disktype":"nvme","topology.kubernetes.io/zone":"ap-northeast-2c"}

kubectl get nodes -l disktype=nvme,topology.kubernetes.io/zone=ap-northeast-2c

No resources found

Removing the conditions one at a time reveals which side is at fault.

kubectl get nodes -l disktype=nvme

NAME           STATUS   ROLES    AGE   VERSION
ip-10-0-1-14   Ready    <none>   9d    v1.31.4

kubectl get nodes -l topology.kubernetes.io/zone=ap-northeast-2c

No resources found

The nvme node exists, but there is not a single node in that zone. Either the zone condition is wrong, or the node group in that zone was scaled down to 0.

The Per-Node Pod Count Ceiling and NotReady

If the message is Too many pods, the problem is a count, not resources.

kubectl get nodes -o custom-columns='NAME:.metadata.name,MAXPODS:.status.allocatable.pods'

NAME           MAXPODS
ip-10-0-1-14   29
ip-10-0-2-77   29
ip-10-0-3-91   29

On EKS with the VPC CNI, the ceiling is determined by how many ENIs and IPs a node can attach, so with a small instance type the node cannot take more pods even when CPU and memory are left over. You have to enable prefix delegation or move to a larger instance type.

It is also common for the node itself to have dropped out.

kubectl get nodes

NAME           STATUS                     ROLES    AGE   VERSION
ip-10-0-1-14   Ready                      <none>   9d    v1.31.4
ip-10-0-2-77   Ready,SchedulingDisabled   <none>   9d    v1.31.4
ip-10-0-3-91   NotReady                   <none>   9d    v1.31.4

SchedulingDisabled means someone cordoned the node. In most cases somebody forgot to uncordon it after maintenance.

Storage and Placement Constraints

An Unbound PVC, and Pending That Is Perfectly Normal

When the Pod is Pending and the PVC is Pending too, most people suspect storage, but there are two situations here that you absolutely must tell apart.

First, the case that is a genuine problem.

kubectl describe pvc clickhouse-data -n analytics | tail -5

Events:
  Type     Reason              Age                From                         Message
  ----     ------              ----               ----                         -------
  Warning  ProvisioningFailed  9m (x14 over 12m)  persistentvolume-controller  storageclass.storage.k8s.io "fast-ssd" not found

The StorageClass name is wrong. This is the classic case where the name changed during a cluster migration.

kubectl get storageclass

NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp3 (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   61d
io2             ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   61d

Now the case that is normal.

kubectl describe pvc report-cache -n analytics | tail -4

Events:
  Type    Reason                Age                From                         Message
  ----    ----                  ----               ----                         -------
  Normal  WaitForFirstConsumer  12s (x8 over 2m)   persistentvolume-controller  waiting for first consumer to be created before binding

This is not an error. A StorageClass whose VOLUMEBINDINGMODE is WaitForFirstConsumer creates the volume in the same zone as the node only after the Pod has been placed on a node. The volume comes into being only once the Pod is placed, so a Pending PVC is exactly as designed.

The trouble starts here. If the Pod cannot be placed for some other reason, the PVC stays Pending forever as well. People then look at the PVC and set out to fix storage, when what actually needs fixing is the scheduling failure of the Pod. If the PVC is waiting on WaitForFirstConsumer, ignore the PVC and look only at the FailedScheduling message on the Pod.

If nodes are replaced after the volume has already been created, the zones can fall out of alignment.

kubectl describe pod clickhouse-0 -n analytics | grep FailedScheduling

  Warning  FailedScheduling  4m  default-scheduler  0/5 nodes are available: 5 node(s) had volume node affinity conflict.

It means the volume lives in ap-northeast-2a but there is no schedulable node in that zone.

kubectl get pv -o custom-columns='NAME:.metadata.name,ZONE:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values' | head -3

NAME                                       ZONE
pvc-8f3c1d5b-2e7a-94c0-f13d-68b5a2e9c47f   [ap-northeast-2a]

Self-Exclusion Caused by Anti-Affinity

If you apply required host-level anti-affinity to 3 replicas but have only 2 nodes, the third Pod stays Pending forever.

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: report-worker
          topologyKey: kubernetes.io/hostname

kubectl describe pod report-worker-7f4c8d9b6-hs2p -n analytics | grep FailedScheduling

  Warning  FailedScheduling  9m  default-scheduler  0/2 nodes are available: 2 node(s) didn't match pod anti-affinity rules.

There are clearly times when this constraint is needed, but the moment the replica count exceeds the topology count it silently blocks the rollout. If availability is the goal, preferred is usually enough.

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app: report-worker
            topologyKey: kubernetes.io/hostname

topologySpreadConstraints

Spread constraints are gentler than anti-affinity, but if you leave whenUnsatisfiable at DoNotSchedule they block placement just the same.

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: report-worker

With 3 zones, if every node in one zone fills up, placement is refused because a skew of 1 cannot be maintained even though there is room in another zone.

kubectl get pods -n analytics -l app=report-worker -o wide

NAME                            READY   STATUS    NODE           ZONE
report-worker-7f4c8d9b6-k9vt    1/1     Running   ip-10-0-1-14   2a
report-worker-7f4c8d9b6-m2xd    1/1     Running   ip-10-0-2-77   2b
report-worker-7f4c8d9b6-hs2p    0/1     Pending   <none>         <none>

Unless availability is an absolute requirement, switching to ScheduleAnyway is the pragmatic move. The constraint then feeds only into scoring and does not block placement.

Why Cluster Autoscaler Does Not Add Nodes

You expect the autoscaler to add capacity when nodes run short, but there are plenty of cases where it does not. Here too the reason is left in an event.

kubectl describe pod clickhouse-0 -n analytics | grep -A2 NotTriggerScaleUp

  Normal   NotTriggerScaleUp  2m17s (x21 over 9m)  cluster-autoscaler  pod didn't trigger scale-up: 1 max node group size reached, 2 node(s) had untolerated taint {dedicated: batch}, 1 node(s) had volume node affinity conflict

It is telling you three things at once. One node group has reached its maximum size, two groups carry a taint this Pod cannot tolerate, and one group has a volume zone that does not match. The autoscaler simulates in advance whether this Pod could be placed assuming a new node were brought up, so once it decides placement is impossible, it does not add nodes.

Look at the controller-side logs and the status ConfigMap as well.

kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200 | grep -i "no expansion\|max size\|scale_up"

I0726 09:52:14.118  scale_up.go:300] Pod analytics/clickhouse-0 can't be scheduled on eks-general-2c. Predicate checking error: node(s) had volume node affinity conflict
I0726 09:52:14.119  scale_up.go:472] No expansion options

kubectl -n kube-system get configmap cluster-autoscaler-status \
  -o jsonpath='{.data.status}' | head -12

Cluster-autoscaler status at 2026-07-26 09:52:14:
Cluster-wide:
  Health:      Healthy
  ScaleUp:     NoActivity (ready=5 registered=5)
  ScaleDown:   NoCandidates

NodeGroups:
  Name:        eks-general-2a
  Health:      Healthy (ready=3 cloudProviderTarget=3 minSize=1 maxSize=3)
  ScaleUp:     NoActivity

When maxSize and cloudProviderTarget are equal, you have hit the ceiling.

A common wrong answer: adding nodes unconditionally whenever Pending shows up. Anti-affinity, spread constraints, and volume zone mismatches either do not clear up when you add nodes, or the nodes land in the wrong zone and only raise the bill. Read the NotTriggerScaleUp message first and remove the reason it names.

Preventing Recurrence

Pending is usually the result of requests being sized with no evidence behind them. Three mechanisms cut it down.

First, enforce defaults and ceilings on the namespace. This keeps Pods that omitted requests from getting in.

apiVersion: v1
kind: LimitRange
metadata:
  name: default-requests
  namespace: analytics
spec:
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 256Mi
      max:
        cpu: 4
        memory: 16Gi

Second, run VPA in recommendation mode to obtain suggested values grounded in actual usage. Even without going as far as automatic application, just seeing the numbers is worth it.

kubectl get vpa spark-executor -n analytics -o jsonpath='{.status.recommendation.containerRecommendations[0]}'

{"containerName":"executor","lowerBound":{"cpu":"180m","memory":"1000Mi"},"target":{"cpu":"320m","memory":"1400Mi"},"upperBound":{"cpu":"1200m","memory":"3Gi"}}

Third, catch the duration of Pending with an alert. Placement failures never show up in application metrics at all, so a separate rule is required.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: scheduling-alerts
  namespace: monitoring
spec:
  groups:
    - name: scheduling
      rules:
        - alert: PodPendingTooLong
          expr: kube_pod_status_phase{phase="Pending"} == 1
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.namespace }}/{{ $labels.pod }} has been Pending for 15m"
        - alert: NodeRequestsNearlyFull
          expr: sum by (node) (kube_pod_container_resource_requests{resource="cpu"}) / on(node) kube_node_status_allocatable{resource="cpu"} > 0.9
          for: 30m
          labels:
            severity: warning

Wrapping Up — the Scheduler Already Told You the Answer

There is almost never a moment in Pending diagnosis that calls for guessing. One sentence in the FailedScheduling event tallies the node count and the elimination reasons, and each reason maps to a single row in the table above. Once you remember that the filter chain stops at the first failure per node, so the reasons surface one after another, the job turns into peeling them off one at a time.

The one sentence to remember is this. Scheduling is decided by the arithmetic of requests, not by actual usage, and just because kubectl top looks idle does not mean there is room on the node.