Kubernetes Pod Pending 状態の解決 — スケジューラーが残した拒否理由の読み方

kubectl get pod -n analytics

NAME                          READY   STATUS    RESTARTS   AGE
clickhouse-0                  0/1     Pending   0          11m
report-worker-7f4c8d9b6-hs2p  0/1     Pending   0          11m
report-worker-7f4c8d9b6-k9vt  1/1     Running   0          11m

kubectl logs clickhouse-0 -n analytics

Error from server (BadRequest): container "clickhouse" in pod "clickhouse-0" is waiting to start: ContainerCreating

kubectl describe pod clickhouse-0 -n analytics | tail -6

Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  11m                   default-scheduler  0/5 nodes are available: 2 Insufficient cpu, 1 node(s) had untolerated taint {dedicated: batch}, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.

kubectl get events -n analytics --field-selector reason=FailedScheduling --sort-by=.lastTimestamp | tail -3

LAST SEEN   TYPE      REASON             OBJECT             MESSAGE
3m41s       Warning   FailedScheduling   pod/clickhouse-0   0/5 nodes are available: 2 Insufficient cpu, ...

kubectl top node

NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ip-10-0-1-14   842m         21%    4118Mi          31%
ip-10-0-2-77   1105m        27%    5203Mi          39%
ip-10-0-3-91   667m         16%    3877Mi          29%

kubectl describe node ip-10-0-1-14 | grep -A10 "Allocated resources"

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests       Limits
  --------           --------       ------
  cpu                3730m (93%)    6200m (155%)
  memory             11544Mi (89%)  15Gi (118%)
  ephemeral-storage  0 (0%)         0 (0%)
  pods               24             24

kubectl get pods -A --field-selector spec.nodeName=ip-10-0-1-14 \
  -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,MEM:.spec.containers[*].resources.requests.memory' \
  | sort -k3 -hr | head -6

NS          POD                            CPU     MEM
analytics   spark-executor-3               1500m   4Gi
analytics   spark-executor-1               1500m   4Gi
search      indexer-6c4d9f7b8-h2klp        500m    2Gi
payments    ledger-5f8b7c6d9-nm3xt         200m    1Gi

kubectl top pod -A --containers --sort-by=cpu | head -6

NAMESPACE   POD                 NAME              CPU(cores)   MEMORY(bytes)
analytics   spark-executor-3    executor          180m         1204Mi
analytics   spark-executor-1    executor          166m         1189Mi
search      indexer-6c4...      indexer           410m         1802Mi

kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key,EFFECT:.spec.taints[*].effect'

NAME           TAINTS                              EFFECT
ip-10-0-1-14   dedicated                           NoSchedule
ip-10-0-2-77   <none>                              <none>
ip-10-0-3-91   node.kubernetes.io/disk-pressure    NoSchedule
ip-10-0-4-22   nvidia.com/gpu                      NoSchedule

spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: batch
      effect: NoSchedule

kubectl get pod clickhouse-0 -n analytics -o jsonpath='{.spec.nodeSelector}'

{"disktype":"nvme","topology.kubernetes.io/zone":"ap-northeast-2c"}

kubectl get nodes -l disktype=nvme,topology.kubernetes.io/zone=ap-northeast-2c

No resources found

kubectl get nodes -l disktype=nvme

NAME           STATUS   ROLES    AGE   VERSION
ip-10-0-1-14   Ready    <none>   9d    v1.31.4

kubectl get nodes -l topology.kubernetes.io/zone=ap-northeast-2c

No resources found

kubectl get nodes -o custom-columns='NAME:.metadata.name,MAXPODS:.status.allocatable.pods'

NAME           MAXPODS
ip-10-0-1-14   29
ip-10-0-2-77   29
ip-10-0-3-91   29

kubectl get nodes

NAME           STATUS                     ROLES    AGE   VERSION
ip-10-0-1-14   Ready                      <none>   9d    v1.31.4
ip-10-0-2-77   Ready,SchedulingDisabled   <none>   9d    v1.31.4
ip-10-0-3-91   NotReady                   <none>   9d    v1.31.4

kubectl describe pvc clickhouse-data -n analytics | tail -5

Events:
  Type     Reason              Age                From                         Message
  ----     ------              ----               ----                         -------
  Warning  ProvisioningFailed  9m (x14 over 12m)  persistentvolume-controller  storageclass.storage.k8s.io "fast-ssd" not found

kubectl get storageclass

NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp3 (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   61d
io2             ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   61d

kubectl describe pvc report-cache -n analytics | tail -4

Events:
  Type    Reason                Age                From                         Message
  ----    ----                  ----               ----                         -------
  Normal  WaitForFirstConsumer  12s (x8 over 2m)   persistentvolume-controller  waiting for first consumer to be created before binding

kubectl describe pod clickhouse-0 -n analytics | grep FailedScheduling

  Warning  FailedScheduling  4m  default-scheduler  0/5 nodes are available: 5 node(s) had volume node affinity conflict.

kubectl get pv -o custom-columns='NAME:.metadata.name,ZONE:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values' | head -3

NAME                                       ZONE
pvc-8f3c1d5b-2e7a-94c0-f13d-68b5a2e9c47f   [ap-northeast-2a]

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: report-worker
          topologyKey: kubernetes.io/hostname

kubectl describe pod report-worker-7f4c8d9b6-hs2p -n analytics | grep FailedScheduling

  Warning  FailedScheduling  9m  default-scheduler  0/2 nodes are available: 2 node(s) didn't match pod anti-affinity rules.

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app: report-worker
            topologyKey: kubernetes.io/hostname

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: report-worker

kubectl get pods -n analytics -l app=report-worker -o wide

NAME                            READY   STATUS    NODE           ZONE
report-worker-7f4c8d9b6-k9vt    1/1     Running   ip-10-0-1-14   2a
report-worker-7f4c8d9b6-m2xd    1/1     Running   ip-10-0-2-77   2b
report-worker-7f4c8d9b6-hs2p    0/1     Pending   <none>         <none>

kubectl describe pod clickhouse-0 -n analytics | grep -A2 NotTriggerScaleUp

  Normal   NotTriggerScaleUp  2m17s (x21 over 9m)  cluster-autoscaler  pod didn't trigger scale-up: 1 max node group size reached, 2 node(s) had untolerated taint {dedicated: batch}, 1 node(s) had volume node affinity conflict

kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200 | grep -i "no expansion\|max size\|scale_up"

I0726 09:52:14.118  scale_up.go:300] Pod analytics/clickhouse-0 can't be scheduled on eks-general-2c. Predicate checking error: node(s) had volume node affinity conflict
I0726 09:52:14.119  scale_up.go:472] No expansion options

kubectl -n kube-system get configmap cluster-autoscaler-status \
  -o jsonpath='{.data.status}' | head -12

Cluster-autoscaler status at 2026-07-26 09:52:14:
Cluster-wide:
  Health:      Healthy
  ScaleUp:     NoActivity (ready=5 registered=5)
  ScaleDown:   NoCandidates

NodeGroups:
  Name:        eks-general-2a
  Health:      Healthy (ready=3 cloudProviderTarget=3 minSize=1 maxSize=3)
  ScaleUp:     NoActivity

apiVersion: v1
kind: LimitRange
metadata:
  name: default-requests
  namespace: analytics
spec:
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 256Mi
      max:
        cpu: 4
        memory: 16Gi

kubectl get vpa spark-executor -n analytics -o jsonpath='{.status.recommendation.containerRecommendations[0]}'

{"containerName":"executor","lowerBound":{"cpu":"180m","memory":"1000Mi"},"target":{"cpu":"320m","memory":"1400Mi"},"upperBound":{"cpu":"1200m","memory":"3Gi"}}

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: scheduling-alerts
  namespace: monitoring
spec:
  groups:
    - name: scheduling
      rules:
        - alert: PodPendingTooLong
          expr: kube_pod_status_phase{phase="Pending"} == 1
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.namespace }}/{{ $labels.pod }} has been Pending for 15m"
        - alert: NodeRequestsNearlyFull
          expr: sum by (node) (kube_pod_container_resource_requests{resource="cpu"}) / on(node) kube_node_status_allocatable{resource="cpu"} > 0.9
          for: 30m
          labels:
            severity: warning