Skip to content
Published on

The Workload/PodGroup API in Kubernetes v1.36 — Gang Scheduling Is Moving Into kube-scheduler

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — What Happens When You Schedule Pods One at a Time

Anyone who has run a distributed training job on Kubernetes knows this failure mode. Training only starts once all 4 workers are up, but the default scheduler places pods one at a time, independently. If only enough GPU capacity for 3 pods is left, those 3 pods claim nodes and sit there, waiting forever for the 4th. Three GPUs sit occupied, doing nothing. If two jobs do this at the same time, you can end up with a deadlock where each holds resources the other needs.

KEP-4671's motivation section states this plainly — parallel applications need communication among all pods to begin execution, and ongoing communication such as barriers or all-reduce to make progress. So all pods need to start at close to the same time, or expensive compute sits idle, or the application dies from a communication timeout.

The solution has existed for a long time: gang scheduling, "all or nothing" placement. The problem is that the solution has always lived outside the cluster — install Volcano, bolt on Kueue, or layer in the coscheduling plugin. That ecosystem side of the story is already covered in Kubernetes AI Training Pipeline: Analyzing Volcano, Training Operator, and Kueue.

This post is about a different story. In Kubernetes v1.36, released April 22, 2026 (codename ハル / Haru, 70 enhancements total — 18 Stable, 25 Beta, 25 Alpha), the work of bringing gang scheduling into kube-scheduler itself entered its second phase. And along the way, the API that had shipped just six months earlier got replaced wholesale.

Why Kubernetes Decided to Step In

Why does the core need to get involved when working implementations already exist outside it? KEP-4671 answers directly: gang scheduling has been implemented at least four times outside kube-scheduler, and some controllers have started supporting multiple gang schedulers simultaneously just to be portable across clusters that use different ones.

That's where the problem shows up. The parties that want gang scheduling are workload controllers — Job, JobSet, LeaderWorkerSet, MPIJob, TrainJob — and each of them has to carry separate code paths for Volcano and for Kueue, because there's no standard interface. The KEP aims at three things: make gang scheduling available on every Kubernetes distribution, let controllers request it the same way against both standard and custom schedulers, and let other components such as the cluster autoscaler understand a workload's requirements too.

That last item matters more than it looks. Today's autoscaler only sees "3 pods are Pending" — it has no idea that "these 3 are part of a gang of 4, and if the 4th can't come up, the other 3 are pointless too." Once the workload is expressed as an API object, that can be read.

Topology comes into it too. Workloads that need a gang usually only perform well when the gang's members sit close together topologically. Existing pod affinity does influence placement, but as the KEP points out, it doesn't treat the gang as the scheduling unit, and it doesn't efficiently try multiple mutually exclusive placement candidates for a set of pods. Affinity is a tool built from the perspective of a single pod.

v1.35 — The First Piece: the Workload API and the Permit Gate

v1.35 (December 17, 2025) shipped the first batch. A Workload resource appeared in scheduling.k8s.io/v1alpha1.

apiVersion: scheduling.k8s.io/v1alpha1
kind: Workload
metadata:
  name: training-job-workload
  namespace: some-ns
spec:
  podGroups:
  - name: workers
    policy:
      gang:
        # The gang is schedulable only if 4 pods can run at once
        minCount: 4

Pods connected to it via workloadRef. The mechanics worked like this.

  1. When a pod is created, the scheduler holds it at PreEnqueue — as long as a Workload object exists, contains the matching pod group, and the number of waiting pods hasn't yet reached minCount.
  2. Once enough pods have gathered, it attempts placement, but instead of binding immediately, it holds them at a Permit gate.
  3. If a valid placement is found for the whole group (at least minCount), the gate opens and they're bound together; if only part of the group gets scheduled within the timeout (5 minutes), all of the group's pods are rejected back to the queue and the reserved resources are released.

Shipping alongside it was opportunistic batching. It needs neither the Workload API nor explicit user opt-in, and in v1.35 it's beta and on by default. It recognizes pods with identical scheduling requirements (container image, resource requests, affinity, and so on) and reuses the feasibility computation done for one pod on the identical pods that follow, speeding up processing. Pods that make up a gang are usually identical to each other, so this pairs well. That said, every field the scheduler uses to find a placement has to match across pods, and using certain features turns batching off for correctness. The docs go out of their way to warn you to check whether your kube-scheduler configuration is implicitly disabling batching.

The v1.35 blog post was explicit that this was a first implementation, and flagged a scheduling phase that processes the whole gang in one cycle, plus workload-level preemption, as next. That's exactly what arrived in v1.36.

v1.36 — Workload as Template, PodGroup as Runtime

v1.36 shipped scheduling.k8s.io/v1alpha2, completely replacing the previous v1alpha1. The API broke just six months in. The reasoning is spelled out in detail in the KEP, and it's convincing.

The original design embedded PodGroup inside the Workload spec. But Workload is a long-lived configuration-intent object, while PodGroup is a transient scheduling unit. Tying a runtime execution unit to a persistent definition object breaks separation of concerns. On top of that, it runs straight into a scalability problem — a large Workload with many PodGroups easily hits etcd's 1.5MB object size limit, and updating the status of a single PodGroup requires a read-modify-write of the huge central Workload object, which creates contention.

So they split it. Workload is now a static template.

apiVersion: scheduling.k8s.io/v1alpha2
kind: Workload
metadata:
  name: training-job-workload
  namespace: some-ns
spec:
  # renamed from v1alpha1's podGroups -> podGroupTemplates
  podGroupTemplates:
  - name: workers
    # renamed from v1alpha1's policy -> schedulingPolicy
    schedulingPolicy:
      gang:
        minCount: 4

And a controller stamps out a runtime PodGroup from this template.

apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata:
  name: training-job-workers-pg
  namespace: some-ns
spec:
  podGroupTemplateRef:
    workload:
      workloadName: training-job-workload
      podGroupTemplateName: workers
  schedulingPolicy:
    gang:
      minCount: 4
status:
  conditions:
  - type: PodGroupScheduled
    status: "True"
    lastTransitionTime: 2026-04-03T00:00:00Z

The pod-side linking field changed too. workloadRef is gone, replaced by schedulingGroup.

apiVersion: v1
kind: Pod
metadata:
  name: worker-0
  namespace: some-ns
spec:
  schedulingGroup:
    podGroupName: training-job-workers-pg

Worth noting: .metadata.ownerReferences still points at the "real" workload object (e.g., a Job). podGroupTemplateRef only says which template this came from. Ownership and provenance are kept separate.

This also simplifies the scheduler's logic. Since a single PodGroup carries everything the scheduler needs, there's no reason to watch or parse the Workload object.

The PodGroup Scheduling Cycle — Evaluating the Group Atomically

This is the real substance of v1.36. Evaluating and reserving resources sequentially, pod by pod, carries deadlock risk, so the scheduler now has a dedicated cycle that evaluates the group as a single unit.

When the scheduler pulls a PodGroup member off the queue, regardless of policy it gathers the group's remaining waiting pods, sorts them deterministically, and runs an atomic cycle.

  1. It takes a single snapshot of cluster state — to prevent race conditions and ensure consistency while evaluating the whole group.
  2. The PodGroup scheduling algorithm searches for a valid node placement for every pod in the group. The filtering and scoring stages reuse the existing pod-based logic as-is.
  3. It applies the result to the whole group atomically.
    • On success, the member pods that can be scheduled move on to the binding stage together, and the remaining pods go back to the queue to wait for resources to free up.
    • On failure, the entire group is treated as unschedulable; no pod is bound, and it retries after a backoff.

There's a quiet but operationally important caveat here. If pods are added to a PodGroup that already has scheduled pods, the cycle evaluates the new pods taking the existing ones into account. And pods already assigned to a node keep running — even if the group fails to meet its requirements in a later cycle, the scheduler does not unassign or evict those pods. In other words, "all or nothing" is atomicity at bind time, not an invariant maintained for the group's whole lifetime.

minCount gating itself still remains. The scheduler still holds pods at PreEnqueue, and only the actual scheduling phase now depends entirely on the new PodGroup cycle. While the algorithm runs, it checks whether the number of schedulable pods satisfies minCount, and if the cluster can't support the minimum, nothing gets bound.

The Algorithm's Honest Limits

This is the most important part of this post. And credit where due — this isn't my observation; it's what the Kubernetes project itself wrote in its own blog post, under a heading titled "Limitations."

The limits of the first version of the PodGroup scheduling cycle are as follows.

  • For homogeneous pod groups — where every pod's scheduling requirements are identical and there are no inter-pod dependencies such as affinity, anti-affinity, or topology spread constraints — the algorithm is expected to find a placement if one exists.
  • For heterogeneous pod groups, there's no guarantee it will find a placement even if one exists — even when the answer looks obvious.
  • There's no guarantee for groups with inter-pod dependencies either.

On top of that, with dependencies inside the group — for instance, where inter-pod affinity makes one pod's schedulability depend on another member — the deterministic processing order can cause it to fail to find a placement regardless of cluster state. The KEP puts the same point this way: the default algorithm may fail to find a valid placement that would have been found had the group's pods been processed in a different order.

Once you read this, the scope of applicability becomes clear. What this feature is properly aimed at right now is PyTorch DDP-style training jobs where every worker is identical. Workloads like LeaderWorkerSet with different leader and worker specs, heterogeneous GPU pools, or ones tangled up with topology spread constraints, are still outside the guarantee. That's not a bug — it's the scope of alpha.

Topology-Aware Scheduling — First Iteration

v1.36 lets you attach topology constraints directly to a PodGroup.

apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata:
  name: topology-aware-workers-pg
spec:
  schedulingPolicy:
    gang:
      minCount: 4
  schedulingConstraints:
    topology:
      - key: topology.kubernetes.io/rack

The scheduler tries node combinations that satisfy the rack constraint, and picks the best placement based on how efficiently the PodGroup uses resources and how many pods fit within that domain. To do this, three placement-based algorithm stages were added to the PodGroup cycle — generating candidate placements (the PlacementGenerate extension point), verifying that the whole group actually fits into each candidate, and scoring the feasible placements to pick the best one (the PlacementScore extension point).

The limits are stated too. Topology-aware scheduling currently does not trigger pod preemption to satisfy a constraint; the plan is to integrate it with workload-aware preemption in a future release. And multiple topology levels, soft constraints (preferences), deep integration with DRA, and robustness when combined with the basic policy all remain future work. So what's usable right now is just one single-level hard constraint.

Workload-Aware Preemption

A new preemption mechanism for when a PodGroup can't be scheduled also arrived. Unlike existing per-pod preemption, it treats the whole PodGroup as a single preemptor unit. Instead of picking victims node by node, it searches across the whole cluster and evicts pods on multiple nodes at once, to make room for the entire group. Without this, gang preemption behaves strangely — nibbling away a little on each node never adds up to enough room for the whole gang.

Two concepts were added to the PodGroup API.

  • priority — overrides the priority of the individual pods that make up the PodGroup.
  • disruptionMode — specifies whether pods within the group can be preempted individually, or must be preempted together, all or nothing.
apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata:
  name: victim-pg
spec:
  priorityClassName: high-priority
  priority: 1000
  disruptionMode: PodGroup

There's a trap here. In v1.36, these two fields are honored only by the workload-aware preemption mechanism. Other disruption paths — including default preemption in the existing per-pod cycle — ignore these fields. Extending them to other disruption sources is a future aspiration. So don't read setting disruptionMode: PodGroup as meaning your gang is wholly protected from every kind of eviction.

Combining With DRA — the 256 Wall Opens Up

DRA graduated to GA in v1.34 (for DRA itself, see Kubernetes Dynamic Resource Allocation and GPU Scheduling), and now PodGroup can serve as the replication unit for a ResourceClaimTemplate.

Previously, referencing a ResourceClaimTemplate created one ResourceClaim per pod. For multiple pods to share the same device, you had to reference a ResourceClaim directly by name, which meant a user had to create and manage that claim by hand. Now, for a ResourceClaimTemplate referenced by a PodGroup's spec.resourceClaims, a single ResourceClaim is created for the whole group, no matter how many pods are in it. If a pod's spec.resourceClaims entry matches the one on its PodGroup, that pod's claim resolves to the ResourceClaim created for the group, and no per-pod claim is created.

The more practical part is this. Previously, kube-scheduler could only list individual pods in a ResourceClaim's status.reservedFor, and that field had a 256-entry limit. Now a single PodGroup reference in status.reservedFor can represent far more than 256 pods. That opens a path to high-cardinality device sharing for large-scale workloads. For training jobs with hundreds to thousands of pods, this isn't a theoretical improvement — it was an actual wall that was blocking things.

Job Controller Integration — the Conditions Are Fairly Strict

If you've read everything so far and are wondering "so do I have to create the Workload and PodGroup myself," v1.36's answer is "not if it's a Job." Turn on the WorkloadWithJob feature gate and the Job controller creates the Workload and runtime PodGroup on its own, sets .spec.schedulingGroup on every pod the Job creates, and sets the owner of the created objects to the Job so they get garbage-collected when the Job is deleted.

But to keep the first iteration predictable, the conditions are narrow. The integration only kicks in when a Job satisfies all of the following.

  • .spec.parallelism is greater than 1
  • .spec.completionMode is Indexed
  • .spec.completions equals .spec.parallelism
  • the pod template doesn't already have schedulingGroup set

The logic is clear — each pod needs a stable identity (Indexed), the gang size needs to be fixed at admission time (parallelism == completions), and no other controller should have already taken over scheduling responsibility. Jobs that don't meet the conditions get scheduled pod-by-pod as before.

The last condition is particularly thoughtful. If you've already set schedulingGroup on the pod template yourself — say, because a higher-level controller is managing the workload — the Job controller leaves the template alone and doesn't create its own Workload/PodGroup either. In other words, it's safe to turn this gate on even in a cluster that already uses an external placement system.

What's missing is clear too. The current constraints limit this integration to static, indexed, fully-parallel Jobs. Support for elastic Jobs and other built-in controllers is tracked in KEP-5547.

How to Turn It On

All of it is alpha in v1.36. Prerequisites first.

  • Workload/PodGroup API — turn on the GenericWorkload gate on both kube-apiserver and kube-scheduler, and enable the scheduling.k8s.io/v1alpha2 API group.

Then, per feature you want.

  • Gang scheduling — GangScheduling on kube-scheduler
  • Topology-aware scheduling — TopologyAwareWorkloadScheduling on kube-scheduler
  • Workload-aware preemption — WorkloadAwarePreemption on kube-scheduler (GangScheduling must also be on)
  • DRA ResourceClaim support — DRAWorkloadResourceClaims on kube-apiserver, kube-controller-manager, kube-scheduler, and kubelet, all of them
  • Job controller integration — WorkloadWithJob on kube-apiserver and kube-controller-manager

Notice that the gate is different per component. Just from this list you can conclude you won't be able to try this on managed clusters (EKS, GKE, AKS) any time soon — you need the ability to turn alpha gates on in the control plane, so a test cluster set up with kubeadm or kind is the realistic place to experiment.

So Should You Rip Out Volcano and Kueue

No. And this isn't my opinion — it's written into KEP-4671's Non-Goals: bringing fairness or multi-workload queuing into kube-scheduler is not a goal, and Kueue and Volcano.sh will continue to provide it.

This draws the boundary precisely. What the core is taking on is the mechanism — "place this bundle of pods together, atomically, with topology in mind." The policy — "how much quota does which team get, who comes off the queue first, when do you reclaim borrowed resources" — is still Kueue and Volcano's territory. And in practice, the latter is why most organizations adopted a gang scheduler in the first place. Fewer teams than you'd think needed only gang scheduling.

The same Non-Goals list also includes: not taking pod-creation responsibility away from controllers; integrating cluster autoscaling with gang scheduling is not (yet) in scope; workload-level preemption is not a goal (of this KEP); and resource contention between different schedulers — including possible deadlock — is not addressed. That last one is telling — problems that arise from running kube-scheduler and Volcano together in one cluster are still your problem.

Why You Shouldn't Use This Right Now

This section is probably worth more to practitioners than the technology itself.

It's all alpha. Only opportunistic batching is beta (on by default); the Workload API, gang scheduling, topology, preemption, DRA integration, and Job integration are all alpha.

The API has already broken once, and is set to break again. v1.35's v1alpha1 was completely replaced by v1alpha2 in v1.36. podGroups became podGroupTemplates, policy became schedulingPolicy, a pod's workloadRef became schedulingGroup, and runtime state moved out into a separate object. In six months.

And the v1.37 plan written into the KEP goes one step further. It plans to promote the API to v1beta1 while simultaneously creating a new v1alpha3 to replace v1alpha2, for backward-incompatible DisruptionMode-related changes to planned alpha features. The KEP spells out what's required to upgrade — v1alpha2 resources must be deleted entirely before upgrading (they're unsupported in v1.37), and converting back from v1alpha3 to v1alpha2 is not supported on downgrade. On top of that, since the GangScheduling gate is being merged into GenericWorkload, you'll need to drop GangScheduling from your configuration when upgrading, and turn it back on if you downgrade to v1.36. (v1.37 is planned for August 26, 2026, and everything written here is a plan as stated in the KEP — it can change before release.)

The algorithm's guarantees are narrow. As covered above, it may fail to find a placement outside of homogeneous groups.

There are risks the KEP itself calls out — the contention window at the binding stage is longer than standard scheduling, the extra objects mean more API calls and more etcd objects, and there are consistency issues between the two objects, Workload and PodGroup. It's mitigated with GC and admission controllers, but it's not free.

So Why Pay Attention to This Anyway

Because the direction is clear. Kubernetes has stood, until now, on the premise that "the pod is the minimum unit of scheduling," and that premise broke down for AI workloads. The response over the past several years has been to work around it entirely from outside. Now the core is in the process of recognizing the workload as a first-class citizen.

The v1.37 roadmap tells you where this is going — promoting the Workload/PodGroup API to beta alongside introducing mutable minCount for elastic jobs, a multi-level hierarchy for complex AI workloads like JobSet- or LWS-based distributed inference, promoting topology-aware scheduling and workload-aware preemption to beta, and a unified controller-integration API for real workload controllers. The project adds that this priority and implementation order can change.

What you should do right now is not migrate. Spin up v1.36 on a test cluster with kind, turn on the gates, and check whether the shape of your training jobs fits the Job controller integration conditions (Indexed, parallelism == completions), and whether your pod groups qualify as "homogeneous." If both hold, you're the exact target of this feature, and once beta arrives you could shed your Volcano dependency fairly early. If either one is off — and if you need quota and queuing — Kueue and Volcano remain the answer for a while yet. That's not a failure; it's exactly how the KEP was designed.

Closing

To sum up: gang scheduling has been implemented at least four times outside kube-scheduler, and Kubernetes started bringing it into the core with the Workload API in v1.35, then fixed the structure in v1.36. Workload is a static template, PodGroup is a runtime object, and there's a PodGroup scheduling cycle that atomically evaluates the whole group over a snapshot. On top of that came topology constraints, workload-level preemption, PodGroup-level ResourceClaim sharing (opening up the 256 wall), and a first iteration of automatic Job controller integration.

In exchange, all of it is alpha, the API broke once in six months and is planned to break again, the algorithm's guarantees are narrow, limited to homogeneous pod groups, and fairness and queuing were never the goal to begin with. So this isn't a "Volcano replacement" — it's "the standard floor Volcano will stand on." Watch that floor get built, but don't put production on top of it yet.

References

Discuss on Twitter • View on GitHub