Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — The Node Boundary Is No Longer the Performance Boundary
Placement Group Recap — All Four Strategies Are Node-Scoped
The Problem — On an NVL Rack, STRICT_PACK Is Impossible and PACK Doesn't Care
Ray 2.56's Answer — A Domain-Level Layer on Top of the Node Strategies
Failure Semantics — The Part the Docs Write Down Honestly
Alpha Means Alpha — The List of Limits
Where This Feature Sits — Ray Scheduling Converging on Labels
When to Use It, When to Ignore It
Closing
References

Introduction — The Node Boundary Is No Longer the Performance Boundary

Distributed schedulers carry one old, implicit assumption: "the fast interconnect ends at the node." In the era of 8-GPU HGX servers that was mostly true — NVLink tied GPUs together inside a node, and between nodes you dropped to InfiniBand or Ethernet. So as long as your scheduler knew how to "cram things onto the same node," fast communication came along for free.

NVIDIA's rack-scale systems broke that assumption. Per NVIDIA's spec sheet (vendor's own figures), GB300 NVL72 binds 72 Blackwell Ultra GPUs and 36 Grace CPUs into a single NVLink domain inside one rack, and quotes NVLink bandwidth of 130 TB/s. Which means the boundary of "the fast interconnect" is now the rack, not the node. And it is a boundary a node-only scheduler cannot see.

Ray 2.56.0, released on June 29, 2026, carried Ray Core's first answer to this shift as a release highlight — GPU-domain-aware placement groups, officially named label locality scheduling. This post lays out what the feature solves, exactly how it works, and why it wears an alpha tag, based on what can be verified in the release notes, the docs, and the source.

Placement Group Recap — All Four Strategies Are Node-Scoped

Ray's placement group is a gang-scheduling primitive that atomically reserves a set of resource bundles. It expresses demands like "give me all 18 bundles of 4 GPUs and 2 CPUs, or give me none." It is the basic tool for workloads where a partial set of workers is meaningless — multi-node training, distributed inference — and it is what runs underneath model serving with Ray Serve and distributed training stacks alike.

There are four placement strategies — PACK (prefer packing onto one node), STRICT_PACK (force one node), SPREAD (prefer spreading across nodes), STRICT_SPREAD (force spreading across nodes). The word to notice here is not the strategy name, it is "node." In all four, the unit of placement is the node. Topology larger than a node — a rack, an NVLink domain — does not exist in this vocabulary.

The Problem — On an NVL Rack, STRICT_PACK Is Impossible and PACK Doesn't Care

Here is the example from the Ray 2.56.0 docs, carried over as-is. In a cluster of 2 racks, 18 nodes per rack, 4 GPUs and 2 CPUs per node, you want to place 18 bundles inside a single rack.

from ray.util.placement_group import placement_group

# Attempt 1: STRICT_PACK -> tries to put all 18 bundles on 'a single node'.
# A node only has 4 GPUs, so this placement group stays pending forever.
pg = placement_group([{"GPU": 4, "CPU": 2}] * 18, strategy="STRICT_PACK")

# Attempt 2: PACK -> spreads across nodes, but has no notion of a 'rack',
# so the scheduler has no complaint even if bundles straddle two racks.
pg = placement_group([{"GPU": 4, "CPU": 2}] * 18, strategy="PACK")

And what happens when you straddle two racks is this — communication across that seam falls outside the NVLink domain, onto the inter-rack network. The scheduler half-nullifies the hardware you bought precisely to stay inside the domain.

There were workarounds, of course. Attach a custom label to each node and pin things with a static label selector — for instance, requiring the rack-1 label on every bundle. The docs spell out two limits of this workaround. First, the selector is static, so it cannot respond to failure — if every node in rack-1 dies, the placement group cannot automatically move to another rack. Second, what you actually want is "any one rack," yet you have to name a specific rack by hand, and once you have many racks that management becomes a burden of its own.

Ray 2.56's Answer — A Domain-Level Layer on Top of the Node Strategies

Label locality scheduling does not replace the existing node-level strategies; it stacks one domain-level scheduling layer on top of them. Per the docs, it works in three steps.

Group candidate nodes by the value of the ray.io/gpu-domain label — same value, same domain.
Pick one domain that can accommodate all the bundles.
Inside that domain, apply the node-level strategy the user specified, unchanged.

At the domain level only STRICT_PACK is supported — either the whole placement group fits inside a single domain, or it does not get scheduled. The trigger condition is that every bundle in the placement group requests the GB200 or GB300 accelerator type in bundle_label_selector.

import ray
from ray.util.placement_group import placement_group

bundles = [{"GPU": 4, "CPU": 2}] * 18
label_selector = [{"ray.io/accelerator-type": "GB300"}] * 18

pg = placement_group(
    bundles=bundles,
    bundle_label_selector=label_selector,
)

ray.get(pg.ready())  # all 18 bundles land inside the same ray.io/gpu-domain

Just how narrow the trigger condition is can be confirmed directly in the source at the 2.56.0 tag — the trigger is literally a hardcoded set of the two values GB200 and GB300, and if even one bundle carries this selector while the rest do not carry the same value, it raises ValueError. Write any other accelerator type, H100 or B200, and it works as a plain static label selector with no domain layer at all.

There is one asymmetry operators absolutely must know here. The two labels come from different places.

ray.io/accelerator-type is attached by Ray automatically — per the labels docs, the default node labels are ray.io/node-id and ray.io/accelerator-type, and the latter is derived from the GPU name NVML reports (it is not attached on CPU nodes).
ray.io/gpu-domain is not attached by Ray automatically. The docs say so explicitly — which rack a node belongs to is something the operator has to fill in.

# You must label the NVLink domain (rack) a node belongs to yourself
ray start --labels="ray.io/gpu-domain=rack-1"

In other words, half of "domain awareness" is the provisioning pipeline's job. Without automation that grabs the rack identifier at node boot and writes it into this label, this feature cannot even get started. For what it is worth, the autoscaler side is ready — changes that pass domain constraints and label selectors through the autoscaler protocol (#61614, #62487) landed in the same release, so scale-up decisions receive domain information too.

Failure Semantics — The Part the Docs Write Down Honestly

Half the reason this feature exists is failure handling (a static selector was the end of the story once a rack died). The semantics the docs specify fork in two.

Partial failure — when some nodes inside the domain die, Ray reschedules the lost bundles onto live nodes in the same domain. Actors and tasks on the surviving bundles keep running. However, if there are not enough resources inside that same domain, those bundles stay queued in the infeasible state, waiting until resources free up in the same domain. If you want to move to a different domain, there is exactly one way — delete it with remove_placement_group and create a new one. And that call force-kills every actor and task that was using that placement group's bundles, and does not restart them. Recovery automation is on you.

Full failure — when every node in the domain dies, Ray clears the domain assignment and reschedules the whole placement group into a different domain. This is precisely what static label selectors could not do.

You can check where it landed. The dashboard's placement group table gained a Label Domain column (#62533), and it shows up in the state API too.

ray list placement-groups --detail

# partial output (condensed from the 2.56.0 docs example)
- placement_group_id: 237f47c3235ac1a96ad423c3f74501000000
  name: gpu-domain-pg
  state: CREATED
  label_domain_key: ray.io/gpu-domain
  label_domain_assignments:
    ray.io/gpu-domain: rack-2

For placement groups that do not use label locality, label_domain_key is an empty string.

Alpha Means Alpha — The List of Limits

This is a feature the docs themselves warn is alpha, and the constraints are sharp. Only the confirmed ones are listed here.

The trigger is GB200/GB300-only. Two hardcoded values in the source, and the docs likewise state that only these two accelerator types are supported. PR #61442, the starting point of the feature (merged April 3, 2026), does not hide its purpose either — it is written for NVML domains (abstracted as GPU domains), and says it targets GB300 racks first.
STRICT_PACK is the only domain-level strategy. The docs mention plans to add more domain-level strategies and to support arbitrary label domains, but for now those are only plans.
The ray.io/gpu-domain label is manual. You have to build the automation that transcribes rack topology into labels yourself, and if the label is wrong, the scheduler believes the wrong topology.
The same-domain-wait semantics on partial failure cut both ways. If in-domain recovery is your goal it is the right default, but for a workload where "come back up quickly, in any domain" matters more, waiting indefinitely in the infeasible queue is a loss — and the only escape hatch is a recreate that costs you every actor.
This is freshly minted code. Within the same release cycle, a bugfix for label-domain placement groups getting stuck in the infeasible queue (#62483, merged April 25) already landed. The alpha tag is not decoration.
There are no published performance numbers. Not in the release notes, not in the docs, not in the PRs — no benchmark for this feature. It is a placement-semantics feature rather than a speed optimization in the first place, and the size of the gain is set by "the gap between in-domain NVLink bandwidth and inter-rack network bandwidth × how much inter-node communication your workload does." That number you have to measure on your own cluster.

Where This Feature Sits — Ray Scheduling Converging on Labels

Label locality is not an isolated feature; it is the latest step in Ray scheduling reorganizing itself on top of label selectors. The verifiable timeline goes like this — label scheduling itself arrived as a beta feature in 2.49 (August 2025) (the docs introduce it as "for controlling scheduling from KubeRay," but general paths like ray start --labels are documented alongside), from 2.50 a dynamic cluster with autoscaler v2 enabled learned to grow nodes in a designated worker group to satisfy label requirements, and 2.53 added label_selector to Ray Train's ScalingConfig. Then 2.56 shipped this feature, layering the notion of a domain on top of labels, alongside the work of replacing the NodeAffinitySchedulingStrategy(soft=False) that Ray's internal libraries used with the ray.io/node-id label selector (#54940). The direction is to absorb special-purpose scheduling strategies into the label vocabulary one by one. The TPU side has been solving the same problem through a separate path called SlicePlacementGroup, and this release attached explicit bundle_label_selector support there too (#63171).

The direction itself is not Ray's alone. Kubernetes is pulling gang scheduling and topology awareness into the core scheduler as well, with the Workload and PodGroup APIs in v1.36. Rack-scale hardware is handing schedulers the same homework, and everyone has started writing the same answer — "a unit larger than a node."

When to Use It, When to Ignore It

The conditions for having a reason to use it are clear — you have GB200/GB300 NVL racks, your cluster has more than one domain, and you run a workload where a single placement group spans several nodes (multi-node training, tensor/expert-parallel inference that crosses nodes). In that case the feature gives you two things. Prevention of domain straddling, and freedom from naming a domain by hand (including the automatic move on full failure).

Conversely, you can ignore it in the following cases.

Your workload fits on a single node — STRICT_PACK is already the answer.
You have no multi-node NVLink domain — on a cluster of 8-GPU servers the node boundary is the domain boundary, so the existing strategies are enough, and if it is not a GB200/GB300 the trigger will not fire anyway.
The whole cluster is one domain — if there is only one domain to choose, the domain-selection layer has nothing to do.
On failure, "restart fast anywhere" matters more than "wait in the same domain" — this feature's partial-failure semantics may actually trip you up. Either come with recreate automation, or keep your distance for now.

Closing

To sum up: Ray's placement groups have until now had only a node-level vocabulary, and NVL72-class rack-scale systems created a boundary that vocabulary cannot express — the multi-node NVLink domain. Ray 2.56.0's label locality scheduling defines a set of nodes sharing the same ray.io/gpu-domain label as a domain, and stacks a layer on top of the node strategies that STRICT_PACKs the whole placement group into one domain. Unlike a static label selector, it moves to another domain automatically when a whole domain fails.

At the same time, this feature is faithful to the definition of alpha. A hardcoded GB200/GB300 trigger, a single domain-level STRICT_PACK strategy, a manual gpu-domain label, a destructive path for forcing a domain move, and no published performance numbers. If you actually operate NVL racks, it is worth starting your evaluation now; if not — the day this feature comes to mean something to you will be announced first by a hardware purchase order.