Skip to content
Published on

CXL Memory Tiering: How Far Has the Kernel Come — What DAMON Merged, and What the Numbers Didn't Measure

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — The Kernel's Part in the CXL Story

We already covered the CXL hardware story once, in Reviving Retired DDR4 — Meta's CXL Bridge Chip, Vistara. It was a hyperscale end-of-life deployment report about reviving retired DDR4 as CXL and using it as a second memory tier.

But a chip being plugged in and that chip actually paying off are two different things. To the operating system, CXL memory just looks like a NUMA node with no CPUs, and deciding which pages are hot and which are cold — then actually moving them — is entirely the kernel's job. Get that judgment wrong and CXL is just slow RAM.

So this post looks at the code running on top of the hardware, not the hardware itself. We check directly against kernel source what's actually landed in Linux mainline as of July 2026, and trace the numbers cited in this space back to their original sources to see exactly what each one measured and didn't. The conclusion up front: what's landed are parts, the policy is still on you, and the bigger a number is, the less methodology tends to come attached to it.

Two Ways the Kernel Moves Pages

First, the lay of the land. There are broadly two mainline mechanisms for moving pages in tiered memory.

First, NUMA balancing's tiering mode (NUMAB-2). It finds hot pages via access-checking page faults (prot_none) and performs promotion in the process's own context — meaning the promotion work happens on the same thread as the application's actual work. Demotion is reclaim-based, handled by kswapd or direct reclaim.

Second, DAMON and DAMOS on top of it. DAMON spins up a kernel thread that samples memory access every 5ms and hands the results to userspace every 100ms. DAMON's creator, SeongJae Park, presented at LSFMM+BPF 2026 that overhead is under 0.1% on a typical system. It first merged in 5.15. DAMOS is the layer that attaches actions to those observations, and the actions used for tiering are migrate_hot (promotion) and migrate_cold (demotion). The key difference is that DAMOS runs on an asynchronous worker thread.

This difference comes back as a number later.

What's Actually Landed in Mainline as of July 2026

Rather than trusting slides or news coverage, I checked kernel source directly. Below is the tag each feature first appears in, confirmed against include/linux/damon.h and the documentation.

6.11    DAMOS_MIGRATE_HOT / DAMOS_MIGRATE_COLD
        (Absent in v6.10 -> appears in v6.11. SK hynix: Hyeongtak Ji, Honggyu Kim, 2024-06-14)
        For HMSDK's CXL capacity expansion. Thresholds are static values a human hand-sets.

6.16    node_mem_used_bp / node_mem_free_bp quota goals
        (Self-tuning tiering, aka TPP-DAMON. SeongJae Park)

6.17    damos->migrate_dests -- multiple destination nodes + weights
        (Micron: Bijan Tabatabai. Virtual address space only)

6.19    node_memcg_used_bp / node_memcg_free_bp
        (cgroup-scoped tiering. Meta collaboration)

7.2-rc1 node_eligible_mem_bp
        (Micron: Ravi Jonnalagadda. Not yet in any officially released kernel)

Two things need pointing out here.

First, there is no tiering kernel module that just works. The string "tier" appears zero times in mm/damon/Kconfig at v7.2-rc3. The only buildable DAMON modules are DAMON_RECLAIM, DAMON_LRU_SORT, and DAMON_STAT, and there is no tiering.c under mm/damon/. In other words, what mainline provides is parts — quota-goal metrics, migration actions, destination weighting — and assembling those into a policy is left to userspace.

This isn't an accident — it's the plan. In the self-tuning tiering RFC cover letter, the author himself wrote that he'd send the quota-goal metric piece first and the tiering module piece "later." The reason given was that the module side only assumes two tiers and requires the user to hand-enter node address ranges — far from a finished state. That "later" still hasn't arrived as of July 2026.

So to actually run this today, you have to hand-write a damo command, as shown in the presentation slides.

damo start \
  --numa_node 0 --monitoring_intervals_goal 4% 3 5ms 10s \
      --damos_action migrate_cold 1 --damos_access_rate 0% 0% \
      --damos_apply_interval 1s \
      --damos_quota_interval 1s --damos_quota_space 200MB \
      --damos_quota_goal node_mem_free_bp 0.5% 0 \
      --damos_filter reject young \
  --numa_node 1 --monitoring_intervals_goal 4% 3 5ms 10s \
      --damos_action migrate_hot 0 --damos_access_rate 5% max \
      --damos_apply_interval 1s \
      --damos_quota_interval 1s --damos_quota_space 200MB \
      --damos_quota_goal node_mem_used_bp 99.7% 0 \
      --damos_filter allow young \
      --damos_nr_quota_goals 1 1 --damos_nr_filters 1 1 \
  --nr_targets 1 1 --nr_schemes 1 1 --nr_ctxs 1 1

This configuration demotes cold pages on node 0 (DRAM) targeting 0.5% free space, and promotes hot pages on node 1 (CXL) targeting 99.7% DRAM utilization. The design intent is to set 0.5% and 99.7% to overlap slightly so promotion and demotion don't stall each other, and the cover letter proposes these two values as the default recommendation. It's not a bad interface, but it's hard to call this "just works."

Second, CONFIG_DAMON isn't on by default. In May 2025, SeongJae Park submitted a patch to enable it by default, on the grounds that major distributions were already building DAMON in and that it only grows the kernel package size by about 0.1%, and it landed in the 6.16-rc1 merge window. Then, on June 10, 2025, Linus Torvalds reverted it. The body of commit aef17cb3d3c4 is a single line — "No, we don't make random features default to being on."

Checking mm/damon/Kconfig at v7.2-rc3, config DAMON still has no default y. The revert has held.

The Number 4.43% — And What It Didn't Measure

The number attached to self-tuning tiering (6.16) is 4.42%. Reading the original RFC cover letter (SeongJae Park, 2025-03-19), all the conditions are spelled out.

Test environment. A machine exposing a 250 GiB DRAM tier and a 50 GiB CXL tier as NUMA nodes 0 and 1 respectively. The kernel is v6.13 with all the DAMON patches from the 2025-03-15 mm tree plus this series layered on top, and the mempolicy() syscall was patched to be ignored so that the application's traditional NUMA optimizations don't interfere. The workload is a modified variant of DCPerf's Taobench, with a memory size of 340 GiB, a 2,500-second warmup, a 1,440-second measurement window, and RSS growing to 270 GiB during warmup. Each configuration was run 5 times.

Results table (verbatim from the cover letter).

Config         Score   Stdev   (%)     Normalized
Baseline       1.6165  0.0319  1.9764  1.0000
Numab_tiering  1.4976  0.0452  3.0209  0.9264
DAMON_tiering  1.6881  0.0249  1.4767  1.0443

Doing the math directly: 1.6881 divided by 1.6165 is 1.0443, so DAMON tiering is a 4.43% improvement; 1.4976 divided by 1.6165 is 0.9264, so NUMAB-2 is a 7.36% regression. (The cover letter's summary text states 4.42% and 7.34%, but the values that actually come out of its own table are 4.43% and 7.36%. "~4.42%" carries through unchanged even into the LSFMM slides a year later. It's a small thing, but it also means the cited number hasn't passed through anyone who recomputed the original table.)

So far this sounds like good news. Now let's look at the caveats the same cover letter wrote down about itself.

The unmodified workload produced no difference at all. In the author's own words, Taobench's access pattern is so static that only about 13% of RSS was frequently accessed from start to finish, and as a result promotion — regardless of design or implementation — produced no meaningful performance difference. So the author patched the kernel to inject a disturbance that demotes up to 10 GiB of hot pages and promotes up to 10 GiB of cold pages once a minute. This was meant to simulate a situation where the access pattern periodically shifts, which is why the workload's name has "modified" attached to it.

In other words, 4.43% is a value produced on a workload artificially perturbed to favor tiering. On the original, unperturbed workload, no tiering implementation made a difference.

Demotion never actually happened. As the cover letter states directly, the simulated disturbance ran from the start of the workload, so DRAM-tier usage stayed below the watermark the whole time, and no actual demotion occurred in any test run. This table does not compare LRU-based demotion against DAMON-based demotion. It only captures the difference between NUMAB-2 promotion and DAMON promotion.

The author's own statement of limitations. The cover letter has a dedicated "Evaluation Limitations" section stating that, depending on one's perspective, this modified Taobench may skew the picture too much, and that it would be better to evaluate with a more realistic workload or a fairer microbenchmark. It's hard to be more honest about your own numbers than that.

And the standard deviation. Standard deviation relative to score ranges from 1.48% to 3.02%. 4.43% sits above that range, but not overwhelmingly so.

What's actually interesting is the other line. On the same test, NUMAB-2 — the tiering feature already in the kernel — actually made performance 7.36% worse. The author speculates this is because NUMAB-2's synchronous promotion blocked Taobench's actual work from progressing. This is where "promotion in the process's own context," noted earlier, exacts its cost.

Summed up in one line: on a workload artificially given a reason to need tiering, the new approach gained 4.43% and the existing approach lost 7.36%.

The Number 25% — And Why It Isn't Actually Tiering

25% is the number attached to 6.17's multi-destination migration. The original cover letter (Bijan Tabatabai, Micron, 2025-07-02) spells out its conditions too, and this one is a completely different animal from the story above.

Test environment. A 128-core, 256-thread AMD CPU, local DDR bandwidth of 72 GB/s, CXL bandwidth of 26 GB/s. The workload is a bandwidth-intensive embedding reduction application that reuses the same allocation across iterations and reports per-iteration time.

Procedure. Since system bandwidth utilization starts low, the interleave weight is initially set to 1:0 with everything allocated to local memory. Once the workload begins, local bandwidth saturates, and at that point the weight is changed to 3:1, having DAMON move data. Iteration time dropped from around 7,600ms to around 5,630ms, and the cover letter records this as roughly a 25% speedup.

Here's the important pivot. This is not "hot goes on the fast tier, cold goes on the slow tier." It's the opposite: deliberately placing hot data on the slower CXL tier to sum the bandwidth of both tiers. This is bandwidth interleaving, not capacity tiering.

I checked directly whether the numbers add up. 72 plus 26 is 98, and 98 divided by 72 is 1.361, so if this is a bandwidth bottleneck, time should scale by 0.735x. 7,620ms times 0.735 is about 5,600ms, and the measured value is about 5,634ms — within 1% error. The 3:1 weight also tracks the bandwidth ratio of 72:26 (roughly 2.8:1). In other words, this 25% is fully explained by bandwidth aggregation, no more and no less. Without a workload that is purely bandwidth-bound, this gain doesn't materialize.

And the part that most demands honesty. The one who changed the weight from 1:0 to 3:1 was a human — at exactly the moment bandwidth saturated. The cover letter doesn't hide this: it states plainly that this patch set only adds a mechanism for changing how data gets interleaved, and leaves the policy of what the weight should be to userspace. It also notes that a userspace program that monitors system bandwidth utilization or access latency to tune the weight is being built, but isn't ready to be published yet.

The piece that closes that loop is node_eligible_mem_bp, from the timeline above. It's a metric that measures, in basis points, what fraction of the memory matching a scheme's access pattern sits on a particular node, and setting it as a goal lets the quota auto-tuner maintain a distribution like "60% of hot data on DRAM, 40% on CXL" on its own. Commit 9138e27a3bc3 (Ravi Jonnalagadda, Micron) is the implementation.

Three things worth noting. First, this commit has an author date of 2026-04-28 and a commit date of 2026-05-29. Since 7.1-rc1 shipped on 2026-04-26, this missed 7.1 — and indeed DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP isn't in v7.0 or v7.1; it first appears in v7.2-rc1. Since v7.2 isn't an official release yet (the latest tag as of writing is v7.2-rc3), there is no officially released kernel with this feature as of today. (The LWN article states this feature merged in 7.1-rc1, but the presenter's own slides say "merged into mm-new after 7.1-rc1," and the source backs the slides. What landed in 7.1-rc1 was the quota-goal tuner algorithm.)

Second, this commit has zero performance numbers. There's a "Testing Results" section, but all it says is that functional verification was done on a 2-node heterogeneous memory system, that the TEMPORAL tuner converges to the target distribution quickly while the CONSIST tuner converges more slowly, and that time-to-target varies with workload intensity. It's a report on convergence behavior, not a speedup report.

Third, it requires CONFIG_DAMON_PADDR. It only operates in physical address space. By contrast, 6.17's weighted interleaving is virtual-address-space only — because a folio's destination node is determined by its mapped offset within the VMA, and figuring that out from a physical-address scheme requires an expensive rmap walk. In other words, the mechanism and its auto-tuning target live in two different address spaces.

To sum up, the 25% is real, but it's a value obtained by a human flipping the switch by hand, and the thing that would flip that switch automatically only arrived in an -rc a year later, and still has no performance numbers attached.

The Number 94% — A Number With No Conditions

The biggest number at LSFMM+BPF 2026 was 94%. TPP-DAMON was too slow single-threaded, so it moved to a multi-threaded model, and this is claimed to deliver a 94% improvement on the llama.cpp benchmark. This is SK hynix's work.

Pulling the original slides the presenter published themselves and checking them, the slide in question looks like this — title "Multi-thread Tiering (SK hynix)," three bullets: per-subregion DAMON worker threads, accelerated hot/cold page discovery, accelerated migration. And a final bullet: "Achieves ~94% llama.cpp speedup."

That's it. No hardware, no model, no baseline, no run count — nothing. I could not find a public cover letter that states this number's conditions. It doesn't appear in the DAMON project's own news list either. The LWN article reports the same 94%, but since that's just transcribing the same talk, it isn't independent confirmation.

I'm not saying this is "fake." But it's worth contrasting. 4.43% comes with hardware specs, workload parameters, 5 repetitions, a standard-deviation column, and a limitations section the author wrote himself. 94% comes with one bullet point. In this space, the size of a number and the amount of methodology attached to it are, generally, inversely proportional. And reading further into the slides, it states that TPP-DAMON's positive test results were "privately shared," that there are no plans for further active development, and that it has entered support mode.

In other words, kernel tiering's flagship auto-tuning module moved into maintenance mode without any public performance evidence, and development effort has shifted to the next thing, NUMA-TPP-DAMON. In the slide's own words, that's "early-stage, brainstorming-level planning."

The Real Problem Isn't the Algorithm — It's the Parameters

Why this community is so fixated on auto-tuning is well explained by a paper from University of Wisconsin researchers. The abstract of From Good to Great: Improving Memory Tiering Performance Through Parameter Tuning (Kanellis et al., arXiv:2504.18714, 2025-04-25) states: existing tiering solutions decide placement and migration using heuristics and pre-set thresholds, and as a result fail to adapt to the workload and hardware, operating suboptimally.

The researchers used Bayesian optimization to search the parameters of existing tiering systems (HeMem and SK hynix's HMSDK), and report that getting the values right yields 2x the performance of the same system's default settings, and 1.56x compared to the then-state-of-the-art tiering system.

Put this sentence next to the 4.43% above and the picture snaps into focus. Changing the algorithm gets you single-digit percentages; getting the same algorithm's parameters right gets you 2x. What determines the payoff in tiering is not the mechanism but the tuning. And the default settings are throwing away half of it.

At the same time, this is also bad news. The way to get that 2x is Bayesian optimization — running an expensive search per workload. Not many organizations can afford to run a parameter sweep per workload in production. The kernel's self-tuning effort is an attempt to replace exactly that search with an online feedback loop, and the trajectory runs from 6.16's utilization/free-space goals to 7.2-rc1's distribution goals. It just hasn't arrived yet.

What's Still Unresolved

The open problems raised in the talk say the most accurate thing about how mature this technology is right now.

DAMON doesn't know who did the accessing. As the self-tuning tiering cover letter states explicitly, this design assumes only a single CPU NUMA node. The author writes directly that because DAMON can't identify the source of an access, applying this to a system with multiple CPU NUMA nodes gets complicated. Given how common 2-socket servers are, this is not a small caveat.

DAMON and NUMA balancing fight over the same bit. Because both use the page-idle bit to check access, each ends up "measuring" faults the other one triggered. The candidate solutions raised in the talk were build-time exclusivity (put only one of the two in the kernel), runtime isolation (only one active at a time), partial isolation, and just letting them interfere. Park's proposal was that once the necessary cleanup work is done, the first implementation should either use build-time exclusivity or just ignore the problem. Going the exclusivity route puts distributions in the awkward position of having to pick one or the other. Davidlohr Bueso's concern during the session, about multiple layers fighting each other over memory placement decisions, is in the same vein.

The next generation is still just an idea. NUMA-TPP-DAMON — the concept of generalizing tiering as a special case of NUMA placement, running separate promotion paths per accessor (CPU, GPU, etc.) — is at the brainstorming stage, with the hard problem of per-accessor monitoring still ahead of it.

And the documentation. node_eligible_mem_bp is in include/linux/damon.h at v7.2-rc3, but it's not yet in the DAMON design document's list of quota-goal metrics. Lagging docs are common enough, but it's still a sign of just how freshly this feature arrived.

So, Should You Turn This On Right Now

When it's worth turning on

  • Your servers are bound not by cores but by memory capacity, and a substantial share of your workload's memory is genuinely cold. (This was the state of the 43.7% of fleet that Meta reported in the Vistara paper.)
  • Your workload is bandwidth-bound and there's something to gain from aggregating CXL bandwidth. In this case, look at weighted interleaving rather than capacity tiering — redo the arithmetic from the 25% case above with your own hardware's numbers and you get the ceiling immediately.
  • You have the capacity to tune parameters against real workloads. Per the paper above, this is where the 2x lives.
  • You have a single CPU NUMA node.

When it's not yet time

  • Your access pattern is static. That was true of the original, unmodified Taobench in the 4.43% experiment, and the result was no meaningful difference under any implementation. Tiering pays off when hot and cold shift over time. If they don't shift, there's no reason to move anything.
  • You're expecting something that "just works when you flip it on." There is no tiering module in mainline — the damo command line and a handful of hand-picked constants are yours to own.
  • You're on 2 sockets or more. The design doesn't yet assume that case.
  • You're running a latency-sensitive service with a tight margin. If demotion gets it wrong, the next access to that page eats the full CXL latency.
  • You think just turning on NUMAB-2 will do it. In the table above, that was a 7.36% loss — at least on that workload.

Whatever you do, there's one order of operations — measure your workload's access pattern first. Fortunately that's easy now. DAMON_STAT landed in 6.17 and is build-enabled in Debian's kernel. Turning on tiering without knowing what percentage of your pages have gone unread for how many hours is, exactly as the experiment above shows, paying migration overhead for nothing.

Closing

To sum up: Linux mainline has steadily received CXL tiering parts from 6.11 in 2024 through 7.2-rc1 in 2026, and most of those parts are code written directly by companies that sell memory (SK hynix, Micron) and a company with money tied up in memory (Meta). The parts themselves are well designed.

But to be honest, mainline today has no tiering module you can just switch on, CONFIG_DAMON is off by default, the last piece of auto-tuning still hasn't reached an official release, and the flagship module moved to support mode without public performance evidence. Of the cited numbers, the best-documented one is 4.43% — and even that only appeared after artificially perturbing the workload — while the biggest number, 94%, has no conditions attached at all.

It's a common line that CXL has been overhyped. The picture from the kernel side is a little different — here, progress is slow but real, and above all, the people doing the work are remarkably honest about their own numbers. These are people who write "this workload may skew the picture too much" in a cover letter, and "results were privately shared" on a slide. The exaggeration mostly gets added not by the people writing this code, but by the layer that summarizes and carries it onward.

So if you're evaluating CXL tiering, read the cover letters instead of the vendor materials. They already come with the number, and a list of everything that number didn't measure.

References

Discuss on Twitter • View on GitHub