Split View: containerd 2.3의 EROFS 네이티브 레이어 — 언팩을 없애는 대신 무엇을 냈나

EROFS Native Layers in containerd 2.3 — What You Get in Exchange for Removing Unpack

• Introduction — "Slow Image Pulls" Is Actually Two Problems
• What Actually Merged in containerd 2.3
• What the Differ Code Tells You — Where Unpack Actually Disappears
• Checking It Directly Against the Registry
• os.features — Serving Two Worlds from One Tag
• +zstd — Compression Comes Back, But Only Half the Unpack Avoidance
• Lazy Loading Doesn't Exist Yet
• dm-verity — Quietly the More Interesting Part
• What You Pay For
• Where the Standard Stands
• So, Should You Turn This On Now?
• Closing
• References

Introduction — "Slow Image Pulls" Is Actually Two Problems

When people say container image pulls are slow, two different problems are actually bundled into one word.

One is download time — the network cost of pulling several GB from a registry. The other is unpack time — the cost of streaming tar+gzip decompression while creating tens of thousands of files on the host filesystem one at a time. The latter is a CPU and filesystem-metadata-journal problem, not a network one, so it stays put even if you lay down a 100Gbps link.

For the past several years, most of the buzz in this space has been about the first problem. Lazy-pull approaches like eStargz, SOCI, and Nydus said "don't fetch everything, fetch only the pieces you need," and AWS's SOCI even added a mode that flips the direction entirely: "just fetch it all in parallel, fast." That lineage was covered in the container runtime comparison post.

containerd 2.3, released on April 30, 2026, goes straight at the second problem. The idea is simple — instead of unpacking a tar to build a filesystem, ship the filesystem itself as the layer. This post is a record of how far that idea has actually come, verified against the code and the registry manifests.

What Actually Merged in containerd 2.3

Let's pin down the dates and PRs first. EROFS itself didn't debut in 2.3.

The pieces that landed in 2.3:

• #12567 (merged 2026-01-06) — adds the application/vnd.erofs.layer.v1 media type. In the PR description's own words, this lets "containerd pull EROFS-native container images directly."
• #13091 (merged 2026-04-01) — has the transfer service select the EROFS snapshotter/differ when os.features includes erofs.
• #12502 (merged 2026-04-01) — adds dm-verity support to the EROFS snapshotter.
• #13185 (merged 2026-04-12) — application/vnd.erofs.layer.v1+zstd, an EROFS layer wrapped in zstd.

For context, 2.3.0 is also the first release under containerd's changed release policy. In the release notes' own phrasing, the project moved to a roughly 4-month cadence aligned with the Kubernetes schedule, and 2.3 is the first annual LTS under that scheme. endoflife.date independently shows the same figures — 2.3 released 2026-04-30, EOL 2028-04-30, marked LTS. Worth noting from the same table: containerd 2.1 already went EOL on 2026-07-03.

What the Differ Code Tells You — Where Unpack Actually Disappears

The payoff of EROFS is clearest not in a benchmark chart but in a few lines of the differ code. containerd 2.3.0's plugins/diff/erofs/differ.go branches three ways.

1. When it receives a plain OCI tar layer — it spins up mkfs.erofs and converts the tar into EROFS on the fly. That's what ConvertTarErofs in internal/erofsutils/mount.go does.

args := append([]string{"--tar=f", "--aufs", "--quiet", "-Enoinline_data"}, mkfsExtraOpts...)
cmd := exec.CommandContext(ctx, "mkfs.erofs", args...)
cmd.Stdin = r

In other words, the tar stream is piped straight into mkfs.erofs's stdin. Rather than creating files on the host FS one by one, it stamps out a single EROFS blob, which means less metadata-journal traffic than overlayfs snapshotter's per-file extraction, and no need to delete tens of thousands of files during GC. This part has been true since 2.1.

2. When it receives an uncompressed native EROFS layer — this is the new path in 2.3, and the code ends like this.

layerBlobPath := path.Join(layer, "layer.erofs")
// Allow copy file range when there is an uncompressed native EROFS layer
if fastcopy {
    f, err := os.Create(layerBlobPath)
    ...
    _, err = io.Copy(f, content.NewReader(ra))
    f.Close()
    ...
    return desc, nil
}

This is the crux of it. No mkfs.erofs, no tar parser, no per-file creation. It just copies the blob from the content store to layer.erofs and returns. The unpack step itself hasn't been eliminated — it has been reduced to a single blob copy. The snapshotter then mounts layer.erofs under fs/ and returns an overlayfs mount combined with the parent layers.

3. When it receives a +zstd layer — covered separately below.

Checking It Directly Against the Registry

Reading the docs alone isn't much fun, so I poked at an actual registry. The EROFS maintainer has uploaded relevant tags to docker.io/hsiangkao/ubuntu as a test repository. What follows is what I pulled directly from registry-1.docker.io on 2026-07-16.

Open the OCI index for the 22.04-platforms tag, and you'll find two amd64 manifests.

{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    { "digest": "sha256:1c4cc37c...",
      "platform": { "architecture": "amd64", "os": "linux" } },
    { "digest": "sha256:06ae2788...",
      "platform": { "architecture": "amd64", "os": "linux",
                    "os.features": ["erofs"] } }
  ]
}

Same architecture, same OS — only os.features differs. Open the layers of each manifest and the media types diverge.

The whole tradeoff sits in those three lines. The uncompressed EROFS layer is about 1.53x the wire bytes of tar+gzip — you spend 53 percent more network to remove unpack. Wrap it in zstd and it actually comes in about 8 percent smaller than tar+gzip.

One honest caveat to attach here: this is not a benchmark, just one manifest I opened. It's a single image (an Ubuntu 22.04 base, one layer), and a test artifact uploaded by the EROFS project itself, and I did not verify whether the EROFS blob's contents are byte-for-byte the same rootfs as the tar layer. Compression ratio swings a lot depending on image content. What you should take away here isn't the exact ratio but the sign — uncompressed native gets bigger, wrapping it in zstd makes it smaller again.

os.features — Serving Two Worlds from One Tag

Let's look at why the index above is elegant. A client that can't use EROFS picks the plain tar+gzip manifest; a client that can picks the EROFS manifest. Same tag, same pull command. The PR description shows syntax like ctr i pull --platform="linux(+erofs)" docker.io/hsiangkao/ubuntu:22.04-platforms.

But was that actually what os.features was meant for? Reading the OCI image-spec's image-index doc, it's defined as follows — this OPTIONAL property is an array of strings for required OS features, and when os is windows, implementations SHOULD understand the value win32k. And then comes the decisive line.

When os is not windows, values are implementation-defined and SHOULD be submitted to this specification for standardization.

In other words, on Linux, os.features is a door the spec deliberately left open — "use it however you like, but come standardize it" — and containerd walked through that door. It hasn't submitted anything yet.

One easy thing to misread — support for os.features is not an automatic switch. PR #13091's description nails it down itself.

Users still need to explicitly specify the EROFS snapshotter to run EROFS native images, and this is by design. This only improves the user experience around the unpack process.

Instead of the opaque extraction error overlayfs throws when it runs into an EROFS image on the default snapshotter, you get a clear error — that's the entire scope of the improvement.

+zstd — Compression Comes Back, But Only Half the Unpack Avoidance

The 53 percent penalty on the uncompressed native layer stings, which is what +zstd is for. The idea is laid out well in RFC #12506 — leave the filesystem itself uncompressed (so runtime reads stay fast), and wrap it in zstd only for transport (so the pull is fast).

But looking at the code that actually landed in 2.3.0, the +zstd path does not take the fastcopy branch. It passes through the standard zstd processor chain before being copied to layer.erofs.

processor := diff.NewProcessorChain(diffLayerType, content.NewReader(ra))
...
if native {
    f, err := os.Create(layerBlobPath)
    _, err = io.Copy(f, rc)
    ...
}

To sum it up:

• mkfs.erofs still doesn't run — no tar parsing, no per-file creation.
• But the full stream decompression pass is back. It receives 27MB, unpacks it to 43MB, and writes that to disk.

So +zstd sits somewhere between uncompressed native and tar+gzip. For reference, the differ only accepts +zstd and rejects other suffixes (unsupported erofs layer suffix). +gzip isn't even on the table — according to PR #13185's description, that's because zstd has skippable frames. Which leads into the next section.

Lazy Loading Doesn't Exist Yet

This is the most important part of this post.

#12703 is a proposal to document the EROFS layer format. Top of the motivation list in its first section reads: "Runtime random access (lazy loading): containers can start immediately without waiting for the entire image to be downloaded or unpacked, and data is fetched on-demand."

The design is concrete, too. Inside a zstd skippable frame (magic 0x184D2A5E) sits a Chunk Mapping Table, so a standard zstd decoder just skips over it while an aware reader maps uncompressed offsets to compressed ranges to fetch specific chunks. The header carries magic/version/total size/chunk size (e.g., 4MiB)/hash algorithm, and each chunk carries an absolute offset within the blob plus an optional checksum. The metadata needed for this is passed through OCI manifest annotations — dev.containerd.erofs.zstd.chunk_table_offset, dev.containerd.erofs.zstd.chunk_digest, dev.containerd.erofs.dmverity.root_digest, and so on.

The problem is that this proposal, opened on 2025-12-18, has not been merged yet. Its own first line reads: "This is a draft proposal, I'm opening as PR rather than issue to make it easier to comment."

The implementation side is even more decisive. #12764 "Seekable-erofs" was opened on 2026-01-09, last updated on 2026-04-13, and is still not merged. The PR body lists what's implemented alongside what isn't, and the "isn't" list reads:

Does NOT implement:

• lazy-loading: the chunk table is written but not consumed
• dm-verity usage: the dm-verity data is decoded but yet not used in the differ / at mount time

The chunk table is written but not read. So as of July 2026, there is no lazy loading in containerd's EROFS path. That's the distance between the proposal's headline claim and what has actually shipped.

So if you boil down containerd 2.3's EROFS native layers to one line, it's this — it's not "fetch lazily," it's "don't unpack." The fact that the whole image is fetched up front is unchanged; what disappeared is the unpack. That's not a bad thing — it's just solving a different problem, and turning it on expecting "image pulls get faster" will leave you disappointed.

dm-verity — Quietly the More Interesting Part

Buried under the lazy-loading story is the part that actually shipped complete in 2.3: integrity.

The EROFS snapshotter now offers three integrity mechanisms.

• set_immutable — sets IMMUTABLE_FL on the layer blob to block delete/rename/modify and flushes dirty data immediately. It doesn't catch data corruption from hardware faults. There's a cost, and the containerd docs recorded their own measurement of it — on EXT4, unpack time for tensorflow:2.19.0 grew 108.86 percent, from 10.090 seconds to 21.074 seconds. This is the project's own measurement, limited to that image and that filesystem, and it's stated to have no effect on runtime performance.
• enable_fsverity — turns on fs-verity at commit time and verifies state before mount. As the docs candidly note, this adds runtime overhead because every image read in the container has to verify the Merkle hash tree first.
• dmverity_mode — the newcomer, added by #12502. Creates a dm-verity device per layer and mounts it read-only.

The dm-verity implementation is fairly clean. Instead of shelling out to the veritysetup CLI, it builds the Merkle tree directly with the containerd/go-dmverity library, appends the tree inline inside the blob, and keeps only the root hash and offset separately in a layer.erofs.dmverity JSON file. The remaining parameters live in the superblock inside the blob and are auto-detected at mount time. Block size is 4096 bytes in normal mode and 512 bytes in tar-index mode (a constraint from dm-verity's logical_block_size).

There are three modes: auto (default — uses .dmverity metadata if present, otherwise mounts as plain EROFS), on (forces it on every layer, fails if metadata is missing), and off (fully disabled).

There's a trap here. The docs bold a warning: if you turn on dmverity_mode = "on" while layers already exist that were unpacked without dm-verity, mounting fails for those layers because they have no .dmverity file. You have to clean up the snapshots and re-pull with the differ's enable_dmverity = true. If you're planning to roll this out to existing nodes, this isn't just a trap — it's a migration you need to plan for.

Proposal #12703 goes one step further here — if dm-verity data exists, its root hash should be used as the layer's DiffID. That would mean the value the kernel verifies block by block and the value the image config points to become the same thing — but this is still just a proposal.

What You Pay For

Here's what you need to know before turning this on, most of it from containerd's EROFS snapshotter docs.

Kernel. You need the EROFS module and Linux 5.4 or later (modprobe erofs). But what you actually want is file-backed mount, and that requires Linux 6.12 or later. Below that, it falls back to a loop device. Using dm-verity requires CONFIG_DM_VERITY and the device-mapper module on top of that.

External binaries. Converting an ordinary (non-native-layer) image into EROFS requires mkfs.erofs. Per the docs: erofs-utils 1.7 or later; -T0 --mkfs-time for reproducible builds needs 1.8 or later; --sort=none, which avoids unnecessary tar reordering, needs 1.8.2 or later. So this doesn't end with a single containerd binary — it adds one more package dependency to your node image.

Explicit opt-in. The EROFS snapshotter isn't the default. You have to pass --snapshotter erofs or set it in config, and you also need to touch the differ priority (default = ["erofs","walking"]).

The maintainers' own label. containerd's EROFS tracking issue #11340 groups the native-blob-related PRs under this line — "Native EROFS blob support in content store and image (experimental)." "Experimental" isn't my judgment — it's the project's own labeling. In the same list, go-erofs native library integration is still unfinished.

Ecosystem. This is probably the biggest wall. The EROFS native images sitting in registries right now are effectively test artifacts uploaded by project insiders. Standard build pipelines don't emit EROFS layers, and there's no guarantee that major registries, scanners, or signing tools understand this vendor media type. The broader image build pipeline was covered in Docker BuildKit and OCI Image Layers, and that world is still tar.

I couldn't verify the benchmarks. The containerd docs include unpack benchmarks for Top25 images and large AI images, but the numbers only exist inside PNG images, not as text. On top of that, the conditions stated in the docs are containerd 2.2.1, a local registry, and no parallel unpack. I won't cite a number I can't verify. Figures like "14 percent improvement for WordPress images" circulate in web search snippets, but pulling the original doc and grepping it turns up no such sentence.

Where the Standard Stands

Step back, and the real subject of this story isn't EROFS — it's the pace of OCI standardization.

The media type containerd used is application/vnd.erofs.layer.v1. Not org.opencontainers.*. The annotations are dev.containerd.* too, and proposal #12703 notes in a footnote — "Future Goal: as adoption widens, transition the dev.containerd.* annotations to a standardized namespace (e.g. org.opencontainers.*)."

So what's the state of the door on the OCI side? Lay out the relevant image-spec issues with their dates and the picture gets sharp.

The proposal to add lazy pull to the spec opened in December 2020 and went untouched for nearly 5 years after October 2021. The EROFS layer type issue has been open for more than 2 years. #1191's problem statement still holds up today — large artifacts (1) take too long to download and (2) take too long to unpack, and this issue starts to address (2) — and that "start" is now 2 years old.

So containerd didn't wait, and shipped with a vendor type instead. That's an observation, not a criticism — a working implementation genuinely does give the standardization discussion something to chew on. But using it does mean depending on something that isn't a standard, which also means the media type or annotation names could change during standardization. The proposal itself notes that possibility.

So, Should You Turn This On Now?

Worth trying

• If measurement has confirmed that unpack is the actual bottleneck on your node. If a multi-GB AI image with a large file count is grinding CPU/IO away on unpack, there's something to gain from just the EROFS snapshotter + differ that's existed since 2.1 (without native layers). Stamping out a blob with mkfs.erofs beats per-file extraction, and GC becomes deleting one blob instead of tens of thousands of files.
• If you're running a VM-based runtime. The containerd docs call this use case out explicitly — passing image layers through as EROFS instead of virtiofs or 9p in a VM container like Kata improves performance and memory footprint, and gVisor supports EROFS too.
• If block-level integrity is a requirement. In a regulated environment where you need to enforce dm-verity or fs-verity per layer, this is something the overlayfs snapshotter doesn't do well. The docs explain why — FS_IMMUTABLE_FL and fs-verity protect individual files, not subtrees, so they're inefficient on an overlayfs layer with tens of thousands of files. EROFS is a natural fit because a layer is a single file.
• If you need to enforce a disk quota on the writable layer. default_size lets you hand overlayfs a fixed-size block device as the upper layer.

Not yet

• If the problem is download, not unpack. Native EROFS layers don't help here, and if uncompressed, actively hurt. If you need lazy loading, look at eStargz/SOCI/Nydus, or just fetch faster in parallel.
• If you want to distribute native EROFS layers. There's no build pipeline, the media type isn't standard, and there's no guarantee the tooling around the registry understands it.
• If your node kernel is below 6.12, or getting erofs-utils onto your node image is impractical.
• If the experimental label is a dealbreaker for you. In production, that label means "the config surface and on-disk format can change in a minor release."

One thing worth adding — turning on the EROFS snapshotter and pulling native EROFS layers are two separate things. The former has existed since 2.1 and is worth evaluating today. The latter is 2.3's new story, and it's still experimental. Don't judge them as one bundle. If you want the fuller picture of containerd's image handling, see the containerd image management post as well.

Closing

To sum up: containerd 2.3 (2026-04-30, the first annual LTS) added the application/vnd.erofs.layer.v1 and +zstd media types, opening a path to pull an EROFS filesystem image straight from the registry and mount it. In the code, unpacking an uncompressed native layer ends with a single blob copy, os.features lets one tag serve both EROFS-aware and plain clients at once, and dm-verity gives you block-level integrity per layer.

In exchange, there's no lazy loading — the chunk table design lives in a proposal, the implementation PR is open, and that PR itself states it "writes the table but doesn't consume it." The media type is a vendor type, and the door on the OCI side has sat open for more than 2 years. Go uncompressed and you spend more wire bytes; claw that back with zstd and the decompression pass returns. It needs kernel 6.12 and erofs-utils, and the maintainers have marked it experimental.

If there's one thing to take from this post, I hope it's this — the next time someone says "faster" in a container image optimization conversation, ask first whether they mean faster download or faster unpack. Most of the buzz in this space over the past 5 years has been about the former, and what containerd 2.3 actually shipped is the latter. Which one hurts on your node is a question your profiler answers, not the release notes.

References

• containerd 2.3.0 release notes — EROFS native images, first annual LTS
• containerd EROFS snapshotter docs (v2.3.0)
• PR #12567 — Add EROFS layer media type
• PR #13185 — erofs-differ: support zstd-wrapped EROFS layers
• PR #13091 — Add os.features support for EROFS native container images
• PR #12502 — Add dmverity support to the erofs snapshotter using go-dmverity
• PR #12703 — EROFS layer format proposal (draft, not merged)
• PR #12764 — Seekable-erofs (not merged, lazy loading not implemented)
• Issue #11340 — EROFS Support and Improvements tracking
• OCI image-spec — os.features definition in image-index.md
• OCI image-spec Issue #1191 — erofs/squashfs layer media types
• OCI image-spec Issue #815 — eStargz lazy pull specification
• containerd release policy — LTS and support windows
• endoflife.date — containerd release/EOL table
• EROFS filesystem official documentation
• Container runtime comparison — containerd, runc, Kata, gVisor (related post)