Skip to content
←

필사 모드: Kafka Diskless Topics (KIP-1150): The Cross-AZ Cost You Trade for Latency — and a Feature Not Yet Shipped

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — The Cost Tiered Storage Didn't Solve

Anyone who has run Kafka in the cloud knows which line on the bill hurts. It isn't disk. It's network.

KIP-405 Tiered Storage is already widely used, and it has cut storage costs sharply by offloading inactive segments to object storage. But that is exactly the gap the motivation section of KIP-1150 points to — Tiered Storage "does not eliminate the need to replicate and durably store the active segment, which is the largest infrastructure cost for Kafka operators on hyperscalers today."

That's because Kafka replication is broker-to-broker communication, and in a cluster spread across availability zones (AZs), all of that traffic is cross-AZ. Regardless of how long you retain data, every incoming byte crosses an AZ boundary once per replication fan-out. Tiered Storage made "old data" cheap; it never made "data arriving right now" cheap.

On March 2, 2026, the Apache Kafka community accepted a proposal aimed squarely at this problem: KIP-1150, Diskless Topics.

What Was Accepted, and What Wasn't

Let's clear up a misunderstanding first: acceptance is not shipping.

The Proposed Changes section of the KIP-1150 text opens with: "This KIP will not require any changes to the codebase or documentation upon acceptance." In other words, acceptance changes not a single line of code. The next sentence is even more explicit: by accepting this KIP, the community is "reaching agreement on the need for this feature and the end-user requirements, not on the concrete implementation details."

The Kafka community calls this a meta-KIP, or a motivational KIP. There's precedent: KIP-500, which removed ZooKeeper in favor of KRaft, worked the same way — agree on the big direction first, then split the actual design into follow-up KIPs.

So here is the current state, precisely stated.

KIP-1150: Diskless Topics       Accepted (2026-03-02)   <- direction only, accepted
  ├─ KIP-1163: Diskless Core         Under Discussion    <- actual design
  ├─ KIP-1164: Diskless Coordinator  Under Discussion    <- actual design
  └─ KIP-1165: Object Compaction     Under Discussion (Re-Opened)

Apache Kafka latest release: 4.3.1 (2026-06-25 released) -> no Diskless

In other words, as of July 2026, Diskless Topics is in no Kafka release. No target release version has been announced. Only the direction has been agreed on.

For a sense of scale — per Aiven's Josep Prat in his post-acceptance writeup, KIP-405 Tiered Storage took 2 years and 3 months from first draft to acceptance (December 14, 2018 → February 18, 2021). And that was only the time to acceptance. KIP-1150 itself went from its first published version on April 16, 2025 to acceptance in about ten and a half months — fast because of the split strategy, not because implementation got faster. In the same post's words: "By having KIP-1150 accepted, we have charted the path and reached base camp. Now the climb begins."

Where the Cost Math Starts — Cross-AZ Traffic

The motivation section of KIP-1150 gives exactly three lines of numbers. Here they are, unchanged.

The last line matters. Only two of the three major clouds charge for cross-AZ traffic. If you run on Azure, Diskless's core economic argument doesn't hold. The KIP acknowledges this and lists benefits that survive even without the network-cost savings — less data to store on disk and rebalance between brokers improves cluster scalability, and object storage is more durable than local disk. That's an honest accounting, but if cost reduction was your team's main motivation, you need to redo the math on Azure.

Diskless promises to eliminate two things.

  • Eliminating cross-zone data-transfer cost from replication
  • Eliminating cross-zone ingress cost from producers and cross-zone egress cost to consumers

The second one stands out. It targets client traffic, not just replication traffic. To see how that's possible, we need to look at the architecture.

Architecture — WAL Segments and the Diskless Coordinator

KIP-1163: Diskless Core holds the actual design. There's one core shift: separate data from metadata.

For a classic topic, the broker that receives a Produce request does six things: validate the data, assign offsets and timestamps, inject offsets into the batch data, write to durable storage, wait for replication to complete, and return a response. In Diskless, offset and timestamp assignment moves to the Diskless Coordinator, and injecting offsets becomes each replica's job.

The Produce path flows like this.

1. The producer sends a Produce request to any broker (doesn't have to be the leader)
2. The broker buffers the request (up to a size or time limit)
3. Once the limit is hit, the broker creates one WAL Segment
   -> batches from multiple topics/partitions get mixed into a single object
4. Upload the WAL Segment to object storage (durability is secured here)
5. The broker commits the batch coordinates to the Diskless Coordinator
6. The Coordinator assigns offsets, persists the coordinates, and responds
7. The broker responds to every Produce request bundled into that object

One design decision here touches cost directly. Unlike a classic segment, a WAL Segment mixes data from multiple partitions into a single object. In the KIP's words, this is "necessary to keep object storage write operation costs at a reasonable level" — if every partition got its own object, PUT request counts would explode in proportion to partition count. But that comes at a price: because one object holds data from multiple topics, physical deletion gets hard. KIP-1163 solves this with logical deletion, and honestly pins the point of physical deletion for compliance purposes at "a bit longer than the longest roll time."

KIP-1150 also spells out what "no disk" actually means. Diskless means broker disks are no longer the primary durable store for user data — not that disks disappear. KRaft metadata, batch metadata, data in transit to Tiered Storage, and caches for serving consumers all still use disk. The KIP puts it neatly: "Diskless's relationship to 'no disks' is the same as Serverless's relationship to 'no servers.'"

On the metadata side, KIP-1164: Diskless Coordinator has an interesting implementation choice. Coordinator state is stored in a new internal topic called __diskless_metadata, sharded across multiple partitions to create multiple coordinators. Local state materialization uses SQLite. The reasoning is spelled out: unlike other coordinators, the Diskless Coordinator's state size is expected to reach "hundreds of megabytes or even gigabytes," making it impractical to hold entirely in memory. If SQLite showing up inside a Kafka broker feels unusual to you, that reaction is part of why these KIPs are still Under Discussion.

The Latency Bill — the Budget KIP-1163 Wrote Down Itself

Now for the most important part of this post. Diskless's price is latency, and KIP-1163 wrote the budget down without hiding it.

Append latency is broadly composed by:
- Buffering: up to 250ms or 4MiB (both configurable)
- Upload to Remote Storage: P99 ~200-400ms, P50 ~100ms
- Batch Coordinates Commit: P99 ~20-50ms, P50 ~10ms (depending on batch count)

We are aiming for a Produce request latency of P50 ~500ms P99 ~1-2 sec

How you read this matters. This is a target, not a measurement. The KIP's own text says "We are aiming for," and I won't reinterpret that as "this is what it delivers." There's no upstream implementation yet, so there can be no measured value.

Still, this budget reveals the structure. The target P50 is about 500ms, and up to 250ms of that is buffering. That means roughly half the latency doesn't come from storage being slow — it comes from a decision to wait: absorbing object storage's write-operation cost requires batching, and batching requires waiting. This is latency born from economics, not physics.

The other half is physical. KIP-1163 writes that "remote storage can have higher latency than local disk, which increases Kafka request and end-to-end data latency," and the WarpStream docs make the same point: "creating a file in object storage is a relatively high-latency operation compared to a local disk write."

So what should this 500ms be compared against? Neither the KIP nor the vendor docs state a baseline for classic Kafka's Produce latency, so I won't invent a number here either. The comparison point is your own current cluster — how many milliseconds your producers are getting responses in right now lives only on your dashboard, and placing that number next to 500ms is the only comparison that means anything.

KIP-1163 doesn't hide this tradeoff — it makes it a design principle. Diskless topics, classic topics, and tiered topics coexist in one cluster, letting application developers "trade off latency and cost on a per-topic basis." In the KIP's description of data flow, low-latency data keeps riding classic topics, and high-latency-tolerant data goes to the Diskless ingestion engine. In other words, this isn't a feature that replaces Kafka — it adds one more dial to Kafka.

How to Read Vendor Numbers

Commercial products already run this same idea in production — WarpStream, Buffstream, Confluent Freight Clusters, Redpanda Cloud Topics, and others. Their published numbers are the only real-world reference point for "what latency does an S3-backed log engine actually produce," but don't forget: every one of them is vendor-measured.

WarpStream's official docs are on the more candid end. Its performance tuning doc says: "The WarpStream Agent has no local disk and writes data directly to object storage. Creating a file in object storage is a relatively high-latency operation compared to a local disk write. For example, in our experience the P99 latency of writing a 4MiB file to S3 is about 400ms." (Tuning for Performance) The qualifier "in our experience" is right there in the original text.

What the doc says next is more practical. To get throughput on top of a 400ms latency, Little's Law says you need concurrency. Carrying over the doc's own example: 1GiB/s of write throughput at a 1MiB batch size requires 400 concurrent in-flight Produce requests. And it states flatly that "not having enough in-flight requests is the single biggest cause of low write throughput when using WarpStream." Moving to an S3-backed engine changes the whole nature of client tuning.

There's a way to push latency lower, but it isn't free. WarpStream's S3 Express doc says that "combined with a reduced batch timeout, S3 Express can bring Produce request P99 latency below 150ms." But the same doc lists the price: since S3 Express One Zone stores data in a single AZ as the name implies, WarpStream has to replicate across a quorum of multiple single-zone buckets, and S3 Express storage cost is already "about 7x regular object storage before accounting for replication." So the recommended setup is S3 Express for ingestion only, with compaction sent down to a regular bucket. Which brings us back to the original story: buying latency costs money.

On the other hand, some numbers need to be read carefully. A related-resources card on Aiven's post-acceptance post carries the phrase "reduce TCO by up to 80%." This is a vendor's own claim, and the post gives no basis whatsoever for what that 80% includes or excludes — which workload, which cloud, which replication factor, which retention period, whether compute is included. It's just a ceiling ("up to") attached to promotional copy. Aiven is both the primary author of KIP-1150 and a company that profits from managed Kafka (the post itself is honest about this interest). TCO calculations from streaming vendors are especially self-serving, so treat a number like this as a directional signal at best, until you verify it against your own workload.

What Breaks Quietly

KIP-1150 states that Diskless topics are intended to be "semantically interchangeable" with classic topics, and lists support for ordering guarantees, idempotence, transactions, consumer groups, queue/share groups, and tiered storage. Most of that is probably true, but reading the design docs closely turns up a few rough edges.

Queues (share groups) don't get the full cost benefit. KIP-1163 says so directly — only the share-partition leader can handle ShareFetch requests, and that leader lives wherever the partition leader lives. So "queue consumers must send ShareFetch requests to the partition leader, and cannot always benefit from the elimination of cross-zone consumer traffic." That's one exception carved into the promise of eliminating cross-zone egress, and rack-aware optimization for queue functionality is pushed out of this KIP's scope.

The meaning of ISR changes. In Diskless, an "in-sync replica" doesn't mean synced with the leader — it means synced with the Diskless Coordinator, since the Coordinator is the source of truth. KIP-1163 draws a sharp implication from this: since the leader is just another replica, the leader itself can be out-of-sync. When that happens, the leader can't efficiently do its own job of offloading segments to tiered storage, and reading from the leader is discouraged, just as with any other out-of-sync replica. This is a point where a longtime Kafka operator's intuition folds over once. For reference, replicas still send FetchRequests to the leader even though they no longer replicate broker-to-broker — the leader returns an empty, record-less response, and that round trip survives purely as an ISR-tracking mechanism.

Older clients need a workaround. New clients send rack information via MetadataRequest v14 and get back which topics are Diskless and which broker is best to produce to. But for legacy or third-party clients, KIP-1163's proposal is to append a rack identifier to the client ID.

client.id = "my-app,diskless_rack_id=use1-az4"

The broker reads this string and fills the response with a replica from that rack as if it were the leader. It works, but as the KIP itself admits, this breaks ShareFetch, which only the real leader can handle. So an operational guideline follows: "set diskless_rack_id for producers and regular consumers, but not for share consumers — the latter must be a separate instance in code." It isn't elegant, and they know it.

So, Should You Wait?

Here's how the decision breaks down.

Likely worth it

  • You run on AWS or GCP, and throughput is high enough that cross-AZ replication traffic shows up as a real line item on the bill.
  • The workload tolerates latency — log collection, metrics, analytics pipelines, batch-style ingest, where hundreds of milliseconds of Produce latency mean nothing.
  • Throughput is high and per-partition data is dense enough that batches fill naturally.

Overkill or harmful

  • You run on Azure. With no cross-AZ charges, the core economic argument disappears.
  • Latency is a product requirement. You can't put order execution or a user-facing request path on a path whose target is P50 500ms.
  • Throughput is low. Low traffic means low cross-AZ cost too, so you'd eat the buffering latency for nothing.
  • Queues/share groups are your main usage pattern. The exception covered above hits you directly.
  • And you need it right now. This is the most practically disqualifying condition.

Let me stress that last item again. As of July 2026, this feature does not exist. Only the direction has been accepted, two design KIPs are still under discussion, and there's no target release. If your cross-AZ bill hurts right now, your options are a commercial product (WarpStream and others), Aiven's open-source experimental fork Inkless, or gritting your teeth through existing techniques (like single-rack topics). KIP-1150's Rejected Alternatives section explicitly dismisses existing workarounds like single-rack topics as forcing "application-specific compromises in durability, availability, semantics, and usability" — and it's ironic that those compromises are exactly what's available to you to choose right now.

The Road Ahead

What makes KIP-1150 interesting has more to do with politics than technology.

The last item in the Rejected Alternatives section is "Do Nothing," and its reasoning is unusually candid — doing nothing would make this "the single biggest missing feature in the upstream implementation," push high-scale, cloud users toward Kafka alternatives, and risk Kafka "losing authority over the Kafka protocol" entirely. And in fact, every cloud-native Kafka-compatible product to emerge in the last few years has been closed source. KIP-1150 is upstream's response to that trend.

How serious that response was shows in the fact that three competing proposals landed at once — alongside KIP-1150, KIP-1176 (Tiered Storage for the Active Log Segment) and KIP-1183 (Unified Shared Storage) were submitted into the same space, and it was ultimately resolved when KIP-1183's authors (Slack) withdrew their own proposal and formally endorsed KIP-1150. The vote restarted on January 9, 2026 and passed on March 2, receiving 9 binding and 5 non-binding votes, per Aiven's tally.

What's left is actually getting KIP-1163 and KIP-1164 passed. Beyond that lie items KIP-1150 leaves as "Further Work" — things that don't even have a KIP yet: converting topic type between classic and Diskless, broker role specialization (dedicated produce/consume/coordination/compaction brokers), parallel Produce processing to raise per-producer throughput in high-latency environments, an Iceberg format, and multi-region active-active via metadata replication. If the last two ever materialize, they'd touch Kafka's identity itself — but for now they're just a line on a list.

Closing

To sum up: Tiered Storage made old data cheap but left active-segment replication traffic untouched, and that traffic is Kafka's largest infrastructure cost on hyperscalers. KIP-1150 is the community agreeing on a direction — using object storage as the source of truth to eliminate that cost — and only the direction.

The price is clear, and the design docs don't hide it: target Produce latency of P50 ~500ms, P99 ~1-2 sec. Roughly half of that is time spent batching to absorb object storage's PUT cost. So Diskless isn't "better Kafka" — it's a dial you can turn per topic. Topics that need low latency stay classic; only high-volume, latency-tolerant topics go to Diskless. That's the picture its designers have drawn.

And that dial isn't in your hands yet. It isn't in Kafka 4.3.1, and no timeline for when it arrives has been announced. So what you should be doing right now isn't a migration plan — it's measurement: how much of your bill your cross-AZ traffic actually is, and whether your producers can tolerate 500ms. Without those two numbers, you can't judge whether to use this feature even after it arrives. The bill comes before the feature, not the other way around.

References

현재 단락 (1/89)

Anyone who has run Kafka in the cloud knows which line on the bill hurts. It isn't disk. It's networ...

작성 글자: 0자원문 글자: 18,328자작성 단락: 0/89