Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — Standing in Front of a Bug That Won't Reproduce
What Deterministic Simulation Testing Actually Is
Aiven's Experiment — Putting Diskless Kafka Into the Simulator
Result 1 — 2,200 Logical Hours, Zero Bugs
Result 2 — KAFKA-19880, Found in the Control Group
What This Case Really Teaches — Reproducibility
And the Ending Isn't Clean
A Case Where the Numbers Look Better — And How to Read Them
The Honest Tradeoffs
So, Should You Use It
Closing
References

Introduction — Standing in Front of a Bug That Won't Reproduce

Something strange happened once in production. It left a trace in the logs, but no matter how hard you try to recreate the same conditions, it never comes back. You run the test 100 times and it's green every time. Eventually you write it off as "transient" and close the ticket. Six months later the same thing comes back, worse.

Aiven's Ivan Yurchenko calls these bugs heisenbugs — bugs produced by the nondeterminism that comes from IO, execution scheduling, and other sources of randomness. Networks delay packets, drop them, deliver them out of order. The OS schedules threads however it pleases. Disks fail. In his words, when one of these bugs goes off in production, understanding it from telemetry alone is effectively impossible.

"Write more tests" doesn't help much here, because you can't test for what you never imagined. This post follows a technique that attacks the problem from a different angle — deterministic simulation testing (DST) — and one Apache Kafka bug it actually found in November 2025. And it explains why that case is far more interesting than a marketing blog post, precisely because the outcome isn't clean.

What Deterministic Simulation Testing Actually Is

Antithesis's documentation defines DST as placing the software under test into a simulated, deterministic environment. Two words carry the weight.

Simulated. Some or all of the system runs on a simulator instead of on real hardware, network, and OS. That lets the test harness control what happens at the simulated layer — when and which failures occur.

Deterministic. Sources of nondeterminism — the clock, thread interleaving, the system's random numbers — are made deterministic. So fixing a seed reproduces the exact same run every time. Why this matters is exactly what the previous section showed: finding the bug wasn't the hard part; recalling it was.

Per the docs, practical adoption of this approach happened around 2010 at both FoundationDB and AWS, close to a case of simultaneous invention (the idea itself predates that). At AWS, Al Vermeulen introduced it to test an early implementation of an internal lock service, and one of the earliest recorded discussions is Will Wilson's 2014 Strange Loop talk. The framework FoundationDB built later became the technical foundation of a company called Antithesis.

Implementation splits roughly into two approaches.

The FoundationDB approach. Design the system so every nondeterministic component is pluggable from the start. Powerful, but it means the system and its entire dependency tree have to be built with DST in mind from day one. In the Antithesis docs' words, this is generally unrealistic for a system that's already in production.
The deterministic hypervisor approach. Run ordinary, nondeterministic software as-is inside a deterministic hypervisor. This is what Antithesis sells, which means you don't have to touch your code.

And there's a point the docs make that's easy to overlook. People treat "building a fully deterministic system" as the main technical challenge, but exploring the state space thoroughly and efficiently is just as hard a problem — because for most software the state space is extremely large. That's why DST travels together with property-based testing/fuzzing and fault injection. (Property-based testing itself has been covered separately.)

One more concept before moving on. When the simulator owns the clock, code doesn't actually have to wait when it does the equivalent of time.Sleep(). That lets you compress far more logical time (application time) than wall-clock time. Every number below is in that unit.

Aiven's Experiment — Putting Diskless Kafka Into the Simulator

On March 5, 2026, Aiven published the process. The target was Inkless — a temporary Apache Kafka fork implementing KIP-1150: Diskless Topics. It's a feature that uses object storage as the primary store instead of the broker's local disk, and the KIP wiki's current status is Accepted.

The motivation is clear. They're adding a large number of new code paths to the existing Kafka codebase, and they need to confirm those paths are correct on their own — but more importantly, that they don't break the existing Kafka paths.

The setup: instead of writing a new harness, they extended Kafka's existing Ducktape system test suite.

14 Ducktape workers inside Docker Compose (running both brokers and clients on the Inkless code), PostgreSQL for the Inkless control plane, and MinIO for storage
A system test that keeps producing to a Diskless topic and checks everything gets consumed (inkless_produce_consume_test.py), into which they imported the Antithesis SDK for extra asserts
Push the image to a registry and kick off a run with a single HTTP POST

What matters here isn't the tooling — it's the invariants. The test only ever checked three things.

The test never halts (Antithesis's reachable operator)
Every message written is eventually read (always_or_unreachable)
Nothing bad happens — messages read out of order, duplicate offsets, or a partition set that changes (unreachable)

DST is not magic. It catches nothing unless you tell it what "wrong" looks like. The simulator just produces strange states; it's these three lines that decide whether a state counts as strange.

Result 1 — 2,200 Logical Hours, Zero Bugs

The test kept producing to and consuming from the Kafka cluster inside the Antithesis supervisor for a total of roughly 2,200 logical hours. And it found no problems.

The sentence Aiven attached to this result is, I think, the most valuable line in the whole piece.

A passing test can be the most suspicious kind of test. It might just mean you haven't found the problem yet.

For reference, Aiven states that they ran roughly 200 wall-clock hours overall, which amounted to about 9,700 logical hours (1.1 years). The 2,200 above is the share of that spent on this particular Inkless system test.

Let's be precise about what "zero bugs" actually proves. It is not proof that Inkless is correct. It means with this workload, these three invariants, and these seeds, no violation was found. Code paths the workload never touches, and properties the invariants don't express, remain blind spots.

Result 2 — KAFKA-19880, Found in the Control Group

So Aiven ran a control. To double-check their own assumptions, they ran part of Kafka's original system tests through Antithesis — not against the Inkless fork, but against unmodified upstream Apache Kafka. It was a sanity check, meant to see whether the harness could actually catch a bug at all.

And that's where KAFKA-19880 came from. Its title:

First batch in producer epoch can be appended out-of-order even for idempotent producer

In Aiven's words, it was "surprising." The apparatus built to validate their own new code instead brought back a bug from the control group they'd set up. To quote Aiven's own conclusion directly, Apache Kafka is very mature, battle-tested software whose community has always been diligent about testing and design.

The mechanism is this. The producer is configured with enable.idempotence=true and max.in.flight.requests.per.connection=5, and it sends enough records to a freshly created, empty topic to form several batches. But there's a connectivity problem between the cluster and the target broker that affects metadata propagation.

Producer: enable.idempotence=true, max.in.flight.requests.per.connection=5
Target: freshly created empty topic input-topic-0

correlationId=21  base sequence 0     -> NotLeaderOrFollowerException  (broker doesn't yet know it's the leader)
correlationId=22  base sequence 1458  -> NotLeaderOrFollowerException
correlationId=23  base sequence 2823  -> NotEnoughReplicasException    (ISR size 1 < min.isr 2)
correlationId=24  base sequence 4188  -> success (!)
correlationId=25  base sequence 5553  -> ...

Result: the base sequence of the first batch ever written to this partition
        is 4188, not 0.
        Even with idempotence enabled, the first batch is accepted out of logical order.

In the issue's own words, by the time correlationId 24 arrives the request succeeds, so the first batch ever written is accepted even though it's logically out of order. The earlier failed batches may later be retried under a new producer epoch — and in the reporter's case, they were — but that doesn't matter, because the intended record order has already been broken.

The condition is clear too. The partition doesn't need to be empty — it's enough for the producer state to be empty and for that epoch's first requests to fail as above. Conversely, if the epoch's first request succeeds, this situation is impossible: if r1 succeeds and r2 fails with NotLeaderOrFollowerException, then r3 gets rejected with OutOfOrderSequenceException. The reporter traces this discrepancy to the validation logic in the broker-side ProducerAppendInfo.java.

What This Case Really Teaches — Reproducibility

This is where DST's value shows up. This is the reporter's own sentence, straight from the issue body.

Under Antithesis I can reproduce this live fairly reliably. I doubt it'd be easy to build a local setup that catches this at a decent rate.

But he didn't stop there — he reduced it down to a single unit test inside ReplicaManagerTest.

// excerpted and condensed from the reproduction code in the KAFKA-19880 issue body
val r0 = MemoryRecords.withIdempotentRecords(
  Compression.NONE, producerId, producerEpoch, 0, new SimpleRecord("record 0".getBytes()))
val r1 = MemoryRecords.withIdempotentRecords(
  Compression.NONE, producerId, producerEpoch, 1, new SimpleRecord("record 1".getBytes()))

// val order = List(r0, r1)  // normal order -- no ordering violation
val order = List(r1, r0)     // r1 is accepted out of order
                             // r0 is retried later under a higher epoch and accepted

This flow is DST's real selling point. The simulator reliably reproduces a rare interleaving → a human looks at it → it gets compressed into a deterministic unit test that runs without the tool at all. The final artifact is 20 lines of Scala that anyone can run without an Antithesis subscription. The tool was the ladder, not the destination.

And the Ending Isn't Clean

This is the part that doesn't make it into the marketing material.

First, the reporter himself isn't sure. The last question in the issue body is this — as far as I've read the docs, this is a bug, I expected enable.idempotence to prevent this, am I right, or is this acceptable behavior and the docs need improving? Aiven's blog post keeps the same tone — whether this is a documentation gap or a strict bug remains to be seen, and they're waiting on the community's answer.

So what's the state of this issue now? Querying JIRA as of July 2026 gives this.

KAFKA-19880
  Summary:     First batch in producer epoch can be appended out-of-order
               even for idempotent producer
  Status:      Open
  Resolution:  Unresolved
  Priority:    Minor
  Reporter:    Ivan Yurchenko
  Created:     2025-11-12
  Affects:     4.0.1, 4.1.0
  Fix Version: (none)
  Components:  core, producer
  Comments:    1

Eight months after it was reported, there's exactly one comment. On March 10, 2026, Travis Bischel (author of franz-go, the Go Kafka client) left a single line — "Likely related: KAFKA-14312."

Open KAFKA-14312 and its title reads: "Kraft + ProducerStateManager: produce requests to new partitions with a non-zero sequence number should be rejected." Produce requests arriving at a new partition with a nonzero sequence number should be rejected. The reporter is the same Travis Bischel, reported on October 18, 2022, priority Major. And its status is Resolved / Won't Fix. It was closed on November 12, 2024.

To be fair, Travis only said "likely related," not that it's the same issue — nobody has made that call yet. But the picture that emerges is this: a state-of-the-art simulator ran for hundreds of hours and dragged back an anomaly, it reproduced, it got compressed into a unit test — and whether it's a bug to fix or a spec that's just poorly documented is a question that's already been raised once, in an adjacent form, and closed as won't-fix, and it's still open now. Priority: Minor.

What DST automates is discovery. Deciding whether something is a bug, prioritizing it, and finding someone to fix it is still a job for people and organizations. Buying the tool solves the first half; the second half stays exactly where it was.

A Case Where the Numbers Look Better — And How to Read Them

If Aiven's "zero bugs" result feels anticlimactic, there's a frequently cited counterexample. WarpStream wrote about this in March 2024. Before reading it, let's be clear about the source — WarpStream sells a Kafka replacement built on object storage (the post itself carries that disclosure), and this is an Antithesis customer writing about Antithesis. Every number below is self-reported.

Antithesis simulated 280 hours of application time while running the WarpStream workload for 6 wall-clock hours.
Of that, it took about 160 application hours before it stalled — stopped discovering new "behaviors." WarpStream's own conclusion here is an honest one — running past 160 hours has diminishing returns, and the investment should instead go into making the test itself more sophisticated.
What got caught on day one was a data race in a metrics instrumentation library. It had been there since the project's first month, and it had run for literally tens of thousands of hours in CI with the Go race detector on, and it was never caught. Antithesis caught it in 233 seconds of runtime.
More interesting is the data-loss bug. To cut S3 PUT costs, the Agent buffers Produce requests from multiple clients in memory for about 250ms, then merges them into a single file and flushes it to object storage. A refactor that added speculative retry to the flush subtly broke error handling, so that for a very brief window a file that had failed to flush was treated as if it had succeeded. The background goroutine that commits this to metadata polls every 5ms, and the gap between the two problematic state transitions is usually under one microsecond — meaning two rare events, a network failure and a specific thread interleaving, have to coincide. They never saw it in staging; inside Antithesis it happened roughly once per wall-clock hour. They report it's fixed now.

What to take from this case isn't the dramatic contrast of "233 seconds vs. tens of thousands of hours," but the structure underneath it. CI that ran for tens of thousands of hours with the race detector on was, in effect, repeating the same scheduling tens of thousands of times. Repetition is not exploration. What DST sells isn't time — it's diversity of interleavings.

In the same post, WarpStream spends several paragraphs arguing that Antithesis beats Jepsen; since that's clearly one party's opinion rather than a verified fact, it's best not to take it at face value. (Chaos-engineering approaches in general were covered separately in the chaos engineering piece.)

The Honest Tradeoffs

Even Antithesis's own documentation writes its limits down plainly — which, coming from the vendor itself, is if anything more trustworthy.

The setup is heavy. The docs say building out a deterministic simulation environment is a complex, resource-intensive undertaking.

Not every system can do this. In the docs' words, not every system can be designed in a way that lets you build DST around it.

External dependencies still have to be mocked. Platforms like Antithesis make DST feasible for most software, but to guarantee determinism, external dependencies still have to be mocked or otherwise swapped out. This is a bigger constraint than it sounds. Look at Aiven's setup again — MinIO instead of S3, a Postgres container instead of a managed DB. What you're simulating isn't production; it's a model of production, and S3's actual consistency behavior, or a managed service's actual failure modes, only show up to the extent the model is faithful to them.

State-space exploration is the real bottleneck. WarpStream's 160-hour stall point shows this. Past some point the simulator stops finding new behaviors, and what's needed from there isn't more time but a better workload and sharper invariants. That's design work done by people, and it doesn't automate.

It isn't free. Antithesis is a commercial product. I haven't been able to confirm a public price sheet, so I won't invent a number here, but the fact that 24/7 simulation costs compute is plain enough on its own. For what it's worth, Aiven was already running a smaller-scale DST setup before adopting Antithesis — property-based writer tests that randomize inputs like the number of producer requests, message arrival times, and payload shapes — and reports that it helped. They add, though, that they had to write and maintain quite a lot of code for those tests themselves, and that parts of it were awkward. The ladder has more than one rung.

And the outcome can end up being a people problem. Exactly as seen in the KAFKA-19880 section.

So, Should You Use It

Likely to pay off

Systems where concurrency, state, and coordination are the whole point. The examples the Antithesis docs give are distributed databases, financial transaction engines, distributed infrastructure, blockchain/consensus protocols, and asynchronous workflows. (Domains with sharp invariants, like a double-entry ledger, fit especially well.)
Unreproducible production anomalies are already causing you pain.
You can write your invariants as sentences — "what's written gets read," "offsets increase monotonically." If you can't do this, turning on DST gives you nothing to catch.
You already do example-based testing and deterministic CI well, and need to go beyond it.

Overkill

Ordinary CRUD applications and most web services. The bugs you find there are usually logic bugs, not interleaving bugs, and logic bugs are caught by far cheaper tools.
You don't yet have the basics of unit and integration testing in place. DST is a layer on top of that, not a replacement for it.
The system's core value lives in external SaaS calls. Mock all of that away for determinism, and what you actually wanted to test disappears with it.
You're short on time and about to write two lines of invariant and call it done. You'll get "zero bugs" — and that's precisely the most suspicious outcome Aiven warned about.

I'd recommend climbing the ladder from the bottom rung. Seed-fixed property-based tests → a homemade lightweight simulator or deterministic fault injection → and only once something still slips through and that bug's expected cost is high enough, the deterministic hypervisor. That's the actual order Aiven followed.

Closing

To sum up: DST traps sources of nondeterminism inside a simulator, deliberately explores rare interleavings, and reproduces whatever it finds with a seed. The value is less "finds more bugs" than "can recall the bug it found." KAFKA-19880 being reduced to 20 lines of deterministic unit test is the evidence.

At the same time, this case is a fairly honest snapshot of where this technology stands today. Running their own code for 2,200 logical hours turned up nothing; running a control against mature upstream code turned up one bug, which has sat at Minor/Open for eight months, and an adjacent issue in the same family was reported in 2022 and closed Won't Fix in 2024. This doesn't mean it's a failure story. It means this is what the tool actually working looks like.

So don't pick the tool first. Start by writing, in one sentence, the thing that must never happen in your system. If that sentence doesn't come out, no simulator will save you — and if it does come out, you're already, surprisingly, halfway there.