Skip to content
Published on

The Truth About Exactly-Once Semantics

🇰🇷KO🇺🇸EN🇯🇵JA
Split View필사 모드
Authors

Introduction — The Temptation of "Exactly Once"

The first time anyone works with a message queue or a payment system, they make the same wish: "I hope each message is processed exactly once." A payment should be charged once, an email sent once, stock decremented once. Charge twice and the customer is furious; charge zero times and the company loses money. So we crave exactly-once.

But study distributed systems and you soon meet a bewildering sentence: "exactly-once delivery is impossible." This seems to flatly deny our wish. Yet at the same time, systems like Kafka advertise "exactly-once semantics." So which is it?

The goal of this post is to clear up that confusion. The key is a single distinction: exactly-once delivery is impossible, but exactly-once processing is achievable. Unpack that sentence fully, and it becomes clear how to design for correctness in payments and messaging.

If you want to see the message flow that connects to these ideas, you can visualize it in this site's Message Queue Playground.

The Three Delivery Guarantees

When a messaging system says it "delivers a message," that guarantee comes in three grades. Distinguishing these precisely is the starting point for everything.

At-most-once. A message is delivered once, or not at all. There is never a duplicate, but there can be loss. The sender "fires and doesn't check." Fast and simple, but suitable only for data you can afford to lose (e.g., bulk metric samples, some logs).

At-least-once. A message is always delivered at least once. There is no loss, but there can be duplicates. The sender retransmits until it receives an acknowledgment (ack). This is the default of most practical messaging systems. You accept "may be duplicated" in exchange for "never lost."

Exactly-once. A message is delivered exactly once, with neither loss nor duplication. It sounds the most ideal, but as we'll see, achieving this purely at the "delivery" level is fundamentally impossible.

  at-most-once : 0 or 1 time   (loss possible, no duplicates)
  at-least-once: 1 or more     (no loss, duplicates possible)
  exactly-once : exactly 1     (no loss, no duplicates — the ideal)

An important intuition here: at-most-once and at-least-once are opposite trade-offs. One tolerates loss to eliminate duplicates; the other tolerates duplicates to eliminate loss. And real reliable systems almost always choose at-least-once, because loss is usually far more catastrophic than duplication. The remaining problem, then, is "how do we make that duplication harmless?"

Why Exactly-Once Delivery Is Impossible

The reason exactly-once delivery is impossible is not a deep theorem — it emerges from a very simple situation: the network fails, and failures are indistinguishable.

Consider a sender A sending a message to receiver B and waiting for an ack. A didn't receive an ack. But A can never distinguish between two cases.

  Case 1: the message never reached B
    A --message--X (lost)     B
    -> B didn't receive it. A must retransmit.

  Case 2: the message arrived but the ack was lost
    A --message--> B (processed it!)
    A <--ack--X (lost)
    -> B already received it. If A retransmits, that's a duplicate!

From A's point of view, the two cases look identical: "I sent a message but no ack came back." Here A faces a dilemma.

  • If it retransmits: Case 1 is resolved, but in Case 2, B receives the message twice (duplicate). → at-least-once
  • If it doesn't retransmit: Case 2 is safe, but in Case 1, the message is lost forever. → at-most-once

So at-least-once and at-most-once fork on a single choice: "whether to retransmit when no ack arrives." And the perfect point in between — "delivery with neither loss nor duplication" — cannot exist at the level of delivery itself, because of this fundamental ambiguity. As long as the network can disconnect at any time, the sender has no way to know for certain "did the other side receive it."

The Two Generals Problem — The Impossibility of Agreement

The classic that shows this ambiguity most sharply is the Two Generals' Problem, a thought experiment that reveals the fundamental difficulty of distributed agreement.

Two generals are each encamped on a hill, with the enemy in the valley between them. Their armies win only if they attack simultaneously. If only one attacks, they lose. The generals communicate by sending a messenger across the valley, but the messenger can be captured and lost.

  General A: "attack at dawn" --messenger--> General B
    (if the messenger is captured? A doesn't know B received it)

  General B receives it and "agreed!" --messenger--> General A
    (if this messenger is captured? B doesn't know A got the agreement)

  A receives the agreement and "confirmed!" --messenger--> ...
    (this too can be captured. an endless confirmation of confirmation...)

The key is this: no matter how many confirmation messages they exchange, the sender can never be certain the last message arrived — because the last messenger might have been captured. So the two generals cannot reach the common knowledge that "we both know for sure" through any finite exchange of messages. This is a proven impossibility.

The lesson the Two Generals Problem gives us is clear. Over an unreliable channel, "certain, single-shot agreement" is impossible. So instead of pursuing certainty, we must find a detour that solves the problem at a different level. That detour is the next part of the story.

Not Delivery but Processing — A Shift in Thinking

Here a decisive shift happens. If exactly-once delivery is impossible, let's change the goal. What we truly want is not "the message is delivered exactly once" but "its effect is applied exactly once."

What the customer wants is not "the payment message crosses the network exactly once." It's "the charge happens exactly once." These are different. Even if the message crosses the network twice, as long as the charge happens once, the customer is satisfied.

This is the idea of exactly-once processing. Leave delivery as at-least-once (that is, allow duplicates), and instead absorb the duplicates harmlessly on the receiving side so that the final effect is exactly once.

  Give up: exactly-once delivery   (fundamentally impossible)
  Adopt:   at-least-once delivery  (duplicates possible, no loss)
           + absorb duplicates on the receiving side
           = exactly-once processing  (final effect is once)

The beauty of this shift is that instead of forcing the impossible, we compose achievable things to produce the desired result. The three key ingredients of that composition are idempotency, deduplication, and the transactional outbox. Let's look at each.

Idempotency — The Same Result No Matter How Many Times

The most important weapon of exactly-once processing is idempotency. An operation being idempotent means that doing it once or many times produces the same result.

Let's get a feel with a few examples.

  • "Set the balance to 200" → idempotent. However many times you run it, the result is a balance of 200.
  • "Subtract 100 from the balance" → not idempotent. Run it twice and 200 is subtracted.
  • "Mark this order as 'paid'" → idempotent. However many times you do it, the state is a single 'paid'.

The key is that even if duplicate delivery occurs, if the processing is idempotent, the duplicate automatically becomes harmless. So the first step of designing exactly-once processing is "make operations idempotent wherever possible."

The problem is inherently non-idempotent operations like "subtract 100." Payments and stock decrements are usually of this kind. In such cases, if you can't make the operation itself idempotent, you must "remember whether it was already processed" and filter out duplicates. That is the next ingredient: deduplication.

Deduplication — Remembering What You've Already Seen

The standard technique for making an inherently non-idempotent operation idempotent is deduplication. The idea is simple: attach a unique identifier to each message (or request), have the receiving side remember a "list of already-processed identifiers," and ignore any identifier that arrives again.

  Attach a unique key to the request:  idempotency-key: "abc-123"

  Receiving-side processing:
    IF "abc-123" has already been processed?
      YES -> do nothing, return the previous result (absorb the duplicate)
      NO  -> process it, and record "abc-123 processed"

The idempotency key you often see in payment APIs is exactly this. When the client sends a payment request with a unique key, the server uses that key to detect duplicates. Even if a network problem makes the client retransmit the same request, the server sees the same key, thinks "ah, this was already processed," and returns the previous result rather than creating a new charge.

There's a subtle but decisive point here. The "record that it was processed" and the "actual processing" must happen atomically together. If it processes but dies just before recording "processed," the next retry processes it again. Conversely, if it records but dies before processing, the processing is lost. So deduplication must be bundled with processing into one transaction — and this is where the third ingredient appears.

The Transactional Outbox — Bundling Processing and Publishing

The outbox pattern covered in the previous post plays a key role again here. In exactly-once processing, the outbox atomically bundles two things: the "result of processing a message (state change)" and the "record that this processing happened (plus, if needed, the next message to publish)."

A typical consumer's exactly-once processing flow looks like this:

  Receive a message (the message has a unique id)
    |
    v
  Within one local transaction:
    BEGIN
      IF this message id is already in the processed table -> do nothing, end
      business processing (e.g., state change)
      INSERT processed_message_id           -- the record that we "saw" it
      INSERT INTO outbox (next event)        -- the next message, if needed
    COMMIT   -- processing, record, and publish all together, or none

The power of this structure is that the "record that it was processed" and the "actual processing" live in the same transaction of the same database, so they can never drift apart. If the transaction commits, all three take effect; if it fails, all three never happened. It stays consistent even if it dies in the middle. And publishing is still at-least-once (that's how the outbox works), but if the next consumer also has the same idempotency/deduplication structure, the whole chain maintains "exactly-once effect" at each stage.

To summarize, exactly-once processing is a composition of three ingredients:

IngredientRole
at-least-once deliveryprevents loss (tolerates duplicates)
idempotency / deduplicationmakes duplicates harmless
transactional outboxatomically bundles processing and record

What Kafka Exactly-Once Actually Guarantees

At this point, a common question: "But doesn't Kafka support exactly-once?" It does. You just need to know precisely what that means.

Kafka's exactly-once semantics (EOS) did not beat the laws of physics by magic. It implements the very principles above at the protocol level. Two axes are key.

  • Idempotent producer: the producer attaches a sequence number to each message, and the broker checks it to eliminate duplicate writes caused by retransmission at the broker-log level. So even if the producer retransmits due to a network problem, the log retains it only once.
  • Transactions: writes across multiple partitions, and the consumer offset commit, are bundled into one atomic transaction. This makes the chain "consume a message → process → produce results → commit the offset" all finalize together or all cancel together.

There's a decisive condition here. Kafka's EOS truly holds within the closed world of "read from Kafka, write to Kafka" — because the processing, offset commit, and output all go into the Kafka transaction. But the moment the processing produces a side effect on an external system (e.g., calling a payment gateway, sending an email, writing to another DB), that external effect is outside the Kafka transaction.

  Where Kafka EOS is complete:
    read Kafka -> process -> write Kafka (+offset)  <- all one transaction

  Where EOS does not automatically cover:
    read Kafka -> call external payment API  <- this call is outside Kafka!
    -> here you still need an idempotency key / deduplication

So even Kafka's exactly-once, the moment there's an external side effect, ultimately needs the idempotency and deduplication we saw earlier to seal that boundary. Kafka EOS is powerful but not omnipotent — it is powerful within the condition of "closed stream processing." Misunderstand this condition and you fall into the dangerous illusion that "since we use Kafka, duplicate worries are over."

In Practice, in Payment Systems

Let's apply these principles to the most sensitive domain: payments. Because in payments a duplicate is literally money, the principles of exactly-once processing matter especially.

  • Require an idempotency key on every payment request. Have the client generate a unique key (often based on the order id) and include it in the payment request, and have the server block duplicate charges by that key. It's the only robust way to prevent a double charge when the client retries after a network timeout.
  • Bundle the "record of charging" with the "charge" atomically. The external payment gateway call is outside the transaction, so it's subtle. A common pattern is to record "request started" first (pending), finalize the state on the gateway's response, and use the idempotency key to detect retries in between.
  • Make webhook receipt idempotent too. The payment-completed webhook the gateway sends is usually at-least-once, so the same event can arrive several times. Webhook processing must also deduplicate based on the event id.
  • Have a reconciliation process. No matter how well designed, distributed systems can drift. A reconciliation procedure that periodically compares your records against the payment provider's records to catch discrepancies is the last safety net.

The core message is this. In payments, "exactly-once" is not something the network magically guarantees — it is a property we create through idempotency keys, deduplication, and reconciliation.

Practical Guidance

Let's compress everything.

Be wary of any system that promises "exactly-once delivery." Pure exactly-once delivery is fundamentally impossible. Such a promise is usually calling "at-least-once + deduplication" by that name, or holds only within a specific closed condition. Always check the condition.

Accept at-least-once as the default. Loss is usually more catastrophic than duplication. So reliable systems choose at-least-once and solve the remaining duplication problem at the processing layer.

Design operations to be idempotent as much as possible. "Set" beats "increment/decrement." If something is inherently non-idempotent, deduplicate with a unique identifier.

Bundle deduplication and the actual processing into one transaction. If the "record that it was seen" and the "processing" drift apart, exactly-once collapses. The outbox pattern gives you this atomicity.

Have reconciliation as the last safety net. Especially where duplicates are catastrophic, like payments, a procedure that catches discrepancies through periodic comparison is essential.

Wrapping Up

The answer to "is exactly-once an illusion?" is "yes and no." Exactly-once delivery is an illusion. As long as the network can disconnect and failures are indistinguishable, certain agreement over an unreliable channel is impossible, just as the Two Generals Problem says. But exactly-once processing is not an illusion. Accept at-least-once delivery, make duplicates harmless with idempotency and deduplication, and atomically bundle processing and record with a transactional outbox, and you really can build a system whose final effect is exactly once.

Even Kafka's exactly-once stands on these principles. It implements these concepts with an idempotent producer and transactions within closed stream processing, and the moment an external side effect enters, we must again guard the boundary with idempotency and deduplication.

So the practical wisdom is not to strive to "be delivered exactly once," but to design so that "even if it arrives many times, only one effect remains." This shift in perspective is precisely the key to making correctness in payments and messaging an engineering discipline rather than an illusion.

References

Discuss on Twitter • View on GitHub