💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

원문 렌더가 준비되기 전까지 텍스트 가이드로 표시합니다.

Introduction — Why We Need Queues

As a service grows, the amount of work that "does not have to happen right now" grows too. Sending a welcome email after signup, encoding an uploaded video, aggregating settlements after a payment. If you do these synchronously during request handling, the user waits forever. So we put the work on a queue and process it later, in the background.

A message queue is that idea turned into a system. A producer puts messages in, a consumer takes them out and processes them. With a queue in between, production and consumption are decoupled, speed differences are absorbed (buffering), and even if a consumer dies the message survives for reprocessing (reliability).

The catch is that under the single phrase "message queue" live tools with fairly different personalities. Kafka, RabbitMQ, and Redis are all called queues, but their design philosophies differ. This post contrasts them one by one, and finally covers Python asyncio, which is often confused with queues.

If you want to see these ideas in motion, you can visualize them interactively in the [Message Queue Playground](/tools/message-queue-playground) newly built on this site.

Starting with a Plain FIFO Queue

The simplest queue is a first-in-first-out (FIFO) data structure. What goes in first comes out first. It is the same queue that ships in a language's standard library.

from collections import deque

q = deque()

q.append("job-1") # enqueue

q.append("job-2")

print(q.popleft()) # dequeue -> job-1

print(q.popleft()) # dequeue -> job-2

This queue has three key properties. Order is preserved, one message goes to exactly one consumer, and once taken out it is gone. The concept is clear, and so is the limit. This `deque` lives only in one process's memory. If the process dies, the queue dies with it, and you cannot share it with a consumer on another server.

Distributed message queue systems exist precisely to cross that limit. They move the queue across the network, store it on disk, and let many producers and consumers attach. But each does it differently.

Kafka — An Unerasable Log

Kafka is not really a "queue" in the traditional sense. Kafka is an **append-only log**. Rather than pulling a message off a queue and destroying it, it keeps appending to the end of a log file, and the consumer only remembers "how far I have read."

Here are the core concepts.

- **Topics and partitions**: A topic is a category of messages, and each topic is split into several partitions. The partition is the unit of parallelism.

- **Offsets**: The sequence number of a message within a partition. A consumer stores its offset and can, if it wants, rewind to an earlier offset and **replay**.

- **Consumer groups**: Consumers in the same group split the partitions among themselves. A partition is assigned to only one consumer within a group. So adding consumers raises throughput, but once the number of consumers exceeds the number of partitions, the extra consumers sit idle.

- **Per-partition ordering**: Order is guaranteed only within a partition. There is no global ordering across the whole topic.

Visually:

topic: orders

┌─────────────────────────────────────────┐

│ partition 0: [o0][o1][o2][o3] ... <- consumer A

│ partition 1: [o0][o1][o2] ... <- consumer B

│ partition 2: [o0][o1][o2][o3][o4] <- consumer C

└─────────────────────────────────────────┘

each consumer remembers its own offset

Kafka is powerful because the log stays. Since a message is not destroyed the moment it is consumed, a new consumer can attach later and read from the beginning. It fits event sourcing, stream processing, and setups where several systems each consume the same events. In return, you must always keep in mind that ordering is per-partition, and that the same key must go to the same partition for its order to hold.

RabbitMQ — Smart Routing

RabbitMQ is a traditional message broker implementing the AMQP protocol. If Kafka is a log, RabbitMQ is a **smart post office**. The key is that the producer does not put messages into a queue directly. The producer sends to an **exchange**, and the exchange **routes** the message to queues by rules.

The routing behavior is set by the exchange type.

- **direct**: sends to queues whose routing key matches exactly.

- **fanout**: ignores the routing key and copies to every bound queue. This is a broadcast.

- **topic**: matches the routing key as a pattern. For example, patterns like `order.*` or `order.#` produce partial matches.

The relationship among producer, exchange, binding, queue, and consumer:

producer --> [exchange] --binding (routing key)--> [queue] --> consumer

│

├── direct : exact key match

├── fanout : copy to all

└── topic : pattern match (order.*, order.#)

Another core piece of RabbitMQ is the **acknowledgment (ack)**. The queue deletes a message only after the consumer processes it and sends an ack. If the consumer dies mid-processing and cannot send an ack, the broker redelivers the message to another consumer. That prevents "loss during processing."

The decisive difference from Kafka is that a message disappears after consumption. RabbitMQ is not built for rewinding and replaying past messages. Instead, it excels at fine-grained message flow control: complex routing, work distribution, priority queues, and delayed queues.

Redis — Light and Versatile, in Three Ways

Redis is originally an in-memory data store, but it is widely used for messaging too. The interesting part is that Redis offers three different messaging styles.

**1. List-based queue (LPUSH / BRPOP).** The simplest work queue. One side pushes onto a list, the other pops. `BRPOP` blocks, so a consumer waits when there is no message.

producer

LPUSH tasks "job-1"

LPUSH tasks "job-2"

consumer (wait up to 5 seconds for a message)

BRPOP tasks 5

**2. Pub/Sub (publish/subscribe).** When you publish to a channel, it is delivered at that moment to all subscribers currently listening. But this is **fire-and-forget**. If there is no subscriber, the message simply vanishes; it is not stored. It fits sending "only to whoever is listening now," like real-time notifications.

subscriber

SUBSCRIBE news

publisher (delivered to all subscribers, not stored)

PUBLISH news "hello"

**3. Streams.** Redis Streams splits the difference between the simplicity of lists and the durability and consumer groups of Kafka. Messages accumulate like a log, are consumed split across a consumer group, and support acknowledgment (ack). It is handy when you want a Kafka-like pattern at a scale that is not Kafka-sized.

append to a stream

XADD mystream * event "signup" user "alice"

create a consumer group, then read

XGROUP CREATE mystream g1 0

XREADGROUP GROUP g1 consumer1 COUNT 1 STREAMS mystream >

Redis's appeal is the lightness of bolting a queue onto "the same Redis you already use as a cache." That said, the three styles have different properties, so choose by being clear about whether loss is acceptable (Pub/Sub) or not (lists, Streams).

When to Use Which — Comparison Table

Here is everything so far in one table.

| --- | --- | --- | --- | --- |

A quick selection guide:

- For a temporary buffer inside one process, a **standard-library queue** is enough.

- When events must be retained for a long time and multiple consumers need to replay and reprocess, use **Kafka**.

- When you need complex routing, work distribution, and solid ack-based processing, use **RabbitMQ**.

- When you already run Redis and want a lightweight queue or real-time notifications, **Redis** is practical.

Async Is Not a Queue — Python asyncio

Here is a common misconception to clear up. "Asynchronous processing" and "message queue" often appear together, but they are not the same thing. A message queue is infrastructure that spans processes and servers; asyncio is a programming model for handling concurrency within a single process.

The core of Python asyncio is a **single-threaded event loop**. It does not add threads; one thread moves back and forth between many tasks. The trick is I/O wait time. While waiting for a network response or a disk, it pauses that task (`await`) and makes progress on another in the meantime.

async def fetch(name, delay):

print(f"start {name}")

await asyncio.sleep(delay) # another coroutine runs during this

print(f"done {name}")

return name

async def main():

run three tasks together (closer to the max wait, not the sum)

results = await asyncio.gather(

fetch("a", 2),

fetch("b", 1),

fetch("c", 3),

)

print(results)

asyncio.run(main())

Let us pin down a few terms.

- **Coroutine**: a function defined with `async def`. It can pause partway and resume.

- **await**: the point that says "wait here, but yield so the event loop can do other work in the meantime."

- **gather**: schedules several coroutines together so they progress concurrently.

The most important distinction is **concurrency versus parallelism**. asyncio gives concurrency. Multiple tasks make overlapping progress, but only one thread actually runs at any given instant. Parallelism, by contrast, is truly running at the same time across multiple CPU cores.

What this means in practice is clear. asyncio is strong for **I/O-bound** work (waiting on network, disk, DB). The more time spent waiting, the bigger the win. But it does nothing for **CPU-bound** work (heavy computation). During computation there is no chance to yield, so the event loop is blocked. CPU-bound work needs multiprocessing or a separate worker.

Queues and asyncio Are Used Together

Being different does not make them unrelated. They are often used together. A typical shape is this. When a web server receives a request, it does not do the heavy work itself; it puts a message on a queue. A separate worker process pulls work off the queue, and inside that worker it handles many messages concurrently with asyncio.

request --> web server --(message)--> [queue: Kafka/RabbitMQ/Redis]

worker process

└ handles many messages

concurrently with asyncio

In other words, the queue is the layer that "hands work off to later, to another process," and asyncio is the layer that "uses wait time efficiently within one process." They sit at different layers, so they complement rather than compete.

Common Pitfalls

Finally, a few points that often trip people up in practice.

- **The exactly-once illusion**: Most queues guarantee "at-least-once" by default. That means the same message can arrive twice. So consumers should be designed to be **idempotent**, producing the same result even when the same message is processed multiple times.

- **Overtrusting order**: Kafka's ordering is per-partition. If you need global order, you either keep a single partition or design keys carefully, which sacrifices parallelism.

- **Expecting durability from Pub/Sub**: Redis Pub/Sub does not store. A missed message is gone. If loss is a problem, use lists, Streams, or a different broker.

- **Blocking calls inside asyncio**: Calling a synchronous blocking function (like a plain `time.sleep` or a blocking DB driver) inside the event loop stalls the whole loop. Use async-capable libraries or hand it off to a separate thread.

- **Ignoring dead letters**: Retrying a perpetually failing message forever clogs the queue. It is safer to have a dead-letter queue that collects failed messages separately.

Wrapping Up

A message queue is not one concept but a spectrum. It starts with a simple FIFO queue inside a process, and runs through an unerasable log (Kafka), smart routing (RabbitMQ), and a lightweight, versatile tool (Redis). And asyncio, different in nature but often used alongside them, is a concurrency model that efficiently handles wait time within a single process.

The key is not "which is correct" but "which fits this problem." Need replay, use Kafka; need routing, use RabbitMQ; need lightness, use Redis; need in-process concurrency, use asyncio. Keep that mapping in your head and most choices get easy.

If you want to move the concepts yourself, visualize how each style behaves differently in the [Message Queue Playground](/tools/message-queue-playground).

References

- Apache Kafka documentation: https://kafka.apache.org/documentation/

- RabbitMQ tutorials: https://www.rabbitmq.com/tutorials

- Redis Streams introduction: https://redis.io/docs/latest/develop/data-types/streams/

- Python asyncio documentation: https://docs.python.org/3/library/asyncio.html

- AMQP 0-9-1 concepts: https://www.rabbitmq.com/tutorials/amqp-concepts