필사 모드: Streaming vs Batch Redefined: Flink, RisingWave, Materialize, CDC, Streaming SQL and the Pragmatism of Real Time (2025)

Season 5 Ep 2 — If Ep 1 was "where data is stored", Ep 2 is "how fast data flows". The real-time frenzy of 2020–2023, and the return to pragmatism in 2024–2025.

• Prologue — "Real time is an option, not a default"
• Chapter 1 · Freshness Tiers
  • 1.1 A Five-tier Framework
  • 1.2 Mapping Each Metric and Table to a Tier
  • 1.3 Decision Principles
• Chapter 2 · The 2025 Version of Lambda and Kappa
  • 2.1 Lambda (2014)
  • 2.2 Kappa (2014, Jay Kreps)
  • 2.3 2025: Unified on Lakehouse
  • 2.4 The "Streaming + Materialized" Pattern
• Chapter 3 · Comparing the Four Major Streaming Engines
  • 3.1 Apache Flink
  • 3.2 Spark Structured Streaming
  • 3.3 Kafka Streams / ksqlDB
  • 3.4 RisingWave
  • 3.5 Materialize
  • 3.6 Comparison Table
  • 3.7 Selection Guide
• Chapter 4 · CDC (Change Data Capture)
  • 4.1 Why CDC Is Central
  • 4.2 Implementation Methods
  • 4.3 Tools
  • 4.4 The CDC → Iceberg Pattern
  • 4.5 Practical Traps
• Chapter 5 · Iceberg v3 and Real-time Upsert
  • 5.1 Iceberg Version History
  • 5.2 Two Kinds of Row-level Delete
  • 5.3 The Real-time Upsert Workflow
  • 5.4 Performance Cautions
• Chapter 6 · The Rise of Streaming SQL
  • 6.1 Why SQL
  • 6.2 Flink SQL
  • 6.3 ksqlDB
  • 6.4 The Postgres Compatibility of RisingWave
  • 6.5 Materialize
• Chapter 7 · The Cost and Latency Trade-off
  • 7.1 Cost Components
  • 7.2 Representative Cost Comparison (monthly, mid-size scale)
  • 7.3 Cost Reduction Techniques
  • 7.4 Recommended Architecture by Latency Target
• Chapter 8 · Observability and Debugging
  • 8.1 Key Metrics
  • 8.2 Observability Tools
  • 8.3 Debugging
  • 8.4 Alerting
• Chapter 9 · Failure, Recovery and SLA
  • 9.1 Designing the SLA
  • 9.2 Recovery Strategy
  • 9.3 Reprocessing
  • 9.4 Multi-region
• Chapter 10 · Practical Patterns for Streaming plus Lakehouse
  • 10.1 Streaming on Medallion
  • 10.2 CDC → Silver
  • 10.3 Event Sourcing
  • 10.4 Real-time Feature Store
  • 10.5 Streaming ETL Pipeline
• Chapter 11 · Three Real-world Cases
  • 11.1 An E-commerce Order Pipeline
  • 11.2 Financial Transaction Monitoring
  • 11.3 Game Telemetry
• Chapter 12 · Streaming at Korean Companies
  • 12.1 Traditional Patterns
  • 12.2 Recent Trends
  • 12.3 Regulatory Considerations
  • 12.4 Obstacles
• Chapter 13 · Ten Anti-patterns
  • 13.1 "Everything in Real Time"
  • 13.2 Blind Faith in Exactly-once
  • 13.3 Skipping the Initial CDC Snapshot
  • 13.4 No Automatic Propagation of Schema Changes
  • 13.5 Kafka Retention Too Short
  • 13.6 Checkpoint Interval Too Long
  • 13.7 Keeping State Without Limits
  • 13.8 No Compaction of Delete Files
  • 13.9 Under-allocating Memory and CPU
  • 13.10 No Observability or Alerting
• Chapter 14 · Checklist — Twelve Points Before Launching Streaming
• Chapter 15 · Next Up — Season 5 Ep 3: "Comparing OLAP Engines in 2025"

Prologue — "Real time is an option, not a default"

Every keynote at a data conference between 2019 and 2022 said "all data must become real time". The reality of 2025 is different:

• Real-time pipelines are 3–10x more expensive than batch
• Most BI dashboards run 15 minutes to 1 hour behind and nobody complains
• What genuinely needs real time is 5–15% of all data

The right answer in 2025:

"Divide data into freshness tiers based on the SLA of each dataset and metric, and use the tool that fits each tier."

This post makes those tiers and tool choices concrete.

Chapter 1 · Freshness Tiers

1.1 A Five-tier Framework

1.2 Mapping Each Metric and Table to a Tier

• Inventory metrics: Fresh
• Login anomaly detection: Real-time
• Monthly revenue: Daily
• Features for model training: a mix of Fresh and Daily

1.3 Decision Principles

• The business question: "what happens if the metric is an hour late?"
• If the answer is "nothing at all" → Daily or Fresh
• If the answer is "a loss of 100,000 won" → Near-real-time
• If the answer is "someone gets hurt or we have a legal problem" → Real-time

Chapter 2 · The 2025 Version of Lambda and Kappa

2.1 Lambda (2014)

• Batch layer + Speed layer + Serving layer
• The same logic implemented twice, a maintenance burden
• Dominant from 2015 to 2020

2.2 Kappa (2014, Jay Kreps)

• Unified on a single streaming log (Kafka)
• Reprocessing happens through streaming too
• Theoretically simple, practically hard

2.3 2025: Unified on Lakehouse

• On top of Iceberg/Delta tables, batch and streaming write at the same time
• Flink + Iceberg, Spark Structured Streaming + Delta
• Serving is an OLAP engine or a materialized view

2.4 The "Streaming + Materialized" Pattern

• Source → streaming pipeline → Lakehouse
• Precompute frequently used queries as materialized views
• Ad-hoc queries hit the OLAP engine directly

Chapter 3 · Comparing the Four Major Streaming Engines

3.1 Apache Flink

• Genuinely based on event time, with watermarks and exactly-once
• Rich keyed state, proven in large-scale production
• Steep learning curve, complex to operate
• Flink CDC 2.x and Flink SQL matured in 2024–2025

3.2 Spark Structured Streaming

• Micro-batch streaming in the Spark ecosystem
• The Databricks standard
• Latency target of 1–60 seconds or more, not truly millisecond class
• SQL and Python make it easy

3.3 Kafka Streams / ksqlDB

• A library built into Kafka, JVM based
• Lightweight, optimal when you concentrate on data already in Kafka
• Distributed operation is something you implement yourself

3.4 RisingWave

• Open sourced in 2022, specialized in Streaming SQL
• Postgres-compatible SQL plus materialized views
• State lives in separate storage (S3), which keeps operations simple
• Rising fast as an alternative to Flink

3.5 Materialize

• Streaming plus incremental view maintenance
• Complex joins and aggregates are refreshed automatically
• Centered on enterprise SaaS

3.6 Comparison Table

3.7 Selection Guide

• Event time and complex state: Flink
• The Databricks ecosystem: Spark SS
• Kafka centric with simple logic: Kafka Streams / ksqlDB
• Streaming apps in SQL alone: RisingWave or Materialize

Chapter 4 · CDC (Change Data Capture)

4.1 Why CDC Is Central

• Replicates in real time the changes in an operational DB (Postgres/MySQL) into the analytics system
• Dramatically lower latency and load than a batch dump
• Keeps the Silver layer of the Lakehouse up to date automatically

4.2 Implementation Methods

• Log-based: Postgres WAL, MySQL binlog — the most efficient
• Trigger-based: DB triggers generate change events — heavy load on the DB
• Query-based: polling on an updated_at column — easy, but it cannot catch deletes

4.3 Tools

• Debezium (OSS): Postgres/MySQL/SQL Server/Oracle/Mongo → Kafka
• Fivetran, Airbyte: managed ELT, hundreds of source connectors
• Striim, HVR: enterprise CDC
• Flink CDC: Flink integration on top of Debezium

4.4 The CDC → Iceberg Pattern

Postgres → Debezium → Kafka → Flink → Iceberg

• Kafka retains the events, so reprocessing is possible
• Flink upserts into Iceberg with MERGE INTO
• Uses Iceberg v2/v3 row-level delete

4.5 Practical Traps

• Handling schema changes (adding or dropping columns, type changes)
• Harmonizing the initial snapshot with incremental snapshots
• Guaranteeing exactly-once (preventing both duplicates and gaps)
• Backfill and reprocessing strategy

Chapter 5 · Iceberg v3 and Real-time Upsert

5.1 Iceberg Version History

• v1: Append-only
• v2: Row-level delete (position/equality), MERGE INTO
• v3 (2024–2025): Deletion vectors, row lineage, V3 partition transforms

5.2 Two Kinds of Row-level Delete

• Position delete: a delete file that points at the file and position of a row
• Equality delete: condition-based deletion (useful for implementing UPDATE)

5.3 The Real-time Upsert Workflow

1. Flink reads the CDC events
2. Generates an equality delete plus an insert based on the PK
3. Iceberg reflects it in a snapshot
4. Periodic compaction cleans up the delete files

5.4 Performance Cautions

• Read performance degrades as delete files pile up
• The compaction interval matters (minutes to hours)
• Running a background job that converts to Copy-on-Write alongside it is recommended

Chapter 6 · The Rise of Streaming SQL

6.1 Why SQL

• The Flink Java API is hard to learn
• A common language that data engineers and analysts can both handle
• Easy to integrate with tools such as dbt and Dagster

6.2 Flink SQL

• Stabilized in 2023–2024
• Event time, windows and state all expressed in SQL
• UDFs and UDAFs are possible

6.3 ksqlDB

• Streaming SQL on top of Kafka
• Clear Table and Stream concepts
• Suited to simple ETL and aggregation

6.4 The Postgres Compatibility of RisingWave

• CREATE MATERIALIZED VIEW → kept up to date in real time
• Postgres tools (Grafana, Superset and so on) work as they are
• Low operational complexity, hence rapid growth in 2025

6.5 Materialize

• Based on Incremental View Maintenance (IVM) research
• Even complex joins are refreshed automatically
• SaaS plus a self-hosted option

Chapter 7 · The Cost and Latency Trade-off

7.1 Cost Components

• Compute (a streaming cluster running 24/7)
• State storage (RocksDB/S3)
• Running Kafka/MSK
• Network (cross-region)

7.2 Representative Cost Comparison (monthly, mid-size scale)

7.3 Cost Reduction Techniques

• Keep the real-time layer thin (core metrics only)
• Check whether the near-real-time layer can take its place
• Scale down in off-peak periods
• The trade-off between state on S3 (the RisingWave way) and local RocksDB (Flink)

7.4 Recommended Architecture by Latency Target

• Under 100ms: in-memory stream processor + Redis/RocksDB
• Seconds: Flink/RisingWave + Kafka
• Minutes: Spark Structured Streaming + Delta/Iceberg
• Hours: micro-batch Airflow + dbt
• Days: batch Spark

Chapter 8 · Observability and Debugging

8.1 Key Metrics

• End-to-end latency (from the event occurring to the result being reflected)
• Lag (Kafka consumer lag, Flink checkpoint lag)
• Throughput (eps, rps)
• Back-pressure (per Flink task)
• State size (RocksDB size, checkpoint size)

8.2 Observability Tools

• Flink UI + Prometheus + Grafana
• Kafka Lag Exporter, Burrow, Conduktor
• Streaming pipeline lineage with OpenLineage
• Datadog/NewRelic APM plus streaming extensions

8.3 Debugging

• Reprocessing based on checkpoints and savepoints
• Debugging CEP (Complex Event Processing) rules
• Tracing events with sampled logs
• Tests run on MiniCluster plus embedded Kafka

8.4 Alerting

• Lag over the threshold → Slack/PagerDuty
• End-to-end latency SLO violation
• Consecutive checkpoint failures

Chapter 9 · Failure, Recovery and SLA

9.1 Designing the SLA

• Availability: 99.9% = 43 minutes of downtime allowed per month
• Freshness: queryable within X seconds of the event occurring
• Correctness: eventual accuracy vs exactly-once

9.2 Recovery Strategy

• Flink: checkpoints (scheduled), savepoints (manual)
• Kafka: Replication factor 3, min.insync.replicas 2
• Iceberg: roll back with snapshots and branches

9.3 Reprocessing

• Keep Kafka retention as long as the reprocessing window (7–30 days is common)
• Or replay the streaming job from the Iceberg source
• Prevent duplicates during reprocessing (idempotent design)

9.4 Multi-region

• Kafka MirrorMaker 2 / Confluent Cluster Linking
• Iceberg relies on storage replication (S3 Cross-region)
• Active-Passive is the norm for streaming jobs

Chapter 10 · Practical Patterns for Streaming plus Lakehouse

10.1 Streaming on Medallion

• Bronze: append the raw events
• Silver: cleansing, deduplication, join upserts
• Gold: aggregates and metrics

10.2 CDC → Silver

• DB change events → Flink → Iceberg Silver upsert
• Silver plays the role of "a copy plus history" of the operational DB
• Gold is produced by dbt/Spark batch

10.3 Event Sourcing

• Keep domain events in Kafka permanently
• Reconstruct state by replaying events
• Long-term retention in Iceberg Bronze

10.4 Real-time Feature Store

• Feast/Tecton + Kafka + Iceberg
• Online features (Redis) plus offline features (Iceberg)
• Monitoring online-offline skew

10.5 Streaming ETL Pipeline

• Source → Bronze (append) → Silver (cleansing) → Gold (aggregation)
• Each stage implemented as a Flink/Spark SS job
• Reprocessing only requires resetting the upstream offset

Chapter 11 · Three Real-world Cases

11.1 An E-commerce Order Pipeline

• Order events into Kafka → Flink → inventory update plus anomaly detection plus analytics
• Inventory: real time at second granularity
• Revenue metrics: 5-minute near-real-time
• Monthly reports: daily batch

11.2 Financial Transaction Monitoring

• Transaction events into Kafka → Flink CEP → fraud score
• Requires latency under 100ms
• State: per-user transaction history (Flink keyed state)
• Results: stored in Redis plus Iceberg

11.3 Game Telemetry

• Client events into Kinesis/Kafka → Flink/Spark SS
• Real-time analysis (concurrent users, DAU) runs near-real-time
• Detailed logs pile up in Iceberg Bronze for later analysis
• A/B tests: Gold aggregates

Chapter 12 · Streaming at Korean Companies

12.1 Traditional Patterns

• Finance: a mix of Tibco EMS, IBM MQ and Kafka
• Telecom: real-time streaming for charging and billing
• Gaming: Kafka + Flink or Kafka Streams
• Commerce: Kafka + Spark SS + ELK

12.2 Recent Trends

• Growing adoption of Confluent Cloud and MSK
• Wider adoption of Flink (especially at Toss, Coupang, Naver and Kakao)
• RisingWave and Materialize are at the early adoption stage in 2024–2025

12.3 Regulatory Considerations

• Finance: an in-house Kafka cluster inside a network-separated environment
• Personal information: PII masking of CDC events is mandatory
• Audit logs: long-term retention and immutability

12.4 Obstacles

• A shortage of data engineers pushes preference toward managed services
• Coexisting with legacy DWs
• A 24/7 on-call culture is still taking root

Chapter 13 · Ten Anti-patterns

13.1 "Everything in Real Time"

Streaming even the tables you do not need, so cost and complexity explode.

13.2 Blind Faith in Exactly-once

Guaranteeing end-to-end exactly-once on both the source and the sink is not easy. Idempotent design is mandatory.

13.3 Skipping the Initial CDC Snapshot

Gaps appear and accuracy drops.

13.4 No Automatic Propagation of Schema Changes

Downstream pipelines break.

13.5 Kafka Retention Too Short

Reprocessing becomes impossible.

13.6 Checkpoint Interval Too Long

Recovery cost and the amount to reprocess explode when something fails.

13.7 Keeping State Without Limits

Flink keyed state grows without bound, leading to OOM.

13.8 No Compaction of Delete Files

Iceberg read performance degrades.

13.9 Under-allocating Memory and CPU

A chain reaction of back-pressure.

13.10 No Observability or Alerting

Customers discover the incident before you do.

Chapter 14 · Checklist — Twelve Points Before Launching Streaming

• Freshness tier mapping (SLA per table and metric)
• Rationale for the engine choice (Flink/Spark SS/RisingWave/Materialize)
• CDC source selection plus an initial snapshot strategy
• Kafka retention plus a reprocessing plan
• Iceberg/Delta table design plus automated compaction
• Checkpoint and savepoint policy
• Idempotency and duplicate handling design
• Observability (lag, latency, throughput, back-pressure)
• SLA/SLO definition plus alerting
• Cost dashboard (compute, storage, network)
• Disaster recovery and multi-region plan
• On-call and incident response playbook

Chapter 15 · Next Up — Season 5 Ep 3: "Comparing OLAP Engines in 2025"

Now that streaming and batch share the same storage, the next question is "who queries fastest on top of it?".

• DuckDB: the revolution of single-node OLAP
• ClickHouse: the standard for real-time OLAP
• Snowflake / BigQuery / Redshift: the managed giants
• Databricks SQL / StarRocks / Doris / Pinot / Druid
• Federated queries with Trino / Presto
• The traps in real benchmarks
• Cost vs latency vs operational burden
• The line between MPP and single node
• A selection guide for Korean companies
• Engine placement patterns for the right tool in the right place

Things get genuinely interesting only after you accept the 2025 reality that "a single engine cannot do everything".

See you in the next post.

Summary: Streaming in 2025 was redefined from "everything in real time" to "freshness tiers based on SLA". You place Flink, Spark SS, RisingWave, Materialize and ksqlDB across the five tiers of Real-time, Near-real-time, Fresh, Daily and Historical, use CDC to flow changes from the operational DB into the Lakehouse, and handle real-time upserts with Iceberg v3 row-level delete. "Unified on Lakehouse" rather than Lambda/Kappa is the dominant pattern, and the trade-offs of cost, latency and complexity are designed deliberately. "Real time is an option, not a default" — that is the pragmatism of 2025.