Streaming vs Batch Redefined: Flink, RisingWave, Materialize, CDC, Streaming SQL and the Pragmatism of Real Time (2025)

Season 5 Ep 2 — If Ep 1 was "where data is stored", Ep 2 is "how fast data flows". The real-time frenzy of 2020–2023, and the return to pragmatism in 2024–2025.

Prologue — "Real time is an option, not a default"
Chapter 1 · Freshness Tiers
Chapter 2 · The 2025 Version of Lambda and Kappa
Chapter 3 · Comparing the Four Major Streaming Engines
Chapter 4 · CDC (Change Data Capture)
Chapter 5 · Iceberg v3 and Real-time Upsert
Chapter 6 · The Rise of Streaming SQL
Chapter 7 · The Cost and Latency Trade-off
Chapter 8 · Observability and Debugging
Chapter 9 · Failure, Recovery and SLA
Chapter 10 · Practical Patterns for Streaming plus Lakehouse
Chapter 11 · Three Real-world Cases
Chapter 12 · Streaming at Korean Companies
Chapter 13 · Ten Anti-patterns
Chapter 14 · Checklist — Twelve Points Before Launching Streaming
Chapter 15 · Next Up — Season 5 Ep 3: "Comparing OLAP Engines in 2025"

Prologue — "Real time is an option, not a default"

Every keynote at a data conference between 2019 and 2022 said "all data must become real time". The reality of 2025 is different:

Real-time pipelines are 3–10x more expensive than batch
Most BI dashboards run 15 minutes to 1 hour behind and nobody complains
What genuinely needs real time is 5–15% of all data

The right answer in 2025:

"Divide data into freshness tiers based on the SLA of each dataset and metric, and use the tool that fits each tier."

This post makes those tiers and tool choices concrete.

Chapter 1 · Freshness Tiers

1.1 A Five-tier Framework

Tier	Latency	Example	Tools
Real-time	ms–seconds	Transaction monitoring, anomaly detection, fraud	Flink, Kafka Streams
Near-real-time	1–5 min	Operational dashboards, alerts	Flink, RisingWave, Materialize
Fresh	5–60 min	Inventory, ad optimization	Streaming append + rollup
Daily	24 hours	BI, reports	Spark, dbt, SQL warehouse
Historical	Weekly/monthly	Analysis, ML training	Batch yearly/monthly

1.2 Mapping Each Metric and Table to a Tier

Inventory metrics: Fresh
Login anomaly detection: Real-time
Monthly revenue: Daily
Features for model training: a mix of Fresh and Daily

1.3 Decision Principles

The business question: "what happens if the metric is an hour late?"
If the answer is "nothing at all" → Daily or Fresh
If the answer is "a loss of 100,000 won" → Near-real-time
If the answer is "someone gets hurt or we have a legal problem" → Real-time

Chapter 2 · The 2025 Version of Lambda and Kappa

2.1 Lambda (2014)

Batch layer + Speed layer + Serving layer
The same logic implemented twice, a maintenance burden
Dominant from 2015 to 2020

2.2 Kappa (2014, Jay Kreps)

Unified on a single streaming log (Kafka)
Reprocessing happens through streaming too
Theoretically simple, practically hard

2.3 2025: Unified on Lakehouse

On top of Iceberg/Delta tables, batch and streaming write at the same time
Flink + Iceberg, Spark Structured Streaming + Delta
Serving is an OLAP engine or a materialized view

2.4 The "Streaming + Materialized" Pattern

Source → streaming pipeline → Lakehouse
Precompute frequently used queries as materialized views
Ad-hoc queries hit the OLAP engine directly

Chapter 3 · Comparing the Four Major Streaming Engines

3.1 Apache Flink

Genuinely based on event time, with watermarks and exactly-once
Rich keyed state, proven in large-scale production
Steep learning curve, complex to operate
Flink CDC 2.x and Flink SQL matured in 2024–2025

3.2 Spark Structured Streaming

Micro-batch streaming in the Spark ecosystem
The Databricks standard
Latency target of 1–60 seconds or more, not truly millisecond class
SQL and Python make it easy

3.3 Kafka Streams / ksqlDB

A library built into Kafka, JVM based
Lightweight, optimal when you concentrate on data already in Kafka
Distributed operation is something you implement yourself

3.4 RisingWave

Open sourced in 2022, specialized in Streaming SQL
Postgres-compatible SQL plus materialized views
State lives in separate storage (S3), which keeps operations simple
Rising fast as an alternative to Flink

3.5 Materialize

Streaming plus incremental view maintenance
Complex joins and aggregates are refreshed automatically
Centered on enterprise SaaS

3.6 Comparison Table

Engine	Latency	Complexity	SQL	Use in Korea	Characteristics
Flink	ms–seconds	High	Yes	Common	Industry standard, strong state management
Spark SS	Seconds–minutes	Medium	Yes	Very common	Databricks friendly
Kafka Streams	Seconds	Medium	ksqlDB	Moderate	Built into Kafka
RisingWave	Seconds	Low	Postgres	Growing	Simple to operate, SaaS/OSS
Materialize	Seconds	Medium	Yes	Rare	Strong incremental views

3.7 Selection Guide

Event time and complex state: Flink
The Databricks ecosystem: Spark SS
Kafka centric with simple logic: Kafka Streams / ksqlDB
Streaming apps in SQL alone: RisingWave or Materialize

Chapter 4 · CDC (Change Data Capture)

4.1 Why CDC Is Central

Replicates in real time the changes in an operational DB (Postgres/MySQL) into the analytics system
Dramatically lower latency and load than a batch dump
Keeps the Silver layer of the Lakehouse up to date automatically

4.2 Implementation Methods

Log-based: Postgres WAL, MySQL binlog — the most efficient
Trigger-based: DB triggers generate change events — heavy load on the DB
Query-based: polling on an updated_at column — easy, but it cannot catch deletes

4.3 Tools

Debezium (OSS): Postgres/MySQL/SQL Server/Oracle/Mongo → Kafka
Fivetran, Airbyte: managed ELT, hundreds of source connectors
Striim, HVR: enterprise CDC
Flink CDC: Flink integration on top of Debezium

4.4 The CDC → Iceberg Pattern

Postgres → Debezium → Kafka → Flink → Iceberg

Kafka retains the events, so reprocessing is possible
Flink upserts into Iceberg with MERGE INTO
Uses Iceberg v2/v3 row-level delete

4.5 Practical Traps

Handling schema changes (adding or dropping columns, type changes)
Harmonizing the initial snapshot with incremental snapshots
Guaranteeing exactly-once (preventing both duplicates and gaps)
Backfill and reprocessing strategy

Chapter 5 · Iceberg v3 and Real-time Upsert

5.1 Iceberg Version History

v1: Append-only
v2: Row-level delete (position/equality), MERGE INTO
v3 (2024–2025): Deletion vectors, row lineage, V3 partition transforms

5.2 Two Kinds of Row-level Delete

Position delete: a delete file that points at the file and position of a row
Equality delete: condition-based deletion (useful for implementing UPDATE)

5.3 The Real-time Upsert Workflow

Flink reads the CDC events
Generates an equality delete plus an insert based on the PK
Iceberg reflects it in a snapshot
Periodic compaction cleans up the delete files

5.4 Performance Cautions

Read performance degrades as delete files pile up
The compaction interval matters (minutes to hours)
Running a background job that converts to Copy-on-Write alongside it is recommended

Chapter 6 · The Rise of Streaming SQL

6.1 Why SQL

The Flink Java API is hard to learn
A common language that data engineers and analysts can both handle
Easy to integrate with tools such as dbt and Dagster

6.2 Flink SQL

Stabilized in 2023–2024
Event time, windows and state all expressed in SQL
UDFs and UDAFs are possible

6.3 ksqlDB

Streaming SQL on top of Kafka
Clear Table and Stream concepts
Suited to simple ETL and aggregation

6.4 The Postgres Compatibility of RisingWave

CREATE MATERIALIZED VIEW → kept up to date in real time
Postgres tools (Grafana, Superset and so on) work as they are
Low operational complexity, hence rapid growth in 2025

6.5 Materialize

Based on Incremental View Maintenance (IVM) research
Even complex joins are refreshed automatically
SaaS plus a self-hosted option

Chapter 7 · The Cost and Latency Trade-off

7.1 Cost Components

Compute (a streaming cluster running 24/7)
State storage (RocksDB/S3)
Running Kafka/MSK
Network (cross-region)

7.2 Representative Cost Comparison (monthly, mid-size scale)

Option	Monthly cost	Latency
Batch (Airflow + Spark, daily)	Low ($1–5k)	24 hours
Micro-batch (5 min)	Medium ($3–10k)	5 min
Structured Streaming	Medium–high ($5–20k)	Seconds–minutes
Flink cluster	High ($10–30k+)	ms–seconds
Managed (RisingWave/Confluent)	Medium–high ($7–25k)	Seconds

7.3 Cost Reduction Techniques

Keep the real-time layer thin (core metrics only)
Check whether the near-real-time layer can take its place
Scale down in off-peak periods
The trade-off between state on S3 (the RisingWave way) and local RocksDB (Flink)

7.4 Recommended Architecture by Latency Target

Under 100ms: in-memory stream processor + Redis/RocksDB
Seconds: Flink/RisingWave + Kafka
Minutes: Spark Structured Streaming + Delta/Iceberg
Hours: micro-batch Airflow + dbt
Days: batch Spark

Chapter 8 · Observability and Debugging

8.1 Key Metrics

End-to-end latency (from the event occurring to the result being reflected)
Lag (Kafka consumer lag, Flink checkpoint lag)
Throughput (eps, rps)
Back-pressure (per Flink task)
State size (RocksDB size, checkpoint size)

8.2 Observability Tools

Flink UI + Prometheus + Grafana
Kafka Lag Exporter, Burrow, Conduktor
Streaming pipeline lineage with OpenLineage
Datadog/NewRelic APM plus streaming extensions

8.3 Debugging

Reprocessing based on checkpoints and savepoints
Debugging CEP (Complex Event Processing) rules
Tracing events with sampled logs
Tests run on MiniCluster plus embedded Kafka

8.4 Alerting

Lag over the threshold → Slack/PagerDuty
End-to-end latency SLO violation
Consecutive checkpoint failures

Chapter 9 · Failure, Recovery and SLA

9.1 Designing the SLA

Availability: 99.9% = 43 minutes of downtime allowed per month
Freshness: queryable within X seconds of the event occurring
Correctness: eventual accuracy vs exactly-once

9.2 Recovery Strategy

Flink: checkpoints (scheduled), savepoints (manual)
Kafka: Replication factor 3, min.insync.replicas 2
Iceberg: roll back with snapshots and branches

9.3 Reprocessing

Keep Kafka retention as long as the reprocessing window (7–30 days is common)
Or replay the streaming job from the Iceberg source
Prevent duplicates during reprocessing (idempotent design)

9.4 Multi-region

Kafka MirrorMaker 2 / Confluent Cluster Linking
Iceberg relies on storage replication (S3 Cross-region)
Active-Passive is the norm for streaming jobs

Chapter 10 · Practical Patterns for Streaming plus Lakehouse

10.1 Streaming on Medallion

Bronze: append the raw events
Silver: cleansing, deduplication, join upserts
Gold: aggregates and metrics

10.2 CDC → Silver

DB change events → Flink → Iceberg Silver upsert
Silver plays the role of "a copy plus history" of the operational DB
Gold is produced by dbt/Spark batch

10.3 Event Sourcing

Keep domain events in Kafka permanently
Reconstruct state by replaying events
Long-term retention in Iceberg Bronze

10.4 Real-time Feature Store

Feast/Tecton + Kafka + Iceberg
Online features (Redis) plus offline features (Iceberg)
Monitoring online-offline skew

10.5 Streaming ETL Pipeline

Source → Bronze (append) → Silver (cleansing) → Gold (aggregation)
Each stage implemented as a Flink/Spark SS job
Reprocessing only requires resetting the upstream offset

Chapter 11 · Three Real-world Cases

11.1 An E-commerce Order Pipeline

Order events into Kafka → Flink → inventory update plus anomaly detection plus analytics
Inventory: real time at second granularity
Revenue metrics: 5-minute near-real-time
Monthly reports: daily batch

11.2 Financial Transaction Monitoring

Transaction events into Kafka → Flink CEP → fraud score
Requires latency under 100ms
State: per-user transaction history (Flink keyed state)
Results: stored in Redis plus Iceberg

11.3 Game Telemetry

Client events into Kinesis/Kafka → Flink/Spark SS
Real-time analysis (concurrent users, DAU) runs near-real-time
Detailed logs pile up in Iceberg Bronze for later analysis
A/B tests: Gold aggregates

Chapter 12 · Streaming at Korean Companies

12.1 Traditional Patterns

Finance: a mix of Tibco EMS, IBM MQ and Kafka
Telecom: real-time streaming for charging and billing
Gaming: Kafka + Flink or Kafka Streams
Commerce: Kafka + Spark SS + ELK

12.2 Recent Trends

Growing adoption of Confluent Cloud and MSK
Wider adoption of Flink (especially at Toss, Coupang, Naver and Kakao)
RisingWave and Materialize are at the early adoption stage in 2024–2025

12.3 Regulatory Considerations

Finance: an in-house Kafka cluster inside a network-separated environment
Personal information: PII masking of CDC events is mandatory
Audit logs: long-term retention and immutability

12.4 Obstacles

A shortage of data engineers pushes preference toward managed services
Coexisting with legacy DWs
A 24/7 on-call culture is still taking root

Chapter 13 · Ten Anti-patterns

13.1 "Everything in Real Time"

Streaming even the tables you do not need, so cost and complexity explode.

13.2 Blind Faith in Exactly-once

Guaranteeing end-to-end exactly-once on both the source and the sink is not easy. Idempotent design is mandatory.

13.3 Skipping the Initial CDC Snapshot

Gaps appear and accuracy drops.

13.4 No Automatic Propagation of Schema Changes

Downstream pipelines break.

13.5 Kafka Retention Too Short

Reprocessing becomes impossible.

13.6 Checkpoint Interval Too Long

Recovery cost and the amount to reprocess explode when something fails.

13.7 Keeping State Without Limits

Flink keyed state grows without bound, leading to OOM.

13.8 No Compaction of Delete Files

Iceberg read performance degrades.

13.9 Under-allocating Memory and CPU

A chain reaction of back-pressure.

13.10 No Observability or Alerting

Customers discover the incident before you do.

Chapter 14 · Checklist — Twelve Points Before Launching Streaming

Chapter 15 · Next Up — Season 5 Ep 3: "Comparing OLAP Engines in 2025"

Now that streaming and batch share the same storage, the next question is "who queries fastest on top of it?".

DuckDB: the revolution of single-node OLAP
ClickHouse: the standard for real-time OLAP
Snowflake / BigQuery / Redshift: the managed giants
Databricks SQL / StarRocks / Doris / Pinot / Druid
Federated queries with Trino / Presto
The traps in real benchmarks
Cost vs latency vs operational burden
The line between MPP and single node
A selection guide for Korean companies
Engine placement patterns for the right tool in the right place

Things get genuinely interesting only after you accept the 2025 reality that "a single engine cannot do everything".

See you in the next post.

Summary: Streaming in 2025 was redefined from "everything in real time" to "freshness tiers based on SLA". You place Flink, Spark SS, RisingWave, Materialize and ksqlDB across the five tiers of Real-time, Near-real-time, Fresh, Daily and Historical, use CDC to flow changes from the operational DB into the Lakehouse, and handle real-time upserts with Iceberg v3 row-level delete. "Unified on Lakehouse" rather than Lambda/Kappa is the dominant pattern, and the trade-offs of cost, latency and complexity are designed deliberately. "Real time is an option, not a default" — that is the pragmatism of 2025.