Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Season 5 Ep 5 — If Ep 4 was the engineering of pipelines, Ep 5 is the 2025 edition of an old promise: "define the meaning of data exactly once."

Prologue — "Why Does Revenue Come Out Different on Every Team?"
Chapter 1 · A History of the Semantic Layer
Chapter 2 · The Anatomy of a Semantic Layer
Chapter 3 · dbt Semantic Layer / MetricFlow
Chapter 4 · Cube — "A Semantic Layer for Developers"
Chapter 5 · Looker·LookML — "The Enterprise Classic"
Chapter 6 · Other Major Tools
Chapter 7 · The Metrics Store Pattern
Chapter 8 · Headless BI
Chapter 9 · Reverse ETL — "Sending Data Back Into Operations"
Chapter 10 · AI·LLMs and the Semantic Layer
Chapter 11 · Three Real Data Activation Scenarios
Chapter 12 · Maturity at Korean Companies
Chapter 13 · Ten Antipatterns
Chapter 14 · Checklist — 12 Items for Semantic Layer·Reverse ETL
Chapter 15 · Next Up — Season 5 Ep 6: "The Fusion of Feature Store and Vector·Graph·Time-Series DBs"

Prologue — "Why Does Revenue Come Out Different on Every Team?"

A classic confusion that many companies lived through in 2015:

Finance: "Revenue this month is 1 billion won"
Marketing: "Revenue this month is 1.15 billion won"
Sales: "Revenue this month is 970 million won"

Same data, different numbers. The reasons:

How refunds are handled
Whether tax is included
The reference date for currency conversion
The point of recognition (at order time vs. at payment time)

Every team writes its own SQL, so the definitions drift apart. Semantic Layer is the attempt to solve this by gathering "metric definitions in one place."

Chapter 1 · A History of the Semantic Layer

1.1 First Generation: Definitions Inside the BI Tool

Business Objects, Cognos, MicroStrategy (1990s–2000s)
Definitions live inside each BI tool → vendor lock-in

1.2 Second Generation: Looker's LookML (2013–)

Metric definitions codified in LookML
Managed in Git, reusable
Downside: Looker only

1.3 Third Generation: Headless / Open Semantic Layer (2020–)

A separate layer independent of BI
dbt Semantic Layer / MetricFlow (2022)
Cube (2019–)
Transform(2021, → acquired by dbt in 2022)
AtScale(enterprise)

1.4 Why It Is Rising Again Now

Fragmentation of the data stack → worse metric duplication
AI and chatbots querying data → need for natural language → metric mapping
The rise of Headless BI

Chapter 2 · The Anatomy of a Semantic Layer

2.1 Core Entities

Entity: business objects (order, customer, product)
Dimension: attributes (country, category, date)
Measure: raw numeric values (amount, quantity)
Metric: measurable indicators (revenue, user count, conversion rate)

2.2 Relationships and Joins

Relationships between entities are declared
Join paths are generated automatically
The right joins run internally when a metric is queried

2.3 Filters and Parameters

Standardized filters for period, country, product, and the like
Per-user permissions (row-level security)
Comparison math (YoY, MoM) built in

2.4 Example (dbt Semantic Layer / MetricFlow)

semantic_models:
  - name: orders
    model: ref('fact_orders')
    entities:
      - name: order_id
        type: primary
      - name: customer_id
        type: foreign
    measures:
      - name: total_amount
        agg: sum
        expr: amount_krw
    dimensions:
      - name: order_date
        type: time
        type_params: {time_granularity: day}

metrics:
  - name: gross_revenue
    type: simple
    type_params:
      measure: total_amount

→ API call: metric('gross_revenue', filters=..., group_by=['order_date'])

Chapter 3 · dbt Semantic Layer / MetricFlow

3.1 Background

2021: Transform is founded (Nick Handel, formerly of the Airbnb experimentation team)
2022: dbt acquires Transform
MetricFlow becomes the query engine of the dbt Semantic Layer

3.2 Structure

Semantic models and metrics defined on top of dbt models
dbt Cloud provides the API service
BI tools, notebooks, and AI query metrics through the API

3.3 Strengths

dbt-friendly, extends an existing project
Open standards (GraphQL, JDBC)
Major BI integrations across 2024–2025 (Tableau, Hex, Mode, and others)

3.4 Limits

Depends on dbt Cloud (self-hosting options are still maturing)
Complex joins and measures are still easier in raw SQL

Chapter 4 · Cube — "A Semantic Layer for Developers"

4.1 Identity

Cube Dev, 2019
Open source plus Cube Cloud
APIs for data apps and Headless BI

4.2 Characteristics

Data models defined in YAML, JS, or Python
REST/GraphQL/SQL APIs
Suited to developers and product teams who build dashboards themselves

4.3 Example

cube(`Orders`, {
  sql: `SELECT * FROM fact_orders`,
  measures: {
    revenue: { sql: 'amount_krw', type: 'sum' },
    count: { type: 'count' }
  },
  dimensions: {
    orderDate: { sql: 'order_date', type: 'time' },
    country: { sql: 'country', type: 'string' }
  }
});

4.4 Where It Is Used

Embedded analytics (shipping dashboards to your customers)
Custom data apps
The metric-query backend behind an AI chatbot

Chapter 5 · Looker·LookML — "The Enterprise Classic"

5.1 Identity

Looker in 2013 → acquired by Google in 2019
LookML: a DSL for metric definitions
Google Cloud integration, paired with Looker Studio

5.2 Strengths

LookML is the most battle-tested in the industry
An enormous set of enterprise references
Natural language queries through the Gemini integration in 2024–2025

5.3 Limits

Hard to reuse LookML outside Looker
Cost and lock-in

5.4 The 2025 Strategy

Run the dbt Semantic Layer alongside LookML
For BigQuery-centric companies, Looker + Gemini is the default

Chapter 6 · Other Major Tools

6.1 AtScale

A specialist enterprise Semantic Layer
Large-scale MDX/XMLA compatibility (Excel, Power BI)
Strong in finance and insurance enterprises

6.2 Lightdash

Open source Headless BI + dbt Semantic
Drawing attention as a Looker alternative
Explosive growth across 2024–2025

6.3 Transform (dbt)

Absorbed into the dbt Semantic Layer after the acquisition

6.4 Metabase·Superset

Some Semantic Layer capability built into the BI tool itself
Expanding dbt Semantic Layer integration

6.5 Thought Spot

Natural language search BI
The front-runner in LLM and AI integration across 2024–2025

Chapter 7 · The Metrics Store Pattern

7.1 Concept

A single store for the company's metrics
Definition, version, owner, related dashboards
A "metric catalog"

7.2 Composition

Metric catalog: name, description, formula
Governance: approval, change, deprecation
API: queried by BI, notebooks, and AI
Lineage: which data a metric came from

7.3 Examples

Airbnb Minerva (internal system, published paper)
Uber uMetric
LinkedIn Datahub metrics
The dbt Semantic Layer standardizing this externally

7.4 Effects

Metrics that agree across teams
Audit and regulatory readiness
Higher accuracy for natural language AI questions

Chapter 8 · Headless BI

8.1 Identity

Query and metric engine ↔ frontend UI, separated
The frontend is free to be an app, a dashboard, or an AI channel
The Semantic Layer serves it through an API

8.2 Advantages

Multiple UIs (web, mobile, Slack, voice)
Easy migration off an existing BI tool
Straightforward AI integration

8.3 Tools

dbt Semantic Layer + Hex/Mode/Tableau
Cube + a custom web frontend
Lightdash + dbt
Metabase + dbt Semantic Layer

8.4 Limits

The cost of custom UX
Maintenance spread thin

Chapter 9 · Reverse ETL — "Sending Data Back Into Operations"

9.1 Why It Is Needed

Precious customer and behavioral data sits in the warehouse
It is not in Salesforce/HubSpot/Marketo
Engineers wire up the APIs → problems of time, cost, and reliability

→ Reverse ETL: syncing data from the warehouse into operational SaaS

9.2 Major Tools

Hightouch: 2020, the industry leader
Census: 2018, another leader
Grouparoo(now part of Airbyte)
RudderStack(CDP + Reverse ETL)
Workato(general iPaaS + data)

9.3 The Pattern

Build customer segment models with dbt
Reverse ETL reads from the warehouse and pushes to Salesforce/HubSpot/Intercom
Sales and marketing put the segments to work immediately

9.4 Cases

Inject customer LTV into Intercom → conversation prioritization
Inject a health score into Salesforce → churn response
Usage-based segments into Marketo → campaigns

9.5 Data Activation

Do not stop at analysis; trigger an operational action
Partly competing with and partly complementing a CDP (Customer Data Platform)
The finishing piece of the 2025 modern data stack

Chapter 10 · AI·LLMs and the Semantic Layer

10.1 Why AI Needs a Semantic Layer

When an LLM tries to turn a natural language question into SQL, it needs the metric definition
The answer to "what is our revenue?" differs from company to company
If an LLM reaches raw tables directly, hallucination and error become a risk

10.2 "Text-to-SQL" vs "Text-to-Metric"

Text-to-SQL: generates SQL from the company schema → risk of metric mismatch
Text-to-Metric: calls the Semantic Layer API → returns only defined metrics → safe

10.3 Cases

Natural language BI with dbt Semantic Layer + GPT/Claude/Gemini
Cube + a custom chatbot
ThoughtSpot AI and Looker Gemini, natively

10.4 Quality Control

Standardizing metric definitions is mandatory
Permissions and auditing managed centrally in the Semantic Layer
Blocking the AI from writing SQL itself is the safe choice

10.5 The 2025 Direction

Headless BI + AI = "every team talks to data in natural language"
The Semantic Layer becomes AI's safe data access layer

Chapter 11 · Three Real Data Activation Scenarios

11.1 Commerce — Personalized Recommendations

Warehouse: customer behavior and purchase history → recommendation scores from an ML model
Reverse ETL: scores → Braze/Iterable/Salesforce Marketing Cloud
Result: automated personalized push and email

11.2 B2B SaaS — Product-led Sales

Warehouse: product usage and feature adoption metrics
Reverse ETL: metrics → Salesforce
The AE instantly sees "which of my accounts is most likely to upgrade"

11.3 Fintech — Risk Scoring

Warehouse: transaction patterns and risk scores
Reverse ETL: scores → customer support tools (Zendesk/Intercom)
CS staff get context when handling VIP or at-risk customers

Chapter 12 · Maturity at Korean Companies

12.1 Current State

Large enterprises and IT: a mix of Looker, Tableau, and Superset; Semantic Layer adoption is early
Startups: dbt + Metabase/Superset is a common combination
Toss, Coupang, Danggeun, Kakao: cases of building an internal Metrics Store
Reverse ETL: sharp expansion of Hightouch and Census in 2024

12.2 The Adoption Pattern

dbt + documented standard metrics
Expose the metrics in Metabase/Superset
Experiment with dbt Semantic Layer / Cube
Wire operations up through Reverse ETL, starting with one team
Company-wide Metrics Store + AI connection

12.3 Considerations

Defining Korean-language business terms
Customization for tax, exchange rate, and accounting policy
The security and permission scheme (by rank and department)
Reflecting audit and regulatory requirements

Chapter 13 · Ten Antipatterns

13.1 Connecting AI Directly Without a Semantic Layer

Metric hallucination and collapse of consistency.

13.2 Depending on LookML Alone

Cut off from other BI and AI channels.

13.3 No Clear Metric Owner

A vacuum of responsibility for changes.

13.4 Changing Without Deprecation

Consumer dashboards break.

13.5 A Sprawl of Reverse ETL Paths

Dozens of sync pipelines out of the warehouse, unmanageable.

13.6 Expecting Real Time From Reverse ETL

Most run in minutes to hours; it is not "real time."

13.7 Running Without Metric Versions

Past reports cannot be reproduced.

13.8 Defining Metrics Only in Superset/Metabase

No independent Semantic Layer → fragmentation with other tools.

13.9 Weak Integrity Checks on Natural Language BI

Trusting AI answers as they come → bad decisions.

13.10 Every Team Computing Metrics With Its Own SQL

Back to square one.

Chapter 14 · Checklist — 12 Items for Semantic Layer·Reverse ETL

Chapter 15 · Next Up — Season 5 Ep 6: "The Fusion of Feature Store and Vector·Graph·Time-Series DBs"

If the Semantic Layer is the language layer of BI, the Feature Store is the language layer of ML. And in 2025 the data database is fusing vector, graph, and time series.

Redefining the Feature Store (Feast/Tecton)
Managing online-offline skew
Real-time feature pipelines
The vector DB landscape (Pinecone/Weaviate/Milvus/Chroma/LanceDB)
Storing multi-modal embeddings
Graph DBs (Neo4j/NebulaGraph/Dgraph) and knowledge graphs
Time-series DBs (InfluxDB/TimescaleDB/VictoriaMetrics)
The arrival of the 2025 "Unified DB"
The shock AI and ML workloads dealt to the DB landscape
What Korean companies are choosing

"Data over models, features over data" is the wisdom of ML in 2025.

See you in the next post.

Summary: The Semantic Layer is the 2025 edition of an old promise — "define a metric once, reuse it in every channel." dbt Semantic Layer, Cube, Looker, Lightdash, and AtScale each stake out a position, and Reverse ETL (Hightouch, Census) sends warehouse data back into operational tools to complete Data Activation. In the age of AI and LLMs the Semantic Layer is promoted to a safe data access layer, and Text-to-Metric is recommended over Text-to-SQL. Korean companies are catching up fast on the dbt + Metabase/Superset + Hightouch/Census combination, and cases of building an internal Metrics Store — Toss, Coupang, Kakao — are spreading. "The layer that speaks the meaning of data" is no longer a luxury; it is infrastructure for the AI era.