Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Season 5 Ep 7 — Through Ep 1–6, data kept growing and getting more complex. Ep 7 is the opposite axis — "how do we govern it". If you do not manage your data, your data will rule your company.

Prologue — "How many places in our company hold a customer email?"
Chapter 1 · Defining Data Governance
Chapter 2 · OpenLineage — The Standard for Lineage
Chapter 3 · The Four Major Data Catalogs
Chapter 4 · Technical Catalogs — Unity/Polaris/Glue
Chapter 5 · PII — Definition and Detection
Chapter 6 · Protecting PII — Masking, Tokenization, Encryption
Chapter 7 · Regulation — GDPR, Korean PIPA, AI Act
Chapter 8 · Data Subject Request (DSR)
Chapter 9 · Extending Governance into the AI Era
Chapter 10 · Practical Architecture — Governance Integration
Chapter 11 · The Reality of Governance at Korean Companies
Chapter 12 · Eight Failure Stories
Chapter 13 · Ten Anti-patterns
Chapter 14 · Checklist — Twelve Points for Data Governance
Chapter 15 · Next Up — Season 5 Ep 8: "Observability 2025 (Logs, Metrics, Traces + LLM)"

Prologue — "How many places in our company hold a customer email?"

The hardest questions for a CTO to answer in 2025:

How many tables and systems hold customer email addresses?
How many of those are PII-masked?
When a user says "delete my data", where do you start deleting?
Which source data did the numbers on this dashboard come from?

If the answer to all four is "I do not know", that company carries regulatory risk and product quality risk at the same time. This post lays out the tools and processes that make those four questions answerable.

Chapter 1 · Defining Data Governance

1.1 The Four Axes of Governance

Catalog: what data do we have
Lineage: where does it come from and where does it go
Quality: is this data trustworthy
Security and regulation (PII, RBAC): who can see it and how long do we keep it

1.2 Why It Matters Now

Fragmentation of the data stack (Ep 1–6) makes management complexity explode
Tightening of the EU AI Act, GDPR and the Korean Personal Information Protection Act
AI models train on and infer from data, raising copyright and PII issues
Customers and employees exercise their data rights more often

1.3 The Governance Spectrum

Catalog: read-oriented, discovery by employees
Active governance: enforcing policy, blocking violations
Federation: unified management across multiple clouds and multiple databases

Chapter 2 · OpenLineage — The Standard for Lineage

2.1 Identity

Open sourced in 2020, under the LF AI & Data Foundation
The industry standard spec for data lineage
Reference implementations in Python, Java and OpenAPI

2.2 Structure

Job: a unit of execution (Airflow DAG, dbt run, Flink job)
Run: an execution instance of a Job
Dataset: input and output datasets
Event: START/COMPLETE/FAIL events

2.3 Integrations

Supports Airflow, dbt, Spark, Flink, Dagster and Prefect
Marquez (the reference metadata store)
Datakin, then Astronomer, provides SaaS support
DataHub, Atlan and Collibra receive OpenLineage events

2.4 Example Event

{
  "eventType": "COMPLETE",
  "job": {"name": "dbt.fact_orders"},
  "run": {"runId": "..."},
  "inputs": [{"name": "raw.orders"}, {"name": "raw.customers"}],
  "outputs": [{"name": "analytics.fact_orders"}]
}

2.5 Value

Automatic lineage collection reduces the documentation burden on people
Portable between tools
Incident analysis and impact assessment

Chapter 3 · The Four Major Data Catalogs

3.1 Collibra

2008, Belgium — the strongest player in enterprise governance
Specialized in finance, public sector and pharmaceuticals
Strong at business glossary and policy management

3.2 Atlan

2018, India — friendly to the modern data stack
Outstanding integration with dbt, Snowflake and Databricks
Excellent collaboration and UX

3.3 DataHub

2020, LinkedIn — then Apache
Open source, can be self-hosted
Backed by Acryl Data (commercial SaaS)

3.4 Alation

2012 — an enterprise collaboration catalog
Positioned as a Data Intelligence Platform

3.5 Others

Secoda, Castor(→ CoreView), OpenMetadata, Amundsen(Lyft)
Informatica Data Governance, IBM Watson Knowledge Catalog

3.6 Comparison

Tool	Strength	Customers	Model
Collibra	Regulation, enterprise	Finance, public sector	SaaS/self
Atlan	Modern stack, UX	SaaS, startups, mid-size	SaaS
DataHub	Open, flexible	Engineering teams	OSS + Acryl
Alation	Collaboration, business users	Mid-size, enterprise	SaaS
OpenMetadata	Open, lightweight	Self-hosted	OSS

Chapter 4 · Technical Catalogs — Unity/Polaris/Glue

4.1 Unity Catalog

The governance layer of Databricks
Open sourced in 2024 (Unity Catalog OSS)
Unifies Table, Volume, Model and Function
Supports both Delta and Iceberg

4.2 Polaris

The open Iceberg REST catalog from Snowflake
Open sourced in 2024
Accessible from external engines
Combines governance with the catalog

4.3 AWS Glue Data Catalog

The default catalog on AWS
Supports Iceberg, Delta and Hudi
Combined with Lake Formation for permissions and auditing

4.4 Nessie / Gravitino

Nessie: Git-like branches and tags
Gravitino: metadata federation (unifying several catalogs)

4.5 Business vs Technical Catalog

Technical catalog: the metadata engines and queries consume
Business catalog: the meaning, owner and SLA that people consume
In 2025 linking the two (OpenLineage, OpenMetadata) is the standard

Chapter 5 · PII — Definition and Detection

5.1 Defining PII

Personally Identifiable Information
Direct identifiers (name, resident registration number, card number)
Indirect identifiers (date of birth plus region, IP, cookie ID)
Sensitive information (health, religion, orientation)

5.2 Automatic Detection Tools

AWS Macie, Google DLP, Azure Purview: cloud DLP
Privacera, BigID, OneTrust: enterprise specialists
Presidio (Microsoft): open source
Snowflake Data Classification: built into the DW

5.3 Detection Methods

Regular expressions (card numbers, resident registration numbers)
Dictionary based (names, addresses)
ML classifiers (context included)
Column sampling plus entropy analysis

5.4 Classification Grades

Public / Internal / Confidential / PII / Sensitive PII / Regulated
Access, log retention and encryption policies differ by grade

Chapter 6 · Protecting PII — Masking, Tokenization, Encryption

6.1 Masking

Email: john@***.com
Card: **** **** **** 1234
Name: J*hn D*e or John OO
Dynamic Data Masking: shows different values per user at query time

6.2 Tokenization

Replace PII with meaningless tokens and keep the mapping in a separate vault
Preserves analytical usefulness while minimizing the impact of a leak
Vault: HashiCorp Vault, AWS KMS plus an in-house service

6.3 Hashing and Anonymization

One-way hashes such as SHA-256: analysis is possible, the original cannot be recovered
k-anonymity, l-diversity, differential privacy
Used for healthcare and public sector data

6.4 Encryption

At rest: S3 SSE, disk encryption
In transit: TLS
Column-level: Parquet Modular Encryption
Field-level: encrypt in the application before storing

6.5 Dynamic Masking Example (Snowflake)

CREATE MASKING POLICY mask_email AS (val STRING)
RETURNS STRING ->
  CASE WHEN CURRENT_ROLE() IN ('ADMIN') THEN val
       ELSE REGEXP_REPLACE(val, '.+@', '***@')
  END;

ALTER TABLE customers MODIFY COLUMN email
  SET MASKING POLICY mask_email;

Data subject rights (access, erasure, portability, rectification, objection to processing)
A legal basis (consent, contract and so on) must be stated
DPIA (impact assessment), DPO (data protection officer)
Violations draw up to 4% of global turnover or 20 million euros

7.2 The Korean Personal Information Protection Act (PIPA)

Enacted in 2011, amended in 2020 and 2023
Consent for processing personal information and a stated purpose are required
The 2023 amendment strengthened data subject rights (access, erasure, portability)
Introduced an administrative fine capped at 3% of revenue
Supervised by the Personal Information Protection Commission

7.3 The Korean AI Framework Act (2024–)

Prior impact assessment for high-risk AI
Guidelines for responsible AI development and operation
Reporting obligations for operators

7.4 Other Regulations

HIPAA, GLBA and CPRA (California) in the United States
PDPA in Singapore, APPI in Japan
By sector: the Financial Supervisory Service, the Medical Service Act, the Regulation on Supervision of Electronic Financial Transactions

7.5 Common Principles

Purpose limitation: no use beyond the purpose of collection
Minimal collection: only as much as necessary
Retention period: destroy once the purpose is achieved
Subject rights: access, rectification, erasure, portability
Transparency: disclose how processing is carried out

Chapter 8 · Data Subject Request (DSR)

8.1 Request Types

Access: view my data
Rectification: correction
Erasure (right to be forgotten): deletion
Portability: move to another service
Object: refuse processing

8.2 Implementation Difficulty

Deleting that user rows in the warehouse
Row-level delete in Lakehouse Iceberg
Handling backups and logs
When the data is part of ML training data (retrain?)
Data shared with third parties

8.3 Practical Patterns

Data Map: a list of the tables and locations that hold PII
Unique ID: search every system with a per-user unique ID
DSR workflow tools: OneTrust, TrustArc, Osano
Log and backup policy: automatic destruction after the statutory period

8.4 AI and Training Data

Deleting an individual from training data is difficult in principle
Remedies: pseudonymization, anonymization, synthetic data
Machine unlearning research is under way

Chapter 9 · Extending Governance into the AI Era

9.1 Model Catalog

The list of models the company uses
Version, data, performance, license, audit
MLflow, Databricks Model Registry, Weights & Biases, HuggingFace Hub

9.2 Prompt and Agent Catalog

System prompt versions
Agent tools and permissions
An extension of Ep 11 (LLMOps)

9.3 Tracking the Provenance of Training Data

Documenting copyright and licenses
Dataset Cards
Presenting the original evidence when an audit asks

9.4 Auditing AI Output

Response logs, citations, the basis for decisions
Reproducibility for postmortems

9.5 The Link to Regulation

EU AI Act: quality management and documentation for high-risk models
The Korean AI Framework Act: impact assessment

Chapter 10 · Practical Architecture — Governance Integration

10.1 Layers

Ingestion: Kafka, Fivetran, Airbyte emit OpenLineage events
Transformation: dbt and Spark emit OpenLineage automatically
Storage: Iceberg and Delta plus a catalog (Unity/Polaris/Glue)
Governance: DataHub/Atlan/Collibra receive and consolidate
Policy: Unity/Lake Formation/Ranger/OPA
PII: Macie/Privacera/DLP

10.2 Policy Example

Raw: original data, restricted access, retained 90 days
Silver: masking plus tokenization, 1 year
Gold: aggregates, permission-based masking, 3 years or more
Backups: encrypted, destroyed automatically after the legal period

10.3 Auditing

Logs of every query and DDL
Access anomaly detection (unusual hour, IP, volume)
A quarterly audit report

10.4 Automation

A governance check in CI when a PR merges
A new table triggers automatic classification and owner assignment
Access requests go through an approval workflow

Chapter 11 · The Reality of Governance at Korean Companies

11.1 Current State

Finance and public sector: in-house builds mixed with Collibra/Informatica
Large enterprises: DataHub open source, then an internal fork
Mid-size companies and startups: growing adoption of Atlan and Secoda
Dedicated observability and PII tools are still at an early stage

11.2 Regulatory Response

Growing numbers of audits and administrative fines from the Personal Information Protection Commission
Guidelines from the Financial Security Institute
The three data laws for public data (Personal Information Protection Act, Credit Information Act, Information and Communications Network Act)

11.3 Challenges

Building a catalog inside a network-separated environment
Support for Korean-language metadata
Automatic collection of history from legacy DWs is hard
Talent shortage: dedicated governance staff are scarce

11.4 Reference Cases

Toss: DSR and catalog integrated into the internal data platform
Kakao: an automatic PII classification pipeline
Naver: a governance standards team plus an in-house catalog
Samsung SDS and LG CNS: governance SI packages for enterprise customers

Chapter 12 · Eight Failure Stories

12.1 PII Exposed on a BI Dashboard

Real names shown without dynamic masking, flagged in an audit.

12.2 Incident Response Without Lineage

Spent half a day hunting for "why this metric is wrong".

12.3 A Deletion Request That Took a Month

Manual work, and simultaneous deletion across several systems failed.

12.4 Thousands of Tables With No Known Owner

Nobody knows who to ask.

12.5 A Catalog That Exists but Is Empty

Built once and never maintained, so it drifted away from reality.

12.6 Fragmented Access Permissions

RBAC configured separately in ten databases, central management failed.

12.7 Contractors With Access to All PII

An audit violation.

12.8 Unknown Provenance for AI Training Data

Helpless in a copyright dispute or a regulatory investigation.

Chapter 13 · Ten Anti-patterns

13.1 "Governance Later"

The larger you grow, the harder it is to tear up and redo. Lay the foundations from the start.

13.2 A Catalog With No Owners

There is documentation, but no accountability.

13.3 A Few Lines of Regex for PII Detection

You need ML classifiers and sampling alongside it.

13.4 Indifference to Regulation for Backups

The "the destruction deadline passed but it is still in the backup" incident.

13.5 Lineage as a Hand-written Document

Adopt automatic collection (OpenLineage).

13.6 Changing Upstream Freely Without a Data Contract

Every consumer breaks.

Public S3 buckets, email attachments.

13.8 Short Retention for Audit Logs

Fails to meet regulatory requirements.

13.9 Running Several Catalog Tools in Parallel

Duplicated management with no consolidation.

13.10 Leaving AI and ML Outside Governance

In 2025 this is the biggest risk of all.

Chapter 14 · Checklist — Twelve Points for Data Governance

Chapter 15 · Next Up — Season 5 Ep 8: "Observability 2025 (Logs, Metrics, Traces + LLM)"

If governance is "how do we manage data", observability is "how are the systems and the data actually running".

The maturing of OpenTelemetry
Unifying the three signals: metrics, logs and traces
Grafana Cloud, Datadog, New Relic, Splunk, Honeycomb, SigNoz
Tempo, Loki, Mimir, Jaeger, Zipkin
LLM observability (LangFuse, LangSmith, Phoenix, Helicone, an extension of Ep 6)
SLOs, SLIs and error budgets
Chaos engineering and resilience
The observability stack at Korean companies
"Observability is not insurance, it is product quality"

"No observability, no operations" — the default of infrastructure in 2025.

See you in the next post.

Summary: Data governance in 2025 rests on four axes — catalog, lineage, quality, and PII and regulation. OpenLineage has become the standard for lineage, Collibra, Atlan, DataHub and Alation are the four catalog options, and Unity, Polaris and Glue handle the technical catalog. PII is managed in five stages of detection, classification, masking, tokenization and encryption, while GDPR, the Korean PIPA and the AI Act form the regulatory triangle. Governance in the AI era now extends to models, prompts and training data, and data subject requests (DSR) demand an automated workflow. For Korean companies, network separation, Korean-language metadata and legacy integration are the challenges, and securing dedicated governance staff is the next competitive edge. "If you do not manage it, it manages you" — the 2025 law of data governance.