필사 모드: Observability 2025 Complete Guide: OpenTelemetry, Grafana / Datadog / Honeycomb / SigNoz, SLO and Error Budget, LLM Observability (2025)

Season 5 Ep 8 — If governance is "management", observability is "grasping reality". In 2025 observability does not stop at log, metric, and trace — it takes in LLMs, data pipelines, and user experience as well.

• Prologue — "Observability is not insurance, it is the product"
• Chapter 1 · Three Signals + α
  • 1.1 The basic three
  • 1.2 Extended signals
  • 1.3 Correlation
• Chapter 2 · OpenTelemetry — a Standard Comes of Age
  • 2.1 What it is
  • 2.2 Structure
  • 2.3 Why it matters
  • 2.4 How hard it is to adopt
• Chapter 3 · The Grafana Stack (Open)
  • 3.1 Components
  • 3.2 Grafana Cloud
  • 3.3 Strengths
  • 3.4 Limits
• Chapter 4 · Datadog, New Relic, Splunk, Dynatrace — the SaaS Giants
  • 4.1 Datadog
  • 4.2 New Relic
  • 4.3 Splunk
  • 4.4 Dynatrace
  • 4.5 Comparison
• Chapter 5 · The New Generation — Honeycomb, SigNoz, Axiom, Tinybird
  • 5.1 Honeycomb
  • 5.2 SigNoz
  • 5.3 Axiom
  • 5.4 Tinybird
  • 5.5 OpenObserve
  • 5.6 What they share
• Chapter 6 · Implementing Logs, Metrics, and Traces
  • 6.1 Metric design
  • 6.2 Log design
  • 6.3 Trace design
  • 6.4 RUM
  • 6.5 Synthetic
• Chapter 7 · SLO, SLI, Error Budget
  • 7.1 Concepts
  • 7.2 Why it matters
  • 7.3 Metrics in practice
  • 7.4 Error budget policy
  • 7.5 Tools
• Chapter 8 · LLM Observability (an extension of Ep 6)
  • 8.1 Additional signals
  • 8.2 Major tools
  • 8.3 OpenLLMetry
  • 8.4 Operating patterns
• Chapter 9 · Chaos Engineering and Resilience
  • 9.1 Philosophy
  • 9.2 Tools
  • 9.3 In practice
  • 9.4 Applying it to data pipelines
  • 9.5 2025 trends
• Chapter 10 · Alerting, On-call, Incident Response
  • 10.1 Alert design
  • 10.2 On-call culture
  • 10.3 Incident response
  • 10.4 Postmortem
  • 10.5 Tools
• Chapter 11 · Cost Optimization
  • 11.1 Log cost
  • 11.2 Metric cost
  • 11.3 Trace cost
  • 11.4 SaaS cost
  • 11.5 A realistic target
• Chapter 12 · Observability at Korean Companies
  • 12.1 Where things stand
  • 12.2 Adopting LLM observability
  • 12.3 Korean particulars
  • 12.4 Reference cases
• Chapter 13 · 10 Antipatterns
  • 13.1 "Add observability after the deploy"
  • 13.2 Every log at DEBUG
  • 13.3 Alert spam
  • 13.4 Operating without SLOs
  • 13.5 Analyzing an incident without traces
  • 13.6 High-cardinality metric labels
  • 13.7 Sensitive information in logs
  • 13.8 The same incident repeating with no postmortem
  • 13.9 Leaving SaaS cost unattended
  • 13.10 Zero LLM observability
• Chapter 14 · Checklist — 12 Things Before an Observability Launch
• Chapter 15 · Next Post Preview — Season 5 Ep 9: "Data Team Organization and Careers"

Prologue — "Observability is not insurance, it is the product"

From 2015 to 2020 many teams treated observability as "insurance for incident response". 2025 is different:

• Finding user problems first = a competitive advantage
• SLO-based operations = both development speed and stability
• The observation data itself is the basis for product decisions
• For LLM products, "is the output right" becomes something you observe

The era of "observability is a cost" is over. "Observability is part of product quality" is the 2025 standard.

---

Chapter 1 · Three Signals + α

1.1 The basic three

• Metric: time series numbers (CPU, QPS, latency)
• Log: event text
• Trace: the path of a distributed request

1.2 Extended signals

• Profile: CPU and memory profiling (continuous profiling)
• Event: business, deployment, and release events
• RUM (Real User Monitoring): what browser and mobile users actually feel
• Synthetic: periodic virtual requests
• LLM events: prompt, response, tokens, cost, feedback

1.3 Correlation

• Jump from a metric spike to the trace and log at that timestamp
• User session → trace → log linkage
• The default UX of 2025 tooling

---

Chapter 2 · OpenTelemetry — a Standard Comes of Age

2.1 What it is

• A CNCF project since 2019 (OpenCensus + OpenTracing merged)
• A standard for metric, trace, log, and profile (2024)
• A vendor-neutral collection pipeline

2.2 Structure

• SDK: per language (Python/Go/JS/Java/Rust and others)
• Collector: collection, processing, routing
• OTLP protocol: gRPC/HTTP
• Exporter: Prometheus, Datadog, Grafana, New Relic, and others

2.3 Why it matters

• Vendor lock-in removed: the code is OTel, the backend is replaceable
• Through 2024–2025 most observability vendors adopted OTel natively
• Semantic Conventions standardize metadata (HTTP, DB, messaging)

2.4 How hard it is to adopt

• Basic collection is easy, but
• Semantic Conventions, sampling, and cost tuning need expertise
• Unifying a fragmented legacy metric/log setup is the real task

---

Chapter 3 · The Grafana Stack (Open)

3.1 Components

• Prometheus (metric): the standard; at scale, Mimir/Thanos/VictoriaMetrics
• Loki (log): low cost by design, "index the metadata only"
• Tempo (trace): OTel-ready, cheap storage
• Grafana (UI): the unified dashboard
• Pyroscope (profile): acquired in 2023, continuous profiling

3.2 Grafana Cloud

• The stack above as a managed SaaS
• A free tier exists, paid plans are usage-based
• The value-for-money option for mid-sized companies

3.3 Strengths

• Open flexibility plus a unified UX
• Cost efficient (log and trace storage)
• A large community

3.4 Limits

• Self-hosting means carrying the burden of several components
• It does not reach the Datadog-grade "all-in-one" experience

---

Chapter 4 · Datadog, New Relic, Splunk, Dynatrace — the SaaS Giants

4.1 Datadog

• Number one in the industry, 600+ integrations
• Infrastructure, APM, logs, RUM, security, LLM observability — all of it
• Cost is the biggest complaint (and also the source of its revenue growth)

4.2 New Relic

• Redesigned in 2020 (NRDB) for unified signals
• Pricing model changed in 2022 (data-based → user-based)
• Kubernetes and OTel friendly

4.3 Splunk

• Strong in logs and security, acquired by Cisco in 2023
• Specialized for enterprise and security
• Consolidating Observability Cloud (AppD, SignalFx)

4.4 Dynatrace

• Built on automation and AI (Davis)
• Enterprise, large and complex systems

4.5 Comparison

---

Chapter 5 · The New Generation — Honeycomb, SigNoz, Axiom, Tinybird

5.1 Honeycomb

• Pioneer of high-cardinality, exploratory observability
• BubbleUp (automatic correlation analysis)
• Large influence on engineering culture (Charity Majors)

5.2 SigNoz

• An open source Datadog alternative
• Built on ClickHouse
• Cost savings through self-hosting

5.3 Axiom

• Serverless logs, ultra-cheap storage
• Event and analytics centric
• Popular with the community and with personal projects

5.4 Tinybird

• ClickHouse, API-first
• Real-time analytics and user-facing metrics
• Not observability-specific, but a similar workload

5.5 OpenObserve

• A full open source stack
• Cost built on ClickHouse

5.6 What they share

• OTel support
• Low-cost storage on ClickHouse/Parquet
• Kafka and Kinesis support

---

Chapter 6 · Implementing Logs, Metrics, and Traces

6.1 Metric design

• RED (rate, errors, duration) for services
• USE (utilization, saturation, errors) for resources
• Golden signals (latency, traffic, errors, saturation)
• Beware high-cardinality labels (Prometheus blowup)

6.2 Log design

• Structured logs (JSON)
• Include trace ID, span ID, user ID
• Consistent levels (DEBUG/INFO/WARN/ERROR)
• Mask sensitive information

6.3 Trace design

• A span at every service boundary
• Auto-instrument DB and external API calls
• Manage cost with sampling (head, tail)
• Include business attributes (org, plan)

6.4 RUM

• Browser: Web Vitals (LCP, CLS, INP)
• Mobile: startup time, crashes, network
• Link user ID, version, and device

6.5 Synthetic

• Key paths every 1–5 minutes
• Checks per region
• Validate multi-step workflows (login → payment)

---

Chapter 7 · SLO, SLI, Error Budget

7.1 Concepts

• SLI (Service Level Indicator): the measured metric (availability, latency)
• SLO (Service Level Objective): the target (99.9% availability)
• Error Budget: the allowed amount of failure (43 minutes a month)

7.2 Why it matters

• 100% is impossible and uneconomical
• A scientific balance between shipping features and stability
• A shared language across teams

7.3 Metrics in practice

• Availability: successful requests / total requests
• Latency: the share within the p95/p99 threshold
• Correctness: the result error rate
• Freshness: data lag
• LLM quality: eval set score (Ep 6)

7.4 Error budget policy

• When the budget is spent, freeze feature deploys and move to stabilization work
• When budget is left over, deploy and experiment aggressively

7.5 Tools

• Grafana SLO, Datadog SLO, Nobl9, Blameless
• Prometheus + Sloth (OSS)
• Monthly and quarterly reviews are mandatory

---

Chapter 8 · LLM Observability (an extension of Ep 6)

8.1 Additional signals

• Prompt and response text
• Tokens and cost
• Latency and TTFT (Time to First Token)
• Eval set scores and user feedback
• Tool calls and agent steps

8.2 Major tools

• LangFuse (open + SaaS)
• LangSmith (LangChain)
• Phoenix/Arize
• Helicone
• Weights & Biases Weave
• Traceloop (built on OpenLLMetry)
• Datadog LLM Observability (2024)

8.3 OpenLLMetry

• Standardizes LLM semantic conventions as an OpenTelemetry extension
• Started in 2024 led by Traceloop → active OTel standardization discussion in 2025

8.4 Operating patterns

• Prompt A/B and canary (Ep 11)
• Periodic eval set runs plus regression alerts
• A loop that feeds user feedback back in
• Cost and token dashboards

---

Chapter 9 · Chaos Engineering and Resilience

9.1 Philosophy

• Failure is inevitable → cause it deliberately and learn from it
• Originates with Netflix Chaos Monkey in 2010

9.2 Tools

• Gremlin, LitmusChaos (K8s), Steadybit
• AWS Fault Injection Service
• Chaos Toolkit

9.3 In practice

• Game days (one team reproduces a failure, the rest respond)
• Runbook testing
• Validate recovery mechanisms (circuit breaker, retry, fallback)

9.4 Applying it to data pipelines

• When a Kafka partition goes down
• When the DB fails over
• When one engine goes down

9.5 2025 trends

• Resilience as code: chaos scenarios managed as code
• CI integration: recovery tests before deployment

---

Chapter 10 · Alerting, On-call, Incident Response

10.1 Alert design

• Symptom-based: based on user impact (a rise in 5XX)
• Avoid cause-based: an alert for every cause is noise
• Severity (P1/P2/P3)
• Throttling and suppression

10.2 On-call culture

• A weekly rotation
• Primary and secondary
• Compensation and rest policy

10.3 Incident response

• Detect → Triage → Mitigate → Resolve → Learn
• A communication channel (a dedicated Slack room)
• Customer communication (status page)

10.4 Postmortem

• Blameless
• Timeline, root cause, action items
• Published (internally) and archived

10.5 Tools

• PagerDuty, Incident.io, FireHydrant, Rootly
• Slack and MS Teams integration
• Jira and Linear links (action tracking)

---

Chapter 11 · Cost Optimization

11.1 Log cost

• Choose what to store by structuring and classifying fields
• Cheap storage (ClickHouse, Loki)
• Long-term archive (S3 Glacier)
• Sensitive information masked already

11.2 Metric cost

• Manage label cardinality
• Optimize the scrape interval
• Aggregation rules

11.3 Trace cost

• Head sampling at 1–10%
• Tail sampling (errors and slow requests first)
• Automatic dropping (exclude health checks)

11.4 SaaS cost

• Watch for surges in usage-based models
• Monthly review plus budget alerts
• Consider open alternatives

11.5 A realistic target

• Observability cost / infrastructure cost = 5–15% is typical
• Going over 20% is a signal that optimization is needed

---

Chapter 12 · Observability at Korean Companies

12.1 Where things stand

• Large corporations: Datadog, Dynatrace, Splunk plus in-house builds
• Internet companies: the Grafana stack and self-run Prometheus plus some SaaS
• Startups: a mix of Grafana Cloud, Datadog, and Axiom
• Finance and public sector: on-premise Elastic, Splunk, Prometheus

12.2 Adopting LLM observability

• Started in 2024, expanded sharply in 2025
• Self-hosting the open source LangFuse
• In-house build cases (Toss, Kakao, Naver)

12.3 Korean particulars

• Fully on-premise requirements in network-separated environments
• Korean-language logs and alert text
• A culture of covering holidays and weekends
• Financial Supervisory Service audit requirements

12.4 Reference cases

• Coupang: an in-house observability platform mixed with open source
• Naver: large-scale Elastic plus an in-house APM
• Samsung, LG, SK: enterprise tools built out with SI
• Toss: leading the SLO-based operating culture

---

Chapter 13 · 10 Antipatterns

13.1 "Add observability after the deploy"

It is only worth something if you instrument from the start.

13.2 Every log at DEBUG

Cost explodes and important signals get buried.

13.3 Alert spam

An alert on every error → alert fatigue syndrome.

13.4 Operating without SLOs

Political fights over "development vs stability".

13.5 Analyzing an incident without traces

Hours spent, cause unknown.

13.6 High-cardinality metric labels

Prometheus blows up.

13.7 Sensitive information in logs

Audit findings and leak incidents.

13.8 The same incident repeating with no postmortem

There is no learning loop.

13.9 Leaving SaaS cost unattended

End-of-month invoice shock.

13.10 Zero LLM observability

Hallucination, refusals, and runaway cost all go undetected.

---

Chapter 14 · Checklist — 12 Things Before an Observability Launch

• OpenTelemetry-based instrumentation (SDK + Collector)
• The three signals plus RUM and Synthetic
• Structured logs linked by trace ID
• SLO/SLI definitions and dashboards
• An error budget policy
• On-call rotation plus runbooks
• Severity and alert design (symptom-based)
• A postmortem process (blameless)
• LLM observability (prompts, tokens, feedback)
• A cost dashboard plus a review cadence
• Chaos engineering and resilience drills
• Security and privacy guidance

---

Chapter 15 · Next Post Preview — Season 5 Ep 9: "Data Team Organization and Careers"

With technology, governance, and observability stacked up, next come the people who build them. Ep 9 is about the organization and careers of data and AI teams.

• Data engineer vs analytics engineer vs platform engineer vs scientist
• The rise of the ML engineer and the AI engineer
• Central vs embedded vs mesh organizational models
• Team size and roles at startups and large companies
• On-call and knowledge sharing culture
• Leadership paths (manager vs IC)
• Global remote work and Korean particulars
• Salary and compensation structures in 2025
• Hiring and interview trends
• Learning and career strategy

"What matters more than tools is the team" — where data organizations stand in 2025.

See you in the next post.

---

Summary: In 2025 observability expanded into "three signals + α" (metric, log, trace plus profile, RUM, synthetic, LLM events), and as OpenTelemetry became the collection standard, vendor lock-in is dissolving fast. The Grafana stack is open and good value, Datadog, New Relic, Splunk, and Dynatrace are integrated and enterprise, and Honeycomb, SigNoz, and Axiom are the new generation of low cost and high explorability. SLO and Error Budget became the language for balancing development against stability, and LLM observability (LangFuse, Phoenix, Helicone, Datadog LLM) went mainstream as an extension of Ep 6. The declaration that "observability is not insurance, it is product quality" is the 2025 default. Korean companies are fusing network separation, the Korean language, and SLO culture, and the next post is about the data teams and people who build that observability.