The Complete LLMOps Guide: Three-Axis Versioning of Model, Prompt, and Eval Set, Canary, Cost Control, and the Platform Team (2025)

Season 4 Ep 11 — If Ep 10 was "the axis that protects," Ep 11 is the axis that keeps things running sustainably. How to avoid "one week to the first release, then a year of hell."

Prologue — "Is LLMOps MLOps or DevOps?"
Chapter 1 · Three-Axis Versioning
Chapter 2 · Deployment Strategy
Chapter 3 · Cost Control
Chapter 4 · Building the Platform Team
Chapter 5 · The Eval Harness — The Heart of LLMOps
Chapter 6 · Observability and Operations
Chapter 7 · Data Governance
Chapter 8 · Ten Failure Cases
Chapter 9 · KPIs and OKRs
Chapter 10 · The Korean and Asian Environment
Chapter 11 · Tooling Roundup (2025)
Chapter 12 · Ten Anti-patterns
Chapter 13 · Checklist — 12 Things Before an LLMOps Launch
Chapter 14 · Next Up — Season 4 Ep 12: "AI Product Design"

Prologue — "Is LLMOps MLOps or DevOps?"

Both, and neither.

Unlike MLOps, model training is rare (what you use is a hosted API or an adapter)
Unlike DevOps, results are probabilistic, which makes testing hard
But deployment, observability, rollback, and cost control need every lesson from both

The core questions of LLMOps in 2025:

Three-axis versioning (model, prompt, eval set) — how do you keep it consistent?
Cost: how do you find and reduce token waste?
Organization: where is the boundary between the AI platform team and the product teams?
Regulation and audit: how do you leave a trace of every deployment and change?

Chapter 1 · Three-Axis Versioning

1.1 The Axes

Model: gpt-4o-2025-03-15, claude-3.7-sonnet, qwen2.5-14b-awq, and so on
Prompt: the Git/registry version of the system prompt
Eval set: the eval set snapshot tied to each release

1.2 Release Metadata

release: v1.4.2
model: anthropic/claude-3.5-sonnet-2025-02
prompt_version: sales_copilot@v23
eval_set: v12 (scored 91.3/100)
adapters:
  - korean_tone_lora@v3
guardrails:
  - llama_guard@v1
  - custom_policy@v8
rollout:
  canary: 5%

Every release has to pin this metadata as immutable.

1.3 Linking to Logs

Include release_id in every request log
Later you can reproduce "which configuration produced this response" with 100% fidelity
The minimum requirement for a regulatory audit

Chapter 2 · Deployment Strategy

2.1 Shadow

Call the new version in parallel against production traffic (the response is not used)
Diff it against the existing response and compare metrics
Confirm regressions and improvements with no user impact

2.2 Canary

Expand in sequence: 1–5% → 10% → 50% → 100%
Automatic gates at each stage (eval set score, p95 latency, error rate)
Roll back to the previous version immediately on failure

2.3 Blue-Green

Two environments (Blue live, Green standby) hold the old version and the new version
The switch is a load balancer flip
Rollback is immediate (a matter of seconds)

2.4 A/B Experiments

Secure statistical significance (sample size, duration)
Metrics: Quality, Cost, Latency, User satisfaction
End-user segmentation (user-based / organization-based)

2.5 Percentage Router

A small LLM gateway service splits traffic by user, organization, or feature
Versions can be changed instantly with a toggle
Open and SaaS gateways such as Helicone, Portkey, Martian, and LiteLLM

Chapter 3 · Cost Control

3.1 Token Audit

Record input/output tokens per request
Aggregate by user, feature, and endpoint
Weekly review of the highest-cost endpoints and prompts

3.2 Caching

Prompt cache: native on Anthropic, OpenAI, and Google
Response cache: reuse a previous response for identical or similar input
RAG retrieval cache: cache the query → document mapping

3.3 Model Routing

Simple query → small model → complex query → large model
The router itself is a classifier (LoRA or rules)
Open routers such as Martian and RouteLLM rose in 2025

3.4 Token Diet

Keep trimming the system prompt
Optimize the number of documents fed into RAG
Split JSON response fields into required and optional
Remove waste with Structured Output (Ep 2)

3.5 Storage and Network

Long-term log retention goes to cold storage
Compress and partition vector DB indexes
Minimize cross-region transfer

3.6 Realistic Savings Targets

First 3 months: 20–40% (low-hanging fruit)
Month 6: another 15–30% (model routing and cache sophistication)
After that: 5–10% of maintenance improvements

Chapter 4 · Building the Platform Team

4.1 Three Layers

Foundation / Infra: GPUs, model serving, observability infrastructure
AI Platform: shared SDK, prompt registry, eval harness, guardrails
Product AI: feature implementation for each product (chatbot, RAG, agents, voice, and so on)

4.2 Interfaces

AI Platform → Product AI: an SDK that is easy to use plus standard observability
Infra → AI Platform: stable serving and a cost dashboard
Manage every boundary like an API that has an SLA plus on-call

4.3 By Size

Startup of 10–50 people: no team split, just "1–2 AI leads plus product engineers"
Mid-size of 50–500: an AI Platform of 5–15, with an AI owner per product
Enterprise of 500+: all three layers split out, with separate security and compliance teams

4.4 Political Traps

"All AI goes through our team" → bottleneck
"Every product builds its own AI infrastructure" → duplicated waste
The middle path: a shared platform plus product autonomy was the dominant model in 2025

Chapter 5 · The Eval Harness — The Heart of LLMOps

5.1 Versioning the Eval Set

Git LFS or a data registry (DVC, Hugging Face Datasets, S3 + metadata)
A version, schema, labeler, and creation date on every eval set
Kept strictly separate from training data

5.2 CI Integration

Run a lightweight eval set automatically on every PR
Run the large eval set asynchronously after merge
Slack/Discord alerts when a regression is detected

5.3 On-demand Experiments

Evaluate new model candidates and prompt variants immediately
A dashboard comparing cost and results
Parallel experiments (several variants at once)

5.4 Production Feedback Loop

User thumbs → label candidates
Labeler review → added to the eval set
A monthly "consolidated incident-case release"

Chapter 6 · Observability and Operations

6.1 Extending Ep 6

Three layers: Trace/Span/Metric
Built on OpenTelemetry
LangFuse/LangSmith/Phoenix/Helicone/W&B

6.2 The Core Dashboards

Quality: eval set score trend, thumbs up/down
Cost: daily/monthly totals, by feature, by user
Latency: p50/p95/p99, TTFT, TTL
Safety: attack attempts, refusal rate, false refusal
Errors: by status, by vendor, retry rate

6.3 On-call

Define SLIs/SLOs for the key metrics
Alert tiers (P1/P2/P3)
Runbooks: vendor outage, cost blowout, quality regression
Postmortem within 24–72h of an incident

6.4 Preparing for Vendor Outages

Multi-provider routing (OpenAI outage → automatic fallback to Anthropic)
A minimum-functionality mode on a local model
A clear escalation path

Chapter 7 · Data Governance

7.1 Data Classification

Public / Internal / Confidential / PII / Regulated
A model, vendor, and log retention policy for each tier
Use an automatic classifier (DLP)

Disclose AI use, be transparent about the scope of log retention
APIs for deletion and data provision requests
Comply with GDPR and personal information protection law

7.3 Handling PII

Detect PII on input, then mask or pseudonymize it
Re-mask on output
Keep audit logs as hashes only

7.4 Licensing

The terms of the model used to generate synthetic data
Contracts covering third-party data use
Responding to user copyright concerns (citing RAG retrieval results)

Chapter 8 · Ten Failure Cases

8.1 Token Blowout

Costs spiked after the system prompt was updated from 3000 to 8000 tokens.

8.2 Prompt Rollback Impossible

Inline strings rather than Git, so the previous version could not be found.

8.3 Model Deprecation

The vendor announced end-of-life for a model, and a regression showed up on the replacement.

8.4 One Region Down, Everything Down

Dependence on a single region. No multi-region or multi-vendor fallback.

8.5 RAG Index Drift

Document updates did not trigger an index rebuild, producing stale answers.

8.6 Over-retention of User Logs

Flagged in a regulatory audit, resulting in a fine.

8.7 Prompt Injection

Internal documents leaked through indirect injection.

8.8 False Refusal Surge

The refusal rate on legitimate requests rose after guardrails were tightened.

8.9 Vector DB Cost Blowout

More chunks were generated than expected, doubling the monthly cost.

8.10 Misreading A/B Results

Too few samples and ignored variance, promoting the wrong version.

Chapter 9 · KPIs and OKRs

9.1 Product KPIs

Task completion rate
Time saved (per user)
NPS / Retention
Adoption / Active usage

9.2 Engineering KPIs

p95 latency
Error rate
Cost per request
Availability (SLA)

9.3 AI Quality KPIs

Eval set score (weekly/monthly)
Hallucination rate
Safety violations
False refusal rate

9.4 Operations KPIs

MTTR (mean time to recovery)
Incident frequency (weekly/quarterly)
Release cadence (times per week)
Test coverage

Chapter 10 · The Korean and Asian Environment

10.1 Cloud

AWS, GCP, Azure, Oracle + domestic (NHN Cloud, KT Cloud, Naver Cloud)
Regulatory audits may require a domestic cloud or on-premises
Prepare for Seoul and Tokyo region latency

10.2 Vendor Diversification

OpenAI, Anthropic, Google, Cohere, Mistral + domestic (Upstage, LG AI Research)
Finance and public sector: preferential terms for domestic vendors

10.3 Employee Training and Culture

AI usage guidelines (preventing internal data leaks)
Basic prompt engineering training (1–2 hours)
A security review before any new product adopts AI

Chapter 11 · Tooling Roundup (2025)

11.1 Gateway and Routing

LiteLLM, Portkey, Helicone, Martian, OpenRouter

11.2 Observability

LangFuse, LangSmith, Phoenix (Arize), Helicone, W&B Weave, Datadog/NewRelic LLM extensions

11.3 Evaluation

RAGAS, DeepEval, Promptfoo, Giskard, PyRIT (attack), OpenAI Evals, Confident AI

11.4 Prompt Registry

LangSmith Prompt Hub, Humanloop, PromptLayer, W&B Weave, Agenta

11.5 Data and Features

Weights & Biases Artifacts, MLflow, DVC, LakeFS

11.6 Serving

vLLM, TGI, SGLang, TensorRT-LLM, Ollama, Modal, Anyscale, Together, Fireworks

11.7 Guardrails and Security

NeMo Guardrails, Guardrails AI, Lakera Guard, Rebuff, Azure Content Safety, Google Cloud Model Armor

11.8 Cost and Resources

Helicone Cost Insights, OpenMeter, CloudZero, Cast AI, Karpenter (K8s)

Chapter 12 · Ten Anti-patterns

12.1 Swapping the Model and Shipping

Rolling out without rechecking the eval set and the prompt.

12.2 Single-vendor Dependence

The risk of outages, price increases, and changes to the terms.

12.3 Prompts Living Outside Git

Config files, inline code, SaaS-only → versioning disappears.

12.4 Not Using Shadow or Canary

Discovering problems in production.

12.5 No Cost Dashboard

A shock when the month-end invoice arrives.

12.6 Eval Set Mixed with Training Data

Inflated evaluation.

12.7 No Plan for Vendor Outages

Downtime users feel keenly.

12.8 An Over-powerful AI Platform Team

The autonomy of Product AI gets suppressed.

12.9 No Regard for Regulation and Audit

Fines or a forced shutdown after launch.

12.10 No Postmortem Written

The same incident repeats.

Chapter 13 · Checklist — 12 Things Before an LLMOps Launch

Chapter 14 · Next Up — Season 4 Ep 12: "AI Product Design"

Once the engineering is stable, what remains is the user experience.

The boundary between deterministic UI and generative UI
Design that builds "trust" (citations, uncertainty, editability)
Feedback UX: patterns beyond 👍/👎
Handling streaming, latency, and feedback
Agent UX: progress, approval, interruption, replay
Voice UX in depth (extending Ep 9)
The UX of data and learning
Onboarding and first-impression design
The UX of failures and outages
Accessibility and inclusion
Ethical AI UX
Specialization for the Korean language and Korean culture

It is not "a good AI product = a good model + good UX" but "a good AI product = UX design that builds trust within constraints."

See you in the next post.

Summary: LLMOps comes down to three-axis versioning, Shadow/Canary, cost control, the platform team, and audit. It absorbs the lessons of MLOps and DevOps, but it needs additional practices that account for the probabilistic nature and vendor dependence unique to LLMs. Building one has become easy, but to build "an LLM product that keeps improving for a year without stopping" the 12-item checklist in this post is the minimum starting point. "AI is not something you deploy, it is something you run" is the lesson of 2025.