필사 모드: What LLM Ops Actually Does — Reproducibility, Contamination, Checkpoints, Promotion, and Rollback

• Introduction — The Moment You Can't Answer "How Was That Model Built"
• The Smallest Unit of Reproducibility — The Run Manifest
• Data Pipelines and Contamination Management
• Checkpoint Management — Interval and Retention Are Different Problems
• Putting Evaluation Into CI
• Registry and Promotion — Where Training Hands Off to Serving
• After Deployment — Regression Detection and Rollback
• Tool Map — By Category
• Closing — A Result You Can't Reproduce Isn't a Result

Introduction — The Moment You Can't Answer "How Was That Model Built"

A problem turns up in a model you shipped three months ago. Someone asks what data it was trained on, what the configuration was, what the evaluation scores were at the time. If the answer that comes back from the team is "it's probably in that one notebook somewhere," that organization does not have LLM Ops.

Learning LLM Ops as a list of tools will not stop this. You can bolt on an experiment tracker, but if it never recorded a data snapshot, you can't reproduce the run; you can automate evaluation, but if the eval set was contaminated, the entire score is void.

So this post organizes things by responsibility, not tools. Each section is structured around "what breaks" and "what you have to record to prevent it." Tools are attached only at the end, sorted by category. No vendor pricing — it goes stale too fast.

The Smallest Unit of Reproducibility — The Run Manifest

First, an honest caveat. Bit-for-bit reproduction is impossible in most cases. Change the GPU count and the reduction order changes, which changes the floating-point result; kernel auto-tuning can pick a different algorithm on every run; a single driver update can shift the outcome. Turning on every determinism flag makes runs reproducible, but the slowdown is large enough that it's unusable for pretraining.

So the realistic goal isn't "the same bits" — it's being able to start again from the same coordinates. Those coordinates are what the run manifest records.

# run_manifest.py — call once from rank 0 at the start of training
import hashlib
import importlib.metadata as md
import json
import os
import platform
import subprocess
import time


def _sh(cmd: str) -> str:
    try:
        return subprocess.check_output(cmd, shell=True, text=True,
                                       stderr=subprocess.DEVNULL).strip()
    except Exception:
        return "unknown"


def build_manifest(config: dict, data_snapshot_id: str) -> dict:
    cfg_bytes = json.dumps(config, sort_keys=True).encode()
    return {
        "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
        "code": {
            "commit": _sh("git rev-parse HEAD"),
            "dirty": _sh("git status --porcelain") != "",
            "remote": _sh("git config --get remote.origin.url"),
        },
        "config": {
            "sha256": hashlib.sha256(cfg_bytes).hexdigest()[:16],
            "values": config,
        },
        "data": {
            "snapshot_id": data_snapshot_id,      # immutable snapshot identifier
            "manifest_sha256": _sh(f"sha256sum data/manifests/{data_snapshot_id}.jsonl | cut -d' ' -f1"),
        },
        "packages": {
            p: md.version(p)
            for p in ("torch", "transformers", "trl", "peft", "accelerate",
                      "deepspeed", "datasets")
            if _safe_version(p)
        },
        "hardware": {
            "gpu": _sh("nvidia-smi --query-gpu=name --format=csv,noheader | head -1"),
            "gpu_count": int(os.environ.get("WORLD_SIZE", "1")),
            "driver": _sh("nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1"),
            "python": platform.python_version(),
        },
        "scheduler": {
            "slurm_job_id": os.environ.get("SLURM_JOB_ID"),
            "nodelist": os.environ.get("SLURM_JOB_NODELIST"),
        },
        "seeds": {"global": config.get("seed"), "data_order": config.get("data_seed")},
    }


def _safe_version(p: str) -> bool:
    try:
        md.version(p)
        return True
    except md.PackageNotFoundError:
        return False

One more thing about seeds. A single global seed isn't enough. You need a separate seed for data ordering, so you can run an experiment that changes only the data order while keeping model initialization fixed. And because each data-loader worker derives its own seed, changing the worker count changes the data order. Worker count is an experimental variable, not a performance knob.

Data Pipelines and Contamination Management

Training data has to be an immutable snapshot, not a "folder." If you only record a path, and the contents at that path change later, the run manifest becomes a lie. In practice, this usually looks like:

• The originals go up to object storage in a single write and are never overwritten.
• What training actually references is a manifest holding the file list, each file's hash, and the sample count.
• The manifest itself is versioned, and the run manifest records the manifest's hash.

Contamination is managed on top of this. If a question or answer from the eval set ends up in the training corpus, the score goes up while the actual capability stays the same. There are two minimum lines of defense.

# Roughly scans for n-gram overlap between the eval set and the training corpus.
# Not a precise tool — a first-pass filter that catches "obvious accidents."
from datasketch import MinHash, MinHashLSH   # pip install datasketch


def shingles(text: str, n: int = 13) -> set:
    toks = text.split()
    return {" ".join(toks[i:i + n]) for i in range(max(1, len(toks) - n + 1))}


def build_index(eval_docs, threshold: float = 0.8):
    lsh = MinHashLSH(threshold=threshold, num_perm=128)
    for i, doc in enumerate(eval_docs):
        m = MinHash(num_perm=128)
        for s in shingles(doc):
            m.update(s.encode())
        lsh.insert(f"eval-{i}", m)
    return lsh


def scan(train_docs, lsh) -> list:
    hits = []
    for j, doc in enumerate(train_docs):
        m = MinHash(num_perm=128)
        for s in shingles(doc):
            m.update(s.encode())
        found = lsh.query(m)
        if found:
            hits.append((j, found))
    return hits

The second line of defense is simpler and more powerful: keep a holdout set that never once goes into the training pipeline. Store it in a separate location, walled off from anything the training cluster can reach. A contamination scan can miss things, but data the pipeline never had access to in the first place cannot be contaminated.

A case from the open-model side that tackled this problem head-on is worth citing. Allen AI's Olmo 3 published its checkpoints, datasets, and dependencies in full, so a third party could directly audit it for contamination. Most internal projects can't match that level of openness, but they can still adopt "is this auditable?" as a design criterion.

Checkpoint Management — Interval and Retention Are Different Problems

Don't conflate the two questions. "How often do you save" is about failure preparedness; "how long do you keep it" is about audit and cost.

Interval is backed out from the failure interval. Young and Daly's first-order approximation says the optimal interval is the square root of twice the product of the save time and the mean time between failures.

import math


def optimal_interval_min(save_minutes: float, mtbf_hours: float) -> float:
    """Young/Daly first-order approximation: T = sqrt(2 * delta * M)"""
    return math.sqrt(2 * save_minutes * mtbf_hours * 60)


def wasted_fraction(interval_min: float, save_minutes: float, mtbf_hours: float) -> float:
    """Fraction = time spent saving + average recomputation time lost to failures"""
    mtbf_min = mtbf_hours * 60
    return save_minutes / interval_min + (interval_min / 2) / mtbf_min


for gpus, mtbf in [(64, 120.0), (512, 15.0), (16384, 3.1)]:
    t = optimal_interval_min(save_minutes=5, mtbf_hours=mtbf)
    print(f"GPU {gpus:6}  MTBF {mtbf:6.1f}h  optimal interval {t:6.1f}min  "
          f"wasted {wasted_fraction(t, 5, mtbf):.1%}")
# GPU     64  MTBF  120.0h  optimal interval  268.3min  wasted   3.7%
# GPU    512  MTBF   15.0h  optimal interval   94.9min  wasted  10.5%
# GPU  16384  MTBF    3.1h  optimal interval   43.1min  wasted  23.2%

The MTBF of 3.1 hours in the last row isn't an arbitrary number. It's the reported 419 unexpected interruptions over 54 days from Llama 3 405B training, converted into an interval. At this scale, even sticking to the optimal interval means more than 20 percent of total time disappears into saving and recomputation. Once you scale up, checkpointing stops being a side task and becomes a major cost line item.

Retention is a separate matter. Saving the full training state of a 70B model comes out to roughly this.

Leave behind 980GB every 43 minutes and that's 33TB a day. Over 20 days, that's 660TB. That's why you need a retention policy. In practice, the common approach is to split it into tiers.

1. Resumption: Keep only the most recent 2-3 and delete the rest immediately. Holds the full state.
2. Milestone: Keep only the inference weights at a fixed token cadence (e.g., every 100B tokens). Used for scaling-curve analysis and postmortems.
3. Release: Permanently retain only what was actually deployed. Always bundle the run manifest and evaluation report with it.

Putting Evaluation Into CI

If a human runs evaluation by hand, something always gets skipped. But running the full evaluation suite on every commit isn't feasible either. So you split it into tiers.

What the smoke tier catches isn't quality, it's accidents. A broken chat template, special tokens added to the tokenizer but not reflected downstream, an end-of-sequence token that changed. These accidents surface as exceptions rather than evaluation scores, so they get caught fast and with certainty.

# Example regression tier. Pin the evaluation tool and task versions together.
pip install "lm-eval==0.4.12"

lm_eval --model hf \
  --model_args pretrained=./out/sft-8b,dtype=bfloat16 \
  --tasks arc_challenge,hellaswag,gsm8k \
  --batch_size 16 \
  --seed 1234 \
  --output_path reports/$(date +%F)-sft-8b.json \
  --log_samples

Turning on --log_samples to keep the generations themselves is important. If you only keep the scores, you can't later answer "why did it drop." Record the evaluation tool version and task definitions alongside the scores too — the same benchmark can produce a different score when the harness version changes. The lm-evaluation-harness version I checked was 0.4.12, released on May 11, 2026.

The evaluation design itself is covered separately in LLM evaluation without the vibes.

Registry and Promotion — Where Training Hands Off to Serving

A model registry isn't a file store, it's an evidence-attachment procedure. All it takes is deciding, for each stage, what has to be attached before you can move to the next one.

The points where the handoff actually breaks are, in most cases, the following six. The order is by frequency, in my experience.

1. Chat template mismatch. The template used during training differs from the one the serving engine applies. A single space or line break out of place, and the model sees a distribution it never trained on. Hash the template string inside the checkpoint's tokenizer_config.json on both the training side and the serving side, and compare.
2. Embedding size mismatch after adding special tokens. Adding tokens during SFT grows the embedding matrix; if the serving-side config still holds the original vocab size, loading fails, or it silently uses the wrong tokens.
3. Missing end-of-sequence token configuration. Generation doesn't stop and runs to the max length. Latency and cost multiply, and users get rambling tacked onto the end of their responses.
4. Whether the LoRA adapter got merged. Hand an unmerged adapter to serving and only the base model loads — the entire fine-tuning effect vanishes. It's especially common for evaluation to run with the adapter attached while serving runs without it.
5. Precision drift. It's common for training to run in bf16, evaluation in fp32, and serving in fp8. You need to run evaluation at least once at the precision and quantization serving will actually use.
6. Padding direction and max position length. The convention is right-padding during training and left-padding during generation; swap them and quality collapses only in batched generation. A single-example test won't catch it.

# Fingerprint comparison before promotion — checks that the training artifact and the serving config are looking at the same thing
import hashlib
import json
from transformers import AutoTokenizer, AutoConfig


def fingerprint(path: str) -> dict:
    tok = AutoTokenizer.from_pretrained(path)
    cfg = AutoConfig.from_pretrained(path)
    tmpl = tok.chat_template or ""
    return {
        "chat_template_sha": hashlib.sha256(tmpl.encode()).hexdigest()[:16],
        "vocab_size_tokenizer": len(tok),
        "vocab_size_config": cfg.vocab_size,
        "eos_token_id": tok.eos_token_id,
        "pad_token_id": tok.pad_token_id,
        "max_position_embeddings": getattr(cfg, "max_position_embeddings", None),
        "torch_dtype": str(getattr(cfg, "torch_dtype", None)),
    }


a, b = fingerprint("./out/sft-8b"), fingerprint("./serving/model")
diff = {k: (a[k], b[k]) for k in a if a[k] != b[k]}
print(json.dumps(diff, indent=2, ensure_ascii=False) if diff else "match")
assert a["vocab_size_tokenizer"] == a["vocab_size_config"], "vocab size mismatch"

After Deployment — Regression Detection and Rollback

The model isn't the only thing that changes after deployment. The serving engine version, prompts, the retrieval index, and upstream application code all change independently. So the ability to pin down what changed when quality degrades is the essence of regression detection.

Wire up three things and you'll catch most of it.

• Golden-set replay. Right after deployment, and at a fixed time every day, run the same few hundred inputs through a deterministic configuration and save the results. Compare the differences against the previous results at the text level. A list of diffs is more useful than a score.
• Output distribution monitoring. Response-length distribution, refusal rate, schema-validation failure rate for structured output, the rate of generations that hit max length without an end-of-sequence token. None of these four are quality metrics, but as accident indicators they're nearly perfect.
• Per-request tracing. Bundle the prompt, model version, prompt version, retrieval document identifiers, latency, and token count into a single trace. Standardization here falls to OpenTelemetry's GenAI semantic conventions; as of July 2026, I found language stating that this convention is still in development and not yet stable. I wasn't able to confirm the status marking directly in the official spec document myself, so check the stability grade in the convention's repository yourself before adopting it. The practical implication is simple: attribute names can change, so keep your instrumentation code behind a thin adapter.

Rollback should be a configuration change, not a rebuild. Store the previous version's weights, tokenizer, prompts, and serving engine version as a single immutable bundle, and make deployment nothing more than flipping a pointer at that bundle. With adapter-based deployment, rollback gets especially cheap — you leave the base model in place and just swap the adapter.

One thing that's often left out of rollback: the prompt and the model have to roll back together. If you tweaked the prompt to match a new model and then only revert the model, that combination has never once been evaluated.

Tool Map — By Category

I won't recommend specific products. Just categories and selection criteria. Versions are as checked on August 2, 2026.

If you had to sum up the selection criteria in one line, it's this: does the record survive even if the tool disappears? If your experiment-tracking server dies but the run manifest JSON is still sitting in object storage, you can recover. If, on the other hand, all your metadata lives only inside a SaaS product, your organization's training history ends the day that contract does.

Closing — A Result You Can't Reproduce Isn't a Result

The tool list for LLM Ops changes every year, but the responsibilities don't. Can you rebuild this run? Can you trust this score? Can you say where this model came from? Can you roll it back within minutes when something goes wrong?

If you can answer "yes" to all four, it doesn't matter which tools you use. If even one is "no," bolting on one more tool won't fix it. The final post in this series looks at how these principles actually broke down, and were recovered, in real large-scale training, drawn from published training case studies.