Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — Where 80% of Model Quality Is Decided
A Bird's-Eye View of the Full Pipeline
① Text Extraction — Only the Real Content from HTML
② Language Identification — A Fast, Quiet Gate
③ Quality Filtering — From Heuristics to Classifiers
④ Deduplication — The Most Underrated Stage for Its Impact
⑤·⑥ PII, Toxicity, and Decontamination
⑦ Tokenization and Sequence Packing — The Final Step
SFT Data Is Different — A Small, Elite World
The Toolbox — No Need to Build Everything Yourself
The Papers That Changed the Field — A Recommended Reading Order
Closing Thoughts
References

Introduction — Where 80% of Model Quality Is Decided

In the AI model development lifecycle post, I wrote that "1GB of good data beats 1TB of garbage." This post is the hands-on companion to that sentence — we walk through the process raw internet text actually goes through to become trainable tokens, along with the tools for each stage and the papers that established it. One of the most interesting findings of recent years is that the gap between frontier labs opens up in this very pipeline, more than in model architecture (the FineWeb and DCLM papers proved it with open data).

A Bird's-Eye View of the Full Pipeline

Collection (CommonCrawl, etc.)
  → ① Text extraction (HTML → text)
  → ② Language identification
  → ③ Quality filtering (heuristics → model classifier)
  → ④ Deduplication (exact → near, MinHash)
  → ⑤ PII & toxicity handling
  → ⑥ Benchmark decontamination
  → ⑦ Tokenization + sequence packing
  → Training

The order is deliberate — cheap filters go first and expensive ones last, so the costly operations (classifiers, MinHash) see fewer documents. Let's look at each stage.

① Text Extraction — Only the Real Content from HTML

Most pretraining data originates from CommonCrawl (a periodic crawl archive of the entire web). The first fork in the road appears here: do you use the pre-extracted WET text, or extract directly from the raw WARC (HTML)? The shared conclusion of RefinedWeb and FineWeb is that extracting directly from WARC with a tool like trafilatura gives distinctly better quality — WET files are packed with boilerplate such as navigation menus, cookie banners, and footers. The quality of text extraction sets the ceiling for every stage that follows.

② Language Identification — A Fast, Quiet Gate

A fastText-family language classifier determines each document's language, and only the target language is kept (e.g., ko for a Korean dataset). It looks simple, but there are traps — misclassification is frequent on short documents and documents that mix code with text, and the lower-resource the language, the more good data gets thrown away at this gate. If you are building a multilingual model, tuning thresholds per language is the standard move.

③ Quality Filtering — From Heuristics to Classifiers

Two generations of techniques are used together.

Heuristic rules (the Gopher rules family) — dozens of statistical rules, such as document length, mean word length, symbol-to-word ratio, bullet and hashtag ratios, and whether the text contains "lorem ipsum", weed out the obvious garbage. The Gopher paper established them, and almost every pipeline uses some variant.

Model-based classifiers — the recent battleground. FineWeb-Edu trained a small classifier on labels produced by having an LLM grade each document's "educational value" and used it to select a high-quality subset, and DCLM showed that a single fastText classifier trained on good data (instruct-style) can beat sophisticated pipeline combinations. The lesson: shift the definition of "quality" from the data itself to the criterion you train the classifier on, and the pipeline gets simpler while performance goes up.

One caution here — filters always bias the distribution. An educational-value filter shaves off colloquial language, minority dialects, and creative writing. The habit of eyeballing samples of what you are throwing away is half of filter tuning.

④ Deduplication — The Most Underrated Stage for Its Impact

A substantial share of any web crawl is duplicates (mirror sites, boilerplate, reposts). Duplicates effectively run certain texts for multiple epochs during training, which inflates memorization and benchmark contamination and wastes compute.

Exact deduplication: remove perfect matches by hashing documents (or paragraphs). Cheap and reliable.
Near deduplication — MinHash LSH: the standard technique for catching "almost identical" documents. Turn each document into a set of n-grams (shingles), approximate Jaccard similarity with MinHash signatures, then use LSH banding to narrow the comparison pairs down to candidates — which is why it works on billions of documents without comparing everything against everything.
Line-level deduplication: remove lines that repeat across many documents (navigation text and so on). A technique RefinedWeb emphasized.

There is an interesting twist as well — FineWeb reported that per-snapshot deduplication produced better results than aggressively deduplicating across all crawl snapshots. Even deduplication is not "the more the better" — it is something to measure and tune.

⑤·⑥ PII, Toxicity, and Decontamination

PII handling: patterns such as emails, phone numbers, and IP addresses are detected with regular expressions, then removed or replaced with placeholders (the Dolma toolkit is a good open implementation of this stage). Legal review not being a luxury is exactly what I said in the model development post.

Benchmark decontamination: if items from evaluation sets (MMLU, GSM8K, and so on) are mixed into the training data, benchmark scores go up without any real capability. Removing training documents that overlap with evaluation-set n-grams is the standard procedure, and the "skepticism when reading a new model's scores" from the quantization post is aimed precisely at the absence of this stage. If you built your own evaluation set, it belongs on the decontamination list too — the data-side version of the eval-first principle.

⑦ Tokenization and Sequence Packing — The Final Step

Cleaned documents become token IDs via a tokenizer (BPE family) and are cut into fixed-length sequences for the trainer. Two practical points here:

Sequence packing: when documents are shorter than the sequence length, padding is wasted. Packing — concatenating multiple documents with EOS tokens to fill each sequence — is the standard, and a growing number of implementations pair it with masking that blocks attention across document boundaries.
Shuffling and mixture ratios: the per-domain mixture (web/code/papers/multilingual) is itself a hyperparameter. Curriculum schemes (annealing) that raise the share of high-quality data late in training are another technique whose benefits were reported in FineWeb-line experiments.

SFT Data Is Different — A Small, Elite World

Fine-tuning (SFT) data follows a different grammar than pretraining. It is a world of thousands to tens of thousands of examples, so per-example quality is everything.

Have humans (or a strong model as a first pass) review responses for accuracy and format consistency,
Ensure instruction diversity (one thousand different patterns beat ten thousand of the same one),
Match the chat template (role markers) exactly between training and inference — template mismatch is a classic cause of silent performance degradation,
Deduplicate and decontaminate just as in pretraining (benchmark items slipping into SFT sets is an especially frequent accident).

The Toolbox — No Need to Build Everything Yourself

Tool                Made by       One-line description
──────────────────  ────────────  ─────────────────────────────────
datatrove           HuggingFace   The pipeline library that built FineWeb
Dolma toolkit       AI2           Filters, dedup, PII taggers (the Dolma dataset process)
NeMo Curator        NVIDIA        GPU-accelerated curation at scale (distributed MinHash, etc.)
trafilatura         Open source   The de facto standard for HTML text extraction
fastText            Meta          Language ID and lightweight quality classifiers

Up to the hundreds-of-gigabytes range, datatrove or the Dolma toolkit is plenty; once you need TB-scale GPU acceleration, NeMo Curator becomes the candidate. Running the pipeline on top of a distributed training cluster is also a common combination.

The Papers That Changed the Field — A Recommended Reading Order

The Pile (2020) — the archetype of open datasets as "a mixture of diverse, high-quality sources."
Gopher/MassiveText (2021) — established the heuristic quality rules.
RefinedWeb (2023) — the twist that "curated web data alone is enough," plus the emphasis on extraction and line-level dedup.
Dolma (2024) — a three-trillion-token process fully in the open — toolkit included.
FineWeb / FineWeb-Edu (2024) — 15 trillion tokens plus published stage-by-stage ablation experiments, and the power of the educational-value classifier.
DCLM (2024) — turned data curation into a benchmark — the power of one good classifier.

Read these six in order, and you will understand as history why today's standard pipeline looks the way it does.

Closing Thoughts

Data preprocessing is not glamorous, but each stage of this process sets one of the model's ceilings — extraction sets the purity of the raw material, filtering the distribution, dedup the memorization, decontamination the honesty of your evaluation, and packing the compute efficiency. And a single habit runs through every stage: keep looking, with your own eyes, at samples of what gets discarded and what remains. The pipeline is code, but the sense of quality ultimately belongs to people.