https://www.youngju.dev/blog 천천히 올바르게. AI Researcher & DevOps Engineer Youngju's tech blog. GPU/CUDA, LLM, MLOps, Kubernetes AI workloads, distributed training, and data engineering. ko fjvbn2003@gmail.com (Youngju Kim) fjvbn2003@gmail.com (Youngju Kim) Thu, 09 Jul 2026 00:00:00 GMT https://www.youngju.dev/blog/2026-07-09-llm-data-preprocessing.en https://www.youngju.dev/blog/2026-07-09-llm-data-preprocessing.en Good models come from good data, and good data comes from a preprocessing pipeline. This post walks through the entire pretraining data process step by step — web crawl collection → text extraction → language identification → heuristic and classifier-based quality filtering → exact and near (MinHash) deduplication → PII and toxicity handling → benchmark decontamination → tokenization and sequence packing — and also covers the essentials of SFT data curation, tools like datatrove, Dolma, and NeMo Curator, and the recent papers that changed the field, from The Pile to FineWeb and DCLM. Thu, 09 Jul 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) aillmdata-engineeringpreprocessingtraining https://www.youngju.dev/blog/2026-07-09-llm-data-preprocessing.ja https://www.youngju.dev/blog/2026-07-09-llm-data-preprocessing.ja よいモデルはよいデータから生まれ、よいデータは前処理パイプラインから生まれます。Webクロール収集 → 本文抽出 → 言語識別 → ヒューリスティック・分類器による品質フィルタリング → 厳密・近似(MinHash)重複除去 → PII・有害性処理 → ベンチマーク汚染除去 → トークナイズとシーケンスパッキングまで、事前学習データの全工程を段階ごとに解説します。あわせてSFTデータ整備の要点、datatrove・Dolma・NeMo Curatorといったツール、そしてThe PileからFineWeb・DCLMまで、この分野の流れを変えた最新論文も紹介します。 Thu, 09 Jul 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) aillmdata-engineeringpreprocessingtraining https://www.youngju.dev/blog/2026-07-09-llm-data-preprocessing https://www.youngju.dev/blog/2026-07-09-llm-data-preprocessing 좋은 모델은 좋은 데이터에서 나오고, 좋은 데이터는 전처리 파이프라인에서 나옵니다. 웹 크롤 수집 → 본문 추출 → 언어 식별 → 휴리스틱·분류기 품질 필터링 → 정확·근사(MinHash) 중복 제거 → PII·독성 처리 → 벤치마크 오염 제거 → 토크나이즈와 시퀀스 패킹까지 사전학습 데이터의 전체 공정을 단계별로 설명하고, SFT 데이터 정제의 요점, datatrove·Dolma·NeMo Curator 같은 도구, 그리고 The Pile부터 FineWeb·DCLM까지 이 분야의 흐름을 바꾼 최신 논문들을 함께 소개합니다. Thu, 09 Jul 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) aillmdata-engineeringpreprocessingtraining