https://www.youngju.dev/blog 천천히 올바르게. AI Researcher & DevOps Engineer Youngju's blog. GPU/CUDA, LLM, MLOps, Kubernetes AI workloads, and data engineering — plus mindset essays on confidence, routines, health, and sport psychology. ko fjvbn2003@gmail.com (Youngju Kim) fjvbn2003@gmail.com (Youngju Kim) Wed, 12 Aug 2026 00:00:00 GMT https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals.en https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals.en How byte-level BPE works, then downloading the actual tokenizer files of Qwen3, DeepSeek-V3, and Mixtral to tokenize the same English and Korean text and compare. Covers the vocabulary-size tradeoff, why the config vocab_size differs from the real vocabulary count, and how token counts feed into cost and the context window. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalstokenizerbpekorean-nlpvocab-size https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals.ja https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals.ja バイトレベル BPE の仕組みを説明し、Qwen3、DeepSeek-V3、Mixtral の実際のトークナイザファイルをダウンロードして同じ意味の英語と韓国語の文章を直接トークン化して比較します。語彙サイズのトレードオフ、config の vocab_size と実際の語彙数が異なる理由、トークン数がコストと文脈窓に与える影響を整理します。 Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalstokenizerbpekorean-nlpvocab-size https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals 바이트 수준 BPE가 어떻게 동작하는지 설명하고, Qwen3와 DeepSeek-V3, Mixtral의 실제 토크나이저 파일을 내려받아 같은 뜻의 영어와 한국어 문장을 직접 토큰화해 비교합니다. 어휘 크기의 트레이드오프, config의 vocab_size와 실제 어휘 수가 다른 이유, 그리고 토큰 수가 비용과 문맥 창에 미치는 영향을 정리합니다. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalstokenizerbpekorean-nlpvocab-size