https://www.youngju.dev/blog 천천히 올바르게. AI Researcher & DevOps Engineer Youngju's blog. GPU/CUDA, LLM, MLOps, Kubernetes AI workloads, and data engineering — plus mindset essays on confidence, routines, health, and sport psychology. ko fjvbn2003@gmail.com (Youngju Kim) fjvbn2003@gmail.com (Youngju Kim) Wed, 12 Aug 2026 00:00:00 GMT https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-attention-variants.en https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-attention-variants.en Comparing MHA, MQA, GQA, and MLA using real config values. Working out with formulas and numbers how the grouped-query attention in Qwen3, Mixtral, and GLM-4.5 and the latent attention in DeepSeek-V3 and Kimi K2 reduce per-token KV cache, and what each design gives up in return. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsattentiongqamlakv-cache https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-attention-variants.ja https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-attention-variants.ja MHA、MQA、GQA、MLA を実際の config 値で比較します。Qwen3、Mixtral、GLM-4.5 のグループクエリアテンションと、DeepSeek-V3、Kimi K2 の潜在アテンションがトークンあたりの KV キャッシュを何倍縮めるのかを式と数字で計算し、その代償に何を差し出すのかを整理します。 Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsattentiongqamlakv-cache https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-attention-variants https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-attention-variants MHA, MQA, GQA, MLA를 실제 config 값으로 비교합니다. Qwen3, Mixtral, GLM-4.5의 그룹 쿼리 어텐션과 DeepSeek-V3, Kimi K2의 잠재 어텐션이 토큰당 KV 캐시를 몇 배 줄이는지 공식과 숫자로 계산하고, 그 대가로 무엇을 내주는지 정리합니다. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsattentiongqamlakv-cache https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-how-to-read-a-config-json.en https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-how-to-read-a-config-json.en How every field in config.json — hidden_size, num_hidden_layers, num_attention_heads versus num_key_value_heads, head_dim, intermediate_size, rope_theta, vocab_size, tie_word_embeddings — shows up in memory and speed, and a hand count of Qwen3-8B and Mixtral-8x7B parameters that matches the published tensor totals exactly. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsconfig-jsontransformerllm-architectureparameter-count https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-how-to-read-a-config-json.ja https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-how-to-read-a-config-json.ja hidden_size、num_hidden_layers、num_attention_heads と num_key_value_heads、head_dim、intermediate_size、rope_theta、vocab_size、tie_word_embeddings まで、config.json の各フィールドがメモリと速度にどう現れるかを説明し、Qwen3-8B と Mixtral-8x7B のパラメータ数を手で数えて公開されたテンソル数と正確に一致させる過程を示します。 Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsconfig-jsontransformerllm-architectureparameter-count https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-how-to-read-a-config-json https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-how-to-read-a-config-json hidden_size, num_hidden_layers, num_attention_heads와 num_key_value_heads, head_dim, intermediate_size, rope_theta, vocab_size, tie_word_embeddings까지 config.json의 모든 필드가 메모리와 속도에 어떻게 나타나는지 설명하고, Qwen3-8B와 Mixtral-8x7B의 파라미터 수를 손으로 세어 공개된 텐서 수와 정확히 일치시키는 과정을 보여줍니다. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsconfig-jsontransformerllm-architectureparameter-count https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-moe-routing.en https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-moe-routing.en Reading the config fields of a mixture-of-experts layer against real models. Comparing Mixtral 2-of-8, Qwen3 8-of-128, the 256 routed experts plus a shared expert in DeepSeek-V3, and the sparsity of 48 in Kimi K2, then working through why active and total parameters produce different costs and what load-balancing losses trade against auxiliary-loss-free bias schemes. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsmixture-of-expertsmoeroutingload-balancing https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-moe-routing.ja https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-moe-routing.ja 専門家混合層の config フィールドを実際のモデルで読みます。Mixtral の 8 個中 2 個、Qwen3 の 128 個中 8 個、DeepSeek-V3 の 256 個のルーティング専門家と共有専門家、Kimi K2 の希薄度 48 を比較し、活性パラメータと全パラメータがなぜ異なるコストを生むのか、ロードバランシング損失と補助損失なしのバイアス方式が何を引き換えにするのかを整理します。 Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsmixture-of-expertsmoeroutingload-balancing https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-moe-routing https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-moe-routing 전문가 혼합 계층의 config 필드를 실제 모델로 읽습니다. Mixtral의 8개 중 2개, Qwen3의 128개 중 8개, DeepSeek-V3의 256개 라우팅 전문가와 공유 전문가, Kimi K2의 희소도 48을 비교하고, 활성 파라미터와 전체 파라미터가 왜 다른 비용을 만드는지, 로드 밸런싱 손실과 보조 손실 없는 편향 방식이 무엇을 맞바꾸는지 정리합니다. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsmixture-of-expertsmoeroutingload-balancing https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation.en https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation.en Confirming from config values why RMSNorm, pre-norm, and SwiGLU became the defaults, then walking through the newer devices that stop attention logits from exploding — the QK-Norm of Qwen3 and the QK-Clip of Kimi K2 — with the figures their reports state. What gets added for stability, and what gets accepted in return. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsrmsnormswiglutraining-stabilityqk-norm https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation.ja https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation.ja RMSNorm とプレノルム、SwiGLU がなぜ現在の標準になったのかを config の値で確認し、Qwen3 の QK-Norm や Kimi K2 の QK-Clip のようにアテンションロジットの暴走を止める新しい仕組みを、各レポートが記す数値とともに整理します。安定性のために何を足し、何を引き受けるのかを扱います。 Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsrmsnormswiglutraining-stabilityqk-norm https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation RMSNorm과 프리노름, SwiGLU가 왜 지금의 표준이 되었는지 config 값으로 확인하고, Qwen3의 QK-Norm과 Kimi K2의 QK-Clip처럼 어텐션 로짓 폭주를 막는 최신 장치를 실제 리포트의 수치와 함께 정리합니다. 안정성을 위해 무엇을 추가하고 무엇을 감수하는지 다룹니다. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsrmsnormswiglutraining-stabilityqk-norm https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-positional-encoding.en https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-positional-encoding.en Showing through wavelength arithmetic how a single rope_theta value acts on context length, then comparing the 500000 of Llama 3, the ABF step in Qwen3, the YaRN settings of DeepSeek-V3 and Kimi K2, and the partial rotation in GLM-4.5 using real configs. Concludes with why long context is not free, in prefill compute and cache cost. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsropepositional-encodinglong-contextyarn https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-positional-encoding.ja https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-positional-encoding.ja rope_theta 一つが文脈長にどう作用するかを波長の計算で示し、Llama 3 の 500000、Qwen3 の ABF、DeepSeek-V3 と Kimi K2 の YaRN 設定、GLM-4.5 の部分回転を実際の config で比較します。長い文脈がなぜ無料でないのかを、プリフィルの演算量とキャッシュのコストから整理します。 Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsropepositional-encodinglong-contextyarn https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-positional-encoding https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-positional-encoding rope_theta 하나가 문맥 길이에 어떻게 작용하는지 파장 계산으로 보여 주고, Llama 3의 500000, Qwen3의 ABF, DeepSeek-V3와 Kimi K2의 YaRN 설정, GLM-4.5의 부분 회전을 실제 config로 비교합니다. 긴 컨텍스트가 왜 공짜가 아닌지 프리필 연산량과 캐시 비용으로 정리합니다. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsropepositional-encodinglong-contextyarn https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-reading-a-tech-report.en https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-reading-a-tech-report.en How to separate the verifiable from the unverifiable in a model technical report. Covers the limits of self-reported benchmarks, the places where a config and a report disagree, values that could not be read from gated repositories, and the verification procedure used throughout this series, illustrated with real cases. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalstech-reportbenchmarksevaluationreproducibility https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-reading-a-tech-report.ja https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-reading-a-tech-report.ja モデル技術レポートで検証可能な項目と検証不可能な項目を区別する方法を整理します。自己申告ベンチマークの限界、config とレポートが食い違う箇所、ゲートされたリポジトリで確認できなかった値、そしてシリーズ全体で用いた確認手順を実例とともに示します。 Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalstech-reportbenchmarksevaluationreproducibility https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-reading-a-tech-report https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-reading-a-tech-report 모델 기술 리포트에서 검증 가능한 항목과 검증 불가능한 항목을 구분하는 법을 정리합니다. 자체 보고 벤치마크의 한계, config와 리포트가 어긋나는 지점, 게이트된 저장소에서 확인할 수 없는 값, 그리고 시리즈 전체에서 쓴 확인 절차를 실제 사례로 보여 줍니다. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalstech-reportbenchmarksevaluationreproducibility https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals.en https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals.en How byte-level BPE works, then downloading the actual tokenizer files of Qwen3, DeepSeek-V3, and Mixtral to tokenize the same English and Korean text and compare. Covers the vocabulary-size tradeoff, why the config vocab_size differs from the real vocabulary count, and how token counts feed into cost and the context window. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalstokenizerbpekorean-nlpvocab-size https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals.ja https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals.ja バイトレベル BPE の仕組みを説明し、Qwen3、DeepSeek-V3、Mixtral の実際のトークナイザファイルをダウンロードして同じ意味の英語と韓国語の文章を直接トークン化して比較します。語彙サイズのトレードオフ、config の vocab_size と実際の語彙数が異なる理由、トークン数がコストと文脈窓に与える影響を整理します。 Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalstokenizerbpekorean-nlpvocab-size https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-tokenizer-internals 바이트 수준 BPE가 어떻게 동작하는지 설명하고, Qwen3와 DeepSeek-V3, Mixtral의 실제 토크나이저 파일을 내려받아 같은 뜻의 영어와 한국어 문장을 직접 토큰화해 비교합니다. 어휘 크기의 트레이드오프, config의 vocab_size와 실제 어휘 수가 다른 이유, 그리고 토큰 수가 비용과 문맥 창에 미치는 영향을 정리합니다. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalstokenizerbpekorean-nlpvocab-size https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-training-recipe.en https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-training-recipe.en Comparing, exactly as written in the reports, the three stages and six context extensions of Llama 3, the three-stage pre-training of Qwen3, the learning-rate schedule and two-phase YaRN extension of DeepSeek-V3, and the data-rephrasing experiment of Kimi K2, then setting out how to read a training-stage description. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalspretrainingtraining-recipedata-mixturelong-context https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-training-recipe.ja https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-training-recipe.ja Llama 3 の三段階と六回の文脈拡張、Qwen3 の三段階事前学習、DeepSeek-V3 の学習率スケジュールと二段階 YaRN 拡張、Kimi K2 のデータ再記述実験を、レポートに書かれたとおりに比較し、学習段階の記述を読む方法を整理します。 Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalspretrainingtraining-recipedata-mixturelong-context https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-training-recipe https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-training-recipe Llama 3의 세 단계와 여섯 번의 컨텍스트 확장, Qwen3의 세 단계 사전학습, DeepSeek-V3의 학습률 스케줄과 두 단계 YaRN 확장, Kimi K2의 데이터 재작성 실험을 리포트에 적힌 그대로 비교하고, 학습 단계 기술을 읽는 법을 정리합니다. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalspretrainingtraining-recipedata-mixturelong-context