https://www.youngju.dev/blog 천천히 올바르게. AI Researcher & DevOps Engineer Youngju's blog. GPU/CUDA, LLM, MLOps, Kubernetes AI workloads, and data engineering — plus mindset essays on confidence, routines, health, and sport psychology. ko fjvbn2003@gmail.com (Youngju Kim) fjvbn2003@gmail.com (Youngju Kim) Wed, 12 Aug 2026 00:00:00 GMT https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation.en https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation.en Confirming from config values why RMSNorm, pre-norm, and SwiGLU became the defaults, then walking through the newer devices that stop attention logits from exploding — the QK-Norm of Qwen3 and the QK-Clip of Kimi K2 — with the figures their reports state. What gets added for stability, and what gets accepted in return. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsrmsnormswiglutraining-stabilityqk-norm https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation.ja https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation.ja RMSNorm とプレノルム、SwiGLU がなぜ現在の標準になったのかを config の値で確認し、Qwen3 の QK-Norm や Kimi K2 の QK-Clip のようにアテンションロジットの暴走を止める新しい仕組みを、各レポートが記す数値とともに整理します。安定性のために何を足し、何を引き受けるのかを扱います。 Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsrmsnormswiglutraining-stabilityqk-norm https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation https://www.youngju.dev/blog/ai-papers/2026-08-12-model-internals-normalization-and-activation RMSNorm과 프리노름, SwiGLU가 왜 지금의 표준이 되었는지 config 값으로 확인하고, Qwen3의 QK-Norm과 Kimi K2의 QK-Clip처럼 어텐션 로짓 폭주를 막는 최신 장치를 실제 리포트의 수치와 함께 정리합니다. 안정성을 위해 무엇을 추가하고 무엇을 감수하는지 다룹니다. Wed, 12 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) ai-papersmodel-internalsrmsnormswiglutraining-stabilityqk-norm