https://www.youngju.dev/blog 천천히 올바르게. AI Researcher & DevOps Engineer Youngju's tech blog. GPU/CUDA, LLM, MLOps, Kubernetes AI workloads, distributed training, and data engineering. ko fjvbn2003@gmail.com (Youngju Kim) fjvbn2003@gmail.com (Youngju Kim) Sat, 16 May 2026 00:00:00 GMT https://www.youngju.dev/blog/culture/2026-05-16-distributed-training-deepspeed-fsdp-megatron-ray-jax-fsdp2-torchtitan-blackwell-2026-deep-dive.en https://www.youngju.dev/blog/culture/2026-05-16-distributed-training-deepspeed-fsdp-megatron-ray-jax-fsdp2-torchtitan-blackwell-2026-deep-dive.en A comparison of DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, and Composer — plus NVIDIA Blackwell GB200 NVL72, AMD MI325X, Intel Gaudi 3, AWS Trainium 2, and Google TPU v5p/v6e Trillium. 3D parallelism, ZeRO/FSDP equivalence, MoE All-to-All, fp8/mxfp4 precision, NCCL tuning, checkpointing, and failure recovery — LLM training infra as of mid-2026. Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) distributed-trainingdeepspeedfsdpmegatron-lmrayjaxlightningacceleratetorchtitancudancclblackwelltpullm-traininggpu-infrastructure https://www.youngju.dev/blog/culture/2026-05-16-distributed-training-deepspeed-fsdp-megatron-ray-jax-fsdp2-torchtitan-blackwell-2026-deep-dive.ja https://www.youngju.dev/blog/culture/2026-05-16-distributed-training-deepspeed-fsdp-megatron-ray-jax-fsdp2-torchtitan-blackwell-2026-deep-dive.ja DeepSpeed/FSDP2/Megatron-LM/Ray Train/JAX/TorchTitan/Composerを比較し、NVIDIA Blackwell GB200 NVL72、AMD MI325X、Intel Gaudi 3、AWS Trainium 2、Google TPU v5p/v6e Trillimuまで。3D並列化、ZeRO-FSDP等価性、MoEのAll-to-All、fp8/mxfp4精度、NCCLチューニング、チェックポイント、障害復旧までLLM学習インフラ2026年現在形。 Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) distributed-trainingdeepspeedfsdpmegatron-lmrayjaxlightningacceleratetorchtitancudancclblackwelltpullm-traininggpu-infrastructure https://www.youngju.dev/blog/culture/2026-05-16-distributed-training-deepspeed-fsdp-megatron-ray-jax-fsdp2-torchtitan-blackwell-2026-deep-dive https://www.youngju.dev/blog/culture/2026-05-16-distributed-training-deepspeed-fsdp-megatron-ray-jax-fsdp2-torchtitan-blackwell-2026-deep-dive DeepSpeed/FSDP2/Megatron-LM/Ray Train/JAX/TorchTitan/Composer를 비교하고, NVIDIA Blackwell GB200 NVL72, AMD MI325X, Intel Gaudi 3, AWS Trainium 2, Google TPU v5p/v6e Trillium까지. 3D 병렬화, ZeRO-FSDP 등가성, MoE All-to-All, fp8/mxfp4 정밀도, NCCL 튜닝, 체크포인팅, 실패 복구까지 LLM 학습 인프라 2026년 현재형. Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) distributed-trainingdeepspeedfsdpmegatron-lmrayjaxlightningacceleratetorchtitancudancclblackwelltpullm-traininggpu-infrastructure