https://www.youngju.dev/blog 천천히 올바르게. AI Researcher & DevOps Engineer Youngju's tech blog. GPU/CUDA, LLM, MLOps, Kubernetes AI workloads, distributed training, and data engineering. ko fjvbn2003@gmail.com (Youngju Kim) fjvbn2003@gmail.com (Youngju Kim) Sat, 16 May 2026 00:00:00 GMT https://www.youngju.dev/blog/culture/2026-05-16-ai-inference-engines-2026-vllm-sglang-llama-cpp-tgi-tensorrt-llm-mlx-mistralrs-deepspeed-aphrodite-deep-dive.en https://www.youngju.dev/blog/culture/2026-05-16-ai-inference-engines-2026-vllm-sglang-llama-cpp-tgi-tensorrt-llm-mlx-mistralrs-deepspeed-aphrodite-deep-dive.en In 2026, LLM engineering is no longer about which model — it is about which inference engine. We dissect vLLM V1, SGLang 0.4, TensorRT-LLM, TGI 3.x, llama.cpp, MLX-LM, mistral.rs, DeepSpeed-MII, Aphrodite, CTranslate2, ExLlamaV3, OpenVINO, AWS Neuron, Triton — 10+ engines through the lens of PagedAttention, Continuous Batching, Speculative Decoding, Disaggregated Inference, KV quantization, NIM, and Groq LPU. Plus self-hosting ROI math and Korean/Japanese inference infrastructure. Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) llm-inferencevllmsglangllama-cpptgitensorrt-llmmlxmistral-rsdeepspeedaphroditeinferenceenglish https://www.youngju.dev/blog/culture/2026-05-16-ai-inference-engines-2026-vllm-sglang-llama-cpp-tgi-tensorrt-llm-mlx-mistralrs-deepspeed-aphrodite-deep-dive.ja https://www.youngju.dev/blog/culture/2026-05-16-ai-inference-engines-2026-vllm-sglang-llama-cpp-tgi-tensorrt-llm-mlx-mistralrs-deepspeed-aphrodite-deep-dive.ja 2026 年の LLM エンジニアリングはもうモデル選定の問題ではなく、エンジン選定の問題になった。vLLM V1、SGLang 0.4、TensorRT-LLM、TGI 3.x、llama.cpp、MLX-LM、mistral.rs、DeepSpeed-MII、Aphrodite、CTranslate2、ExLlamaV3、OpenVINO、AWS Neuron、Triton — 10 以上のエンジンを PagedAttention・Continuous Batching・Speculative Decoding・Disaggregated Inference・KV 量子化・NIM・Groq LPU の観点から横並びで比較する。セルフホスト ROI 計算と日韓の推論インフラまで。 Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) llm-inferencevllmsglangllama-cpptgitensorrt-llmmlxmistral-rsdeepspeedaphroditeinference日本語 https://www.youngju.dev/blog/culture/2026-05-16-ai-inference-engines-2026-vllm-sglang-llama-cpp-tgi-tensorrt-llm-mlx-mistralrs-deepspeed-aphrodite-deep-dive https://www.youngju.dev/blog/culture/2026-05-16-ai-inference-engines-2026-vllm-sglang-llama-cpp-tgi-tensorrt-llm-mlx-mistralrs-deepspeed-aphrodite-deep-dive 2026년 LLM 추론은 더 이상 모델 선택의 문제가 아니라 엔진 선택의 문제가 됐다. vLLM V1, SGLang 0.4, TensorRT-LLM, TGI 3.x, llama.cpp, MLX-LM, mistral.rs, DeepSpeed-MII, Aphrodite, CTranslate2, ExLlamaV3, OpenVINO, AWS Neuron, Triton — 10개+ 엔진을 PagedAttention·Continuous Batching·Speculative Decoding·Disaggregated Inference·KV 양자화·NIM·Groq LPU 관점에서 한 줄로 비교한다. 자가 호스팅 ROI 계산과 한국·일본 추론 인프라까지. Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) llm-inferencevllmsglangllama-cpptgitensorrt-llmmlxmistral-rsdeepspeedaphroditeinference https://www.youngju.dev/blog/culture/2026-05-16-edge-ai-tinyml-2026-litert-executorch-edge-impulse-jetson-coral-hailo-sipeed-k230-llama-cpp-deep-dive.en https://www.youngju.dev/blog/culture/2026-05-16-edge-ai-tinyml-2026-litert-executorch-edge-impulse-jetson-coral-hailo-sipeed-k230-llama-cpp-deep-dive.en A full-stack map of the 2026 Edge AI / TinyML ecosystem — the dual standard formed after TFLite Micro was rebranded as LiteRT and ExecuTorch reached GA, the TinyML cloud workflow created by Edge Impulse, the accelerator showdown between NVIDIA Jetson Orin Nano/NX/Thor/AGX and Coral Dev Board, the rise of Israeli Hailo-15/8 and Chinese Sipeed K230 (RISC-V + NPU), the SBC standardization of Rockchip RK3588, MaixPy / Arduino Nano 33 BLE Sense / Seeed Wio AI, MicroPython for ML, and on-phone MLC LLM / llama.cpp / Whisper.cpp / GGUF — covering Phi-3/3.5/4, Gemma 2/3, Llama 3.2 1B/3B — and how IoT / mobile / automotive engineers should pick tools in the age of Always-on AI, including case studies from Korea (ETRI / Samsung / LG) and Japan (Sony AI / NTT). Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) edge-aitinymltflite-microlitertexecutorchedge-impulsejetson-orincoral-dev-boardhailosipeed-k230rockchip-rk3588maixpymicropython-mlonnx-runtime-mobilecore-mltensorrtapache-tvmmlc-llmwhisper-cppllama-cppggufphi-3phi-4gemmallama-3-22026deep-diveenglish https://www.youngju.dev/blog/culture/2026-05-16-edge-ai-tinyml-2026-litert-executorch-edge-impulse-jetson-coral-hailo-sipeed-k230-llama-cpp-deep-dive.ja https://www.youngju.dev/blog/culture/2026-05-16-edge-ai-tinyml-2026-litert-executorch-edge-impulse-jetson-coral-hailo-sipeed-k230-llama-cpp-deep-dive.ja 2026 年のエッジ AI / TinyML エコシステムをフルスタックで俯瞰 — TFLite Micro が LiteRT へリブランドされ ExecuTorch が GA となって形成された二大標準、Edge Impulse が築いた TinyML クラウドワークフロー、NVIDIA Jetson Orin Nano/NX/Thor/AGX と Coral Dev Board のアクセラレータ対決、イスラエルの Hailo-15/8 と中国の Sipeed K230(RISC-V+NPU)の台頭、Rockchip RK3588 の SBC 標準化、MaixPy / Arduino Nano 33 BLE Sense / Seeed Wio AI、ML 向け MicroPython、そしてスマホ上で動く MLC LLM / llama.cpp / Whisper.cpp / GGUF — Phi-3/3.5/4、Gemma 2/3、Llama 3.2 1B/3B まで — Always-on AI の時代に IoT / モバイル / 自動車エンジニアがどう道具を選ぶべきかを、韓国(ETRI / Samsung / LG)と日本(Sony AI / NTT)の事例とともにまとめます。 Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) edge-aitinymltflite-microlitertexecutorchedge-impulsejetson-orincoral-dev-boardhailosipeed-k230rockchip-rk3588maixpymicropython-mlonnx-runtime-mobilecore-mltensorrtapache-tvmmlc-llmwhisper-cppllama-cppggufphi-3phi-4gemmallama-3-22026deep-dive日本語 https://www.youngju.dev/blog/culture/2026-05-16-edge-ai-tinyml-2026-litert-executorch-edge-impulse-jetson-coral-hailo-sipeed-k230-llama-cpp-deep-dive https://www.youngju.dev/blog/culture/2026-05-16-edge-ai-tinyml-2026-litert-executorch-edge-impulse-jetson-coral-hailo-sipeed-k230-llama-cpp-deep-dive 2026년 엣지 AI / TinyML 생태계의 풀스택 지도 — TFLite Micro 가 LiteRT 로 리브랜드된 후 ExecuTorch 가 GA 가 되며 형성된 듀얼 표준, Edge Impulse 가 만든 TinyML 클라우드 워크플로, NVIDIA Jetson Orin Nano/NX/Thor/AGX 와 Coral Dev Board 의 가속기 격돌, 이스라엘 Hailo-15/8 과 중국 Sipeed K230(RISC-V+NPU) 의 부상, Rockchip RK3588 의 SBC 표준화, MaixPy/Arduino Nano 33 BLE Sense/Seeed Wio AI, MicroPython for ML, 그리고 폰 위에서 도는 MLC LLM / llama.cpp / Whisper.cpp / GGUF — Phi-3/3.5/4, Gemma 2/3, Llama 3.2 1B/3B 까지 — Always-on AI 의 시대에 IoT / 모바일 / 자동차 엔지니어가 어떤 도구를 어떻게 골라야 하는지 한국(ETRI/삼성/LG)과 일본(Sony AI/NTT) 사례까지 모두 정리합니다. Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) edge-aitinymltflite-microlitertexecutorchedge-impulsejetson-orincoral-dev-boardhailosipeed-k230rockchip-rk3588maixpymicropython-mlonnx-runtime-mobilecore-mltensorrtapache-tvmmlc-llmwhisper-cppllama-cppggufphi-3phi-4gemmallama-3-22026deep-dive https://www.youngju.dev/blog/culture/2026-05-16-llm-serving-local-inference-2026-vllm-llama-cpp-mlx-ollama-lm-studio-sglang-tgi-deep-dive.en https://www.youngju.dev/blog/culture/2026-05-16-llm-serving-local-inference-2026-vllm-llama-cpp-mlx-ollama-lm-studio-sglang-tgi-deep-dive.en A map of the 2026 LLM serving and inference landscape. Datacenter camp (vLLM, SGLang, TGI, Triton, TensorRT-LLM), local camp (llama.cpp, MLX, llamafile, Ollama, LM Studio, GPT4All), emerging camp (KTransformers, MLC LLM, Modular MAX), and cloud serving SaaS (Together, Fireworks, Groq, Cerebras, SambaNova, Lepton acquired by NVIDIA). Quantization formats (GGUF Q4_K_M, AWQ, GPTQ, FP8), plus Korean and Japanese model ecosystems (Upstage Solar, KT Mi:dm, Sakana, NTT Tsuzumi, ELYZA) — who should pick what. Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) llmservinginferencevllmllama-cppmlxllamafileollamalm-studiogpt4allsglangtgiktransformerstensorrt-llmmodular-maxquantization2026deep-diveenglish https://www.youngju.dev/blog/culture/2026-05-16-llm-serving-local-inference-2026-vllm-llama-cpp-mlx-ollama-lm-studio-sglang-tgi-deep-dive.ja https://www.youngju.dev/blog/culture/2026-05-16-llm-serving-local-inference-2026-vllm-llama-cpp-mlx-ollama-lm-studio-sglang-tgi-deep-dive.ja 2026 年の LLM サービング・推論フレームワークの地図を描く。データセンター陣営(vLLM・SGLang・TGI・Triton・TensorRT-LLM)、ローカル陣営(llama.cpp・MLX・llamafile・Ollama・LM Studio・GPT4All)、新興陣営(KTransformers・MLC LLM・Modular MAX)、そしてクラウドサービング SaaS(Together・Fireworks・Groq・Cerebras・SambaNova・NVIDIA が 2025 年に買収した Lepton)。量子化フォーマット(GGUF Q4_K_M・AWQ・GPTQ・FP8)、韓国・日本のモデル生態系(Upstage Solar・KT Mi:dm・Sakana・NTT つづみ・ELYZA)まで — 誰が何を選ぶべきか。 Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) llmservinginferencevllmllama-cppmlxllamafileollamalm-studiogpt4allsglangtgiktransformerstensorrt-llmmodular-maxquantization2026deep-dive日本語 https://www.youngju.dev/blog/culture/2026-05-16-llm-serving-local-inference-2026-vllm-llama-cpp-mlx-ollama-lm-studio-sglang-tgi-deep-dive https://www.youngju.dev/blog/culture/2026-05-16-llm-serving-local-inference-2026-vllm-llama-cpp-mlx-ollama-lm-studio-sglang-tgi-deep-dive 2026년 LLM 서빙·추론 프레임워크의 지도를 그린다. 데이터센터 진영(vLLM·SGLang·TGI·Triton·TensorRT-LLM), 로컬 진영(llama.cpp·MLX·llamafile·Ollama·LM Studio·GPT4All), 신생 진영(KTransformers·MLC LLM·Modular MAX), 그리고 클라우드 서빙 SaaS(Together·Fireworks·Groq·Cerebras·SambaNova·Lepton-NVIDIA 인수). 양자화 포맷(GGUF Q4_K_M·AWQ·GPTQ·FP8), 한국·일본 모델 생태계(Upstage Solar·KT Mi:dm·Sakana·NTT Tsuzumi·ELYZA)까지 — 누가 무엇을 골라야 하나. Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) llmservinginferencevllmllama-cppmlxllamafileollamalm-studiogpt4allsglangtgiktransformerstensorrt-llmmodular-maxquantization2026deep-dive https://www.youngju.dev/blog/culture/2026-05-16-local-ai-on-device-llms-2026-ollama-lm-studio-jan-msty-open-webui-gpt4all-anythingllm-faraday-deep-dive.en https://www.youngju.dev/blog/culture/2026-05-16-local-ai-on-device-llms-2026-ollama-lm-studio-jan-msty-open-webui-gpt4all-anythingllm-faraday-deep-dive.en By May 2026, local AI is no longer a hobby. An M4 Max MacBook Pro runs Llama 4 Scout 109B MoE at 24 tokens per second. Desktop runtimes like Ollama, LM Studio, Jan, and Msty unify GUI and CLI, while Open WebUI, AnythingLLM, and LibreChat deliver ChatGPT-class interfaces. The backends — llama.cpp, MLX-LM, vLLM — hold up the stack, and quantization formats (GGUF, AWQ, GPTQ, EXL3, MXFP4, BitNet) let an 8GB laptop infer 13B models. Llama 3.3 70B, DeepSeek R1 Distill, Qwen 3, Phi-4, Gemma 3, MiniCPM 3, DeepSeek Coder V2 — open models competitive with the cloud ship every week. Meanwhile Apple Intelligence (iOS 18), Phi Silica (Windows Copilot+), and Gemini Nano (Android / Chrome) have standardized OS-level on-device AI. This piece maps the entire local AI landscape as of May 2026 — runtimes, GUIs, backends, quantization, recommended models, and Korean / Japanese local stacks. Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) local-aion-device-llmollamalm-studiojanmstyopen-webuigpt4allanythingllmfaradayllama-cppmlxggufquantizationragprivategptlibrechatkhojreorpinokiochatboxapple-intelligencephi-silicagemini-nano2026deep-diveenglish https://www.youngju.dev/blog/culture/2026-05-16-local-ai-on-device-llms-2026-ollama-lm-studio-jan-msty-open-webui-gpt4all-anythingllm-faraday-deep-dive.ja https://www.youngju.dev/blog/culture/2026-05-16-local-ai-on-device-llms-2026-ollama-lm-studio-jan-msty-open-webui-gpt4all-anythingllm-faraday-deep-dive.ja 2026年5月、ローカルAIはもはや「趣味」ではない。M4 Max MacBook ProがLlama 4 Scout 109B MoEを毎秒24トークンで動かす時代だ。Ollama、LM Studio、Jan、MstyのようなデスクトップランタイムがGUI/CLIを統一し、Open WebUI、AnythingLLM、LibreChatがChatGPT級のインターフェースを提供する。バックエンドはllama.cpp、MLX-LM、vLLMが支え、GGUF / AWQ / GPTQ / EXL3 / MXFP4 / BitNet という量子化フォーマットが、8GB VRAMのノートでも13Bモデルの推論を可能にする。Llama 3.3 70B、DeepSeek R1 Distill、Qwen 3、Phi-4、Gemma 3、MiniCPM 3、DeepSeek Coder V2 — クラウドに匹敵するオープンモデルが毎週リリースされる。一方でApple Intelligence(iOS 18)、Phi Silica(Windows Copilot+)、Gemini Nano(Android / Chrome)はOSレベルのオンデバイスAIを標準化した。本稿はランタイム → GUI → バックエンド → 量子化 → 推奨モデル → 韓国・日本のローカル事例まで、2026年5月時点のローカルAI全景を一気に整理する。 Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) local-aion-device-llmollamalm-studiojanmstyopen-webuigpt4allanythingllmfaradayllama-cppmlxggufquantizationragprivategptlibrechatkhojreorpinokiochatboxapple-intelligencephi-silicagemini-nano2026deep-dive日本語 https://www.youngju.dev/blog/culture/2026-05-16-local-ai-on-device-llms-2026-ollama-lm-studio-jan-msty-open-webui-gpt4all-anythingllm-faraday-deep-dive https://www.youngju.dev/blog/culture/2026-05-16-local-ai-on-device-llms-2026-ollama-lm-studio-jan-msty-open-webui-gpt4all-anythingllm-faraday-deep-dive 2026년, 로컬 AI는 더 이상 "취미"가 아니다. M4 Max MacBook Pro에서 Llama 4 Scout 109B MoE가 24토큰/초로 도는 시대다. Ollama, LM Studio, Jan, Msty 같은 데스크탑 런타임이 GUI/CLI를 통일하고, Open WebUI, AnythingLLM, LibreChat이 ChatGPT급 인터페이스를 제공한다. 백엔드는 llama.cpp, MLX-LM, vLLM이 떠받치고, GGUF / AWQ / GPTQ / EXL3 / MXFP4 / BitNet 양자화 포맷이 8GB VRAM 노트북도 13B 모델 추론을 가능케 한다. Llama 3.3 70B, DeepSeek R1 Distill, Qwen 3, Phi-4, Gemma 3, MiniCPM 3, DeepSeek Coder V2 — 클라우드와 견줄 만한 오픈 모델이 매주 쏟아진다. 한편 Apple Intelligence(iOS 18), Phi Silica(Windows Copilot+), Gemini Nano(Android/Chrome)는 OS 레벨 온디바이스 AI를 표준화했다. 이 글은 런타임 → GUI → 백엔드 → 양자화 → 추천 모델 → 한국·일본 로컬 사례까지, 2026년 5월 기준 로컬 AI 풍경을 한 번에 정리한다. Sat, 16 May 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) local-aion-device-llmollamalm-studiojanmstyopen-webuigpt4allanythingllmfaradayllama-cppmlxggufquantizationragprivategptlibrechatkhojreorpinokiochatboxapple-intelligencephi-silicagemini-nano2026deep-dive