Split View: Apple Silicon에서 LLM 서빙하기: M4/M5 칩의 비밀과 한계

Traditional PC Architecture:
┌─────────────────────┐   PCIe x16 (~32 GB/s)   ┌──────────────────────────┐
│ CPU                 │◄──────────────────────►  │ GPU                      │
│ Intel/AMD           │                           │ NVIDIA RTX 4090          │
│ DDR5 64GB           │                           │ GDDR6X 24GB              │
│ 89 GB/s bandwidth   │                           │ 1008 GB/s bandwidth      │
│                     │                           │                          │
│ Model slice during  │   Data must be COPIED!    │ Model slice during        │
│ inference           │                           │ inference                │
└─────────────────────┘                           └──────────────────────────┘

Problems:
- During LLM inference: weights on GPU VRAM, KV cache partly in CPU RAM → copy overhead
- PCIe bandwidth 32 GB/s is 30x slower than GPU VRAM bandwidth of 1 TB/s
- A 70B parameter model (FP16) = ~140GB → simply won't fit in 24GB VRAM!

Apple M4 Max (Unified Memory):
┌─────────────────────────────────────────────────────────────────┐
│                    128GB Unified Memory Pool                    │
│                      546 GB/s bandwidth                         │
│                                                                 │
│   ┌────────────┐   ┌──────────────┐   ┌──────────────────────┐ │
│   │   CPU      │   │     GPU      │   │   Neural Engine      │ │
│   │ 14 cores   │   │  40 GPU cores│   │   38 TOPS            │ │
│   │ (4P + 10E) │   │              │   │   INT8/FP16 dedicated│ │
│   └────────────┘   └──────────────┘   └──────────────────────┘ │
│                                                                 │
│   All processors access the SAME physical memory directly!     │
│   No copying, no latency, no synchronization overhead          │
└─────────────────────────────────────────────────────────────────┘

Memory requirements by model size (FP16):
Llama 3.2 3B   → ~6GB   (fits easily in M4 Pro 24GB)
Llama 3.1 8B   → ~16GB  (fits in M4 Pro 24GB)
Llama 3.1 70B  → ~140GB (needs Q4 quantization for M4 Max 128GB, ~40GB)
Llama 3.1 405B → ~810GB (even M4 Ultra 192GB needs quantization)

Components inside Apple M4 Max and their roles:
┌────────────────────────────────────────────────────────────────┐
│                       Apple M4 Max                             │
│                                                                │
│  CPU (4P + 10E cores)                                          │
│  - General purpose computation, OS, app logic                  │
│  - Handles tokenization during prefill phase                   │
│                                                                │
│  GPU (40 cores)                                                │
│  - Programmable via Metal API                                  │
│  - Handles matrix multiply, attention in LLM inference         │
│  - Primary compute target for llama.cpp, MLX                   │
│                                                                │
│  Neural Engine (38 TOPS)                                       │
│  - Exclusive to CoreML models                                  │
│  - INT8/FP16 systolic array MAC units                          │
│  - NOT directly programmable (no CUDA-like general API)        │
│  - Used for Whisper, Face ID, Siri processing                  │
│                                                                │
└────────────────────────────────────────────────────────────────┘

Systolic Array (simplified):
Input matrix → [MAC][MAC][MAC][MAC] → Output
                ↓    ↓    ↓    ↓
Input matrix → [MAC][MAC][MAC][MAC]
                ↓    ↓    ↓    ↓
Input matrix → [MAC][MAC][MAC][MAC]

Each MAC unit: multiply + accumulate simultaneously
Data flows left→right, weights flow top→bottom

# Converting an LLM to CoreML (conceptual example)
import coremltools as ct
import torch

model = load_llm_model()
traced_model = torch.jit.trace(model, example_input)

# Specify compute units during CoreML conversion
mlmodel = ct.convert(
    traced_model,
    compute_units=ct.ComputeUnit.ALL,  # CPU + GPU + ANE
    # Or: ct.ComputeUnit.CPU_AND_NE    # CPU + ANE only
    minimum_deployment_target=ct.target.macOS14,
)
mlmodel.save("llm_model.mlpackage")

# Using PyTorch MPS backend
import torch

# Check MPS availability
print(torch.backends.mps.is_available())   # True on Apple Silicon
print(torch.backends.mps.is_built())       # True if compiled with MPS

device = torch.device("mps")

# Move model to MPS
model = YourTransformerModel()
model = model.to(device)

# Tensor operations run on MPS
x = torch.randn(1, 512, device=device)
output = model(x)

# What happens under the hood:
# PyTorch → MPS backend → Metal API → GPU kernel execution
# Kernels written in MSL (Metal Shading Language), not PTX

import mlx.core as mx
import mlx.nn as nn

# MLX's key innovation: lazy evaluation
a = mx.array([[1.0, 2.0], [3.0, 4.0]])
b = mx.array([[1.0, 0.0], [0.0, 1.0]])

# No computation happens yet!
c = a @ b        # Only builds computation graph
d = mx.exp(c)    # Adds to graph
e = d.sum()      # Adds to graph

# mx.eval() computes the entire optimized graph at once
mx.eval(e)       # GPU execution happens here

print(e)  # [[2.71828...]]

# Transformer attention in MLX
import mlx.core as mx
import mlx.nn as nn
import math

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model: int, num_heads: int):
        super().__init__()
        self.num_heads = num_heads
        self.d_head = d_model // num_heads
        self.scale = math.sqrt(self.d_head)

        self.q_proj = nn.Linear(d_model, d_model)
        self.k_proj = nn.Linear(d_model, d_model)
        self.v_proj = nn.Linear(d_model, d_model)
        self.out_proj = nn.Linear(d_model, d_model)

    def __call__(self, x: mx.array, mask=None):
        B, T, C = x.shape

        q = self.q_proj(x).reshape(B, T, self.num_heads, self.d_head).transpose(0, 2, 1, 3)
        k = self.k_proj(x).reshape(B, T, self.num_heads, self.d_head).transpose(0, 2, 1, 3)
        v = self.v_proj(x).reshape(B, T, self.num_heads, self.d_head).transpose(0, 2, 1, 3)

        scores = (q @ k.transpose(0, 1, 3, 2)) / self.scale

        if mask is not None:
            scores = scores + mask

        weights = mx.softmax(scores, axis=-1)
        out = (weights @ v).transpose(0, 2, 1, 3).reshape(B, T, C)
        return self.out_proj(out)

# Build llama.cpp with Metal support
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_METAL=ON
cmake --build build --config Release -j

# Run Llama 3.1 8B on M4 Pro
./build/bin/llama-cli \
  -m models/llama-3.1-8b-q4_k_m.gguf \
  -p "Explain transformer architecture" \
  -n 200 \
  --n-gpu-layers 999  # Offload all layers to GPU (Metal)
# Expected output: ~45-50 tok/s (M4 Pro 48GB)

# Enable Metal debug logging
GGML_METAL_LOG_LEVEL=1 ./build/bin/llama-cli -m model.gguf ...

# Install and use Ollama
brew install ollama
ollama serve  # Start background server

# Performance by model (measured on M4 Pro 48GB)
ollama run llama3.2         # 3B, Q4_K_M: ~80-85 tok/s
ollama run llama3.1:8b      # 8B, Q4_K_M: ~45-50 tok/s
ollama run llama3.1:70b     # 70B, Q4_K_M: borderline on 48GB, ~6-8 tok/s
# On M4 Max 128GB: llama3.1:70b runs at ~8-10 tok/s comfortably

# Using Ollama from Python
import ollama

response = ollama.chat(
    model='llama3.1:8b',
    messages=[{'role': 'user', 'content': 'Explain RLHF in detail'}]
)
print(response['message']['content'])

# Q4_K_M quantization process (simplified)
import torch

def quantize_q4_k(weight_tensor, block_size=32):
    """
    K-Quants: per-block scale and min stored in FP16
    Actual weights stored as 4-bit integers
    """
    B, N = weight_tensor.shape
    num_blocks = N // block_size

    quantized_blocks = []
    scales = []
    mins = []

    for i in range(num_blocks):
        block = weight_tensor[:, i*block_size:(i+1)*block_size]

        block_min = block.min(dim=-1, keepdim=True).values
        block_max = block.max(dim=-1, keepdim=True).values

        # Normalize to 0-15 range (4bit = 16 levels)
        scale = (block_max - block_min) / 15.0
        # Convert to 4-bit integer
        q = torch.round((block - block_min) / scale).clamp(0, 15).to(torch.uint8)

        quantized_blocks.append(q)
        scales.append(scale.to(torch.float16))
        mins.append(block_min.to(torch.float16))

    return quantized_blocks, scales, mins

# Dequantize during inference:
def dequantize_q4_k(quantized, scales, mins):
    """Restore 4-bit integers to FP16"""
    return quantized.float() * scales + mins

# Memory savings calculation:
# FP16 70B model: 70B * 2 bytes = 140 GB
# Q4_K_M 70B model: ~40 GB (approx 71% reduction)

# Quantization type comparison on M4 Max 128GB, Llama 3.1 70B:
# Q2_K   : ~18GB, significant quality loss, ~12 tok/s
# Q4_0   : ~35GB, acceptable quality, ~8 tok/s
# Q4_K_M : ~42GB, good quality, ~8 tok/s  (recommended)
# Q5_K_M : ~52GB, very good quality, ~6 tok/s
# Q8_0   : ~70GB, near FP16 quality, ~4 tok/s
# FP16   : ~140GB (won't fit in 128GB!)

# Conclusion: Q4_K_M is the optimal quality/speed balance point

M4:           10 CPU cores (4P+6E), 10 GPU cores, 38 TOPS ANE, 16-32GB
M4 Pro:       14 CPU cores (10P+4E), 20 GPU cores, 38 TOPS ANE, 24-64GB
M4 Max:       14 CPU cores (10P+4E), 40 GPU cores, 38 TOPS ANE, 48-128GB
M4 Ultra:     28 CPU cores, 80 GPU cores, 76 TOPS ANE, 192GB (two M4 Max dies)
Memory BW:    M4=120GB/s, M4 Pro=273GB/s, M4 Max=546GB/s, M4 Ultra=819GB/s

Expected M5 specs (N3P process, late 2025):
- TSMC 3nm N3P process (~15-20% power efficiency improvement vs M4's N3E)
- CPU: M5 Pro estimated at 20 cores (12P + 8E)
- GPU: M5 Max estimated at 50 GPU cores
- Neural Engine: 50+ TOPS estimated
- Memory: M5 Max up to 192GB (from M4 Max's 128GB)
- Memory bandwidth: M5 Max 700+ GB/s estimated (~30% improvement vs M4 Max's 546)

Apple Intelligence architecture (inferred):
User Request
     ↓
On-Device model (~3B, ANE execution)
     ↓ (complex requests only)
Private Cloud Compute (larger models, Apple Silicon servers)

# Install mlx-lm
pip install mlx-lm

# Convert Hugging Face model to MLX format and run
python -m mlx_lm.convert \
  --hf-path meta-llama/Llama-3.1-8B-Instruct \
  --mlx-path mlx_models/llama-3.1-8b \
  --quantize \
  --q-bits 4

# Run the converted MLX model
python -m mlx_lm.generate \
  --model mlx_models/llama-3.1-8b \
  --prompt "Explain attention mechanism" \
  --max-tokens 500

# Using MLX-LM from Python
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Llama-3.1-8B-Instruct-4bit")

prompt = "You are a helpful assistant. Explain what makes Apple Silicon unique for AI inference."

response = generate(
    model,
    tokenizer,
    prompt=prompt,
    max_tokens=500,
    verbose=True,  # Prints token generation speed
)

# Monitor GPU utilization on Apple Silicon
# Option 1: Activity Monitor > GPU History window
# Option 2: powermetrics (requires root)
sudo powermetrics --samplers gpu_power -i 500 -n 5

# Option 3: asitop (third-party, pip install asitop)
sudo asitop
# Real-time display of CPU/GPU/ANE utilization,
# memory bandwidth, power consumption