Skip to content
Published on

Running LLMs on Apple Silicon: Inside M4/M5 Architecture for AI Inference

🇰🇷KO🇺🇸EN🇯🇵JA
Split View
Authors

Why Apple Silicon Is Disrupting the AI Inference Market

Late 2024, a startup ML engineer asked me: "Why would you run LLMs on a MacBook when you have a workstation with an RTX 4090?" The expression changed after watching Llama 3.1 70B run at 8 tokens/sec on an M4 Max MacBook. A single RTX 4090 has only 24GB of VRAM — the 70B model simply cannot load into it.

Apple Silicon's LLM inference performance means more than benchmark numbers. It represents a fundamentally different architectural approach. This post dissects the internals of M4/M5 chips and explains exactly what happens when you run an LLM on them.

1. The Core Differentiator: Unified Memory Architecture (UMA)

The most important characteristic of Apple Silicon is its Unified Memory Architecture (UMA). To understand why this matters, compare it to traditional PC architecture.

The Bottleneck in Traditional PC Architecture

In conventional systems, CPU and GPU use physically separate memory pools.

Traditional PC Architecture:
┌─────────────────────┐   PCIe x16 (~32 GB/s)   ┌──────────────────────────┐
│ CPU                 │◄──────────────────────►  │ GPU                      │
│ Intel/AMD           │                           │ NVIDIA RTX 4090          │
│ DDR5 64GB           │                           │ GDDR6X 24GB              │
│ 89 GB/s bandwidth   │                           │ 1008 GB/s bandwidth      │
│                     │                           │                          │
│ Model slice during  │   Data must be COPIED!    │ Model slice during        │
│ inference           │                           │ inference                │
└─────────────────────┘                           └──────────────────────────┘

Problems:
- During LLM inference: weights on GPU VRAM, KV cache partly in CPU RAM → copy overhead
- PCIe bandwidth 32 GB/s is 30x slower than GPU VRAM bandwidth of 1 TB/s
- A 70B parameter model (FP16) = ~140GB → simply won't fit in 24GB VRAM!

Apple M4 Max's Approach

Apple M4 Max (Unified Memory):
┌─────────────────────────────────────────────────────────────────┐
│                    128GB Unified Memory Pool                    │
│                      546 GB/s bandwidth                         │
│                                                                 │
│   ┌────────────┐   ┌──────────────┐   ┌──────────────────────┐ │
│   │   CPU      │   │     GPU      │   │   Neural Engine      │ │
│   │ 14 cores   │   │  40 GPU cores│   │   38 TOPS            │ │
│   │ (4P + 10E) │   │              │   │   INT8/FP16 dedicated│ │
│   └────────────┘   └──────────────┘   └──────────────────────┘ │
│                                                                 │
│   All processors access the SAME physical memory directly!     │
│   No copying, no latency, no synchronization overhead          │
└─────────────────────────────────────────────────────────────────┘

Why This Is a Game Changer for LLM Inference

The core bottleneck in LLM inference is not compute bound but memory bandwidth bound. To understand this, look at the Roofline model.

Arithmetic Intensity = FLOPs / Memory Bytes

In Transformer attention and linear layers:

  • Matrix-vector multiply (batch size 1): read weight once, multiply once → Arithmetic Intensity ≈ 1
  • Matrix-matrix multiply (large batch): Arithmetic Intensity increases

M4 Max peak FP16 performance is ~14.2 TFLOPS, memory bandwidth is 546 GB/s. Roofline crossover: 14.2 TFLOPS / 0.546 TB/s = ~26 FLOP/byte

For single-query (batch=1) inference, Arithmetic Intensity is ~1 FLOP/byte — fully memory bandwidth limited. The theoretical throughput ceiling is determined by how fast you can stream weights through the chip.

From this perspective, M4 Max's 546 GB/s is lower than RTX 4090's 1008 GB/s, but when the model actually fits in memory, the effective utilization is far more favorable.

Memory requirements by model size (FP16):
Llama 3.2 3B   → ~6GB   (fits easily in M4 Pro 24GB)
Llama 3.1 8B   → ~16GB  (fits in M4 Pro 24GB)
Llama 3.1 70B  → ~140GB (needs Q4 quantization for M4 Max 128GB, ~40GB)
Llama 3.1 405B → ~810GB (even M4 Ultra 192GB needs quantization)

2. Dissecting the Apple Neural Engine (ANE)

Many people assume "Apple Silicon AI performance = Neural Engine." The reality is far more nuanced.

ANE Is Not the GPU

The Neural Engine (ANE) is Apple's dedicated AI accelerator, included since the A11 Bionic (2017). It was designed for an entirely different purpose from the GPU.

Components inside Apple M4 Max and their roles:
┌────────────────────────────────────────────────────────────────┐
│                       Apple M4 Max                             │
│                                                                │
│  CPU (4P + 10E cores)                                          │
│  - General purpose computation, OS, app logic                  │
│  - Handles tokenization during prefill phase                   │
│                                                                │
│  GPU (40 cores)                                                │
│  - Programmable via Metal API                                  │
│  - Handles matrix multiply, attention in LLM inference         │
│  - Primary compute target for llama.cpp, MLX                   │
│                                                                │
│  Neural Engine (38 TOPS)                                       │
│  - Exclusive to CoreML models                                  │
│  - INT8/FP16 systolic array MAC units                          │
│  - NOT directly programmable (no CUDA-like general API)        │
│  - Used for Whisper, Face ID, Siri processing                  │
│                                                                │
└────────────────────────────────────────────────────────────────┘

ANE Internal Architecture

The ANE consists of a systolic array of MAC (Multiply-Accumulate) units. M4's ANE can process 38 TOPS (INT8), using pipelined matrix multiply.

Systolic Array (simplified):
Input matrix → [MAC][MAC][MAC][MAC] → Output
                ↓    ↓    ↓    ↓
Input matrix → [MAC][MAC][MAC][MAC]
                ↓    ↓    ↓    ↓
Input matrix → [MAC][MAC][MAC][MAC]

Each MAC unit: multiply + accumulate simultaneously
Data flows left→right, weights flow top→bottom

Leveraging CoreML and ANE

To use the ANE, you must go through CoreML:

# Converting an LLM to CoreML (conceptual example)
import coremltools as ct
import torch

model = load_llm_model()
traced_model = torch.jit.trace(model, example_input)

# Specify compute units during CoreML conversion
mlmodel = ct.convert(
    traced_model,
    compute_units=ct.ComputeUnit.ALL,  # CPU + GPU + ANE
    # Or: ct.ComputeUnit.CPU_AND_NE    # CPU + ANE only
    minimum_deployment_target=ct.target.macOS14,
)
mlmodel.save("llm_model.mlpackage")

Practical limitation: most LLM serving frameworks (llama.cpp, vLLM, Ollama) do not use the ANE. ANE is CoreML-exclusive, and CoreML conversion doesn't support all LLM architectures. In practice, most LLM inference uses the GPU path.

3. Metal Performance Shaders and MLX

Metal: The Programming Layer for Apple GPUs

Just as NVIDIA provides CUDA, Apple provides Metal. PyTorch's MPS (Metal Performance Shaders) backend uses Metal to leverage Apple GPUs.

# Using PyTorch MPS backend
import torch

# Check MPS availability
print(torch.backends.mps.is_available())   # True on Apple Silicon
print(torch.backends.mps.is_built())       # True if compiled with MPS

device = torch.device("mps")

# Move model to MPS
model = YourTransformerModel()
model = model.to(device)

# Tensor operations run on MPS
x = torch.randn(1, 512, device=device)
output = model(x)

# What happens under the hood:
# PyTorch → MPS backend → Metal API → GPU kernel execution
# Kernels written in MSL (Metal Shading Language), not PTX

MLX: A Machine Learning Framework Built for Apple Silicon

Released by Apple in late 2023, MLX is a machine learning framework designed from the ground up for Apple Silicon. It provides a NumPy-like API while fully leveraging unified memory.

import mlx.core as mx
import mlx.nn as nn

# MLX's key innovation: lazy evaluation
a = mx.array([[1.0, 2.0], [3.0, 4.0]])
b = mx.array([[1.0, 0.0], [0.0, 1.0]])

# No computation happens yet!
c = a @ b        # Only builds computation graph
d = mx.exp(c)    # Adds to graph
e = d.sum()      # Adds to graph

# mx.eval() computes the entire optimized graph at once
mx.eval(e)       # GPU execution happens here

print(e)  # [[2.71828...]]

Benefits of lazy evaluation:

  1. Graph optimization: eliminates unnecessary intermediate allocations
  2. Kernel fusion: merges multiple small ops into a single GPU kernel
  3. Unified memory exploitation: no explicit data movement between CPU and GPU
# Transformer attention in MLX
import mlx.core as mx
import mlx.nn as nn
import math

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model: int, num_heads: int):
        super().__init__()
        self.num_heads = num_heads
        self.d_head = d_model // num_heads
        self.scale = math.sqrt(self.d_head)

        self.q_proj = nn.Linear(d_model, d_model)
        self.k_proj = nn.Linear(d_model, d_model)
        self.v_proj = nn.Linear(d_model, d_model)
        self.out_proj = nn.Linear(d_model, d_model)

    def __call__(self, x: mx.array, mask=None):
        B, T, C = x.shape

        q = self.q_proj(x).reshape(B, T, self.num_heads, self.d_head).transpose(0, 2, 1, 3)
        k = self.k_proj(x).reshape(B, T, self.num_heads, self.d_head).transpose(0, 2, 1, 3)
        v = self.v_proj(x).reshape(B, T, self.num_heads, self.d_head).transpose(0, 2, 1, 3)

        scores = (q @ k.transpose(0, 1, 3, 2)) / self.scale

        if mask is not None:
            scores = scores + mask

        weights = mx.softmax(scores, axis=-1)
        out = (weights @ v).transpose(0, 2, 1, 3).reshape(B, T, C)
        return self.out_proj(out)

4. LLM Serving Tools and Real Benchmarks

llama.cpp + Metal Backend

llama.cpp is the most battle-tested approach for running LLMs on Apple Silicon. The Metal backend routes GPU workloads through Metal.

# Build llama.cpp with Metal support
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_METAL=ON
cmake --build build --config Release -j

# Run Llama 3.1 8B on M4 Pro
./build/bin/llama-cli \
  -m models/llama-3.1-8b-q4_k_m.gguf \
  -p "Explain transformer architecture" \
  -n 200 \
  --n-gpu-layers 999  # Offload all layers to GPU (Metal)
# Expected output: ~45-50 tok/s (M4 Pro 48GB)

# Enable Metal debug logging
GGML_METAL_LOG_LEVEL=1 ./build/bin/llama-cli -m model.gguf ...

Ollama: Production-Ready Wrapper Around llama.cpp

# Install and use Ollama
brew install ollama
ollama serve  # Start background server

# Performance by model (measured on M4 Pro 48GB)
ollama run llama3.2         # 3B, Q4_K_M: ~80-85 tok/s
ollama run llama3.1:8b      # 8B, Q4_K_M: ~45-50 tok/s
ollama run llama3.1:70b     # 70B, Q4_K_M: borderline on 48GB, ~6-8 tok/s
# On M4 Max 128GB: llama3.1:70b runs at ~8-10 tok/s comfortably

# Using Ollama from Python
import ollama

response = ollama.chat(
    model='llama3.1:8b',
    messages=[{'role': 'user', 'content': 'Explain RLHF in detail'}]
)
print(response['message']['content'])

Performance Comparison Table

ModelQuantizationM4 Pro 48GBM4 Max 128GBRTX 4090 24GB
Llama 3.2 3BQ4_K_M~80 tok/s~100 tok/s~130 tok/s
Llama 3.1 8BQ4_K_M~45 tok/s~60 tok/s~110 tok/s
Llama 3.1 8BFP16~20 tok/s~35 tok/s~85 tok/s
Llama 3.1 70BQ4_K_M~6 tok/s~8 tok/sOOM (24GB insufficient)
Llama 3.1 70BQ8OOM~4 tok/sOOM
Mistral 7BQ4_K_M~50 tok/s~65 tok/s~120 tok/s
Qwen2.5 72BQ4_K_MOOM~7 tok/sOOM

Measured values; varies with prompt length and KV cache state

5. Why GGUF Quantization Shines on Apple Silicon

Understanding K-Quants

GGUF's Q4_K_M is not simple 4-bit quantization. K-quants apply mixed precision per block, storing different precision for different parts of the weight tensors.

# Q4_K_M quantization process (simplified)
import torch

def quantize_q4_k(weight_tensor, block_size=32):
    """
    K-Quants: per-block scale and min stored in FP16
    Actual weights stored as 4-bit integers
    """
    B, N = weight_tensor.shape
    num_blocks = N // block_size

    quantized_blocks = []
    scales = []
    mins = []

    for i in range(num_blocks):
        block = weight_tensor[:, i*block_size:(i+1)*block_size]

        block_min = block.min(dim=-1, keepdim=True).values
        block_max = block.max(dim=-1, keepdim=True).values

        # Normalize to 0-15 range (4bit = 16 levels)
        scale = (block_max - block_min) / 15.0
        # Convert to 4-bit integer
        q = torch.round((block - block_min) / scale).clamp(0, 15).to(torch.uint8)

        quantized_blocks.append(q)
        scales.append(scale.to(torch.float16))
        mins.append(block_min.to(torch.float16))

    return quantized_blocks, scales, mins

# Dequantize during inference:
def dequantize_q4_k(quantized, scales, mins):
    """Restore 4-bit integers to FP16"""
    return quantized.float() * scales + mins

# Memory savings calculation:
# FP16 70B model: 70B * 2 bytes = 140 GB
# Q4_K_M 70B model: ~40 GB (approx 71% reduction)

x86 vs Apple Silicon for Quantized Inference

On x86 CPUs, dequantization creates a bottleneck. Even with AVX-512, 4-bit operation efficiency is poor.

Apple Silicon GPU, by contrast:

  1. Metal shaders with optimized 4-bit operation support
  2. Unified memory: dequantized values available immediately for matrix multiply (no copy)
  3. INT8/INT4 compute units integrated directly in GPU
# Quantization type comparison on M4 Max 128GB, Llama 3.1 70B:
# Q2_K   : ~18GB, significant quality loss, ~12 tok/s
# Q4_0   : ~35GB, acceptable quality, ~8 tok/s
# Q4_K_M : ~42GB, good quality, ~8 tok/s  (recommended)
# Q5_K_M : ~52GB, very good quality, ~6 tok/s
# Q8_0   : ~70GB, near FP16 quality, ~4 tok/s
# FP16   : ~140GB (won't fit in 128GB!)

# Conclusion: Q4_K_M is the optimal quality/speed balance point

6. M5 Chip Predictions and Apple's AI Roadmap

Confirmed M4 Specifications (2024)

M4:           10 CPU cores (4P+6E), 10 GPU cores, 38 TOPS ANE, 16-32GB
M4 Pro:       14 CPU cores (10P+4E), 20 GPU cores, 38 TOPS ANE, 24-64GB
M4 Max:       14 CPU cores (10P+4E), 40 GPU cores, 38 TOPS ANE, 48-128GB
M4 Ultra:     28 CPU cores, 80 GPU cores, 76 TOPS ANE, 192GB (two M4 Max dies)
Memory BW:    M4=120GB/s, M4 Pro=273GB/s, M4 Max=546GB/s, M4 Ultra=819GB/s

M5 Outlook (Expected 2025-2026)

Based on industry analyst consensus:

Expected M5 specs (N3P process, late 2025):
- TSMC 3nm N3P process (~15-20% power efficiency improvement vs M4's N3E)
- CPU: M5 Pro estimated at 20 cores (12P + 8E)
- GPU: M5 Max estimated at 50 GPU cores
- Neural Engine: 50+ TOPS estimated
- Memory: M5 Max up to 192GB (from M4 Max's 128GB)
- Memory bandwidth: M5 Max 700+ GB/s estimated (~30% improvement vs M4 Max's 546)

The biggest interest for M5 is memory capacity expansion. A 192GB M5 Max would allow Llama 3.1 70B to run in Q8 (~70GB) or even approach FP16 (~140GB) comfortably.

Apple's On-Device AI Strategy

Apple pursues a hybrid strategy between Private Cloud Compute (PCC) and on-device models. The iPhone 16 carries a ~3B parameter Apple Intelligence model for ANE execution; Mac runs larger models.

Apple Intelligence architecture (inferred):
User Request
     ↓
On-Device model (~3B, ANE execution)
     ↓ (complex requests only)
Private Cloud Compute (larger models, Apple Silicon servers)

For LLM training, Apple Silicon still significantly trails NVIDIA H100/H200. The M4 Ultra's 76 TOPS ANE is not comparable to the H100's 3958 TOPS Tensor Cores.

7. Practical Setup: Building a Local LLM Environment on Mac

Running Models with MLX-LM

# Install mlx-lm
pip install mlx-lm

# Convert Hugging Face model to MLX format and run
python -m mlx_lm.convert \
  --hf-path meta-llama/Llama-3.1-8B-Instruct \
  --mlx-path mlx_models/llama-3.1-8b \
  --quantize \
  --q-bits 4

# Run the converted MLX model
python -m mlx_lm.generate \
  --model mlx_models/llama-3.1-8b \
  --prompt "Explain attention mechanism" \
  --max-tokens 500

# Using MLX-LM from Python
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Llama-3.1-8B-Instruct-4bit")

prompt = "You are a helpful assistant. Explain what makes Apple Silicon unique for AI inference."

response = generate(
    model,
    tokenizer,
    prompt=prompt,
    max_tokens=500,
    verbose=True,  # Prints token generation speed
)

Monitoring Memory and GPU Usage

# Monitor GPU utilization on Apple Silicon
# Option 1: Activity Monitor > GPU History window
# Option 2: powermetrics (requires root)
sudo powermetrics --samplers gpu_power -i 500 -n 5

# Option 3: asitop (third-party, pip install asitop)
sudo asitop
# Real-time display of CPU/GPU/ANE utilization,
# memory bandwidth, power consumption

8. Apple Silicon vs NVIDIA: An Honest Decision Framework

Comparison Table

Use CaseApple SiliconNVIDIA GPURecommendation
Local LLM experimentation (70B)M4 Max 128GB works24GB VRAM insufficientApple Silicon
Local LLM experimentation (8B)M4 Pro sufficientRTX 4090 finePreference/budget
Production serving (large scale)InefficientA100/H100 optimalNVIDIA
Model fine-tuning (70B)Very slow8x H100 recommendedNVIDIA
Battery / portabilityMacBook at 15-20WDesktop at 300-400WApple Silicon
Memory capacity (single device)M4 Ultra 192GBH100 SXM 80GBApple Silicon
Cost efficiency (inference)MacBook $3-6K totalRTX 4090 $1.6K + powerComparable
Software ecosystemLimited but growing fastCUDA dominantNVIDIA
CUDA code compatibilityNot supportedFully supportedNVIDIA

Honest Conclusion

Apple Silicon wins when:

  • You want to run large models (30B-70B) locally as a solo developer
  • Mobility and battery life matter
  • You are already in the Mac ecosystem
  • You need a quiet, power-efficient machine for day-to-day inference experiments

Choose NVIDIA when:

  • Throughput at scale is critical for production serving
  • Model training and fine-tuning is your primary workload
  • You need immediate access to the latest PyTorch/CUDA ecosystem features
  • Your team needs a uniform development environment

Personally, the most practical setup in 2026 is Mac with Apple Silicon for local development and experimentation, combined with cloud A100/H100 access for large-scale training and production. This combination gets the best of both worlds.

Conclusion

Apple Silicon's UMA is not marketing fluff. It eliminates the PCIe bottleneck, provides a large unified memory pool, and enables practical LLM inference at low power. The combination of llama.cpp + Metal + GGUF Q4_K_M is the most proven approach for running LLMs on Mac in 2026.

When M5 ships with 700+ GB/s memory bandwidth, even faster 70B inference on a single laptop will become reality. As Apple continues growing the MLX ecosystem, future Apple Silicon will become an increasingly compelling local AI inference platform.

The long hegemony of CUDA and NVIDIA finally has a meaningful competitor. As an ML engineer, that's genuinely exciting.

Discuss on Twitter • View on GitHub