Mastering the Segment Anything Model: Paper Analysis and Practical Guide from SAM 1 to SAM 2 to SAM 3

┌─────────────────────────────────────────────────────────┐
│                      SAM Architecture                    │
│                                                          │
│  ┌──────────────┐   ┌────────────────┐                   │
│  │ Image Encoder │   │ Prompt Encoder │                   │
│  │  (ViT-H/L/B) │   │ (Point/Box/    │                   │
│  │  MAE Pretrain │   │  Mask/Text)    │                   │
│  └──────┬───────┘   └───────┬────────┘                   │
│         │                   │                            │
│         └─────────┬─────────┘                            │
│                   ▼                                      │
│          ┌────────────────┐                              │
│          │  Mask Decoder   │                              │
│          │ (Transformer    │                              │
│          │  2-layer)       │──→ 3 masks + IoU scores      │
│          └────────────────┘                              │
└─────────────────────────────────────────────────────────┘

Phase 1: Assisted-Manual        Phase 2: Semi-Automatic       Phase 3: Fully Automatic
─────────────────────          ──────────────────────         ──────────────────────
• SAM + human annotators        • SAM auto-suggests masks     • 32×32 grid points auto-generated
• Browser-based tool            • Humans add missed objects   • NMS + confidence filtering
• 120K images / 4.3M masks     • 180K images / 5.9M masks    • Majority of 1.1B masks

# Installation
pip install git+https://github.com/facebookresearch/segment-anything.git

# Optional dependencies
pip install opencv-python pycocotools matplotlib onnxruntime onnx

from segment_anything import SamPredictor, sam_model_registry
import numpy as np

# Load model
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

# Create predictor
predictor = SamPredictor(sam)

# Set image (image embedding computed once)
predictor.set_image(image)  # numpy array (H, W, 3) RGB

# Predict with point prompt
masks, scores, logits = predictor.predict(
    point_coords=np.array([[500, 375]]),  # (N, 2) coordinates
    point_labels=np.array([1]),            # 1=foreground, 0=background
    multimask_output=True,                 # Return 3 candidate masks
)

from segment_anything import SamAutomaticMaskGenerator, sam_model_registry

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

mask_generator = SamAutomaticMaskGenerator(sam)
masks = mask_generator.generate(image)
# Each mask: segmentation, area, bbox, predicted_iou, stability_score

┌────────────────────────────────────────────────────────────────────┐
│                    SAM 2 Streaming Architecture                    │
│                                                                    │
│  Frame t-2      Frame t-1      Frame t (current)                   │
│     │            │            │                                    │
│     ▼            ▼            ▼                                    │
│  ┌──────┐    ┌──────┐    ┌──────────────┐                          │
│  │Hiera │    │Hiera │    │ Hiera Image  │                          │
│  │Enc.  │    │Enc.  │    │   Encoder    │                          │
│  └──┬───┘    └──┬───┘    └──────┬───────┘                          │
│     │           │               │                                  │
│     ▼           ▼               ▼                                  │
│  ┌──────────────────────────────────────────┐                      │
│  │            Memory Bank                    │                      │
│  │  ┌──────────────┐  ┌───────────────────┐ │                      │
│  │  │ FIFO Queue   │  │ Prompted Frames   │ │                      │
│  │  │ (N recent    │  │ (M prompted       │ │                      │
│  │  │  frames)     │  │  frames)          │ │                      │
│  │  └──────────────┘  └───────────────────┘ │                      │
│  │  ┌──────────────────────────────────────┐│                      │
│  │  │     Object Pointers (256-dim)        ││                      │
│  │  └──────────────────────────────────────┘│                      │
│  └──────────────────┬───────────────────────┘                      │
│                     ▼                                              │
│  ┌──────────────────────────────┐   ┌───────────────┐              │
│  │    Memory Attention Module    │←──│Prompt Encoder │              │
│  │ (L Transformer Blocks:       │   │(Point/Box/    │              │
│  │  Self-Attn + Cross-Attn)     │   │ Mask)         │              │
│  └──────────────┬───────────────┘   └───────────────┘              │
│                 ▼                                                   │
│  ┌──────────────────────────┐                                      │
│  │      Mask Decoder        │──→ Mask + IoU + Occlusion Score      │
│  │ (+ Skip Connections)     │                                      │
│  └──────────────────────────┘                                      │
└────────────────────────────────────────────────────────────────────┘

# Installation
git clone https://github.com/facebookresearch/sam2.git && cd sam2
pip install -e .

# Download checkpoints
cd checkpoints && ./download_ckpts.sh && cd ..

import torch
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor

checkpoint = "./checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"
predictor = SAM2ImagePredictor(build_sam2(model_cfg, checkpoint))

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(image)
    masks, _, _ = predictor.predict(
        point_coords=np.array([[500, 375]]),
        point_labels=np.array([1]),
    )

import torch
from sam2.build_sam import build_sam2_video_predictor

checkpoint = "./checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"
predictor = build_sam2_video_predictor(model_cfg, checkpoint)

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(video_path)

    # Add prompt to a specific frame
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(
        state, frame_idx=0, obj_id=1,
        points=np.array([[500, 375]]),
        labels=np.array([1]),
    )

    # Propagate through entire video
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        # Process masks for each frame
        process_masks(frame_idx, masks)

from sam2.sam2_image_predictor import SAM2ImagePredictor
predictor = SAM2ImagePredictor.from_pretrained("facebook/sam2-hiera-large")

from sam2.sam2_video_predictor import SAM2VideoPredictor
predictor = SAM2VideoPredictor.from_pretrained("facebook/sam2-hiera-large")

SAM 1: Point/Box → "Isolate the object at this location"
SAM 2: Point/Box + Video Tracking → "Track this object through the video"
SAM 3: Text/Image Exemplar → "Find everything matching this concept"

┌───────────────────────────────────────────────────────────────────────┐
│                       SAM 3 Architecture                              │
│                                                                       │
│  ┌────────────┐  ┌────────────┐  ┌──────────────────┐                 │
│  │ Text       │  │ Exemplar   │  │ Visual Prompts   │                 │
│  │ Encoder    │  │ Encoder    │  │ (Point/Box/Mask) │                 │
│  │ "school    │  │ [image]    │  │                  │                 │
│  │  bus"      │  │            │  │                  │                 │
│  └─────┬──────┘  └─────┬──────┘  └────────┬─────────┘                 │
│        └────────────┬───┘                  │                          │
│                     ▼                      │                          │
│  ┌──────────────────────────────┐          │                          │
│  │   Meta Perception Encoder    │          │                          │
│  │   (Vision-Language joint     │          │                          │
│  │    embedding space)          │          │                          │
│  └──────────────┬───────────────┘          │                          │
│                 │                          │                          │
│        ┌────────┴────────┐                 │                          │
│        ▼                 ▼                 │                          │
│  ┌───────────┐    ┌─────────────────┐      │                          │
│  │ Presence  │    │  Fusion Encoder │      │                          │
│  │   Head    │    │  (Conditional   │      │                          │
│  │ "yes/no"  │    │   features)     │      │                          │
│  └───────────┘    └────────┬────────┘      │                          │
│                            │               │                          │
│                            ▼               │                          │
│               ┌──────────────────┐         │                          │
│               │  DETR Detector   │←────────┘                          │
│               │  (Transformer-   │                                    │
│               │   based detect.) │                                    │
│               └────────┬─────────┘                                    │
│                        │                                              │
│                        ▼                                              │
│  ┌──────────────────────────────────────┐                             │
│  │    SAM 2-inspired Tracker            │                             │
│  │    (Memory Bank + temporal           │                             │
│  │     consistency)                     │                             │
│  └──────────────────┬───────────────────┘                             │
│                     ▼                                                 │
│              Masks + Boxes + Scores                                   │
└───────────────────────────────────────────────────────────────────────┘

Previous approach: Detector simultaneously "finds" and "judges existence" → optimization conflict
SAM 3:            Presence Head first judges "does it exist?" → Detector focuses solely on "finding"

# Create environment
conda create -n sam3 python=3.12
conda activate sam3

# Install PyTorch
pip install torch==2.7.0 torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/cu126

# Install SAM 3
git clone https://github.com/facebookresearch/sam3.git
cd sam3
pip install -e .

# Optional: for notebooks or development
pip install -e ".[notebooks]"
pip install -e ".[train,dev]"

from PIL import Image
from sam3.model_builder import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor

model = build_sam3_image_model()
processor = Sam3Processor(model)
image = Image.open("path/to/image.jpg")

# Concept segmentation with text prompt
inference_state = processor.set_image(image)
output = processor.set_text_prompt(
    state=inference_state,
    prompt="yellow school bus"
)
masks, boxes, scores = output["masks"], output["boxes"], output["scores"]

from sam3.model_builder import build_sam3_video_predictor

video_predictor = build_sam3_video_predictor()
response = video_predictor.handle_request(
    request=dict(
        type="start_session",
        resource_path="path/to/video.mp4"
    )
)

pip install -U ultralytics

from ultralytics.models.sam import SAM3SemanticPredictor

overrides = dict(
    conf=0.25, task="segment", mode="predict",
    model="sam3.pt", half=True
)
predictor = SAM3SemanticPredictor(overrides=overrides)
predictor.set_image("path/to/image.jpg")

# Segment multiple concepts simultaneously
results = predictor(text=["person", "bus", "glasses"])

1. SAM 1 paper (2023) → Establish foundational concepts
   ↓
2. SAM 2 paper (2024) → Understand video extension
   ↓
3. SAM 3 paper (2025) → Understand concept awareness
   ↓
4. Hands-on notebooks from each GitHub repository