Split View: Fine-tuning 실전 가이드: LoRA와 QLoRA로 나만의 모델 만들기

Full Fine-tuning Llama 3.1 70B:
- VRAM required: ~560GB (FP32)
  -> Needs 7x H100 80GB GPUs
- Training time: days
- Cloud cost: thousands of dollars

LoRA Fine-tuning Llama 3.1 70B:
- VRAM required: ~48GB (QLoRA: ~20GB!)
  -> A single RTX 3090 or A10 works
- Training time: hours on 1 GPU
- Cloud cost: $20-100 (A100 rental)

Full fine-tuning:
W_new = W_original + delta_W
(delta_W is the same size as the original matrix = 70B parameter updates)

LoRA: decompose delta_W into two small matrices
delta_W = A x B
  A: (d x r) matrix, B: (r x d) matrix
  r = rank (typically 4-64; smaller = more memory-efficient)

Example: d=4096, r=16
- Full delta_W: 4096 x 4096 = 16.7M parameters
- LoRA delta_W: A(4096x16) + B(16x4096) = 131K parameters
- 128x fewer parameters to train for equivalent effect!

During training: freeze W_original, only train A and B
During inference: W_new = W_original + A x B (merge or keep separate)

Standard LoRA:
- Base model: FP16
- LoRA adapters: FP16
- VRAM for 70B model: ~140GB

QLoRA:
- Base model: 4-bit (NF4 quantization)
- LoRA adapters: BF16/FP16 (full precision)
- VRAM for 70B model: ~20GB (!!)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset, Dataset

# ============================================================
# Step 1: Load model with 4-bit quantization (QLoRA)
# ============================================================
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # NF4 beats FP4 in quality
    bnb_4bit_compute_dtype=torch.bfloat16,  # compute in BF16
    bnb_4bit_use_double_quant=True          # extra memory savings
)

model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"  # Important: left-padding causes instability

# ============================================================
# Step 2: Configure LoRA
# ============================================================
lora_config = LoraConfig(
    r=16,                   # rank: higher = more expressive, more VRAM
    lora_alpha=32,          # scaling factor (typically 2x rank)
    target_modules=[        # which layers to apply LoRA to
        "q_proj", "v_proj", "k_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"  # include MLP layers too
    ],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Example output: trainable params: 167,772,160 || all params: 8,201,441,280 || trainable%: 2.05
# Only 2% of parameters are trained — 98% are frozen

# ============================================================
# Step 3: Prepare dataset
# ============================================================
raw_data = [
    {
        "instruction": "Classify the sentiment of the following customer review.",
        "input": "The shipping took 3 extra days with no communication. Very disappointing.",
        "output": "Negative (frustrated, disappointed)"
    },
    # ... more examples
]

def format_instruction(example):
    """Convert to Llama 3.1 instruction format"""
    if example.get("input"):
        text = f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful AI assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
{example['instruction']}

{example['input']}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{example['output']}<|eot_id|>"""
    else:
        text = f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful AI assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
{example['instruction']}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{example['output']}<|eot_id|>"""
    return {"text": text}

dataset = Dataset.from_list(raw_data)
dataset = dataset.map(format_instruction)
train_test = dataset.train_test_split(test_size=0.1)

# ============================================================
# Step 4: Train
# ============================================================
training_args = SFTConfig(
    output_dir="./lora-output",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,     # effective batch = 4 x 4 = 16
    gradient_checkpointing=True,       # save VRAM (~20% speed tradeoff)
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    fp16=True,
    logging_steps=10,
    eval_steps=50,
    save_steps=100,
    eval_strategy="steps",
    load_best_model_at_end=True,
    max_seq_length=2048,
    dataset_text_field="text",
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_test["train"],
    eval_dataset=train_test["test"],
    tokenizer=tokenizer,
)

trainer.train()

# ============================================================
# Step 5: Save and optionally merge
# ============================================================
# Save only the LoRA adapter (small file)
model.save_pretrained("./lora-adapter")
tokenizer.save_pretrained("./lora-adapter")

# Optional: merge into base model for deployment
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged-model")

# Criteria for good training data
good_data_checklist = {
    "consistency": "Same question always answered in the same style",
    "diversity": "Covers all use cases you care about",
    "accuracy": "No incorrect information",
    "format": "Matches target model response style",
    "length": "Not too short, not unnecessarily long",
}

# Simple data quality check
def check_data_quality(examples):
    issues = []
    for i, ex in enumerate(examples):
        if len(ex["output"]) < 10:
            issues.append(f"Example {i}: response too short")
        if len(ex["output"]) > 2000:
            issues.append(f"Example {i}: response too long")
        if i > 0 and ex["output"] == examples[i-1]["output"]:
            issues.append(f"Example {i}: possible duplicate response")
    return issues

from unsloth import FastLanguageModel
import torch

# Much faster than standard transformers + PEFT
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-Instruct",
    max_seq_length=2048,
    load_in_4bit=True,
)

# LoRA config with Unsloth optimizations
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,     # Unsloth recommends 0
    bias="none",
    use_gradient_checkpointing="unsloth",  # Unsloth's optimized checkpointing
    random_state=3407,
)

Learning Rate:
- Too high: catastrophic forgetting (model loses existing capabilities)
- Too low: doesn't learn target behavior
- Recommended range: 1e-4 to 3e-4 (LoRA can use higher LR than full FT)

LoRA Rank (r):
- r=4: fast and lightweight, simple tasks
- r=8: good starting point for most use cases
- r=16: complex tasks, style learning
- r=64: approaching full fine-tuning quality, VRAM increases

Number of epochs:
- Small dataset (< 1,000 examples): 3-5 epochs
- Medium dataset (1,000-10,000 examples): 1-3 epochs
- Large dataset (> 10,000 examples): 1 epoch often sufficient

Effective batch size = per_device_batch x gradient_accumulation:
- Too small: unstable training
- Too large: risk of overfitting
- Typically 16-32 recommended

# Use early stopping
training_args = SFTConfig(
    ...
    eval_strategy="steps",
    eval_steps=50,
    load_best_model_at_end=True,   # auto-select best checkpoint
    metric_for_best_model="eval_loss",
    greater_is_better=False,
)

# Use tokenizer.apply_chat_template instead of manual formatting
messages = [
    {"role": "system", "content": "You are a professional translator."},
    {"role": "user", "content": "Translate this to French: Hello"},
    {"role": "assistant", "content": "Bonjour"}
]
formatted = tokenizer.apply_chat_template(messages, tokenize=False)