Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — Not the big models, the small ones, by hand
Part 0 — First lesson: a benchmark without warmup lies by 38x
Part 1 — microGPT: GPT from scratch, in 28 seconds
Part 2 — OCR showdown: dedicated model vs. general-purpose VLM
Part 3 — Music generation: 8 seconds of music in 1.9 seconds
Closing — The age of small models: a single GPU is enough
References

Introduction — Not the big models, the small ones, by hand

The whole conversation these days is 700B parameters and thousands of GPUs. But for learning computer science, the opposite is better — training a palm-sized model on a single GPU by hand, taking it apart, and hitting its limits. As it happened, a single RTX 5090 (Blackwell, 32GB) was sitting on the other side of an SSH connection, so I ran a trio of small models by hand: a char-level GPT trained from scratch, a dedicated OCR vs. small VLM showdown, and text-to-music. Every number is measured. And the honest traps I ran into along the way turned out to be better teaching material than the results themselves.

Part 0 — First lesson: a benchmark without warmup lies by 38x

Before running any models, I checked that the GPU was actually working with a bf16 matmul. The first measurement was 6.1 TFLOP/s. There's no way a 5090 does that... so I added a warmup loop and measured again.

                 First (no warmup)    Measured properly
bf16 matmul       6.1 TFLOP/s          231.8 TFLOP/s     ← 38x difference

The culprit was that the first call swept in the entire cost of cuBLAS initialization and kernel autotuning. The iron rule of GPU benchmarking — warm up a few times, wait for the kernels to actually finish with torch.cuda.synchronize(), and then measure. 231.8 TFLOP/s also lines up with the 5090's bf16 dense spec (~210 TFLOPS). Every latency measurement in this post includes this warmup.

For the record, there was also some upfront slog just to use this server: the 5090 is a Blackwell sm_120 architecture, so older PyTorch says "CUDA available" but then silently fails because the kernels don't exist. Only with torch 2.10.0+cu128 (the CUDA 12.8 build) did sm_120 show up in the architecture list. Half the battle of using the newest GPU is this environment fight.

Part 1 — microGPT: GPT from scratch, in 28 seconds

First, following Karpathy's nanoGPT lineage, I trained a char-level GPT from scratch in pure PyTorch. I fed it 1MB of Shakespeare text at the character level (vocab of 65) and ran a 6-layer, 6-head, 384-dimensional transformer (about 10.75M parameters) for 2000 steps.

iter     1 | train 3.7921 | val 3.8103 |   10.0k tok/s
iter   500 | train 1.5798 | val 1.7738 | 1006.2k tok/s
iter  1000 | train 1.2769 | val 1.5302 | 1119.1k tok/s
iter  1750 | train 1.1221 | val 1.4425 | 1174.8k tok/s   ← val minimum
iter  2000 | train 1.1150 | val 1.4552 | 1184.7k tok/s   ← val rebounds (overfitting begins)

trained 2000 iters in 28.0s | peak VRAM 1.73 GB | 1.17M tokens/s

28 seconds, 1.73GB of VRAM. Loss dropped from 3.79 to 1.12, and generated samples come out in this Shakespearean style:

The people, could be speak'd of this course,
Who shall be at bold well away and a chief.

Second Citizen:
This in the morniest hour of Turningh; down princ

The grammar is sloppy, but you can see it learned the dialogue format, character names, and archaic diction at the character level. And there's a hidden lesson here — val loss rebounded from 1.4425 at step 1750 to 1.4552 at step 2000. Train loss keeps falling while val rises: this is the textbook start of overfitting. With 10M parameters on 1MB of data, it starts memorizing quickly. It's the moment you see early stopping in the flesh. If you want to see the transformer architecture in a bigger picture, read this alongside the AI Model Development Lifecycle post.

The core code is spare, as befits modern PyTorch — attention isn't masked by hand but left to the flash attention kernel:

# Causal self-attention: is_causal=True replaces the triangular mask
y = F.scaled_dot_product_attention(q, k, v, is_causal=True)

Part 2 — OCR showdown: dedicated model vs. general-purpose VLM

This is the most fun experiment. I put a dedicated OCR model (Microsoft TrOCR) and a small vision-language model (Qwen2-VL-2B) head-to-head on the same images. The metric is CER (Character Error Rate, lower is better). The test images are one line of English, one line of Korean, and a receipt mixing Korean, English, and numbers.

Model                 Image     Latency  CER      Read result
────────────────────  ────────  ───────  ──────  ──────────────────────────────
TrOCR-printed         English   69ms     0.791   THE QUICK BROWN FOX ... LAY DOG
TrOCR-printed         Korean    40ms     1.429   CHANGE @ SUBJECT EXCLIP
TrOCR-handwritten     English   44ms     0.000   The quick brown fox ... lazy dog
Qwen2-VL-2B (VLM)     English   90ms     0.000   The quick brown fox ... lazy dog
Qwen2-VL-2B (VLM)     Korean   143ms     0.071   다람쥐 헌 챗바퀴에 타고파
Qwen2-VL-2B (VLM)     Receipt  453ms      —      (KR/EN/num multi-line nearly perfect)

Three stories come out of this.

First, don't take a metric at face value. If you see TrOCR-printed's English CER of 0.791 and conclude it's "terrible," you're wrong. The actual output was THE QUICK BROWN FOX JUMPS OVER THE LAY DOG — it read almost everything correctly and just put it out in all caps (and dropped one z in lazy). Because CER counts uppercase and lowercase as different characters, a single capitalization inflated the score to 0.79. Ignore case and the CER is around 0.05. A bad metric doesn't mean a bad model — you have to know what the metric is counting.

Second, dedicated models are narrow. TrOCR was trained only on Latin script, so on a Korean image it produces complete nonsense like CHANGE @ SUBJECT EXCLIP (CER 1.429 — it exceeds 1 because of insertion errors). The model can't say "I don't know" and instead forces the input into characters it does know: the classic failure of out-of-domain input.

Third, an entertaining twist — the "handwriting" model TrOCR-handwritten read printed English perfectly at CER 0.000, even getting the case right. The "print" model smeared it into all caps while the "handwriting" one nailed it exactly. A name tag (the model name) doesn't guarantee performance — you don't know until you measure on real data.

And the general-purpose VLM's landslide win: Qwen2-VL-2B read English perfectly, missed only one character in Korean (챗/쳇), and above all transcribed the receipt mixing Korean, English, and numbers nearly verbatim, preserving even the line breaks. On top of that, it understands a natural-language instruction like "transcribe this receipt as-is" — something a dedicated OCR model can't do.

To sum up, it's the classic dedicated vs. general-purpose trade-off:

              Dedicated OCR (TrOCR)    General-purpose VLM (Qwen2-VL-2B)
──────────  ───────────────────────  ─────────────────────────────
Speed         40~69ms (fast)           90~453ms (slow)
VRAM          1.39 GB (small)          4.49 GB (3x)
Language      Latin only               Multilingual
Layout        Single line only         Multi-line, tables, mixed
Instruction   Not possible             Instructable in natural language

If you're mass-processing single-line Latin documents at ultra-low latency and low cost, go with dedicated OCR; if you need multilingual support, complex layouts, and flexibility, go with a VLM — that's the intuition.

Part 3 — Music generation: 8 seconds of music in 1.9 seconds

Last is text-to-music. I threw two prompts at Meta's MusicGen-small (587M).

Prompt                                            Gen time   Music len   Real-time factor
────────────────────────────────────────────────  ────────  ────────  ──────────
"upbeat lofi hip hop beat, warm piano, relaxing"   1.90s     7.9s       4.19x
"epic orchestral cinematic trailer, drums, brass"  1.85s     7.9s       4.30x

peak VRAM 1.37 GB

8 seconds of music in 1.9 seconds — generated at 4.2x real-time speed. MusicGen converts audio into EnCodec tokens (about 50 per second), produces them autoregressively like a language model, and finally decodes back into a waveform. It pulls out music as "tokens" the way an LLM pulls out text tokens — the grammar of generative AI carries over here intact. The output is saved as 32kHz mono WAV, and when you actually play it, you get the lofi beat and orchestra as prompted.

Closing — The age of small models: a single GPU is enough

What the three experiments have in common: all of them ran in under 5GB of VRAM and finished in seconds.

Experiment      Parameters  VRAM        Key figure
────────────  ─────────  ─────────  ────────────────────────
microGPT       10.75M     1.73 GB    2000-step training in 28s
TrOCR          334M       1.39 GB    single line read 40~69ms
Qwen2-VL-2B    2B         4.49 GB    receipt transcription 453ms
MusicGen       587M       1.37 GB    8s of music in 1.9s

Big models grab the headlines, but the best place to learn and experiment is these small models. Training a GPT from scratch and watching overfitting with your own eyes, learning "dedicated vs. general-purpose" and "the trap of metrics" in your bones through an OCR showdown, confirming the token grammar of generative AI through music generation — all of it takes not a notebook but 30 seconds and a single GPU. If you're curious about the next step of moving to multiple GPUs, the Distributed Training Platforms post is the continuing story. And only the models you've run by hand stay with you as real intuition — because even a single benchmark number lies by 38x without warmup.