- Published on
Running Small Models Hands-On with a Single RTX 5090 — microGPT, OCR, Music Generation
- Authors
- Name
- Youngju Kim
- @fjvbn20031
- Introduction — Not the big models, the small ones, by hand
- Part 0 — First lesson: a benchmark without warmup lies by 38x
- Part 1 — microGPT: GPT from scratch, in 28 seconds
- Part 2 — OCR showdown: dedicated model vs. general-purpose VLM
- Part 3 — Music generation: 8 seconds of music in 1.9 seconds
- Closing — The age of small models: a single GPU is enough
- References
Introduction — Not the big models, the small ones, by hand
The whole conversation these days is 700B parameters and thousands of GPUs. But for learning computer science, the opposite is better — training a palm-sized model on a single GPU by hand, taking it apart, and hitting its limits. As it happened, a single RTX 5090 (Blackwell, 32GB) was sitting on the other side of an SSH connection, so I ran a trio of small models by hand: a char-level GPT trained from scratch, a dedicated OCR vs. small VLM showdown, and text-to-music. Every number is measured. And the honest traps I ran into along the way turned out to be better teaching material than the results themselves.
Part 0 — First lesson: a benchmark without warmup lies by 38x
Before running any models, I checked that the GPU was actually working with a bf16 matmul. The first measurement was 6.1 TFLOP/s. There's no way a 5090 does that... so I added a warmup loop and measured again.
First (no warmup) Measured properly
bf16 matmul 6.1 TFLOP/s 231.8 TFLOP/s ← 38x difference
The culprit was that the first call swept in the entire cost of cuBLAS initialization and kernel autotuning. The iron rule of GPU benchmarking — warm up a few times, wait for the kernels to actually finish with torch.cuda.synchronize(), and then measure. 231.8 TFLOP/s also lines up with the 5090's bf16 dense spec (~210 TFLOPS). Every latency measurement in this post includes this warmup.
For the record, there was also some upfront slog just to use this server: the 5090 is a Blackwell sm_120 architecture, so older PyTorch says "CUDA available" but then silently fails because the kernels don't exist. Only with torch 2.10.0+cu128 (the CUDA 12.8 build) did sm_120 show up in the architecture list. Half the battle of using the newest GPU is this environment fight.
Part 1 — microGPT: GPT from scratch, in 28 seconds
First, following Karpathy's nanoGPT lineage, I trained a char-level GPT from scratch in pure PyTorch. I fed it 1MB of Shakespeare text at the character level (vocab of 65) and ran a 6-layer, 6-head, 384-dimensional transformer (about 10.75M parameters) for 2000 steps.
iter 1 | train 3.7921 | val 3.8103 | 10.0k tok/s
iter 500 | train 1.5798 | val 1.7738 | 1006.2k tok/s
iter 1000 | train 1.2769 | val 1.5302 | 1119.1k tok/s
iter 1750 | train 1.1221 | val 1.4425 | 1174.8k tok/s ← val minimum
iter 2000 | train 1.1150 | val 1.4552 | 1184.7k tok/s ← val rebounds (overfitting begins)
trained 2000 iters in 28.0s | peak VRAM 1.73 GB | 1.17M tokens/s
28 seconds, 1.73GB of VRAM. Loss dropped from 3.79 to 1.12, and generated samples come out in this Shakespearean style:
The people, could be speak'd of this course,
Who shall be at bold well away and a chief.
Second Citizen:
This in the morniest hour of Turningh; down princ
The grammar is sloppy, but you can see it learned the dialogue format, character names, and archaic diction at the character level. And there's a hidden lesson here — val loss rebounded from 1.4425 at step 1750 to 1.4552 at step 2000. Train loss keeps falling while val rises: this is the textbook start of overfitting. With 10M parameters on 1MB of data, it starts memorizing quickly. It's the moment you see early stopping in the flesh. If you want to see the transformer architecture in a bigger picture, read this alongside the AI Model Development Lifecycle post.
The core code is spare, as befits modern PyTorch — attention isn't masked by hand but left to the flash attention kernel:
# Causal self-attention: is_causal=True replaces the triangular mask
y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
Part 2 — OCR showdown: dedicated model vs. general-purpose VLM
This is the most fun experiment. I put a dedicated OCR model (Microsoft TrOCR) and a small vision-language model (Qwen2-VL-2B) head-to-head on the same images. The metric is CER (Character Error Rate, lower is better). The test images are one line of English, one line of Korean, and a receipt mixing Korean, English, and numbers.
Model Image Latency CER Read result
──────────────────── ──────── ─────── ────── ──────────────────────────────
TrOCR-printed English 69ms 0.791 THE QUICK BROWN FOX ... LAY DOG
TrOCR-printed Korean 40ms 1.429 CHANGE @ SUBJECT EXCLIP
TrOCR-handwritten English 44ms 0.000 The quick brown fox ... lazy dog
Qwen2-VL-2B (VLM) English 90ms 0.000 The quick brown fox ... lazy dog
Qwen2-VL-2B (VLM) Korean 143ms 0.071 다람쥐 헌 챗바퀴에 타고파
Qwen2-VL-2B (VLM) Receipt 453ms — (KR/EN/num multi-line nearly perfect)
Three stories come out of this.
First, don't take a metric at face value. If you see TrOCR-printed's English CER of 0.791 and conclude it's "terrible," you're wrong. The actual output was THE QUICK BROWN FOX JUMPS OVER THE LAY DOG — it read almost everything correctly and just put it out in all caps (and dropped one z in lazy). Because CER counts uppercase and lowercase as different characters, a single capitalization inflated the score to 0.79. Ignore case and the CER is around 0.05. A bad metric doesn't mean a bad model — you have to know what the metric is counting.
Second, dedicated models are narrow. TrOCR was trained only on Latin script, so on a Korean image it produces complete nonsense like CHANGE @ SUBJECT EXCLIP (CER 1.429 — it exceeds 1 because of insertion errors). The model can't say "I don't know" and instead forces the input into characters it does know: the classic failure of out-of-domain input.
Third, an entertaining twist — the "handwriting" model TrOCR-handwritten read printed English perfectly at CER 0.000, even getting the case right. The "print" model smeared it into all caps while the "handwriting" one nailed it exactly. A name tag (the model name) doesn't guarantee performance — you don't know until you measure on real data.
And the general-purpose VLM's landslide win: Qwen2-VL-2B read English perfectly, missed only one character in Korean (챗/쳇), and above all transcribed the receipt mixing Korean, English, and numbers nearly verbatim, preserving even the line breaks. On top of that, it understands a natural-language instruction like "transcribe this receipt as-is" — something a dedicated OCR model can't do.
To sum up, it's the classic dedicated vs. general-purpose trade-off:
Dedicated OCR (TrOCR) General-purpose VLM (Qwen2-VL-2B)
────────── ─────────────────────── ─────────────────────────────
Speed 40~69ms (fast) 90~453ms (slow)
VRAM 1.39 GB (small) 4.49 GB (3x)
Language Latin only Multilingual
Layout Single line only Multi-line, tables, mixed
Instruction Not possible Instructable in natural language
If you're mass-processing single-line Latin documents at ultra-low latency and low cost, go with dedicated OCR; if you need multilingual support, complex layouts, and flexibility, go with a VLM — that's the intuition.
Part 3 — Music generation: 8 seconds of music in 1.9 seconds
Last is text-to-music. I threw two prompts at Meta's MusicGen-small (587M).
Prompt Gen time Music len Real-time factor
──────────────────────────────────────────────── ──────── ──────── ──────────
"upbeat lofi hip hop beat, warm piano, relaxing" 1.90s 7.9s 4.19x
"epic orchestral cinematic trailer, drums, brass" 1.85s 7.9s 4.30x
peak VRAM 1.37 GB
8 seconds of music in 1.9 seconds — generated at 4.2x real-time speed. MusicGen converts audio into EnCodec tokens (about 50 per second), produces them autoregressively like a language model, and finally decodes back into a waveform. It pulls out music as "tokens" the way an LLM pulls out text tokens — the grammar of generative AI carries over here intact. The output is saved as 32kHz mono WAV, and when you actually play it, you get the lofi beat and orchestra as prompted.
Closing — The age of small models: a single GPU is enough
What the three experiments have in common: all of them ran in under 5GB of VRAM and finished in seconds.
Experiment Parameters VRAM Key figure
──────────── ───────── ───────── ────────────────────────
microGPT 10.75M 1.73 GB 2000-step training in 28s
TrOCR 334M 1.39 GB single line read 40~69ms
Qwen2-VL-2B 2B 4.49 GB receipt transcription 453ms
MusicGen 587M 1.37 GB 8s of music in 1.9s
Big models grab the headlines, but the best place to learn and experiment is these small models. Training a GPT from scratch and watching overfitting with your own eyes, learning "dedicated vs. general-purpose" and "the trap of metrics" in your bones through an OCR showdown, confirming the token grammar of generative AI through music generation — all of it takes not a notebook but 30 seconds and a single GPU. If you're curious about the next step of moving to multiple GPUs, the Distributed Training Platforms post is the continuing story. And only the models you've run by hand stay with you as real intuition — because even a single benchmark number lies by 38x without warmup.