Choosing Open Text Generation Models by Size Class

• Size Is a Deployment Constraint, Not a Quality Tier
• Small: On-Device and Edge
• Mid: One Single GPU
• Large: The Server-Class Band
• MoE Changes the Memory Math
• Context Extension Is Not Free
• Do Not Ignore the Runtime Conditions the Card Demands
• Invocation Examples
• The Order to Decide In
• Try It Yourself
• Series Navigation
• References

Model details were read directly from the Hugging Face pages on 2026-08-12. Model cards and licenses change, so check the original again before you use anything.

Size Is a Deployment Constraint, Not a Quality Tier

The reason to sort open models by size is not that bigger means better. Size decides where you can put the model, how long a single request may take, and how many replicas you have to run. For the same task, a 300-millisecond latency budget and a 3-second one lead to completely different answers.

So this post does not rank models. It lists the card values actually verified in each band and points out how those values feed a deployment decision.

Small: On-Device and Edge

In this band, read the limitations the cards state about themselves first. HuggingFaceTB/SmolLM2-1.7B-Instruct says to treat it as an assistive tool rather than a definitive source of information, and that output may not be factually accurate, logically consistent, or free from biases, so important information should be verified. microsoft/Phi-4-mini-instruct states that model size limits how much factual knowledge it can store, that quality varies for non-English languages, and that the majority of its code training was Python.

meta-llama/Llama-3.2-1B-Instruct requires accepting conditions and sharing contact information before file access. On-device deployment usually bakes weights in CI, and this repository simply fails in a token-free pipeline.

Mid: One Single GPU

The expectation that most often breaks in this band is language coverage. Korean and Japanese are not among the languages the meta-llama/Llama-3.1-8B-Instruct card officially lists. Qwen/Qwen3-8B and google/gemma-3-4b-it state 100+ and 140+ languages respectively. But those sentences are declarations of coverage, not quality guarantees, so you have to measure with sentences from your own domain.

The mistralai/Mistral-7B-Instruct-v0.3 card states it has no moderation mechanisms at all. Fine for an internal tool; put it at a customer touchpoint and you are committing to a separate filter layer.

Large: The Server-Class Band

The roughly 55 GB figure on the mistralai/Mistral-Small-24B-Instruct-2501 card is the most practical piece of information in this band. It is common to see 24B parameters, picture a single GPU, and discover right before deployment that bf16 means one 80GB-class card or several 24GB ones.

MoE Changes the Memory Math

openai/gpt-oss-20b states 21B parameters with 3.6B active. The confusing part is that compute behaves like a 3.6B model while all 21B of the weights still have to sit in memory. The card adding that it runs within 16GB assumes MXFP4 quantization.

So for an MoE model, compute speed from the active parameters and memory from the total parameters plus the quantization format, separately. Collapse them into one number and you will be wrong in one direction or the other.

Context Extension Is Not Free

The Qwen3 cards are unusually candid about extension. Qwen/Qwen3-8B, Qwen/Qwen3-4B, and Qwen/Qwen3-32B all state a 32,768 default extendable to 131,072 with YaRN, while warning that the static YaRN implemented in open-source frameworks keeps the scaling factor constant regardless of input length, which can affect performance on shorter texts.

Practically, that means requests needing 128K and requests finishing in 2K should not share an endpoint. Splitting long-document serving from general chat serving is usually cheaper and steadier.

Do Not Ignore the Runtime Conditions the Card Demands

Three get ignored most often.

First, the openai/gpt-oss-20b card states the model was trained on the harmony response format and should only be used with it, as it will not work correctly otherwise. Concluding that quality is poor without matching the format is a bad judgment.

Second, the Qwen3 cards explicitly say not to use greedy decoding in thinking mode, warning it can lead to endless repetitions. Qwen/Qwen3-32B records temperature 0.6 and TopP 0.95 as the thinking-mode settings.

Third, base and instruct variants differ by a single word in the name. The Qwen/Qwen3-4B card states that thinking-mode switching is controlled by the enable_thinking argument with /think and /no_think soft switches, so there is no mystery when a runtime that does not support the argument fails to switch modes.

Invocation Examples

# Example: call an instruct model through its chat template
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Qwen/Qwen3-4B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "Write a three-line summary of these minutes."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256, temperature=0.6, top_p=0.95)
print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Served as an endpoint, the runtime manages context and batching for you.

# Example: serve behind an OpenAI-compatible endpoint
vllm serve Qwen/Qwen3-8B --max-model-len 32768

The Order to Decide In

1. Fix the latency budget and the deployment location first. The size band falls out of that.
2. Within the band, filter candidates by license and gating.
3. Check the language coverage each survivor states, then measure with sentences from your own domain.
4. Check whether your context requirement exceeds the default, and if so read the extension mechanism and its warnings.
5. Compare quality only after honoring the format and sampling conditions the card demands.

Try It Yourself

• GPU VRAM Calculator for LLMs — enter parameters and context to see which band fits your GPU.
• LLM API Cost Calculator — use it as the baseline when comparing self-hosting against API calls.
• Browser AI Lab — feel the actual latency of a small model yourself.

Series Navigation

• Previous: How to Read a Hugging Face Model Card
• Next: Embeddings and Rerankers: What Actually Matters in RAG

References

• Every value in the tables was read directly from that model page on Hugging Face on 2026-08-12. Anything absent from the page is written as not stated.
• Benchmark numbers on the cards are self-reported by publishers and are not independent evaluations. This post builds no cross-model score comparison.
• Read the full license text yourself and put commercial use through legal review.