Skip to content
←

필사 모드: Inside the Tokenizer — Why Korean Costs More Tokens, and What That Costs

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction

The same content is 100 tokens in English and 190 tokens in Korean. API billing and the rate at which you fill the context window differ by exactly that much. This post explains where that difference comes from, using numbers measured by downloading the actual tokenizers.

The measurements here are not paper citations but results from runs performed directly. Public tokenizer files were downloaded and the same sentences tokenized, so anyone can reproduce them.

All figures were verified directly against papers, official reports, and config.json files on 2026-08-12. Models get updated, so check the originals again.

Byte-Level BPE

Nearly every model shipping today uses byte-level BPE. The Qwen3 report states that the Qwen tokenizer implements byte-level byte-pair encoding with a vocabulary size of 151,669 (arXiv:2505.09388).

The mechanism is easiest to understand in two steps. First, all text is converted to UTF-8 bytes. Then byte pairs that frequently co-occur in the training corpus are repeatedly merged to form longer pieces. Since any character is representable at least at the byte level, the out-of-vocabulary problem disappears.

This is where the disadvantage for Korean begins. ASCII letters are 1 byte, but Hangul syllables and most CJK characters occupy 3 bytes in UTF-8. Any piece for which no merge rule was learned gets split back into bytes, and one character becomes three tokens.

Measuring It Directly

Tokenizer files were downloaded from public repositories and used to tokenize text with the same meaning. The paragraphs carry identical content in 392 English characters and 199 Korean characters.

                English 392 ch     Korean 199 ch
                (UTF-8 392B)       (UTF-8 489B)

Qwen3-8B          76 tokens          148 tokens      -> 1.95x
DeepSeek-V3       76 tokens          142 tokens      -> 1.87x
Mixtral-8x7B      81 tokens          218 tokens      -> 2.69x

characters per token (Korean)
Qwen3-8B      1.34 ch/token
DeepSeek-V3   1.40 ch/token
Mixtral-8x7B  0.91 ch/token

Look at the last line for Mixtral. Characters per token falls below 1. That means representing a single Hangul character takes more than one token, a signal that merge rules are scarce and text is being split at the byte level.

A per-language comparison on a short sentence, one sentence with the same meaning throughout.

language   chars   UTF-8   Qwen3   DeepSeek-V3   Mixtral
English      73     73B      13         13          13
Korean       29     73B      22         19          32
Japanese     28     84B      19         20          29
Chinese      19     57B      12         10          20

Chinese is the most efficient. Each character carries more meaning, and frequently used characters are merged into single tokens. Korean and Japanese both spend more tokens than English here, and which of the two fares better flips depending on the tokenizer. This is a single-sentence measurement, so take it as a tendency only.

The Vocabulary-Size Tradeoff

The vocabulary sizes of the three tokenizers are 151,669, 128,815, and 32,000. The difference shows up directly in the results. Mixtral, with a vocabulary of 32,000, spends 218 tokens on the Korean paragraph, while Qwen3 at around 150,000 spends 148.

So should vocabulary just grow without limit? No. There is a cost.

First, the embedding matrix grows. Qwen3-8B has a width of 4096, so a single embedding is about 620 million parameters. Doubling the vocabulary adds 600 million parameters from this alone. The smaller the model, the larger that share becomes.

Second, the output softmax grows. Probabilities are computed over the entire vocabulary at every token, so vocabulary size becomes the compute of the final layer directly.

Third, rare tokens receive little training signal. If a token created by expanding the vocabulary appears rarely in the corpus, its embedding never trains adequately.

The Llama 3 case illustrates this balance well. The report sets the vocabulary size at 128,000, adds 28K tokens to an existing tokenizer to better support non-English languages, and states that the compression rate on a sample of English data improved from 3.17 to 3.94 characters per token. The same passage notes that adding 28K tokens from select non-English languages improved both compression ratios and downstream performance, with no impact on English tokenization (arXiv:2407.21783).

The config vocab_size Is Not the Vocabulary Count

Here we confirm the trap flagged in the first post. The Qwen3-8B config gives vocab_size as 151936. Yet the vocabulary size stated in the report is 151,669, and opening the tokenizer file directly also yields 151,669.

model            config vocab_size    actual tokenizer vocabulary
Qwen3-8B             151,936                  151,669
DeepSeek-V3          129,280                  128,815
Mixtral-8x7B          32,000                   32,000

The gap is padding that rounds the embedding matrix up to a hardware-friendly size. Sometimes, as with Mixtral, it lines up exactly. Use the config value when counting parameters, and the tokenizer value when discussing token counts.

What This Means in Practice

Token efficiency turns into cost in three places.

First, the context window. Even with a 128K context, feeding Korean documents fits roughly half the source material of English into the same window. As seen in the previous post, longer context raises prefill compute quadratically and the KV cache linearly. Twice the tokens means twice the cache memory.

Second, generation speed. Generation happens token by token, so a Korean answer of the same length requires more decoding steps than an English one.

Third, the efficiency of training data. When the Llama 3 report describes its compression improvement as enabling the model to read more text for the same amount of training compute, this is precisely what it means.

What to Check When Choosing a Model

If you work heavily in Korean, there is something to check before benchmark scores: tokenize the documents you will actually handle with that model's tokenizer. Tokenizer files are usually public, so you can check without downloading the model weights.

The same document becomes 1.9 times longer on one model and 2.7 times on another. That difference never appears in a benchmark table, but it lands directly in operating cost.

Closing

The tokenizer draws the least attention of any part of model architecture, yet for Korean speakers it is the most keenly felt. Under byte-level BPE, three-byte characters are structurally disadvantaged, and raising vocabulary size improves matters while raising embedding and softmax costs alongside. Measuring it yourself is the surest approach, and reproducing it is not hard.

References

Try It Yourself

Series

현재 단락 (1/53)

The same content is 100 tokens in English and 190 tokens in Korean. API billing and the rate at whic...

작성 글자: 0자원문 글자: 6,408자작성 단락: 0/53