Split View: 한국어를 지원하는 오픈 모델과 토크나이저 비용

Open Models That Support Korean, and the Cost of Tokenization

• Supporting Korean Means One of Two Things
• Models Built with Korean in Mind
• How to Check the Korean Line on a Multilingual Card
• How a Tokenizer Splits Korean Is the Cost
• How to Measure Token Counts Yourself
• Context Length Is Counted in Tokens, Not Characters
• Where Licenses Diverge Most
• When You Design Korean Retrieval Alongside
• Try It Yourself
• Series Navigation
• References

Model details were read directly from the Hugging Face pages on 2026-08-12. Model cards and licenses change, so check the original again before you use anything.

Supporting Korean Means One of Two Things

A card listing Korean means one of two things: that Korean made up a meaningful share of the training data, or that Korean was present somewhere in a multilingual corpus. The real quality gap between those two cases is large, and the card wording does not distinguish them.

The reverse, however, is a much clearer signal. The languages the meta-llama/Llama-3.1-8B-Instruct card officially lists are English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai — Korean is not among them. meta-llama/Llama-3.2-1B-Instruct uses the same list and marks use in languages beyond those explicitly referenced as out of scope.

Models Built with Korean in Mind

Several facts only surface if you read the card. K-intelligence/Midm-2.0-Mini-Instruct states it is optimized for on-device environments and systems with limited GPU resources, and its sibling K-intelligence/Midm-2.0-Base-Instruct describes itself as Korea-centric AI that internalizes the values and commonsense reasoning of Korean society. Both cards state that training was limited to English and Korean so other languages are not guaranteed, and that the model is not guaranteed to give reliable advice in fields requiring professional expertise such as law, medicine, or finance.

naver-hyperclovax/HyperCLOVAX-SEED-Text-Instruct-1.5B requires agreeing to share contact information for access, and the card states that vulnerability to long-context handling was observed in smaller models prior to reinforcement training. upstage/SOLAR-10.7B-Instruct-v1.0 is easily mistaken for a Korean model because of its origin, but the language its card states is English, and it notes the model was fine-tuned primarily for single-turn conversation, making it less suitable for multi-turn chat.

How to Check the Korean Line on a Multilingual Card

Korean-focused models are not the only candidates. The microsoft/Phi-4-mini-instruct card explicitly includes Korean in its language list, and mistralai/Mistral-Small-24B-Instruct-2501 lists Korean as well. Qwen/Qwen3-8B states more than 100 languages and google/gemma-3-4b-it more than 140.

What separates them is the specificity of the claim. A card that enumerates languages one by one probably had them in view during evaluation; a card that states only a count says nothing beyond corpus inclusion. Neither replaces measuring with sentences from your own domain.

How a Tokenizer Splits Korean Is the Cost

After quality, the thing that matters most for Korean is token efficiency, for a simple reason: API pricing, context limits, and generation speed are all denominated in tokens rather than characters.

In most BPE tokenizers an English word costs one or two tokens. Korean is agglutinative — particles and endings attach to stems, so the same stem appears in different surface forms depending on context, and a sentence carrying the same meaning frequently splits into more tokens. That difference becomes cost in three places at once: input tokens per request rise, the context window fills faster, and output tokens grow, which raises latency.

Vocabulary size affects this but not proportionally. The config.json for Qwen/Qwen3-8B carries a vocab_size of 151936. A larger vocabulary creates room to express a sentence in fewer tokens, but how that vocabulary was allocated across languages cannot be read off the number. In the end there is no substitute for measuring.

How to Measure Token Counts Yourself

# Example: split the same sentence with several tokenizers and compare
from transformers import AutoTokenizer

repos = [
    "Qwen/Qwen3-8B",
    "microsoft/Phi-4-mini-instruct",
    "K-intelligence/Midm-2.0-Mini-Instruct",
]

samples = [
    "연차 이월 규정은 취업규칙 제12조에 따라 운영됩니다.",
    "Annual leave carryover follows Article 12 of the employment rules.",
]

for repo in repos:
    tok = AutoTokenizer.from_pretrained(repo)
    counts = [len(tok.encode(s, add_special_tokens=False)) for s in samples]
    print(repo, "ko:", counts[0], "en:", counts[1], "vocab:", tok.vocab_size)

Run this over about 200 real sentences from your own service and you get the Korean token cost of each candidate immediately. It is a number you can obtain much faster than a leaderboard score, and it stays valid much longer.

For gated repositories this code fails without an access token. The meta-llama family, google/gemma-3-27b-it, and naver-hyperclovax/HyperCLOVAX-SEED-Text-Instruct-1.5B all state on their pages that accepting conditions or sharing contact information is required.

Context Length Is Counted in Tokens, Not Characters

The 32,768 of LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct and the 131,072 of LGAI-EXAONE/EXAONE-4.0-32B are both token counts. Two models with the same limit but different token efficiency hold different amounts of actual Korean text.

So for a service handling long documents, any comparison of context limits has to carry the measurement from the previous section along with it. A bigger number on the card is no guarantee that more of your documents fit. Some models do not state a context length at all: the K-intelligence pages and kakaocorp/kanana-nano-2.1b-instruct do not.

Where Licenses Diverge Most

Korean models carry an unusually wide spread of license types. K-intelligence/Midm-2.0-Base-Instruct and K-intelligence/Midm-2.0-Mini-Instruct are marked MIT. kakaocorp/kanana-nano-2.1b-instruct is cc-by-nc-4.0 and its card states the non-commercial condition. upstage/SOLAR-10.7B-Instruct-v1.0 is cc-by-nc-4.0 as well.

The EXAONE family uses its own license. LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct is marked EXAONE AI Model License Agreement 1.1 - NC and LGAI-EXAONE/EXAONE-4.0-32B as 1.2 - NC, and the latter page states that the license restricts use toward developing models that compete with EXAONE. naver-hyperclovax/HyperCLOVAX-SEED-Text-Instruct-1.5B uses its own hyperclovax-seed identifier, and commercial terms are not separately stated on the page. Bllossom/llama-3.2-Korean-Bllossom-3B follows llama3.2, and its card contains a sentence saying commercial use is possible.

Read the full license text yourself and put commercial use through legal review. This post does not decide which model you may use commercially; it only reports what the cards say.

When You Design Korean Retrieval Alongside

For Korean RAG, the generation model is only half the job. Embeddings trained further for Korean include nlpai-lab/KURE-v1, nlpai-lab/KoE5, and dragonkue/BGE-m3-ko, whose dimensions and maximum input lengths are laid out in part 3 of this series. Note in particular that nlpai-lab/KoE5 inherits the 512-token limit of its base model, so with a poor token-efficiency pairing even a single paragraph may not fit.

Try It Yourself

• LLM API Cost Calculator — feed in your token measurements to price a Korean workload.
• GPU VRAM Calculator for LLMs — check whether the candidates fit the hardware you have.
• Prompt Engineering Guide — practice writing the same instruction in fewer tokens.

Series Navigation

• Previous: Embeddings and Rerankers: What Actually Matters in RAG
• Next: Choosing Speech Models: Practical Criteria for STT and TTS

References

• Every value in the tables was read directly from that model page on Hugging Face on 2026-08-12. Anything absent from the page is written as not stated.
• No token-count figures are published in this post. They vary heavily with sentence composition, so measuring on your own data is more accurate.
• Read the full license text yourself and put commercial use through legal review.