Skip to content
←

필사 모드: Positional Encoding — RoPE and the Price of Context Extension

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction

Transformer attention has no sense of order on its own. Positional encoding is the mechanism that supplies token order, and virtually every open-source model shipping today uses RoPE, rotary position embeddings.

In the config there are only two or three related fields: rope_theta, max_position_embeddings, and, when present, rope_scaling. This post is about how those few numbers lead to the outcome we call context length.

All figures were verified directly against papers, official reports, and config.json files on 2026-08-12. Models get updated, so check the originals again.

What RoPE Does

RoPE rotates the query and key vectors by an angle proportional to position. The dot product of two tokens then depends only on the difference between their positions, so relative distance enters attention naturally.

The key point is that each dimension rotates at a different speed. Early dimensions rotate quickly and resolve short distances finely, while later dimensions rotate very slowly and distinguish long distances. The value that sets those speeds is rope_theta.

Reading rope_theta as a Wavelength

Converting the number into a wavelength builds intuition. The wavelength of a dimension pair is computed like this.

wavelength(i) = 2 x pi x theta^(2i / head_dim)

With head_dim = 128, the wavelength of the slowest dimension:
  theta =    10,000  ->        54,410 positions
  theta =    50,000  ->       265,295 positions
  theta =   500,000  ->     2,559,196 positions
  theta = 1,000,000  ->     5,063,256 positions

If a wavelength is shorter than the context length, that dimension's positional signal wraps around. Two different positions end up pointing at the same angle, making them hard for the model to tell apart. Raising theta pushes the point where this wrapping happens further out.

Here are the real values. Llama 3 uses 500,000. The report states that the RoPE base frequency hyperparameter was increased to 500,000 to better support longer contexts, citing prior work showing this value effective for context lengths up to 32,768 (arXiv:2407.21783). The configs of Qwen3, Qwen2.5, Mixtral, and GLM-4.5 all use 1,000,000.

Changing theta During Training

Qwen3 does not set this value high from the start but raises it mid-training. According to the report, in the final long-context stage the RoPE base frequency is increased from 10,000 to 1,000,000 using the ABF technique (arXiv:2505.09388). By the table above, that stretches the slowest dimension's wavelength by roughly 93 times.

The ordering matters. At short context a smaller theta resolves nearby distances more finely, and theta is raised only when long context becomes necessary, enabling long-distance discrimination. This dovetails with a structure where most training happens cheaply at short context and only the final stage pays for expensive long-context training.

An Interesting Exception: Low theta With YaRN

Look at the DeepSeek-V3 config and rope_theta is 10,000. By the table above that gives a wavelength around fifty thousand, yet the model advertises a 128K context. The answer is in the rope_scaling entry of the same config.

{
  "rope_theta": 10000,
  "rope_scaling": {
    "type": "yarn",
    "factor": 40,
    "original_max_position_embeddings": 4096,
    "beta_fast": 32,
    "beta_slow": 1
  }
}

This means the original positional range of 4096 is extended by a factor of 40. The report likewise states that after pre-training, YaRN was applied through two phases of 1000 steps each, widening the window from 4K to 32K and then to 128K. The scale is 40, alpha is 1, and beta is 32, identical across both phases, and the extension is applied exclusively to the decoupled shared key (arXiv:2412.19437).

Kimi K2 pairs a rope_theta of 50,000 with a YaRN factor of 32. The same family of design, but with a base frequency set five times higher.

The YaRN paper reports that the method extends the context window with 10x fewer tokens and 2.5x fewer training steps than previous methods (arXiv:2309.00071). That says the extension itself is cheap, not that using the extended context is cheap.

Rotating Only Half

The GLM-4.5 config carries partial_rotary_factor set to 0.5. This means rotation is applied to only half the head dimensions while the other half is left position-independent. It is a design that keeps some components untied to position, and it reduces rotation compute along the way.

A similar split appears in models using MLA. DeepSeek-V3 keeps qk_nope_head_dim 128 and qk_rope_head_dim 64 separately. This divides the portion that receives no rotation from the portion that does, a structure that exists because positional information is hard to recover from a compressed latent vector, so the position-carrying component is pulled out and handled on its own.

Why Long Context Is Not Free

Switch on rope_scaling and the context grows on paper. But cost appears in three places.

First, prefill compute. Attention score computation scales with the square of sequence length. Quadruple the context and this portion becomes sixteen times the work.

Second, the KV cache. As computed in the previous post, the cache grows linearly with context length. For Qwen3-8B the fp16 cache is 0.56 GiB at 4,096 tokens, 4.50 GiB at 32,768 tokens, and 18.00 GiB at 131,072 tokens. There comes a point where the cache outweighs the weights.

Third, architectural choices themselves hinge on context length. The Kimi K2 report states that at a sequence length of 128k, raising attention heads from 64 to 128 increases inference FLOPs by 83 percent, and on that basis the model keeps 64 heads (arXiv:2507.20534). Presuming long context changes the design decisions.

And there is no guarantee that extension techniques preserve quality, which is why reports include separate long-context evaluations after extending.

Growing It in Stages

Llama 3 handled extension through training. According to the report, pre-training of the 405B model consists of three stages — initial pre-training, long-context pre-training, and annealing — and the context length was increased gradually in six stages from the original 8K to a final 128K, with roughly 800B tokens spent in that long-context stage.

So there are broadly two roads to long context: grow it by training, or grow it by an extension technique. The former is expensive, the latter cheap but in greater need of verification.

Closing

When you look at rope_theta, convert it into a wavelength. Whether that value is comfortably larger than the context length is the first check. If rope_scaling is present, that model's context length is an extended value rather than a trained one. And when extending context, remember to put prefill compute and cache memory into the same calculation.

References

Try It Yourself

Series

현재 단락 (1/51)

Transformer attention has no sense of order on its own. Positional encoding is the mechanism that su...

작성 글자: 0자원문 글자: 6,669자작성 단락: 0/51