Skip to content
Published on

Do You Need a New Learning Rate When You Change LoRA Rank? — The Two Regimes μA (2026) Splits, and the Limits of That Evidence

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — I Raised the Rank and Had to Re-Find the Learning Rate

Anyone who has actually run LoRA in production has hit this. A recipe that worked fine at rank 16 starts spiking the loss or stalling training once you bump it to rank 64. You have to re-sweep the learning rate. But talk to another team and you hear the opposite — "You can just raise the rank, no need to touch the learning rate."

Both are genuinely observed phenomena. And this isn't a contradiction born from practitioners being sloppy. Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning (Nan Chen, Soledad Villar, Soufiane Hayou, arXiv:2602.06204), posted to arXiv on February 5, 2026, says the two beliefs are true statements about different setups. Which one applies to you is decided by the α convention and the initialization you use.

This post lays out exactly what the paper derived, what its experiments measured, and what they didn't. The short version: a useful rule falls out of it. But the evidence behind it is a single seed (42) on a 2x-spaced grid, and the authors themselves spell out the limits. Carrying those limits along too is the point of this post.

The Shape of the Problem — α, Initialization, Rank, and Learning Rate Are Tangled Together

Let's align on the paper's notation first. LoRA adds a low-rank term to frozen pretrained weights.

W  ←  W + α·B·A

  A ∈ R^(r×n)   (down-projection)
  B ∈ R^(n×r)   (up-projection)
  n = model width,  r = LoRA rank,  α = LoRA multiplier

There are four knobs a practitioner turns here — rank r, multiplier α, learning rate η, and initialization. To make the adapter a no-op at the start, one factor is set to zero and only the other is filled with random values; the paper names these two choices as follows.

  • Init[A] — A is random, B = 0. This is what the original LoRA uses, and it's the default in most implementations.
  • Init[B] — B is random, A = 0. The flipped version.

The multiplier α isn't singular either. Per the paper's Section 2 summary, the original LoRA (Hu et al., 2022) uses α = r^(-1) to keep the update size down; rsLoRA (Kalajdzievski, 2023) proposed α = r^(-1/2) for stability at large rank; and more recent work increasingly uses α = 1.

This is where practitioners get confused. Hugging Face PEFT doesn't take α directly. Open the PEFT v0.17.0 source (the version the paper actually used) and you find this.

# peft/tuners/lora/layer.py — update_layer()
if use_rslora:
    self.scaling[adapter_name] = lora_alpha / math.sqrt(r)
else:
    self.scaling[adapter_name] = lora_alpha / r

That is, when use_rslora=False, the effective multiplier is lora_alpha / r. Converted into the paper's α, that gives:

lora_alpha = 32 (a constant, independent of rank)  ->  effective α = 32/r   ∝ r^(-1)
lora_alpha = r  (scaled up together with rank)      ->  effective α = 1      (constant)
lora_alpha = 2r                                     ->  effective α = 2      (constant)

That means the fixed lora_alpha=32 common in tutorials, and the increasingly common advice to "match lora_alpha to rank (or double it)," are different mathematical regimes. As the paper's Section 1 puts it, the LoRA update is bilinear in A and B, so initialization, α, rank, and η are entangled and jointly determine the effective update size. That's why hyperparameter tuning breaks so easily.

μA — Where the Two Regimes Split

The paper computes the size of a single-step feature update for LoRA in the limit where width and rank go to infinity simultaneously (n, r → ∞). It's named μA (Maximal-Update Adaptation) because it carries pretraining's μP (Maximal-Update Parametrization) over to fine-tuning. The goal is to find the learning rate at which the feature update neither blows up nor vanishes — stays Θ(1).

The assumptions have to come first. Section 4 of the paper spells out three.

  1. At every step, the forward-pass inputs and backward-pass gradients are bounded at Θ(1) (Assumption 4.1)
  2. The optimizer is approximated as momentum-free Adam = SignSGD (Assumption 4.2). The experiments themselves actually run on AdamW.
  3. The loss is computed on a single sample basis (following Hayou et al., 2024b)

Two results fall out on top of this.

Init[A] (Corollary 4.4). Set α = r^(-γ) with γ between 0 and 1, and the learning rate required for stable feature learning is

η = Θ( n^(-1/2) · r^(-(1-γ)/2) )

  γ=0   (α = 1)          ->  η ∝ n^(-1/2) · r^(-1/2)     rank-dependent
  γ=1/2 (α = r^(-1/2))   ->  η ∝ n^(-1/2) · r^(-1/4)     weakly rank-dependent
  γ=1   (α = r^(-1))     ->  η ∝ n^(-1/2)                rank-independent

Carrying over the paper's own reading: at α = 1, quadrupling the rank means you have to halve the learning rate; at α = r^(-1), the rank dependence gets absorbed into the multiplier and the learning rate becomes rank-independent.

Init[B], α = 1 (Corollary 4.6).

η = Θ( n^(-1) )      rank-independent, depends only on width

This is where the paper's central observation lands. Under the same SignSGD approximation, full fine-tuning (FFT) of a linear layer also gives η ∝ n^(-1) (Appendix A.6, Theorem A.23). The width scaling matches — this is the necessary condition for transferring a learning rate between LoRA and FFT.

There's a lesser-known byproduct too. In Init[A], as rank grows, A's contribution vanishes and learning effectively happens through B alone. In the paper's phrasing, the adapter increasingly behaves like "a random projection with A frozen at its initial value," and this imbalance doesn't disappear for any of Init[A]'s standard α choices. In Init[B], it's the reverse — when r is much smaller than n, B's update becomes negligible and learning happens mainly through A.

Why the "10x Learning Rate" Folklore Kept Surviving

Now the earlier contradiction resolves.

Thinking Machines' LoRA Without Regret (John Schulman et al.), posted last year on September 29, wrote this — they used W' = W + (α/r)BA with α fixed at 32, and found that "the 1/r scaling factor makes the optimal learning rate roughly independent of rank." Early in training, the loss curves were identical across different ranks, to the point that they even write they initially suspected a bug where the rank factor was being ignored.

Translated into μA coordinates, Schulman's setup is effective α = 32/r, i.e., γ=1 — exactly the cell where Corollary 4.4 predicts rank-independence. Of the two competing beliefs, the "rank-independent" one is true — under the condition that you use the classic 1/r convention. And the moment you start scaling lora_alpha with rank, you've moved into the γ=0 cell, where quadrupling the rank means halving the learning rate. This is why the same person, using the same library, can have opposite experiences.

But there's one thing μA doesn't resolve. The famous rule Schulman reported — "LoRA's optimal learning rate is 10x FullFT's, for the same application." Here's the evidence behind it. They swept LoRA and FullFT learning rates separately across 14 models in the Llama and Qwen families on the Tulu3 dataset, and fit a function predicting the optimal learning rate from hidden size and model family (Llama/Qwen). That fit found a LoRA multiplier of 9.8, and LoRA's hidden-size dependence exponent matched FullFT's (the fit gave LoRA pow = 0). They also noted a preliminary observation that the multiplier is larger on short runs (~15x, with a footnote saying "this is anecdotal evidence and holds for roughly under 100 steps").

What Schulman himself attached to this rule matters — "we don't yet have a satisfying theoretical explanation for this observation." In fact, item 2 on the blog's list of open questions is "a more complete theory explaining the ratio of LoRA to FullFT learning rates."

μA does not answer that question. The paper's Section 2 mentions Schulman's result only in passing, as a "heuristic rule (e.g., dividing the LoRA learning rate by 10 to get the FFT learning rate)." Instead, what μA does is a workaround — instead of explaining the ratio, it finds a setup where the ratio is 1.

The Workaround — Carrying a Learning Rate Straight from LoRA to FFT

Init[B] + α = 1 is that setup. The paper's logic goes like this: the required learning rate for this setup is η ∝ n^(-1), and the required learning rate for FFT of a linear layer is also η ∝ n^(-1). Since the width scaling matches, the optimal learning rates can align, and if they do, a learning rate tuned on LoRA can be used as-is on FFT.

The experiments largely support this prediction. Carrying over Section 5.2's own wording — for most models, the optimal learning rate for Init[B] (α=1) aligns closely with FFT's optimal learning rate. The exception is RoBERTa-large, where FFT preferred a slightly smaller learning rate than Init[B]. The paper attributes this to the added classification head — its learning rate was fixed at 10^(-3), but training it alongside the adapter can shift the optimum.

This is where the authors' proposed practical workflow comes from. In the RLVR experiment fine-tuning Llama-3.1-8B on GSM8k with GRPO, FFT used more than 2x the GPU memory of LoRA (r=256), with most of the excess going to gradient buffers and optimizer state (wall-clock time per step was similar between the two in the same experiment). So the pitch is: sweep the learning rate with Init[B] (α=1) on a relatively accessible mid-tier GPU, then carry that value over to FFT on bigger hardware, saving the cost of re-tuning.

Two things need to be flagged honestly here.

First, the paper's Section 6 limitations nail this down — this analysis establishes a necessary condition for learning-rate transfer, and finding and proving a sufficient condition is left as future work. Matching width scaling means "can align," not "aligns." The experiments show alignment, but experiments are experiments.

Second, Init[B] is not PEFT's default. Look at reset_lora_parameters in the v0.17.0 source, and the default path (init_lora_weights=True) puts kaiming_uniform_ on lora_A and zeros_ on lora_B — that's Init[A]. What's more, the paper states it used Kaiming normal on the non-zero factor, matching variance 1/n for A's components in Init[A] and variance 1/r for B's components in Init[B]. PEFT's default is Kaiming uniform. In short, reproducing this paper's setup takes more than flipping a few LoraConfig flags — you have to touch the initialization code directly.

What the Experiments Actually Measured, and What They Didn't

This is the most important part of this post.

What they ran. Five SFT setups — Llama-3.2-1B / Tulu-3 SFT mixture (320k train, 32k validation, max 1024 tokens), Qwen2.5-3B-Instruct / OpenThoughts-114k (71k / 6k, 8192-token packing), RoBERTa-large / ANLI (163k / 3k), ViT-Huge/14 / ImageNet-1K (1.28M / 50k), Qwen3-VL-2B-Instruct / LLaVA-Instruct-Mix (198k / 17k). On top of that, RLVR with Llama-3.1-8B / GSM8k (a DAPO variant of GRPO), and a diffusion model, Stable Diffusion v1.5 / Naruto-BLIP-Captions (1.2k pairs, 10,000 steps). Hardware was a 4x H200 node. Coverage is genuinely broad, sweeping language, vision, vision-language, image generation, and reinforcement learning.

But here's the statistics.

  • Paper Appendix B.1: "We set the random seed to 42 for all experiments." One seed. No confidence intervals, no error bars, no seed-to-seed variance reported.
  • The learning-rate grid is on a log2 scale with integer exponents only, meaning adjacent points differ by 2x. The reported "optimum" is the best value on this discrete grid.
  • Only 5 ranks per setup. Init[A] uses 4, 16, 64, 256, 1024. Init[B] uses 16, 32, 64, 128, 256 for SFT. Init[A] for RLVR and the diffusion model uses 1, 4, 16, 64, 256.
  • RLVR training draws 500 problems from GSM8k (50 for test).

The paper's own Section 5.1 writes this limitation down. The scaling rule is a leading-order dependence as (n, r) → ∞, and at finite width and rank, constant and lower-order terms can shift the optimum, with the effect larger at small rank. And "if the loss landscape is flat near the optimum, nearby grid points perform similarly, so training noise can determine which one looks best." So the authors declare they evaluate by qualitative trend, not exact numerical agreement — a setup predicted to be rank-independent passes if the optima cluster within one grid step (2x); a rank-dependent setup passes if the optima move monotonically with rank and span multiple grid points.

In other words, this paper's "rank-independent" means "within one step of a 2x-spaced grid, on a single seed." A useful claim, but not a precise one. And the paper doesn't hide this — not hiding it is this paper's virtue.

The paper also self-reports mismatches. For Qwen2.5-3B-Instruct's Init[A] (α=1), r=16 and r=64 hit minimum loss at the same learning rate (the prediction says they should differ). The paper blames grid discretization. And for Init[A] (α = r^(-1)), small rank (r=4) preferred a slightly smaller learning rate than large rank — the opposite direction from what was observed at α=1. The paper explains this as a finite-size effect. Worth noting that both explanations are post hoc.

And what this paper didn't measure. This is not a "LoRA does as well as FFT" paper. What's measured is where the optimal learning rate sits, not whether LoRA reaches FFT's final loss. It's about the location of the bottom of the U-shaped learning-rate-sweep curve, not a claim about the depth of that bottom. The one exception is Appendix D.1 for RLVR, which observes "peak performance is similar across ranks" and notes agreement with Schulman — but that's a side observation, not this paper's claim. Where LoRA falls short of FFT is still a question for separate evidence like LoRA Learns Less and Forgets Less (Biderman et al., TMLR 2024) — which reports that in programming and math domains, LoRA with standard low-rank settings falls substantially short of full fine-tuning, while preserving out-of-domain performance better.

One more thing. On decoder-only LMs, LoRA was attached only to the MLP (gate_proj, up_proj, down_proj). This isn't so much a flaw as alignment with current best practice — Schulman's blog reported that attention-only LoRA is markedly worse than MLP-only, and that adding attention on top of MLP brings no gain, and it notes Biderman et al. got the same result. But whether the scaling rule holds under other placement strategies is not something these experiments answer.

Where the Two Results Diverge on Width Scaling

This section is what I found by reading the two sources side by side, and I want to flag upfront that neither paper addresses the other at this specific point.

In μA coordinates, Schulman's setup is Init[A], α ∝ r^(-1). μA's Table 1 gives η ∝ n^(-1/2) for that cell. And μA's Theorem A.23 says FFT gives η ∝ n^(-1). Dividing the two yields the prediction that the LoRA/FFT learning-rate ratio is proportional to n^(1/2) — meaning the ratio should grow as the model gets wider.

But Schulman's fit result is the opposite. Across 14 models, the optimization found LoRA pow = 0 — meaning LoRA's hidden-size dependence is identical to FullFT's, with the multiplier fixed at 9.8.

These two don't line up as they stand. But there's a reason not to hastily say "one of them is wrong."

  • μA's theory is asymptotic. The paper's Section 6 limitations state directly that "our theory is asymptotic, and at finite (n, r) it may be sensitive to noise."
  • Doing the arithmetic, n^(1/2) means hidden size has to grow 4x for the ratio to change 2x. And 2x is exactly one step on a log2 grid. In other words, this predicted difference could be a size that neither study's measurement resolution can catch.
  • Schulman's fit assigns a separate exponent per Llama/Qwen family, so it's hard to call it a pure test of width scaling either.

The honest conclusion is this — no number that resolves this discrepancy has been published yet. μA doesn't explain the 10x rule, and Schulman's fit wasn't designed to test μA's width exponent. Someone needs to align the two coordinate systems and sweep a wide range of hidden sizes across multiple seeds to get an answer. As far as I know, that experiment doesn't exist.

So What Should You Actually Do

1. Check your α convention first. This is the cheapest, most reliable win in this post.

fixed lora_alpha (e.g. 32) + use_rslora=False   ->  effective α ∝ r^(-1)
   => keep the learning rate when you change rank. (what Schulman saw)

lora_alpha = r or 2r                             ->  effective α = constant
   => μA prediction: halve the learning rate every 4x in rank.

The paper checks in Appendix C.1 that the same scaling holds when α is 2 or 4 as well. In other words, the moment you follow the advice to "scale lora_alpha with rank," there's a learning-rate rule that's supposed to come with it — and that rule is usually missing from the advice.

2. If you want to find an FFT learning rate cheaply, Init[B] + α=1 is a candidate. But (a) it's only a necessary condition, (b) it's not PEFT's default, so you have to write the initialization code yourself, and (c) the evidence is a single seed. On models with a classification head, the paper itself reported a mismatch. Use it as a starting point, but verify it.

3. In RL, a learning-rate mistake costs you twice. This is what Section 5.3 of the paper observes — in SFT, too small a learning rate just slows convergence, but in RLVR, either too large or too small badly damages the training reward. And it costs you wall-clock time on top of that. In the Appendix D.2 case study, at the optimal learning rate (2^(-18)), the model exits the initial exploration phase (where completion length pins to the 1024-token cap) around step 100 and moves to concise ~200-token solutions, taking 4-5 seconds per step. At a suboptimal learning rate (2^(-16)), it instead gets stuck in the exploration phase, keeps producing max-length completions, and takes 15-22 seconds per step — 3-5x slower. The reward collapses toward zero. The authors also note a failure in the opposite direction: give Init[B], α=1 a higher learning rate (2^(-15)) and the model produces about 2 tokens and stops generating — which also speeds up the steps. The point being: faster steps aren't automatically good news.

When you don't need any of this. If you never change rank and your current recipe already works, this paper gives you nothing. If your model and task are fixed, a sweep you've already run beats the asymptotic n, r → ∞ rule. And every rule here is Θ(·) — it tells you the exponent, not the constant. Either way, running one baseline sweep is still on you. What μA cuts down is how many times you repeat that sweep, not the sweep itself.

Closing

To sum up: as of February 2026, the answer to "do you need to re-find the learning rate when you change LoRA rank" is "it depends on your α convention." With the classic 1/r convention, you don't need to change it; with the increasingly common practice of scaling lora_alpha with rank, you need to halve it every 4x in rank. And in the less-common setup of Init[B] + α = 1, the optimal learning rate has the same width scaling as full fine-tuning, so carrying a value tuned on LoRA over to FFT is possible in principle.

At the same time, you need to know the exact size of that evidence. A single seed (42), a 2x-spaced grid, five ranks per setup, asymptotic theory, and the authors' own caveat that it's "only a necessary condition." The authors themselves declared they evaluate by qualitative trend rather than exact numerical agreement, and they cleared that bar. You shouldn't read more than that into this paper.

And there's one thing nobody knows yet. Schulman's observation that "LoRA's learning rate is 10x FullFT's" remains unexplained. Instead of explaining that ratio, μA moved to a coordinate system where the ratio becomes 1. That's a clever enough workaround in practice, but Schulman's open question is still exactly that — open. LoRA has been a tool we've used every day since 2021, and even in mid-2026, why its learning rate has to be what it is still isn't fully explained.

References

Discuss on Twitter • View on GitHub