LoRA / PEFT: Low-Rank Fine-Tuning

Full fine-tuning updates every weight in the model. That means storing a gradient for every weight, optimizer state for every weight, and a full-size checkpoint for every task — plus a separate deployment for every adaptation.

The memory arithmetic is large. A 16-bit weight takes 2 bytes. Its gradient takes 2 more. The two standard Adam optimizer tables take 8 (4 bytes each). That is about 12 bytes per parameter, so a 7-billion-parameter model needs on the order of 84 gigabytes before activations — the intermediate values produced during a forward pass — are even counted.

LoRA starts from an observation: most of the useful change is low-rank. Adapting a pretrained model to one task does not need to move every direction independently. It needs to move a few directions a lot.

The method does exactly that. Freeze the original weight matrix W and learn only a correction, ΔW = B·A. Here A is a small matrix with r rows, B is a small matrix with r columns, and r — the rank — is tiny, say 16. The layer computes h = Wx + (α/r)·BAx, where x is the layer's input, h is its output, and α (alpha) is a scaling number that keeps the correction's size steady when you change r.

B starts at zero, so the model begins as the pretrained model, and A starts with small random numbers so the first learning signal is informative. The number of trained parameters drops from d_in·d_out to r(d_in + d_out), where d_in and d_out are the input and output widths of the layer: the product becomes a sum, times the small rank.

With width 4096 and rank 16, that is 131,000 parameters instead of 16.8 million — about 0.8 percent. And the frozen base needs no optimizer state. Training a 7-billion-parameter model on one 24-gigabyte card becomes routine rather than heroic.

QLoRA pushes the frozen base even smaller. Store it in a 4-bit format, unpack each value on the fly during the forward pass, and keep the adapter — the learned pair A and B — in 16-bit. Paged optimizers absorb memory spikes, and the base costs half a byte per parameter.

The 4-bit rounding is fixed, because the frozen base never changes; the adapters learn around it. A smarter initialization called LoftQ starts the factor pair from a quantization-aware decomposition instead of zeros and recovers a little of the gap.

Which layers you adapt matters more than the rank. Early LoRA attached only to the query and value projections in attention. Current guidance is every linear layer — the attention projections and the feed-forward network — because that is where the useful change lives.

The canvas shows the target update ΔW, its best rank-r approximation B·A, and the two merged together. Raise the rank and the error collapses: most of the change's energy sits in the first component, and B·A is the provably best rank-r approximation, so no other rank-r pair can do better.

The calculator applies the same arithmetic to real model shapes. At width 4096 and rank 16, the adapter is a rounding error next to the dense matrix, and the QLoRA toggle shows where a 7-billion-parameter base actually fits. Flip Merge adapter and watch W + BA fold into one matrix: after merging there is nothing extra to run, no additional kernel, no added latency.

Serving is where LoRA pays off twice. A merged adapter adds zero inference overhead. An unmerged one can be swapped per request, so a single base model serves many tasks from a library of small adapters.

The memory economics are the same as prefix caching, where the repeated opening of a prompt is computed once and reused. In agent and multi-tenant workloads, prompt openings repeat constantly, and cache hit rates of 60 to 85 percent mean most of the prompt is never recomputed. Adapters change which weights answer the request; prefix caches avoid re-reading the context that leads up to it.

A rank of 8 to 16, with alpha around twice the rank and all linear layers as targets, is the 2026 default. Bigger ranks help less than a better target set, and they cost checkpoint size and overfitting risk.

The failure modes are ordinary training failures with a low-rank accent. Too much rank on too little data overfits. Too few target layers plateau at a mediocre loss. Merging into a quantized base without unpacking it first corrupts the weights. Adapters fitted on drifting data go stale like any model.

One honest rule: LoRA is not a free replacement for full fine-tuning when the task needs a genuinely new capability. It steers what is already there. Prompt tuning is the even lighter cousin — a handful of learned vectors and no weight changes at all — and it is the right tool when the shift is style or format rather than knowledge.

The problems below count LoRA and adapter parameters, size a QLoRA run, merge the update back into W, and compute a prefix cache hit ratio.