QLoRA
Quantise the frozen base to 4 bits, keep the adapters in bf16, and backpropagate through the dequantisation. Three tricks make the accuracy loss negligible.
Standing
Current standardPart of the default recipe. Present in most frontier models trained today, and the thing a new design departs from rather than argues for.
How fine-tuning happens on one GPU. The quantised base plus low-rank update combination is now routine.
judged as of 2026-09 · what the labels mean
Theory
NF4 is an information-theoretically optimal code for zero-centred normal data: its sixteen levels are the quantiles of a standard normal, so each level receives equal probability mass under the assumption that pretrained weights are approximately Gaussian per block.
Double quantisation then quantises the per-block scale factors themselves, which at block size 64 costs 0.5 bits per parameter before compression and 0.127 after. Paged optimiser state moves Adam moments to host memory on gradient-checkpointing spikes rather than failing the allocation.
The base weights never leave 4 bits in memory; they are dequantised into bf16 tile by tile inside the matmul. This is why QLoRA is compute-bound where LoRA is not — you pay a dequantisation per forward and per recompute.
Implementation
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B", quantization_config=bnb
)
model.gradient_checkpointing_enable()
model = get_peft_model(model, LoraConfig(
r=64, lora_alpha=16, lora_dropout=0.05,
target_modules="all-linear", # every projection, not just q/v
))Keep compute_dtype at bf16; fp16 reintroduces the overflow problems that the NF4
block scales were meant to remove.