QLoRA

NF4 is an information-theoretically optimal code for zero-centred normal data: its sixteen levels are the quantiles of a standard normal, so each level receives equal probability mass under the assumption that pretrained weights are approximately Gaussian per block.

Double quantisation then quantises the per-block scale factors themselves, which at block size 64 costs 0.5 bits per parameter before compression and 0.127 after. Paged optimiser state moves Adam moments to host memory on gradient-checkpointing spikes rather than failing the allocation.

The base weights never leave 4 bits in memory; they are dequantised into bf16 tile by tile inside the matmul. This is why QLoRA is compute-bound where LoRA is not — you pay a dequantisation per forward and per recompute.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B", quantization_config=bnb
)
model.gradient_checkpointing_enable()

model = get_peft_model(model, LoraConfig(
    r=64, lora_alpha=16, lora_dropout=0.05,
    target_modules="all-linear",   # every projection, not just q/v
))

Keep compute_dtype at bf16; fp16 reintroduces the overflow problems that the NF4 block scales were meant to remove.