PEFT and TRL
Two Hugging Face libraries covering the two halves of post-training: which parameters you are allowed to move, and what objective moves them.
Standing
Current standardPart of the default recipe. Present in most frontier models trained today, and the thing a new design departs from rather than argues for.
The reference implementations of LoRA and of DPO. Faster wrappers exist and almost all of them are built on these two.
judged as of 2026-09 · what the labels mean
Theory
PEFT implements the parameter-efficient methods — LoRA, QLoRA, DoRA, prefix tuning, IA³ — as a wrapper that freezes a model and attaches trainable adapters.
TRL implements the post-training objectives — SFT, DPO, PPO, GRPO, reward modelling — as trainers that take a model and a dataset.
They are designed to compose, and the composition is the normal way to do alignment on hardware that is not a cluster.
from peft import LoraConfig
from trl import DPOConfig, DPOTrainer
peft_config = LoraConfig(
r=16,
lora_alpha=32, # scaling is alpha/r; 2r is the usual choice
target_modules=[ # all seven, not just q and v
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
lora_dropout=0.05,
task_type="CAUSAL_LM",
)
trainer = DPOTrainer(
model=model, # ref_model=None: PEFT gives the reference
args=DPOConfig(beta=0.1, learning_rate=5e-6, output_dir="out"),
train_dataset=pairs, # prompt / chosen / rejected
peft_config=peft_config,
)
trainer.train()The trick in that snippet
ref_model=None. DPO needs a frozen
reference policy, which normally means a second copy of the model in memory. With
LoRA the base weights are the reference — disable the adapters and you have it
— so TRL computes reference log-probabilities by toggling them off. One model in
memory instead of two, which for a 70B is the difference between fitting and not.
Choosing an objective
| Have | Use | Cost |
|---|---|---|
| demonstrations | SFTTrainer | one model |
| preference pairs | DPOTrainer | one model with LoRA |
| a reward model, online | PPOTrainer | four models |
| a verifiable reward | GRPOTrainer | two models, k samples |
The cost column is the practical determinant. PPO needs policy, reference, reward and value models resident at once, which is why DPO displaced it for most purposes despite being the less general method.
What they are not
Optimised. Both are reference implementations, written for clarity and coverage. Unsloth is roughly twice as fast at the same task with less memory, and config-driven wrappers remove the Python. Both of those are built on PEFT and TRL, so the concepts transfer unchanged.