Parameter-Efficient Fine-Tuning

The Story That Explains Parameter-Efficient Fine-Tuning

A modern LLM like GPT-3 or LLaMA-3 has billions of parameters. Fine-tuning all of them on a task-specific dataset is called full fine-tuning, and it demands the same GPU memory and compute as pre-training. For most teams, that is simply impossible.

PEFT is a family of techniques that fine-tune only a tiny fraction of parameters — sometimes less than 0.1% of the model — while keeping the vast majority frozen. The result: near-full-fine-tuning performance at a fraction of the cost.

The Problem — Why Full Fine-Tuning Is Impractical

Before understanding PEFT, you need to feel the pain it solves. Here is what full fine-tuning actually requires:

LoRA — Low-Rank Adaptation

The Core Idea — Low-Rank Decomposition

LoRA, introduced by Hu et al. (2021), rests on a key insight: the weight updates during fine-tuning have a low intrinsic rank. This means the change matrix ΔW doesn't need to be full-sized — it can be expressed as the product of two much smaller matrices.

What Gets LoRA Applied To?

In practice, LoRA is applied to the query (Q) and value (V) projection matrices inside every transformer attention layer. Some implementations also target the key (K) and the feed-forward layers. The original paper found Q + V sufficient for most tasks.

LoRA Hyperparameters

Full LoRA Implementation with HuggingFace PEFT

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from datasets import load_dataset
from trl import SFTTrainer

# ── 1. Load base model (frozen) ───────────────────────────
model_id = "meta-llama/Llama-3.2-3B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token  # LLaMA has no pad token by default

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",       # auto-shard across available GPUs
    trust_remote_code=True
)

# ── 2. Define LoRA configuration ──────────────────────────
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,                        # rank: expressiveness of the low-rank update
    lora_alpha=32,               # scaling factor (alpha/r = 2.0 here)
    target_modules=[              # apply LoRA to these projection layers
        "q_proj", "k_proj",
        "v_proj", "o_proj"
    ],
    lora_dropout=0.05,           # light dropout for regularisation
    bias="none",                 # keep biases frozen
    use_rslora=False,            # rank-stabilised LoRA (set True for large r)
)

# ── 3. Wrap model with LoRA adapters ──────────────────────
model = get_peft_model(model, lora_config)

# Inspect how many parameters are actually trainable
model.print_trainable_parameters()

# ── 4. Load a sample instruction dataset ─────────────────
dataset = load_dataset("tatsu-lab/alpaca", split="train[:5000]")

def format_instruction(example):
    return {"text": f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"}

dataset = dataset.map(format_instruction)

# ── 5. Training arguments ─────────────────────────────────
training_args = TrainingArguments(
    output_dir="./lora-llama3-alpaca",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,   # effective batch = 16
    warmup_ratio=0.03,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=50,
    save_strategy="epoch",
    optim="adamw_8bit",              # bitsandbytes 8-bit optimiser saves ~50% memory
    report_to="none",
)

# ── 6. Supervised Fine-Tuning Trainer ────────────────────
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=512,
)

trainer.train()

# ── 7. Save only the LoRA adapter weights (~30 MB) ───────
model.save_pretrained("./lora-adapter")
tokenizer.save_pretrained("./lora-adapter")

Merging LoRA Back Into the Base Model (for deployment)

from peft import PeftModel

# Load base model + LoRA adapter, then merge into a single model
base_model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.float16, device_map="auto"
)
model_with_lora = PeftModel.from_pretrained(base_model, "./lora-adapter")

# Merge: absorbs B·A into W₀ permanently — no inference overhead
merged_model = model_with_lora.merge_and_unload()

# The merged model is identical to a fully fine-tuned model at inference
merged_model.save_pretrained("./merged-model")

# Inference
inputs = tokenizer(
    "### Instruction:\nExplain gradient descent in simple terms\n\n### Response:\n",
    return_tensors="pt"
).to("cuda")

with torch.no_grad():
    output = merged_model.generate(**inputs, max_new_tokens=200, temperature=0.7)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Prefix Tuning

How Prefix Tuning Works

Introduced by Li and Liang (2021), Prefix Tuning prepends trainable continuous vectors (called the prefix) to the key (K) and value (V) matrices of every transformer attention layer. These are not real tokens from the vocabulary — they are free-floating vectors optimised end-to-end.

Prefix Tuning vs Prompt Tuning

Prefix Tuning Implementation

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PrefixTuningConfig, get_peft_model, TaskType
import torch

# ── 1. Load a seq2seq model (T5-base for summarisation) ───
model_id = "t5-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

# ── 2. Prefix Tuning config ───────────────────────────────
prefix_config = PrefixTuningConfig(
    task_type=TaskType.SEQ_2_SEQ_LM,
    num_virtual_tokens=30,          # 30 virtual prefix tokens per layer
    encoder_hidden_size=768,          # match T5-base hidden dim
    prefix_projection=True,          # MLP re-parameterisation for stability
)

# ── 3. Apply prefix tuning ────────────────────────────────
model = get_peft_model(model, prefix_config)
model.print_trainable_parameters()

# ── 4. Forward pass example ───────────────────────────────
# Prefix tokens are automatically prepended inside the PEFT wrapper
input_text = "summarize: The rapid advancement of large language models has transformed..."
inputs = tokenizer(input_text, return_tensors="pt", max_length=512, truncation=True)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=60, num_beams=4)
    summary = tokenizer.decode(outputs[0], skip_special_tokens=True)
    print(f"Summary: {summary}")

# ── 5. Training loop (simplified) ────────────────────────
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments
from datasets import load_dataset

dataset = load_dataset("cnn_dailymail", "3.0.0", split="train[:2000]")

training_args = Seq2SeqTrainingArguments(
    output_dir="./prefix-t5-cnn",
    num_train_epochs=5,
    per_device_train_batch_size=8,
    predict_with_generate=True,
    learning_rate=1e-3,           # prefix tuning uses a HIGHER lr than LoRA
    warmup_steps=100,
    fp16=True,
    save_strategy="epoch",
)

Adapters — Bottleneck Layers Between Frozen Weights

Adapter Architecture — The Bottleneck Module

Houlsby et al. (2019) introduced the original Adapter architecture. Each adapter is a down-project → non-linearity → up-project bottleneck, with a residual connection:

Adapter Implementation with PEFT

from transformers import AutoModelForSequenceClassification, AutoTokenizer, TrainingArguments, Trainer
from peft import AdaptionPromptConfig, get_peft_model, TaskType
from peft.tuners.adaption_prompt import AdaptionPromptConfig
from datasets import load_dataset
import numpy as np

# ── Alternatively: use AdapterHub / adapter-transformers ──
# pip install adapter-transformers  (separate library with more adapter types)
import transformers.adapters as adapters
from transformers import RobertaAdapterModel, RobertaTokenizer

# ── Load model with adapter support ──────────────────────
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
model = RobertaAdapterModel.from_pretrained("roberta-base")

# ── Add a Houlsby-style bottleneck adapter ────────────────
# reduction_factor=16 means bottleneck dim = hidden/16 = 768/16 = 48
model.add_adapter("sentiment-adapter", config="houlsby")

# Add classification head on top
model.add_classification_head("sentiment-adapter", num_labels=2)

# Activate the adapter — freezes base model, trains adapter only
model.train_adapter("sentiment-adapter")

# Inspect trainable parameters
total = sum(p.numel() for p in model.parameters())
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Trainable: {trainable:,} / {total:,} ({trainable/total*100:.2f}%)")

# ── Dataset preparation ───────────────────────────────────
dataset = load_dataset("sst2", split="train[:3000]")

def tokenize(batch):
    return tokenizer(batch["sentence"], truncation=True, padding="max_length", max_length=128)

dataset = dataset.map(tokenize, batched=True)
dataset = dataset.rename_column("label", "labels")
dataset.set_format("torch", columns=["input_ids", "attention_mask", "labels"])

# ── Training ──────────────────────────────────────────────
training_args = TrainingArguments(
    output_dir="./adapter-roberta-sst2",
    num_train_epochs=5,
    per_device_train_batch_size=32,
    learning_rate=1e-4,           # adapters use higher lr than LoRA
    fp16=True,
    logging_steps=50,
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
)

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return {"accuracy": (predictions == labels).mean()}

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    compute_metrics=compute_metrics,
)
trainer.train()

# ── Save only the adapter (very small checkpoint) ─────────
model.save_adapter("./saved-adapter", "sentiment-adapter")
print("Adapter saved!")

# ── Load later: combine any adapter with the base model ───
model2 = RobertaAdapterModel.from_pretrained("roberta-base")
model2.load_adapter("./saved-adapter", load_as="sentiment-adapter")
model2.set_active_adapters("sentiment-adapter")

Head-to-Head — LoRA vs Prefix Tuning vs Adapters

QLoRA — Quantised LoRA (The Game-Changer)

QLoRA (Dettmers et al. 2023) stacks three innovations on top of LoRA:

QLoRA Implementation

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, TaskType

# ── 1. BitsAndBytes 4-bit quantisation config ─────────────
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,              # load base model in 4-bit
    bnb_4bit_use_double_quant=True,  # double-quantise the quant constants
    bnb_4bit_quant_type="nf4",       # NormalFloat-4 (optimal for Gaussian weights)
    bnb_4bit_compute_dtype=torch.bfloat16,  # dequantise to bfloat16 for compute
)

# ── 2. Load 7B model in 4-bit (only ~5 GB VRAM!) ─────────
model_id = "meta-llama/Llama-3.1-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto"
)

# ── 3. Prepare model — cast LayerNorm to fp32 etc. ────────
model = prepare_model_for_kbit_training(model)

# ── 4. Apply LoRA on top of the 4-bit frozen base ─────────
lora_config = LoraConfig(
    r=64,                            # higher r than usual, base is compressed
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],  # include FFN for qlora
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# Memory summary
if torch.cuda.is_available():
    allocated = torch.cuda.memory_allocated() / 1e9
    print(f"GPU memory used: {allocated:.2f} GB")

Decision Guide — Which PEFT Method for Your Task?

Golden Rules — Non-Negotiables for PEFT in Practice

Quick Reference — Install, Import, Run

Environment Setup

# Install core PEFT dependencies
pip install transformers peft datasets trl accelerate
pip install bitsandbytes  # for QLoRA (NF4 quantisation + 8-bit Adam)
pip install adapter-transformers  # for Houlsby-style adapters (optional)
pip install sentencepiece protobuf  # needed by some tokenisers (LLaMA, T5)

Minimal LoRA Cheatsheet

from peft import LoraConfig, get_peft_model, TaskType, PeftModel

# Standard config — works for 90% of instruction tuning tasks
cfg = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
    bias="none"
)
model = get_peft_model(model, cfg)    # wraps model; freezes base
model.print_trainable_parameters()     # always sanity-check this
model.save_pretrained("./lora-out")   # saves adapter only (~10-50 MB)

# Load and merge for deployment
base = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)
peft = PeftModel.from_pretrained(base, "./lora-out")
merged = peft.merge_and_unload()      # now identical to full FT model