When to open: You want to build a domain model / fine-tune a chatbot / do RLHF / choose between LoRA and full-parameter fine-tuning / choose between PPO and DPO. This wiki organizes the 19 ai-interview PDFs into a practical roadmap.
One-sentence core:Choose full-parameter vs PEFT based on VRAM + LoRA is mainstream + QLoRA enables 65B on a single GPU + PPO is hard to tune, DPO is simple → mainstream shifted to DPO after 2024 + Use Base for domain fine-tuning / use Chat for strong instruction following.
0. Perspective note
Main source:LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning 19 PDFs.
1. LLM Fine-tuning Basics (8 high-frequency questions)
LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning §1:
| Q | Key points |
|---|---|
| How much VRAM does full-parameter fine-tuning require? | Model fp16 + gradients fp16 + Adam states (m,v) fp32 + activations — approximately 16–20x model size |
| Why does the LLM seem dumber after SFT? | Catastrophic forgetting / SFT overfitting to short patterns / RL stage not yet applied |
| How to construct SFT instruction fine-tuning data? | Diverse instructions + high-quality answers + appropriate length + refusal samples |
| Data selection for domain Continue PreTrain? | 70% domain + 30% general to prevent forgetting |
| How to mitigate degradation of general capability after domain training? | LoRA incremental training + data mixing + progressive fine-tuning |
| How to learn more during Continue PreTrain? | High-quality data + sufficient training + mixing of different tasks |
| Use Chat or Base model for SFT? | Use Base for domain fine-tuning / use Chat for strong instruction-following capability |
| Input format for domain fine-tuning data? | Alpaca / ShareGPT / Vicuna instruction formats |
2. PEFT (Parameter-Efficient Fine-Tuning)
2.1 Full-parameter vs PEFT
| Method | Advantages | Disadvantages |
|---|---|---|
| Fine-tune (full-parameter) | Good performance | Slow + expensive + catastrophic forgetting |
| PEFT | Fast / memory-efficient / less catastrophic forgetting | Slight performance loss |
2.2 Comparison of PEFT methods
| Method | Idea |
|---|---|
| Prefix-Tuning | Add trainable prefixes at each layer |
| P-Tuning | Add trainable prompts at the input end |
| Prompt-Tuning | Simplified P-Tuning |
| LoRA | Low-rank matrix factorization |
| AdaLoRA | Adaptive LoRA rank |
| Adapter | Insert small modules |
| IA³ | Three vectors multiplied with K/V/FFN |
Experience:
- Lianjia BELLE report: FT slightly outperforms LoRA
- But LoRA has huge advantages in speed and VRAM → mainstream choice
3. LoRA Family (mainstream)
3.1 Core idea of LoRA
Full-parameter update → low-rank matrix factorization:
W' = W + ΔW
ΔW = A * B (A: r×d, B: d×r, r << d)
Only train two small matrices A and B (parameters << full-parameter fine-tuning).
3.2 LoRA characteristics
| Feature | Description |
|---|---|
| Training | Only update A, B (0.1–1% of full parameters) |
| Inference | A*B can be merged into W,zero additional inference overhead |
| Multi-task | Switch between multiple LoRA sets (swappable adapters) |
| rank | Typically r=8/16/32; the more complex the task, the larger r |
3.3 LoRA application layers
Usually applied to:
q_proj/k_proj/v_proj/o_proj(attention)gate_proj/up_proj/down_proj(FFN, often added in newer versions)
Experience: applying to FFN usually yields better results than applying only to attention.
4. QLoRA (Fine-tune 65B on a single 24GB GPU)
4.1 Core idea
Q = Quantization.
- Quantize the base model to 4-bit (NF4 format)
- Perform LoRA on the quantized model
- Fine-tune 65B on a single 24GB GPU
4.2 Key technologies
| Technology | Function |
|---|---|
| NF4(4-bit NormalFloat) | Better accuracy than fp4 (optimized for normal distribution) |
| Double Quantization | Re-quantize quantization constants themselves → saves VRAM |
| Paged Optimizer | Swap to CPU when VRAM is insufficient |
4.3 QLoRA training VRAM estimation
| Model | Full-parameter FT | LoRA | QLoRA |
|---|---|---|---|
| 7B | 100GB+ | 24GB | 8GB |
| 13B | 200GB+ | 40GB | 16GB |
| 65B | 1TB+ | 200GB | 24GB |
5. Continued Pretraining
5.1 When to do it?
- Domain knowledge must be injected(medical / legal / finance)
- New languages —— languages not supported by the main model
- New vocabulary —— large number of domain-specific terms
5.2 Data ratio
领域数据 70% + 通用数据 30% — 防灾难遗忘
5.3 Continued pretraining sample concatenation
LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning §file_33:
- Packing —— concatenate multiple documents into the same sequence (saves padding)
- Document Boundary Token ——
<eos>separated - Attention Mask —— documents do not see each other (prevents attention cross-talk)
5.4 LoRA secondary pretraining
LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning §file_34:
- Using LoRA for Continue Pretrain → VRAM-friendly + easy to roll back
- Key:rank must be large enough(r=64+), otherwise domain knowledge cannot be learned
6. RLHF / DPO / PPO Roadmap
6.1 Overall RLHF pipeline (classic three stages)
1. SFT(Supervised Fine-tuning)
↓
2. RM(Reward Model 训练)
- 人类对回答打分 / 排序
- 训练奖励模型
↓
3. RL 训练
- PPO 或变种
- LLM 策略 vs Reward Model
6.2 RLHF variants
LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning §file_37:
| Method | Characteristics |
|---|---|
| PPO | Standard RLHF / hard to tune / large VRAM / 4 models: Reward Model + Critic + Actor + Reference |
| DPO(Direct Preference Optimization) | No Reward Model / train directly with preference pairs / mainstream in 2024 |
| IPO(Identity Preference Optimization) | DPO improvement / reduces overfitting |
| KTO(Kahneman-Tversky Optimization) | Uses single-sample feedback / does not require pairwise preferences |
| GRPO(Group Relative Policy Optimization) | Proposed by DeepSeek / used by R1 / reduces Critic model |
| ORPO(Odds Ratio Preference) | Train SFT + preference merging in one step |
| SimPO | DPO improvement / no reference model needed |
6.3 PPO difficulties
LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning §file_38:
| Pain point | Mitigation |
|---|---|
| VRAM explosion from 4 models(Actor + Critic + Reward + Reference) | LoRA / ZeRO-3 / parameter sharing |
| Hyperparameter-sensitive(KL coefficient / clip range) | Multiple experiments + early stopping |
| Reward Hacking | Multi Reward Model ensemble |
| Training instability | gradient clipping + warmup |
6.4 Why did DPO become mainstream later?
- No Reward Model needed → training pipeline cut in half
- No RL needed → direct supervised learning
- Performance is not worse → close to PPO on most tasks
But DPO also has problems:
- Prone to overfitting to preference data
- Lacks online exploration
- → IPO / SimPO / GRPO continue to improve
7. Training Set / SFT Data Construction
7.1 SFT data sources
LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning §file_40:
| Source | Characteristics |
|---|---|
| Human annotation | High quality / expensive |
| LLM distillation (GPT-4 / Claude) | Fast / cheap / has License risk |
| Self-Instruct | LLM self-generation → train on itself |
| Evol-Instruct(WizardLM) | Use LLM to turn simple instructions into complex ones |
| OASST / OpenAssistant | Open-source human annotation |
7.2 LLM-generated SFT data
LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning §file_41:
Key techniques:
- Diversify seed instructions(domain / difficulty / style)
- Post-processing filtering(deduplication / refusal / safety filter)
- Quality grading(EasyData / DataGPT scoring)
- Mix human + synthetic(pure synthetic data is prone to overfitting)
7.3 Data scale experience
| Stage | Recommended data volume |
|---|---|
| SFT | 10K–100K high-quality instructions |
| Continue Pretrain | 1–10 billion tokens |
| RLHF/DPO | 10K–100K preference pairs |
8. VRAM Optimization Strategies
8.1 VRAM usage estimation
LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning §file_42:
显存 ≈ 模型参数 + 梯度 + 优化器状态 + 激活 + 缓存
| Item | fp16 |
|---|---|
| Parameters | 2 bytes/param |
| Gradients | 2 bytes/param |
| Adam m+v | 8 bytes/param(fp32) |
| Activations | Depends on batch / seq_len |
Rough estimate for full-parameter fine-tuning of a 7B model:7B × (2+2+8) = 84 GB + activations → single A100 80G is not enough → distributed training or PEFT is required.
8.2 8 major VRAM optimization techniques
LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning §file_43:
- Gradient Checkpointing —— recompute activations to trade VRAM (2x time for ~50% VRAM)
- Gradient Accumulation —— simulate large batch with small batch
- Mixed Precision(AMP) —— fp16/bf16 training
- Optimizer Offloading —— swap Adam states to CPU (DeepSpeed)
- ZeRO 1/2/3 —— distribute optimizer / gradients / parameters across devices
- LoRA / QLoRA —— PEFT
- Flash Attention —— do not store attention matrix
- Sequence Parallelism —— split along seq dimension
9. Evaluation
LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning §file_35:
9.1 Automatic metrics
| Metric | Use case |
|---|---|
| BLEU / ROUGE / METEOR | Translation / summarization (outdated) |
| Perplexity | Language modeling (pretraining) |
| Exact Match / F1 | QA |
| Pass@k | Code (HumanEval / MBPP) |
9.2 LLM-as-Judge
- GPT-4 as judge —— mainstream
- Arena mode —— Pairwise voting
- Chatbot Arena ELO —— LMSys public leaderboard
9.3 Chinese benchmarks
| Name | Coverage |
|---|---|
| C-Eval | Chinese academic subjects |
| CMMLU | Comprehensive Chinese |
| GSM8K-zh | Chinese mathematics |
10. Inference Optimization
LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning §file_31:
See LLM Basics (Transformer / Attention / Three Types of PLMs) (Cross-Author Topic Synthesis) §9.3:
| Technology | Effect |
|---|---|
| KV Cache | Baseline |
| vLLM PagedAttention | 2-3x |
| Quantization(GPTQ / AWQ / INT4) | 1.5–2x speed / 4x VRAM savings |
| Speculative Decoding | 2-3x |
| Continuous Batching | 2-5x throughput |
11. How to Use — Fine-tuning Path by Role
| Who you are | Primary method | Starting point |
|---|---|---|
| Want to build a domain chatbot | SFT + LoRA + DPO | Base model + 10k high-quality instructions + preference pairs |
| Fine-tune 65B on a single GPU | QLoRA | NF4 + Double Quant + Paged Optimizer |
| Need to inject domain knowledge | Continue Pretrain | 70% domain + 30% general + LoRA secondary pretraining |
| Want to do RLHF | Prioritize DPO; use PPO if you have budget | SFT → preference pairs → DPO |
| Training VRAM overflows | 8 major VRAM techniques | Gradient Checkpointing + LoRA + ZeRO |
| Need fast evaluation | LLM-as-Judge | GPT-4 judge + Chatbot Arena |
General rules:
- Use Base model for domain fine-tuning(already aligned Chat will conflict)
- 70% domain + 30% general(prevents catastrophic forgetting)
- LoRA is mainstream / QLoRA is the single-GPU killer app
- PPO is hard to tune → DPO is mainstream (2024+)
- Do not use pure synthetic data for SFT(prone to overfitting)
- If VRAM overflows, first apply Gradient Checkpointing + LoRA
12. Reference List
- LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning —— 19 PDFs / fine-tuning + PEFT + LoRA family + QLoRA + continued pretraining + RLHF variants + PPO + VRAM optimization + inference + Prompting + SFT data + training sets
- newtype · AI Coding Practice and Tool Stack Evolution —— newtype fine-tuning practice (partial)
- LLM Basics (Transformer / Attention / Three Types of PLMs) (Cross-Author Topic Synthesis) —— basic architecture (read together)
- LLM Distributed Training / GPU Clusters / Inference Optimization (Cross-Author Topic Synthesis) —— distributed training (read together)
Sources and related materials
- LLM Interview / Complete Special Topic on Fine-tuning and Reinforcement Learning
- newtype · AI Coding Practice and Tool Stack Evolution
- LLM Basics (Transformer / Attention / Three Types of PLMs) (Cross-Author Topic Synthesis)
- LLM Distributed Training / GPU Clusters / Inference Optimization (Cross-Author Topic Synthesis)
- RAG / Knowledge Base Engineering (Cross-Author Topic Synthesis)