Fine-tuning / RLHF / inference optimization section in LLM interviews. AiGC Interview Bible (Ningjing Zhiyuan). 19 PDFs in total.
1. LLM Fine-Tuning Basics (file_9)
8 high-frequency fine-tuning questions
| Q | Key points |
|---|---|
| How much GPU memory does full-parameter fine-tuning require? | Model parameters fp16 + gradients fp16 + Adam states (m, v) fp32 + activations — approximately 16–20x the model size |
| Why does the LLM seem to become less capable after SFT? | Catastrophic forgetting / SFT overfitting to short patterns / the RL stage has not been applied yet |
| How to construct SFT instruction fine-tuning data? | Diverse instructions + high-quality answers + appropriate length + refusal samples |
| How to select data for domain model Continue PreTrain? | 70% domain data + 30% general data to prevent forgetting |
| How to mitigate the decline in general capability after domain training? | LoRA continued pre-training + data mixing + progressive fine-tuning |
| How to learn more during Continue PreTrain? | High-quality data + sufficient training + mixing of different tasks |
| Should SFT use a Chat model or a Base model? | Use Base for domain fine-tuning / use Chat for strong instruction-following capability |
| Input format for domain fine-tuning data? | Instruction formats such as Alpaca / ShareGPT / Vicuna |
2. PEFT Parameter-Efficient Fine-Tuning (file_26)
Full fine-tuning vs PEFT
| Method | Advantages | Disadvantages |
|---|---|---|
| Fine-tune (full parameter) | Good performance | Slow + expensive + catastrophic forgetting |
| PEFT | Fast / memory-efficient / minimal catastrophic forgetting | Slight performance loss |
Comparison of PEFT methods
| Method | Idea |
|---|---|
| Prefix-Tuning | Add trainable prefixes at each layer |
| P-Tuning | Add trainable prompts at the input end |
| Prompt-Tuning | Simplified version of P-Tuning |
| LoRA | Low-rank matrix factorization |
| AdaLoRA | Adaptive LoRA rank |
| Adapter | Insert small modules |
| IA³ | Three vectors multiplied by K/V/FFN |
Experience:
- Lianjia BELLE report: FT is slightly better than LoRA
- But LoRA has huge advantages in speed and GPU memory
3. LoRA Series (file_29 + file_30)
Core idea of LoRA
Full-parameter update → decompose matrix into low-rank matrix updates:
W' = W + ΔW
ΔW = A * B (A: r×d, B: d×r, r << d)
Only train two small matrices A and B (parameters << full-parameter fine-tuning).
LoRA characteristics
- Training:Only update A, B(0.1–1% of full parameters)
- Inference:A*B can be merged into W,Zero additional inference overhead
- Multi-task:Switch between multiple sets of LoRA(swappable adapters)
QLoRA
Q = Quantization.
- Quantize the base model to 4-bit (NF4 format)
- Apply LoRA on the quantized model
- Can fine-tune a 65B model on a single 24GB GPU
Key technologies:
- NF4 (4-bit NormalFloat) — better accuracy than fp16
- Double Quantization — requantize quantization constants
- Paged Optimizer — page Adam states to CPU
How to use LoRA in the PEFT library (file_30)
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=8, # 秩
lora_alpha=32, # 缩放
target_modules=["q_proj","v_proj"], # 应用在哪
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
model = get_peft_model(base_model, config)
Main components of GPU memory usage:
- Model weights (fp16: 2x)
- Gradients (fp16: 2x)
- Adam states (fp32 m, v: 8x)
- Activations (variable)
- Input batch
Optimization strategies:
- Quantize base model (4-bit) → -75% GPU memory
- LoRA → greatly reduces gradient + Adam memory usage
- Gradient Accumulation
- Gradient Checkpointing (trade time for memory)
4. Adapter-tuning (file_27)
Adapter idea
Insert one at each Transformer layer small bottleneck structure:
down-project (高维 → 低维) → 非线性 → up-project (低维 → 高维)
Adapter advantages
- Extremely few parameters(0.5-5%)
- Fast training + memory-efficient
- Multi-task switching Simple (one set of adapters per task)
Adapter vs LoRA
| Dimension | Adapter | LoRA |
|---|---|---|
| Inference overhead | Yes (additional layers) | No (can be merged) |
| Flexibility | General-purpose | Mainly used for attention |
| Performance | Close to LoRA | Slightly better |
5. Prompting / Prompt Learning (file_28)
Why prompt learning is needed
- Traditional fine-tune requires large amounts of labeled data
- Prompting uses natural language descriptions of tasks + few-shot examples,enabling zero-shot / few-shot use of LLMs
Prompting categories
| Type | Example |
|---|---|
| Zero-shot | "Please translate the following English into Chinese:..." |
| Few-shot | Provide 1–5 examples |
| Chain-of-Thought (CoT) | "Let's think step by step..." |
| Self-Consistency | Sample multiple times and take the majority |
| Tree of Thoughts | Tree of thoughts |
| ReAct | Reasoning + Acting |
Prompting advantages
- No training required
- Rapid trial and error
- Preserves the model's general capabilities
6. Continued Pre-Training (file_32 + file_33 + file_34)
Why continued pre-training?
Full pre-training is expensive: Training a 7B–65B model from scratch costs millions of dollars.
Advantages of continued pre-training:
- Continue training based on an existing model
- Add domain knowledge
- Adapt to new languages / new modalities
Preparation for continued pre-training
- Data preparation (domain + partial general data)
- Whether to expand tokenizer vocabulary (for new languages)
- Training framework (DeepSpeed / FSDP)
- Learning rate (usually 10x smaller than initial pre-training)
- Checkpoint strategy
Continued pre-training process
1. 加载基座模型
2. 准备增量数据(可分 phase:领域 → 指令 → 偏好)
3. 学习率 warmup(短)+ 余弦衰减
4. 训练 + 定期 eval
5. 检查指标(loss / 通用能力)
6. 选最佳 checkpoint
Data volume experience
- Continued pre-training:10B-100B token order of magnitude (domain)
- Full pre-training:1T-15T token
Pretrain sample concatenation (file_33)
Why concatenation is needed? — individual samples are short, leading to low GPU utilization.
4 concatenation methods:
| Method | Idea |
|---|---|
| Random Concatenate | Randomly concatenate to max_length |
| Random + NoiseMask | Concatenate + add noise at boundaries |
| Random + Cluster | Cluster by topic then concatenate |
| IN-CONTEXT PRETRAINING | Keep long documents, concatenate short documents from the same source |
LoRA secondary pre-training for llama2 (file_34)
Goal: Add Chinese language capability to LLaMA 2 using LoRA.
Idea:
- Vocabulary expansion (add Chinese tokens)
- LoRA training
- Data: Chinese encyclopedia + Chinese books + Chinese web novels
Parameters:
- r=64 / alpha=128 (a large rank is suitable for continued pre-training)
- target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"]
- Learning rate 1e-4 → 1e-5
7. Reinforcement Learning and RLHF
file_36 Reinforcement Learning Interview Questions
Brief introduction to RL = An agent learns the optimal policy by interacting with the environment.
Brief introduction to RLHF = Reinforcement Learning from Human Feedback, train a reward model using human feedback, then optimize the LLM with RL.
Shortcomings of RLHF in practice:
- Human preference data is expensive + slow
- Reward Hacking
- Training instability
- Prone to over-optimization (alignment tax)
file_37 Classic RLHF Pipeline
1. Pre-training:无监督预训练
2. Supervised Fine-tuning (SFT):指令微调
3. Reward Model (RM):用人类偏好对训练奖励模型
4. PPO:用 RM 作奖励信号优化 LLM
RLHF variants
| Method | Innovation |
|---|---|
| DPO (Direct Preference Optimization) | Skip RM, directly optimize using preference pairs |
| IPO | Improved version of DPO |
| KTO | Single-sample preference (no pairs needed) |
| ORPO | Joint SFT + preference |
| RLAIF | Use AI instead of humans to provide feedback |
file_38 Detailed Explanation of PPO
Main steps:
- Sampling: generate trajectory using current policy
- Calculate advantage: GAE
- Update policy: CLIP loss
- Repeat
Key concepts:
- CLIP: prevent the policy from deviating too much
- KL divergence penalty: constrain the new policy to not move far from the reference policy
- Value Function: estimate state value
file_39 RL Applications in NLP
| Task | RL application |
|---|---|
| Text generation | RLHF / DPO |
| Dialogue systems | RLHF + dialogue reward |
| Translation | Use BLEU as reward |
| Summarization | Use ROUGE / information coverage as reward |
| Recommendation | CTR / Reward Model |
8. SFT Data (file_40 + file_41)
file_40 Training Set Format
| Stage | Data format |
|---|---|
| SFT | One question one answer (instruction, output) |
| RM (Reward Model) | One question + good answer + bad answer |
| PPO | One question (sampled from prompt set) |
file_41 SFT Data Generation
Two methods:
- Human annotation — suitable for vertical domains, high quality, expensive
- LLM generation (GPT-4) — large volume, suitable for general purposes
Self-Instruct framework:
1. 种子任务(175 个)
2. LLM 生成新指令
3. LLM 生成对应输出
4. 过滤 + 去重
5. 用于 SFT
9. GPU Memory and Inference Optimization
file_42 GPU Memory Issues
| Q | A |
|---|---|
| How large is the model file? | 2n GB in fp16 (n billion parameters) |
| GPU memory during inference? | 2n GB in fp16 + KV cache + activations |
| Can 4*V100 32GB train vicuna 65B? | No(65B in fp16 = 130GB; 4×32=128GB cannot even fit the model, let alone the gradients) |
file_43 GPU Memory Optimization Strategies
| Strategy | Idea |
|---|---|
| Gradient Accumulation | Accumulate gradients over multiple mini-batches before updating, simulating a large batch |
| Gradient Checkpointing | Do not save some activations, recompute during backward pass (trade time for memory) |
| Mixed Precision | fp16 computation + fp32 master weights |
| DeepSpeed ZeRO | Distribute optimizer states / gradients / parameters across multiple GPUs |
| CPU Offload | Place Adam states on CPU |
| Quantization | int8 / int4 inference |
| Flash Attention | Reduce attention memory usage |
file_31 LLM Inference
| Q | A |
|---|---|
| Why does inference memory usage increase so much and stay occupied? | KV cache accumulation + PyTorch memory pool does not release memory |
| GPU vs CPU inference speed? | GPU is 10–100x faster (depends on the model) |
| int8 vs fp16 inference? | int8 is faster + memory-efficient + slight precision loss |
| How to accelerate inference? | KV cache / Flash Attention / quantization / speculative decoding |
| Batch inference vs single sample? | Batching amortizes fixed overhead, resulting in higher throughput |
10. LLM Evaluation (file_35)
Difficulties in evaluation
- SuperGLUE / GLUE / CLUE are not very suitable for evaluating large models
- Evaluation Reasoning ability + multi-turn dialogue ability are the core
Evaluation system
| Dimension | Benchmark |
|---|---|
| Commonsense reasoning | MMLU / HellaSwag / ARC |
| Mathematics | GSM8K / MATH |
| Code | HumanEval / MBPP |
| Multi-turn dialogue | MT-Bench / AlpacaEval |
| Long context | Needle In Haystack / LongBench |
| Chinese | C-Eval / CMMLU / GAOKAO-Bench |
| Safety | TruthfulQA / RealToxicity |
LLM-as-Judge
- Use GPT-4 as a judge
- Advantages: batch processing + low cost
- Disadvantages: GPT-4 has its own preferences
See also
- LLM Interviews / Basic Architecture and Components — Basic Architecture
- LLM Interviews / Complete Special Topic on RAG Retrieval-Augmented Generation — RAG
- LLM Interviews / 8 Illustrated Special Topics on Distributed Training — Distributed Training
- AI Product Manager Interviews and Career Transition / Compilation of 12 Materials — AI PM Questions
- Foundation Models for NLP (Springer Textbook Close Reading) — Complete Textbook Version
High-Frequency Exam Point Priority
- 🔥🔥🔥 LoRA + QLoRA — Must-Know
- 🔥🔥🔥 RLHF 4 Stages + DPO
- 🔥🔥 PEFT Method Comparison
- 🔥🔥 8 Tricks for GPU Memory Optimization
- 🔥🔥 SFT Data Construction + Self-Instruct
- 🔥 Continue Pretrain Catastrophic Forgetting Mitigation
- 🔥 CoT / Self-Consistency / ToT