When to open:You are preparing for LLM engineering interviews / want to understand the architectural differences between mainstream models / want to get started via nanochat / are working on attention optimization. This wiki organizes the AI interview 8 must-knows together with the Karpathy nanochat paradigm.
One-sentence core:The difference between the three PLM architectures = different attention masks + LLaMA uses RMS Norm + SwiGLU + RoPE + GQA + Mainstream inference optimization = Flash Attention + KV Cache + quantization + Karpathy reproduces the full LM algorithm in 243 lines.
0. Perspective explanation
1. Three Decoder architectures (basic interview point 1)
1.1 Classification of mainstream open-source models
| Type |
Representative models |
Attention |
| Prefix Decoder |
ChatGLM / ChatGLM2 / U-PaLM |
Bidirectional input + unidirectional output |
| Causal Decoder |
LLaMA-7B / LLaMA derivatives / GPT series |
Unidirectional left-to-right |
| Encoder-Decoder |
T5 / Flan-T5 / BART |
Bidirectional input + unidirectional output (independent encoder) |
1.2 The difference between the three = different attention masks
- Prefix Decoder:Bidirectional attention on the prefix (prompt), unidirectional for subsequent generation
- Causal Decoder:Fully unidirectional, left-to-right throughout
- Encoder-Decoder:Encoder is bidirectional, Decoder is unidirectional, Decoder can access the full Encoder output
1.3 Why is Causal Decoder the mainstream?
- High training efficiency —— Single objective / clean pipeline
- Strong zero-shot capability —— GPT-3 emergence path
- Natural long-text generation —— Autoregressive generation in one go
2. Layer Normalization
2.1 Layer Norm(LN)
LN(x) = γ * (x - μ) / σ + β
2.2 RMS Norm
RMS Norm(x) = γ * x / RMS(x)
RMS(x) = sqrt(mean(x²))
Compared to LN:
- Simplification: removes mean shift (only keeps scaling, no shift)
- Faster
- Equivalent / slightly better performance
2.3 Deep Norm
- Up-scale residual connections before performing LN (alpha > 1)
- Down-scale parameters during initialization (beta < 1)
- Solves the training stability problem of 1000-layer Transformers (Microsoft 2022)
2.4 Which Norm do mainstream LLMs use?
| Model |
Norm |
| GPT series |
Layer Norm |
| BERT |
Layer Norm |
| LLaMA / GLM |
RMS Norm |
| GPT-NeoX / GLM-130B |
Deep Norm |
3. Activation functions (FFN block)
3.1 Basic FFN formula
FFN(x) = max(0, xW_1 + b_1) * W_2 + b_2
中间维度通常为 4h
3.2 Main activation functions
| Function |
Formula |
Properties |
| GeLU |
x * Φ(x) |
Smooth replacement for ReLU |
| Swish |
x * σ(x) |
Friendly to automatic differentiation |
| GLU |
(x * W) ⊗ σ(x * V) |
Linear gating |
| GeGLU |
(x * W) ⊗ GeLU(x * V) |
GLU + GeLU |
| SwiGLU |
(x * W) ⊗ Swish(x * V) |
GLU + Swish / Used by LLaMA |
Intermediate dimension of GLU series = 4h * 2/3 (because there are 3 weight matrices).
3.3 Which activation do mainstream LLMs use?
| Model |
Activation function |
| GPT series |
GeLU |
| BERT |
GeLU |
| LLaMA |
SwiGLU |
| GLM |
GeGLU |
| PaLM |
SwiGLU |
4. Attention upgrades
4.1 Standard Multi-Head Attention (MHA)
Attention(Q, K, V) = softmax(QK^T / sqrt(d)) * V
Each head has independent Q/K/V projections → memory / compute intensive.
4.2 Multi-Query Attention(MQA)
- Multi-Query / share 1 Key and 1 Value
- Reduced memory → faster inference
- Slight performance drop
4.3 Grouped-Query Attention(GQA)
- Middle ground between MHA and MQA
- Multiple Q heads, grouped K/V (g groups)
- LLaMA-2 / 3 use GQA
- Inference speed close to MQA, performance close to MHA
4.4 Flash Attention
- Do not explicitly store the Attention Matrix
- Block-wise computation + softmax recomposition
- Speeds up both training and inference + large memory reduction
- Stanford Tri Dao 2022
4.5 Linear attention / sparse attention
| Type |
Idea |
| Linear Attention |
Approximate softmax with kernel functions → O(n) |
| Sparse Attention |
Only compute partial token pairs → Longformer / BigBird |
| Sliding Window |
Local window → Mistral |
5. Positional encoding
| Type |
Representative |
| Absolute Position Embedding |
BERT / original Transformer |
| Relative Position |
T5 |
| RoPE(Rotary Position Embedding) |
LLaMA / GLM / mainstream |
| ALiBi |
BLOOM |
Advantages of RoPE:
- Naturally supports long-sequence extrapolation
- Rotary matrix encodes relative positions
- Naturally fused with attention computation
6. Quick reference for mainstream LLM configurations
| Model |
Norm |
Activation |
Attention |
Positional encoding |
| GPT-3 |
LN |
GeLU |
MHA |
Absolute |
| LLaMA-1 |
RMS |
SwiGLU |
MHA |
RoPE |
| LLaMA-2 |
RMS |
SwiGLU |
GQA |
RoPE |
| LLaMA-3 |
RMS |
SwiGLU |
GQA |
RoPE |
| ChatGLM |
LN |
GeGLU |
MHA |
RoPE |
| Mistral |
RMS |
SwiGLU |
Sliding Window |
RoPE |
| DeepSeek-V2 |
RMS |
SwiGLU |
MLA (exclusive) |
RoPE |
7. Loss functions + similarity
7.1 Loss functions
| Function |
Formula |
Use case |
| Cross-Entropy (CE) |
-Σ y log(ŷ) |
Classification / next-token prediction |
| KL divergence |
Σ p log(p/q) |
Distillation / RLHF reference alignment |
7.2 Similarity
| Function | Formula | Use case |
|---|---|---|
| Cosine similarity | (a·b) / (|a||b|) | Vector retrieval / RAG |
| Euclidean distance | sqrt(Σ(a-b)²) | k-NN |
| Jaccard | |A∩B| / |A∪B| | Jaccard similarity / text deduplication / set similarity |
8. Advanced LLM interview topic: repetition problem
LLM interviews / basic architecture and components §Advanced LLMs:
8.1 What is the repetition problem?
LLMs get stuck in repetitive patterns during generation:
- Repeating the same phrase ("I don't know. I don't know. I don't know...")
- Repeating template answers
- Repeating user input
8.2 8 major mitigation strategies
- Temperature parameter —— Increase temperature to boost diversity
- Top-K / Top-P sampling —— Restrict candidate set
- Repetition Penalty —— Penalize already appeared tokens
- Beam Search control —— diverse beam search
- Deduplication (no-repeat-ngram) —— Prohibit repeated n-grams
- SFT data diversity —— Mix diverse answers during training
- DPO / RLHF preference alignment —— Penalize repetition via user feedback
- LLM-as-Judge filtering —— Post-inference processing
9.1 Tokenizer
| Algorithm |
Representative |
| BPE(Byte Pair Encoding) |
GPT-2 / LLaMA |
| SentencePiece |
T5 / LLaMA |
| WordPiece |
BERT |
| Tiktoken |
GPT-3/4 |
9.2 KV Cache (key to inference optimization)
- Cache K / V —— Avoid recomputation every step
- Memory explosion → GPUs cannot handle long sequences
- Solutions:GQA + quantization + PagedAttention (vLLM)
9.3 Inference speed optimization
| Technique |
Speedup factor |
| KV Cache |
Baseline |
| Flash Attention |
1.5-2x |
| PagedAttention(vLLM) |
2-3x |
| Quantization(INT8/INT4) |
1.5-2x |
| Speculative Decoding |
2-3x |
| Continuous Batching |
2-5x throughput |
10. Karpathy nanochat beginner path
Karpathy · Software in the era of AI(Software 3.0) + Xiaohongshu / Vibecoding aesthetics + creator economy collection:
10.1 Value of nanochat / nanoGPT
- Full algorithm in 243 lines of PyTorch —— Reproduced by Karpathy
- The extra hundreds of thousands of lines in GPT are only there to make it run faster(Systems engineering, not algorithm)
- Learning path:Read nanoGPT → nanochat → truly understand Transformer
10.2 Xiaohongshu zero-basis explanation (supporting prompts)
- Student and teacher grading fill-in-the-blanks metaphor for backpropagation
- 10 browser modules + new word explanations
- Let ordinary users follow along with Karpathy's YouTube walkthrough
10.3 Recommended Karpathy YouTube series
- Let's build GPT: from scratch, in code, spelled out —— Transformer core
- Let's reproduce GPT-2 —— Engineering-level reproduction
- Let's build the GPT Tokenizer —— Detailed BPE explanation
- nanochat —— Complete chatbot training
11. LLM training experience posts
LLM interviews / basic architecture and components §10:
11.1 Training stage division
| Stage |
Goal |
Data |
| Pretrain |
Learn language + knowledge |
Trillions of tokens / Common Crawl + high-quality data |
| Continue Pretrain |
Inject domain knowledge |
70% domain + 30% general |
| SFT(Supervised Fine-tuning) |
Learn instruction following |
High-quality instruction pairs |
| RLHF / DPO |
Preference alignment |
Human preference pairs |
11.2 Common pitfalls
- Loss suddenly becomes NaN → Deep Norm / gradient clipping / fp32 accumulation
- Model becomes dumb after SFT → Catastrophic forgetting / SFT overfitting
- Continue Pretrain fails to learn domain knowledge → Data quality / ratio / training steps
- General capability drops after domain fine-tuning → LoRA increment + general data mixing
12. How to use —— LLM fundamentals path by identity
| Who you are |
Main method |
Starting point |
| Want to interview for LLM engineering |
llm-interview-basics 12 PDF |
Three Decoder types + Norm + activation + Attention |
| Want to start from zero |
Karpathy nanochat |
nanoGPT → nanochat → YouTube series |
| Want to choose a model |
Mainstream configuration quick reference |
LLaMA-3 / Mistral / DeepSeek-V2 |
| Need to work on long sequences |
RoPE + Flash + GQA |
LLaMA-2 / 3 route |
| Need to do inference optimization |
KV Cache + vLLM + quantization |
PagedAttention + INT8 |
| Want to build a chatbot |
Causal Decoder + SFT + DPO |
LLaMA / Mistral + LoRA |
General principles:
- Difference between three PLMs = different masks(basic interview point 1)
- LLaMA paradigm = RMS Norm + SwiGLU + RoPE + GQA
- The full algorithm is 243 lines, everything else is engineering(Karpathy)
- Flash Attention / vLLM are mandatory for inference optimization
- Solve the repetition problem with a combination of the 8 major strategies
13. Reference list
Sources and related materials
