LLM Interviews / Infrastructure and Components

The "Fundamentals and Architecture" section of LLM large model interview questions. Source: AIGC Interview Bible (Ningjing Zhiyuan), totaling 9 PDFs plus 3 top-level comprehensive interview question sets.

I. Three Decoder Architectures (Fundamental Interview Topic 1)

Classification of mainstream open-source model families

Type	Representative models	Attention
Prefix Decoder family	ChatGLM / ChatGLM2 / U-PaLM	Bidirectional on input + unidirectional on output
Causal Decoder family	LLaMA-7B / LLaMA derivatives	Unidirectional left-to-right
Encoder-Decoder	T5 / Flan-T5 / BART	Bidirectional on input + unidirectional on output (independent encoder)

The difference between the three lies in different attention masks

Prefix Decoder:Applies bidirectional attention to the prefix (prompt), and unidirectional attention for subsequent generation
Causal Decoder:Fully unidirectional, left-to-right throughout
Encoder-Decoder:Encoder is bidirectional, Decoder is unidirectional, and the Decoder can access the full output of the Encoder

II. Layer Normalization (LN/RMS/Deep Norm)

1. Layer Norm formula

LN(x) = γ * (x - μ) / σ + β
其中 μ, σ 是 x 的均值和标准差(layer 内每个样本的特征维度)

2. RMS Norm

RMS Norm(x) = γ * x / RMS(x)
其中 RMS(x) = sqrt(mean(x²))

Characteristics compared to LN:

Simplification: removes the mean-centering shift term(only retains scaling, no shift)
Faster:Faster computation speed
Comparable or even slightly better performance

3. Deep Norm

Idea:

Up-scale the residual connection before performing LN (alpha > 1)
Down-scale model parameters during initialization (beta < 1)

Advantage:Solves the training stability problem for 1000-layer Transformers. Microsoft 2022 paper.

Which Norm do mainstream LLMs use?

Model	Norm
GPT series	Layer Norm
BERT	Layer Norm
LLaMA / GLM	RMS Norm
GPT-NeoX / GLM-130B	Deep Norm

III. Activation Functions

Basic FFN block formula

FFN(x) = max(0, xW_1 + b_1) * W_2 + b_2
中间维度通常为 4h

Main activation functions

Function	Formula	Properties
GeLU	x * Φ(x), where Φ is the Gaussian CDF	Smooth replacement for ReLU
Swish	x * σ(x)	Friendly to automatic differentiation
GLU	(x * W) ⊗ σ(x * V)	Linear Gated Unit
GeGLU	(x * W) ⊗ GeLU(x * V)	GLU + GeLU
SwiGLU	(x * W) ⊗ Swish(x * V)	GLU + Swish,Used by LLaMA

Intermediate dimension of GLU series = 4h * 2/3 (because there are 3 weight matrices instead of 2).

Which activation function do mainstream LLMs use?

Model	Activation
GPT-3	GeLU
LLaMA / LLaMA-2 / LLaMA-3	SwiGLU
PaLM	SwiGLU
GLM-130B	GeGLU

IV. Attention Upgrades

Problems with traditional Attention

Context length constraint(O(n²) memory usage)
Slow speed, high memory footprint

Attention optimization directions

Increase context length
Accelerate + reduce memory footprint

List of Attention variants

Variant	Idea
Sparse attention	Introduce sparse bias to reduce complexity
Linearized attention	Decouple the attention matrix from kernel feature maps, and compute in reverse order to achieve linear complexity
Prototype / memory compression	Reduce the number of Q / KV
Low-rank self-Attention	Capture low-rank properties
Attention and priors	Use priors to supplement or replace standard attention
Improve multi-head mechanism	MQA / GQA, etc.

Multi-Query Attention (MQA)

All heads share the same K and V, only retaining multiple Q heads
Effect:Greatly accelerates inference, significantly reduces KV cache memory usage
Cost: slight accuracy loss (later improved by GQA)

Grouped-Query Attention (GQA)

Compromise between MQA and MHA:Group Q heads, each group shares one pair of K and V
Effect:Accuracy close to MHA + memory usage close to MQA
Used by default in LLaMA-2 / 3

Flash Attention

Leverage GPU memory hierarchy,reduce HBM access
Tiled computation, without instantiating the full attention matrix
2-4x speedup, near-linear memory usage

V. Transformers Operations (Coding Questions)

Load Bert using transformers

from transformers import BertModel, BertTokenizer
model_name = 'bert-base-uncased'
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertModel.from_pretrained(model_name)

Output specified hidden_state

output = model(input_ids, output_hidden_states=True)
# output.last_hidden_state 最后一层
# output.hidden_states 每一层 list

BERT last-layer / per-layer vector outputs

last_hidden_state = last layer of the encoder
pooler_output = special processed output of <[BOS_never_used_51bce0c785ca2f68081bfa7d91973934]>
hidden_states[i] = i-th layer

VI. Loss Functions

KL divergence

KL(P || Q) = Σ P(x) log(P(x) / Q(x))

Measures the difference between two probability distributions. Asymmetric(KL(P||Q) ≠ KL(Q||P)).

Cross-entropy loss

CE = -Σ y_true * log(y_pred)

Physical meaning:Measures the difference between the model's predicted distribution and the true distribution.

Difference between KL divergence and cross-entropy

CE = H(P) + KL(P||Q)
H(P) is the entropy of the true distribution (a constant)
Optimizing CE is equivalent to optimizing KL(P||Q)

Use CE instead of MSE for classification

CE has large gradients(increasing with prediction error), leading to fast convergence
MSE suffers from gradient vanishing when using sigmoid, resulting in slow training

Softmax numerical stability issue

If e^x exceeds float range,subtract the maximum value before applying softmax(equivalent but avoids overflow):

softmax(x_i) = exp(x_i - max(x)) / Σ exp(x_j - max(x))

Excessively large loss differences in multi-task learning

Method	Idea
Weighting	Manually tune weights
Uncertainty weighting (Kendall et al.)	Learn the uncertainty for each task
GradNorm	Gradient normalization
PCGrad	Project conflicting gradients

VII. Similarity Functions

What else is there besides cosine similarity?

| Method | Formula | |---|---| | Cosine similarity | cos(θ) = A·B / (||A|| ||B||) | | Euclidean distance | √Σ(a_i - b_i)² | | Manhattan distance | Σ|a_i - b_i| | | Jaccard similarity | |A∩B| / |A∪B| | | Pearson correlation coefficient | cov(X,Y) / (σ_X σ_Y) |

Contrastive Learning

Unsupervised learning:Train the model to make representations of the same sample closer and different samples farther apart
Network:Siamese / Triplet
Loss:InfoNCE / Triplet Loss

Negative samples in contrastive learning

Importance:Depends on the specific task and data.

How to solve the high cost of constructing negative samples?:

Hard Negative Mining(Hard Negative Mining): select samples where the model performs worst as negatives each time
In-batch Negatives:Use non-self samples within the batch as negatives
MoCo / SimCLR:Use a momentum queue to expand the number of negative samples

VIII. Advanced LLM Interview Topics (Generative Models + Repetition Problem)

What is a generative large model?

Autoregressive generation:Predict the next token one by one based on previous tokens.

How do large models make generation rich instead of monotonous?

Sampling strategies:Temperature / Top-K / Top-P / Nucleus
Diversity enhancement:Multi-beam Beam Search / Diverse Beam Search

LLM repetition problem

Symptom:Repeats the same sentence / paragraph during generation.

8 major mitigation strategies:

Strategy	Idea
Unlikelihood Training	Penalize already generated tokens during training
Introduce noise	Add noise to training data to improve robustness
Repetition Penalty	Reduce the probability of already appeared tokens during inference
Contrastive Search	Compare with context and reduce similarity
Beam Search	Better coverage with multi-beam search
Top-K Sampling	Limit the selection space
Nucleus Sampler (Top-P)	Cumulative probability truncation
Temperature	Adjust distribution smoothness
No repeat ngram size	Forbid repeated N-grams
Repetition rate metric detection	Monitor repetition rate during training as a loss term

IX. LLM Training Experience Posts (Practical Orientation)

Distributed training framework selection

Use DeepSpeed more, use PyTorch native torchrun less. There is little difference when the number of nodes is small; DeepSpeed is powerful at hundreds of nodes (easy to launch + performance profiling).

Training advice checklist

Elastic fault tolerance and automatic restart — Large model training takes weeks or even months, and machine failures occur frequently
Regular model saving (checkpointing) — Essential for resuming from interruptions
Think clearly before starting training — Large model training is costly; clarify the purpose first, and record parameters and intermediate results
Learning rate warmup — Prevents loss explosion at the beginning of training
Gradient clipping — Prevents gradient explosion
Mixed Precision — fp16 for acceleration, fp32 for stability
Gradient Accumulation — Accumulate gradients to simulate large batches when memory is insufficient
Gradient Checkpointing — Trade time for memory

X. Comprehensive Interview Question Sets (Top 3 Major PDFs)

97 Transformer Interview Questions Summary

Covers:

Transformer architecture principles (self-attention / multi-head / positional encoding)
Differences between BERT / GPT / T5
Position Encoding (absolute / relative / rotary RoPE)
Training details (layer norm position / dropout / warmup)
Inference optimization (KV cache / FlashAttention)
Multilingual / multimodal Transformers

AI Large Model Interview Questions (102 Questions)