LLM Distributed Training / GPU Clusters / Inference Optimization (Cross-Author Topic Synthesis)

When to open: You need to train LLMs (>13B) / do multi-node multi-GPU training / choose between DeepSpeed and Megatron / build inference services. This wiki organizes 9 PDFs into a practical stack.

One-sentence core:4 major parallelisms = DP + MP + PP + TP + Three stages of ZeRO + 3D Parallelism = TP * PP * DP + DeepSpeed for general use + Megatron for cutting-edge training + vLLM for inference.

0. Perspective note

Main source:LLM Interview / 8 Illustrated Specials on Distributed Training 9 PDFs / illustrated specials.

1. Why distributed training is needed

LLM Interview / 8 Illustrated Specials on Distributed Training §1:

Single-GPU bottlenecks	Solutions
Insufficient VRAM — A 65B model in fp16 takes 130GB, which cannot fit on an A100 80GB	Model parallelism / ZeRO
Overly long computation time — Pre-training a 13B model on a single GPU takes 100+ days	Data parallelism
Too small batch size leads to instability — Per-GPU batch=8 has high noise	Scale batch size via distributed training
Multi-node coordination — Inter-node communication	NCCL / Gloo / MPI

2. 4 major parallelism strategies

Strategy	Idea	Communication overhead	Applicable scenarios
Data Parallel(DP)	Each GPU holds a full copy of the model, processing different data shards	AllReduce gradients	Model fits on a single GPU
Model Parallel(MP)	Different parts of the model reside on different GPUs	Forward / backward activations	Extremely large models
Pipeline Parallel(PP)	Different layers reside on different GPUs, executed in a pipeline	Inter-layer activations	Across nodes
Tensor Parallel(TP)	Single-layer tensors split across multiple GPUs	Intra-layer AllReduce	Within a single node

2.1 Practical combination (3D Parallelism)

Used by Megatron-Turing NLG 530B:

TP=8(Within a single 8-GPU node)
PP=35(Across nodes)
DP = replacement

Total GPUs = TP × PP × DP.

3. Pipeline Parallelism (PP)

LLM Interview / 8 Illustrated Specials on Distributed Training §2:

3.1 Pipeline idea

模型: Layer1 → Layer2 → Layer3 → Layer4
分到 4 个 GPU:
  GPU0: Layer1
  GPU1: Layer2
  GPU2: Layer3
  GPU3: Layer4

mini-batch 切成 micro-batch:
  micro1, micro2, micro3, micro4

执行流水线:
  t=1: GPU0: micro1
  t=2: GPU0: micro2 | GPU1: micro1
  t=3: GPU0: micro3 | GPU1: micro2 | GPU2: micro1
  ...

3.2 Pain point: Bubble

GPUs sit idle waiting for data → utilization drops.

3.3 Solutions

Scheduling	Characteristics
GPipe(Google)	Synchronous pipeline / wait for mini-batch to complete
PipeDream(Microsoft)	Asynchronous pipeline / 1F1B scheduling
1F1B(one-forward-one-backward)	Reduces bubbles / primarily used by Megatron
Interleaved 1F1B	Further reduces bubbles
DeepSpeed PP	Integrates ZeRO

4. PyTorch data parallelism series

LLM Interview / 8 Illustrated Specials on Distributed Training §3:

4.1 nn.DataParallel (DP, obsolete)

model = nn.DataParallel(model)

Characteristics:

Single-process multi-threading
GIL lock limitation
High VRAM pressure on the primary GPU (aggregates all gradients)
Not recommended(obsolete)

4.2 nn.parallel.DistributedDataParallel (DDP, mainstream)

import torch.distributed as dist
dist.init_process_group(backend="nccl")
model = DDP(model, device_ids=[local_rank])

Characteristics:

Multi-process(no GIL lock)
Ring AllReduce communication
Each GPU computes gradients independently
Mainstream choice

4.3 torch.multiprocessing

LLM Interview / 8 Illustrated Specials on Distributed Training §file_48:

import torch.multiprocessing as mp
mp.spawn(train_worker, args=(world_size,), nprocs=world_size)

Used to launch multi-process training (paired with DDP).

5. AMP mixed precision training

LLM Interview / 8 Illustrated Specials on Distributed Training §file_49:

5.1 fp16 vs bf16

Format	Range	Precision	Applicable scenarios
fp16	Relatively narrow	High	Works for both training and inference / prone to overflow
bf16	Same as fp32	Relatively low	Recommended for training / no overflow
fp32	Large	High	Accumulation / optimizer states

5.2 AMP workflow

forward: fp16/bf16
loss: fp32
backward: fp16/bf16(梯度)
optimizer step: fp32(参数 + Adam 状态)
gradient scaler: 防止 fp16 下溢

5.3 Key APIs

from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
with autocast():
    output = model(input)
    loss = criterion(output, target)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

6. DeepSpeed

LLM Interview / 8 Illustrated Specials on Distributed Training §file_50:

6.1 What is DeepSpeed

A training framework released by Microsoft that integrates:

ZeRO(three-stage partitioning of optimizer / gradients / parameters)
Pipeline Parallel
Mixed Precision
CPU/NVMe Offload

6.2 Three stages of ZeRO

LLM Interview / 8 Illustrated Specials on Distributed Training §file_52:

Stage	What is partitioned	VRAM savings	Communication overhead
ZeRO-1	Optimizer states (Adam m, v)	4x	Baseline
ZeRO-2	+ Gradients	8x	Slight increase
ZeRO-3	+ Parameters	N x (N = number of GPUs)	Significant increase

ZeRO-Offload: Swap optimizer states / parameters to CPU (saves more VRAM, slower).

6.3 When to use ZeRO

Scenario	Choice
<13B models / single-node multi-GPU	ZeRO-1 or ZeRO-2
13B-65B / multi-node	ZeRO-3
>100B	3D Parallelism + ZeRO
Extremely large single-GPU models	ZeRO-Offload

6.4 DeepSpeed Config example

{
  "fp16": {"enabled": true},
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {"device": "cpu"},
    "offload_param": {"device": "cpu"}
  },
  "gradient_accumulation_steps": 4
}

7. Megatron-LM

7.1 Characteristics

Released by NVIDIA
Strengths: Tensor Parallel + Pipeline Parallel
Cutting-edge training(widely used for 530B / GPT-3 175B)

7.2 Megatron vs DeepSpeed

Dimension	Megatron	DeepSpeed
Strengths	TP + PP	ZeRO
Barrier to entry	High (C++ kernels)	Low (Python API)
Applicable scenarios	Training cutting-edge large models	Small and medium models + Offload
Mixed usage	Megatron-DeepSpeed	Combine the strengths of both

8. Accelerate(HuggingFace)

LLM Interview / 8 Illustrated Specials on Distributed Training §file_51:

8.1 One-sentence positioning

A lightweight distributed wrapper released by HuggingFace:Convert PyTorch code to multi-node multi-GPU with 5 lines of changes.

8.2 Core APIs

from accelerate import Accelerator
accelerator = Accelerator()
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

for batch in dataloader:
    output = model(batch)
    loss = criterion(output, batch.labels)
    accelerator.backward(loss)
    optimizer.step()

8.3 When to use

You do not want to write DeepSpeed configs
Simple DDP / fp16 / multi-GPU training
Suitable for small and medium training tasks such as SFT / LoRA

9. Communication backends

Backend	Applicable scenarios
NCCL	NVIDIA GPUs (mainstream)
Gloo	CPU training / cross-GPU-vendor setups
MPI	HPC clusters

Key parameters:

world_size = total number of GPUs
rank = global GPU index
local_rank = local GPU index on this node
MASTER_ADDR / MASTER_PORT = rendezvous

10. Inference optimization stack (exclusive integration)

10.1 Mainstream inference engines

Engine	Characteristics
vLLM	PagedAttention / Continuous Batching / Throughput king
TensorRT-LLM	NVIDIA / extreme low latency
TGI(Text Generation Inference)	Released by HuggingFace / easy to use
llama.cpp	CPU + quantization / Mac-friendly
MLC-LLM	Cross-platform / WebGPU
SGLang	Structured output / Radix Tree cache

10.2 PagedAttention (core of vLLM)

Manages KV Cache in pages (similar to OS virtual memory)
Solves the fragmentation problem of traditional KV Cache
2-3x throughput improvement

10.3 Continuous Batching

Does not wait for all requests in a batch to finish
Immediately places new requests into completed slots
2-5x throughput

10.4 Quantized inference

Method	Precision	Speedup
GPTQ	INT4 / post-training	1.5-2x
AWQ	INT4 / retains high precision for important weights	1.5-2x
GGUF(llama.cpp)	INT4-INT8	CPU-friendly
SmoothQuant	INT8 / smooths weights and activations	1.5x

11. Empirical estimation of training resources

11.1 Training time formula

训练 token 数 / (GPU 数 × 每 GPU 吞吐 token/s) = 总时间

11.2 Training cost of mainstream models (reference)

Model	GPU	Time	Estimated cost
GPT-3(175B)	1024 A100	1 month	~$5M
LLaMA-2 70B	2048 A100	Several weeks	~$3M
GPT-4	Tens of thousands of H100s	Several months	~$100M
DeepSeek-V3	2048 H800	2 months	~$5M (exclusive optimization)

12. How to use — distributed paths by role

Who you are	Primary method	Starting point
Getting started with multi-GPU training	DDP + Accelerate	accelerator.prepare()
Training a 13B model	ZeRO-2 + DeepSpeed	stage=2 + AMP
Training 65B+ models	3D Parallelism + Megatron-DeepSpeed	TP=8 + PP=8 + ZeRO-3
Single-GPU fine-tuning of large models	QLoRA + Offload	NF4 + ZeRO-Offload
Building inference services	vLLM + PagedAttention	OpenAI-compatible API
Running locally on Mac	llama.cpp / MLC	GGUF quantization
Need structured output	SGLang	Radix Tree cache

General rules:

Do not use nn.DataParallel(obsolete)
DDP + bf16 is the standard entry-level setup
Choose ZeRO stage based on model size(1/2/3)
3D Parallelism is mandatory for 100B+ models
vLLM is the top choice for inference services
PagedAttention + Continuous Batching = the king of throughput
Quantization (GPTQ/AWQ) delivers almost free 1.5-2x speedup in inference

13. Reference list

LLM Interview / 8 Illustrated Specials on Distributed Training — 9 PDFs / overview + pipeline + DP/DDP + multiprocessing + AMP + DeepSpeed + accelerate + ZeRO
LLM Fundamentals (Transformer / Attention / Three Types of PLMs) (Cross-Author Topic Synthesis) — foundational architecture (read alongside)
LLM Fine-tuning / LoRA / RLHF / DPO (Cross-Author Topic Synthesis) — fine-tuning VRAM + LoRA applications
Scaling Law / AGI Timeline / Software 3.0 (Cross-Author Topic Synthesis) — Scaling Law / training scale / compute economics (read alongside)