Foundation Models for NLP (In-Depth Reading of a Springer Textbook)

When to open this: You want to build a academic foundation (not industry news, not product reviews) for "large models". This wiki is an in-depth reading of the open-access Springer textbook, authored by Gerhard Paaß (senior scientist at Fraunhofer IAIS) and Sven Giesselbach (head of the IAIS NLU team), completed in February 2023,436 pages / 8 chapters / full in-depth reading.

Differences from other wikis:

AI Agent Technology Architecture Evolution 2025 (In-Depth Research Report): Industry research (January 2026), technology stack + frameworks + cases

In-Depth Reading of 3 Agent Research Papers (2024–2025): In-depth reading of 3 research papers (More Agents / ChatQA / CAMEL)

This wiki: academic textbook, covering the complete academic evolution from 70 years of NLP to Foundation Models,mathematical foundational theory

Reading all three together = academia + papers + industry = a complete knowledge stack.

1. Positioning of the book & authors

1.1 Author backgrounds

Gerhard Paaß: Senior Scientist at Fraunhofer IAIS, PhD in mathematics and computer science (University of Bonn), core figure in AI education
Sven Giesselbach: Head of the IAIS NLU team, researcher in Informed Machine Learning
Institution: Fraunhofer IAIS (German national laboratory level)
Funding: Funded by the German Federal Ministry of Education and Research under ML2R
Features:Open Access published, freely shared with NLP researchers and engineers worldwide

1.2 Core thesis of this book

Software has followed a paradigm that has persisted for 70 years: Software 1.0 (writing code). In the last 5 years, a "language revolution" has occurred: through embedding + self-attention, models can learn from unlabeled text. Models such as GPT-3 demonstrate "emergent" behavior — the same architecture can process text, as well as images, audio, video, DNA, control signals, and any other kind of "sequence".

Therefore, the term "Pre-trained Language Model(PLM)" is already too narrow; Bommasani et al. (2021) coined the term "Foundation Model"to emphasize that a single model can provide the foundation for many downstream tasks. This book is the complete academic presentation.

1.3 Structure of the book (8 chapters)

Chapter	Topic	Key models
Ch 1	Classical NLP fundamentals (reference for comparison)	RNN, CNN, Word2Vec, GloVe, ELMo
Ch 2	Three major PLM architectures + training	BERT, GPT-2, Transformer
Ch 3	Six major directions for improving PLMs	RoBERTa, ELECTRA, XLNet, T5, GPT-3, PaLM, MoE, distillation
Ch 4	Knowledge evaluation + reasoning	GLUE, SuperGLUE, BIG-bench
Ch 5	Information extraction applications	XLNet, DeBERTa, BART for IE
Ch 6	Text generation applications	WebGPT, Retro, LaMDA, BlenderBot 3
Ch 7	Multimodality (speech/image/video/control/DNA)	ViT, DALL-E 2, GLIDE, NÜWA, Flamingo, GATO, DNABERT
Ch 8	Summary + economic impact + harms + future	—

2. Ch 1: Classical NLP fundamentals (reference for comparison)

2.1 Text Preprocessing

Tokenization:Byte-Pair Encoding (BPE) (used by GPT) / WordPiece (used by BERT) / SentencePiece (for multilingual and Asian languages)
Key judgment:Subword tokenization avoids OOV, common words are kept as whole words, rare words are split
Modern PLMs no longer require stemming / stop word removal / lemmatization — LLMs automatically learn these features

2.2 Vector Space Models

bag-of-words + tf-idf → logistic classifier (classical baseline)
Loss function: Maximum Entropy / Cross Entropy L_MC(w) = -Σ log p(y_i | x_i; w)
Three training paradigms:
- Supervised (labeled)
- Unsupervised (unlabeled, clustering)
- Self-supervised(deriving supervision signals from the data itself, such as predicting the next token) — the foundation of PLMs

2.3 Static Word Embeddings(Word2Vec / GloVe / FastText)

Core idea (Firth): "a word is characterized by the company it keeps" — the meaning of a word is determined by its neighbors
Word2Vec: uses the center word embedding to predict neighboring words (logistic classifier + noise contrastive estimation for acceleration)
Analogical relations:vec(Germany) - vec(Berlin) + vec(Paris) ≈ vec(France)
Fundamental limitation of static embeddings: "bank" has only one vector, unable to distinguish between the financial / river meanings → leading to contextual embeddings

2.4 RNN(GRU / LSTM)

Recurrent neural networks process sequences:h_{t+1} = RNN(h_t, x_t)
LSTM / GRU solve "vanishing/exploding gradients", but still struggle when sequence length is long
bidirectional RNN(biLSTM): runs forward + backward simultaneously
ELMo(2018): uses biLSTM to generate contextual embeddings — a harbinger of the PLM era

2.5 CNN

AlexNet (2012): 1000-class image classification, the tipping point for Deep Learning
ResNet: residual connections enable training networks with hundreds of layers
CNNs in NLP: Kim (2014) used CNNs for sentence classification; Wavenet used dilated causal convolutions for text-to-speech

3. Ch 2: Three major PLM architectures

The most central chapter. All modern LLMs are variants of these three architectures.

3.1 BERT(Autoencoder)— 2.1

Bidirectional Encoder Representations from Transformers(Devlin et al., 2018)

3.1.1 Core mathematics of Self-Attention

Q = X·W^Q   K = X·W^K   V = X·W^V
Attention(Q,K,V) = softmax(Q·K^T / √d_k) · V

Query / Key / Value are the results of projecting input embeddings through 3 matrices
Scaled dot-product attention:(q^T · k) / √d_k(to avoid softmax saturation)
Asymmetric (because W^Q ≠ W^K), suitable for expressing asymmetric semantic roles in "Fred gave roses to Mary"

3.1.2 Multi-Head Attention

BERT_BASE: 12 heads, each head uses different W^Q / W^K / W^V matrices
Different heads capture different semantic relations (syntax, semantics, coreference, etc.)
Outputs are concatenated and passed through an FCL to obtain the next layer of embeddings

3.1.3 Encoder Block

input embedding(token + position + segment)
   ↓
multi-head self-attention
   ↓  (residual + layer norm)
position-wise FCL (2 层,中间维度 4*D,ReLU)
   ↓  (residual + layer norm)
output embedding

BERT_BASE: 12 encoder blocks × 12 heads = 110M parameters.

3.1.4 Training: Masked Language Model (MLM)

Randomly mask 15% of tokens
Of these, 80% are replaced with [MASK], 10% with a random token, 10% left unchanged (to avoid mismatch when there is no mask during fine-tuning)
Use a logistic classifier to predict the masked token
Additional task:Next Sentence Prediction(later shown to provide little benefit)
Training data: Wikipedia + book corpus = 3.3B words
Training time: 4 days × 64 TPUs

3.1.5 Fine-tuning for 5 downstream tasks

Task type	Input	Output method
Text Classification	Text	`[CLS]` embedding → logistic
Text Pair Classification	Text A + `[SEP]` + Text B	`[CLS]` embedding → logistic
Word Annotation(NER)	Text	Each token embedding → logistic
Span Prediction(QA)	Question + `[SEP]` + Context	2 logistic classifiers predict span start and end

3.1.6 Computational complexity (comparison)

Layer Type	Complexity / Layer	Sequential Ops	Max Path Length
Self-attention	O(T²·D)	O(1)	O(1)
Recurrent	O(T·D²)	O(T)	O(T)
Fully Connected	O(T²·D²)	O(1)	O(1)
Convolutional	O(K·T·D²)	O(1)	O(log_K(T))

Self-attention's maximum path length O(1) is key — distant tokens are directly connected, while RNNs require O(T) steps to propagate.

3.1.7 Achievements of BERT

GLUE average score: 75.2 → 82.1(human 87.1)
SQuAD 1.0:91.7 → 93.2(human 91.2)
Became the general-purpose backbone model for NLP, extensively inherited by RoBERTa / DeBERTa / ALBERT / DistilBERT

3.2 GPT(Autoregressive)— 2.2

Generative Pre-trained Transformer(Radford et al., 2018)

3.2.1 Core differences (vs BERT)

BERT: bidirectional (looks at left and right)
GPT:Unidirectional(only looks at the left side,causal/masked self-attention)
Training:predict the next token, rather than fill-in-the-blank
Directly derived from the Transformer decoder

3.2.2 Probability factorization

p(v_1, ..., v_T) = p(v_T|v_1,...,v_{T-1}) · ... · p(v_2|v_1) · p(v_1)

Autoregressive → predicts each token in sequence
Loss: negative log-likelihood = -Σ log p(v_t | v_1, ..., v_{t-1})

3.2.3 Perplexity evaluation

perplexity = p(v_1, ..., v_T)^(-1/T)

GPT-2 perplexity on Penn Treebank:46.5 → 35.8(large reduction = the model "understands" language better)

3.2.4 Sampling strategies for text generation

Method	Description	Applicability
Random sampling	Completely random according to the probability distribution	Often selects ultra-low-probability words, poor quality
Top-k sampling	Only considers top-k tokens, renormalizes	Standard for practical generation
Top-p (nucleus) sampling	Tokens with cumulative probability exceeding p	Standard for practical generation
Entmax	Sparse version of softmax	Experimental

3.2.5 GPT-2 details

1.5B parameters (largest), 48 layers, 1600 tokens context
Training: 40GB of Reddit text
First zero-shot generation of coherent articles (almost indistinguishable from humans)
Demonstrated zero-shot learning (can solve tasks without specialized training)

3.3 Transformer(Encoder-Decoder)— 2.3

The original paper by Vaswani et al. (2017) "Attention Is All You Need"

3.3.1 Architecture

Source: "The mouse likes cheese"          Target: "Die Maus mag Käse"
   ↓                                                 ↑
[Encoder: 6 blocks]                          [Decoder: 6 blocks]
                                                     ↑
   contextual embeddings ──── cross-attention ───────┘

Encoder: Same as BERT, generates contextual embeddings for the source language
Decoder: Autoregressive like GPT, but adds cross-attention to attend to encoder outputs
Used for: seq2seq tasks such as translation / summarization / QA

3.3.2 Cross-Attention

Decoder layer 自己的 embedding → Query
Encoder 最后层 embedding → Key + Value

Note: This is the key mechanism by which the Decoder "sees" the input text.

3.3.3 Decoding strategies

Method	Description
Greedy decoding	Selects the highest-probability token at each step, simple but prone to getting stuck in suboptimal outputs
Beam search(beam=4)	Keeps k candidates, balances quality and cost (used in practical translation)

3.3.4 Translation evaluation metrics

Metric	Properties	Description
Bleu	precision	Compares 1-gram to 4-gram matches against reference translations
Rouge	recall	unigram / bigram / longest common subsequence
Meteor	F-mean	word-to-word alignment, including stemming + synonymy
BERTscore	embedding similarity	Compute cosine similarity using BERT embeddings

3.3.5 Comparison of the Three Architectures

Model	Type	Training Task	Applicable to
BERT	Autoencoder	Predict masked tokens	NLU (understanding tasks)
GPT-2	Autoregressive	Predict the next token	NLG (generation tasks)
Transformer	Encoder-Decoder	Input → output seq2seq	Translation / summarization / QA

Key insight: these 3 types collectively are referred to as Pre-trained Language Models(PLM); when parameters >= 1B they are often called Foundation Models.

3.4 Training & Evaluation — Section 2.4

3.4.1 Evolution of SGD Optimizers

SGD → Momentum → AdaGrad → AdaDelta → RMSProp → Adam → LAMB

Adam: default for most PLMs (combines AdaGrad and RMSProp)
LAMB: reduced BERT training time from 3 days → 76 minutes(batch size 32868)

3.4.2 Parallelization (essential for GPT-3 scale)

Data parallelism: same model, different data
Pipeline parallelism: different layers on different GPUs
Within-layer parallelism: weights of the same layer distributed across multiple GPUs
DeepSpeed / GSPMD: tools

3.4.3 Regularization

Method	Purpose
Dropout	Randomly mask hidden variables
Layer Norm	Replaces BatchNorm, default for PLMs
RMSNorm	Simplified version of Layer Norm, divides only by root mean square
Label Smoothing	Change labels from one-hot to soft
Residual connections(introduced by ResNet)	Train hundreds of layers

3.4.4 Three Methods for Uncertainty Estimation

Bayesian Neural Networks: posterior is hard to compute, use MCMC / variational inference / Laplace approximation
Single Deterministic(Dirichlet, SNGP): single network outputs uncertainty
Ensemble: train multiple networks and average (most reliable in practice, but expensive)

3.4.5 Five Methods for Interpretability

Feature importance: gradient of input tokens with respect to output
Counterfactual: modify input and observe how output changes
Surrogate models(LIME): approximate with a simple linear model
Example-driven: show similar training samples
Source citation: WebGPT style — attach document citations to answers
Chain of Thought(CoT): let the model generate reasoning steps (demonstrated by PaLM)

IV. Ch 3: Six Major Directions for Improving PLMs

4.1 Modifying Pre-training Tasks (3.1)

Category	Representative Models	Innovation
BERT-like autoencoder	RoBERTa	Larger data + remove NSP + longer training
	ELECTRA	Replaced token detection (more efficient)
	DeBERTa / DeBERTaV3	disentangled attention
	ALBERT	parameter sharing
	XLNet	permutation-based
GPT-like autoregressive	GPT-3(175B)	Emergence of few-shot prompting
	PaLM(540B)	Pathway scaling, outperforms human average on BIG-bench
	Gopher / Chinchilla	scaling laws
	MT-NLG(530B)	DeepSpeed training
	Wu Dao(1.75T MoE)	Chinese large models
Encoder-Decoder	T5	text-to-text unified
	BART	denoising autoencoder
	mT5 / mBART	multilingual

4.2 Handling Longer Inputs (3.2)

self-attention has O(T²) complexity, explodes when input is too long. Solutions:

Sparse attention: Longformer / BigBird / Sparse Transformer (only attend to partial positions)
Hashing / Low-rank:Reformer(LSH) / Linformer / Performer
Empirical results: can handle 4K–16K context (as of the book's 2023 timeframe)

4.3 Multilinguality (3.3)

mBERT / XLM-R: multilingual autoencoder
mT5 / mBART: multilingual seq2seq
XGLM: multilingual autoregressive
Strong transfer ability from high-resource languages to low-resource languages
PaLM covers 100+ languages, but 78% English / 3.5% German

4.4 Knowledge Injection (3.4)

Method	Description
Knowledge Base embeddings(KGE)	Treat KB entities as "virtual tokens"
Graph Learning	Graph structure + PLM
Textual encoding of tables	TAPAS / TaBERT
KB relations textually encoded	KEPLER
Retrieval-augmented(Critical!)	REALM / RAG / Retro — supplement knowledge with retrieval

4.5 Changing Model Size (3.5)

Bigger is better:Scaling Laws(Kaplan, Hoffmann)
Mixture-of-Experts(MoE):Activation sparsity, 1.6T parameters but only ~10B are used each time
- Switch Transformer / GShard / GLaM
Compression / distillation: DistilBERT (66M parameters, 90% performance)
Low-rank factorization: ALBERT's factorized embedding

4.6 Fine-tuning (3.6)

Variant	Description
Full fine-tuning	Update all parameters (original approach)
Adapter	Add small modules, train only these
Prompt tuning	Only tune prompt embeddings
LoRA(Low-Rank Adaptation)	Low-rank matrix fine-tuning (efficient and popular)
P-tuning v2	Hybrid prompt + adapter

4.6.1 Few-shot Prompts (key to GPT-3's emergent abilities)

In-context learning: provide a few examples in the prompt, the model learns from them
Chain-of-Thought(CoT): prompt the model to reason step by step (PaLM)
Instruction tuning: fine-tune with instruction-style data (InstructGPT / FLAN)
RLHF(became popular after this book was published): followed by a large number of papers

4.6.2 Using Foundation Models to Generate Labeled Data (3.6.6)

GPT-3 few-shot generates training data → small model learns from it
A new paradigm of data augmentation + distillation

V. Ch 4: Knowledge Evaluation of PLMs

5.1 Major Benchmarks

Benchmark	Number of tasks	Nature
GLUE	9	Classic NLU
SuperGLUE	8	Harder NLU
LAMBADA	1	Predict the last word of a paragraph (tests long-distance understanding)
BIG-bench	200+	Human-like multifaceted test (PaLM outperforms human average)
HELM	Multiple	Systematic evaluation
MMLU	57 subjects	Multi-domain knowledge

5.2 Probing Classifiers

Do not modify the PLM, only use its embeddings to train a simple classifier
Tests syntactic knowledge(POS / dependencies) / commonsense knowledge(WSC) / logical consistency
Finding:BERT is strong at syntax, but still weak at commonsense and logic

5.3 Transferability & Reproducibility

Results on different benchmarks do not necessarily transfer across tasks
NLP papers have low reproducibility(sensitive to random seeds / hyperparameters)

VI. Ch 5: Information Extraction

6.1 Text Classification

Multiclass(mutually exclusive classes): in most cases use BERT + [CLS]
Multilabel(multi-label): independent sigmoid for each label
Few/Zero-shot: Prompt-based, relies on existing knowledge in PLMs

6.2 Word Sense Disambiguation(WSD)

Task: is "bank" sloping land or a financial institution?
Classic task, models are evaluated on SemCor
The BERT series raised WSD accuracy to 83%

6.3 Named Entity Recognition(NER)

Subtask	Description
Flat NER	Flat tagging
Nested NER	Nested entities (e.g., NYU in "NYU School of Law")
Entity Linking	Link to KBs such as Wikipedia / Wikidata

CoNLL-2003 NER: BERT raised F1 from 92.6 to 92.8(gradual progress)

6.4 Relation Extraction

Subtask	Difficulty
Coreference Resolution	Medium
Sentence-level RE	Medium
Document-level RE	High
Joint Entity + Relation	High
Distant Supervision (automatic KB annotation)	Medium
Layout-aware RE(LayoutLM etc.)	Tables / receipts

VII. Ch 6: Text Generation

7.1 Document Retrieval(6.1)

From BM25 / TF-IDF → Dense Retrieval(using embeddings)
DPR(Dense Passage Retrieval): dual encoder + dot product
ColBERT:token-level interaction
FAISS: large-scale nearest neighbor search
MS-MARCO dense retrieval significantly outperforms keyword-based methods on benchmarks

7.2 Question Answering(6.2)

Type	Description
Closed-book(knowledge from training data)	T5 fine-tune,GPT-3 prompt
Open-domain(retrieve + read)	DPR + reader, FB Hybrid (67.4%, close to expert level)
Long-form QA(retrieve + generate passage)	RAG / Retro / WebGPT

7.3 Neural Machine Translation(6.3)

Monolingual pairs vs Multilingual (100+) mT5 / NLLB-200
Back-translation: augment training with monolingual data
Document-level(full documents rather than single sentences)
Some language pairs reach/exceed human reference translations

7.4 Text Summarization(6.4)

Type	Model
Short document summarization	BART / Pegasus
Long documents (>4K)	BigBird / LongT5
Multi-document	Hierarchical / FiD

RLHF on summarization: in OpenAI experiments, 70% of users preferred RL-tuned outputs over reference summaries

7.5 Text Generation(6.5)

7.5.1 Three Ways to Control Generation

Style tags(CTRL)
Storyline control(Plot keywords)
Few-shot prompting(GPT-3 style, recommended)

7.5.2 Fake News & Code

GPT-2 can generate almost indistinguishable fake news → risk of misuse
Code generation: Codex / GitHub Copilot / AlphaCode (2022 level)

7.6 Dialog Systems(6.6)

Old style: Pipeline modules (intent recognition + slot filling + answer generation)
New style end-to-end PLMs:BlenderBot 3 / LaMDA / Sparrow
LaMDA(137B): add safety + quality + groundedness fine-tuning,approaches human level
Limitations: hallucination + bias + toxicity

VIII. Ch 7: Multimodality (the most interesting chapter in the book)

8.1 Speech(7.1)

HuBERT / wav2vec 2.0 / Whisper: self-supervised speech + transcribe
LibriSpeech WER 1.4% (superhuman)
WaveNet: first model capable of synthesizing realistic 16kHz speech
AudioLM: speech model (analogous to text models)

8.2 Image(7.2)

8.2.1 Vision Transformer (ViT)

Cut images into 16×16 patches → treat each patch as a "token"
Process with BERT-style self-attention
Surpasses ResNet on ImageNet (and trains faster)

8.2.2 Image Generation Model Timeline

Model	Type	Key point
DALL-E(2021)	autoregressive	First large text-to-image model
GLIDE(2021)	diffusion	OpenAI
DALL-E 2(2022)	CLIP + diffusion	1024×1024 high resolution
Imagen(2022)	diffusion + T5	Google
Stable Diffusion(2022)	latent diffusion	Open source, game-changing

8.2.3 General-purpose (OFA / Flamingo)

OFA: unify text and image tokens, perform captioning / VQA / image generation
Flamingo: video + text foundation model

8.3 Video(7.3)

Tubelet input (image patches across multiple frames)
NÜWA: uniformly process text/image/sketch/video, perform 9 generation tasks
Imagen Video / Make-A-Video: high-definition text-to-video

8.4 Control(7.4)

8.4.1 Decision Transformer

Treat control problems as "sequence modeling"
Input:(reward_t, state_t, action_t) triples
Training: predict the next action (given desired future reward)
Applications: Atari games / board games / robotics

8.4.2 GATO(DeepMind)

A single Foundation Model, simultaneously performing:
- Text QA
- Image captioning
- Atari games (same architecture + different tokens)
- Robot control
1.2B parameters,PoC for multimodal general intelligence

8.5 DNA / Protein(7.5)

DNABERT: treat DNA as a "language", predict promoter regions, motifs
AminoBERT: directly predict 3D structure from protein sequences (outperforms AlphaFold2 on some tasks)
A new paradigm for biomedicine

IX. Ch 8: Summary & Future

9.1 Foundation Models Are a New Paradigm (8.1)

Four generations of AI model evolution

Generation	Description	Examples
1980s	Expert Systems	rule-based
~2000s	Machine Learning	SVM, random forest, feature engineering
2010s	Deep Learning	CNN, RNN
2020s	Foundation Models	BERT, GPT, PaLM, DALL-E

Fei-Fei Li's judgment:Foundation Models = "phase change in AI"

9.2 Economic Applications (8.1.4) — 8 Major Domains

Search & Retrieval: embedding-based search replaces keyword search
Writing: automated drafting, checking, information supplementation
Translation: across 100+ languages, low-resource languages benefit the most
Chatbots: LaMDA / BlenderBot, customer service / pre-sales
Healthcare: $4T market — medical record analysis, imaging, drug discovery, psychological counseling (Woebot)
Genomics / Proteomics: DNABERT / AminoBERT, accelerating drug development
Legal: 1.3M lawyers in the United States, automation of contracts / cases / legal research
Education: personalized teaching, reducing student loan burden

9.3 Five Major Potential Harms (8.2)

Harm	Mechanism	Examples
1. Bias / Toxic Language	Training data reflects social biases	GPT-3: 83% of occupations are associated with males; Islam → terrorist at 60% frequency
2. Intentional Harm	Deliberately train fake news / attack code	Fine-tune GPT-2 for extreme far-right propaganda
3. Overreliance / Anthropomorphism	Users treat conversational models as real people	XiaoIce has 660 million users, average 23 dialogue turns (more than interpersonal conversations)
4. Privacy Leakage	Models memorize training data	PaLM reproduces 2.4% of training samples
5. Societal Harm	Energy consumption / inequality / cultural homogenization	PaLM training emits 271 tons of CO₂ (= one SFO–JFK flight)

9.4 Legal Regulation (end of 8.2.5)

EU AI Act: 4 risk levels (minimal / limited / high / unacceptable)
US: FTC + Commerce Department (voluntary framework)

9.5 Advanced AI (8.3) — 5 Future Directions

9.5.1 Can Foundation Models Innovate?

Opposing side (Marcus, Bender):stochastic parrots(stochastic parrot, only capable of parroting/repeating)
Supporting side:Novel combinations within the contextual embedding space = innovation (e.g., "corgi playing a flame-throwing trumpet")

9.5.2 Grounding Language to the World

"language modeling task, because it only uses linguistic forms as training data, cannot in principle lead to learning of meaning" — Bender et al. 2020

Babies learn while watching (self-supervised perception)
LeCun proposes: self-supervised learning of the physical world from video self-supervised learning of the physical world (NÜWA has already started)
5 grounding scopes (Bisk 2020):
1. Curated corpora(BERT)
2. Web-scale text(GPT-3, PaLM)
3. Mix with other modalities (OFA, NÜWA) — partially implemented
4. Embodiment(sensors + motor control) — future
5. Social interaction — more distant future

9.5.3 Kahneman Fast & Slow Thinking

System	Speed	Description
System 1	Fast	Intuition, instinct, automatic — current PLMs are close to this
System 2	Slow	Reasoning, reflection, logic — current models still fall short

LeCun: System 2 requires planning + internal world models + model predictive control.

9.5.4 Planning Strategies

Decision Transformer / GATO is an early attempt at Planning
LeCun's vision:High-level configurator + multiple world models + dynamic scheduling
SayCan: verbally expressing PLM + robotic skills (early 2022 PoC)
Most researchers believe that Symbolic and Neural approaches are not necessarily contradictory — Hinton:"symbols just exist out there in the external world, and we do internal operations on big vectors"

9.6 LeCun's Judgment

"The ability to construct models of the world is basically the essence of intelligence."

Challenges for AI in the next 10 years:Learning world models and handling uncertainty.

10. Limitations of This Book & My Judgment

10.1 Publication Time Point of This Book (2023-02)

This book was written in the second half of 2022 and finalized in 2023-02. Therefore:

✅ Fully covers BERT (2018) → GPT-3 (2020) → PaLM (2022) → DALL-E 2 (2022)
✅ Mentions LaMDA, BlenderBot 3, Retro, WebGPT
❌ Does not cover: GPT-4 (2023-03), the phenomenal popularity of ChatGPT, Claude, the Llama series, the phenomenal explosion of Stable Diffusion, in-depth RLHF methodology, InstructGPT, Constitutional AI
❌ Does not cover: Agent engineering, MCP, Multi-Agent Orchestration, tool use engineering

10.2 Connections to Other Wikis

Wiki	Focus on
AI Agent Technical Architecture Evolution 2025 (In-Depth Research Report)	2026-01 Industry Landscape (filling the gap after the book's 2023 endpoint)
Close Reading of 3 Agent Research Papers (2024-2025)	Close Reading of 3 Key Papers
Karpathy · Software in the era of AI(Software 3.0)	Karpathy's Software 1.0/2.0/3.0 Paradigm (the "philosophized" expression of this book)
newtype · MCP Protocol and Ecosystem in Practice	MCP Protocol Ecosystem
newtype · RAG in Practice and Knowledge Base Evolution	RAG in Practice (Sections 3.4.5 + 6.1-6.2 of this book form its academic foundation)
Claude Code Best Practices (Cal Rueb · Anthropic Internal Talk)	Cal's internal talk — judgment on Agentic Search vs RAG; this book is the academic foundation of RAG
Wang Kai's Claude Code Terminal Workflow Methodology (Full Version)	Wang Kai's Claude Code Overview

10.3 Application Advice for Independent Developers

You don't need to read the whole book — you don't need to understand the math of self-attention from scratch. But Ch 2 at least skim through it to know the differences between BERT and GPT
Jump to Ch 6 to choose an application direction — QA / translation / summarization / dialogue / code, see which one matches your product
Ch 7 Multimodal section — see which combination of media + text presents your opportunity (e-commerce images, video education, medical imaging, DNA)
Ch 8 Economic applications + harms — useful for explaining to your boss / writing PRDs / writing BD proposals
Important reference: Bommasani et al. 2021 ("On the Opportunities and Risks of Foundation Models" arXiv:2108.07258) — the "bible" of the entire Stanford CRFM, repeatedly cited in this book

History

2026-05-17: Phase 4 ingest is a complete close reading of 06_Foundation_Models_NLP.pdf(arXiv:2302.08575, Springer open textbook, 436 pages). All 8 chapters are covered: Ch1 Classical Foundations + Ch2 Three Major PLMs + Ch3 Improvement Directions + Ch4 Evaluation + Ch5 IE + Ch6 Text Generation + Ch7 Multimodality + Ch8 Conclusion. It forms the "Foundation Models Academic Foundation" layer of the newtype knowledge base.