Close Reading of 3 Agent Research Papers (2024–2025)

When to open: You want academic references for judgments about Agents / Multi-Agent / RAG. This wiki is a close reading of 3 papers / talks,not a survey(for the survey see AI Agent Technical Architecture Evolution 2025 (In-Depth Research Report)); it is the research itself.

A single thread runs through all three papers:the feasibility and practical path of using multiple Agents to replace a single stronger LLM.

1. "Finding the Scaling Laws of Agents" (CAMEL-AI Wendong Fan, 2025)

1.1 Talk Background

Speaker: Wendong Fan (core member of CAMEL-AI, Technical Lead at Eigent AI)
Host organization: CAMEL-AI.org — an open-source research organization dedicated to Multi-Agent Systems
Core thesis:Where is the scaling law for Agents?

1.2 Citing Marvin Minsky's 1986 insight

"Agents are mindless processes. Agent by itself can only do some simple things. Joining these agents in societies leads to true intelligence."

"The trick is that there is no trick. The power of intelligence stems from our vast diversity, not from any single, perfect principle." — Marvin Minsky, The Society of Mind, p. 308

This aligns with the Software 3.0 / Agentic Decade mindset — see Karpathy · Software in the era of AI(Software 3.0).

1.3 RL Agent vs LLM Agent

RL Agent(Traditional):

Agent ↔ Environment(actions / states / rewards)
Applications: Go / autonomous driving / games

LLM Agent(New):

Language as Input / Output
Both State and Action are expressed in natural language
Generalizability(no longer tied to a specific environment)

Three characteristics of LLM Agents:

Memory:Manages chat history + context window
Tools:Internal + external function calls
Step Loop: multi-turn interaction to complete a single task

1.4 Agent Scaling Law Hypothesis (core)

Traditional LLM Scaling Law	Agent Scaling Law (hypothesis)
# Parameters	# Agents?
Data	Environments?
Training	Memory / Interaction?

Key question:Is a Multi-Agent System with LLM Agents the new generation of scaling?

1.5 Timeline of LLM-based Agent Systems

The First:CAMEL (2023-03-21,Multi-Agent role-playing)
       AutoGPT (2023-03-30)
       Baby AGI (2023-04-03)
       ChatDev (2023-07-16)
       MetaGPT (2023-08-01)
       CrewAI (2023-11-15)

1.6 CAMEL Framework (NeurIPS 2023, arXiv:2303.17760)

Role-playing framework:

Idea
Role assignment
Task agents
Chat agents

Workforce (hierarchical architecture):

Hierarchical Architecture
Task Planning and Decomposition
Task Channel Communication
Robust Failure Handling

Key experiments:

CAMEL multi-Agent > single Agent on 200 tasks over 70%
GPT-4 evaluation is consistent with human evaluation

1.7 OWL(Optimized Workforce Learning)

GAIA benchmark Average score 58.18
Ranked first among open-source frameworks 🏆

1.8 CRAB(NeurIPS 2024)

Cross-environment Agent Benchmark for Multimodal LMs

Evaluating cross-environment capabilities of multimodal Agents
Task example: open Slack, navigate to a channel, summarize messages, send a text to a contact
arxiv:2407.01511

Simulating social media,up to 1 million Agents
Reproducing social science experiments
Exploring Agent social dynamics
arxiv:2411.11581

1.10 GraphRAG + CAMEL

Role-Playing Scraper for Report & Knowledge Graph Generation:

Autonomous Task Solving
Knowledge Generation
Scalable Solution

1.11 Agentic Data Generation (feeding back into training)

AI Society:50 assistant roles × 50 user roles × 10 tasks
Code:20 languages × 50 domains × 50 tasks
Generate data from GPT-3.5/4 → fine-tune Llama
On Hugging Face, 180+ models use CAMEL datasets (MosaicML MPT-30B-Chat, HuggingFaceM4/idefics2-8b)

1.12 Answer to the Scaling Law hypothesis

Dimension	CAMEL framework	OASIS framework	LOONG	OWL	DataGen	GraphRAG
# Parameters → # Agents	✓	✓
Data → Environments			✓	✓
Training → Evolution					✓	✓

Conclusion(Wendong Fan): CAMEL-AI is simultaneously exploring the scaling law of Agents along these 3 directions..

2. "More Agents Is All You Need" (Tencent ICML 2024)

2.1 Paper Information

Authors: Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, Deheng Ye (Tencent)
arXiv:2402.05120v1
Date:2024-02-03
The title pays homage to "Attention Is All You Need"

2.2 Core Thesis (one sentence)

Through the simplest sampling-and-voting, LLM performance scales with the number of Agents. and this is orthogonal to existing complex methods(CoT, ReAct, Reflection, Debate, etc. can all be stacked on top of our method for further improvement).

2.3 Sampling-and-Voting Method (Algorithm 1)

Require: Query x, number of samples N, LLM M (or LLM+method fM)

S ← ∅
for i = 1 to N do
    Generate sample s_i ← M(x)
    Add to S
end for
for each sample s_i in S:
    V(s_i) ← Σ sim(s_i, s_j)   # 累积相似度
end for
A ← argmax_{s_i} V(s_i)         # 选累积相似度最高的
return A

Similarity calculation:

Numerical answers: mathematical equivalence comparison
Multiple-choice questions: answer frequency
Code:BLEU score(sacreBLEU)

2.4 Core Experimental Results (Llama2-13B / 70B + GPT-3.5/4)

Tasks: GSM8K (arithmetic) / MATH (mathematics) / Chess / MMLU (general knowledge) / HumanEval (code)

Model	GSM8K Single	GSM8K +Ours	MATH	Chess	MMLU	HumanEval
Llama2-13B	0.35	0.59 (+0.24)	+0.06	+0.04	+0.09	+0.04
Llama2-70B	0.54	0.74 (+0.20)	+0.06	+0.01	+0.05	+0.09
GPT-3.5-Turbo	0.73	0.85 (+0.12)	+0.10	+0.04	+0.11	+0.06

Counterintuitive findings:Llama2-13B + 15 agents reaches the single-model level of Llama2-70B on GSM8K(59% vs 54%)! Llama2-70B + 15 agents approaches GPT-3.5-Turbo on some tasks!

2.5 Orthogonality with Other Methods (Table 3)

Method	GSM8K Standalone	+ Ours	Increment
CoT	0.39 → 0.56	(+0.17)	Large
ZS-CoT	0.40 → 0.61	(+0.21)	Large
SPP	0.19 → 0.42	(+0.23)	Large
Debate	0.38 → 0.48	(+0.10)	Medium
Reflection	0.36 → 0.59	(+0.23)	Large

Conclusion:All methods are further improved by sampling-and-voting.

2.6 Three Dimensions That Affect Gain Magnitude

Through controlled experiments, the authors isolated 3 orthogonal dimensions:

Dimension 1:Inherent Difficulty(inherent difficulty)

Increasing difficulty → gain first rises then falls
Gain is largest at moderate difficulty
When a task is too hard, the model's own reasoning ability is insufficient, and scaling cannot save it

Dimension 2:Number of Steps(number of steps)

More steps → larger gain
Errors accumulate at each step, and sampling-and-voting corrects errors at each step

Dimension 3:Prior Probability(prior probability)

Fewer options → better performance
If there are too many options (K = 32), split into multiple high-probability subtasks (K = 8) and then combine

2.7 Derived Methods (based on the 3 dimensions)

Step-wise sampling-and-voting:

Have the LLM output step by step
Use sampling-and-voting at each step to select the best
Prevent cumulative error
Compared to single-layer sampling-and-voting, another 15–42% improvement

Hierarchical sampling-and-voting:

Split high-difficulty tasks into multiple high-probability subtasks
Homogeneous combination(all GPT-3.5): 21% → 31%
Heterogeneous combination(GPT-3.5 finds intermediate answers + GPT-4 final): 35% → 47%
Key: use cheap models for simple subtasks, expensive models for hard subtasks,for optimal cost-effectiveness

2.8 Implications (for independent developers)

N small models + sampling-and-voting > 1 large model — holds in many scenarios
DeepSeek + multiple instances + voting may offer better cost-effectiveness than running GPT-4 alone
Most current Agent frameworks (CrewAI / AutoGen / LangGraph)have not yet integrated this simple trick
When writing Wrappers / SaaS products, sampling-and-voting can be treated as secret sauce

3. "ChatQA: Building GPT-4 Level Conversational QA Models" (NVIDIA 2024)

3.1 Paper Information

Authors:Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro(NVIDIA)
arXiv:2401.10225v2
Date:2024-01-23

3.2 Core Results

ChatQA-70B across 10 conversational QA datasets:

Average score 54.1454.14 > GPT-4 53.90 (slight edge)
Average score > GPT-3.5-Turbo 50.37 (significant)
No OpenAI synthetic data required

This is the first time an open-source 70B model has comprehensively matched GPT-4 on conv QA.

3.3 Core Method 1: Two-Stage Instruction Tuning

Foundation LLM
   ↓
Stage-1: Supervised Fine-tuning(SFT)
  数据:Soda + ELI5 + FLAN + Dolly + OpenAssistant
  目的:获得 conversational instruction-following 能力
   ↓
Stage-2: Context-Enhanced Instruction Tuning
  数据:NarrativeQA + DROP + Quoref + ROPES + SQuAD + NewsQA + TAT-QA + ConversationalQA
  目的:增强基于上下文 / RAG 的 QA 能力
   ↓
ChatQA

3.4 Core Method 2: Fine-tune Retriever to Replace Query Rewriting

Problem: Follow-up questions in Conversational QA (containing pronouns like "it") lack sufficient information, so direct retrieval is inaccurate.

Two solutions:

A. Query Rewriting(GPT-3.5-turbo rewriting) — requires extra LLM calls, high cost
B. Fine-tune Retriever(paper's method) — fine-tune a single-turn retriever on multi-turn data

Results (Dragon retriever):

Method	Avg Top-1 Recall	Avg Top-5 Recall
Dragon Standalone	46.29	73.09
Dragon + Query Rewriting(GPT-3.5)	54.46	80.13
Dragon + Fine-tune (paper's method)	52.72	80.67

Conclusion: The performance of the fine-tuned retriever is on par with GPT-3.5 query rewriting, but saves API call costs.

3.5 Core Method 3: Unanswerable Samples to Eliminate Hallucinations

Problem: When there is no answer in the document, LLMs tend to make things up (hallucination). Approach: Add 1.5K unanswerable samples to the instruction tuning data, and use "Sorry. I cannot find the answer based on the context""unanswerable"

Ablation:

Number of samples	QuAC Avg	DoQA Avg	Avg-CQA
1k unanswerable	80.89	72.88	54.16
1.5k unanswerable	80.76	73.74	54.14
2k unanswerable	80.82	73.38	53.86
2.5k unanswerable	78.81	72.93	53.78

Surprise:More unanswerable samples are not necessarily better; 1.5K is the optimal point.

3.6 The Importance of Stage-1 SFT

Removing Stage-1 and going directly to Stage-2:

Average score 54.08 → 52.18(drop of 1.9)
Conclusion:Establishing instruction-following ability first is critical for Stage-2

3.7 Helpfulness of Single-Turn QA Data for Multi-Turn QA

Adding single-turn QA data:

Average score improves by 1.83
Improvements also appear on table-based datasets (ConvFinQA / SQA / HybridDial)
Because single-turn QA improves the general ability of "extracting answers from context" general-purpose capabilities

3.8 Ablation on Retrieved Chunk Count and Order

Chunk count:

top-3: average 41.91
top-5: average 42.31(optimal)
top-10: average 40.71 (worse, "lost in the middle")

Chunk order:

Sequential / Reverse / Swing are close
Random shuffle is slightly worse
Conclusion:The paper's model is robust to chunk order

3.9 Human Evaluation Comparison (A/B Test ChatQA-70B vs GPT-4)

Average	Ours Win	Tie	GPT-4 Win
All	13.81%	69.09%	17.10%

Tie accounts for 69%, GPT-4 is slightly ahead by about 3.3% — 70B has basically caught up with GPT-4.

3.10 Implications (for self-built RAG systems)

No need to rely on OpenAI to synthesize data: 7K human-annotated + 7K GPT-synthesized, with comparable performance
Fine-tuning the Retriever yourself > rewriting with GPT-3.5: saves money and latency
Add 1.5K unanswerable samples = significantly reduces hallucinations
Stage-1 + Stage-2 two-stage approach = a single stage is insufficient
top-5 chunks is the sweet spot, don’t be greedy for more

Complementary to newtype · RAG in Practice and Knowledge Base Evolution — that piece is newtype’s post-level practical guide to RAG, while this section presents paper-level methodology.

IV. Aggregated Insights from the 3 Papers

4.1 Common Themes

All 3 papers answer one core question:How to achieve performance close to large models in a cheaper / smaller / more open-source way?**

Paper	Approach
Scaling Laws of Agents	Multiple Agents collaborating > a single stronger model
More Agents Is All You Need	Repeating the same model multiple times + voting > calling an expensive model once
ChatQA	70B + careful fine-tuning > using GPT-4

4.2 Summary of Counterintuitive Findings

Llama2-13B + 15 agents = Llama2-70B run alone (More Agents)
Fine-tuned retriever is on par with GPT-3.5 query rewriting (ChatQA)
1.5K unanswerable samples = significant hallucination reduction, but 2K+ makes it worse instead (ChatQA)
CAMEL multi-Agent > single Agent holds in 70%+ scenarios across 200 tasks (CAMEL)
Hierarchical sampling-and-voting using GPT-3.5 for intermediate steps + GPT-4 for final = optimal cost-effectiveness (More Agents)

4.3 Engineering Key Points

Practice	Source	Applicable Scenarios
Integrate sampling-and-voting into the Agent loop	More Agents	Any LLM task, especially reasoning tasks
Hierarchical mixing of cheap + expensive models	More Agents	High-difficulty multi-step tasks
Stage-1 + Stage-2 fine-tune	ChatQA	Self-built conversational AI
1.5K unanswerable samples	ChatQA	Preventing hallucinations in RAG systems
Multi-Agent role-playing	CAMEL	Complex tasks with clearly defined roles
Agentic Data Generation feeding back into training	CAMEL	Having large-model budgets but wanting to build small models

4.4 Connections to Existing Wiki Content

AI Agent Technical Architecture Evolution 2025 (In-Depth Research Report) — survey-level research report (this wiki is a paper-level close reading)
newtype · RAG in Practice and Knowledge Base Evolution — newtype’s post-level practical RAG guide (the ChatQA section here is paper-level)
newtype Model Evaluation, Fine-tuning, and Hardware Evolution — model evaluation (More Agents provides a comparison of "small model + multiple runs" vs "large model + single run")
AWS Agent Lecture Series 2025 (Compilation of 4 Talks) — AWS Agent lectures (this wiki covers papers; that one is industry lectures)
Wang Kai’s Multi-Agent Browser Matrix Architecture (v1.1 Practical Guide) — Wang Kai’s browser Agent matrix (this wiki covers academic theory; that one covers hands-on implementation)

History

2026-05-17: Phase 4 ingest created from 3 newtype PDFs (05/07/08). Fully covers the CAMEL-AI talk (Scaling Laws of Agents hypothesis + OWL/CRAB/OASIS/DataGen) + More Agents Is All You Need (sampling-and-voting + 3-dimensional gain analysis) + ChatQA (70B QA model + two-stage tuning + fine-tune retriever + unanswerable samples).