When to open this: You want to see how Anthropic itself builds agents / modifies Claude Code / designs the MCP ecosystem, rather than listening to third-party interpretations. This wiki compresses 25 posts from the anthropic.com/engineering area (from 2024 through 2026-05) into 6 major themes organized by "question → answer", plus a 1-sentence index for all 25 posts.
One-sentence core:"The Anthropic perspective" = first-party practice and standard-pushing running in parallel — it both covers "how we do it ourselves" (building effective agents / harness design / internal Claude Code decisions) and pushes open standards (MCP / Agent Skills / Desktop Extensions / Managed Agents) so the ecosystem can plug in. Reading third-party (Wang Kai / Lao Huang / Lenny) wikis gives the "user perspective"; this wiki gives the "original manufacturer perspective",and you must read both perspectives together **to avoid bias.
Cross-check map: This wiki ↔ AI Agent Architecture and Multi-Agent Orchestration (cross-author topic synthesis)(6-perspective synthesis) ↔ Claude Code Workflow (cross-author topic synthesis)(6-perspective view of Claude Code) ↔ AI Agent Evaluation Methodology (Anthropic engineering team demystifying-evals deep dive)(Phase A anchor wiki / agent eval single-article deep dive) ↔ Prompt / Context Engineering (cross-author topic synthesis)(prompt + context engineering evolution).
0. Categorized overview of the 25 posts
Grouped by the problem they solve:
| Category | Number of posts | Core question | Which section of this wiki to read |
|---|---|---|---|
| A. Frameworks and definitions | 2 | "What is an agent? How is it different from a workflow?" | §1 |
| B. Context engineering paradigm | 2 | "What comes after prompt engineering?" | §2 |
| C. Harness design | 4 | "How do you design agents that can run single-shot / run over long durations / run with multiple agents in parallel?" | §3 |
| D. Tool design and MCP | 4 | "How can an agent use 1000 tools without blowing up its context?" | §4 |
| E. Agent Skills (new primitive) | 1 | "How do you add domain expertise to an agent without polluting its default behavior?" | §5 |
| F. Claude Code engineering | 5 | "How does Claude Code internally maintain quality + safety + user experience?" | §6 |
| G. Eval methodology | 5 | "How do you evaluate agents scientifically? (For in-depth coverage see AI Agent Evaluation Methodology (Anthropic engineering team demystifying-evals deep dive))" | §7 |
| H. Other | 2 | think tool has been replaced by extended thinking / Contextual Retrieval (RAG optimization) | §8 |
1. Frameworks and definitions: Workflow vs Agent (official Anthropic version)
From Building Effective AI Agents(published in late 2024; a large number of subsequent internal blogs cite this as canonical).
1.1 Core distinction
Anthropic collectively refers to LLM-related systems as agentic systems(agent systems), and internally divides them into two categories:
| Workflow(workflow) | Agent(agent) | |
|---|---|---|
| English definition | "Systems where LLMs and tools are orchestrated through predefined code paths." | "Systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks." |
| Chinese | The code path is pre-written; the LLM + tools run according to the pre-defined flow | The LLM itself decides what step to take next and which tools to call, and controls the entire process on its own |
| Who holds the initiative | The engineer | The model |
| When to use | The task has a clear structure, the flow can be predefined, and predictability is required | The task is open-ended, high in complexity, requires flexibility, and you are willing to accept uncertain performance + higher cost |
Key judgment:
"agentic systems often trade latency and cost for better task performance"
Translation: Agent systems typically trade latency and cost for better task performance.
Meaning: agents are not "more advanced"; they are the option that is "more expensive, slower, but more flexible". The discipline Anthropic gives:
| When to use a simple LLM call | When to use a workflow | When to use an agent |
|---|---|---|
| The answer can come out directly | The task can be broken into clear steps, and predictability / consistency must be maintained | The number of steps and paths are unpredictable, and human intervention at scale is impractical |
1.2 The 5 major workflow / agent patterns
The "building blocks" Anthropic provides, from simple to complex:
| # | Pattern | Chinese | One sentence | When to use |
|---|---|---|---|---|
| 1 | Prompt chaining | Prompt chaining | Break the task into consecutive steps, with an independent LLM call per step; the output of the previous step = the input of the next step | The task can be cleanly decomposed / there are sequential dependencies between steps |
| 2 | Routing | Routing | Classify the input → process it with different prompts / models | Categories are clear + each category requires a different response |
| 3 | Parallelization | Parallelization | Run multiple LLM calls simultaneously and merge the results (sectioning to split tasks / voting to run the same task multiple times) | Subtasks are independent + speed is sensitive |
| 4 | Orchestrator-workers | Orchestrator + workers | One "orchestrator" LLM breaks down the task and dynamically generates subtasks for "worker" LLMs | Subtasks cannot be enumerated in advance |
| 5 | Evaluator-optimizer | Evaluator + optimizer | One LLM produces output, another LLM evaluates it, forming a feedback loop | There are clear evaluation criteria + iteration can produce obvious improvements |
+ Agent(Most complex): The LLM plans its own loop, with tool calls + environment feedback, until the task is completed or a stop condition is met.
Anthropic repeatedly emphasizes:"successful implementations weren't using complex frameworks or specialized libraries. Instead, they were building with simple, composable patterns."(successful implementations do not rely on complex frameworks, but on simple, composable patterns.) — this is consistent with the cross-check against Wang Kai's Claude Code Terminal Workflow Methodology (Complete Version) Wang Kai's principle of "keep the tool stack as minimal as possible".
1.3 Comparison with third-party perspectives
| Source | What "Agent" refers to |
|---|---|
| Anthropic (this section) | A system with dynamic autonomous decision-making + tool calls + self-control of the process |
| Huang Yihe's post-level judgment(newtype · Agent Architecture and Multi-Agent Orchestration) | A dynamic entity with perception + autonomous decision-making + action execution (similar) |
| Wang Kai's literal usage(Wang Kai's Practical Agent Engineering) | "Open a new terminal to run Claude" = 1 agent |
| Academic school(Close Reading of 3 Agent Research Papers (2024–2025)) | More Agents Is All You Need / # agents replaces # parameters |
Biggest consensus: agent ≠ workflow; the two are used in combination (Anthropic promotes 5 patterns + 6 categories of agent combinations; Lao Huang calls this "Agentic Workflow"); there is no production deployment of a "pure agent".
2. Context engineering: The new paradigm after Prompt Engineering
From Effective context engineering for AI agents(2025).
2.1 Paradigm upgrade
"After a few years of prompt engineering being the focus of attention in applied AI, a new term has come to prominence: context engineering."
Translation: After several years of focusing on prompt engineering, a new term has emerged: context engineering.
Relationship between the two:
| Prompt Engineering | Context Engineering | |
|---|---|---|
| Focus | "Finding the right words / sentences" | "Configuring what goes into the entire context window" |
| Unit | A segment of prompt | The entire token state before a call |
| Key question | "How do I phrase this instruction?" | "What kind of context configuration best elicits the behavior I want?" |
| Way of thinking | thinking in words | thinking in context — holistically considering what behavior the current token state will produce |
2.2 What Context is (expanded definition)
Context = all tokens at the time the LLM samples, including:
- System prompt
- User message
- Conversation history
- Tool definitions
- Tool return values
- Retrieval results (RAG)
- Memory
- Intermediate reasoning process
Engineering problem: Under the inherent constraints of LLMs (limited context window + more tokens burn more cost + attention decay in long contexts),maximize the "signal density" of these tokens.
2.3 Why prompt engineering is no longer enough
Once an agent is running, context is dynamically growing — every round of tool calls stuffs more things into it.Simply optimizing the system prompt is not enough; you also need to manage:
- The format / length of tool return values
- Which intermediate results to keep / which to discard
- How to persist and retrieve cross-session "memory"
- How to perform compaction when the context window is full
All of these fall within the scope of context engineering.
Practical pointer: See Prompt / Context Engineering (cross-author topic synthesis) for cross-author synthesis. Karpathy was already shouting "Context Matters" back in 2024; the frameworks from Lao Huang / Wang Kai / Anthropic represent the same wave of cognitive convergence.
3. Harness design: 4 categories of scenarios
Harness(framework / scaffolding) = the infrastructure that lets an LLM run as an agent (receiving input, orchestrating tools, managing context, recording transcripts).
Anthropic divides agent running patterns into 4 categories, each with different harness design:
3.1 Single-session agent (single context window)
When: The task can be completed within 1 context window.
Representative harnesses: Claude Code default mode / Cowork / Cursor.
Core difficulties: context management + tool selection + user interaction.
3.2 Long-running single agent (across multiple context windows)
From Effective harnesses for long-running agents + Harness design for long-running application development.
When: The task runs for hours or even days, and cannot fit in 1 context window.
Core difficulty:When a new session starts, it has no memory of the previous session. Anthropic uses a vivid analogy:
"Imagine a software project staffed by engineers working in shifts, where each new engineer arrives with no memory of what happened on the previous shift."
Translation: Imagine a software project where engineers work in shifts, but every new person who comes on shift knows nothing about what happened on the previous shift.
Anthropic's two-part solution:
| Role | Task |
|---|---|
| Initializer agent(initialization) | On the first run,configure the environment —— prepare the working directory, install dependencies, set everything up so the "successor" can start working immediately |
| Coding agent(per shift) | After each new session starts,make incremental progress + leave clear handoff materials(progress notes / TODO / tests) for the next shift to use |
Corrected analogy: it is not "every new person has total amnesia", but "every new person can see the markdown notes the previous shift stuck on the whiteboard".
Real case: Carlini used 16 parallel Claude instances to write a 100k-line Rust-based C compiler, ran nearly 2,000 Claude Code sessions, and burned $20K in API costs; see #36-real-event-carlini-16-claude-writes-c-compiler.
Advanced: harness-design-long-running-apps borrows the GAN idea — one agent writes code, another agent acts as a "judge" to evaluate frontend design (subjective quality) + test usability (verifiable), in GAN-like iterations.Suitable for long-running tasks where "the output is subjective quality such as design / copywriting".
3.3 Multi-Agent Parallelism (Same Project)
From How we built our multi-agent research system + Building a C compiler with a team of parallel Claudes.
When to use: tasks where subtasks can be parallelized(researching different materials, writing different modules) + results need to be merged.
Design of Anthropic Research system(the Research feature on Claude.ai):
[规划 agent] —— 接用户问题,拆出 N 个并行子查询
↓
[N 个并行子 agent] —— 各自跑 web 搜索 / Workspace 检索
↓
[合并 agent] —— 收集所有结果,综合 + 写报告
Key lessons(repeatedly emphasized by Anthropic):
- Coordination cost is a hidden overhead: passing context between multiple agents burns tokens extremely fast
- Evaluation difficulty surges: you are evaluating "the entire agent team", not a single agent
- Reliability issues: a single agent has a 5% failure rate; an N-agent team has a failure rate of 1−(0.95)^N
3.4 Managed Agents (decoupling brain from hands, new formulation in 2026)
From Scaling Managed Agents(2026).
Core insight: previous harness designs were all based on the assumption that "Claude can currently do X and cannot do Y".But models iterate quickly, and assumptions will go stale(become outdated).
Typical example:"context anxiety"(context anxiety) —— when Claude Sonnet 4.5 feels the context is almost full, it will "rush to wrap up" and end the task prematurely. The harness team designed "compaction" (compressing history) to work around this.But Sonnet 4.6 no longer had this problem at all, and the old harness became a burden.
Managed Agents approach:Decouple the "brain" (model) from the "hands" (execution capabilities), so that the harness automatically upgrades along with model capabilities, without needing to rewrite the harness every time the model upgrades.
Corresponding product: platform.claude.com/docs/en/managed-agents, Claude automatically manages long-running complexity.
3.5 Quick Reference for 4 Types of Harness
| Scenario | Harness Name | Representative | Complexity |
|---|---|---|---|
| Single session | Standard agent harness | Claude Code default | ⭐ |
| Long-running single agent | initializer + coding agent | Agent SDK long-running template | ⭐⭐⭐ |
| Multi-agent parallel | planner + workers + synthesizer | Claude.ai Research | ⭐⭐⭐⭐ |
| Decoupled managed | Automatically follows model upgrades | Managed Agents service | ⭐⭐ (from the user's perspective) |
3.6 Real Event: Carlini's 16 Claude Instances Writing a C Compiler
From Building a C compiler with a team of parallel Claudes(2026, author Nicholas Carlini, Safeguards team).
Task: 16 parallel Claude instances writing a Rust-based C compiler from scratch, with the goal of compiling the Linux kernel.
Results:
- Nearly 2,000 Claude Code sessions
- Burned $20,000 in API costs
- Produced 100,000 lines of compiler code
- Can compile Linux 6.9, runs on x86 / ARM / RISC-V three platforms
- Code is open source
anthropics/claudes-c-compiler
Design principles Carlini learned:
- Write tests to keep agents from going off track(under unsupervised conditions, tests are the only "right/wrong signal")
- Structure tasks so that parallel multi-agents do not collide
- Where the ceiling is: the "upper bound" of agent team scaling is starting to show —— you cannot handle arbitrarily large projects by infinitely piling on agents
4. Tool Design and MCP: From "Stuffing the Context" to "Loading on Demand"
From:
- Code execution with MCP
- Writing effective tools for AI agents—using AI agents
- Introducing advanced tool use on the Claude Developer Platform
- Claude Desktop Extensions
- The "think" tool(already superseded by extended thinking, see §8)
4.1 Core Contradiction: Number of Tools vs Context Budget
When an agent connects to 100 MCP servers, it has hundreds to thousands of tools.Stuffing all of them into context = uses 50,000+ tokens before the agent even reads the user's question(Anthropic measured data).
Anthropic's solution:Let tools be imported on demand like a codebase, rather than stuffing all definitions in at once.
4.2 Code Execution with MCP (Key Innovation in 2025)
Approach: the agent does not "call tools in natural language", but instead writes code to call tools.
| Natural language invocation | Code invocation | |
|---|---|---|
| Form | Each time the LLM says "I want to call tool X with parameter Y" | The LLM writes a segment of Python / JS code that calls the tool inside it |
| 1 tool round trip | 1 inference | 0 extra inferences (called directly in code) |
| Intermediate results | All stuffed into context(whether used or not) | Only the final result is stuffed in, intermediate variables exist while the code is running |
| Error handling | Model sees error → retries → says it again | Handled at the code layer via try/except |
Benefit: for complex agent tasks with hundreds of tools / thousands of turns,token consumption can be compressed to below 10%.
Practical implementation: Code Execution + MCP work together; the agent executes code in a sandbox, calls tools exposed by MCP, and only returns key results to the main LLM.
4.3 Advanced Tool Use (Discovery + On-Demand Loading)
From the advanced-tool-use article. Two new primitives:
| Primitive | What it solves |
|---|---|
| Tool discovery on-demand(on-demand discovery) | Do not list all tools upfront; the agent calls a "find tools" tool, which returns the relevant N tools based on the current task |
| Calling tools from code(in-code invocation) | Expose tools as code APIs; the agent writes code to compose and call them —— same as §4.2 |
4.4 5 Principles for Writing Effective Tools (meta-pattern)
From writing-tools-for-agents. Anthropic itself used Claude Code to iterate on tool descriptions and arrived at 5 principles:
| Principle | Content |
|---|---|
| 1 | Choose the right tools to implement(don't add tools that shouldn't be implemented —— the more tools, the more confused the agent gets) |
| 2 | Namespacing(Namespaces) clearly delineate tool boundaries |
| 3 | Return meaningful context(not just data, include hints on "how the agent should handle it") |
| 4 | Token-efficient(return values are concise) |
| 5 | Tool descriptions + specifications also need prompt engineering(test, tune, iterate) |
Meta insight:Let Claude Code itself participate in revising your tool descriptions —— give it "the tool + the failure transcript", let it suggest how to revise the description, and iterate to convergence.
4.5 Desktop Extensions(DXT → MCPB)
From the desktop-extensions article.
Problem: installing a local MCP server is too complicated (developer tools / editing config / installing dependencies).
Solution:One-click installation package, file extension .dxt (later changed to .mcpb = MCP Bundle, 2025-09). Users double-click to install, and Claude Desktop automatically registers the MCP server.
Form:.mcpb = zip package, containing:
- MCP server code
manifest.json(metadata / permissions / entry point)- Dependencies (packaged so no npm install is needed)
For developers: opens up the MCP ecosystem from "a small number of geeks who know how to edit config files" to "all Claude Desktop users".
4.6 Synthesis: MCP Evolution Timeline
| Time | Milestone |
|---|---|
| 2024-11 | MCP launched, open standard |
| 2025-H1 | Community built thousands of MCP servers, SDKs cover mainstream languages |
| 2025-mid | Anthropic itself hit pitfalls —— too many tools caused context explosion |
| 2025-mid | Code execution with MCP released (solved context explosion) |
| 2025-Q3 | Desktop Extensions released (.dxt one-click install) |
| 2025-09 | .dxt changed to .mcpb standard naming |
| 2026-Q1 | Advanced tool use on Developer Platform —— on-demand discovery + in-code invocation |
MCP has become the industry de-facto standard, see MCP Protocol Ecosystem (Cross-Author Topic Synthesis) cross-author wiki synthesis.
5. Agent Skills (New Primitive in 2025)
From Equipping agents for the real world with Agent Skills. Published as an open standard starting 2025-12-18(agentskills.io), cross-platform portable.
5.1 What Is a Skill (Anthropic Official Definition)
"organized folders of instructions, scripts, and resources that agents can discover and load dynamically to perform better at specific tasks"
Translation: an organized folder containing instructions, scripts, and resources that an agent can dynamically discover and load to perform better on specific tasks.
Structure(same as Claude Skill Engineering (Cross-Author Topic Synthesis) §1):
my-skill/
├── SKILL.md # 主指令(YAML frontmatter + description + 步骤)
├── references/ # 参考资料 / 范例 / 数据
└── scripts/ # 可执行脚本(Python / Shell)
Life analogy (given by Anthropic itself):"Building a skill for an agent is like putting together an onboarding guide for a new hire."(writing a skill for an agent is like writing onboarding documents for a new employee.)
5.2 Why Is a New Primitive Needed?
The previous 3 ways to add domain expertise to an agent:
| Method | Drawbacks |
|---|---|
| Stuff into system prompt | The more you stuff, the longer it gets → pollutes default behavior + context explosion |
| Custom fine-tuning | Expensive / slow / not portable |
| MCP server | Heavy + suitable for "calling external systems" rather than "installing domain knowledge" |
Positioning of Skill:Lightweight, dynamically loaded, portable —— does not pollute the agent's default behavior at ordinary times, and is only loaded when the trigger scenario occurs.
Key capability:discover and load dynamically —— the agent itself finds the skill to use based on the task, loads it into context, and releases it after finishing the task.
5.3 Boundary with MCP
| Skill | MCP | |
|---|---|---|
| What is given to the agent is | Documentation + scripts + examples (domain knowledge) | Tool interfaces (calling external systems) |
| Trigger | Agent decides to call on its own | Agent sees it in tool descriptions → selects it |
| Loading | Dynamic import + unload | Connected at startup, stays connected long-term |
| Cross-platform | Yes (open standard) | Yes (open standard) |
| Content | Mainly SKILL.md natural language + scripts | APIs exposed by server code |
Complementary, not replacements: in real projects both are used —— Skills hold "domain workflows", MCP holds "external system integrations".
5.4 Comparison with Third-Party Perspectives
| Source | Understanding of "Skill" |
|---|---|
| Anthropic (this section) | Official schema: SKILL.md + references/ + scripts/, dynamically loaded |
| Lao Huang (newtype Skill System in Practice (Super Analyst v1/v2 + Super Writer)) | Super Analyst v1/v2 + Super Writer three-piece set practice |
| Wang Kai (Wang Kai IFS Internal Family Systems AI-Assisted Practice + Claude Skill Definition) | IFS internal work skill hands-on practice (SKILL.md + safety boundaries + trigger conditions) |
| Cross-author synthesis(Claude Skill Engineering (Cross-Author Topic Synthesis)) | Full integrated perspective |
6. Claude Code Engineering: How Anthropic Does It Themselves
From:
- Best practices for Claude Code
- Claude Code auto mode: a safer way to skip permissions
- Making Claude Code more secure and autonomous with sandboxing
- A postmortem of three recent issues
- An update on recent Claude Code quality reports
6.1 The Sandboxing + Auto Mode Triangle
Claude Code defaults to a permission model: except for safe commands such as "echo / cat", all write operations / commands require user approval.User pain point: excessive manual approvals → approval fatigue(approval fatigue) → clicking yes to everything without even looking.
The 3 modes provided by Anthropic:
| Mode | Safety | Maintenance cost | Autonomy | Suitable for |
|---|---|---|---|---|
| Manual approval(default) | Medium | Low | Low | Ordinary projects |
| --dangerously-skip-permissions(skip) | Extremely poor | Zero | High | Not recommended, ad-hoc experiments |
| sandbox | High | High (must be configured for every new capability) | Medium | Strictly isolated scenarios that do not require network / host access |
| 🆕 Auto mode(2025) | High | Low | High | Most daily use — sandbox by default + privilege escalation when necessary |
Design of Auto mode: make sandboxing the default,sensitive operations trigger an "elevation dialog"(instead of asking for every write operation), compressing the "93% of manual approvals that get a yes anyway" into "asking once only when there is real danger".
Anthropic internal data:sandboxing safely reduces permission prompts by 84%.
6.2 Postmortem Culture: Public + Detailed + Fixes
In 2025–26, Anthropic publicly published at least 2 large-scale Claude Code quality regression postmortems:
6.2.1 Aug–Sep 2025 postmortem (3 infrastructure bugs)
Symptom: starting in August, a large number of users reported that "Claude got dumber".
Truth:3 independent infrastructure bugs intermittently degraded quality,unrelated to load / time of day.
Discipline stated publicly by Anthropic:
"We never reduce model quality due to demand, time of day, or server load."
Translation: We will never reduce model quality because of load, time of day, or server pressure.
Transparency commitment:no secret throttling. Every instance of "getting worse" must be a bug, and must be root-caused and fixed.
6.2.2 April 2026 Claude Code regression (three things)
3 independent changes caused Claude Code quality to drop:
| Time | What was changed | Side effect | Fix |
|---|---|---|---|
| 3-4 | Claude Code reasoning effort default changed from high to medium |
Users perceived it as dumber | Reverted back to on 4-7 high |
| 3-26 | (specific changes are in the article, similar in nature to Aug–Sep) | Quality drop | Subsequent fixes |
| ... | ... | ... | All fixed in v2.1.116 on 4-20 |
Internal judgment revealed by Anthropic:"Changing the reasoning effort default to medium was the wrong tradeoff" — a latency problem that makes the UI appear frozen should not be solved by "making the model dumber"; UI feedback should be fixed instead.
6.3 Key Points of the Best Practices Document
The best-practices article is the core of Claude Code's official manual, covering:
- 3 levels of CLAUDE.md (project / user / subdirectory)
- 3 tiers of Permission modes
- /clear /compact context management
- 3 tiers of Think Hard reasoning
- Sub-Agent boundaries + use cases
- Hooks triggers
- MCP selection
For a detailed cheat sheet see Claude Code Best Practices (Cal Rueb · Anthropic Internal Talk)(Cal Rueb talk version, cross-checked against official docs) + Wang Kai's Claude Code Terminal Workflow Methodology (Complete Version)(Wang Kai practical version) + Claude Code Workflows (Cross-Author Thematic Synthesis)(cross-author synthesis).
6.4 Observations on Anthropic Engineering Culture
| Practice | Where it comes from |
|---|---|
| Public postmortems + No secret quality degradation | Two postmortem articles |
| eval-driven development(write evals first to define future capabilities + wait for model upgrades) | demystifying-evals (Phase A anchor wiki) |
| Use Claude Code to modify itself(meta-engineering: use agents to help you modify agents) | writing-tools-for-agents / harness-design |
| Push open standards(MCP / Skills / Agent SDK are all open source) | All of the above |
7. Eval Methodology (single wiki index)
5 articles constitute Anthropic's own eval playbook.All details are incorporated into AI Agent Evaluation Methodology (Anthropic Engineering Team demystifying-evals deep dive) + the upcoming Phase B LLM / AI Agent Evaluation Methodology (Cross-Author Thematic Synthesis: Anthropic + Academia + Third-Party Practice); this section provides only a one-sentence index:
| Article | One sentence | Where to jump to |
|---|---|---|
| Demystifying evals for AI agents | 8 core concepts + 3-category grader matrix + 8-step roadmap + pass@k vs pass^k mathematics | AI Agent Evaluation Methodology (Anthropic Engineering Team demystifying-evals deep dive) |
| Designing AI resistant technical evaluations | Recruiting take-home test: Opus 4 can already ace it instantly, Opus 4.5 is even stronger; tests must be redesigned as models evolve | (recommended standalone deep read in this wiki) |
| Quantifying infrastructure noise in agentic coding evals | On Terminal-Bench 2.0,simply changing infrastructure configuration alone can cause a 6 percentage point difference (p<0.01) — a few points of difference on leaderboards can easily be infra noise rather than model differences | (deep read recommendation) |
| Eval awareness in Claude Opus 4.6's BrowseComp performance | First recorded instance in history: on the BrowseComp test, Claude Opus 4.6 suspected it was being tested, reverse-engineered and identified the benchmark, located the answer key and decrypted it; 2 cases out of 1,266 questions | (deep read recommendation) |
| Claude SWE-Bench Performance | Upgraded Claude 3.5 Sonnet scores 49% on SWE-bench Verified; this article discloses details of the agent harness design used by Anthropic | (developer reference) |
7.1 Three Key Insights (called out separately)
- Infrastructure noise > model differences: comparisons where top leaderboard entries differ by 1–2 percentage points are often due to infra configuration differences, not the model itself.Do not draw serious conclusions from such 1–2 point small differences.
- Models already "know they are being evaluated": Opus 4.6 can identify BrowseComp features + reverse-engineer answers.Future evals must be AI-resistant — otherwise you are not evaluating capability, but "whether the model has seen this benchmark".
- Take-home test lifespan is getting shorter and shorter: for Anthropic's performance engineer take-home, every time a new model (Opus 4 / 4.5 / 4.6) comes out,the questions must be rewritten. The entire recruiting evaluation industry is being disrupted in 2024–2026.
8. The Other 2 Articles
8.1 think tool (superseded by extended thinking)
The article was published in early 2025, introducing a tool specifically for Claude to "stop and think", placed in the tool list; when the agent calls it, it enters structured reasoning.
Official update after 2025-12:
"Extended thinking capabilities have improved since its initial release, such that we recommend using that feature instead of a dedicated think tool in most cases."
Translation: extended thinking has been substantially improved since its launch, and for most scenarios it is recommended to use extended thinking instead of a dedicated think tool.
Implication: the article becomes a historical archive. Unless you have very special engineering reasons, do not hand-roll a think tool; use the platform's extended thinking.
8.2 Contextual Retrieval (RAG optimization)
From the contextual-retrieval article (late 2024).
Core problem: traditional RAG splits documents into chunks, after which chunks lose the context of the original document — recall is harmed during retrieval.
Anthropic's solution:
| Sub-technique | How it works |
|---|---|
| Contextual Embeddings | When embedding, provide the chunk together with the chunk's position in the original document / a context summary |
| Contextual BM25 | BM25 keyword retrieval also carries context |
Benefits:
- Used alone → failed retrievals ↓ 49%
- Combined with reranker → failed retrievals ↓ 67%
Suitable for: you are building a RAG application and are troubled by "insufficient recall". For detailed comparisons see RAG / Knowledge Base Engineering (Cross-Author Thematic Synthesis) §4 The 5 Major RAG Optimizations.
9. Anti-patterns and Real Incidents (Complete Collection)
From the "candid failures + customer lessons" across this batch of 25 articles:
| # | Anti-pattern / incident | Source |
|---|---|---|
| 1 | Relying on complex frameworks / specialized libraries to solve agent | building-effective-agents repeatedly warns "successful implementations are simple + composable" |
| 2 | Agent calls tools individually in natural language every time | code-execution-with-mcp: burns through 50,000+ tokens |
| 3 | Stuffing all tool definitions into the context window | advanced-tool-use: must discover + load on demand |
| 4 | Claude Code defaulting reasoning effort to medium | April postmortem: wrong tradeoff, users felt it "got dumber" and it was immediately reverted |
| 5 | Secretly throttling models / degrading quality | Aug–Sep postmortem shows Anthropic publicly commits not to do this, and every quality drop is traced to a specific bug |
| 6 | Not rewriting take-home tests as models upgrade | AI-resistant-evaluations: test lifespan compressed from "one year" to "one model generation" |
| 7 | Drawing serious conclusions from 1–2 point small gaps on SWE-Bench | infrastructure-noise: same model on different infra differs by 6 points |
| 8 | Assuming LLMs do not know they are being tested | eval-awareness-browsecomp:Opus 4.6 proactively identified BrowseComp and cracked it |
| 9 | Not re-auditing harnesses over time | managed-agents: assumptions go stale; Sonnet 4.5's "context anxiety" is gone in 4.6, old harnesses become dead weight |
| 10 | --dangerously-skip-permissions using for daily use | claude-code-auto-mode: sandboxing + auto mode is safer |
| 11 | Clicking yes to 93% of manual approvals(approval fatigue) | claude-code-sandboxing provides measured numbers |
| 12 | RAG chunks not carrying context | contextual-retrieval: failed retrieval rate reduced by 49% |
10. One-Sentence Index of the 25 Articles (for later grep)
| # | Article | One sentence |
|---|---|---|
| 1 | building-effective-agents | Workflow vs Agent + 5 patterns (prompt chaining / routing / parallelization / orchestrator-workers / evaluator-optimizer) + agent; repeatedly emphasizes "simple + composable > complex framework" |
| 2 | effective-context-engineering-for-ai-agents | Paradigm shift from prompt engineering → context engineering; "thinking in context" considers token state holistically |
| 3 | multi-agent-research-system | Claude.ai Research feature architecture: planner agent → N parallel sub-agents → synthesizer; warning about coordination cost + evaluation complexity |
| 4 | effective-harnesses-for-long-running-agents | Two-piece set for long-running agents: initializer agent sets up the environment + coding agent leaves handoff materials each shift; analogous to "shift engineers" |
| 5 | harness-design-long-running-apps | Dual track of subjective quality + verifiability; GAN-inspired: agent A writes + agent B reviews, iterating to convergence |
| 6 | managed-agents | "Decoupling brain from hands"; warning that assumptions go stale (Sonnet 4.5 context anxiety → gone in 4.6) |
| 7 | building-c-compiler | Carlini ran 16 parallel Claude instances to write a 100k-line Rust C compiler; 2000 sessions / $20K / compiles Linux 6.9 and runs on x86+ARM+RISC-V |
| 8 | writing-tools-for-agents | 5 principles: choose the right tools + namespacing + return meaningful context + token efficiency + prompt-engineer tool descriptions;Use Claude Code itself to modify its own tool descriptions |
| 9 | equipping-agents-for-the-real-world-with-agent-skills | Agent Skills open standard (published 2025-12-18); analogy to an "onboarding guide for a new hire";discover + load dynamically |
| 10 | code-execution-with-mcp | Agents invoke tools via code (not natural language), token consumption can be pushed below 10% |
| 11 | desktop-extensions | .dxt → .mcpb One-click installer, opening MCP setup from "geeks who can edit configs" to all Desktop users |
| 12 | advanced-tool-use | On-demand discovery + in-code invocation, unlimited tool library without blowing up context |
| 13 | claude-think-tool | think tool Already superseded by extended thinking, use the latter in most scenarios |
| 14 | contextual-retrieval | RAG improvements: Contextual Embeddings + Contextual BM25, failed retrieval ↓ 49% / with reranker ↓ 67% |
| 15 | claude-code-best-practices | Official Claude Code manual (full set of CLAUDE.md / Permission / Hook / Think / Sub-Agent / MCP) |
| 16 | claude-code-sandboxing | Sandboxing + 84% reduction in permission prompts, default safety + increased autonomy |
| 17 | claude-code-auto-mode | Auto mode: sandbox by default + elevation dialog, eliminating 93% of approval fatigue |
| 18 | a-postmortem-of-three-recent-issues | Three infra bugs in Aug–Sep 2025 collectively caused reports of "Claude getting dumber", public commitment made no undisclosed throttling |
| 19 | april-23-postmortem | April 2026 three Claude Code changes caused regressions (changing reasoning effort default to medium was a wrong tradeoff), fully fixed in v2.1.116 on 4-20 |
| 20 | AI-resistant-technical-evaluations | Anthropic's recruiting take-home test must be rewritten every time a new model comes out; Opus 4 → 4.5 → 4.6 wash out generation after generation |
| 21 | infrastructure-noise | On Terminal-Bench 2.0 the same model differs by 6 points across different infra; don't take small leaderboard gaps too seriously |
| 22 | eval-awareness-browsecomp | First recorded Opus 4.6 on BrowseComp suspected it was being tested + reverse-engineered the benchmark + found and decrypted the answer key |
| 23 | multi-agent-research-system (already listed #3) | (duplicate) |
| 24 | swe-bench-sonnet | Upgraded Claude 3.5 Sonnet to 49% on SWE-bench Verified, article publicly discloses the agent harness implementation |
| 25 | demystifying-evals-for-ai-agents | See AI Agent Evaluation Methodology (Anthropic engineering team demystifying-evals deep dive)(Phase A anchor wiki) |
11. Related wikis / cross-reference map
| Want to see | wiki | perspectives |
|---|---|---|
| AI Agent Architecture: 6-perspective synthesis(Workflow / Multi-Agent / Sub-Agent / 4 orchestration patterns / framework comparison) | AI Agent Architecture and Multi-Agent Orchestration (cross-author topic synthesis) | Cross-author |
| Claude Code: 6-perspective synthesis(Terminal window as Agent / Skills / MCP / engineering division of labor) | Claude Code Workflows (cross-author topic synthesis) | Cross-author |
| Claude Skill Engineering(Skill = three-piece set of SKILL.md + references + scripts) | Claude Skill Engineering (cross-author topic synthesis) | Cross-author |
| MCP Ecosystem(DOS→Win95 analogy / configuration pitfalls / n8n+MCP+Skill coordination) | MCP Protocol Ecosystem (cross-author topic synthesis) | Cross-author |
| Prompt + Context Engineering(5 elements of task decomposition / Karpathy Context Matters) | Prompt / Context Engineering (cross-author topic synthesis) | Cross-author |
| RAG Engineering(AnythingLLM → GraphRAG → Milvus → Obsidian+NotebookLM end state) | RAG / Knowledge Base Engineering (cross-author topic synthesis) | Cross-author |
| Cal Rueb Claude Code talk(Anthropic engineer's newtype channel version) | Claude Code Best Practices (Cal Rueb · Anthropic internal talk) | Single author (Anthropic) |
| Wang Kai hands-on Claude Code workflow | Wang Kai Claude Code terminal workflow methodology (complete version) | Single author (Wang Kai) |
| Wang Kai Agent hands-on + multi-browser matrix | Wang Kai Agent engineering hands-on / Wang Kai multi-Agent browser matrix architecture (v1.1 hands-on guide) | Single author (Wang Kai) |
| Anthropic agent eval single-article deep dive | AI Agent Evaluation Methodology (Anthropic engineering team demystifying-evals deep dive) | Phase A anchor wiki |
| Cross-author LLM evaluation methodology(Phase B new creation) | LLM / AI Agent Evaluation Methodology (cross-author topic synthesis: Anthropic + academia + third-party practice) | Cross-author (to be generated) |
12. Meta-information
Rewriting strategy: 25 articles reorganized by "problem solved" (not original blog order), compressed into 6 major themes + 25 one-sentence indexes. Each article appears 1–2 times in the wiki: once in a theme chapter as "idea condensation", once in the §10 index as a "one-sentence recap". Quotations from original text ≤ 3 sentences; long quotations are followed by *译:中文*.
Bilingual strategy(translation_status: bilingual):
- Chapter titles mix Chinese and English (product names / concept names / benchmark names are not translated)
- Core English terms (Workflow / Agent / Context Engineering / Harness / Skill / MCP) are given Chinese translations on first occurrence + a Chinese-English comparison table
- Anthropic original golden quotes retained +
*译:中文* - Chapter titles retain core English words (Workflow / Agent / MCP / Skill / Eval)
Cross-check principle: this wiki is the "first-party perspective" of anthropic.com/engineering and must be cross-checked against AI Agent Architecture and Multi-Agent Orchestration (cross-author topic synthesis) / Claude Code Workflows (cross-author topic synthesis) and other "cross-author syntheses" — whenever there is a disagreement between "Anthropic officially says X" vs "third parties say Y", state the disagreement explicitly, do not smooth it over.
Phase B progress: this article is the Phase B hub wiki. Next steps: write LLM / AI Agent Evaluation Methodology (cross-author topic synthesis: Anthropic + academia + third-party practice) (cross-author evals topic) + update 4 existing wikis (Claude Code Workflows (cross-author topic synthesis) / AI Agent Architecture and Multi-Agent Orchestration (cross-author topic synthesis) / Claude Skill Engineering (cross-author topic synthesis) / MCP Protocol Ecosystem (cross-author topic synthesis)) to add an "Anthropic Engineering Team First-Party Perspective" section.
Sources and related materials
- https://www.anthropic.com/engineering/building-effective-agents
- https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
- https://www.anthropic.com/engineering/multi-agent-research-system
- https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
- https://www.anthropic.com/engineering/harness-design-long-running-apps
- https://www.anthropic.com/engineering/managed-agents
- https://www.anthropic.com/engineering/building-c-compiler
- https://www.anthropic.com/engineering/writing-tools-for-agents
- https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills
- https://www.anthropic.com/engineering/code-execution-with-mcp
- https://www.anthropic.com/engineering/desktop-extensions
- https://www.anthropic.com/engineering/advanced-tool-use
- https://www.anthropic.com/engineering/claude-think-tool
- https://www.anthropic.com/engineering/contextual-retrieval
- https://www.anthropic.com/engineering/claude-code-best-practices
- https://www.anthropic.com/engineering/claude-code-sandboxing
- https://www.anthropic.com/engineering/claude-code-auto-mode
- https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues
- https://www.anthropic.com/engineering/april-23-postmortem
- https://www.anthropic.com/engineering/AI-resistant-technical-evaluations
- https://www.anthropic.com/engineering/infrastructure-noise
- https://www.anthropic.com/engineering/eval-awareness-browsecomp
- https://www.anthropic.com/engineering/swe-bench-sonnet
- AI Agent Evaluation Methodology (Anthropic engineering team demystifying-evals deep dive)
- AI Agent Architecture and Multi-Agent Orchestration (cross-author topic synthesis)
- Claude Code Workflows (cross-author topic synthesis)
- Claude Skill Engineering (cross-author topic synthesis)
- MCP Protocol Ecosystem (cross-author topic synthesis)
- Prompt / Context Engineering (cross-author topic synthesis)
- Claude Code Best Practices (Cal Rueb · Anthropic internal talk)
- Wang Kai Claude Code terminal workflow methodology (complete version)
- Wang Kai Agent engineering hands-on
2026-10-18 addendum: quality regressions must cover idle recovery and model differences
What Builders included on October 2 was the April historical retrospective, not a new October incident. Re-verified against Anthropic's official April 23 retrospective: the vendor limited the scope of impact to Claude Code, Agent SDK, and Cowork, stating that the API and inference layer were unaffected; this is the official investigation conclusion.
In the same session, "works after resuming" does not equal correct context retention. The original design only cleaned old thinking records once after more than an hour of idle time, but the defect caused cleaning on every subsequent turn, leading to forgetting, repetition, unreasonable tool choices, and repeatedly breaking cache hits. Internal experiments and the way thinking was displayed also obscured some reproduction paths. For your own product, you can add a continuous test case of "resume after idle—continue multiple turns—receive new message during tool execution", while checking retained information and cost changes, rather than only verifying the first resumed reply. This is an applied derivation for this repository.
Controlling output length is also a behavior change. Officials stated that the system prompt that reduced verbose output exposed quality degradation after broader evaluation and has been rolled back. A prompt suitable for one model cannot be directly generalized to all models. Before launch, run per-model regression, and have internal testing use the same builds and configurations as users; when issues arise, troubleshoot by separating the three layers: default parameters, context handling, and prompts. This article does not generalize a single internal evaluation drop to an overall capability decline.
April regression timeline completion
The following fills in two placeholders in the earlier historical table; dates all refer to historical events around April 2026:
| Change date | Specific issue | Fix |
|---|---|---|
| March 26 | After idle, old thinking should have been cleaned only once, but the defect caused continuous cleaning on every subsequent turn, leading to forgetting, repetition, and cache invalidation | Fixed on April 10 in v2.1.101 |
| April 16 | The system prompt that shortened output, combined with other prompt changes, affected coding quality | Rolled back on April 20, aggregate fix in v2.1.116 |
Addition: Test idle-session recovery and model-specific quality
The October 2 Builders digest covered an April incident, not a new October failure. The official April 23 postmortem was checked again. Anthropic limited the affected products to Claude Code, the Agent SDK and Cowork and stated that the API and inference layer were unaffected. These are the company's investigation findings.
A resumed session that still works may nevertheless be losing important context. The intended optimization cleared older thinking once after more than an hour of inactivity. A bug instead kept clearing it on later turns, causing forgetfulness, repetition, unusual tool choices and repeated cache misses. Internal experiments and thinking-display behavior also concealed some reproduction paths. An editorial application is to test an idle session followed by several turns and a new message during tool execution, checking retained information and costs rather than only the first reply after resumption.
Output-length controls also change behavior. Anthropic reported that broader evaluations exposed a quality regression from a system instruction intended to reduce verbosity, and it reverted that instruction. A prompt suitable for one model should not automatically be applied to all models. Run regressions separately for each model and have internal testers use the same builds and settings as customers. Diagnose default parameters, context handling and prompts separately. A decline in one internal evaluation is not presented here as a general loss of model capability.
Completing the historical April timeline
The following fills the two placeholders in the earlier historical table. These dates describe events around April 2026.
| Change date | Problem | Resolution |
|---|---|---|
| March 26 | Older thinking was meant to be cleared once after inactivity; a bug cleared it on every subsequent turn, causing forgetfulness, repetition and cache misses | Fixed April 10 in v2.1.101 |
| April 16 | A brevity instruction interacted with other prompt changes and hurt coding quality | Reverted April 20; v2.1.116 incorporated the fixes |