AI Engineering ／ Anthropic and Claude
AI Agent Evaluation Methodology (A Deep Dive into Anthropic Engineering Team's demystifying-evals)
A systematic framework for agent evaluation released by the Anthropic engineering team in 2026. It covers an 8-entry concept glossary, a 3-type grader matrix, the dual-track Capability vs Regression evaluation design, 4 sets of evaluation playbooks for different agent categories (coding / conversational / research / computer use), an 8-step roadmap for building evaluations from scratch, the mathematical intuition behind pass@k vs pass^k, the evaluation framework ecosystem, and common anti-patterns. Phase A anchors the wiki, and sets the density and bilingual style baseline for the anthropic-* series of content.