Huang Yihe's RAG (Retrieval-Augmented Generation) practice from 2024-03 to 2026-05. This wiki is based on curated highlight posts plus cross-bucket cross-referenced materials v1 version; the complete full set (341 posts) will be left for a later session to upgrade to v2.
When to open this:
- You want to build a knowledge base and are stuck on chunk size / embedding model selection
- You use RAG but the answers are always inaccurate, and you suspect it is a tool problem
- You have heard of GraphRAG / LightRAG / Contextual Retrieval and don't know whether they are worth using
- You want to understand why RAG is gradually being replaced by NotebookLM / Long Context models
0. RAG evolution timeline (Lao Huang's perspective)
graph LR
A["2024-03~05<br/>RAG 入门<br/>AnythingLLM 默认 LanceDB +<br/>nomic-embed-text"] --> B["2024-06~07<br/>GraphRAG 风暴<br/>但成本极高 $11/问<br/>(帖 360)"]
B --> C["2024-04~05<br/>Cohere Command R+<br/>(RAG 增强模型,帖 62)"]
C --> D["2025-02~03<br/>老黄做老黄项目<br/>用 Pinecone→pgvector<br/>(早期套壳)"]
D --> E["2025-04<br/>Milvus 本地 + MCP<br/>'数据库 + 模型分离'<br/>(帖 1083)"]
E --> F["2025-08<br/>Obsidian + Claude<br/>+ 自家 Obsidian MCP"]
F --> G["2026-01<br/>终态:自己→Obsidian<br/>别人→NotebookLM<br/>(帖 2201)"]
1. Basic principles of RAG (Lao Huang's popular science version, Post 36)
Lao Huang's "plain language" explanation of RAG written in 2024-03:
"Put simply, when a large model processes a document, it first performs chunking. For example, it cuts the document into chunks of 1000 tokens each. Otherwise, the document is too large for the large model to process. These cut text chunks are then stored.
After you ask a question, the large model performs semantic retrieval to find matching text chunks. These text chunks serve as reference information,are sent to the large model together with your question, and finally an answer is produced.
Therefore, if the content in the document is ambiguous, it will definitely affect the answer quality of the large model.I always try to manually revise documents into a simple form, for example one concept or question corresponding to one answer. "
Core steps:
- Chunking (chunk size + overlap)
- Embedding (Embedding model converts text into vectors)
- Storage (vector database)
- Retrieval (semantic matching + optional full-text search)
- Assemble back into Prompt (retrieval results + user question → model)
- Model generates answer (with citation when necessary)
2. Five key points of RAG optimization
2.1 Document preprocessing (most important)
- PDF → Markdown(Using Marker / MinerU / Markitdown / Microsoft markitdown-mcp)
- Direct use of PPT is not recommended(models cannot understand visual logic) → write down the text of each page separately
- Use Excel with caution(RAG handles structured data poorly)
- Each section should be self-contained + have clear boundaries(ambiguous documents will definitely break RAG)
2.2 Chunk Size + Overlap
- The default 1000-token chunk is generally OK
- Too small → cuts off semantics; LightRAG Post 864 mentions this is a common problem
- Too large → retrieval precision drops, and the model's Context is limited
2.3 Embedding model selection
| Scenario | Recommendation |
|---|---|
| Chinese + local | nomic-embed-text / mxbai-embed-large |
| Multilingual + high precision | OpenAI text-embedding-3-large |
| Chinese + cloud | OpenAI 3-large is also OK |
| Do not use defaults blindly | The default embedding in AnythingLLM sometimes fails to download (fetch failed) |
2.4 Model reasoning ability is the ceiling (key insight from Post 1076)
Real-world test(submitted by user):
- Public Qwen Max: high accuracy rate
- Local DeepSeek 70B / Qwen 2.5 70B: accuracy rate not high
Lao Huang's explanation:
"The comprehensive capabilities of models are worlds apart. Qwen Max's training resources and inference resources far exceed those of quantized versions run locally. Its instruction-following ability is much stronger. 70B open-source models easily deviate from the given context, or rely on internal knowledge rather than retrieved information.
In addition, Qwen Max also has stronger ability in understanding subtle contextual differences, integrating multiple retrieved fragments, handling contradictory information, and tolerating imperfect context quality."
Conclusion: 7B-14B local models are very weak in RAG scenarios.First use GPT-4 to validate the workflow, then downgrade to local models.
2.5 RAG is not omnipotent (Lao Huang's repeated warning)
"The RAG technology used in these current applications does not produce good answers for questions that require global understanding and the like, and there is no way around that.When I use it personally, I only put QA and documents where local text contains complete meaning (such as notes) into the knowledge base. "
Fatal weaknesses of RAG:
- Global questions (such as "what is the main idea of this article") → RAG only looks at fragments and cannot answer accurately
- Cross-paragraph reasoning → retrieval may miss key fragments
- Tables / charts → RAG handles them poorly
3. GraphRAG: expensive but useful (2024-07 Posts 358, 360 highlights)
Advantages: global questions global questions (such as "what is the main idea of this book").
Fatal disadvantage (2024-07): Extremely high cost.
- Single document + single question-and-answer = $11(using OpenAI API)
- Lao Huang tried to connect local Ollama, but the embedding step always threw errors
When to use: when global understanding is needed + cost does not matter (enterprise-level in-depth research).
When not to use: personal knowledge base / frequent Q&A scenarios.
4. Cohere Command R+: the first RAG-augmented model (2024-04 Post 62 highlight)
Features:
- 104 billion parameters
- Supports 10 languages including English / Chinese / French / German
- Deeply optimized for RAG, reducing hallucinations
- Performance second only to GPT-4 Turbo, higher than most open-source models on the market
- Significantly lower cost
Why it is a milestone: the first open-weight model explicitly designed for RAG. Subsequent Llama-3-ChatQA (fine-tuned by NVIDIA) and Gemini Long Context all followed a similar path.
See newtype · AI Tool Stack Practice and Evolution §3 Frontend chat layer — AnythingLLM added Cohere API in 2024-05.
5. Lao Huang's "Milvus + MCP" self-built knowledge base (2025-04 Post 1083 key)
Approach:
- Store documents in Milvus, vectorize locally
- Through Milvus MCP,any AI client (Cursor / Claude Desktop / ChatWise) can access it
Why this combination is good:
"This is more flexible than previous solutions, because it separates the database from the model. The model side can be replaced freely; as long as it can call MCP, it can be connected."
Comparison with "traditional RAG applications (AnythingLLM, etc.)":
| Dimension | Traditional application | Milvus + MCP |
|---|---|---|
| Database + model coupling | Yes | Decoupled |
| Model switching | Only those supported by the application | Any LLM can connect via MCP |
| Multi-client sharing | No | OK |
| Cost of upgrading the model | Modify the application | 0 |
Related: this is also the same core idea as OpenMemory MCP (Post 1338). See newtype · MCP Protocol and Ecosystem in Practice §3.2 Knowledge base / memory-type MCP.
6. Code retrieval methods of AI programming tools (inspiration for RAG, 2025-06 Post 1392)
| Tool | Method | Pros / Cons |
|---|---|---|
| Cline | Real-time search (file content + regex) | Accurate + real-time / High Token cost |
| Claude Code | Agentic Search(dynamically decides content/filenames/multi-turn) | Highest accuracy / Even higher Token cost |
| Cursor | Classic Embeddings + RAG | High cost-effectiveness / Limited Embedding comprehension |
| WindSurf | Same as Cursor but lower latency + enterprise | Enterprise-grade / Slow initial indexing |
Cline's reason for rejecting RAG:
"A function, defined at line 200, called at line 300, with related dependencies at line 400. RAG can only read fragments,which breaks the logic, so it is not suitable."
Inspiration for personal RAG:
- For code-like, strongly structured documents → real-time search may be better than RAG
- For short notes, QA format → RAG is suitable
- Cross-fragment reasoning → prioritize Long Context models (Gemini 1M+)
7. Lao Huang's evaluation of RAG tools (by period)
Early 2024 (2024-03~05)
| Tool | Evaluation |
|---|---|
| AnythingLLM | Top choice for beginners, average RAG effect, rudimentary Agent features (Post 449) |
| Open WebUI + Ollama | Built-in RAG, sufficient for simple scenarios |
| QAnything (NetEase Youdao) | Cloud version works well, local deployment is painful (spent a long time on WSL and gave up) (Post 60 highlight) |
Mid-2024 (2024-06~08)
| Tool | Evaluation |
|---|---|
| GraphRAG | Strong global Q&A but $11 per query, local connection to Ollama has some bugs |
| MaxKB | Suitable for enterprise internal training, one-click installation on iPanel (Post 149) |
| RAGFlow | Diverse document parsing, better effect than AnythingLLM (user comparison in Post 490) |
First half of 2025 (2025-01~04)
| Tool | Evaluation |
|---|---|
| AnythingLLM | "Demoted to backup, mainly using ChatWise / Obsidian" (Post 1176) |
| Obsidian Copilot | Main recommendation + local configuration (LM Studio localhost:1234) |
| Milvus + MCP | Lao Huang's self-built solution (Post 1083) |
| Metaso Search + special topics | Team / RAG as Service (Post 646) |
2026 final state (2026-01 Post 2201)
My own stuff → Obsidian (local-first) Other people's stuff → NotebookLM (strict Source-Grounding, no nonsense)
See newtype · AI Tool Stack Practice and Evolution §4 Knowledge base layer.
8. Lao Huang's philosophy of "AI knowledge base system" (2024-05 Post 147 highlight)
Lao Huang's complete idea of a "personal knowledge base system" for students:
Starting point + solution
- Information overload → AI-assisted summarization and distillation for quick general grasp
- The human brain is not suited for memorization → "Second Brain" storage + AI semantic retrieval
- Note-taking is preprocessing → information is internalized only when you type it yourself
Two subsystems
1) External information processing (AnythingLLM + large model, 2024 period) / Obsidian MCP + Claude (after 2025)
- Like a "slaughterhouse": throw daily information in, and use the large model as a "knife" to "dismember" it
- Manually put valuable content into notes (you must type the words yourself)
- Use professional tools for professional materials (such as txyz.ai for processing AI papers)
2) Note generation (Obsidian + plugins)
- Create 4 folders + subfolders PAFP according to [PARA?] logic
- Use 3 layers of association: folders + tags + inter-note links
- Form a knowledge graph, rather than scattered points
Core mindset:
"Continuously transform external information into my own things in a steady stream. This feeling of gradual accumulation and internalization is very delightful."
See newtype Knowledge System and Full Evolution of Life OS (to be written, Phase 1.1 wiki #9).
9. RAG vs Long Context vs NotebookLM (2025-2026 evolution)
Comparison of three knowledge access methods:
| Method | Capacity | Suitable for |
|---|---|---|
| RAG | Arbitrarily large (stored in chunks) | Local precise retrieval, but weak globally |
| Long Context(Gemini 1M+) | 1M tokens (one book) | Strong global reasoning, but cost rises with length |
| NotebookLM | 50-300 documents / notebook | Source-Grounding with no nonsense, balanced |
Lao Huang's choice (2026-01):
- Own notes < 100MB → directly Obsidian + Copilot/MCP,sufficient
- Other people's materials / papers → NotebookLM (free version 50 documents, premium version 300)
- Truly large-scale → Gemini Pro (1M Context) fed directly
Where RAG still has value:
- Total document volume exceeds the Long Context limit (several million tokens)
- Multi-tenant isolation (independent RAG library per user)
- Frequently updated real-time indexes
10. Anti-patterns
| ❌ | ✅ |
|---|---|
| RAG perfectionism (chasing the strongest Embedding / optimal chunk size) | Get GPT-4 + default configuration working first, then downgrade and optimize |
| Document dumping (thousands of PDFs ingested directly into the library) | Curate + preprocess + convert to Markdown format |
| Expecting GPT-4-level performance from a 7B model | Model reasoning capability is the ceiling; validate with GPT-4 first |
| Using GraphRAG for personal scenarios | Too expensive; use source-grounded NotebookLM as an alternative |
| Stuffing all information into the knowledge base | Only include QA-formatted + self-contained + high-quality content |
| Expecting global reasoning from RAG | Use Long Context or GraphRAG for global questions |
| Coupling the database and the model | Decouple with Milvus + MCP to allow any LLM to connect |
| Using RAG to process code | Cline real-time search / Claude Code Agentic Search is more accurate |
| Skipping citation verification | Always check whether citations are correct; RAG errors are very hard to self-detect |
| Treating "setting up RAG tools" as the goal | The essence of a knowledge base is the knowledge itself; tools are only the carrier |
References and further reading
Main references: curated posts from the rag-knowledge topic bucket plus cross-bucket materials. Curated post numbers: 36, 60, 62, 147, 149, 198, 212, 215, 234, 290, 300, 358, 360, 444, 445, 449, 469, 481, 483, 488, 490, 510, 535, 573, 580, 593, 612, 614, 646, 673, 715, 717, 724, 731, 764, 770, 772, 777, 786, 791, 794, 795, 824, 830, 831, 836, 838, 841, 854, 874, 878, 882, 883, 886, 893, 895, 932, 1037, 1071, 1076, 1083, 1085, 1147, 1150, 1154, 1336, 1392, 1495, 1626, 1700, 2074, 2148, 2201, etc.
Note: This wiki is the v1 version, integrated from cross-bucket curated materials (already Read buckets: mcp + prompt-engineering + tools-stack + ai-coding, containing roughly 70% RAG-related content). The complete v2 version requires subsequent session Reads of bucket-rag-knowledge.md(249KB / 341 posts) full text plus supp-rag-knowledge.md(317 posts) non-curated portions.
Companion wikis:
- newtype · MCP Protocol and Ecosystem in Practice — Integrating Milvus MCP / OpenMemory MCP / Knowledge Base MCP
- newtype · AI Tool Stack in Practice and Evolution — Evaluations of AnythingLLM / Open WebUI / RAGFlow / NotebookLM
- newtype · AI Coding in Practice and Tool Stack Evolution — Retrieval methods in Cursor / Cline / Claude Code
- (TODO)newtype Knowledge System and the Full Evolution of Life OS — Complete Obsidian note-taking system solution
- (TODO)newtype Model Evaluation, Fine-tuning, and Hardware Evolution — Embedding model evaluation / RAG models (Command R+/ChatQA)
History
- 2026-05-16 v1.0 created — Phase 1.1 wiki #8 v1 edition. Integrated from curated posts plus cross-bucket materials. The complete full set (341 posts in the rag-knowledge bucket) is reserved for the v2 upgrade.