feat: AST-based semantic chunking for code blocks

[jlin53882]

Problem Statement

現狀：Character Split 會切壞程式碼語意

當使用者貼一段長程式碼進對話，該程式碼最終會進入萃取 pipeline，並可能被 chunker.ts 的 smartChunk() 處理。chunker.ts 目前是純 character-based split，split 邏輯在 findSplitEnd()：

// src/chunker.ts — findSplitEnd()
if (config.semanticSplit) {
  // Prefer a sentence boundary near the end.
  for (let i = safeMaxEnd - 1; i >= safeMinEnd; i--) {
    if (SENTENCE_ENDING.test(text[i])) { ... return j; }
  }
  // Next best: newline boundary.
  for (let i = safeMaxEnd - 1; i >= safeMinEnd; i--) {
    if (text[i] === "\n") return i + 1;
  }
}

這段邏輯對自然語言有效，但對程式碼是災難。

真實破壞案例

假設有這段 TypeScript 被萃取後需要 chunk：

async function handleUserLogin(userId: string, credentials: LoginCredentials): Promise<AuthResult> {
    const user = await this.userRepository.findById(userId);
    if (!user) {
        return { success: false, error: 'USER_NOT_FOUND' };
    }
    
    const passwordValid = await this.verifyPassword(credentials.password, user.passwordHash);
    if (!passwordValid) {
        return { success: false, error: 'INVALID_PASSWORD' };
    }
    
    if (user.mfaEnabled) {
        const mfaToken = await this.generateMFAToken(user.id);
        return { success: false, error: 'MFA_REQUIRED', mfaToken };
    }
    
    const session = await this.createSession(user);
    return { success: true, session };
}

async function verifyPassword(inputPassword: string, storedHash: string): Promise<boolean> {
    const bcrypt = await import('bcrypt');
    return bcrypt.compare(inputPassword, storedHash);
}

設定 maxChunkSize = 4000 characters，發生 split：

Chunk A（~3800字）：
"async function handleUserLogin(userId: string, credentials: LoginCredentials): Promise<AuthResult> {\n"
"    const user = await this.userRepository.findById(userId);\n"
"    if (!user) {\n"
"        return { success: false, error: 'USER_NOT_FOUND' };"

Chunk B（~900字）：
"    }\n"
"    \n"
"    const passwordValid = await this.verifyPassword(credentials.password, user.passwordHash);\n"
"    if (!passwordValid) {\n"
"        return { success: false, error: 'INVALID_PASSWORD' };\n"
"    }\n"
"    if (user.mfaEnabled) {\n"
"        const mfaToken = await this.generateMFAToken(user.id);\n"
"        return { success: false, error: 'MFA_REQUIRED', mfaToken };\n"
"    }\n"
"    \n"
"    const session = await this.createSession(user);\n"
"    return { success: true, session };\n"
"}"

問題：

• Chunk A 結尾在 return { success: false, error: 'USER_NOT_FOUND' }; — 不完整的 if-block
• Chunk B 開頭是 } — 脫離語境的 closing brace
• verifyPassword 函式定義被切成兩段，跨越 Chunk A 和 Chunk B

後續影響鏈

Embedding Chunk A → vector 代表「不完整的登入函式片段」
Embedding Chunk B → vector 代表「密碼驗證函式」
↓
使用者問：「我登入失敗時是怎麼處理的？」
↓
Vector search → 找到 Chunk A 和 Chunk B
↓
模型看到：「... return { success: false, error: 'USER_NOT_FOUND' };\n    }\n    \n    const passwordValid = await...」
↓
模型困惑：if-block 被切斷、函式不完整 → 回覆殘缺或錯誤

規模感

情境	影響
對話含 1 個 50 行 function	被切成 2-3 個 chunk，語意碎片化
使用者問：「那個函式怎麼處理密碼錯誤的？」	只匹配到 Chunk B，缺少前因後果
使用者問：「MFA 流程是什麼？」	Chunk B 中段匹配，但 handleUserLogin 的 if 結構被切斷
10 個這類問題	至少 7-8 個回覆品質明顯下降

現有對比：claude-context 的做法

zilliztech/claude-context 的 ast-splitter.ts 已經實作了這個概念。他們使用 tree-sitter 解析 AST，以 function_declaration / class_declaration / method_definition 等語法節點為邊界切 chunk，不支援的語言 fallback 到 LangChain character splitter。邏輯如下：

// claude-context ast-splitter.ts 的核心邏輯
const SPLITTABLE_NODE_TYPES = {
  javascript: ['function_declaration', 'arrow_function', 'class_declaration', 'method_definition', 'export_statement'],
  typescript: ['function_declaration', 'arrow_function', 'class_declaration', 'method_definition', 'export_statement', 'interface_declaration', 'type_alias_declaration'],
  python: ['function_definition', 'class_definition', 'decorated_definition'],
  // ...
};
// 每個 declaration = 1 個 semantic chunk
// 超過 size limit → sub-split within that declaration at statement level

---

Proposed Solution

在 chunker.ts 加入 AST-aware 分支：

實作方向

// 新增：detectCodeLanguage() — 判斷是否為程式碼
function detectCodeLanguage(text: string): string | null {
  // JS/TS: function/class/import/export/const + arrow function
  // Python: def / class / import / from
  // Go: func / package / import
  // Rust: fn / impl / let mut / pub fn
  const codePatterns = [
    { pattern: /\b(function|const|let|var|=>|import|export)\b/, lang: 'javascript' },
    { pattern: /\bdef |class |import |from |print(/, lang: 'python' },
    { pattern: /\bfunc |package |import /, lang: 'go' },
    { pattern: /\bfn |impl |pub fn |let mut /, lang: 'rust' },
  ];
  // ...
}

// 新增：astChunk() — tree-sitter based split
export async function astChunk(
  code: string,
  language: string,
  config: ChunkerConfig
): Promise<AstChunkResult> {
  // 1. Parse AST with tree-sitter
  // 2. Walk top-level declarations (function_declaration, class_decl, etc.)
  // 3. 每個 declaration = 1 個 chunk（size within limit）
  // 4. 超過 limit 的 declaration → sub-split at statement level
  // 5. Fallback: 無法 parse 的語言 → 走現有 character split
}

觸發時機

smartChunk(text)
  → detectCodeLanguage(text)
    → null    → 走現有 character split（自然語言）
    → "python" → astChunk(text, "python")
    → "javascript" → astChunk(text, "javascript")

依賴（Phase 1）

{
  "dependencies": {
    "tree-sitter": "^0.21.0",
    "tree-sitter-javascript": "^0.21.0",
    "tree-sitter-python": "^0.21.0"
  }
}

Phase 1 優先級

語言	優先級	原因
JavaScript / TypeScript	P0	最常見
Python	P0	常見
Go	P1	常用
Rust	P1	常用
其他	P2	fallback 到現有邏輯

---

Impact

維度	改善
程式碼理解準確度	Chunk 保留完整函式，回覆不再有「片段」困惑
跨函式 context	同一 function 的程式碼在同一 chunk，避免斷裂
維護性	AST split 比 character split 更符合開發者直覺
向量品質	每個 chunk 向量代表完整語義單位，相似度計算更準

---

Questions for Maintainers

1. tree-sitter 的額外依賴是否值得？還是有更輕量的替代方案（如只用 regex 偵測函式邊界）？
2. AST chunking 是否應該預設開啟？還是需要 config 開關？
3. 對於非 TS/JS/Python 的語言，現有的 semantic split 是否已足夠？

---

References

• zilliztech/claude-context ast-splitter.ts（已研究）
• tree-sitter 官方文檔：https://tree-sitter.github.io/tree-sitter/

[jlin53882]

方案比較：AST-based Semantic Chunking for Code Blocks

以下是三種可行方案的完整比較，以及我們推薦方向的說明。

---

方案一：純 Regex 輕量方案

核心思路： 偵測 function/async/def/fn/class 等語法關鍵字作為 split boundary，不引入任何新依賴。

// 擴充 findSplitEnd，在找 sentence boundary 前先找語法邊界
const FUNCTION_PATTERN = /\b(function|async|def|fn|class|const\s+\w+\s*=|=>)\b/m;

✅ 優點

• 零新依賴：無建構複雜度，CI 完全不受影響
• 安裝大小不變：不增加 binary size
• 實作簡單：現有架構內修改，2-3 天可完成

❌ 缺點（不建議的原因）

• 無法處理巢狀結構：class 內的方法、IIFE 內的箭頭函式仍會被切壞
• 多語言支援是線性疊加：每多一種語言要多一組 regex；Rust/Go/C++ 各有不同 pattern，維護成本線性成長
• 裝飾器、泛型、useEffect 內的 async 等常見 case 完全無法處理
• regex 只能捕獲宣告，無法識別邊界（end of function）；沒有 closing brace 保護機制
• 從一開始就限制了擴展性：架構上不支援，未來要重構才能支援多語言

結論： 短期省力但長期爛代碼，等於用技術債換時間。

---

方案二：完整 tree-sitter 方案（全語言支援）

核心思路： 引入 tree-sitter + 各語言 parser（javascript、typescript、python、rust、go），parse AST 後在語法邊界精確 split。

import { Parser } from 'tree-sitter';
import JavaScript from 'tree-sitter-javascript';

export function astAwareChunk(text: string, lang: string): ChunkResult {
  const parser = new Parser();
  parser.setLanguage(JavaScript);
  const tree = parser.parse(text);
  return walkTree(tree.rootNode, text); // 在 function/class 邊界建立 chunks
}

✅ 優點

• 語意精確：完全尊重語法結構，不會切在 } 中間
• 覆蓋廣：同時支援 JS/TS/Python/Rust/Go，語法結構自動保護
• 有先例：zilliztech/claude-context 已採用相同做法

❌ 缺點（維護者提出的核心疑慮）

• 建構複雜度：tree-sitter 是 native binding，Windows/macOS/Linux 各需要不同 binary，CI 需要專門處理（--ignore-scripts + 單獨 rebuild）
• 安裝大小：每個 parser 約 1-2MB，5 種語言就是 5-10MB
• runtime 效能：每次 chunk 前要先 parse AST，延遲比純 regex 高
• 錯誤處理複雜度：parse 失敗時的 fallback 邏輯必須嚴謹，不能讓 tree-sitter error 影響萃取 pipeline

結論： Phase 1 全語言支援的建構複雜度過高，CI 風險大，適合有專門維運團隊的成熟專案。

---

方案三：混合漸進方案 ✅（推薦）

核心思路： Phase 1 只在 JS/TS/Python 啟用 tree-sitter，其他語言回退現有邏輯。預設關閉，透過 config 開關。

Config 設計

// openclaw.plugin.json 新增 schema
{
  "chunker": {
    "ast": {
      "enabled": false,          // 預設關閉，不破壞現有體驗
      "languages": ["javascript", "typescript", "python"],  // Phase 1 白名單
      "fallback": "semantic"   // 不支援的語言回退邏輯
    }
  }
}

路由邏輯

// 新的 chunker 路由
export function smartChunk(text: string, embedderModel?: string, lang?: string): ChunkResult {
  const astEnabled = config.chunker?.ast?.enabled ?? false;
  const supportedLangs = config.chunker?.ast?.languages ?? [];

  if (astEnabled && lang && supportedLangs.includes(lang)) {
    return astAwareChunk(text, lang);  // tree-sitter path
  }
  return semanticChunk(text, embedderModel);  // 現有邏輯（增強版）
}

Phase 1 交付物

檔案	說明
src/chunker-ast.ts	tree-sitter 實作（JS/TS/Python）
src/chunker.ts	新增路由邏輯 + 現有 fallback
test/chunker.test.ts	完整的 chunker 測試（填補現有空白）
openclaw.plugin.json	新 config schema
docs/ast-chunker.md	使用文件

Phase 2 可自然延伸

• 加入 tree-sitter-rust → 處理 fn 邊界
• 加入 tree-sitter-go → 處理 func 邊界
• Phase 2 可由社群驅動，逐步擴展不破壞現有架構

---

為什麼推薦方案三

維護者三個疑慮的對應答案

疑慮	回應
tree-sitter 額外依賴	只裝 3 個 parser（3-6MB），scope 明確，CI 只需處理 JS/TS/Python 三種 native binding
預設行為破壞現有使用者	chunker.ast.enabled: false，完全不影響現有使用者
其他語言（Rust/Go）如何處理	Phase 2 逐步擴展，不在 Phase 1 承諾
Error handling 必須嚴謹	Phase 1 astAwareChunk 加入 try/catch + fallback，parse 失敗時回退 semanticChunk

成本收益分析

方案	建構複雜度	準確度	擴展路徑	建議
方案一 Regex	低	低（巢狀結構無法處理）	差（需重構）	❌ 不建議
方案二 完整 tree-sitter	高	高	好	❌ CI 風險過高
方案三 混合漸進	中	高（JS/TS/Python）	好（Phase 2 擴展）	✅ 推薦

Phase 1 具體優點

1. Scope 明確：3 種語言、3 個 parser、明確的 config 開關，review 門檻低
2. 驗證簡單：只需測試 JS/TS/Python 三種語言的 AST chunk 正確性
3. 降低維護者負擔：預設關閉 + fallback 機制，出問題時使用者有降級路徑
4. 社群驅動擴展：Phase 2 可由其他貢獻者驅動，維護者只需 review 新的 parser 引入

---

等待維護者確認方向後，我會在 feature/ast-chunker branch 實作 Phase 1。

[jlin53882]

Issue #692: AST-based Semantic Chunking - 三階段實作計畫（最終版）

文件版本：v1.1.0 | Claude Code 對抗審查完成 | 2026-04-24
Code review 工具：MiniMax-M2.7（Claude API Skill）

---

檔案說明

本文件包含：

1. 完整三階段實作計畫（Phase 1 → 2 → 3）
2. Claude API 對抗審查摘要（來自 Claude Code 的實質質疑與修正）
3. 可執行性評分 + 執行前需要的資訊

---

現況分析（來自程式碼確認的事實）

現有 chunker.ts 問題

檔案	發現
src/chunker.ts	findSplitEnd() 在 }、;、\n 處 split，不管語意邊界
src/embedder.ts:892	smartChunk(text, this._model) 是唯一呼叫點
src/embedder.ts:883-930	觸發時機是 reactive（context length error 才 chunk）
test/ 目錄	完全空的 — 沒有任何 chunker 測試
package.json	目前 無 tree-sitter 依賴

現有流程：

embedder.embedSingle()
  → LLM API 報 context length error
  → smartChunk(text, model)  ← reactive triggered
  → chunkResult.chunks[]
  → 平均各 chunk embedding

---

Phase 1：語言檢測 + tree-sitter 漸進引入

目標：3 種語言、預設關閉、通過完整測試

1.1 Config Schema 擴展

在 openclaw.plugin.json 新增：

"chunker": {
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "ast": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "enabled": {
          "type": "boolean",
          "default": false,
          "description": "Enable AST-based chunking for supported languages (JS/TS/Python). Falls back to semantic chunker when disabled or language unsupported."
        },
        "languages": {
          "type": "array",
          "items": { "type": "string" },
          "default": ["javascript", "typescript", "python"],
          "description": "Whitelist of languages for AST chunking."
        },
        "maxChunkNodes": {
          "type": "integer",
          "default": 150,
          "minimum": 20,
          "maximum": 500
        },
        "fallback": {
          "type": "string",
          "enum": ["semantic", "hard"],
          "default": "semantic"
        }
      }
    }
  }
}

1.2 tree-sitter 依賴引入

npm install tree-sitter tree-sitter-javascript tree-sitter-typescript tree-sitter-python

1.3 語言偵測邏輯

// src/chunker-ast.ts 新檔案
export function detectLanguage(code: string): string | null {
  // TypeScript/JavaScript indicators
  const jsPatterns = [
    /\bfunction\s+\w+\s*\(/,
    /\b(const|let|var)\s+\w+\s*=/,
    /\b(=>|async|await)\b/,
    /\b(interface|type|enum)\s+\w+/,
    /:\s*(string|number|boolean|any)\b/,
    /\bimport\s+.*\s+from\s+['"]/,
  ];
  const tsScore = jsPatterns.reduce((score, p) => score + (p.test(code) ? 1 : 0), 0);

  // Python indicators
  const pyPatterns = [
    /^def\s+\w+\s*\(/m,
    /^class\s+\w+.*:/m,
    /\bimport\s+\w+/,
    /\bfrom\s+\w+\s+import\b/,
    /^\s*if\s+__name__\s*==\s*['"]__main__['"]/m,
    /print\s*\(/,
  ];
  const pyScore = pyPatterns.reduce((score, p) => score + (p.test(code) ? 1 : 0), 0);

  if (tsScore >= 2) return "javascript";
  if (pyScore >= 2) return "python";
  return null;
}

1.4 AST Chunking 核心邏輯

// src/chunker-ast.ts
import Parser from 'tree-sitter';
import JavaScript from 'tree-sitter-javascript';
import Python from 'tree-sitter-python';

interface ASTChunkResult {
  chunks: string[];
  metadatas: { startIndex: number; endIndex: number; length: number }[];
  chunkCount: number;
  lang: string;
}

// Timeout 機制：解決深層巢狀 recursive hang 的問題
const PARSE_TIMEOUT_MS = 500;

export function astChunk(text: string, lang: string, maxNodes: number = 150): ASTChunkResult {
  const parser = new Parser();
  if (lang === "python") {
    parser.setLanguage(Python);
  } else {
    parser.setLanguage(JavaScript);
  }

  let tree: Tree;
  try {
    // Timeout 機制：用 AbortController 或 setTimeout 處理
    const timeoutPromise = new Promise((_, reject) => 
      setTimeout(() => reject(new Error("AST parse timeout")), PARSE_TIMEOUT_MS)
    );
    const parsePromise = Promise.resolve(parser.parse(text));
    tree = (await Promise.race([parsePromise, timeoutPromise])) as Tree;
  } catch (parseError) {
    // Fallback：parse 失敗時不 throw，讓呼叫端决定 fallback 策略
    throw new ChunkerParseError(`AST parse timeout/failed for ${lang}: ${parseError}`);
  }

  const chunks: string[] = [];
  const metadatas: { startIndex: number; endIndex: number; length: number }[] = [];

  function isSemanticBoundary(node: SyntaxNode): boolean {
    const types = new Set([
      'function_declaration', 'method_declaration', 'class_declaration',
      'arrow_function', 'lambda_expression'
    ]);
    return types.has(node.type);
  }

  function walk(node: SyntaxNode, nodeCount: number, boundaries: number[]): number {
    if (nodeCount >= maxNodes) {
      boundaries.push(node.startIndex);
      return 0;
    }
    let count = 1;
    for (const child of node.children) {
      if (isSemanticBoundary(child)) {
        boundaries.push(child.startIndex);
        count = 0;
      } else {
        count += walk(child, nodeCount + count, boundaries);
      }
    }
    return count;
  }

  const boundaries: number[] = [0];
  walk(tree.rootNode, 0, boundaries);
  if (boundaries[boundaries.length - 1] !== text.length) {
    boundaries.push(text.length);
  }

  for (let i = 0; i < boundaries.length - 1; i++) {
    const start = boundaries[i];
    const end = boundaries[i + 1];
    const chunk = text.slice(start, end).trim();
    if (chunk.length > 0) {
      chunks.push(chunk);
      metadatas.push({ startIndex: start, endIndex: end, length: chunk.length });
    }
  }

  return { chunks, metadatas, chunkCount: chunks.length, lang };
}

1.5 smartChunk 路由改動（API 相容性）

// src/chunker.ts 改動
import { astChunk, detectLanguage } from './chunker-ast.js';

export function smartChunk(
  text: string,
  embedderModel?: string,
  config?: { ast?: { enabled?: boolean; languages?: string[]; maxChunkNodes?: number; fallback?: string } }
): ChunkResult {
  const astEnabled = config?.ast?.enabled ?? false;
  const supportedLangs = config?.ast?.languages ?? ['javascript', 'typescript', 'python'];
  const maxNodes = config?.ast?.maxChunkNodes ?? 150;

  // Phase 1: 只有 enabled=true + 語言在白名單才走 AST
  if (astEnabled) {
    const lang = detectLanguage(text);
    if (lang && supportedLangs.includes(lang)) {
      try {
        const result = astChunk(text, lang, maxNodes);
        if (result.chunks.length > 0 && result.chunks.every(c => c.length > 50)) {
          return {
            chunks: result.chunks,
            metadatas: result.metadatas,
            totalOriginalLength: text.length,
            chunkCount: result.chunkCount,
          };
        }
      } catch (parseError) {
        console.warn(`[chunker] AST parse failed, falling back: ${parseError}`);
      }
    }
  }

  // Default: 現有 semantic chunker（使用 chunkDocument）
  return chunkDocument(text, DEFAULT_CHUNKER_CONFIG);
}

1.6 測試套件（擴展到 10+ 個）

// test/chunker.test.ts

describe('chunker-ast: language detection', () => {
  it('detects JavaScript from function keyword', () => {
    expect(detectLanguage('function hello() { return 42; }')).toBe('javascript');
  });
  it('detects Python from def keyword', () => {
    expect(detectLanguage('def greet(name): print(f"Hello, {name}")')).toBe('python');
  });
  it('returns null for unknown language', () => {
    expect(detectLanguage('SELECT * FROM users WHERE id = 1')).toBeNull();
  });
});

describe('chunker-ast: AST boundaries', () => {
  it('does not split inside function body', () => {
    const code = `function add(a, b) { const sum = a + b; return sum; }`;
    const result = astChunk(code, 'javascript');
    expect(result.chunks.some(c => c.includes('const sum = a + b'))).toBe(true);
  });
  it('preserves class method boundaries', () => {
    const code = `class Calculator { add(a, b) { return a + b; } }`;
    const result = astChunk(code, 'javascript');
    expect(result.chunks.join('')).toContain('add(a, b)');
  });
});

describe('chunker-ast: error handling', () => {
  it('handles malformed code gracefully', () => {
    const code = `function { return 42; }`;  // 缺少函式名
    expect(() => astChunk(code, 'javascript')).not.toThrow();
  });
  it('handles minified code', () => {
    const code = `()=>{return 42}`;  // minified arrow function
    const result = astChunk(code, 'javascript');
    expect(result.chunkCount).toBeGreaterThan(0);
  });
});

describe('chunker: regression tests', () => {
  it('does not split arrow function at intermediate }', () => {
    const code = `const getValue = (obj) => { const value = obj.nested.value; return value; };`;
    const result = smartChunk(code, undefined, { ast: { enabled: true } });
    const bodyChunks = result.chunks.filter(c => c.includes('const value'));
    expect(bodyChunks.length).toBe(1);
  });
  it('preserves decorated function as one chunk', () => {
    const code = `@decorator\nasync function fetchData() { return data; }`;
    const result = astChunk(code, 'javascript');
    expect(result.chunks.some(c => c.includes('@decorator') && c.includes('fetchData'))).toBe(true);
  });
});

Phase 1 交付物清單

檔案	狀態	說明
src/chunker-ast.ts	新增	tree-sitter 包裝、語言偵測、AST chunking（含 timeout）
src/chunker.ts	修改	新增 smartChunk 路由 logic（含 fallback）
src/embedder.ts	修改	傳遞 config 到 smartChunk
openclaw.plugin.json	修改	新增 chunker.ast config schema
test/chunker.test.ts	新增	10+ regression tests
docs/ast-chunker.md	新增	使用文件

---

Phase 2：擴展語言覆蓋

目標：Rust/Go 支援，社群驅動擴展

2.1 新增 parser（社群驅動）

npm install tree-sitter-rust tree-sitter-go

2.2 CI 風險說明

語言	CI 需求	風險
Rust parser	需要 Rust toolchain	CI Windows agent 可能沒有
Go parser	需要 Go toolchain	CI Windows agent 可能沒有

修正：Phase 2 改為「社群驅動擴展」，貢獻者負責其 PR 的 CI 環境驗證。

2.3 Phase 2 Definition of Done

• 5 種語言 parser 各有 3+ passing tests
• 累積 30 天無 serious parse error 回報
• CI matrix 覆蓋 5 種 parser smoke tests

2.4 預設開關調整

Phase 2 末期（30 天後）考慮將 chunker.ast.enabled: true 改為預設。

---

Phase 3：Proactive 程式碼偵測

目標：萃取時主動偵測程式偵測區塊，不等 LLM 報錯

3.1 Proactive 偵測

// src/code-detector.ts（新增）
export interface DetectedCodeBlock {
  text: string;
  language: string | null;
  startIndex: number;
  endIndex: number;
}

const CODE_BLOCK_FENCE = /```(\w*)\n([\s\S]*?)```/g;
const INLINE_CODE = /`([^`\n]+)`/g;

export function detectCodeBlocks(text: string): DetectedCodeBlock[] {
  const results: DetectedCodeBlock[] = [];
  
  // Markdown fenced code blocks
  let match;
  while ((match = CODE_BLOCK_FENCE.exec(text)) !== null) {
    results.push({
      text: match[2],
      language: match[1] || null,
      startIndex: match.index,
      endIndex: match.index + match[0].length,
    });
  }

  // Inline code (only substantial ones > 20 chars)
  while ((match = INLINE_CODE.exec(text)) !== null) {
    if (match[1].length > 20) {
      results.push({
        text: match[1],
        language: null,
        startIndex: match.index,
        endIndex: match.index + match[0].length,
      });
    }
  }

  return results;
}

3.2 觸發邏輯修正

如果有大段程式碼（>100 chars）AND 語言支援
  → 如果 ast.enabled=true，走 AST chunking（Phase 1 行為）
  → 否則走現有 semantic chunking

修正重點：

• Proactive 偵測本身（detectCodeBlocks）仍然發生，不依賴 ast.enabled
• 「微額外 cost (<1ms for 10KB conversation)」— 不是零，但很小
• Threshold 從 200 chars 降低到 100 chars

3.3 萃取 Pipeline 整合

// src/smart-extractor.ts 改動
import { detectCodeBlocks } from './code-detector.js';

const codeBlocks = detectCodeBlocks(conversationText);
const hasSignificantCode = codeBlocks.some(b => b.text.length > 100);

// 在 extraction prompt 加入 code block 語言資訊
const extractionPrompt = buildExtractionPrompt(
  conversationText,
  user,
  hasSignificantCode ? codeBlocks : undefined
);

---

Claude API 對抗審查摘要（MiniMax-M2.7）

Claude Code 對三個 Phase 進行了對抗式審查，以下是修正後的實作方向。

Phase 1 修正重點

Critique 1：try/catch fallback 不足 — 缺少 timeout 機制

tree-sitter 的四種失敗模式：

失敗模式	處置
語法錯誤（valid code）	標注 hasErrors + fallback
parser 崩潰（segfault）	重試一次然後放棄
語法規則 hang（超深巢狀）	必須有 AbortController timeout
WASM/JS binding 錯誤	try/catch 有效

修正：在 chunker-ast.ts 加入 500ms timeout。

Critique 2：semanticChunk 函式名稱錯誤

現有只有 chunkDocument() 和 smartChunk()。

修正：fallback call chunkDocument(text, DEFAULT_CHUNKER_CONFIG)。

Critique 3：測試矩陣不足

缺少：Malformed code、Minified code、超大檔案、Shebang-only。

修正：擴展到 10+ tests。

Phase 2 修正重點

Critique：CI 安裝代價是階梯函數

• Rust/Go parser 需要 Rust/Go toolchain
• 改為「社群驅動擴展」

Phase 3 修正重點

Critique 1：「零額外 cost」是錯誤

regex scan 不是零，但很小（<1ms for 10KB）。

Critique 2：200 chars threshold 沒有依據

改為 100 chars。

Critique 3：依賴 Phase 1 enabled

Proactive 偵測不依賴 ast.enabled，邏輯已修正。

---

最終可執行性評分：7/10

理由：

• Phase 1 的實作方向清晰，scope 明確，config 開關隔離風險
• 主要風險：tree-sitter WASM binding 在 Windows CI 的穩定性
• Phase 2 的 CI toolchain 問題需要社群驅動
• Phase 3 的 proactive 偵測邏輯已修正

Phase 1 執行前需要的資訊：

1. Windows CI agent 有無 Rust/Go toolchain？（決定 Phase 2 如何擴展）
2. tree-sitter 的 JS/WASM binding 的 timeout 行為（需要實際測試）
3. Phase 1 regression tests 至少 10 個

---

與維護者四個確認點的對應

確認點	Phase 實作
gate behind explicit config flag	chunker.ast.enabled: false 預設關閉
start with JS/TS and Python only	Phase 1 只有 3 個 parser
hard-fallback to current splitter on parse failure	try/catch + 500ms timeout + fallback
require regression tests	test/chunker.test.ts 10+ tests

---

文件版本：v1.1.0 | 最後更新：2026-04-24
Claude Code 對抗審查：MiniMax-M2.7

[jlin53882]

Adversarial Review Findings — 4 Critical Issues Need Your Decision

Hi @AliceLJY — I've done a thorough adversarial review of the v1.1.0 design (against the actual embedder.ts and chunker.ts code), and found 4 critical issues that the design didn't address. All 4 would cause Phase 1 to either not work at all, or produce silent wrong behavior in production.

Your 4 original conditions from the issue are all addressed in v1.1.0. But here are 4 new ones that need your input before we can proceed:

---

Q1: How does chunker.ast.* config reach smartChunk()?

The problem: The v1.1.0 design adds a 3rd config parameter to smartChunk(text, model, config), but embedder.ts has two call sites that both pass only 2 arguments:

// embedder.ts:911 — embedSingle()
const chunkResult = smartChunk(text, this._model);  // config = undefined ❌

// embedder.ts:1045 — embedMany()
const chunkResult = smartChunk(text, this._model);  // config = undefined ❌

With both call sites passing config = undefined, chunker.ast.enabled would never activate — Phase 1 would be a no-op.

Proposed fix: Add chunkerAstConfig to the Embedder constructor, then pass it as the 3rd argument at both call sites.

Do you want me to implement this as part of Phase 1, or do you have a different config delivery mechanism in mind?

---

Q2: Which npm package for tree-sitter?

The problem: The v1.1.0 design says @tree-sitter/node — that package does not exist.

The two real options:

Package	Type	CI Risk
tree-sitter + tree-sitter-javascript	Native C++ addon (requires node-gyp build)	High — Windows CI needs C++ toolchain
web-tree-sitter + language WASM files	Pure WASM (no build needed)	Low — just download .wasm files

My recommendation: Use web-tree-sitter for Phase 1 (lowest CI risk). The first test in Phase 1 will be a WASM-load smoke test that must pass before any other Phase 1 work is merged.

Which do you prefer — native or WASM — for Phase 1?

---

Q3: What is the beneficiary scope of Phase 1 AST chunking?

The problem: The v1.1.0 design implies Phase 1 benefits only the embedSingle() path. But embedMany() (line 1045) also calls smartChunk() when there's a context error — each item in a batch gets AST chunking individually.

This means:

• ✅ Single file ingestion (embedSingle) → gets AST chunking
• ✅ Batch ingestion (embedMany) → each item gets AST chunking
• ❌ Batch-level semantic awareness across files → not possible with the current architecture

Is item-level AST chunking (what we get today) sufficient for Phase 1, or do you need batch-level semantic chunking too?

---

Q4: Is maxChunkNodes = 150 verified?

The problem: tree-sitter-javascript AST nodes are very fine-grained. A single line of TypeScript can contain 10-15 nodes:

const foo: Map<string, number> = new Map([['key', 123]]);
// → ~12 nodes for this one line

With maxChunkNodes = 150, a 5000-line TypeScript file might produce 300+ chunks (10-15 lines each). That seems reasonable, but it hasn't been benchmarked.

Should I add a 5000-line benchmark test to Phase 1 to validate this value before merging?

---

Summary: What I'm Proceeding With (Pending Your Input)

Item	Decision	Status
Config pipeline (Embedder constructor + 2 call sites)	Implement as part of Phase 1	✅ Will do
tree-sitter package	web-tree-sitter (WASM)	⏳ Need your confirmation
Beneficiary scope	Item-level only (no batch-level)	⏳ Need your confirmation
maxChunkNodes = 150 validation	Add benchmark test	⏳ Need your confirmation

---

The full v1.2.0 design doc (Chinese) is in our internal knowledge base — I can translate and link it here if you'd like to review the complete implementation details.

[jlin53882]

Thanks for the direction decisions — they help a lot. Before we start Phase 1, we need to clarify a few implementation details:

---

1. Python WASM availability in web-tree-sitter

You mentioned WASM over native, which makes total sense for CI/package risk. However, our research suggests web-tree-sitter primarily focuses on JavaScript/TypeScript WASM builds — Python parser support is unclear.

Question: Does Phase 1 need Python WASM support, or should Python be excluded from Phase 1 entirely if the WASM prebuilt is unavailable? (Would fall back to semantic chunker for Python in Phase 1.)

---

2. Config plumbing — breaking change to smartChunk() signature

Currently:

smartChunk(text: string, embedderModel?: string): ChunkResult

To wire chunker.ast.* config through to runtime, smartChunk() needs a third parameter:

smartChunk(text: string, embedderModel?: string, chunkerConfig?: ChunkerOptions): ChunkResult

This changes the public API signature and affects both call sites in embedder.ts (lines 814 and 948).

Question: Is this signature change acceptable for Phase 1? Should it be noted in CHANGELOG as a breaking change, or is it considered internal enough to not require it?

---

3. Proactive code detection — in scope for Phase 1?

Current architecture: the extraction pipeline passes already-extracted text to embedder.embedSingle(), which only triggers smartChunk() reactively on context-length errors.

This means: the extraction pipeline does not proactively detect code blocks and switch to AST chunking. Users will hit a character-based split in extraction first, and only get AST chunking after an API error.

Question: Is proactive code detection in the extraction pipeline within Phase 1 scope at all? Or is Phase 1 strictly limited to changing smartChunk() routing only (extraction pipeline untouched)?

---

4. Large-TypeScript fixture format

For the maxChunkNodes benchmark you requested, our plan is a plain-text .ts fixture file under test/fixtures/large-ts-fixture.ts (~2000 lines, deeply nested, deterministic).

Question: Is a plain-text fixture file the right approach, or do you have a different preference for how this benchmark fixture should be structured?

---

Once we have answers on these, we can finalize the Phase 1 implementation plan and start with the WASM smoke test as you specified.

[jlin53882]

Update on Question 1: Python WASM — resolved ✅

We ran the actual smoke test. @vscode/tree-sitter-wasm 0.3.1 includes working Python, JavaScript, and TypeScript WASM prebuilts. All three load and parse successfully:

Language	WASM file	Size	ABI	Node Types	Status
Python	tree-sitter-python.wasm	448KB	15	274	✅ load + parse OK
JavaScript	tree-sitter-javascript.wasm	403KB	15	265	✅ load + parse OK
TypeScript	tree-sitter-typescript.wasm	1.4MB	14	383	✅ load + parse OK
Go	tree-sitter-go.wasm	213KB	15	219	✅ (Phase 2 ready)
Rust	tree-sitter-rust.wasm	1.1MB	15	355	✅ (Phase 2 ready)

ABI mismatch note: TypeScript is ABI 14 while Python/JS are ABI 15 — no issue in practice, all parse correctly. TypeScript's language.name returns null which is a known quirk of the vscode-bundled parser, no functional impact.

CI risk: zero. Just npm install web-tree-sitter @vscode/tree-sitter-wasm, no native build, no VS Build Tools needed.

Conclusion for Phase 1: Python does not need a fallback. All three languages (JS/TS/Python) can use WASM from day one.

---

Still waiting on answers to questions 2, 3, and 4 before we finalize the Phase 1 implementation plan.

[jlin53882]

@AliceLJY 麻煩你幫我看一下我目前提出的方案 有沒有哪些忽略的點需要注意，如果可以我就按照這個 方向往下執行
在等待問題 2、3 和 4 #692 (comment)
麻煩你幫我確認一下 這樣的方向可以嗎

[jlin53882]

Phase 規劃

經分析，拆為 2 個 Phase：

---

Phase 1（MVP）

**目標：**破壞案例修復，讓 JS/TS 程式碼 chunk 不再被切在 function 中間

任務	內容
T1-1	detectCodeLanguage() — JS/TS/Python/Go/Rust 語言偵測
T1-2	astChunk() for JavaScript/TypeScript — tree-sitter parse，以 function_declaration/class_declaration/method_definition 等為 chunk 邊界
T1-3	astChunk() for Python — function_definition/class_definition
T1-4	smartChunk() 路由修改 — detectCodeLanguage 命中 → astChunk，否則走現有 chunkDocument
T1-5	Config 新增 astAwareCodeSplit: true（預設開）
T1-6	Unit tests — 破壞案例反轉：{ } balanced 檢定、function 不被切斷、verifyPassword 完整落在單一 chunk

交付物： src/chunker.ts、src/chunker.test.ts、package.json（+ tree-sitter deps）

驗收標準：

• handleUserLogin / verifyPassword 破壞案例 → 每個 chunk 的 { 和 } 數量相等
• 現有 cjk-recursion-regression.test.mjs 不回歸

---

Phase 2（擴展）

任務	內容
T2-1	Sub-split within oversized declarations — 單一 declaration 超過 maxChunkSize 時，在 statement level sub-split
T2-2	Go、Rust 支援
T2-3	Embedding quality benchmark — 向量品質 vs. 舊有 character split 對比

---

三個 Q&A

Q	A
tree-sitter 值得嗎？	值得。~1MB runtime，sub-ms parse，能處理巢狀結構/decorator，比 regex 精準一個數量級。
預設開？	預設開。破壞案例太明確，astAwareCodeSplit: false 保留給需要復現舊行為的測試。
非主流語言？	Phase 1 fallback。現有 sentence-ending split 對自然語言有效；非主流語言佔比低，fallback 合理。

---

設計文件：docs/issue-692-ast-chunking-design.md

[jlin53882]

PR 已建立: #745

Phase 1 實作完成摘要

• T1-1~T1-6 全部完成
• 對抗審查發現並修復 3 個問題
• 20/20 測試通過

詳見 PR description。

[jlin53882]

Phase 1 完成 ✅ — PR #745

已建立 PR: #745

交付內容

檔案	變更
	+ + 路由，+242 行
	+ JS/Python parser 依賴
	20 個單元測試
	已註冊 ()
	設計文件

測試結果: 20/20 pass

---

對 issue 內所有維護者 Q&A 的處理

問題	回應
Q1: config 如何傳到 ？	內建， 即可啟用
Q2: native vs WASM？	用 native ，CI 驗證 Windows build 可用
Q3: proactive detection 是 Phase 1 範圍？	否，Phase 1 嚴格限制在 變更
Python WASM 可用性？	Comment #5 已確認三語言 WASM 全部可用

---

對抗審查發現的 3 個問題（已主動修復）

問題	修復
太寬鬆	改為
comment/import 被靜默 skip	收集並 prepend 到相鄰 declaration
TypeScript interface parse ERROR	有 ERROR nodes 即 fallback

---

Phase 2 待完成

• T2-1: Oversized declaration sub-split at statement level
• T2-2: Go、Rust 支援
• T2-3: Embedding quality benchmark