mzw: add knowledge evolution support

CLAUDE.md

@@ -4,6 +4,57 @@
 - 文件系统即数据库；server 与 orchestrator 解耦，通过文件系统传递状态
 - 多节点通过共享文件系统协同，每个节点只写自己的 `nodes/<node_id>/`
 
+## 数据一致性：单一数据源（Single Source of Truth）
+
+**文件系统即数据库**这一选择的代价是：失去数据库内置的一致性约束。任何"同一份事实"被存到多个文件，都会在并发/重启/部分写入下漂移，最终表现为难以排查的功能 bug。下列原则**强制执行**：
+
+### 原则
+
+1. **每份事实有且只有一个权威文件**（source of truth）。其他文件需要这份信息时，要么从权威源读取后派生（运行时计算），要么显式声明为"不可回读的历史快照"（写完只用于展示/审计，不再驱动逻辑）。
+2. **严禁双向同步**。如果 A 是权威、B 是缓存，B 只能由 A 单向派生；绝不存在"B 改了回写 A"或"A、B 互相更新"的路径。
+3. **冷重启路径必须重新走权威源**。任何在内存/进程里持有的状态（limit、pid、status）一旦进程退出就丢失；重启时只能从权威文件读，不能从 requirements.json / form 副本读"为了方便"。
+4. **新增字段时先问"谁是权威"**。不要图省事把值复制到第二个文件——短期的省事会变成长期的 bug 工厂。
+5. **历史快照必须标注**。某文件如果只是建任务时的表单记录（之后不再驱动运行时），必须在 schema 注释里写明："historical record, runtime reads from <other_file>"。
+
+### 已确立的权威源（参考）
+
+| 数据 | 权威源 | 历史快照 / 派生 |
+|---|---|---|
+| 预算阈值 | `token_budget.json::config.max_cost_usd` | `requirements.json::token_budget_max_cost_usd`（建任务时表单值，运行时不再读） |
+| 预算累计 | `token_budget.json::totals` | `timeline.jsonl` 的 `token_usage` 事件（展示用，从权威派生） |
+| 运行时状态 | `run.json`（phase / iteration / **finished / final_status**） | `registry.json`（**仅身份**：id/type/label/state_dir/workspace_dir/created_at/launcher。**绝不**缓存进程状态） |
+| 任务规格 | `requirements.json`（task_type / form / label / created_at） | `registry.json::type/label`（缓存）；run.json 不再存 task_type |
+| 进程存活 | OS 进程表（`/proc/<pid>`）+ `orchestrator.pid`（pid / started_at / finished_at / exit_hint） | `runtime.json::tasks.<id>`（仅 WebUI session 用 boot_id 标记归属，不作为状态查询源）；**registry.json 不存进程状态** |
+| 进程死亡清理 | `launcher._reap_dead_pid_file()`（单一 reap 路径） | reconcile / liveness / kill 都**调它**，禁止另写 `_write_pid_file_finished` 这种只更新部分文件的简化版 |
+
+### 已知反模式（**禁止**）
+
+- **双写**：同一字段被两个文件各持一份，且都被运行时读取 → 必然漂移。
+  - 已修复的例子：`requirements.json::token_budget_max_cost_usd` 和 `token_budget.json::config.max_cost_usd` 曾经都被读，导致 WebUI 调整预算后冷重启失效（commit 待补）。
+  - 已修复的例子：`task_type` 曾经同时存在 requirements.json / run.json / registry.json，已从 run.json 移除（orchestrator 加载时 load_run 过滤未知字段，兼容旧文件）。
+  - 已修复的例子：`created_at` 曾经同时存在 registry.json / run.json，已从 run.json 移除（registry.json::created_at 是唯一权威源）。
+  - 已修复的例子：进程状态 (pid / started_at / finished_at) 曾经**三处存储** —— `orchestrator.pid` / `runtime.json::tasks.<id>` / `registry.json::tasks[]`。registry 那份名义上是"派生缓存",实际**没有任何派生函数**,reconcile / _reap_dead_pid_file / kill 各自选择性同步;`tasks.update_task` 里 `if v is None: continue` 还静默吞掉了 `pid=None` 的清除语义,导致死任务的 registry 永远显示 stale pid,liveness 用它做 pre-filter 时直接走错路。已**从 registry 移除 pid/started_at/finished_at 字段**,所有进程状态查询只走 `launcher.status()` 读 `orchestrator.pid`;旧 registry.json 通过 `_strip_legacy` 兼容。
+- **多条 reap 路径效果不一致**：reconcile 原来用自己的 `_write_pid_file_finished`(只碰 orchestrator.pid),而 liveness 用 `launcher._reap_dead_pid_file`(还会刷 run.json + 写 timeline)。两条路径 → 同样的死亡事件,UI 拿到的信号不一致。**任何"清理死亡任务"的代码都必须调 `launcher._reap_dead_pid_file`**,禁止另写简化版。
+- **构造函数参数压过文件**：构造函数从 A 文件读值传入，`_load()` 看到"非 None"就跳过 B 文件——这等价于把 A 钉死为权威。正确做法是构造函数只传"env override"，文件值由 `_load()` 单独决定。
+- **多 task 包复制同一份解析逻辑**：每个 task orchestrator 自己实现一遍 cascade → 修一个 bug 要改 N 处。共享逻辑下沉到 `metainfer/orchestrator/` 公共层。
+- **字段别名 + 多 reader 各写一份 fallback 链**：例如 requirements.json 曾经既支持扁平 `target_model` 又支持嵌套 `answers.target_model` / `form.target_model`，每个 reader 自己写 `req.get("x") or (req.get("answers") or {}).get("x")` —— 12+ 处复制，每处 null 处理略有不同。已加 `metainfer.orchestrator.requirements.req_field()` 统一读取，所有 task 包的读取都应走这个 helper。
+
+### requirements.json 扁平化规约
+
+WebUI 的 `create_task` 把表单 answers **扁平展开**到顶层（`{"task_id":..., "target_model":..., "max_iterations":"50", ...}`），没有 `answers` 或 `form` 子键。
+
+- **写**：只写扁平。新代码不要在 requirements.json 里塞 `answers` / `form` 子字典。
+- **读**：用 `metainfer.orchestrator.requirements.req_field(req, key)` / `req_field_int` / `req_field_float`。helper 内部保留对历史嵌套形式的兼容（旧文件、test fixture），但 production 路径只走扁平。
+- **新加字段**：在 task 的 `form.yaml` 里声明 → WebUI 自动写入扁平顶层 → reader 用 `req_field` 读。不需要改 requirements.json 的 schema 文档。
+
+### Code review 检查清单
+
+提交前自问：
+- [ ] 我新增/修改的字段，是否已经有别的文件存了？如果是，谁是权威？
+- [ ] 我的代码读这个字段时，读的是权威源，还是某个缓存？
+- [ ] 冷重启后，我的逻辑还能拿到正确值吗？（写一个测试覆盖 restart 场景）
+- [ ] 我有没有把"派生量"当"权威量"写到磁盘？（派生量应每次计算，不持久化）
+
 ## 运行时目录结构
 
 每个 task 占用 **两个并列子树**，挂在 `$METAINFER_ROOT/nodes/<node_id>/` 下：

README.md

@@ -30,9 +30,13 @@
 </p>
 
 <p align="center">
-  <a href="https://star-history.com/#MetaInfer/MetaInfer&Date">
-    <img src="https://api.star-history.com/svg?repos=MetaInfer/MetaInfer&type=Date" alt="Star History" width="600">
-  </a>
+ <a href="https://www.star-history.com/?type=date&repos=MetaInfer%2FMetaInfer">
+ <picture>
+   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=MetaInfer/MetaInfer&type=date&theme=dark&legend=top-left&sealed_token=7N_57a34GhT7taYXyy9U_E1V_9P1i7A_0PK4Am3dOHxXcNvtk9CuxadGB6B1ZCyS0Zsa2rq_z1U0OmRgz9YDhWs5IaomukOlrF5zq5eapw47cM1rYdOKnQ" />
+   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=MetaInfer/MetaInfer&type=date&legend=top-left&sealed_token=7N_57a34GhT7taYXyy9U_E1V_9P1i7A_0PK4Am3dOHxXcNvtk9CuxadGB6B1ZCyS0Zsa2rq_z1U0OmRgz9YDhWs5IaomukOlrF5zq5eapw47cM1rYdOKnQ" />
+   <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=MetaInfer/MetaInfer&type=date&legend=top-left&sealed_token=7N_57a34GhT7taYXyy9U_E1V_9P1i7A_0PK4Am3dOHxXcNvtk9CuxadGB6B1ZCyS0Zsa2rq_z1U0OmRgz9YDhWs5IaomukOlrF5zq5eapw47cM1rYdOKnQ" />
+ </picture>
+</a>
 </p>
 
 ---
@@ -75,6 +79,14 @@
 
 ## Quick start
 
+### Step 1: Install ccb (MetaInfer currently uses the open-source Claude Code CLI; other coding agents are not yet supported — contributions welcome)
+
+Open-source ccb repository: https://github.com/claude-code-best/claude-code
+```
+npm i -g claude-code-best
+```
+
+### Step 2: Install MetaInfer
 ```bash
 git clone https://github.com/MetaInfer/MetaInfer.git
 cd MetaInfer
@@ -83,7 +95,7 @@ pip install -r requirements.txt
 ```
 
 Open [http://127.0.0.1:8765](http://127.0.0.1:8765), click **+ New Task**,
-pick a task type, describe your requirements, and the LLM gets to work.
+pick a task type, fill in your requirements, and the LLM gets to work.
 
 ```bash
 # Other ways to start
@@ -96,5 +108,26 @@ python -m metainfer.server.app
 
 MIT
 
+## Academic Research
+
+The initial ideas and experimental data of MetaInfer are publicly available
+at https://arxiv.org/abs/2607.12875. The related code is on the `arxiv-paper` branch.
+
+Citation:
+
+```
+@misc{miao2026metainferknowledgellminference,
+      title={MetaInfer: A Knowledge Only LLM Inference Engine Generator SKILL Toolbox}, 
+      author={Zhenwen Miao and Honglin Wang and Mingheng Mi},
+      year={2026},
+      eprint={2607.12875},
+      archivePrefix={arXiv},
+      primaryClass={cs.MA},
+      url={https://arxiv.org/abs/2607.12875}, 
+}
+```
+
+## Contributing
+
 See [CONTRIBUTING.md](CONTRIBUTING.md) for architecture details, design
 principles, and how to add new task types.

README_CN.md

@@ -30,9 +30,13 @@
 </p>
 
 <p align="center">
-  <a href="https://star-history.com/#MetaInfer/MetaInfer&Date">
-    <img src="https://api.star-history.com/svg?repos=MetaInfer/MetaInfer&type=Date" alt="Star History" width="600">
-  </a>
+  <a href="https://www.star-history.com/?type=date&repos=MetaInfer%2FMetaInfer">
+  <picture>
+   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=MetaInfer/MetaInfer&type=date&theme=dark&legend=top-left&sealed_token=7N_57a34GhT7taYXyy9U_E1V_9P1i7A_0PK4Am3dOHxXcNvtk9CuxadGB6B1ZCyS0Zsa2rq_z1U0OmRgz9YDhWs5IaomukOlrF5zq5eapw47cM1rYdOKnQ" />
+   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=MetaInfer/MetaInfer&type=date&legend=top-left&sealed_token=7N_57a34GhT7taYXyy9U_E1V_9P1i7A_0PK4Am3dOHxXcNvtk9CuxadGB6B1ZCyS0Zsa2rq_z1U0OmRgz9YDhWs5IaomukOlrF5zq5eapw47cM1rYdOKnQ" />
+   <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=MetaInfer/MetaInfer&type=date&legend=top-left&sealed_token=7N_57a34GhT7taYXyy9U_E1V_9P1i7A_0PK4Am3dOHxXcNvtk9CuxadGB6B1ZCyS0Zsa2rq_z1U0OmRgz9YDhWs5IaomukOlrF5zq5eapw47cM1rYdOKnQ" />
+ </picture>
+</a>
 </p>
 
 ---
@@ -75,6 +79,14 @@
 
 ## 快速开始
 
+### 第一步，安装ccb。（本项目目前使开源的claude code版本，暂不支持其他coding agent，欢迎贡献代码以支持更多coding agent）
+
+开源ccb项目地址：https://github.com/claude-code-best/claude-code
+```
+npm i -g claude-code-best
+```
+
+### 第二步，安装MetaInfer
 ```bash
 git clone https://github.com/MetaInfer/MetaInfer.git
 cd MetaInfer
@@ -96,4 +108,22 @@ python -m metainfer.server.app
 
 MIT
 
+## 学术研究
+
+MetaInfer的最初想法和实验数据已经公开在https://arxiv.org/abs/2607.12875，相关代码位于`arxiv-paper`分支。
+
+引用信息：
+```
+@misc{miao2026metainferknowledgellminference,
+      title={MetaInfer: A Knowledge Only LLM Inference Engine Generator SKILL Toolbox}, 
+      author={Zhenwen Miao and Honglin Wang and Mingheng Mi},
+      year={2026},
+      eprint={2607.12875},
+      archivePrefix={arXiv},
+      primaryClass={cs.MA},
+      url={https://arxiv.org/abs/2607.12875}, 
+}
+```
+
+## 如何贡献
 架构细节、设计理念和新任务类型添加方法见 [CONTRIBUTING.md](CONTRIBUTING.md)。

metainfer/orchestrator/requirements.py

@@ -0,0 +1,134 @@
+"""Helpers for reading ``requirements.json`` consistently across task
+orchestrators.
+
+Single source of truth for the requirements.json schema
+-------------------------------------------------------
+
+The WebUI's ``create_task`` endpoint writes a **flat** JSON object — the
+form answers are spread to the top level alongside ``task_id`` /
+``task_type`` / ``label`` / ``raw_request``:
+
+    {
+      "task_id": "...",
+      "task_type": "...",
+      "label": "...",
+      "raw_request": "...",
+      "target_model": "...",       ← was in form "answers"
+      "max_iterations": "50",      ← was in form "answers"
+      ...
+    }
+
+There is NO ``answers`` or ``form`` sub-key. Some early code and test
+fixtures wrote a nested ``{"form": {...}}`` or ``{"answers": {...}}``
+shape; that is legacy and not produced by the WebUI anymore.
+
+To avoid every orchestrator re-implementing the same "flat key first,
+fall back to nested for legacy" logic (which drifts — see the
+``token_budget`` cascade bug for how that pattern burns), all readers
+should go through :func:`req_field`.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, Optional
+
+
+# Keys that the orchestrator framework itself writes — task code should
+# treat these as opaque identity/metadata, not as part of the form.
+RESERVED_KEYS = frozenset({"task_id", "task_type", "label", "raw_request"})
+
+
+def req_field(
+    req: Dict[str, Any], key: str, default: Any = None,
+) -> Any:
+    """Read one field from ``requirements.json`` — single source of truth.
+
+    Resolution order:
+      1. Top-level flat key (canonical — what the WebUI writes).
+      2. ``req["form"][key]`` (legacy nested form).
+      3. ``req["answers"][key]`` (legacy nested answers).
+      4. ``default``.
+
+    The legacy nested fallbacks exist ONLY for backward compatibility
+    with old task files on disk / hand-written test fixtures. Production
+    WebUI output is flat, so resolution stops at step 1 in practice.
+
+    Always use this helper instead of hand-writing
+    ``req.get("x") or (req.get("answers") or {}).get("x")`` — that
+    pattern got duplicated across 12+ call sites, each with subtly
+    different null-handling, and was the root cause of at least one
+    "limit silently lost" bug. Centralizing the read keeps the schema
+    honest.
+    """
+    if not isinstance(req, dict):
+        return default
+    if key in req:
+        return req[key]
+    for ns in ("form", "answers"):
+        bucket = req.get(ns)
+        if isinstance(bucket, dict) and key in bucket:
+            return bucket[key]
+    return default
+
+
+def req_field_int(
+    req: Dict[str, Any], key: str, default: Optional[int] = None,
+) -> Optional[int]:
+    """Like :func:`req_field` but coerces to int. Returns ``default``
+    on missing/unparseable values. Useful for fields like
+    ``max_iterations`` that the form may emit as a string."""
+    v = req_field(req, key, None)
+    if v is None:
+        return default
+    try:
+        return int(v)
+    except (TypeError, ValueError):
+        return default
+
+
+def req_field_float(
+    req: Dict[str, Any], key: str, default: Optional[float] = None,
+) -> Optional[float]:
+    """Like :func:`req_field` but coerces to float."""
+    v = req_field(req, key, None)
+    if v is None:
+        return default
+    try:
+        return float(v)
+    except (TypeError, ValueError):
+        return default
+
+
+def req_summary_lines(req: Dict[str, Any]) -> List[str]:
+    """Format the flat requirement fields as ``- key: value`` bullet lines
+    for agent prompts.  Handles the legacy ``answers`` dict nesting for
+    backwards-compat (old WebUI versions that didn't flatten).
+
+    Callers should NOT implement their own iteration + skip set logic —
+    this function is the single source for the per-task frozen-requirements
+    section that appears at the top of orchestrator prompts.
+    """
+    lines = [
+        f"- task_type: {req.get('task_type', '?')}",
+        f"- task_id: {req.get('task_id', '?')}",
+        f"- raw_request: {req.get('raw_request', '')}",
+    ]
+    _skip = {"task_type", "task_id", "raw_request", "answers"}
+    for k, v in req.items():
+        if k in _skip:
+            continue
+        # Type hint / non-single-value fields — format inline; the prompt
+        # engine (claude) handles lists and tuples natively anyway, but
+        # the "- k: v" table makes it skimmable.
+        if isinstance(v, (list, tuple)):
+            v = ", ".join(str(x) for x in v) if v else "(none)"
+        lines.append(f"- {k}: {v}")
+    # Backwards-compat: also surface legacy ``answers`` nesting if present.
+    answers = req.get("answers") or {}
+    for k, v in answers.items():
+        if k in _skip:
+            continue
+        if isinstance(v, (list, tuple)):
+            v = ", ".join(str(x) for x in v) if v else "(none)"
+        lines.append(f"- {k}: {v} (legacy)")
+    return lines