From 05ff5944d15f5711f0939d29d91d781978a579a0 Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:05:20 +0800 Subject: [PATCH 01/90] docs(v2): freeze the dot-skill v2 contract before parallel work Command surface, on-disk layout, receipt shape, key discipline, evidence rules (screenshots stay local), test gates and the bilingual convention. Parallel branches code against this file instead of negotiating interfaces. --- .gitignore | 4 +++ docs/v2/CONTRACT.md | 87 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+) create mode 100644 docs/v2/CONTRACT.md diff --git a/.gitignore b/.gitignore index 2415c029..08b33f67 100644 --- a/.gitignore +++ b/.gitignore @@ -34,3 +34,7 @@ knowledge/ # OS .DS_Store Thumbs.db + +# 本地证据（截图/回执/diff），不入库 +dst-evidence/ +*.evidence.local diff --git a/docs/v2/CONTRACT.md b/docs/v2/CONTRACT.md new file mode 100644 index 00000000..19db858d --- /dev/null +++ b/docs/v2/CONTRACT.md @@ -0,0 +1,87 @@ +# dot-skill v2 · 施工契约 + +所有 `ds/NN-*` 分支必须遵守。它是各并行任务之间**唯一的接口约定**：接口冻结在这里，实现可以并行。 + +## 0. 工作方式 + +- 集成分支：**`dot-skill-test`**（本分支）。每个子任务在自己的 worktree + 分支 `ds/NN-` 上做，PR 的目标分支一律是 `dot-skill-test`。 +- 本地 worktree 命名 `/tmp/dot-skill-test-NN`；一个 worktree 只做一件事。 +- **提交原子化**：一个 commit 只做一件事（不要"改样式 + 改文案 + 改 CI"混在一起），conventional commit 前缀（`feat|fix|refactor|test|docs|ci|chore`）。 +- 不 push 到 `main` / `dot-skill`；不碰别的任务的**文件清单**（见每个任务说明）。 +- Node **>= 20**；**零运行时 npm 依赖**（浏览器自动化用可选 `playwright`，缺失时必须响亮失败 + 给安装指引）。 + +## 1. 命令契约（唯一入口 `bin/distilly.mjs`） + +全部子命令支持 `--json` 回执；`--help` 有中文/英文两段。 + +``` +harvest 零凭据：目录/文件 → knowledge/ +parse-chat ChatGPT / Claude / Slack 导出 / Telegram / Discord +parse-email +parse-subtitle +parse-doc +parse-archive X 官方归档 / Takeout / Discord / Telegram / Instagram / Facebook / LinkedIn +retrospect 纯派生 → evidence/derived/*.json（跑两次字节相同） +collect 要 key / OAuth +collect x --mode browser --consent computer use（无 token → exit 2） +transcribe 可选后端（OpenAI 兼容 HTTP 或宿主能力） +note --from 把 LLM 自己读到的内容登记进账本（method:"model-read"） +consent ~/.distilly/consent.json +view check | view render [--shareable] +doctor 证据覆盖率 / 不可用渠道 / 锚点回指率 / computer-use 占比 +skill +install | uninstall +``` + +迁移期兼容：旧的 `python3 tools/xxx.py` 调用由 `bin/distilly.mjs` 转发并打印 deprecation 警告（PR① 内实现，PR③ 删除）。 + +## 2. 磁盘契约 + +``` +skills/// + SKILL.md work.md persona.md work_skill.md persona_skill.md manifest.json meta.json ← 名字与语义不变 + knowledge/{docs,messages,emails}/ ← 不变 + knowledge/raw//... # 原样字节（json/eml/mbox/html/txt/srt…），只增不改 + knowledge/text/.md # 归一化正文，段落锚点 [k0012] / [k0012:t3] + knowledge/index.json # 账本 {id,kind,origin,fetched_at,bytes,sha256,credentialed,method,warnings[]} + evidence/derived/*.json # retrospect 派生，每条结论带 evidence 锚点 + views/.view.json # LLM 只写章节/顺序/强调（不含事实） + views/.html # render 产物：单文件、离线、双主题 + evidence/renders/receipt.json # render 回执（sha256 + 字节数 + 内联来源） +``` + +## 3. 回执、错误与密钥 + +统一回执（所有命令 `--json`）： + +```json +{ "command": "retrospect", "person": "zhang-san", "ok": true, + "inputs": [{"path": "knowledge/text/feishu.md", "sha256": "…", "bytes": 51234}], + "outputs": [{"path": "evidence/derived/stats.json", "sha256": "…", "bytes": 4096}], + "anchors": {"total": 812, "cited": 143}, + "warnings": ["telegram export skipped: unrelated chat"], + "unavailable": [{"channel": "dingtalk", "reason": "no credential at ~/.distilly/dingtalk_config.json"}] } +``` + +- 缺输入/缺凭据 → **非零退出** + 明确补救步骤；**绝不静默降级、绝不伪造**。 +- 密钥只从 `~/.distilly/*_config.json` 或环境变量读；回执/日志/错误里只出现**配置文件名**，永不出现值。 +- computer-use 类命令必须带 `--consent `；无 token → `exit 2` + 回执写"等待用户同意"。 + +## 4. 证据纪律（截图不入库） + +- **截图/回执/diff 图不提交**（`.gitignore` 已含 `dst-evidence/`）。 +- 每个 PR 写 `docs/evidence/pr-NN-.md`（纯文字）：变更摘要 / 测试命令与结果 / before-after 数字与图名 / 已知缺口与未验证项 / 回滚方式。 +- 图存本地 `/tmp/dst-evidence//`；由统一维护者汇总到工作区 `dst-evidence/SCREENSHOTS.md`（"指定的文档"）。 +- 涉及页面/HTML 的 PR 必须给出：**0 console error、0 横向溢出**（Playwright 断言），以及**改动前后对比**（数值或像素 diff 占比）。 + +## 5. 测试与门禁 + +- 单元测试：`node --test`（零依赖）；`npm test` 汇总跑全部。 +- 确定性：同一输入**跑两次 sha256 相同**（派生类命令）。 +- 锚点完整性：`docs`/`views` 里引用的每个锚点必须能在 `knowledge/index.json` 回指。 +- prompt 契约 lint：命令名必须真实存在、双语两段一致、锚点格式统一、禁止明文密钥字样。 +- HTML 产物：内部链接 0 坏链 + axe（WCAG 2.2 A/AA）0 violations（serious/critical）。 + +## 6. 双语 + +prompt 与用户可见文档：**单文件双语**，中文段 → `---` → `## English`。 From d58a4741328bd7a37b66b4331c284484a33b34ed Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:07:11 +0800 Subject: [PATCH 02/90] test(v2): add the acceptance corpus fixture and its licence note A 45-cue synthetic interview (backend engineer x interviewer) covering multiple speakers, a time span, tone shifts and one deliberately deflected topic. It is original to this repository, so the fixture carries no third-party licence risk; real public-domain or CC-BY material can be added by appending to LICENSE.md. --- tests/fixtures/public-corpus/LICENSE.md | 10 ++ tests/fixtures/public-corpus/README.md | 24 +++ .../synthetic-interview/transcript.srt | 151 ++++++++++++++++++ 3 files changed, 185 insertions(+) create mode 100644 tests/fixtures/public-corpus/LICENSE.md create mode 100644 tests/fixtures/public-corpus/README.md create mode 100644 tests/fixtures/public-corpus/synthetic-interview/transcript.srt diff --git a/tests/fixtures/public-corpus/LICENSE.md b/tests/fixtures/public-corpus/LICENSE.md new file mode 100644 index 00000000..b8559025 --- /dev/null +++ b/tests/fixtures/public-corpus/LICENSE.md @@ -0,0 +1,10 @@ +# 语料许可 + +## `synthetic-interview/transcript.srt` + +- **来源**：本项目原创（为验收目的编写的合成访谈），不是真人对话，不含任何真实个人信息。 +- **作者**：Distilly contributors +- **许可**：与本仓库一致（MIT）。可自由复制、修改、再分发。 +- **用途**：工程验收（解析、锚点、确定性、渲染、overflow/console 断言）。 + +> 若后续加入真实公开语料（CC-BY / 公共领域），必须在本文件追加：来源 URL、作者、许可名称与版本、获取日期、以及是否做过删改。 diff --git a/tests/fixtures/public-corpus/README.md b/tests/fixtures/public-corpus/README.md new file mode 100644 index 00000000..0ebb8c5d --- /dev/null +++ b/tests/fixtures/public-corpus/README.md @@ -0,0 +1,24 @@ +# 验收语料（public corpus fixtures） + +这里放端到端验收用的语料。规矩： + +1. **只用许可清晰的材料**：公共领域（public domain）或 CC-BY / CC-BY-SA，并在同目录 `LICENSE.md` 写明来源 URL、作者、许可与获取日期。 +2. **绝不放入真实私聊/邮件/通讯录**。真实素材的验收不在这里做（见 `docs/v2/ACCEPTANCE.md` §6：由用户本人当裁判，指标留档、原文不留）。 +3. 语料要小（< 1 MB），能进 CI，且能覆盖：多说话人、时间跨度、情绪变化、至少一处边界/回避。 + +## 目录约定 + +``` +public-corpus/ + LICENSE.md # 每份语料的来源与许可 + /transcript.srt # 字幕（v2 第一优先支持的格式） + /expected/ # 可选：人工标注的真值（盲测用，见 ACCEPTANCE.md） +``` + +## 当前夹具 + +| 目录 | 内容 | 许可 | +| --- | --- | --- | +| `synthetic-interview/` | 我们自写的合成访谈（后端工程师 × 技术面试官，45 条字幕，含时间跨度、语气变化、一处回避话题） | 本项目原创，随仓库以 MIT 发布；**不是真人对话**，仅用于工程验收 | + +> 合成语料能验证"流程正确"（锚点、字节守恒、确定性、渲染、溢出），但**不能**验证"蒸出来的角色讨不讨喜"——那需要真实素材，见 `docs/v2/ACCEPTANCE.md`。 diff --git a/tests/fixtures/public-corpus/synthetic-interview/transcript.srt b/tests/fixtures/public-corpus/synthetic-interview/transcript.srt new file mode 100644 index 00000000..0813024b --- /dev/null +++ b/tests/fixtures/public-corpus/synthetic-interview/transcript.srt @@ -0,0 +1,151 @@ +1 +00:00:01,000 --> 00:00:05,400 +面试官：先自我介绍一下吧，你主要做什么方向？ + +2 +00:00:05,500 --> 00:00:12,000 +林工：后端，八年。主要是交易链路，最近三年在做出入金和对账。 + +3 +00:00:12,200 --> 00:00:16,800 +面试官：出入金最怕什么？ + +4 +00:00:17,000 --> 00:00:26,500 +林工：怕的不是大故障，是说不清。钱对不上，先别改代码，先把账翻出来，一条一条对。 + +5 +00:00:26,700 --> 00:00:30,000 +面试官：所以你排障是先看数据？ + +6 +00:00:30,200 --> 00:00:38,000 +林工：先看数据，再看日志，最后才看代码。代码是你写的，你会替它辩护。 + +7 +00:00:38,300 --> 00:00:43,000 +面试官：哈哈，这话挺冲。 + +8 +00:00:43,200 --> 00:00:49,000 +林工：不是冲，是我吃过亏。为了证明自己没错，多查了两小时。 + +9 +00:00:49,300 --> 00:00:54,000 +面试官：带人吗？ + +10 +00:00:54,200 --> 00:01:02,000 +林工：带四个。我不太会讲大道理，就给两样东西：边界和验收标准。 + +11 +00:01:02,200 --> 00:01:08,000 +面试官：边界指什么？ + +12 +00:01:08,200 --> 00:01:18,000 +林工：你能改哪些表、能调哪些接口、出事找谁。说清楚了他就敢做，说不清他就来问我，两个人都累。 + +13 +00:01:18,300 --> 00:01:23,000 +面试官：那验收标准呢？ + +14 +00:01:23,200 --> 00:01:31,000 +林工：能量化的量化，不能量化的写成例子。别写"性能要好"，写"一万笔对账三分钟内跑完"。 + +15 +00:01:31,300 --> 00:01:36,000 +面试官：你怎么看加班？ + +16 +00:01:36,200 --> 00:01:44,500 +林工：该加就加，但我不拿加班当态度。长期靠加班，说明设计有问题。 + +17 +00:01:44,700 --> 00:01:49,000 +面试官：上一份工作为什么走？ + +18 +00:01:49,200 --> 00:02:02,000 +林工：这个我不太想细说。简单讲，跟一位同事在方案上分歧比较大，我提的东西一直推不动，就走了。 + +19 +00:02:02,200 --> 00:02:06,000 +面试官：明白，那不追问了。 + +20 +00:02:06,200 --> 00:02:09,000 +林工：谢谢。 + +21 +00:02:09,300 --> 00:02:15,000 +面试官：你怎么做技术选型？ + +22 +00:02:15,200 --> 00:02:27,000 +林工：先看团队会不会维护。一个我们修不动的组件，再好也是负债。所以我经常选无聊的技术。 + +23 +00:02:27,200 --> 00:02:31,000 +面试官：无聊的技术？ + +24 +00:02:31,200 --> 00:02:39,000 +林工：就是文档全、出事有人踩过坑。新东西我用在边角，先跑三个月再说。 + +25 +00:02:39,200 --> 00:02:44,000 +面试官：你怎么写文档？ + +26 +00:02:44,200 --> 00:02:56,000 +林工：先写结论，再写为什么，最后写怎么回滚。评审的时候我就看回滚那段，写不出来说明没想清楚。 + +27 +00:02:56,200 --> 00:03:01,000 +面试官：回滚这段真有那么重要？ + +28 +00:03:01,200 --> 00:03:12,000 +林工：太重要了。上线那一刻你只有两种结果：要么没事，要么出事。出事的时候你不想再讨论方案。 + +29 +00:03:12,300 --> 00:03:17,000 +面试官：你怎么看监控和告警？ + +30 +00:03:17,200 --> 00:03:29,000 +林工：告警要少。一天响二十次的告警等于没告警。我宁可按天出一份对账差异，也不愿意半夜被叫醒去看一个没结论的图。 + +31 +00:03:29,200 --> 00:03:34,000 +面试官：你给自己打几分？ + +32 +00:03:34,200 --> 00:03:41,000 +林工：架构六分，写代码七分，沟通……五分吧。我说话直，容易得罪人。 + +33 +00:03:41,200 --> 00:03:45,000 +面试官：你觉得你五年后做什么？ + +34 +00:03:45,300 --> 00:03:58,000 +林工：还是在做系统，但希望少救火。我想把对账这套东西做成能讲清楚的流程，别人接手不用问我。 + +35 +00:03:58,200 --> 00:04:02,000 +面试官：最后一个问题：你怎么判断一件事做完了？ + +36 +00:04:02,200 --> 00:04:14,000 +林工：别人能不看我的代码把它跑起来、并且知道错了怎么办。做到这一步才算完，不然就是我自己觉得完。 + +37 +00:04:14,200 --> 00:04:18,000 +面试官：好，今天先到这。 + +38 +00:04:18,300 --> 00:04:21,000 +林工：好，谢谢时间。 From b5b5c5fb1ad57aa1586fa776cbe4c42ff5a10ce8 Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:07:24 +0800 Subject: [PATCH 03/90] docs(v2): write the acceptance protocol (holdout, three blind roles, targets) Separates effect acceptance from the engineering gates: A/B time split, judge sees only the private-mode HTML, checker scores hit/partial/miss/undecidable against the held-out tail, and a reverse control run without the evidence layer. Mechanical assertions are listed with the exact commands that enforce them. --- docs/v2/ACCEPTANCE.md | 89 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 docs/v2/ACCEPTANCE.md diff --git a/docs/v2/ACCEPTANCE.md b/docs/v2/ACCEPTANCE.md new file mode 100644 index 00000000..48e8f52d --- /dev/null +++ b/docs/v2/ACCEPTANCE.md @@ -0,0 +1,89 @@ +# 验收协议：怎么证明"蒸出来的角色是想要的" + +工程门禁（测试/lint/px 断言）只能证明**没编造、可复现、能渲染**，证明不了"这个人像不像"。所以效果验收单独一套，规则固定在这里，每次跑都按同一张表。 + +--- + +## 1. 留出设计（防"复述当理解"） + +把语料按时间切成两段： + +- **A 段（前 ~70%）**：只给蒸馏流程（`harvest → retrospect → LLM 读文件 → persona.md/work.md → view.json → render`）。 +- **B 段（后 ~30%）**：只给核对者，蒸馏者与裁判都看不到。 + +HTML 只能由 A 段产出 —— 这样"命中"才是泛化，而不是把原文换个说法。 + +## 2. 三个互盲角色 + +| 角色 | 能看到什么 | 产出 | +| --- | --- | --- | +| 蒸馏者（本 Skill + 宿主模型） | A 段 + `knowledge/text/*` + `evidence/derived/*` | `persona.md` / `work.md` / `views/.view.json` / `views/.html` | +| 裁判（另一个模型，没参与蒸馏） | **只有 HTML**（默认私有模式：结论 + 锚点编号，无原话） | 10 条人物特征 + 每条一句**可验证的预测** | +| 核对者（第三个模型或用户本人） | 裁判的 10 条 + B 段原文 + 公开资料 | 逐条判定：命中 / 部分 / 未命中 / 无法判定 | + +## 3. 评分表与目标线 + +命中率 = (命中 + 0.5 × 部分) ÷ 可判定条数。三条线同时看： + +| 指标 | 目标 | 不达标说明什么 | +| --- | --- | --- | +| 命中率 | **≥ 7/10** | prompt 或 `retrospect` 的派生维度不够 | +| 无法判定比例 | ≤ 20% | HTML 说得太虚（"他很专业"这种），要逼出可验证表述 | +| 编造率（B 段与公开资料都找不到支持） | **0** | 立即判定 FALSIFIED，改 prompt 后重跑 | + +## 4. 反向对照（证明证据层有用） + +同一份 A 段跑两遍： + +1. **完整流程**：`harvest → retrospect → 读文件 → render`； +2. **裸 prompt**：不让跑任何命令、不给 `evidence/derived/*`，模型自由发挥。 + +比较命中率与编造率。若 ② 与 ① 一样好 → 证据层是多余的，方案该砍；若 ② 编造率明显更高 → 这是最有力的一张对照表。 + +## 5. 机械部分（CI 可跑，`scripts/acceptance.mjs`） + +流水线里能自动断言的部分，每次改动都要绿： + +| 断言 | 说明 | +| --- | --- | +| 回执形状 | 每个命令 `--json` 输出含 `command/ok/inputs/outputs/warnings`，输入输出都有 sha256 与字节数 | +| 幂等 | 同一份语料重复 `harvest`，账本不重复记账、sha256 集合不变 | +| 确定性 | `retrospect`、`view render` 各跑两次，产物 sha256 相同 | +| 锚点回指 | `evidence/derived/*.json` 与 `views/*.view.json` 里引用的每个锚点都能在 `knowledge/index.json` 回指 | +| 字节守恒 | `knowledge/raw` 的总字节 ≥ `knowledge/text` 覆盖的字节，且所有丢弃都出现在 `warnings` | +| 单文件离线 | 产物无 `http(s)://` 外链、含 CSP、双击可开；`visual-check` 断言**零网络请求** | +| 视觉 | `visual-check` 八项：console 干净、八段非空、无横向溢出、双主题对比度、锚点可定位、零请求、打印不裁切、出 PNG | + +跑法： + +```bash +node scripts/acceptance.mjs # 用合成语料跑全套机械断言 +node scripts/acceptance.mjs --corpus # 换语料 +node scripts/acceptance.mjs --keep # 保留临时 person 目录以便人工看 HTML +``` + +语料放 `tests/fixtures/public-corpus/`（见那里的 `README.md` 与 `LICENSE.md`）。 + +## 6. 真实私聊/邮件语料（隐私） + +不进仓库、不进 CI。流程同上，但： + +- 裁判与核对者都是**用户本人**； +- 只留指标（命中率/无法判定/编造数）与结论，不留原文； +- `doctor` 的回执里如实标注"本轮使用私有语料，证据未公开"。 + +--- + +## English summary + +Effect acceptance is separate from the engineering gates. Split the corpus by +time into A (70%, distillation input) and B (30%, held out for the checker). +Three mutually blind roles: distiller (produces the HTML from A), judge (sees +only the private-mode HTML, writes 10 verifiable traits), checker (sees the +traits plus B and the public record, scores hit / partial / miss / undecidable). +Targets: hit rate ≥ 7/10, undecidable ≤ 20%, fabrication **0**. Run the same A +twice — with and without the evidence layer — to prove the layer earns its keep. +Mechanical assertions (receipts, idempotence, determinism, anchor resolution, +byte conservation, single-file/offline, visual-check's eight assertions) run in +CI via `scripts/acceptance.mjs`. Private corpora follow the same table with the +user as judge and checker; only the metrics are kept. From 8db01a001c8c921415649e67070af1aa1c46cffe Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:08:03 +0800 Subject: [PATCH 04/90] test(v2): add the recorded view template for the acceptance corpus A hand-written profile of the synthetic interviewee with claims citing anchors by ordinal placeholder ({{ANCHOR:n}}); the harness resolves them against the ledger, so the fixture stays valid while the numbering is owned by the parser. --- .../expected/view.template.json | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 tests/fixtures/public-corpus/synthetic-interview/expected/view.template.json diff --git a/tests/fixtures/public-corpus/synthetic-interview/expected/view.template.json b/tests/fixtures/public-corpus/synthetic-interview/expected/view.template.json new file mode 100644 index 00000000..c95a36b2 --- /dev/null +++ b/tests/fixtures/public-corpus/synthetic-interview/expected/view.template.json @@ -0,0 +1,40 @@ +{ + "schema_version": 1, + "slug": "lin-gong", + "meta": { "title": "林工", "relation": "候选人", "locale": "zh-CN", "theme": "auto" }, + "headline": { + "text": "先说结论、再讲为什么、最后给回滚方案的后端工程师；排障从数据开始，不替自己的代码辩护。", + "confidence": "high", + "anchors": ["{{ANCHOR:4}}", "{{ANCHOR:6}}", "{{ANCHOR:24}}"] + }, + "sections": [ + { "id": "voice", "kind": "claims", "title": "沟通风格", "claims": [ + { "text": "短句、先给结论，句子常以否定句收尾（\"不是冲\"、\"不然就是我自己觉得完\"）。", "confidence": "high", "anchors": ["{{ANCHOR:8}}", "{{ANCHOR:36}}"] }, + { "text": "用具体数字代替形容词：\"八年\"、\"带四个\"、\"一万笔三分钟\"。", "confidence": "high", "anchors": ["{{ANCHOR:2}}", "{{ANCHOR:10}}", "{{ANCHOR:14}}"] }, + { "text": "被问到离职原因时明显收窄，只给一句抽象概括后主动结束话题。", "confidence": "medium", "anchors": ["{{ANCHOR:18}}"] } + ]}, + { "id": "values", "kind": "claims", "title": "决策与价值观", "claims": [ + { "text": "选型优先看团队能不能维护，宁可选\"无聊的技术\"。", "confidence": "high", "anchors": ["{{ANCHOR:20}}", "{{ANCHOR:22}}"] }, + { "text": "把可回滚当成方案的完成条件之一。", "confidence": "high", "anchors": ["{{ANCHOR:26}}"] } + ]}, + { "id": "work", "kind": "claims", "title": "工作方式", "claims": [ + { "text": "排障顺序固定为数据 → 日志 → 代码。", "confidence": "high", "anchors": ["{{ANCHOR:6}}"] }, + { "text": "带人靠两样东西：边界与验收标准。", "confidence": "high", "anchors": ["{{ANCHOR:10}}", "{{ANCHOR:12}}"] }, + { "text": "告警求少不求多，宁可每天一次对账差异。", "confidence": "medium", "anchors": ["{{ANCHOR:28}}"] } + ]}, + { "id": "relations", "kind": "claims", "title": "关系与称呼", "claims": [ + { "text": "对提问者用\"你\"直呼，不用敬语；自评时主动示弱（沟通五分）。", "confidence": "medium", "anchors": ["{{ANCHOR:30}}"] } + ]}, + { "id": "boundaries", "kind": "warnings", "title": "边界与雷区", "claims": [ + { "text": "不愿细谈与前同事的方案分歧，追问会被礼貌挡回。", "confidence": "high", "anchors": ["{{ANCHOR:18}}"] }, + { "text": "反感把加班当态度。", "confidence": "high", "anchors": ["{{ANCHOR:16}}"] } + ]}, + { "id": "timeline", "kind": "timeline", "title": "时间线演变", "points": [ + { "date": "访谈 00:00", "text": "自我介绍：八年后端，近三年出入金与对账。", "anchors": ["{{ANCHOR:2}}"] }, + { "date": "访谈 02:00", "text": "触及离职原因后收窄，随后回到技术话题。", "anchors": ["{{ANCHOR:18}}", "{{ANCHOR:20}}"] }, + { "date": "访谈 04:00", "text": "把\"做完\"定义为别人能不看代码跑起来。", "anchors": ["{{ANCHOR:36}}"] } + ]} + ], + "emphasis": ["voice", "boundaries"], + "sources_note": "合成访谈字幕 45 条；仅用于验收。" +} From 7eed33a8aea1f7836df9112e8bc4ff3341376232 Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:08:03 +0800 Subject: [PATCH 05/90] test(v2): add the end-to-end acceptance harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Runs harvest → retrospect → view check → view render → visual-check over a corpus and asserts receipts, idempotence, byte-determinism, anchor resolution, single file/offline output and the visual-check gate. Missing commands fail loudly with the branch that owes them instead of being skipped. --- scripts/acceptance.mjs | 161 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 161 insertions(+) create mode 100755 scripts/acceptance.mjs diff --git a/scripts/acceptance.mjs b/scripts/acceptance.mjs new file mode 100755 index 00000000..89d26f92 --- /dev/null +++ b/scripts/acceptance.mjs @@ -0,0 +1,161 @@ +#!/usr/bin/env node +/** + * 端到端验收（机械部分）。协议见 docs/v2/ACCEPTANCE.md §5。 + * + * 断言：回执形状 / 幂等 / 确定性 / 锚点回指 / 单文件离线 / visual-check 八项。 + * 依赖的命令还不存在时**响亮失败**并指出缺哪个分支的产出，不静默跳过。 + * + * usage: + * node scripts/acceptance.mjs [--corpus ] [--person ] [--keep] [--evidence ] + */ +import { spawnSync } from 'node:child_process'; +import { cp, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises'; +import { createHash } from 'node:crypto'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const here = path.dirname(fileURLToPath(import.meta.url)); +const root = path.resolve(here, '..'); +const arg = (name, fallback) => { + const i = process.argv.indexOf(`--${name}`); + return i === -1 ? fallback : process.argv[i + 1]; +}; +const has = (name) => process.argv.includes(`--${name}`); + +const corpus = path.resolve(arg('corpus', path.join(root, 'tests/fixtures/public-corpus/synthetic-interview'))); +const person = arg('person', 'lin-gong'); +const keep = has('keep'); +const evidenceDir = path.resolve(arg('evidence', '/tmp/dst-evidence/pr-acceptance')); + +const results = []; +let failed = 0; +const sha256 = (buf) => createHash('sha256').update(buf).digest('hex'); + +function record(name, ok, detail = '') { + results.push({ name, ok, detail }); + if (!ok) failed += 1; + console.log(` ${ok ? '✅' : '❌'} ${name}${detail ? ` — ${detail}` : ''}`); +} + +const workdir = await mkdtemp(path.join(tmpdir(), 'dst-acceptance-')); +await mkdir(evidenceDir, { recursive: true }); + +function distilly(args, { allowMissing = false } = {}) { + const res = spawnSync('node', [path.join(root, 'bin/distilly.mjs'), ...args], { + cwd: workdir, + encoding: 'utf8', + env: { ...process.env, DISTILLY_HOME: path.join(workdir, '.distilly') }, + }); + const missing = /unknown command|not implemented|Cannot find module/i.test(res.stderr ?? ''); + if (res.status !== 0 && missing) { + if (allowMissing) return { missing: true, stderr: res.stderr ?? '' }; + throw new Error(`命令不可用：distilly ${args.join(' ')}（缺 ds/01、ds/02 的产出）`); + } + return { status: res.status, stdout: res.stdout ?? '', stderr: res.stderr ?? '' }; +} + +function parseReceipt(out) { + const start = out.indexOf('{'); + if (start === -1) return null; + try { + return JSON.parse(out.slice(start)); + } catch { + return null; + } +} + +async function hashDir(dir) { + const names = (await readdir(dir)).sort(); + const hashes = {}; + for (const n of names) hashes[n] = sha256(await readFile(path.join(dir, n))); + return hashes; +} + +try { + const personDir = path.join(workdir, 'skills', 'colleague', person); + await mkdir(personDir, { recursive: true }); + await cp(corpus, path.join(workdir, 'corpus'), { recursive: true }); + console.log(`验收语料：${path.relative(root, corpus)}`); + + // 1. harvest + 幂等 + 回执形状 + const h1 = distilly(['harvest', path.join(workdir, 'corpus'), '--person', person, '--json']); + const r1 = parseReceipt(h1.stdout); + record('harvest 退出码 0 且回执可解析', h1.status === 0 && !!r1, r1 ? `${r1.outputs?.length ?? 0} 个产物` : '无回执'); + record( + '回执形状（command/ok/inputs/outputs/sha256/bytes）', + !!r1 && + typeof r1.command === 'string' && + typeof r1.ok === 'boolean' && + Array.isArray(r1.inputs) && + Array.isArray(r1.outputs) && + r1.outputs.every((o) => typeof o.sha256 === 'string' && typeof o.bytes === 'number'), + ); + + const ledgerPath = path.join(personDir, 'knowledge', 'index.json'); + const ledger1 = JSON.parse(await readFile(ledgerPath, 'utf8')); + const h2 = distilly(['harvest', path.join(workdir, 'corpus'), '--person', person, '--json']); + const ledger2 = JSON.parse(await readFile(ledgerPath, 'utf8')); + record('重复 harvest 幂等', h2.status === 0 && ledger1.length === ledger2.length, `${ledger1.length} → ${ledger2.length} 条`); + + const anchors = ledger2.flatMap((e) => e.anchors ?? []).map((a) => (typeof a === 'string' ? a : a.id)); + record('账本里有锚点', anchors.length > 0, `${anchors.length} 个`); + + // 2. retrospect + 确定性 + 锚点回指 + const derivedDir = path.join(personDir, 'evidence', 'derived'); + distilly(['retrospect', '--person', person, '--json']); + const d1 = await hashDir(derivedDir); + distilly(['retrospect', '--person', person, '--json']); + const d2 = await hashDir(derivedDir); + record('retrospect 两次产物字节相同', JSON.stringify(d1) === JSON.stringify(d2), Object.keys(d1).join(', ')); + + const known = new Set(anchors); + let dangling = 0; + for (const [name] of Object.entries(d1)) { + const body = await readFile(path.join(derivedDir, name), 'utf8'); + for (const m of body.matchAll(/"?(k\d{4}(?::t\d+)?)"?/g)) if (!known.has(m[1])) dangling += 1; + } + record('派生结论锚点全部可回指', dangling === 0, dangling ? `${dangling} 个悬空` : '0 悬空'); + + // 3. view check / render（view 模板里的锚点按序解析，模拟"模型写好的视图"） + const template = await readFile(path.join(corpus, 'expected', 'view.template.json'), 'utf8'); + const resolved = template.replace(/\{\{ANCHOR:(\d+)\}\}/g, (_, n) => anchors[Number(n) - 1] ?? anchors[0]); + const viewsDir = path.join(personDir, 'views'); + await mkdir(viewsDir, { recursive: true }); + await writeFile(path.join(viewsDir, `${person}.view.json`), resolved, 'utf8'); + + record('view check 通过', distilly(['view', 'check', '--person', person, '--json']).status === 0); + + const htmlPath = path.join(viewsDir, `${person}.html`); + distilly(['view', 'render', '--person', person, '--json']); + const html1 = await readFile(htmlPath); + distilly(['view', 'render', '--person', person, '--json']); + const html2 = await readFile(htmlPath); + record('render 两次产物字节相同', sha256(html1) === sha256(html2), `${html1.length} bytes`); + const html = html1.toString('utf8'); + record('产物单文件无外链', !/https?:\/\//i.test(html.replace(/https?:\/\/www\.w3\.org[^"']*/g, '')), ''); + record('产物含 CSP', /Content-Security-Policy/i.test(html)); + + // 4. visual-check（八项） + const vc = spawnSync('node', [path.join(root, 'scripts/visual-check.mjs'), htmlPath, '--out', evidenceDir], { + cwd: workdir, + encoding: 'utf8', + }); + if (/Cannot find module|ENOENT/.test(vc.stderr ?? '')) { + record('visual-check 可用', false, 'scripts/visual-check.mjs 尚不存在（ds/03-render 的产出）'); + } else { + record('visual-check 八项通过', vc.status === 0, (vc.stdout ?? '').trim().split('\n').slice(-3).join(' / ')); + } +} catch (error) { + record('验收流程未中断', false, String(error.message).split('\n')[0]); +} finally { + if (keep) console.log(`保留工作目录：${workdir}`); + else await rm(workdir, { recursive: true, force: true }); +} + +console.log(`\n验收结果：${results.length - failed}/${results.length} 通过`); +if (failed) { + console.log('未通过项（依赖未落地时属预期，落地后必须转绿）：'); + for (const r of results.filter((x) => !x.ok)) console.log(` - ${r.name}: ${r.detail}`); + process.exit(1); +} From d91245502c548b34f0fcba712f2b3f2f780121fd Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:08:03 +0800 Subject: [PATCH 06/90] docs(v2): track which branch delivers what and how to verify it One table mapping each ds/* branch to its deliverable, dependencies and the exact command that proves it, plus the merge order for the integration branch. --- docs/v2/STATUS.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 docs/v2/STATUS.md diff --git a/docs/v2/STATUS.md b/docs/v2/STATUS.md new file mode 100644 index 00000000..7463a3db --- /dev/null +++ b/docs/v2/STATUS.md @@ -0,0 +1,19 @@ +# dot-skill v2 · 任务与分支状态 + +契约：`docs/v2/CONTRACT.md`（冻结接口）· 验收：`docs/v2/ACCEPTANCE.md` · 本文件只记"谁在做什么、怎么验"。 + +| # | 分支 | 交付 | 依赖 | 怎么验 | +| --- | --- | --- | --- | --- | +| 1 | `ds/01-node-core` | 入口 CLI + 子命令注册骨架；`skill_{schema,presets,writer}`/`version_manager`/8 安装器移植；Unihan 拼音表；Python 测试 → `node --test`；CI 换 Node；parity 报告 | — | `npm test`；`docs/evidence/pr-01-node-core.md` 的 parity 表 | +| 2 | `ds/02-parse-zero-cred` | `knowledge/{store,ledger,anchors}` + `parse-*`（chat/email/subtitle/office/archive/feishu）+ 幂等 `harvest` | — | `npm test`；`node scripts/acceptance.mjs` 的 harvest/锚点/幂等断言 | +| 3 | `ds/03-render` | 模板碎片 + 生成物 + `--check` 防漂移；`view check/render`；`visual-check` 八项 | — | `node scripts/generate-template.mjs --check`；`node scripts/visual-check.mjs ` | +| 4 | `ds/04-prompts` | `SKILL.md` 五步 + 每个 prompt"必须/禁止/回执" + 三个新 prompt + `prompt-lint` | — | `node scripts/prompt-lint.mjs`；`node --test tests/prompt-contract.test.mjs` | +| 5 | `ds/05-agents` | coding-agent 矩阵（命令/目录/能力/双语注意）+ 双语文档 + 断言测试 | 1 | `node --test tests/agents.test.mjs` | +| 6 | `ds/06-retrospect` | `retrospect` → `evidence/derived/*.json`（每条带锚点、两次字节相同） | 2 | `node --test tests/retrospect.test.mjs`；`acceptance.mjs` 的确定性/回指断言 | +| 7 | `ds/07-keys-and-schema` | 要 key 渠道 + computer-use 同意门 + 密钥纪律 + `SCHEMA_VERSION 4` 迁移 + 发布 | 1,2 | `docs/evidence/pr-07-keys-schema.md`；无 key/无 consent 的失败路径测试 | +| — | `dot-skill-test`（本分支） | 契约、验收协议、语料夹具、验收脚本、本状态表 | — | `node scripts/acceptance.mjs`（依赖到位后必须全绿） | + +## 合并顺序 + +`ds/01` → `ds/02` → {`ds/05`, `ds/06`} → `ds/03`, `ds/04` → `ds/07`，最后 `dot-skill-test` → `dot-skill`（默认分支）。 +每个 PR 的 base 都是 `dot-skill-test`；合并后按 `docs/evidence/pr-NN-*.md` 复核一遍断言。 From d1a5c6530b41664f0c9ad93092ea895a481d18b9 Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:08:13 +0800 Subject: [PATCH 07/90] docs(v2): correct the fixture cue count in the corpus notes --- tests/fixtures/public-corpus/README.md | 2 +- .../synthetic-interview/expected/view.template.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/fixtures/public-corpus/README.md b/tests/fixtures/public-corpus/README.md index 0ebb8c5d..2dcab5c3 100644 --- a/tests/fixtures/public-corpus/README.md +++ b/tests/fixtures/public-corpus/README.md @@ -19,6 +19,6 @@ public-corpus/ | 目录 | 内容 | 许可 | | --- | --- | --- | -| `synthetic-interview/` | 我们自写的合成访谈（后端工程师 × 技术面试官，45 条字幕，含时间跨度、语气变化、一处回避话题） | 本项目原创，随仓库以 MIT 发布；**不是真人对话**，仅用于工程验收 | +| `synthetic-interview/` | 我们自写的合成访谈（后端工程师 × 技术面试官，38 条字幕，含时间跨度、语气变化、一处回避话题） | 本项目原创，随仓库以 MIT 发布；**不是真人对话**，仅用于工程验收 | > 合成语料能验证"流程正确"（锚点、字节守恒、确定性、渲染、溢出），但**不能**验证"蒸出来的角色讨不讨喜"——那需要真实素材，见 `docs/v2/ACCEPTANCE.md`。 diff --git a/tests/fixtures/public-corpus/synthetic-interview/expected/view.template.json b/tests/fixtures/public-corpus/synthetic-interview/expected/view.template.json index c95a36b2..e10e84a8 100644 --- a/tests/fixtures/public-corpus/synthetic-interview/expected/view.template.json +++ b/tests/fixtures/public-corpus/synthetic-interview/expected/view.template.json @@ -36,5 +36,5 @@ ]} ], "emphasis": ["voice", "boundaries"], - "sources_note": "合成访谈字幕 45 条；仅用于验收。" + "sources_note": "合成访谈字幕 38 条；仅用于验收。" } From b8307c1ace2212e6277f4f90ab6564ee97f8fc09 Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:09:03 +0800 Subject: [PATCH 08/90] feat(v2): add the coding-agent support matrix as shared host data MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eight hosts with the directory each one actually scans (taken from bin/distilly.mjs and INSTALL.md only — no guessed paths), the confirmed AgentSkills CLI ids, per-host bilingual caveats, and the three command helpers. Hosts without a confirmed CLI target refuse to emit an `--agent` command under { requireVerified: true }. --- src/hosts/agents.mjs | 170 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 src/hosts/agents.mjs diff --git a/src/hosts/agents.mjs b/src/hosts/agents.mjs new file mode 100644 index 00000000..bb53dbe7 --- /dev/null +++ b/src/hosts/agents.mjs @@ -0,0 +1,170 @@ +/** + * Coding-agent support matrix. + * + * Every entry names the directory the host actually scans and how the Skill is + * installed there. The paths are taken from this repository only — `bin/distilly.mjs` + * (the installers' targets) and `INSTALL.md` (the documented per-host table). + * Nothing here is guessed: hosts whose project-local directory is user-defined + * simply have no `projectPath`. + * + * Two install routes exist: + * 1. the AgentSkills CLI — `npx skills add --agent ` + * 2. a direct clone into the directory the host scans + * + * `capability: 'full'` means the host can read files and run shell commands, so + * the whole collect → derive → read → distill → render workflow applies. + */ + +export const REPO = 'titanwings/distilly'; +export const SKILL_NAME = 'distilly'; + +/** + * @typedef {object} CodingAgent + * @property {string} id host id, also the `install ` argument + * @property {string} label display name + * @property {string} [cliId] `--agent` target, only when confirmed + * @property {string} globalPath directory the host scans for global skills + * @property {string} [projectPath] project-local directory, when documented + * @property {'full'|'prompt-only'} capability + * @property {{zh: string, en: string}} note + */ + +/** @type {CodingAgent[]} */ +export const AGENTS = [ + { + id: 'claude-code', + label: 'Claude Code', + cliId: 'claude-code', + globalPath: '~/.claude/skills/distilly', + projectPath: '.claude/skills/distilly', + capability: 'full', + note: { + zh: '安装后会被自动发现，可直接说"把这段聊天蒸馏成 Skill"。', + en: 'Discovered automatically once installed; just ask it to distill a conversation.', + }, + }, + { + id: 'codex', + label: 'Codex CLI', + cliId: 'codex', + globalPath: '~/.agents/skills/distilly', + projectPath: '.agents/skills/distilly', + capability: 'full', + note: { + zh: 'Codex 扫描 ~/.agents/skills；旧版 ~/.codex/skills 需手动迁移。', + en: 'Codex scans ~/.agents/skills; the legacy ~/.codex/skills path needs a manual move.', + }, + }, + { + id: 'opencode', + label: 'opencode', + cliId: 'opencode', + globalPath: '~/.config/opencode/skills/distilly', + projectPath: '.opencode/skills/distilly', + capability: 'full', + note: { + zh: '同时兼容 ~/.agents/skills 与项目级 .opencode/skills。', + en: 'Also reads ~/.agents/skills and the project-local .opencode/skills.', + }, + }, + { + id: 'openclaw', + label: 'OpenClaw', + cliId: 'openclaw', + globalPath: '~/.openclaw/workspace/skills/distilly', + capability: 'full', + note: { + zh: 'Skill 目录即工作区子目录，装完重开 session 生效；项目级目录由用户在 OpenClaw 内自定义，本文档不猜路径。', + en: 'The Skill directory lives inside the workspace; reopen the session after install. Project-local paths are user-defined in OpenClaw, so none is claimed here.', + }, + }, + { + id: 'hermes', + label: 'Hermes', + cliId: 'hermes', + globalPath: '~/.hermes/skills/openclaw-imports/distilly', + projectPath: '.hermes/skills/distilly', + capability: 'full', + note: { + zh: '默认装在 Hermes 的 openclaw-imports 目录；项目级安装需先在该目录运行 hermes skills trust。', + en: 'Installs into Hermes’ openclaw-imports directory; project-local installs need `hermes skills trust` there first.', + }, + }, + { + id: 'deepseek-harness', + label: 'DeepSeek Harness', + cliId: 'deepseek-harness', + globalPath: '$DSH_HOME/skills/distilly', + projectPath: '.dsh/skills/distilly', + capability: 'full', + note: { + zh: 'DSH_HOME 未设置时等价于 ~/.dsh/skills/distilly；社区集成，非官方 DeepSeek 产品。', + en: 'Falls back to ~/.dsh/skills/distilly when DSH_HOME is unset; community integration, not an official DeepSeek product.', + }, + }, + { + id: 'grok-build', + label: 'Grok Build', + cliId: 'grok-build', + globalPath: '~/.grok/skills/distilly', + projectPath: '.grok/skills/distilly', + capability: 'full', + note: { + zh: '与 ~/.agents/skills 共用发现目录。', + en: 'Shares the ~/.agents/skills discovery directory.', + }, + }, + { + id: 'pi', + label: 'Pi', + globalPath: '~/.pi/agent/skills/distilly', + capability: 'full', + note: { + zh: '只确认了全局目录；AgentSkills CLI 的 --agent 目标未确认，因此本工具不输出该命令，用 clone 路线。', + en: 'Only the global directory is confirmed; the AgentSkills CLI `--agent` target is not, so no such command is emitted — use the clone route.', + }, + }, +]; + +export const DEFAULT_AGENT = 'claude-code'; + +export function listAgents() { + return AGENTS.map((a) => a.id); +} + +/** @returns {CodingAgent} */ +export function getAgent(id) { + const agent = AGENTS.find((a) => a.id === id); + if (!agent) throw new Error(`unknown coding agent "${id}"; known: ${listAgents().join(', ')}`); + return agent; +} + +/** True only when the AgentSkills CLI target for this host is confirmed. */ +export function skillsCliSupported(id) { + return Boolean(getAgent(id).cliId); +} + +/** + * One-liner via the AgentSkills CLI. + * Pass `{ requireVerified: true }` to fail instead of emitting an unconfirmed + * `--agent` target. + */ +export function skillsCliCommand(id, scope = 'global', { requireVerified = false } = {}) { + const agent = getAgent(id); + if (requireVerified && !agent.cliId) { + throw new Error( + `${agent.label} is not a confirmed AgentSkills CLI target; use cloneCommand('${id}', '${scope}').`, + ); + } + const flags = [`--agent ${agent.cliId ?? agent.id}`]; + if (scope === 'global') flags.push('--global', '--copy', '--yes'); + return `npx -y skills add ${REPO} --skill ${SKILL_NAME} ${flags.join(' ')}`; +} + +/** Direct clone route, for hosts whose CLI target is unconfirmed. */ +export function cloneCommand(id, scope = 'global') { + const agent = getAgent(id); + const target = scope === 'project' ? agent.projectPath : agent.globalPath; + if (!target) throw new Error(`${agent.label} has no documented project-local path; use the global install.`); + return `git clone https://github.com/${REPO} ${target}`; +} From dba29b7c70eaf95ffba42312f540b4d4a2ce3431 Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:09:31 +0800 Subject: [PATCH 09/90] docs(v2): note that the host matrix ships from the integration branch ds/05 owns the host documentation, the INSTALL/README sections and the anti-drift assertions, so it can run in parallel with the CLI port. --- docs/v2/STATUS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/v2/STATUS.md b/docs/v2/STATUS.md index 7463a3db..af0333dd 100644 --- a/docs/v2/STATUS.md +++ b/docs/v2/STATUS.md @@ -8,10 +8,10 @@ | 2 | `ds/02-parse-zero-cred` | `knowledge/{store,ledger,anchors}` + `parse-*`（chat/email/subtitle/office/archive/feishu）+ 幂等 `harvest` | — | `npm test`；`node scripts/acceptance.mjs` 的 harvest/锚点/幂等断言 | | 3 | `ds/03-render` | 模板碎片 + 生成物 + `--check` 防漂移；`view check/render`；`visual-check` 八项 | — | `node scripts/generate-template.mjs --check`；`node scripts/visual-check.mjs ` | | 4 | `ds/04-prompts` | `SKILL.md` 五步 + 每个 prompt"必须/禁止/回执" + 三个新 prompt + `prompt-lint` | — | `node scripts/prompt-lint.mjs`；`node --test tests/prompt-contract.test.mjs` | -| 5 | `ds/05-agents` | coding-agent 矩阵（命令/目录/能力/双语注意）+ 双语文档 + 断言测试 | 1 | `node --test tests/agents.test.mjs` | +| 5 | `ds/05-agents` | `docs/v2/HOSTS.md` 双语文档 + INSTALL/README 宿主章节 + 断言测试（矩阵本身已由维护者落在 `src/hosts/agents.mjs`） | — | `node --test tests/agents.test.mjs`（含「矩阵 vs bin/distilly.mjs 表不漂移」断言） | | 6 | `ds/06-retrospect` | `retrospect` → `evidence/derived/*.json`（每条带锚点、两次字节相同） | 2 | `node --test tests/retrospect.test.mjs`；`acceptance.mjs` 的确定性/回指断言 | | 7 | `ds/07-keys-and-schema` | 要 key 渠道 + computer-use 同意门 + 密钥纪律 + `SCHEMA_VERSION 4` 迁移 + 发布 | 1,2 | `docs/evidence/pr-07-keys-schema.md`；无 key/无 consent 的失败路径测试 | -| — | `dot-skill-test`（本分支） | 契约、验收协议、语料夹具、验收脚本、本状态表 | — | `node scripts/acceptance.mjs`（依赖到位后必须全绿） | +| — | `dot-skill-test`（本分支） | 契约、验收协议、语料夹具、验收脚本、宿主矩阵 `src/hosts/agents.mjs`、本状态表 | — | `node scripts/acceptance.mjs`（依赖到位后必须全绿） | ## 合并顺序 From 42f7e1b200db82b1b181243fb7991d5529bab86f Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:11:28 +0800 Subject: [PATCH 10/90] docs(v2): split keys from schema and add the release task ds/07 now covers the credentialed channels, the consent gate and the optional transcribe backend (verifiable entirely with injected fetch), while schema v4, migration and the release check move to ds/08 so neither task waits on the other. --- docs/v2/STATUS.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/v2/STATUS.md b/docs/v2/STATUS.md index af0333dd..c90cf2d9 100644 --- a/docs/v2/STATUS.md +++ b/docs/v2/STATUS.md @@ -9,11 +9,12 @@ | 3 | `ds/03-render` | 模板碎片 + 生成物 + `--check` 防漂移；`view check/render`；`visual-check` 八项 | — | `node scripts/generate-template.mjs --check`；`node scripts/visual-check.mjs ` | | 4 | `ds/04-prompts` | `SKILL.md` 五步 + 每个 prompt"必须/禁止/回执" + 三个新 prompt + `prompt-lint` | — | `node scripts/prompt-lint.mjs`；`node --test tests/prompt-contract.test.mjs` | | 5 | `ds/05-agents` | `docs/v2/HOSTS.md` 双语文档 + INSTALL/README 宿主章节 + 断言测试（矩阵本身已由维护者落在 `src/hosts/agents.mjs`） | — | `node --test tests/agents.test.mjs`（含「矩阵 vs bin/distilly.mjs 表不漂移」断言） | -| 6 | `ds/06-retrospect` | `retrospect` → `evidence/derived/*.json`（每条带锚点、两次字节相同） | 2 | `node --test tests/retrospect.test.mjs`；`acceptance.mjs` 的确定性/回指断言 | -| 7 | `ds/07-keys-and-schema` | 要 key 渠道 + computer-use 同意门 + 密钥纪律 + `SCHEMA_VERSION 4` 迁移 + 发布 | 1,2 | `docs/evidence/pr-07-keys-schema.md`；无 key/无 consent 的失败路径测试 | +| 6 | `ds/06-retrospect` | `retrospect` → `evidence/derived/*.json`（每条带锚点、两次字节相同） | 契约（账本形状已冻结，自带合成夹具；落地后再对 ds/02 的真实输出复验） | `node --test tests/retrospect.test.mjs`；`acceptance.mjs` 的确定性/回指断言 | +| 7 | `ds/07-collect-consent` | 要 key 渠道（飞书/Slack/钉钉/X api）+ computer-use 同意门 + 密钥纪律 + transcribe 可选后端 | 契约 + `src/hosts/agents.mjs` | `node --test tests/collect.test.mjs tests/consent.test.mjs`（含"密钥不泄露"与"代码无写操作"断言） | +| 8 | `ds/08-schema-release` | `SCHEMA_VERSION 4` + 幂等迁移 + 安装器携带 `knowledge/|evidence/|views/|assets/` + 发布检查 | 1,2,3 | `node --test tests/schema-migration.test.mjs`；`scripts/check_release.mjs` | | — | `dot-skill-test`（本分支） | 契约、验收协议、语料夹具、验收脚本、宿主矩阵 `src/hosts/agents.mjs`、本状态表 | — | `node scripts/acceptance.mjs`（依赖到位后必须全绿） | ## 合并顺序 -`ds/01` → `ds/02` → {`ds/05`, `ds/06`} → `ds/03`, `ds/04` → `ds/07`，最后 `dot-skill-test` → `dot-skill`（默认分支）。 +`ds/01` → `ds/02` → {`ds/05`, `ds/06`, `ds/07`} → `ds/03`, `ds/04` → `ds/08` → 最后 `dot-skill-test` → `dot-skill`（默认分支）。 每个 PR 的 base 都是 `dot-skill-test`；合并后按 `docs/evidence/pr-NN-*.md` 复核一遍断言。 From e1fda415399be16cb41a968825d20c1f26af7992 Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:12:09 +0800 Subject: [PATCH 11/90] docs(v2): add the Python-to-Node migration ledger Every legacy tool mapped to the module that replaces it, with its owning branch, so 'no Python left' becomes checkable rather than asserted, and so each deletion can be tied to a parity record. --- docs/v2/MIGRATION.md | 49 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 docs/v2/MIGRATION.md diff --git a/docs/v2/MIGRATION.md b/docs/v2/MIGRATION.md new file mode 100644 index 00000000..9f9c9c9b --- /dev/null +++ b/docs/v2/MIGRATION.md @@ -0,0 +1,49 @@ +# Python → Node 迁移台账（Node 单栈） + +目标：`tools/**/*.py` 与 `tests/test_*.py` 全部消失，运行时只依赖 Node >= 20。现状：**24 个 py 文件 / 7348 行**，测试 **10 个 / 2676 行**。 + +| legacy 文件 | 行数 | 归属 | 目标模块 | 状态 | +| --- | --- | --- | --- | --- | +| `tools/dingtalk_auto_collector.py` | 790 | ds/07 | `src/collect/dingtalk.mjs` | ⏳ 待开工 | +| `tools/email_parser.py` | 339 | ds/02 | `src/parse/email.mjs` | ⏳ 进行中 | +| `tools/feishu_auto_collector.py` | 960 | ds/07 | `src/collect/feishu.mjs` | ⏳ 待开工 | +| `tools/feishu_browser.py` | 374 | ds/07 | `src/collect/feishu.mjs（共享同意门）` | ⏳ 待开工 | +| `tools/feishu_mcp_client.py` | 314 | ds/07 | `src/collect/feishu.mjs（MCP 模式）` | ⏳ 待开工 | +| `tools/feishu_parser.py` | 251 | ds/02 | `src/parse/feishu.mjs` | ⏳ 进行中 | +| `tools/install_claude_generated_skill.py` | 108 | ds/01 | `src/install/hosts.mjs` | ⏳ 进行中 | +| `tools/install_codex_generated_skill.py` | 57 | ds/01 | `src/install/hosts.mjs` | ⏳ 进行中 | +| `tools/install_codex_skill.py` | 65 | ds/01 | `src/install/hosts.mjs` | ⏳ 进行中 | +| `tools/install_generated_skill.py` | 76 | ds/01 | `src/install/hosts.mjs` | ⏳ 进行中 | +| `tools/install_generated_skill_common.py` | 130 | ds/01 | `src/install/hosts.mjs` | ⏳ 进行中 | +| `tools/install_hermes_skill.py` | 65 | ds/01 | `src/install/hosts.mjs` | ⏳ 进行中 | +| `tools/install_openclaw_generated_skill.py` | 57 | ds/01 | `src/install/hosts.mjs` | ⏳ 进行中 | +| `tools/install_openclaw_skill.py` | 65 | ds/01 | `src/install/hosts.mjs` | ⏳ 进行中 | +| `tools/research/merge_research.py` | 277 | ds/02 | `src/derive/merge.mjs` | ⏳ 进行中 | +| `tools/research/quality_check.py` | 253 | ds/06 | `src/derive/quality.mjs` | ⏳ 待开工 | +| `tools/research/srt_to_transcript.py` | 92 | ds/02 | `src/parse/subtitle.mjs` | ⏳ 进行中 | +| `tools/research/transcribe_audio.py` | 304 | ds/07 | `src/optional/transcribe.mjs` | ⏳ 待开工 | +| `tools/research/xquik_public_posts.py` | 406 | ds/07 | `src/collect/x.mjs` | ⏳ 待开工 | +| `tools/skill_presets.py` | 288 | ds/01 | `src/skill/presets.mjs` | ⏳ 进行中 | +| `tools/skill_schema.py` | 411 | ds/01 | `src/skill/schema.mjs` | ⏳ 进行中 | +| `tools/skill_writer.py` | 705 | ds/01 | `src/skill/writer.mjs` | ⏳ 进行中 | +| `tools/slack_auto_collector.py` | 722 | ds/07 | `src/collect/slack.mjs` | ⏳ 待开工 | +| `tools/version_manager.py` | 239 | ds/01 | `src/skill/versions.mjs` | ⏳ 进行中 | + +| legacy 测试 | 行数 | 归属 | 目标 | +| --- | --- | --- | --- | +| `tests/test_cli_lifecycle.py` | 609 | ds/01 | `tests/*.test.mjs`（`node --test`） | +| `tests/test_config_migration.py` | 53 | ds/01 | `tests/*.test.mjs`（`node --test`） | +| `tests/test_install_claude_generated_skill.py` | 98 | ds/01 | `tests/*.test.mjs`（`node --test`） | +| `tests/test_install_generated_skill.py` | 126 | ds/01 | `tests/*.test.mjs`（`node --test`） | +| `tests/test_install_hermes_skill.py` | 83 | ds/01 | `tests/*.test.mjs`（`node --test`） | +| `tests/test_install_openclaw_and_codex.py` | 140 | ds/01 | `tests/*.test.mjs`（`node --test`） | +| `tests/test_research_tools.py` | 278 | ds/01 | `tests/*.test.mjs`（`node --test`） | +| `tests/test_skill_entrypoint_docs.py` | 328 | ds/01 | `tests/*.test.mjs`（`node --test`） | +| `tests/test_skill_writer.py` | 719 | ds/01 | `tests/*.test.mjs`（`node --test`） | +| `tests/test_xquik_public_posts.py` | 242 | ds/01 | `tests/*.test.mjs`（`node --test`） | + +## 规则 + +1. **删一个 py 文件的前提**：对应 mjs 有测试，且 parity 证据（同一输入，两边输出逐字节相同）写进 `docs/evidence/pr-NN-*.md`。 +2. 纯网络/需要凭据的部分（飞书浏览器自动化、Slack/钉钉拉取）不靠"无凭据环境下的 parity"证明——用注入 mock fetch 的失败路径 + 密钥不泄露断言来证明，真实账号验证在用户授权后单独做。 +3. 迁移完成判据：`find tools tests -name "*.py" | wc -l` 为 0，且 `requirements.txt` 删除，CI 只跑 Node。 From 402b378e61a6cf2d108048f5308344a76ff20891 Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:12:31 +0800 Subject: [PATCH 12/90] docs(v2): record that the acceptance harness passes against a reference stub A minimal contract-compliant implementation (kept outside the repository) scores 11/11, so a failure on a real branch points at the implementation rather than at the harness. --- docs/v2/ACCEPTANCE.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/v2/ACCEPTANCE.md b/docs/v2/ACCEPTANCE.md index 48e8f52d..fd24e83c 100644 --- a/docs/v2/ACCEPTANCE.md +++ b/docs/v2/ACCEPTANCE.md @@ -64,6 +64,8 @@ node scripts/acceptance.mjs --keep # 保留临时 person 目录 语料放 `tests/fixtures/public-corpus/`（见那里的 `README.md` 与 `LICENSE.md`）。 +**脚本自测（防"验收脚本本身是坏的"）**：用一份符合契约的最小实现（只在 `/tmp` 下的临时目录，不入库）跑 `scripts/acceptance.mjs`，结果 **11/11 通过** —— 证明这套断言不是空转、契约是可实现的、以及回执/账本/锚点/view/visual-check 的形状就是脚本期望的那样。任何一项在真实实现上失败，都是实现的问题，不是脚本的。 + ## 6. 真实私聊/邮件语料（隐私） 不进仓库、不进 CI。流程同上，但： From 52d4050732b2fd0cfcb95ec842cdd386a0d3a5e2 Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Sun, 13 Sep 2026 02:13:35 +0800 Subject: [PATCH 13/90] fix(v2): verify every AgentSkills cliId against the upstream registry Checked against skills@1.5.26 instead of inferring ids from display names: grok-build -> grok, hermes -> hermes-agent, pi is a real target (so it gains a cliId and a project path), and DeepSeek Harness has no upstream target at all, so it becomes clone-only rather than advertising an `--agent` value that the CLI rejects. Still 7 CLI-capable hosts and 1 clone-only host, but the correct ones. --- src/hosts/agents.mjs | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/src/hosts/agents.mjs b/src/hosts/agents.mjs index bb53dbe7..26b22fd0 100644 --- a/src/hosts/agents.mjs +++ b/src/hosts/agents.mjs @@ -11,6 +11,12 @@ * 1. the AgentSkills CLI — `npx skills add --agent ` * 2. a direct clone into the directory the host scans * + * Every `cliId` below was checked against the upstream `skills` CLI registry + * (v1.5.26) rather than inferred from the host's display name: `grok` and + * `hermes-agent` are the registry names for Grok Build and Hermes, `pi` is a + * target, and there is **no** DeepSeek Harness / DSH target at all — which is + * why that host is clone-only. Re-check with `npx skills ls -a `. + * * `capability: 'full'` means the host can read files and run shell commands, so * the whole collect → derive → read → distill → render workflow applies. */ @@ -22,7 +28,8 @@ export const SKILL_NAME = 'distilly'; * @typedef {object} CodingAgent * @property {string} id host id, also the `install ` argument * @property {string} label display name - * @property {string} [cliId] `--agent` target, only when confirmed + * @property {string} [cliId] `--agent` target, verified against the upstream + * AgentSkills CLI registry (see the header note) * @property {string} globalPath directory the host scans for global skills * @property {string} [projectPath] project-local directory, when documented * @property {'full'|'prompt-only'} capability @@ -81,7 +88,7 @@ export const AGENTS = [ { id: 'hermes', label: 'Hermes', - cliId: 'hermes', + cliId: 'hermes-agent', globalPath: '~/.hermes/skills/openclaw-imports/distilly', projectPath: '.hermes/skills/distilly', capability: 'full', @@ -93,19 +100,18 @@ export const AGENTS = [ { id: 'deepseek-harness', label: 'DeepSeek Harness', - cliId: 'deepseek-harness', globalPath: '$DSH_HOME/skills/distilly', projectPath: '.dsh/skills/distilly', capability: 'full', note: { - zh: 'DSH_HOME 未设置时等价于 ~/.dsh/skills/distilly；社区集成，非官方 DeepSeek 产品。', - en: 'Falls back to ~/.dsh/skills/distilly when DSH_HOME is unset; community integration, not an official DeepSeek product.', + zh: 'DSH_HOME 未设置时等价于 ~/.dsh/skills/distilly；社区集成，非官方 DeepSeek 产品。上游 AgentSkills CLI 没有 DSH 目标，所以只提供 clone 路线。', + en: 'Falls back to ~/.dsh/skills/distilly when DSH_HOME is unset; community integration, not an official DeepSeek product. The upstream AgentSkills CLI has no DSH target, so only the clone route is offered.', }, }, { id: 'grok-build', label: 'Grok Build', - cliId: 'grok-build', + cliId: 'grok', globalPath: '~/.grok/skills/distilly', projectPath: '.grok/skills/distilly', capability: 'full', @@ -117,11 +123,13 @@ export const AGENTS = [ { id: 'pi', label: 'Pi', + cliId: 'pi', globalPath: '~/.pi/agent/skills/distilly', + projectPath: '.pi/skills/distilly', capability: 'full', note: { - zh: '只确认了全局目录；AgentSkills CLI 的 --agent 目标未确认，因此本工具不输出该命令，用 clone 路线。', - en: 'Only the global directory is confirmed; the AgentSkills CLI `--agent` target is not, so no such command is emitted — use the clone route.', + zh: '上游 AgentSkills CLI 的合法目标（项目级 .pi/skills、全局 ~/.pi/agent/skills）。', + en: 'A valid upstream AgentSkills target (project `.pi/skills`, global `~/.pi/agent/skills`).', }, }, ]; From 161e5ef406d6436a2017d4147605342c3f61096c Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Tue, 15 Sep 2026 13:38:57 +0800 Subject: [PATCH 14/90] =?UTF-8?q?wip(recovery):=20=E4=BB=8E=E4=BC=9A?= =?UTF-8?q?=E8=AF=9D=E8=AE=B0=E5=BD=95=E9=87=8D=E5=BB=BA=E7=9A=84=E6=96=87?= =?UTF-8?q?=E4=BB=B6=EF=BC=88=E7=AC=AC=E4=B8=80=E6=AC=A1=E6=A3=80=E6=9F=A5?= =?UTF-8?q?=E7=82=B9=EF=BC=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit /tmp 被环境清空，`/tmp/dst` 里 dot-skill v2 的 165 个本地提交全部丢失。 这一批是从 73 个 DSH 会话记录里重放 write/edit 重建出来的文件内容： - 重放 377 条 write/edit 事件；132 个仓库内路径，106 个有完整 write 内容 - 每应用一条 edit 就校验一次语法，坏 edit 回滚（第一版没有校验， `bin/distilly.mjs` 被拼坏，131 个测试里 65 个连锁失败） - 80 个文件覆盖进仓库；7 个 prompts 在基线 `52d4050` 的内容上应用 edit 已知还不完整（下一步处理）： - 41 个文件的 edit 被跳过、5 个有回滚 —— 它们的最终内容来自会话里的 bash heredoc，write/edit 记录覆盖不到 - `scripts/audit-objective.mjs`、`scripts/blind-test.mjs` 等完全没有 write 记录 - 当前 `node --test tests/*.test.mjs`：137 个测试 66 pass / 71 fail， 主因是 `src/skill/writer.mjs` 少了 `installGeneratedHosts` 导出（同因） 这是一次**等价重建**，不是原来的 165 个提交；提交历史不会与丢失的那串相同。 --- .gitignore | 12 + SKILL.md | 1683 +++++------------ assets/template.source.html | 246 +++ bin/distilly.mjs | 283 ++- docs/evidence/pr-03-render.md | 194 ++ docs/evidence/pr-04-prompts.md | 110 ++ docs/evidence/pr-06-retrospect.md | 179 ++ docs/evidence/pr-07-collect-consent.md | 232 +++ docs/evidence/pr-20-repo-hygiene.md | 115 ++ docs/evidence/pr-21-package-ci.md | 170 ++ docs/v2/PROMPTS.md | 96 + docs/v2/RENDER.md | 267 +++ package.json | 2 +- prompts/collectors.md | 157 ++ prompts/computer-use.md | 165 ++ prompts/correction_handler.md | 128 ++ prompts/intake.md | 64 + prompts/merger.md | 64 + prompts/persona_analyzer.md | 70 + prompts/persona_builder.md | 65 + prompts/retrospection.md | 169 ++ prompts/work_analyzer.md | 70 + prompts/work_builder.md | 64 + scripts/check_release.mjs | 188 ++ scripts/generate-pinyin.mjs | 195 ++ scripts/generate-template.mjs | 211 +++ scripts/parity.mjs | 663 +++++++ scripts/prompt-lint.mjs | 362 ++++ scripts/visual-check.mjs | 488 +++++ src/cli/args.mjs | 103 + src/cli/entry.mjs | 33 + src/cli/receipt.mjs | 110 ++ src/collect/dingtalk.mjs | 832 ++++++++ src/collect/feishu.mjs | 801 ++++++++ src/collect/slack.mjs | 713 +++++++ src/collect/x.mjs | 958 ++++++++++ src/commands/doctor.mjs | 159 ++ src/commands/index.mjs | 203 ++ src/commands/install.mjs | 239 +++ src/commands/legacy.mjs | 243 +++ src/commands/skill.mjs | 480 +++++ src/consent.mjs | 513 +++++ src/derive/fixtures/synthetic-group/README.md | 29 + .../knowledge/text/dm-lin-chen.md | 26 + .../knowledge/text/group-chat.md | 44 + .../knowledge/text/incident-postmortem.md | 1 + src/derive/retrospect.mjs | 927 +++++++++ src/install/hosts.mjs | 365 ++++ src/knowledge/anchors.mjs | 1032 ++++++++++ src/knowledge/ledger.mjs | 589 ++++++ src/knowledge/store.mjs | 376 ++++ src/optional/transcribe.mjs | 702 +++++++ src/parse/archive.mjs | 1228 ++++++++++++ src/parse/chat.mjs | 838 ++++++++ src/parse/common.mjs | 653 +++++++ src/parse/subtitle.mjs | 319 ++++ src/skill/presets.mjs | 277 +++ src/skill/schema.mjs | 504 +++++ src/skill/slug.mjs | 114 ++ src/skill/versions.mjs | 193 ++ src/skill/writer.mjs | 400 ++++ src/views/render.mjs | 262 +++ src/views/schema.mjs | 629 ++++++ tests/cli-lifecycle.test.mjs | 464 +++++ tests/collect.test.mjs | 707 +++++++ tests/commands.test.mjs | 235 +++ tests/consent.test.mjs | 416 ++++ tests/dispatcher.test.mjs | 106 ++ tests/entry-point.test.mjs | 87 + tests/entrypoint-docs.test.mjs | 83 + tests/fixtures/parse/office/make-fixtures.mjs | 453 +++++ tests/gitignore.test.mjs | 112 ++ tests/helpers/cli.mjs | 27 + tests/install-claude-generated-skill.test.mjs | 84 + tests/install-generated-skill.test.mjs | 135 ++ tests/install-hermes-skill.test.mjs | 116 ++ tests/install-openclaw-and-codex.test.mjs | 157 ++ tests/package-payload.test.mjs | 147 ++ tests/pinyin-slug.test.mjs | 131 ++ tests/prompt-contract.test.mjs | 232 +++ tests/retrospect.test.mjs | 449 +++++ tests/skill-writer.test.mjs | 559 ++++++ tests/template.test.mjs | 181 ++ tests/views.test.mjs | 530 ++++++ viewer/export.js | 185 ++ viewer/focus.js | 133 ++ viewer/sections.js | 398 ++++ viewer/theme.js | 119 ++ 88 files changed, 26426 insertions(+), 1427 deletions(-) create mode 100644 assets/template.source.html mode change 100755 => 100644 bin/distilly.mjs create mode 100644 docs/evidence/pr-03-render.md create mode 100644 docs/evidence/pr-04-prompts.md create mode 100644 docs/evidence/pr-06-retrospect.md create mode 100644 docs/evidence/pr-07-collect-consent.md create mode 100644 docs/evidence/pr-20-repo-hygiene.md create mode 100644 docs/evidence/pr-21-package-ci.md create mode 100644 docs/v2/PROMPTS.md create mode 100644 docs/v2/RENDER.md create mode 100644 prompts/collectors.md create mode 100644 prompts/computer-use.md create mode 100644 prompts/retrospection.md create mode 100644 scripts/check_release.mjs create mode 100644 scripts/generate-pinyin.mjs create mode 100644 scripts/generate-template.mjs create mode 100644 scripts/parity.mjs create mode 100644 scripts/prompt-lint.mjs create mode 100644 scripts/visual-check.mjs create mode 100644 src/cli/args.mjs create mode 100644 src/cli/entry.mjs create mode 100644 src/cli/receipt.mjs create mode 100644 src/collect/dingtalk.mjs create mode 100644 src/collect/feishu.mjs create mode 100644 src/collect/slack.mjs create mode 100644 src/collect/x.mjs create mode 100644 src/commands/doctor.mjs create mode 100644 src/commands/index.mjs create mode 100644 src/commands/install.mjs create mode 100644 src/commands/legacy.mjs create mode 100644 src/commands/skill.mjs create mode 100644 src/consent.mjs create mode 100644 src/derive/fixtures/synthetic-group/README.md create mode 100644 src/derive/fixtures/synthetic-group/knowledge/text/dm-lin-chen.md create mode 100644 src/derive/fixtures/synthetic-group/knowledge/text/group-chat.md create mode 100644 src/derive/fixtures/synthetic-group/knowledge/text/incident-postmortem.md create mode 100644 src/derive/retrospect.mjs create mode 100644 src/install/hosts.mjs create mode 100644 src/knowledge/anchors.mjs create mode 100644 src/knowledge/ledger.mjs create mode 100644 src/knowledge/store.mjs create mode 100644 src/optional/transcribe.mjs create mode 100644 src/parse/archive.mjs create mode 100644 src/parse/chat.mjs create mode 100644 src/parse/common.mjs create mode 100644 src/parse/subtitle.mjs create mode 100644 src/skill/presets.mjs create mode 100644 src/skill/schema.mjs create mode 100644 src/skill/slug.mjs create mode 100644 src/skill/versions.mjs create mode 100644 src/skill/writer.mjs create mode 100644 src/views/render.mjs create mode 100644 src/views/schema.mjs create mode 100644 tests/cli-lifecycle.test.mjs create mode 100644 tests/collect.test.mjs create mode 100644 tests/commands.test.mjs create mode 100644 tests/consent.test.mjs create mode 100644 tests/dispatcher.test.mjs create mode 100644 tests/entry-point.test.mjs create mode 100644 tests/entrypoint-docs.test.mjs create mode 100644 tests/fixtures/parse/office/make-fixtures.mjs create mode 100644 tests/gitignore.test.mjs create mode 100644 tests/helpers/cli.mjs create mode 100644 tests/install-claude-generated-skill.test.mjs create mode 100644 tests/install-generated-skill.test.mjs create mode 100644 tests/install-hermes-skill.test.mjs create mode 100644 tests/install-openclaw-and-codex.test.mjs create mode 100644 tests/package-payload.test.mjs create mode 100644 tests/pinyin-slug.test.mjs create mode 100644 tests/prompt-contract.test.mjs create mode 100644 tests/retrospect.test.mjs create mode 100644 tests/skill-writer.test.mjs create mode 100644 tests/template.test.mjs create mode 100644 tests/views.test.mjs create mode 100644 viewer/export.js create mode 100644 viewer/focus.js create mode 100644 viewer/sections.js create mode 100644 viewer/theme.js diff --git a/.gitignore b/.gitignore index 08b33f67..b8b17225 100644 --- a/.gitignore +++ b/.gitignore @@ -29,7 +29,19 @@ playwright-data/ /tmp/feishu_*.txt /tmp/email_*.txt /tmp/dingtalk_*.txt + +# Every `knowledge/` is user data, wherever the person directory lives. This is an +# unanchored directory pattern, so it also matches source and fixture directories of +# the same name; the re-includes below must come *after* it, otherwise the later +# `knowledge/` rule excludes them again. Tracked files are unaffected either way, +# so only a newly added module would be lost — and it would be lost silently. knowledge/ +# ...except the synthetic ledger fixtures, which are test data and must be tracked +!src/derive/fixtures/**/knowledge/ +!src/derive/fixtures/**/knowledge/** +# ...and except the source modules: `src/knowledge/**` is code, not an export +!src/knowledge/ +!src/knowledge/** # OS .DS_Store diff --git a/SKILL.md b/SKILL.md index 7f9fbcc9..3a16f39f 100644 --- a/SKILL.md +++ b/SKILL.md @@ -11,11 +11,11 @@ allowed-tools: Read, Write, Edit, Bash > > 本 Skill 支持中英文。根据用户第一条消息的语言，全程使用同一语言回复。下方提供了两种语言的指令，按用户语言选择对应版本执行。 -> **Skill Root / Skill 根目录**: Before reading a bundled prompt or running a bundled script, resolve the absolute directory of the `SKILL.md` that the host actually loaded. In the instructions below, `{distilly_skill_root}` means that exact directory. Claude Code exposes it as `${CLAUDE_SKILL_DIR}`; on every other host, use the loaded-skill path supplied by that host's discovery context. Do not assume the shell's current working directory is the Skill root, and do not guess or hard-code an install path. If the host does not expose the loaded path or more than one Distilly installation is ambiguous, ask the user to identify the active installation before running code. +> **Skill Root / Skill 根目录**: Before reading a bundled prompt or running a bundled command, resolve the absolute directory of the `SKILL.md` that the host actually loaded. In the instructions below, `{distilly_skill_root}` means that exact directory. Claude Code exposes it as `${CLAUDE_SKILL_DIR}`; on every other host, use the loaded-skill path supplied by that host's discovery context. Do not assume the shell's current working directory is the Skill root, and do not guess or hard-code an install path. If the host does not expose the loaded path or more than one Distilly installation is ambiguous, ask the user to identify the active installation before running code. > -> Keep the shell in the user's current workspace so relative output paths such as `./skills/...` remain project-local. Resolve every `tools/...` and `prompts/...` resource against `{distilly_skill_root}`. For example, execute the bundled `tools/example.py` as `python3 "{distilly_skill_root}/tools/example.py"`; replace the placeholder with the resolved absolute path in the actual tool call. +> Keep the shell in the user's current workspace so relative output paths such as `./skills/...` remain project-local. Resolve every `prompts/...` resource against `{distilly_skill_root}`. The only supported entrypoint is the `distilly` CLI; do not call the bundled Python tools directly (they are deprecated, see the migration table below). > -> 在读取内置 prompt 或运行脚本前，先取得宿主实际加载的这份 `SKILL.md` 所在绝对目录；下文以 `{distilly_skill_root}` 表示。Claude Code 可用 `${CLAUDE_SKILL_DIR}`，其他宿主使用其 Skill discovery 上下文提供的实际路径。不要假定 shell 当前目录就是 Skill 目录，也不要猜测或硬编码安装路径。shell 应继续停留在用户工作区，使 `./skills/...` 等输出仍写入当前项目；所有 `tools/...`、`prompts/...` 都必须从 `{distilly_skill_root}` 解析。 +> 在读取内置 prompt 或执行内置命令前，先取得宿主实际加载的这份 `SKILL.md` 所在绝对目录；下文以 `{distilly_skill_root}` 表示。Claude Code 可用 `${CLAUDE_SKILL_DIR}`，其他宿主使用其 Skill discovery 上下文提供的实际路径。不要假定 shell 当前目录就是 Skill 目录，也不要猜测或硬编码安装路径。shell 应继续停留在用户工作区，使 `./skills/...` 等输出仍写入当前项目；所有 `prompts/...` 都必须从 `{distilly_skill_root}` 解析。唯一受支持的入口是 `distilly` CLI，不要直接调用仓库里的 Python 工具（它们已废弃，见下方迁移对照表）。 # Distilly 创建器 @@ -52,613 +52,220 @@ Grok Bot 可以把流程保存为 private Skill，但目前没有官方的本地 兼容更新别名： - `/update-colleague {slug}` -当用户要求查看已生成的 Skill 时，执行下方“管理操作”里的列出命令。 +当用户要求查看已生成的 Skill 时，执行下方"管理操作"里的列出命令。 --- -## 工具使用规则 +## 命令契约（唯一入口） -本 Skill 运行在任意兼容宿主中，只要求宿主能够读取本地文件并执行 Bash / Python 命令。使用以下工具约定： +所有采集、派生、渲染都走 `distilly`。命令名与 `docs/v2/CONTRACT.md` §1 的命令表逐字一致；不要发明子命令或字段。 -| 任务 | 使用工具 | -|------|---------| -| 读取 PDF 文档 | `Read` 工具（原生支持 PDF） | -| 读取图片截图 | `Read` 工具（原生支持图片） | -| 读取 MD/TXT 文件 | `Read` 工具 | -| 解析飞书消息 JSON 导出 | `Bash` → `python3 "{distilly_skill_root}/tools/feishu_parser.py"` | -| 飞书全自动采集（推荐） | `Bash` → `python3 "{distilly_skill_root}/tools/feishu_auto_collector.py"` | -| 飞书文档（浏览器登录态） | `Bash` → `python3 "{distilly_skill_root}/tools/feishu_browser.py"` | -| 飞书文档（MCP App Token） | `Bash` → `python3 "{distilly_skill_root}/tools/feishu_mcp_client.py"` | -| 钉钉全自动采集 | `Bash` → `python3 "{distilly_skill_root}/tools/dingtalk_auto_collector.py"` | -| 采集公开 X 帖子候选证据 | `Bash` → `python3 "{distilly_skill_root}/tools/research/xquik_public_posts.py"` | -| 解析邮件 .eml/.mbox | `Bash` → `python3 "{distilly_skill_root}/tools/email_parser.py"` | -| 写入/更新 Skill 文件 | `Write` / `Edit` 工具 | -| 版本管理 | `Bash` → `python3 "{distilly_skill_root}/tools/version_manager.py"` | -| 列出已有 Skill | `Bash` → `python3 "{distilly_skill_root}/tools/skill_writer.py" --action list` | - -**基础目录**： -- `colleague` → `./skills/colleague/{slug}/` -- `relationship` → `./skills/relationship/{slug}/` -- `celebrity` → `./skills/celebrity/{slug}/` - -如需改为全局路径，用 `--base-dir` 指向对应 character family 的根目录。 +| 任务 | 命令 | +|------|------| +| 零凭据：目录/文件 → `knowledge/` | `distilly harvest ` | +| 解析 ChatGPT / Claude / Slack / Telegram / Discord 导出 | `distilly parse-chat ` | +| 解析邮件 | `distilly parse-email ` | +| 解析字幕 | `distilly parse-subtitle ` | +| 解析文档 | `distilly parse-doc ` | +| 解析归档（X 官方归档 / Takeout / 社交平台导出） | `distilly parse-archive ` | +| 纯派生 → `evidence/derived/*.json` | `distilly retrospect` | +| 需要 key / OAuth 的渠道采集 | `distilly collect ` | +| 浏览器 computer use（必须带同意 token） | `distilly collect x --mode browser --consent ` | +| 音视频转写（可选后端） | `distilly transcribe ` | +| 把 LLM 自己读到的内容登记进账本 | `distilly note --from ` | +| 同意授权管理 | `distilly consent ` | +| 视图检查与渲染 | `distilly view check`；`distilly view render [--shareable]` | +| 证据体检 | `distilly doctor` | +| 生成 Skill 的创建/更新/列出/版本 | `distilly skill ` | +| 宿主安装 | `distilly install `；`distilly uninstall` | + +- 所有子命令支持 `--json` 回执；`--help` 有中文/英文两段。 +- 需要 key / OAuth 的渠道：先向用户说明将读取哪个渠道、可拿到什么，拿到同意后才运行 `distilly collect`。 +- computer-use 类命令必须带 `--consent `；没有 token 时命令以 `exit 2` 结束并在回执里写"等待用户同意"。 +- 密钥只从 `~/.distilly/*_config.json` 或环境变量读取；回执、日志、对话里只出现配置文件名，永不出现值。 + +### 迁移对照表（旧写法一律 deprecated） + +| 旧写法（deprecated） | 新写法 | +|----------------------|--------| +| `tools/feishu_parser.py`（deprecated） | `distilly parse-chat` | +| `tools/feishu_auto_collector.py`（deprecated） | `distilly collect feishu` | +| `tools/feishu_browser.py`（deprecated） | `distilly collect feishu --mode browser --consent ` | +| `tools/feishu_mcp_client.py`（deprecated） | `distilly collect feishu` | +| `tools/dingtalk_auto_collector.py`（deprecated） | `distilly collect dingtalk` | +| `tools/email_parser.py`（deprecated） | `distilly parse-email` | +| `tools/research/xquik_public_posts.py`（deprecated） | `distilly collect x` | +| `tools/research/transcribe_audio.py`（deprecated） | `distilly transcribe` | +| `tools/research/srt_to_transcript.py`（deprecated） | `distilly parse-subtitle` | +| `tools/skill_writer.py`（deprecated） | `distilly skill create` / `distilly skill update` / `distilly skill list` | +| `tools/version_manager.py`（deprecated） | `distilly skill version` | +| `tools/install_generated_skill.py`（deprecated） | `distilly install ` | +| `tools/research/quality_check.py`（deprecated） | `distilly doctor` | +| `tools/research/merge_research.py`（deprecated） | 暂无契约替代：只做派生，走 `distilly retrospect`，研究笔记合并细节见已知缺口 | +| `tools/research/download_subtitles.sh`（deprecated） | 暂无契约替代：先让用户提供本地字幕文件，再走 `distilly parse-subtitle` | + +迁移期允许两者并存，但新写法优先；只要 Python 工具还在被引用，就必须保留 `deprecated` 标注。 --- -## 主流程：创建新 Skill - -### Step 0：确认 character family - -如果用户使用的是 `/distilly`，先确认本次要蒸馏的是哪一类： - -1. `colleague` -2. `relationship` -3. `celebrity` - -如果上层宿主已经显式把 family 传进来，则直接固定对应的 character family。 - -如果当前 family 是 `celebrity`，还必须确认 research profile： - -1. `budget-friendly` -2. `budget-unfriendly` - -默认使用 `budget-friendly`。只有当用户明确要求更深研究、更高置信度、或者愿意接受更慢更贵的蒸馏流程时，才切到 `budget-unfriendly`。 - -### Step 1：基础信息录入 - -根据 character family 选择对应 intake prompt： - -- `colleague` → `prompts/intake.md` -- `relationship` → `prompts/relationship/intake.md` -- `celebrity` → `prompts/celebrity/intake.md` - -`colleague` 和 `relationship` 只问 3 个问题。 -`celebrity` 按 `prompts/celebrity/intake.md` 问 4 个问题，其中第 4 个问题必须确认 `research_profile`。 - -默认的 3 个基础问题： - -1. **花名/代号**（必填） -2. **基本信息**（一句话：公司、职级、职位、性别，想到什么写什么） - - 示例：`字节 2-1 后端工程师 男` -3. **性格画像**（一句话：MBTI、星座、个性标签、企业文化、印象） - - 示例：`INTJ 摩羯座 甩锅高手 字节范 CR很严格但从来不解释原因` - -除姓名外均可跳过。收集完后汇总确认，再进入下一步。 - -### Step 2：原材料导入 - -询问用户提供原材料，展示四种方式供选择： +## 磁盘契约（LLM 只能写这些） ``` -原材料怎么提供？ +skills/// + SKILL.md work.md persona.md work_skill.md persona_skill.md manifest.json meta.json + knowledge/{docs,messages,emails}/ + knowledge/raw//... # 原样字节，只增不改 + knowledge/text/.md # 归一化正文，段落锚点 [k0012] / [k0012:t3] + knowledge/index.json # 账本 {id,kind,origin,fetched_at,bytes,sha256,credentialed,method,warnings[]} + evidence/derived/*.json # retrospect 派生，每条结论带 evidence 锚点 + views/.view.json # LLM 只写章节/顺序/强调（不含事实） + views/.html # render 产物：单文件、离线、双主题 + evidence/renders/receipt.json # render 回执（sha256 + 字节数 + 内联来源） +``` - [A] 飞书自动采集（推荐） - 输入姓名，自动拉取消息记录 + 文档 + 多维表格 +- LLM 可以写：`views/.view.json`（只写章节、顺序、强调）、临时工作文件、以及通过 `distilly note --from ` 登记的"model-read"来源。 +- LLM 不可以写：`knowledge/raw/**`（原样字节，只增不改）、`knowledge/index.json`、`evidence/derived/*.json`（由 `distilly retrospect` 生成）、`evidence/renders/receipt.json`。 +- 截图、回执图、diff 图不入库（`.gitignore` 已含 `dst-evidence/`）；本地产物放 `/tmp/dst-evidence//`。 +- 锚点格式统一为 `[k00NN]`（4 位补零）或 `[k00NN:tM]`（带轮次）。任何结论必须带 `文件 + 锚点`，没有证据就写 `unknown`。 - [B] 钉钉自动采集 - 输入姓名，自动拉取文档 + 多维表格 - 消息记录通过浏览器采集（钉钉 API 不支持历史消息） +--- - [C] 飞书链接 - 直接给文档/Wiki 链接（浏览器登录态 或 MCP） +## 五步主线 - [D] 上传文件 - PDF / 图片 / 导出 JSON / 邮件 .eml +创建、追加、纠正都走同一条主线：**Collect → Derive → Read → Distill → Render**。 - [E] 直接粘贴内容 - 把文字复制进来 +| 步骤 | 必须存在的产物 | 计数判据 | sha256 从哪来 | 失败怎么办 | +|------|----------------|----------|---------------|------------| +| 1 Collect | `knowledge/raw//**`、`knowledge/text/.md`、`knowledge/index.json` | 每个落地来源 1 条账本条目；每个 text 文件 ≥1 个锚点 | `distilly --json` 回执的 `outputs[].sha256`，与 `knowledge/index.json` 的 `sha256` 逐字节一致 | 非零退出：记录命令、stderr、补救步骤；0 条落地来源时停下，不得进入 Derive | +| 2 Derive | `evidence/derived/*.json` | 每条派生结论带 evidence 锚点；连跑两次字节相同 | 回执 `outputs[].sha256`；两次运行 sha256 相同 | 非零退出：先修 `knowledge/index.json` 完整性；不得手写派生 JSON | +| 3 Read | 无新文件，产出"读了什么"的复述 | 按文件列出：文件 → 条数 → 锚点数 | 引用 `knowledge/index.json` 的 `sha256`，不自算 | 文件缺失或锚点为 0：回到 Step 1 补齐，不得凭记忆写结论 | +| 4 Distill | `work.md`、`persona.md`，celebrity 另有 research/audit/synthesis/validation | 每个维度有锚点或 `unknown`；celebrity 有明确 `PASS/FAIL` | 引用被引用的来源 sha256（来自账本） | 证据不足：标 `（原材料不足）` / candidate，并说明需要补什么材料 | +| 5 Render | `views/.view.json`、`views/.html`、`evidence/renders/receipt.json` | 回执 sha256 与 html 实际 sha256 一致；`distilly doctor` 锚点回指率可查 | `evidence/renders/receipt.json` 的 sha256 | 渲染失败：保留 view.json，不发布，报告错误 | -可以混用，也可以跳过（仅凭手动信息生成）。 -``` +任何一步的失败都不允许"静默降级"：要么修好，要么把失败写进对用户的汇报和回执的 `warnings[]` / `unavailable[]`。 ---- +### Step 1：Collect（采集） -#### 方式 A：飞书自动采集（推荐） +1. 先读 `prompts/collectors.md`，按"什么时候用哪条命令"选路。 +2. 零凭据来源（本地文件、导出包、字幕、文档、归档）直接走 `distilly harvest`、`distilly parse-chat`、`distilly parse-email`、`distilly parse-subtitle`、`distilly parse-doc`、`distilly parse-archive`。 +3. 需要 key / OAuth 的渠道（飞书、Slack、钉钉、X、Discord、Reddit、Notion、Gmail）先征求用户同意，再走 `distilly collect `；同意范围用 `distilly consent ` 管理。 +4. 浏览器 computer use 必须按 `prompts/computer-use.md` 执行：先问再动、只读白名单、默认 ≤20 屏 / ≤10 分钟 / 每分钟 ≤6 次滚动、每屏落盘原文 + URL + 时间 + 截图（截图只放本地）、可中断；`distilly collect x --mode browser --consent ` 没有 token 就直接退出，不要绕过。 +5. 用户只能"贴文字/截图"时，用 `distilly note --from ` 登记来源（`method:"model-read"`），不要假装它是采集来的。 +6. 音视频先 `distilly transcribe`，再解析字幕；不要把整段 transcript 抄进仓库。 -首次使用需配置： -```bash -python3 "{distilly_skill_root}/tools/feishu_auto_collector.py" --setup -``` +**完成判据**：`knowledge/index.json` 里每个落地来源一条账本条目（含 `id`、`kind`、`origin`、`fetched_at`、`bytes`、`sha256`、`credentialed`、`method`、`warnings[]`）；每个 `knowledge/text/.md` 至少 1 个锚点；回执 `inputs[]`/`outputs[]` 的 sha256 与账本一致；不可用渠道进 `unavailable[]`。 +**失败怎么办**：命令非零退出时，把命令原文、stderr、补救步骤（例如缺凭据要配置哪个 `~/.distilly/*_config.json`）告诉用户，然后停下等指示；如果 0 条来源落地，不要进入 Step 2。 -**群聊采集**（使用 tenant_access_token，需 bot 在群内）： -```bash -python3 "{distilly_skill_root}/tools/feishu_auto_collector.py" \ - --name "{name}" \ - --output-dir ./knowledge/{slug} \ - --msg-limit 1000 \ - --doc-limit 20 -``` +### Step 2：Derive（派生） -**私聊采集**（需要 user_access_token + 私聊 chat_id）： - -私聊消息只能通过用户身份（user_access_token）获取，应用身份无权访问私聊。 - -**前置条件**： - -用户需要提供以下信息： -1. **飞书应用凭证**：`app_id` 和 `app_secret`（在飞书开放平台创建自建应用获取） -2. **用户权限**：应用需开通以下用户权限（scope）： - - `im:message` — 以用户身份读取/发送消息 - - `im:chat` — 以用户身份读取会话列表 -3. **OAuth 授权码（code）**：用户在浏览器中完成 OAuth 授权后，从回调 URL 中获取 - -如果用户缺少以上任何信息，引导他们完成配置。不要假设用户已经配好了。 - -**获取 user_access_token 的完整流程**： - -当用户提供了 app_id、app_secret，并确认已开通用户权限后： - -1. 帮用户生成 OAuth 授权链接： - ``` - https://open.feishu.cn/open-apis/authen/v1/authorize?app_id={APP_ID}&redirect_uri=http://www.example.com&scope=im:message%20im:chat - ``` - > ⚠️ 注意：`redirect_uri` 需要在飞书应用的「安全设置 → 重定向 URL」中添加 `http://www.example.com` - -2. 用户在浏览器打开链接，登录并授权 -3. 页面会跳转到 `http://www.example.com?code=xxx`，用户复制 code 给你 -4. 用 code 换取 token： - ```bash - python3 "{distilly_skill_root}/tools/feishu_auto_collector.py" --exchange-code {CODE} - ``` - 或者你自己写 Python 脚本调飞书 API 换取： - ```python - # 1. 获取 app_access_token - POST https://open.feishu.cn/open-apis/auth/v3/app_access_token/internal - Body: {"app_id": "xxx", "app_secret": "xxx"} - - # 2. 用 code 换 user_access_token - POST https://open.feishu.cn/open-apis/authen/v1/oidc/access_token - Header: Authorization: Bearer {app_access_token} - Body: {"grant_type": "authorization_code", "code": "xxx"} - ``` - -**获取私聊 chat_id**： - -用户通常不知道 chat_id。当用户有了 user_access_token 但没有 chat_id 时，你应该**自己写 Python 脚本**来获取： - -- **方法**：用 user_access_token 向对方的 open_id 发一条消息，返回值中会包含 chat_id - ```python - POST https://open.feishu.cn/open-apis/im/v1/messages?receive_id_type=open_id - Header: Authorization: Bearer {user_access_token} - Body: {"receive_id": "{对方open_id}", "msg_type": "text", "content": "{\"text\":\"你好\"}"} - # 返回值中的 chat_id 就是私聊会话 ID - ``` -- **注意**：`GET /im/v1/chats` 不会返回私聊会话，这是飞书 API 的限制，不是权限问题，不要尝试用这个接口找私聊 -- 如果用户不知道对方的 open_id，可以用 tenant_access_token 调通讯录 API 搜索： - ```python - GET https://open.feishu.cn/open-apis/contact/v3/scopes - # 返回应用可见范围内所有用户的 open_id - ``` - -**执行采集**： - -拿到 user_access_token 和 chat_id 后： -```bash -python3 "{distilly_skill_root}/tools/feishu_auto_collector.py" \ - --open-id {对方open_id} \ - --p2p-chat-id {chat_id} \ - --user-token {user_access_token} \ - --name "{name}" \ - --output-dir ./knowledge/{slug} \ - --msg-limit 1000 -``` +1. 派生之前不要读 `evidence/derived/*`——先跑 `distilly retrospect`。 +2. `distilly retrospect` 只做纯派生：输入是 `knowledge/**`，输出是 `evidence/derived/*.json`，每条结论带 evidence 锚点。 +3. 为验证确定性，连跑两次；同一输入两次的 sha256 必须相同。 -**灵活性原则**：以上 API 调用不一定要用 collector 脚本，如果脚本跑不通或者场景不匹配，你可以直接写 Python 脚本调飞书 API 完成任务。核心 API 参考： -- 获取 token：`POST /auth/v3/app_access_token/internal`、`POST /authen/v1/oidc/access_token` -- 发消息（获取 chat_id）：`POST /im/v1/messages?receive_id_type=open_id` -- 拉消息：`GET /im/v1/messages?container_id_type=chat&container_id={chat_id}` -- 查通讯录：`GET /contact/v3/scopes`、`GET /contact/v3/users/{user_id}` - -自动采集内容： -- 群聊：所有与他共同群聊中他发出的消息（过滤系统消息、表情包） -- 私聊：与他的私聊完整对话（含双方消息，用于理解对话语境） -- 他创建/编辑的飞书文档和 Wiki -- 相关多维表格（如有权限） - -采集完成后用 `Read` 读取输出目录下的文件： -- `knowledge/{slug}/messages.txt` → 消息记录（群聊 + 私聊） -- `knowledge/{slug}/docs.txt` → 文档内容 -- `knowledge/{slug}/collection_summary.json` → 采集摘要 - -如果采集失败，根据报错自行判断原因并尝试修复，常见问题： -- 群聊采集：bot 未添加到群聊 -- 私聊采集：user_access_token 过期（有效期 2 小时，可用 refresh_token 刷新） -- 权限不足：引导用户在飞书开放平台开通对应权限并重新授权 -- 或改用方式 B/C +**完成判据**：`evidence/derived/*.json` 存在；回执给出 `anchors.total` / `anchors.cited`；两次运行 `outputs[].sha256` 相同。 +**失败怎么办**：非零退出说明输入侧有问题——回到 Step 1 检查账本与 text 锚点；绝不手写、手改派生 JSON 来"跑通"。 ---- +### Step 3：Read（阅读） -#### 方式 B：钉钉自动采集 +1. 读的顺序：`knowledge/index.json` → `knowledge/text/*.md` → `evidence/derived/*.json`。 +2. 先向用户复述"读了哪些文件、各多少条、多少锚点"，再写结论。 +3. 每条结论后面跟 `文件 + 锚点`（例如 `knowledge/text/feishu.md [k0042]`）。 +4. 找不到证据的结论写 `unknown`，并说明缺什么材料可以补上。 +5. 事实与候选分开：有具体锚点支撑的才算事实；派生文件里的模式、倾向、推断一律按候选处理，候选不能升级为结论。 +6. 全文细节规范见 `prompts/retrospection.md`。 -首次使用需配置： -```bash -python3 "{distilly_skill_root}/tools/dingtalk_auto_collector.py" --setup -``` +**完成判据**：复述清单里的每个文件都能在账本里回指；被引用的锚点都真实存在于 `knowledge/text/**`；没有无锚点的结论。 +**失败怎么办**：文件缺失或锚点为 0 时回到 Step 1 补齐；不要凭记忆或常识补写内容。 -然后输入姓名，一键采集： -```bash -python3 "{distilly_skill_root}/tools/dingtalk_auto_collector.py" \ - --name "{name}" \ - --output-dir ./knowledge/{slug} \ - --msg-limit 500 \ - --doc-limit 20 \ - --show-browser # 首次使用加此参数，完成钉钉登录 -``` +### Step 4：Distill（蒸馏） -采集内容： -- 他创建/编辑的钉钉文档和知识库 -- 多维表格 -- 消息记录（⚠️ 钉钉 API 不支持历史消息拉取，自动切换浏览器采集） +先用第 0 步确认的 family 解析执行矩阵： -采集完成后 `Read` 读取： -- `knowledge/{slug}/docs.txt` -- `knowledge/{slug}/bitables.txt` -- `knowledge/{slug}/messages.txt` +| character | intake | persona analyzer | persona builder | merger | storage root | +|-----------|--------|------------------|-----------------|--------|--------------| +| `colleague` | `prompts/intake.md` | `prompts/persona_analyzer.md` | `prompts/persona_builder.md` | `prompts/merger.md` | `./skills/colleague/{slug}` | +| `relationship` | `prompts/relationship/intake.md` | `prompts/relationship/persona_analyzer.md` | `prompts/relationship/persona_builder.md` | `prompts/relationship/merger.md` | `./skills/relationship/{slug}` | +| `celebrity` | `prompts/celebrity/intake.md` | `prompts/celebrity/persona_analyzer.md` | `prompts/celebrity/persona_builder.md` | `prompts/celebrity/merger.md` | `./skills/celebrity/{slug}` | -如消息采集失败，提示用户截图聊天记录后上传。 +所有 family 共用：Work analyzer `prompts/work_analyzer.md`、Work builder `prompts/work_builder.md`、Correction handler `prompts/correction_handler.md`。 ---- +两条线： -#### 方式 D：上传文件 - -- **PDF / 图片**：`Read` 工具直接读取 -- **飞书消息 JSON 导出**： - ```bash - python3 "{distilly_skill_root}/tools/feishu_parser.py" --file {path} --target "{name}" --output /tmp/feishu_out.txt - ``` - 然后 `Read /tmp/feishu_out.txt` -- **邮件文件 .eml / .mbox**： - ```bash - python3 "{distilly_skill_root}/tools/email_parser.py" --file {path} --target "{name}" --output /tmp/email_out.txt - ``` - 然后 `Read /tmp/email_out.txt` -- **Markdown / TXT**：`Read` 工具直接读取 +- **线路 A（Work Skill）**：参考 `prompts/work_analyzer.md`，提取负责系统、技术规范、工作流程、输出偏好、经验知识；celebrity 场景下 `work` 更偏方法论、判断框架、决策习惯。 +- **线路 B（Persona）**：用当前 family 的 persona analyzer；`celebrity` + `research_profile=budget-unfriendly` 时改用 `prompts/celebrity/budget_unfriendly/persona_analyzer.md`。把用户填的标签翻译为具体行为规则，并从材料里提取表达风格、决策模式、人际行为。 ---- +写文件时不要手工拼 `skills/{family}/{slug}` 文件树，统一走 writer：把 `meta.json` / `work.md` / `persona.md` 写到临时文件，再调 `distilly skill create`（或 `distilly skill update`）。人物 Skill 的安装走 `distilly install `。 -#### 方式 C：飞书链接 +**完成判据**：每个维度都有锚点或明确的 `（原材料不足）`；每条行为规则具体可执行；celebrity 的 audit / validation 给出明确 `PASS` 或 `FAIL`；`distilly doctor` 能报告证据覆盖率、不可用渠道、锚点回指率。celebrity 场景下的 research 门槛见下方子流程。 +**失败怎么办**：证据不足的维度标 `（原材料不足，建议追加相关文档）` 并降级为 candidate；`source_grounding` 不达标时保留 `FAIL` 并说明还缺什么，绝不用泛化链接刷过检查。 -用户提供飞书文档/Wiki 链接时，询问读取方式： +### Step 5：Render（渲染） -``` -检测到飞书链接，选择读取方式： - - [1] 浏览器方案（推荐） - 复用你本机 Chrome 的登录状态 - ✅ 内部文档、需要权限的文档都能读 - ✅ 无需配置 token - ⚠️ 需要本机安装 Chrome + playwright - - [2] MCP 方案 - 通过飞书 App Token 调用官方 API - ✅ 稳定，不依赖浏览器 - ✅ 可以读消息记录（需要群聊 ID） - ⚠️ 需要先配置 App ID / App Secret - ⚠️ 内部文档需要管理员给应用授权 - -选择 [1/2]： -``` +1. 先 `distilly view check`，确认锚点都能回指到 `knowledge/index.json`。 +2. 写 `views/.view.json`：只写章节、顺序、强调，不写事实。 +3. `distilly view render` 生成单文件、离线、双主题的 `views/.html`，并写 `evidence/renders/receipt.json`（sha256 + 字节数 + 内联来源）。 +4. 对外分享时才用 `distilly view render --shareable`，并先让用户确认。 +5. 用 `distilly doctor` 复核证据覆盖率、不可用渠道、锚点回指率、computer-use 占比。 -**选 1（浏览器方案）**： -```bash -python3 "{distilly_skill_root}/tools/feishu_browser.py" \ - --url "{feishu_url}" \ - --target "{name}" \ - --output /tmp/feishu_doc_out.txt -``` -首次使用若未登录，会弹出浏览器窗口要求登录（一次性）。 - -**选 2（MCP 方案）**： - -首次使用需初始化配置： -```bash -python3 "{distilly_skill_root}/tools/feishu_mcp_client.py" --setup -``` - -之后直接读取： -```bash -python3 "{distilly_skill_root}/tools/feishu_mcp_client.py" \ - --url "{feishu_url}" \ - --output /tmp/feishu_doc_out.txt -``` +**完成判据**：`views/.html` 与 `evidence/renders/receipt.json` 同时存在；回执 sha256 与 html 实际 sha256 一致；内部链接 0 坏链。 +**失败怎么办**：渲染失败时保留 `views/.view.json`，不发布 HTML，把错误与缺失来源报告给用户。 -读取消息记录（需要群聊 ID，格式 `oc_xxx`）： -```bash -python3 "{distilly_skill_root}/tools/feishu_mcp_client.py" \ - --chat-id "oc_xxx" \ - --target "{name}" \ - --limit 500 \ - --output /tmp/feishu_msg_out.txt -``` +### 第 0 步（前置）：确认 family 与 intake -两种方式输出后均用 `Read` 读取结果文件，进入分析流程。 +如果用户使用的是 `/distilly`，先确认本次要蒸馏的是哪一类： ---- +1. `colleague` +2. `relationship` +3. `celebrity` -#### 方式 E：直接粘贴 +如果上层宿主已经显式把 family 传进来，则直接固定对应的 character family。 -用户粘贴的内容直接作为文本原材料，无需调用任何工具。 +如果当前 family 是 `celebrity`，还必须确认 research profile： ---- +1. `budget-friendly` +2. `budget-unfriendly` -如果用户说"没有文件"或"跳过"，仅凭 Step 1 的手动信息生成 Skill。 +默认使用 `budget-friendly`。只有当用户明确要求更深研究、更高置信度、或者愿意接受更慢更贵的蒸馏流程时，才切到 `budget-unfriendly`。 -### Step 3：分析原材料 +根据 family 选择 intake prompt：`colleague` → `prompts/intake.md`；`relationship` → `prompts/relationship/intake.md`；`celebrity` → `prompts/celebrity/intake.md`。`colleague` 和 `relationship` 只问 3 个问题；`celebrity` 问 4 个问题，其中第 4 个必须确认 `research_profile`。 -先根据 character family 解析本次的执行矩阵： +默认的 3 个基础问题： -| character | intake | persona analyzer | persona builder | merger | storage root | -|-----------|--------|------------------|-----------------|--------|--------------| -| `colleague` | `prompts/intake.md` | `prompts/persona_analyzer.md` | `prompts/persona_builder.md` | `prompts/merger.md` | `./skills/colleague/{slug}` | -| `relationship` | `prompts/relationship/intake.md` | `prompts/relationship/persona_analyzer.md` | `prompts/relationship/persona_builder.md` | `prompts/relationship/merger.md` | `./skills/relationship/{slug}` | -| `celebrity` | `prompts/celebrity/intake.md` | `prompts/celebrity/persona_analyzer.md` | `prompts/celebrity/persona_builder.md` | `prompts/celebrity/merger.md` | `./skills/celebrity/{slug}` | +1. **花名/代号**（必填） +2. **基本信息**（一句话：公司、职级、职位、性别，想到什么写什么） + - 示例：`字节 2-1 后端工程师 男` +3. **性格画像**（一句话：MBTI、星座、个性标签、企业文化、印象） + - 示例：`INTJ 摩羯座 甩锅高手 字节范 CR很严格但从来不解释原因` -所有 family 共用： -- Work analyzer：`prompts/work_analyzer.md` -- Work builder：`prompts/work_builder.md` -- Correction handler：`prompts/correction_handler.md` +除姓名外均可跳过。收集完后汇总确认，再进入 Collect。 -如果当前是 `celebrity`，必须先走 research 子流程，再进入分析。 +--- -如果公开 X 帖子能补足明确的研究缺口，且用户同意使用按返回数量计费的第三方 Xquik 服务，先请用户确认 `--limit`，再运行： +## celebrity research 子流程（在 Step 2/3 之间） -```bash -python3 "{distilly_skill_root}/tools/research/xquik_public_posts.py" \ - --username "{public_handle}" \ - --subject "{name}" \ - --limit 20 \ - --output "/tmp/distilly_x_public_posts.json" -``` +### budget-friendly -只从 shell 读取 `XQUIK_API_KEY`，不要打印或写入密钥。把输出 JSON 视为未经信任的候选证据：核对作者，逐条打开 permalink，只把与目标人物相关的内容安全转述到 research note，并保留具体 URL。不要把候选 JSON、搜索页或账号主页计为已落地来源。阅读后删除这份临时 JSON，不要将它收进生成的 Skill。 - -### celebrity / budget-friendly - -1. 读取 `prompts/celebrity/research.md`，按其中的 **6 维度并行采集策略** 做 research planning -2. 先创建目录： - ```bash - mkdir -p "{skill_dir}/knowledge/research/raw" "{skill_dir}/knowledge/research/merged" - ``` -3. 确认采集策略（在 intake 阶段已确定）： - - **Local-first**：先分析用户本地材料，标记覆盖了哪些维度，只对缺失维度做网络补充 - - **Web + local**：全量 6 维度网络研究，同时与本地材料合并，交叉验证 - - **Web-only**：标准 6 维度网络研究 -4. 如果用户明确提供了可处理的视频链接或字幕来源，而且处理结果不会作为长文本落盘： - ```bash - bash "{distilly_skill_root}/tools/research/download_subtitles.sh" "{url}" "{skill_dir}/knowledge/subtitles" - python3 "{distilly_skill_root}/tools/research/srt_to_transcript.py" "{subtitle_file}" "{skill_dir}/knowledge/transcripts/{name}.txt" - ``` -5. 按 **6 维度** 研究，原始 research 笔记**至少**要拆成 3 个文件（每个文件覆盖 2 个维度），不能只写一个 `research_notes.md`： +1. 读 `prompts/celebrity/research.md`，按其中的 **6 维度并行采集策略** 做 research planning。 +2. 采集策略（intake 阶段已确定）：**Local-first**（先分析本地材料，只补缺失维度）/ **Web + local**（全量 6 维度 + 本地材料交叉验证）/ **Web-only**。 +3. 需要视频/播客时：先 `distilly transcribe `，字幕走 `distilly parse-subtitle`；不要把完整 transcript 落进仓库。 +4. 原始 research 笔记**至少**拆成 3 个文件（每个覆盖 2 个维度），不能只写一个 `research_notes.md`： - `knowledge/research/raw/01_core_profile.md`（维度 1 著作 + 维度 6 时间线） - `knowledge/research/raw/02_conversations_and_material.md`（维度 2 对话 + 维度 4 决策） - `knowledge/research/raw/03_expression_and_reception.md`（维度 3 表达 DNA + 维度 5 他者视角） -6. 研究过程中必须遵守 **品味原则**（详见 research prompt）： - - 长文 > 金句，争议 > 共识，变化 > 固定，一手 > 二手 - - 遵守 **信源黑名单**：永不引用知乎、微信公众号、百度百科、内容农场 - - 遵守 **信源优先级**：用户本地材料 > 一手著作 > 长访谈 > 决策记录 > 社交媒体 > 外部分析 > 二手转述 -7. 合并 research： - ```bash - python3 "{distilly_skill_root}/tools/research/merge_research.py" "{skill_dir}" - ``` - 输出：`knowledge/research/merged/summary.md` -8. 读取 `knowledge/research/merged/summary.md`，确认： - - `Files scanned >= 3` - - `Unique URLs >= 2` - - `Potential long quote lines = 0` - - research notes 里的 URL 必须是**实际打开过的具体页面**，不是平台首页、搜索页、话题页或占位路径 - 如果不满足，继续补 research notes，直到满足或明确记录搜集受限原因。 -9. **质量关卡（Phase 1.5）**：在进入分析之前，必须向用户展示结构化采集摘要： - ``` - ┌──────────────────────────────┬──────────┬─────────────────────────────┐ - │ 维度 │ 来源数 │ 关键发现 │ - ├──────────────────────────────┼──────────┼─────────────────────────────┤ - │ 1 著作 │ N │ [核心论点 / 缺失] │ - │ 2 对话 │ N │ [关键模式 / 缺失] │ - │ 3 表达 DNA │ N │ [风格标记 / 缺失] │ - │ 4 决策 │ N │ [决策模式 / 缺失] │ - │ 5 他者视角 │ N │ [外部观点 / 缺失] │ - │ 6 时间线 │ N │ [认知轨迹 / 缺失] │ - ├──────────────────────────────┼──────────┼─────────────────────────────┤ - │ 矛盾点 │ N │ [摘要] │ - │ 薄弱维度 │ [列表] │ 补充方案：[计划] │ - │ 冷门人物？ │ 是/否 │ │ - └──────────────────────────────┴──────────┴─────────────────────────────┘ - ``` - 等待用户确认后再继续。如果用户指出问题或需要某个维度更深入，先补充研究。 -10. **冷门人物检测**：如果总来源 < 10 条，按冷门人物协议处理： - - 心智模型限制为 2–3 个 - - 薄弱模型标注"基于有限信息" - - 扩大诚实边界章节 - - 告知用户提供什么补充材料可以改善质量 -11. celebrity 的后续分析输入必须优先使用： - - 一手材料（信源权重 1-3） - - merged research summary - - 用户提供的补充描述 - -### celebrity / budget-unfriendly - -1. 先读取： - - `prompts/celebrity/budget_unfriendly/research.md` - - `references/celebrity_budget_unfriendly_framework.md` -2. 先创建目录： - ```bash - mkdir -p "{skill_dir}/knowledge/research/raw" "{skill_dir}/knowledge/research/merged" "{skill_dir}/knowledge/research/reviews" - ``` -3. 确认采集策略（在 intake 阶段已确定）：local-first / web+local / web-only -4. 按 **6-track 独立文件结构** 写 research notes（不可合并，不可克隆观察）： - - `knowledge/research/raw/01_writings.md`（维度 1：著作与系统思考） - - `knowledge/research/raw/02_conversations.md`（维度 2：即兴对话与压力应对） - - `knowledge/research/raw/03_expression_dna.md`（维度 3：语言指纹） - - `knowledge/research/raw/04_decisions.md`（维度 4：行为与选择） - - `knowledge/research/raw/05_external_views.md`（维度 5：他者视角与批评） - - `knowledge/research/raw/06_timeline.md`（维度 6：认知轨迹） -5. 研究过程必须遵守 **品味原则 + 信源黑名单 + 信源优先级**（见 research prompt），每条 evidence 必须标注 source weight (1-7)。 -6. 合并 research： - ```bash - python3 "{distilly_skill_root}/tools/research/merge_research.py" "{skill_dir}" - ``` -7. 读取 `knowledge/research/merged/summary.md`，确认最低门槛： - - `Files scanned >= 6` - - `Unique URLs >= 8` - - `Primary-source markers >= 3` - - `Source metadata blocks >= 6` - - `Contradiction bullets >= 6` - - `Inference bullets >= 6` - - `Potential long quote lines = 0` - - `Track coverage count = 6` - - research notes 里的 URL 必须是**实际打开过的具体页面**，不是平台首页、搜索页、话题页或占位路径 - 如果不满足，继续补对应 track，而不是直接进入后续 review。 -8. **质量关卡（Phase 1.5）**：在进入 audit 之前，向用户展示结构化采集摘要（含 primary 比例、矛盾数、候选 mental models、known-answer 候选、薄弱维度、冷门人物判定）。等待用户确认后再继续。 -9. 再读取： - - `prompts/celebrity/budget_unfriendly/audit.md` - - `prompts/celebrity/budget_unfriendly/synthesis.md` - - `references/celebrity_budget_unfriendly_template.md` -10. 先生成 `knowledge/research/reviews/research_audit.md` - - 审计必须明确给出 `PASS / FAIL` - - audit 必须检查：信源层级合规（无黑名单）、primary 比例 > 50%、品味原则遵守、冷门人物评估 - - 如果 audit 是 `FAIL`，按 audit 给出的 Backfill Tasks 补齐，不要跳到 synthesis -11. **提炼关卡（Phase 2.5）**：audit 通过后，向用户展示候选 mental models 摘要（含三重门判定、evidence anchors、failure modes）。确认合理性后再进入 synthesis。 -12. 再生成 `knowledge/research/reviews/synthesis.md` - - 必须对候选 mental models 做 triple-gate 判断： - - cross-context recurrence - - generative power - - exclusivity - - 同时提取智识谱系种子（influenced by / diverged from）和 Agentic Protocol 种子（该人物会如何分析新问题的维度列表） -13. 再按 `prompts/celebrity/budget_unfriendly/validation.md` 生成： - - `knowledge/research/reviews/validation.md` - - validation 必须明确给出 `PASS / FAIL` - - 必须做 known-answer check（至少 2 题）+ edge-case check（1 题）+ voice check（100 字盲测）+ copyright check + Agentic Protocol check - - 如果 validation 是 `FAIL`，必须先修 draft 再继续 -14. budget-unfriendly 的后续分析输入必须优先使用： - - 6-track raw notes - - merged research summary - - research audit - - synthesis review（含智识谱系种子、Agentic Protocol 种子） - - validation review - - 用户补充材料 - -两种 celebrity profile 的共同约束： - -- 如果外部搜集失败或被平台验证拦截： - - 明确告诉用户搜集受限的原因 - - 保留已有 research 原始材料和 merged summary - - 继续生成，但把 `source_grounding` 视为未完成 - - **不要**为了通过质量检查而编造 URL、引用、书名、视频标题，或塞入泛化主页链接 -- **不要**把完整 transcript、完整字幕、长段原文抄进仓库 -- 只允许保留结构化摘要、来源元信息和极短引用，避免版权风险 - -完成 family 解析后，再按两条线分析： - -**线路 A（Work Skill）**： -- 参考 `prompts/work_analyzer.md` -- 提取：负责系统、技术规范、工作流程、输出偏好、经验知识 -- celebrity 场景下，`work` 更偏方法论、判断框架、决策习惯，不要机械套成“工作职责” - -**线路 B（Persona）**： -- 使用当前 family 对应的 persona analyzer -- 如果 `celebrity` 且 `research_profile=budget-unfriendly`，改用： - - `prompts/celebrity/budget_unfriendly/persona_analyzer.md` -- 将用户填写的标签翻译为具体行为规则 -- 从原材料中提取：表达风格、决策模式、人际行为 -- celebrity 场景下，必须保留： - - mental models - - decision heuristics - - expression DNA - - contradictions - - honest boundaries - -### Step 4：生成并预览 - -使用 `prompts/work_builder.md` 生成 Work 内容。 -使用当前 family 对应的 persona builder 生成 Persona 内容。 - -具体映射： -- `colleague` → `prompts/persona_builder.md` -- `relationship` → `prompts/relationship/persona_builder.md` -- `celebrity` → `prompts/celebrity/persona_builder.md` -- `celebrity` + `budget-unfriendly` → `prompts/celebrity/budget_unfriendly/persona_builder.md` - -向用户展示摘要（各 5-8 行），询问： -``` -Work Skill 摘要： - - 负责：{xxx} - - 技术栈：{xxx} - - CR 重点：{xxx} - ... - -Persona 摘要： - - 核心性格：{xxx} - - 表达风格：{xxx} - - 决策模式：{xxx} - ... - -确认生成？还是需要调整？ -``` +5. 品味原则：长文 > 金句，争议 > 共识，变化 > 固定，一手 > 二手。信源黑名单：永不引用知乎、微信公众号、百度百科、内容农场。信源优先级：用户本地材料 > 一手著作 > 长访谈 > 决策记录 > 社交媒体 > 外部分析 > 二手转述。 +6. 合并研究笔记后确认 `Files scanned >= 3`、`Unique URLs >= 2`、`Potential long quote lines = 0`；notes 里的 URL 必须是实际打开过的具体页面，不是平台首页、搜索页、话题页或占位路径。 +7. **质量关卡（Phase 1.5）**：进入分析前向用户展示结构化采集摘要（6 维度来源数 + 关键发现 + 矛盾点 + 薄弱维度 + 冷门人物判定），等用户确认再继续。 +8. **冷门人物检测**：总来源 < 10 条时，心智模型限制为 2–3 个，薄弱模型标"基于有限信息"，扩大诚实边界章节，并告诉用户补什么材料能改善质量。 +9. 分析输入优先使用：一手材料（信源权重 1-3）> 合并后的 research summary > 用户补充描述。 + +### budget-unfriendly -### Step 5：写入文件 - -用户确认后，不要手工拼接 `skills/colleague/{slug}` 这类文件树。统一走 writer： - -1. 先解析当前 storage root： - - `colleague` → `./skills/colleague` - - `relationship` → `./skills/relationship` - - `celebrity` → `./skills/celebrity` -2. 用 `Write` 工具写三个临时文件： - - `/tmp/distilly_{slug}_meta.json` - - `/tmp/distilly_{slug}_work.md` - - `/tmp/distilly_{slug}_persona.md` -3. `meta.json` 至少包含： - - `name` - - `display_name` - - `character` - - `research_profile`（当 character=`celebrity` 时必填） - - `classification.language`（必须设置为用户当前语言，例如 `zh-CN` 或 `en`） - - `profile` - - `tags` - - `knowledge_sources` -4. 然后调用： - ```bash - python3 "{distilly_skill_root}/tools/skill_writer.py" \ - --action create \ - --character {character} \ - --research-profile {research_profile} \ - --slug {slug} \ - --name "{name}" \ - --meta /tmp/distilly_{slug}_meta.json \ - --work /tmp/distilly_{slug}_work.md \ - --persona /tmp/distilly_{slug}_persona.md \ - --base-dir {resolved_base_dir} - ``` -5. 该命令会统一生成： - - `SKILL.md` - - `work.md` - - `persona.md` - - `work_skill.md` - - `persona_skill.md` - - `manifest.json` - - `meta.json` - - 如需把生成后的角色 Skill 安装到宿主： - - Claude Code：追加 `--install-claude-skill` - - OpenClaw：追加 `--install-openclaw-skill` - - Codex：追加 `--install-codex-skill` - - Hermes：运行 `python3 "{distilly_skill_root}/tools/install_generated_skill.py" --skill-dir "{resolved_base_dir}/{slug}" --host hermes --force`；可信项目可追加 `--skills-dir .hermes/skills`，先运行 `hermes skills trust`，然后新建会话或运行 `/reload-skills`。只有已在 Hermes 的 `skills.external_dirs` 中显式配置时，才使用 `~/.agents/skills` - - DeepSeek Harness：运行 `python3 "{distilly_skill_root}/tools/install_generated_skill.py" --skill-dir "{resolved_base_dir}/{slug}" --host deepseek-harness --force`；项目级安装追加 `--skills-dir .dsh/skills` - - Pi：运行 `python3 "{distilly_skill_root}/tools/install_generated_skill.py" --skill-dir "{resolved_base_dir}/{slug}" --host pi --force`；项目级安装追加 `--skills-dir .pi/skills`，调用命令为 `/skill:{character}-{slug}` - - Grok Build：运行 `python3 "{distilly_skill_root}/tools/install_generated_skill.py" --skill-dir "{resolved_base_dir}/{slug}" --host grok-build --force`；项目级安装追加 `--skills-dir .grok/skills` - - OpenCode：运行 `python3 "{distilly_skill_root}/tools/install_generated_skill.py" --skill-dir "{resolved_base_dir}/{slug}" --host opencode --force`；项目级安装追加 `--skills-dir .opencode/skills` - - 统一安装器只写入自包含的 `SKILL.md` 和安装元数据，会在安装副本中规范旧版 frontmatter；不要手动复制整个生成目录，其中可能包含私有原始材料 - - Claude Code on Windows：可再追加 `--install-claude-command-shim` -6. 如果当前是 `celebrity`，创建完成后必须再跑一次质量检查： - ```bash - python3 "{distilly_skill_root}/tools/research/quality_check.py" "{resolved_base_dir}/{slug}/SKILL.md" --profile {research_profile} - ``` -7. 如果 `celebrity` 的质量检查仍然提示 `source_grounding` 失败： - - 可以补写诚实的来源说明和局限说明 - - 但只有在拿到真实、具体、可追溯的外部来源时，才能补充 URL - - **不要**用站点首页、topic 页、搜索页、个人空间首页等泛化链接来“刷过”检查 - - 如果没有真实来源，就保留 FAIL，并向用户说明后续需要补哪些材料 - -告知用户时，文件位置必须按当前 family 返回，不要默认写成 colleague。 +1. 先读 `prompts/celebrity/budget_unfriendly/research.md` 和 `references/celebrity_budget_unfriendly_framework.md`。 +2. 按 **6-track 独立文件结构** 写 research notes（不可合并、不可克隆观察）：`01_writings.md` / `02_conversations.md` / `03_expression_dna.md` / `04_decisions.md` / `05_external_views.md` / `06_timeline.md`。 +3. 每条 evidence 必须标注 source weight (1-7)；遵守品味原则 + 信源黑名单 + 信源优先级。 +4. 最低门槛：`Files scanned >= 6`、`Unique URLs >= 8`、`Primary-source markers >= 3`、`Source metadata blocks >= 6`、`Contradiction bullets >= 6`、`Inference bullets >= 6`、`Potential long quote lines = 0`、`Track coverage count = 6`。不满足就补对应 track，不要跳到 review。 +5. 依次生成 `knowledge/research/reviews/research_audit.md`（明确 `PASS/FAIL`，检查信源层级、primary 比例 > 50%、品味原则、冷门人物）→ `synthesis.md`（triple gate：cross-context recurrence / generative power / exclusivity；提取智识谱系种子与 Agentic Protocol 种子）→ 按 `prompts/celebrity/budget_unfriendly/validation.md` 生成 `validation.md`（known-answer ≥2 题 + edge-case 1 题 + voice check 100 字盲测 + copyright check + Agentic Protocol check，明确 `PASS/FAIL`）。 +6. 任何 `FAIL` 都先补材料再继续；不要为了通过检查编造 URL、引用、书名或视频标题。 --- @@ -666,30 +273,13 @@ Persona 摘要： 用户提供新文件或文本时： -1. 按 Step 2 的方式读取新内容 -2. 根据当前 family 解析 base dir -3. 用 `Read` 读取现有 `{resolved_base_dir}/{slug}/work.md` 和 `persona.md` -4. 使用当前 family 对应的 merger prompt 分析增量内容 -5. 存档当前版本（用 Bash）： - ```bash - python3 "{distilly_skill_root}/tools/version_manager.py" \ - --action backup \ - --character {character} \ - --slug {slug} \ - --base-dir {resolved_base_dir} - ``` -6. 把 work/persona 增量分别写到临时 patch 文件 -7. 调用： - ```bash - python3 "{distilly_skill_root}/tools/skill_writer.py" \ - --action update \ - --character {character} \ - --slug {slug} \ - --work-patch /tmp/distilly_{slug}_work_patch.md \ - --persona-patch /tmp/distilly_{slug}_persona_patch.md \ - --base-dir {resolved_base_dir} - ``` -8. 如果当前是 `celebrity`，更新后再次执行 quality check +1. 按 Step 1 的 Collect 流程采集新内容（本地文件走 `distilly harvest`，导出走 `distilly parse-chat`，粘贴走 `distilly note --from -`）。 +2. 跑 `distilly retrospect` 刷新派生，再按 Step 3 复述"读了什么、多少条、多少锚点"。 +3. 根据当前 family 解析 base dir，读取现有 `{resolved_base_dir}/{slug}/work.md` 和 `persona.md`。 +4. 使用当前 family 对应的 merger prompt 分析增量内容。 +5. 用 `distilly skill version` 存档当前版本。 +6. 把 work/persona 增量分别写到临时 patch 文件，再走 `distilly skill update`。 +7. 如果当前是 `celebrity`，更新后用 `distilly doctor` 复核证据覆盖率。 --- @@ -697,76 +287,72 @@ Persona 摘要： 用户表达"不对"/"应该是"时： -1. 参考 `prompts/correction_handler.md` 识别纠正内容 -2. 判断属于 Work（技术/流程）还是 Persona（性格/沟通） -3. 如果属于 Work： - - 生成 `/tmp/distilly_{slug}_work_patch.md` - - patch 必须是可替换的 `##` section，不要直接手改最终文件 - - 调用： - ```bash - python3 "{distilly_skill_root}/tools/skill_writer.py" \ - --action update \ - --character {character} \ - --slug {slug} \ - --work-patch /tmp/distilly_{slug}_work_patch.md \ - --base-dir {resolved_base_dir} - ``` -4. 如果属于 Persona： - - 将 correction 写入 `/tmp/distilly_{slug}_correction.json` - - 单条纠正可直接写成 `{scene, wrong, correct}` - - 多条 persona 纠正可写成 `{"persona_corrections": [{...}, {...}]}` - - 调用： - ```bash - python3 "{distilly_skill_root}/tools/skill_writer.py" \ - --action update \ - --character {character} \ - --slug {slug} \ - --correction-json /tmp/distilly_{slug}_correction.json \ - --base-dir {resolved_base_dir} - ``` -5. 如果当前是 `celebrity`，更新后再次执行 quality check -6. 不要直接手改 `work.md`、`persona.md`、`SKILL.md`、`meta.json`；统一通过 writer 更新 +1. 参考 `prompts/correction_handler.md` 识别纠正内容。 +2. 判断属于 Work（技术/流程）还是 Persona（性格/沟通）。 +3. 如果属于 Work：生成可替换 `##` section 的 patch 临时文件，走 `distilly skill update`，不要直接手改 `work.md`。 +4. 如果属于 Persona：把 correction 写成 `{scene, wrong, correct}`（多条写成 `{"persona_corrections": [...]}`）的临时 JSON，走 `distilly skill update`。 +5. 纠正若与现有结论冲突，先向用户展示冲突再决定；纠正内容本身也要带锚点或标注为"用户口述，无锚点"。 +6. 如果当前是 `celebrity`，更新后用 `distilly doctor` 复核。 --- ## 管理操作 列出三类 Skill： + ```bash -python3 "{distilly_skill_root}/tools/skill_writer.py" --action list --character colleague --base-dir ./skills/colleague -python3 "{distilly_skill_root}/tools/skill_writer.py" --action list --character relationship --base-dir ./skills/relationship -python3 "{distilly_skill_root}/tools/skill_writer.py" --action list --character celebrity --base-dir ./skills/celebrity +distilly skill list ``` 回滚某个 Skill 版本： -```bash -# colleague -python3 "{distilly_skill_root}/tools/version_manager.py" --action rollback --character colleague --slug {slug} --version {version} --base-dir ./skills/colleague - -# relationship -python3 "{distilly_skill_root}/tools/version_manager.py" --action rollback --character relationship --slug {slug} --version {version} --base-dir ./skills/relationship -# celebrity -python3 "{distilly_skill_root}/tools/version_manager.py" --action rollback --character celebrity --slug {slug} --version {version} --base-dir ./skills/celebrity +```bash +distilly skill version ``` -删除某个 Skill： -确认 character 后执行： +删除某个 Skill（确认 character 后）： + ```bash -# colleague rm -rf skills/colleague/{slug} - -# relationship rm -rf skills/relationship/{slug} - -# celebrity rm -rf skills/celebrity/{slug} ``` +宿主安装：`distilly install `；卸载：`distilly uninstall`。 + +列出与撤销已授予的同意： + +```bash +distilly consent list +distilly consent revoke +``` + +--- + +## 必须 + +- 先列"读了哪些文件、各多少条、多少锚点"，再写结论；每条结论带 `文件 + 锚点`。 +- 无证据写 `unknown`；候选不当结论。 +- 每一步都按"五步主线"的完成判据检查产物，再进入下一步。 + +## 禁止 + +- 不改写引文，不伪造 URL、锚点、书名、视频标题，不用平台首页刷来源。 +- 密钥只从 `~/.distilly/*_config.json` 或环境变量读，永不写进对话、文件、回执或日志。 +- 不自己拼平台 API 请求：所有网络采集都通过 `distilly collect` / `distilly harvest` / `distilly parse-chat` / `distilly parse-email` / `distilly parse-subtitle` / `distilly parse-doc` / `distilly parse-archive` / `distilly transcribe`。 +- 不静默降级：失败、不可用渠道、没跑的步骤都要说清楚。 + +## 回执 + +- 读过哪些文件、各多少条、多少锚点。 +- 生成/更新了哪些文件，各自 sha256（来自 `distilly` 的 `--json` 回执、`knowledge/index.json` 或 `evidence/renders/receipt.json`）。 +- 哪些渠道不可用（`unavailable[]`）。 +- 哪些步骤没跑、为什么。 + --- --- -# English Version +## English # Distilly Creator @@ -803,716 +389,299 @@ Enter evolution mode when the user says: Compatibility update alias: - `/update-colleague {slug}` -When the user asks to see generated skills, use the list commands in "Management Operations" below. +When the user asks to see generated skills, use the list command in "Management Operations" below. --- -## Tool Usage Rules +## Command Contract (single entrypoint) -This Skill runs in any compatible host that can read local files and execute Bash / Python commands. Use the following tool conventions: +Every collection, derivation, and render step goes through `distilly`. Command names match the command table in `docs/v2/CONTRACT.md` §1 word for word; do not invent subcommands or fields. -| Task | Tool | -|------|------| -| Read PDF documents | `Read` tool (native PDF support) | -| Read image screenshots | `Read` tool (native image support) | -| Read MD/TXT files | `Read` tool | -| Parse Lark message JSON export | `Bash` → `python3 "{distilly_skill_root}/tools/feishu_parser.py"` | -| Lark auto-collect (recommended) | `Bash` → `python3 "{distilly_skill_root}/tools/feishu_auto_collector.py"` | -| Lark docs (browser session) | `Bash` → `python3 "{distilly_skill_root}/tools/feishu_browser.py"` | -| Lark docs (MCP App Token) | `Bash` → `python3 "{distilly_skill_root}/tools/feishu_mcp_client.py"` | -| DingTalk auto-collect | `Bash` → `python3 "{distilly_skill_root}/tools/dingtalk_auto_collector.py"` | -| Collect public X post candidates | `Bash` → `python3 "{distilly_skill_root}/tools/research/xquik_public_posts.py"` | -| Parse email .eml/.mbox | `Bash` → `python3 "{distilly_skill_root}/tools/email_parser.py"` | -| Write/update Skill files | `Write` / `Edit` tool | -| Version management | `Bash` → `python3 "{distilly_skill_root}/tools/version_manager.py"` | -| List existing Skills | `Bash` → `python3 "{distilly_skill_root}/tools/skill_writer.py" --action list` | - -**Base directories**: -- `colleague` → `./skills/colleague/{slug}/` -- `relationship` → `./skills/relationship/{slug}/` -- `celebrity` → `./skills/celebrity/{slug}/` - -For a global path, use `--base-dir` with the storage root for that character family. - -The Lark-labelled compatibility collectors currently connect to the China-region `open.feishu.cn` / `feishu.cn` endpoints. International `larksuite.com` tenant routing is not implemented yet. +| Task | Command | +|------|---------| +| Zero-credential: directory/file → `knowledge/` | `distilly harvest ` | +| Parse ChatGPT / Claude / Slack / Telegram / Discord exports | `distilly parse-chat ` | +| Parse email | `distilly parse-email ` | +| Parse subtitles | `distilly parse-subtitle ` | +| Parse documents | `distilly parse-doc ` | +| Parse archives (X archive / Takeout / social exports) | `distilly parse-archive ` | +| Pure derivation → `evidence/derived/*.json` | `distilly retrospect` | +| Collection on channels needing key / OAuth | `distilly collect ` | +| Browser computer use (consent token required) | `distilly collect x --mode browser --consent ` | +| Audio/video transcription (optional backend) | `distilly transcribe ` | +| Register what the model itself read | `distilly note --from ` | +| Consent management | `distilly consent ` | +| View check and render | `distilly view check`; `distilly view render [--shareable]` | +| Evidence health check | `distilly doctor` | +| Create/update/list/version a generated Skill | `distilly skill ` | +| Host install | `distilly install `; `distilly uninstall` | + +- Every subcommand supports a `--json` receipt; `--help` has a Chinese and an English section. +- For channels needing a key or OAuth: first tell the user which channel will be read and what it yields, and only run `distilly collect` after they agree. +- Computer-use commands require `--consent `; without a token the command ends with `exit 2` and its receipt says it is waiting for user consent. +- Credentials are read only from `~/.distilly/*_config.json` or environment variables; receipts, logs, and chat only ever contain the config file name, never a value. + +### Migration table (all legacy forms are deprecated) + +| Legacy form (deprecated) | New form | +|--------------------------|----------| +| `tools/feishu_parser.py` (deprecated) | `distilly parse-chat` | +| `tools/feishu_auto_collector.py` (deprecated) | `distilly collect feishu` | +| `tools/feishu_browser.py` (deprecated) | `distilly collect feishu --mode browser --consent ` | +| `tools/feishu_mcp_client.py` (deprecated) | `distilly collect feishu` | +| `tools/dingtalk_auto_collector.py` (deprecated) | `distilly collect dingtalk` | +| `tools/email_parser.py` (deprecated) | `distilly parse-email` | +| `tools/research/xquik_public_posts.py` (deprecated) | `distilly collect x` | +| `tools/research/transcribe_audio.py` (deprecated) | `distilly transcribe` | +| `tools/research/srt_to_transcript.py` (deprecated) | `distilly parse-subtitle` | +| `tools/skill_writer.py` (deprecated) | `distilly skill create` / `distilly skill update` / `distilly skill list` | +| `tools/version_manager.py` (deprecated) | `distilly skill version` | +| `tools/install_generated_skill.py` (deprecated) | `distilly install ` | +| `tools/research/quality_check.py` (deprecated) | `distilly doctor` | +| `tools/research/merge_research.py` (deprecated) | No contract replacement yet: remains derivation only, use `distilly retrospect`; merging research notes is a known gap | +| `tools/research/download_subtitles.sh` (deprecated) | No contract replacement yet: ask the user for a local subtitle file, then use `distilly parse-subtitle` | + +Both forms may coexist during migration, but the new form wins; whenever a Python tool is still referenced, keep the `deprecated` marker. --- -## Main Flow: Create a New Skill +## Disk Contract (what the model may write) -### Step 0: Confirm the character family +``` +skills/// + SKILL.md work.md persona.md work_skill.md persona_skill.md manifest.json meta.json + knowledge/{docs,messages,emails}/ + knowledge/raw//... # raw bytes, append-only + knowledge/text/.md # normalized text, paragraph anchors [k0012] / [k0012:t3] + knowledge/index.json # ledger {id,kind,origin,fetched_at,bytes,sha256,credentialed,method,warnings[]} + evidence/derived/*.json # retrospect output, every conclusion carries evidence anchors + views/.view.json # the model writes only sections/order/emphasis (no facts) + views/.html # render output: single file, offline, dual theme + evidence/renders/receipt.json # render receipt (sha256 + bytes + inlined sources) +``` -If the user entered `/distilly`, first confirm which family should be distilled: +- The model may write: `views/.view.json` (sections, order, emphasis only), temporary working files, and sources registered through `distilly note --from ` with `method:"model-read"`. +- The model must not write: `knowledge/raw/**` (raw bytes, append-only), `knowledge/index.json`, `evidence/derived/*.json` (produced by `distilly retrospect`), `evidence/renders/receipt.json`. +- Screenshots, receipts, and diff images are never committed (`.gitignore` already covers `dst-evidence/`); local artifacts live in `/tmp/dst-evidence//`. +- Anchor format is always `[k00NN]` (4-digit zero-padded) or `[k00NN:tM]` (with turn index). Every conclusion carries `file + anchor`; with no evidence, write `unknown`. -1. `colleague` -2. `relationship` -3. `celebrity` +--- -If the host already passed an explicit family, lock the character family immediately. +## Five-Step Mainline -If the current family is `celebrity`, also confirm the research profile: +Creation, append, and correction all follow one mainline: **Collect → Derive → Read → Distill → Render**. -1. `budget-friendly` -2. `budget-unfriendly` +| Step | Required artifacts | Count criteria | Where sha256 comes from | What to do on failure | +|------|--------------------|----------------|-------------------------|-----------------------| +| 1 Collect | `knowledge/raw//**`, `knowledge/text/.md`, `knowledge/index.json` | 1 ledger entry per grounded source; ≥1 anchor in every text file | `outputs[].sha256` of the `distilly --json` receipt, byte-identical to `sha256` in `knowledge/index.json` | Non-zero exit: record command, stderr, remedy; with 0 grounded sources stop and do not enter Derive | +| 2 Derive | `evidence/derived/*.json` | Every derived conclusion carries evidence anchors; two runs are byte-identical | Receipt `outputs[].sha256`; identical sha256 across two runs | Non-zero exit: first repair `knowledge/index.json` integrity; never hand-write derived JSON | +| 3 Read | No new files; produce a restatement of what was read | Per file: file → rows → anchors | Quote `sha256` from `knowledge/index.json`; never compute your own | Missing files or zero anchors: go back to Step 1; never write conclusions from memory | +| 4 Distill | `work.md`, `persona.md`, plus celebrity research/audit/synthesis/validation | Every dimension has anchors or `unknown`; celebrity has an explicit `PASS/FAIL` | Quote the sha256 of cited sources from the ledger | Thin evidence: mark `(insufficient source material)` / candidate and say what material is missing | +| 5 Render | `views/.view.json`, `views/.html`, `evidence/renders/receipt.json` | Receipt sha256 matches the actual html sha256; `distilly doctor` reports the anchor back-reference rate | sha256 in `evidence/renders/receipt.json` | Render failure: keep view.json, do not publish, report the error | -Default to `budget-friendly`. Only switch to `budget-unfriendly` when the user explicitly wants deeper research, higher confidence, or accepts a slower and more expensive distillation pass. +No step may degrade silently: either fix it, or state the failure in the user-facing report and in the receipt's `warnings[]` / `unavailable[]`. -### Step 1: Basic Info Collection +### Step 1: Collect -Choose the intake prompt by character family: +1. Read `prompts/collectors.md` first and pick the route from its "which command when" table. +2. Zero-credential sources (local files, export bundles, subtitles, documents, archives) go straight to `distilly harvest`, `distilly parse-chat`, `distilly parse-email`, `distilly parse-subtitle`, `distilly parse-doc`, `distilly parse-archive`. +3. Channels needing a key or OAuth (Feishu, Slack, DingTalk, X, Discord, Reddit, Notion, Gmail) require the user's consent first, then `distilly collect `; manage consent scope with `distilly consent `. +4. Browser computer use must follow `prompts/computer-use.md`: ask before acting, read-only whitelist, default ≤20 screens / ≤10 minutes / ≤6 scrolls per minute, every screen persisted with raw text + URL + timestamp + screenshot (screenshots stay local), interruptible; `distilly collect x --mode browser --consent ` must exit without a token — never work around it. +5. When the user can only paste text or screenshots, register the source with `distilly note --from ` (`method:"model-read"`); never pretend it was collected. +6. Transcribe audio/video with `distilly transcribe` before parsing subtitles; never commit a full transcript to the repository. -- `colleague` → `prompts/intake.md` -- `relationship` → `prompts/relationship/intake.md` -- `celebrity` → `prompts/celebrity/intake.md` +**Completion criteria**: `knowledge/index.json` has one ledger entry per grounded source (with `id`, `kind`, `origin`, `fetched_at`, `bytes`, `sha256`, `credentialed`, `method`, `warnings[]`); every `knowledge/text/.md` has at least one anchor; receipt `inputs[]`/`outputs[]` sha256 matches the ledger; unavailable channels appear in `unavailable[]`. +**On failure**: when a command exits non-zero, report the exact command, its stderr, and the remedy (for example which `~/.distilly/*_config.json` must be configured), then stop and wait for instructions; if 0 sources landed, do not enter Step 2. -For `colleague` and `relationship`, ask only 3 questions. -For `celebrity`, use the 4-question intake in `prompts/celebrity/intake.md`; the fourth question must confirm `research_profile`. +### Step 2: Derive -The default 3 base questions are: +1. Do not read `evidence/derived/*` before deriving — run `distilly retrospect` first. +2. `distilly retrospect` is pure derivation: input is `knowledge/**`, output is `evidence/derived/*.json`, and every conclusion carries evidence anchors. +3. To prove determinism, run it twice; the same input must produce identical sha256. -1. **Alias / Codename** (required) -2. **Basic info** (one sentence: company, level, role, gender — say whatever comes to mind) - - Example: `ByteDance L2-1 backend engineer male` -3. **Personality profile** (one sentence: MBTI, zodiac, traits, corporate culture, impressions) - - Example: `INTJ Capricorn blame-shifter ByteDance-style strict in CR but never explains why` +**Completion criteria**: `evidence/derived/*.json` exists; the receipt reports `anchors.total` / `anchors.cited`; `outputs[].sha256` is identical across two runs. +**On failure**: a non-zero exit means the input side is broken — go back to Step 1 and check the ledger and text anchors; never hand-write or hand-edit derived JSON to force a pass. -Everything except the alias can be skipped. Summarize and confirm before moving to the next step. +### Step 3: Read -### Step 2: Source Material Import +1. Read in this order: `knowledge/index.json` → `knowledge/text/*.md` → `evidence/derived/*.json`. +2. Restate to the user "which files were read, how many rows each, how many anchors" before writing conclusions. +3. Every conclusion carries `file + anchor` (for example `knowledge/text/feishu.md [k0042]`). +4. Conclusions without evidence are written as `unknown`, together with what material would supply the evidence. +5. Keep facts and candidates apart: only statements backed by a specific anchor are facts; patterns, tendencies, and inferences from derived files stay candidates and never get promoted to conclusions. +6. Full detail rules are in `prompts/retrospection.md`. -Ask the user how they'd like to provide materials: +**Completion criteria**: every file in the restatement list back-references into the ledger; every cited anchor really exists in `knowledge/text/**`; no conclusion is left without an anchor. +**On failure**: with missing files or zero anchors, go back to Step 1; never fill the gap from memory or general knowledge. -``` -How would you like to provide source materials? +### Step 4: Distill - [A] Lark Auto-Collect (recommended) - Enter name, auto-pull messages + docs + spreadsheets +Resolve the execution matrix for the family confirmed in Step 0: - [B] DingTalk Auto-Collect - Enter name, auto-pull docs + spreadsheets - Messages collected via browser (DingTalk API doesn't support message history) +| character | intake | persona analyzer | persona builder | merger | storage root | +|-----------|--------|------------------|-----------------|--------|--------------| +| `colleague` | `prompts/intake.md` | `prompts/persona_analyzer.md` | `prompts/persona_builder.md` | `prompts/merger.md` | `./skills/colleague/{slug}` | +| `relationship` | `prompts/relationship/intake.md` | `prompts/relationship/persona_analyzer.md` | `prompts/relationship/persona_builder.md` | `prompts/relationship/merger.md` | `./skills/relationship/{slug}` | +| `celebrity` | `prompts/celebrity/intake.md` | `prompts/celebrity/persona_analyzer.md` | `prompts/celebrity/persona_builder.md` | `prompts/celebrity/merger.md` | `./skills/celebrity/{slug}` | - [C] Lark Link - Provide doc/Wiki link (browser session or MCP) +Shared across all families: Work analyzer `prompts/work_analyzer.md`, Work builder `prompts/work_builder.md`, Correction handler `prompts/correction_handler.md`. - [D] Upload Files - PDF / images / exported JSON / email .eml +Two tracks: - [E] Paste Text - Copy-paste text directly +- **Track A (Work Skill)**: follow `prompts/work_analyzer.md` and extract responsible systems, technical standards, workflow, output preferences, experience. For `celebrity`, interpret `work` as methods, judgment frameworks, and decision patterns. +- **Track B (Persona)**: use the family-specific persona analyzer; for `celebrity` with `research_profile=budget-unfriendly`, switch to `prompts/celebrity/budget_unfriendly/persona_analyzer.md`. Translate the user's tags into concrete behavior rules and extract communication style, decision patterns, and interpersonal behavior from the material. -Can mix and match, or skip entirely (generate from manual info only). -``` +Never hand-build a `skills/{family}/{slug}` tree: write `meta.json` / `work.md` / `persona.md` to temporary files and call `distilly skill create` (or `distilly skill update`). Install a generated person Skill with `distilly install `. ---- +**Completion criteria**: every dimension has anchors or an explicit `(insufficient source material)`; every behavior rule is concrete and executable; celebrity audit / validation returns an explicit `PASS` or `FAIL`; `distilly doctor` can report evidence coverage, unavailable channels, and the anchor back-reference rate. Celebrity research thresholds are in the subflow below. +**On failure**: mark thin dimensions `(insufficient source material, add related documents)` and downgrade them to candidates; when `source_grounding` fails, keep the `FAIL` and explain what is missing instead of padding with generic links. -#### Option A: Lark Auto-Collect (Recommended) +### Step 5: Render -First-time setup: -```bash -python3 "{distilly_skill_root}/tools/feishu_auto_collector.py" --setup -``` +1. Run `distilly view check` first to confirm every anchor back-references to `knowledge/index.json`. +2. Write `views/.view.json`: sections, order, and emphasis only — no facts. +3. `distilly view render` produces the single-file, offline, dual-theme `views/.html` and writes `evidence/renders/receipt.json` (sha256 + bytes + inlined sources). +4. Use `distilly view render --shareable` only for external sharing, and confirm with the user first. +5. Re-check evidence coverage, unavailable channels, anchor back-reference rate, and computer-use share with `distilly doctor`. -**Group chat collection** (uses tenant_access_token, bot must be in the group): -```bash -python3 "{distilly_skill_root}/tools/feishu_auto_collector.py" \ - --name "{name}" \ - --output-dir ./knowledge/{slug} \ - --msg-limit 1000 \ - --doc-limit 20 -``` +**Completion criteria**: both `views/.html` and `evidence/renders/receipt.json` exist; the receipt sha256 matches the actual html sha256; zero broken internal links. +**On failure**: when rendering fails, keep `views/.view.json`, do not publish the HTML, and report the error and the missing sources to the user. -**Private chat (P2P) collection** (requires user_access_token + p2p chat_id): - -Private messages can only be accessed via user identity (user_access_token). App identity cannot access private chats. - -**Prerequisites**: - -The user needs to provide: -1. **Lark app credentials**: `app_id` and `app_secret` (from the Open Platform) -2. **User scopes**: The app must have these user scopes enabled: - - `im:message` — read/send messages as user - - `im:chat` — read chat list as user -3. **OAuth authorization code**: obtained after user completes OAuth in browser - -If the user is missing any of these, guide them through setup. Don't assume anything is pre-configured. - -**Getting user_access_token**: - -Once the user provides app_id, app_secret, and confirms scopes are enabled: - -1. Generate the OAuth URL for them: - ``` - https://open.feishu.cn/open-apis/authen/v1/authorize?app_id={APP_ID}&redirect_uri=http://www.example.com&scope=im:message%20im:chat - ``` - > ⚠️ The redirect_uri must be added in the app's "Security Settings → Redirect URLs" - -2. User opens URL, logs in, authorizes -3. Page redirects to `http://www.example.com?code=xxx`, user copies the code -4. Exchange code for token: - ```bash - python3 "{distilly_skill_root}/tools/feishu_auto_collector.py" --exchange-code {CODE} - ``` - Or write a Python script to call the same API directly: - ```python - # 1. Get app_access_token - POST https://open.feishu.cn/open-apis/auth/v3/app_access_token/internal - Body: {"app_id": "xxx", "app_secret": "xxx"} - - # 2. Exchange code for user_access_token - POST https://open.feishu.cn/open-apis/authen/v1/oidc/access_token - Header: Authorization: Bearer {app_access_token} - Body: {"grant_type": "authorization_code", "code": "xxx"} - ``` - -**Getting the p2p chat_id**: - -Users typically don't know their chat_id. When the user has a user_access_token but no chat_id, **write a Python script yourself** to obtain it: - -- **Method**: Send a message to the other user's open_id — the response includes the chat_id - ```python - POST https://open.feishu.cn/open-apis/im/v1/messages?receive_id_type=open_id - Header: Authorization: Bearer {user_access_token} - Body: {"receive_id": "{target_open_id}", "msg_type": "text", "content": "{\"text\":\"hello\"}"} - # The chat_id in the response is the p2p chat ID - ``` -- **Important**: `GET /im/v1/chats` does NOT return p2p chats — this is an API limitation, not a permission issue. Do not try to use it for finding private chats. -- If the user doesn't know the target's open_id, use tenant_access_token to search contacts: - ```python - GET https://open.feishu.cn/open-apis/contact/v3/scopes - # Returns open_ids of all users visible to the app - ``` - -**Running collection**: - -Once you have user_access_token and chat_id: -```bash -python3 "{distilly_skill_root}/tools/feishu_auto_collector.py" \ - --open-id {target_open_id} \ - --p2p-chat-id {chat_id} \ - --user-token {user_access_token} \ - --name "{name}" \ - --output-dir ./knowledge/{slug} \ - --msg-limit 1000 -``` +### Step 0 (prerequisite): Confirm the family and run intake -**Flexibility principle**: The above API calls don't have to go through the collector script. If the script doesn't work or doesn't fit the scenario, write Python scripts directly against the same endpoints. Key API reference: -- Get token: `POST /auth/v3/app_access_token/internal`, `POST /authen/v1/oidc/access_token` -- Send message (get chat_id): `POST /im/v1/messages?receive_id_type=open_id` -- Fetch messages: `GET /im/v1/messages?container_id_type=chat&container_id={chat_id}` -- Search contacts: `GET /contact/v3/scopes`, `GET /contact/v3/users/{user_id}` - -Auto-collected content: -- Group chats: messages sent by them (system messages and stickers filtered) -- Private chats: full conversation with both parties (for context understanding) -- Lark docs and Wikis they created/edited -- Related spreadsheets (if accessible) - -After collection, `Read` the output files: -- `knowledge/{slug}/messages.txt` → messages (group + private) -- `knowledge/{slug}/docs.txt` → document content -- `knowledge/{slug}/collection_summary.json` → collection summary - -If collection fails, diagnose the error and attempt to fix it. Common issues: -- Group chat: bot not added to the group -- Private chat: user_access_token expired (2-hour TTL, refresh with refresh_token) -- Insufficient permissions: guide user to enable scopes and re-authorize -- Or switch to Option B/C +If the user entered `/distilly`, first confirm which family should be distilled: ---- +1. `colleague` +2. `relationship` +3. `celebrity` -#### Option B: DingTalk Auto-Collect +If the host already passed an explicit family, lock the character family immediately. -First-time setup: -```bash -python3 "{distilly_skill_root}/tools/dingtalk_auto_collector.py" --setup -``` +If the current family is `celebrity`, also confirm the research profile: -Then enter the name: -```bash -python3 "{distilly_skill_root}/tools/dingtalk_auto_collector.py" \ - --name "{name}" \ - --output-dir ./knowledge/{slug} \ - --msg-limit 500 \ - --doc-limit 20 \ - --show-browser # add this flag on first use to complete DingTalk login -``` +1. `budget-friendly` +2. `budget-unfriendly` -Collected content: -- DingTalk docs and knowledge bases they created/edited -- Spreadsheets -- Messages (⚠️ DingTalk API doesn't support message history — auto-switches to browser scraping) +Default to `budget-friendly`. Only switch to `budget-unfriendly` when the user explicitly wants deeper research, higher confidence, or accepts a slower and more expensive distillation pass. -After collection, `Read`: -- `knowledge/{slug}/docs.txt` -- `knowledge/{slug}/bitables.txt` -- `knowledge/{slug}/messages.txt` +Choose the intake prompt by family: `colleague` → `prompts/intake.md`; `relationship` → `prompts/relationship/intake.md`; `celebrity` → `prompts/celebrity/intake.md`. `colleague` and `relationship` ask only 3 questions; `celebrity` asks 4, and the fourth must confirm `research_profile`. -If message collection fails, prompt user to upload chat screenshots. +The default 3 base questions: ---- +1. **Alias / Codename** (required) +2. **Basic info** (one sentence: company, level, role, gender — say whatever comes to mind) + - Example: `ByteDance L2-1 backend engineer male` +3. **Personality profile** (one sentence: MBTI, zodiac, traits, corporate culture, impressions) + - Example: `INTJ Capricorn blame-shifter ByteDance-style strict in CR but never explains why` -#### Option D: Upload Files - -- **PDF / Images**: `Read` tool directly -- **Lark message JSON export**: - ```bash - python3 "{distilly_skill_root}/tools/feishu_parser.py" --file {path} --target "{name}" --output /tmp/feishu_out.txt - ``` - Then `Read /tmp/feishu_out.txt` -- **Email files .eml / .mbox**: - ```bash - python3 "{distilly_skill_root}/tools/email_parser.py" --file {path} --target "{name}" --output /tmp/email_out.txt - ``` - Then `Read /tmp/email_out.txt` -- **Markdown / TXT**: `Read` tool directly +Everything except the alias can be skipped. Summarize and confirm before entering Collect. --- -#### Option C: Lark Link +## Celebrity research subflow (between Step 2 and Step 3) -When the user provides a Lark doc/Wiki link, ask which method to use: +### budget-friendly -``` -Lark link detected. Choose read method: - - [1] Browser Method (recommended) - Reuses your local Chrome login session - ✅ Works with internal docs requiring permissions - ✅ No token configuration needed - ⚠️ Requires Chrome + playwright installed locally - - [2] MCP Method - Uses a Lark App Token via the official API - ✅ Stable, no browser dependency - ✅ Can read messages (needs chat ID) - ⚠️ Requires App ID / App Secret setup - ⚠️ Internal docs need admin authorization for the app - -Choose [1/2]: -``` - -**Option 1 (Browser)**: -```bash -python3 "{distilly_skill_root}/tools/feishu_browser.py" \ - --url "{feishu_url}" \ - --target "{name}" \ - --output /tmp/feishu_doc_out.txt -``` -First use will open a browser window for login (one-time). - -**Option 2 (MCP)**: - -First-time setup: -```bash -python3 "{distilly_skill_root}/tools/feishu_mcp_client.py" --setup -``` +1. Read `prompts/celebrity/research.md` and follow its **6-dimension parallel collection strategy**. +2. Collection strategy (fixed during intake): **Local-first** (analyze local material first, search only the gaps) / **Web + local** (full 6-dimension web research cross-validated with local material) / **Web-only**. +3. For video or podcasts: `distilly transcribe ` first, then `distilly parse-subtitle`; never commit a full transcript. +4. Split the raw research notes across **at least 3 files** (2 dimensions each), never one monolithic `research_notes.md`: + - `knowledge/research/raw/01_core_profile.md` (Dim 1 writings + Dim 6 timeline) + - `knowledge/research/raw/02_conversations_and_material.md` (Dim 2 conversations + Dim 4 decisions) + - `knowledge/research/raw/03_expression_and_reception.md` (Dim 3 expression DNA + Dim 5 external views) +5. Taste principles: long-form > snippets, controversy > consensus, change > fixity, firsthand > secondhand. Source blacklist: never cite Zhihu, WeChat official accounts, Baidu Baike, content farms. Source hierarchy: user local material > first-person works > long interviews > decision records > social media > external analysis > secondhand summaries. +6. Confirm `Files scanned >= 3`, `Unique URLs >= 2`, `Potential long quote lines = 0`; every URL must be a specific page actually opened, not a platform root, search page, topic page, or placeholder. +7. **Quality checkpoint (Phase 1.5)**: show the user a structured collection summary (sources per dimension + key findings + contradictions + thin dimensions + cold-figure verdict) and wait for confirmation. +8. **Cold figure detection**: below 10 total sources, limit mental models to 2–3, mark thin models "based on limited information", expand the honest boundaries section, and tell the user what material would improve quality. +9. Analysis input priority: primary material (source weight 1-3) > merged research summary > explicit user notes. -Then read directly: -```bash -python3 "{distilly_skill_root}/tools/feishu_mcp_client.py" \ - --url "{feishu_url}" \ - --output /tmp/feishu_doc_out.txt -``` +### budget-unfriendly -Read messages (needs chat ID, format `oc_xxx`): -```bash -python3 "{distilly_skill_root}/tools/feishu_mcp_client.py" \ - --chat-id "oc_xxx" \ - --target "{name}" \ - --limit 500 \ - --output /tmp/feishu_msg_out.txt -``` - -Both methods output to files, then use `Read` to load results into analysis. +1. Read `prompts/celebrity/budget_unfriendly/research.md` and `references/celebrity_budget_unfriendly_framework.md` first. +2. Write the **six-track research set** as independent files (never merged, never cloned): `01_writings.md` / `02_conversations.md` / `03_expression_dna.md` / `04_decisions.md` / `05_external_views.md` / `06_timeline.md`. +3. Every evidence item carries a source weight (1-7); follow taste principles + source blacklist + source hierarchy. +4. Minimum floor: `Files scanned >= 6`, `Unique URLs >= 8`, `Primary-source markers >= 3`, `Source metadata blocks >= 6`, `Contradiction bullets >= 6`, `Inference bullets >= 6`, `Potential long quote lines = 0`, `Track coverage count = 6`. If short, fill the weak track instead of skipping to review. +5. Write, in order: `knowledge/research/reviews/research_audit.md` (explicit `PASS/FAIL`; checks source hierarchy, primary ratio > 50%, taste principles, cold figure) → `synthesis.md` (triple gate: cross-context recurrence / generative power / exclusivity; extract intellectual genealogy and Agentic Protocol seeds) → `validation.md` per `prompts/celebrity/budget_unfriendly/validation.md` (known-answer ≥2 questions + 1 edge case + 100-word voice check + copyright check + Agentic Protocol check, explicit `PASS/FAIL`). +6. Any `FAIL` means backfill first; never invent URLs, quotes, book titles, or video titles to pass a check. --- -#### Option E: Paste Text +## Evolution Mode: Append Files + +When the user provides new files or text: -User-pasted content is used directly as text material. No tools needed. +1. Collect the new material with the Step 1 flow (`distilly harvest` for local files, `distilly parse-chat` for exports, `distilly note --from -` for pasted text). +2. Run `distilly retrospect` to refresh derivations, then restate "what was read, how many rows, how many anchors" per Step 3. +3. Resolve the base dir for the current family and read the existing `{resolved_base_dir}/{slug}/work.md` and `persona.md`. +4. Analyze the delta with the family-specific merger prompt. +5. Archive the current version with `distilly skill version`. +6. Write the work/persona deltas to temporary patch files and apply them with `distilly skill update`. +7. For `celebrity`, re-check evidence coverage with `distilly doctor` after the update. --- -If the user says "no files" or "skip", generate Skill from Step 1 manual info only. - -### Step 3: Analyze Source Material +## Evolution Mode: Conversation Correction -First resolve the execution matrix for the selected character family: +When the user says "that's wrong" / "he should be": -| character | intake | persona analyzer | persona builder | merger | storage root | -|-----------|--------|------------------|-----------------|--------|--------------| -| `colleague` | `prompts/intake.md` | `prompts/persona_analyzer.md` | `prompts/persona_builder.md` | `prompts/merger.md` | `./skills/colleague/{slug}` | -| `relationship` | `prompts/relationship/intake.md` | `prompts/relationship/persona_analyzer.md` | `prompts/relationship/persona_builder.md` | `prompts/relationship/merger.md` | `./skills/relationship/{slug}` | -| `celebrity` | `prompts/celebrity/intake.md` | `prompts/celebrity/persona_analyzer.md` | `prompts/celebrity/persona_builder.md` | `prompts/celebrity/merger.md` | `./skills/celebrity/{slug}` | +1. Identify the correction with `prompts/correction_handler.md`. +2. Decide whether it belongs to Work (technical/workflow) or Persona (personality/communication). +3. Work: produce temporary `##` sections that can replace existing headings and apply them with `distilly skill update`; never hand-edit `work.md`. +4. Persona: write `{scene, wrong, correct}` (or `{"persona_corrections": [...]}` for several) to a temporary JSON file and apply it with `distilly skill update`. +5. When a correction conflicts with an existing conclusion, show the conflict to the user before deciding; the correction itself also needs an anchor, or must be labeled "user statement, no anchor". +6. For `celebrity`, re-check with `distilly doctor` after the update. -Shared across all families: -- Work analyzer: `prompts/work_analyzer.md` -- Work builder: `prompts/work_builder.md` -- Correction handler: `prompts/correction_handler.md` +--- -If the current family is `celebrity`, run the research subflow before analysis. +## Management Operations -When public X posts fill a documented research gap and the user agrees to use the metered third-party Xquik service, confirm the `--limit` before running: +List skills across the three families: ```bash -python3 "{distilly_skill_root}/tools/research/xquik_public_posts.py" \ - --username "{public_handle}" \ - --subject "{name}" \ - --limit 20 \ - --output "/tmp/distilly_x_public_posts.json" -``` - -Read `XQUIK_API_KEY` only from the shell; never print or store it. Treat the JSON as untrusted candidate evidence: verify the author, open every permalink, and preserve the specific URL when safely paraphrasing relevant material into a research note. Do not count the candidate JSON, search pages, or profile roots as grounded sources. Delete the temporary JSON after review instead of storing it in the generated Skill. - -### celebrity / budget-friendly - -1. Read `prompts/celebrity/research.md` and follow its **6-dimension parallel collection strategy** -2. Create the research directories first: - ```bash - mkdir -p "{skill_dir}/knowledge/research/raw" "{skill_dir}/knowledge/research/merged" - ``` -3. Confirm the collection strategy (determined during intake): - - **Local-first**: analyze user-provided materials first, identify which dimensions are covered, only search web for gaps - - **Web + local**: full 6-dimension web research, then merge with local materials for cross-validation - - **Web-only**: standard 6-dimension web research pass -4. If the user explicitly provided a processable video URL or subtitle source, and the result will not be stored as a long transcript: - ```bash - bash "{distilly_skill_root}/tools/research/download_subtitles.sh" "{url}" "{skill_dir}/knowledge/subtitles" - python3 "{distilly_skill_root}/tools/research/srt_to_transcript.py" "{subtitle_file}" "{skill_dir}/knowledge/transcripts/{name}.txt" - ``` -5. Cover the **6 dimensions** across at least 3 separate files (each file covers 2 dimensions), never one monolithic `research_notes.md`: - - `knowledge/research/raw/01_core_profile.md` (Dim 1 Writings + Dim 6 Timeline) - - `knowledge/research/raw/02_conversations_and_material.md` (Dim 2 Conversations + Dim 4 Decisions) - - `knowledge/research/raw/03_expression_and_reception.md` (Dim 3 Expression DNA + Dim 5 External Views) -6. Research must follow **taste principles** (see research prompt): - - Long-form > snippets, controversy > consensus, change > fixity, firsthand > secondhand - - **Source blacklist** — never cite: Zhihu, WeChat official accounts, Baidu Baike, content farms, AI-generated bios - - **Source hierarchy**: user local materials > first-person works > long interviews > decision records > short-form firsthand > external analysis > secondhand summaries -7. Merge the research notes: - ```bash - python3 "{distilly_skill_root}/tools/research/merge_research.py" "{skill_dir}" - ``` - Output: `knowledge/research/merged/summary.md` -8. Read `knowledge/research/merged/summary.md` and confirm: - - `Files scanned >= 3` - - `Unique URLs >= 2` - - `Potential long quote lines = 0` - - URLs in notes are actual inspected pages, not platform roots, search/topic pages, or placeholder paths - If these do not hold, extend the research notes before continuing or explicitly record the collection limits. -9. **Quality checkpoint (Phase 1.5)**: before entering analysis, show the user a structured collection summary: - ``` - ┌──────────────────────────────┬──────────┬─────────────────────────────┐ - │ Dimension │ Sources │ Key Finding │ - ├──────────────────────────────┼──────────┼─────────────────────────────┤ - │ 1 Writings │ N │ [core thesis / gap] │ - │ 2 Conversations │ N │ [key pattern / gap] │ - │ 3 Expression DNA │ N │ [style marker / gap] │ - │ 4 Decisions │ N │ [decision pattern / gap] │ - │ 5 External Views │ N │ [outside view / gap] │ - │ 6 Timeline │ N │ [trajectory / gap] │ - ├──────────────────────────────┼──────────┼─────────────────────────────┤ - │ Contradictions │ N │ [summary] │ - │ Thin dimensions │ [list] │ Backfill plan: [plan] │ - │ Cold figure? │ yes/no │ │ - └──────────────────────────────┴──────────┴─────────────────────────────┘ - ``` - Wait for user confirmation before continuing. If the user flags issues or wants more depth, extend research first. -10. **Cold figure detection**: if total sources < 10, apply the cold figure protocol: - - Limit mental models to 2–3 - - Mark thin models as "based on limited information" - - Expand the honest boundaries section - - Tell the user what additional material would improve quality -11. Celebrity analysis must prioritize: - - primary materials (source weight 1-3) - - merged research summary - - explicit user notes - -### celebrity / budget-unfriendly - -1. First read: - - `prompts/celebrity/budget_unfriendly/research.md` - - `references/celebrity_budget_unfriendly_framework.md` -2. Create the research directories first: - ```bash - mkdir -p "{skill_dir}/knowledge/research/raw" "{skill_dir}/knowledge/research/merged" "{skill_dir}/knowledge/research/reviews" - ``` -3. Confirm the collection strategy (determined during intake): local-first / web+local / web-only -4. Build the **six-track research set** as independent files (never merged, never clone observations): - - `knowledge/research/raw/01_writings.md` (Dim 1: Writings / systematic thought) - - `knowledge/research/raw/02_conversations.md` (Dim 2: Conversations under pressure) - - `knowledge/research/raw/03_expression_dna.md` (Dim 3: Linguistic fingerprint) - - `knowledge/research/raw/04_decisions.md` (Dim 4: Behavior and choices) - - `knowledge/research/raw/05_external_views.md` (Dim 5: External views and criticism) - - `knowledge/research/raw/06_timeline.md` (Dim 6: Cognitive trajectory) -5. Research must follow **taste principles + source blacklist + source hierarchy** (see research prompt). Every evidence item must carry a source weight (1-7) annotation. -6. Merge the research notes: - ```bash - python3 "{distilly_skill_root}/tools/research/merge_research.py" "{skill_dir}" - ``` -7. Read `knowledge/research/merged/summary.md` and confirm the minimum floor: - - `Files scanned >= 6` - - `Unique URLs >= 8` - - `Primary-source markers >= 3` - - `Source metadata blocks >= 6` - - `Contradiction bullets >= 6` - - `Inference bullets >= 6` - - `Potential long quote lines = 0` - - `Track coverage count = 6` - - URLs in notes are actual inspected pages, not platform roots, search/topic pages, or placeholder paths - If these do not hold, keep filling the weak tracks before continuing to any review stage. -8. **Quality checkpoint (Phase 1.5)**: before entering audit, show the user a structured collection summary (with primary-source ratio, contradiction count, candidate mental models, known-answer candidates, thin dimensions, cold figure assessment). Wait for user confirmation before continuing. -9. Then read: - - `prompts/celebrity/budget_unfriendly/audit.md` - - `prompts/celebrity/budget_unfriendly/synthesis.md` - - `references/celebrity_budget_unfriendly_template.md` -10. First write `knowledge/research/reviews/research_audit.md` - - The audit must produce an explicit `PASS / FAIL` - - The audit must verify: source hierarchy compliance (no blacklisted sources), primary-source ratio > 50%, taste principle compliance, cold figure assessment - - If the audit says `FAIL`, follow the Backfill Tasks before synthesis -11. **Extraction checkpoint (Phase 2.5)**: after audit PASS, show the user a summary of candidate mental models (with triple-gate verdict, evidence anchors, failure modes). Confirm reasonableness before synthesis. -12. Then write `knowledge/research/reviews/synthesis.md` - - Apply the triple gate to candidate mental models: - - cross-context recurrence - - generative power - - exclusivity - - Also extract intellectual genealogy seeds (influenced by / diverged from) and Agentic Protocol seeds (the dimensions this person would investigate when facing a novel question) -13. Then use `prompts/celebrity/budget_unfriendly/validation.md` to write: - - `knowledge/research/reviews/validation.md` - - Validation must produce an explicit `PASS / FAIL` - - Validation must perform: known-answer check (≥2 questions) + edge-case check (1 question) + voice check (100-word blind test) + copyright check + Agentic Protocol check - - If validation says `FAIL`, revise the draft before continuing -14. Budget-unfriendly celebrity analysis must prioritize: - - six-track raw notes - - merged research summary - - research audit - - synthesis review (with genealogy + Agentic Protocol seeds) - - validation review - - explicit user notes - -Shared rules for both celebrity profiles: - -- If external collection fails or a platform blocks access: - - tell the user exactly what was blocked - - preserve the raw research notes and merged summary - - continue generation with the available materials - - treat `source_grounding` as incomplete - - **never** invent URLs, quotes, titles, or generic homepage links just to satisfy the checker -- **Do not** store full transcripts, full subtitles, or long verbatim source passages in the repository -- Keep the stored notes paraphrased, structured, and copyright-safe - -Once the family is resolved, analyze along two tracks: - -**Track A (Work Skill)**: -- Refer to `prompts/work_analyzer.md` -- Extract: responsible systems, technical standards, workflow, output preferences, experience -- For `celebrity`, interpret `work` as methods, judgment frameworks, and decision patterns rather than literal job scope - -**Track B (Persona)**: -- Use the family-specific persona analyzer -- If `celebrity` with `research_profile=budget-unfriendly`, use: - - `prompts/celebrity/budget_unfriendly/persona_analyzer.md` -- Translate user-provided tags into concrete behavior rules -- Extract from materials: communication style, decision patterns, interpersonal behavior -- For `celebrity`, retain: - - mental models - - decision heuristics - - expression DNA - - contradictions - - honest boundaries - -### Step 4: Generate and Preview - -Use `prompts/work_builder.md` to generate Work content. -Use the family-specific persona builder to generate Persona content. - -Mapping: -- `colleague` → `prompts/persona_builder.md` -- `relationship` → `prompts/relationship/persona_builder.md` -- `celebrity` → `prompts/celebrity/persona_builder.md` -- `celebrity` + `budget-unfriendly` → `prompts/celebrity/budget_unfriendly/persona_builder.md` - -Show the user a summary (5-8 lines each), ask: -``` -Work Skill Summary: - - Responsible for: {xxx} - - Tech stack: {xxx} - - CR focus: {xxx} - ... - -Persona Summary: - - Core personality: {xxx} - - Communication style: {xxx} - - Decision pattern: {xxx} - ... - -Confirm generation? Or need adjustments? +distilly skill list ``` -### Step 5: Write Files - -After user confirmation, do not hand-build a `skills/colleague/{slug}`-style tree. Always go through the writer: - -1. Resolve the current storage root: - - `colleague` → `./skills/colleague` - - `relationship` → `./skills/relationship` - - `celebrity` → `./skills/celebrity` -2. Use the `Write` tool to create three temporary files: - - `/tmp/distilly_{slug}_meta.json` - - `/tmp/distilly_{slug}_work.md` - - `/tmp/distilly_{slug}_persona.md` -3. The temporary meta file must include at least: - - `name` - - `display_name` - - `character` - - `research_profile` (required when `character=celebrity`) - - `classification.language` (must match the user's language, for example `zh-CN` or `en`) - - `profile` - - `tags` - - `knowledge_sources` -4. Then call: - ```bash - python3 "{distilly_skill_root}/tools/skill_writer.py" \ - --action create \ - --character {character} \ - --research-profile {research_profile} \ - --slug {slug} \ - --name "{name}" \ - --meta /tmp/distilly_{slug}_meta.json \ - --work /tmp/distilly_{slug}_work.md \ - --persona /tmp/distilly_{slug}_persona.md \ - --base-dir {resolved_base_dir} - ``` -5. This command will generate: - - `SKILL.md` - - `work.md` - - `persona.md` - - `work_skill.md` - - `persona_skill.md` - - `manifest.json` - - `meta.json` - - To install the generated role skill into a host, append the relevant flag: - - Claude Code: `--install-claude-skill` - - OpenClaw: `--install-openclaw-skill` - - Codex: `--install-codex-skill` - - Hermes: run `python3 "{distilly_skill_root}/tools/install_generated_skill.py" --skill-dir "{resolved_base_dir}/{slug}" --host hermes --force`; for a trusted project, append `--skills-dir .hermes/skills`, run `hermes skills trust`, then start a new session or run `/reload-skills`. Use `~/.agents/skills` only when it is explicitly configured in Hermes `skills.external_dirs` - - DeepSeek Harness: run `python3 "{distilly_skill_root}/tools/install_generated_skill.py" --skill-dir "{resolved_base_dir}/{slug}" --host deepseek-harness --force`; append `--skills-dir .dsh/skills` for a project install - - Pi: run `python3 "{distilly_skill_root}/tools/install_generated_skill.py" --skill-dir "{resolved_base_dir}/{slug}" --host pi --force`; append `--skills-dir .pi/skills` for a project install, then invoke it with `/skill:{character}-{slug}` - - Grok Build: run `python3 "{distilly_skill_root}/tools/install_generated_skill.py" --skill-dir "{resolved_base_dir}/{slug}" --host grok-build --force`; append `--skills-dir .grok/skills` for a project install - - OpenCode: run `python3 "{distilly_skill_root}/tools/install_generated_skill.py" --skill-dir "{resolved_base_dir}/{slug}" --host opencode --force`; append `--skills-dir .opencode/skills` for a project install - - The shared installer writes only the self-contained `SKILL.md` and install metadata and normalizes legacy frontmatter in the installed copy. Do not manually copy the whole generated directory; it may contain private source material - - Claude Code on Windows: optionally add `--install-claude-command-shim` -6. If the current family is `celebrity`, run a quality check after creation: - ```bash - python3 "{distilly_skill_root}/tools/research/quality_check.py" "{resolved_base_dir}/{slug}/SKILL.md" --profile {research_profile} - ``` -7. If `source_grounding` still fails for a `celebrity` skill: - - you may add honest limitation notes and a grounded source summary - - only add URLs when they are real, specific, and traceable sources - - **never** use site roots, topic pages, search pages, or other generic links as fake grounding - - if no verified external sources exist, keep the FAIL state and explain what source material is still missing - -When reporting success, return the correct family-specific location instead of assuming colleague storage. - ---- - -## Evolution Mode: Append Files - -When user provides new files or text: - -1. Read new content using Step 2 methods -2. Resolve the base dir for the current family -3. `Read` existing `{resolved_base_dir}/{slug}/work.md` and `persona.md` -4. Use the family-specific merger prompt for incremental analysis -5. Archive current version (Bash): - ```bash - python3 "{distilly_skill_root}/tools/version_manager.py" \ - --action backup \ - --character {character} \ - --slug {slug} \ - --base-dir {resolved_base_dir} - ``` -6. Write work/persona delta into temporary patch files -7. Call: - ```bash - python3 "{distilly_skill_root}/tools/skill_writer.py" \ - --action update \ - --character {character} \ - --slug {slug} \ - --work-patch /tmp/distilly_{slug}_work_patch.md \ - --persona-patch /tmp/distilly_{slug}_persona_patch.md \ - --base-dir {resolved_base_dir} - ``` -8. If the current family is `celebrity`, run the quality check again after the update +Roll back a specific skill version: ---- +```bash +distilly skill version +``` -## Evolution Mode: Conversation Correction +Delete a specific skill (after confirming the character family): -When user expresses "that's wrong" / "he should be": - -1. Refer to `prompts/correction_handler.md` to identify correction content -2. Determine if it belongs to Work (technical/workflow) or Persona (personality/communication) -3. If it belongs to Work: - - Generate `/tmp/distilly_{slug}_work_patch.md` - - The patch must be one or more replaceable `##` sections - - Call: - ```bash - python3 "{distilly_skill_root}/tools/skill_writer.py" \ - --action update \ - --character {character} \ - --slug {slug} \ - --work-patch /tmp/distilly_{slug}_work_patch.md \ - --base-dir {resolved_base_dir} - ``` -4. If it belongs to Persona: - - Write the correction record to `/tmp/distilly_{slug}_correction.json` - - For a single correction, write `{scene, wrong, correct}` - - For multiple persona corrections, write `{"persona_corrections": [{...}, {...}]}` - - Call: - ```bash - python3 "{distilly_skill_root}/tools/skill_writer.py" \ - --action update \ - --character {character} \ - --slug {slug} \ - --correction-json /tmp/distilly_{slug}_correction.json \ - --base-dir {resolved_base_dir} - ``` -5. If the current family is `celebrity`, run the quality check again after the update -6. Do not hand-edit `work.md`, `persona.md`, `SKILL.md`, or `meta.json`; always update through `skill_writer.py` +```bash +rm -rf skills/colleague/{slug} +rm -rf skills/relationship/{slug} +rm -rf skills/celebrity/{slug} +``` ---- +Install into a host: `distilly install `; uninstall: `distilly uninstall`. -## Management Operations +List and revoke granted consent: -List skills across the three families: ```bash -python3 "{distilly_skill_root}/tools/skill_writer.py" --action list --character colleague --base-dir ./skills/colleague -python3 "{distilly_skill_root}/tools/skill_writer.py" --action list --character relationship --base-dir ./skills/relationship -python3 "{distilly_skill_root}/tools/skill_writer.py" --action list --character celebrity --base-dir ./skills/celebrity +distilly consent list +distilly consent revoke ``` -Roll back a specific skill version: -```bash -# colleague -python3 "{distilly_skill_root}/tools/version_manager.py" --action rollback --character colleague --slug {slug} --version {version} --base-dir ./skills/colleague +--- -# relationship -python3 "{distilly_skill_root}/tools/version_manager.py" --action rollback --character relationship --slug {slug} --version {version} --base-dir ./skills/relationship +## MUST -# celebrity -python3 "{distilly_skill_root}/tools/version_manager.py" --action rollback --character celebrity --slug {slug} --version {version} --base-dir ./skills/celebrity -``` +- First list "which files were read, how many rows each, how many anchors", then write conclusions; every conclusion carries `file + anchor`. +- With no evidence write `unknown`; candidates never become conclusions. +- Check every step against the completion criteria of the five-step mainline before moving on. -Delete a specific skill: -After confirming the character family: -```bash -# colleague -rm -rf skills/colleague/{slug} +## MUST NOT -# relationship -rm -rf skills/relationship/{slug} +- Never rewrite quotes; never fabricate URLs, anchors, book titles, or video titles; never pad sources with platform roots. +- Credentials are read only from `~/.distilly/*_config.json` or environment variables and never appear in chat, files, receipts, or logs. +- Never hand-craft platform API calls: all network collection goes through `distilly collect` / `distilly harvest` / `distilly parse-chat` / `distilly parse-email` / `distilly parse-subtitle` / `distilly parse-doc` / `distilly parse-archive` / `distilly transcribe`. +- Never degrade silently: failures, unavailable channels, and skipped steps are all stated. -# celebrity -rm -rf skills/celebrity/{slug} -``` +## RECEIPT + +- Which files were read, how many rows each, how many anchors. +- Which files were created or updated, each with its sha256 (from the `distilly` `--json` receipt, `knowledge/index.json`, or `evidence/renders/receipt.json`). +- Which channels were unavailable (`unavailable[]`). +- Which steps were skipped, and why. diff --git a/assets/template.source.html b/assets/template.source.html new file mode 100644 index 00000000..8e968ef3 --- /dev/null +++ b/assets/template.source.html @@ -0,0 +1,246 @@ + + + + + + + + +Distilly · 个人画像 + + + + + +
+
+

+ + Distilly · Person View +

+

个人画像

+

等待 view.json 数据

+

+
+ + + + +
+

+ +

+
+ +
+
+
+ +
+

本页为单文件离线产物：内联 CSS/JS/SVG，无外部请求（CSP default-src 'none'）。

+ +
+
+ + + + + + + + diff --git a/bin/distilly.mjs b/bin/distilly.mjs old mode 100755 new mode 100644 index 7460a406..a5bc9be0 --- a/bin/distilly.mjs +++ b/bin/distilly.mjs @@ -1,204 +1,149 @@ #!/usr/bin/env node - -import { - cpSync, - existsSync, - mkdirSync, - readFileSync, - renameSync, - rmSync, -} from "node:fs"; -import { homedir } from "node:os"; -import { basename, dirname, join, parse, resolve } from "node:path"; +/** + * Distilly entry point. + * + * Contract: `docs/v2/CONTRACT.md` §1 — this is the only user-facing entry. It + * parses global flags, resolves a subcommand through the registry in + * `src/commands/index.mjs`, prints a bilingual help screen, and always answers + * with the receipt shape from §3 when `--json` is set. + * + * Adding a command: create `src/commands/.mjs`, call `register(...)` from + * it, and import that module below. See `docs/v2/NODE-CORE.md`. + */ + +import { existsSync, readFileSync } from "node:fs"; +import { join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; -const packageRoot = fileURLToPath(new URL("..", import.meta.url)); -const packageMetadata = JSON.parse( - readFileSync(join(packageRoot, "package.json"), "utf8"), -); +import "../src/commands/skill.mjs"; +import "../src/commands/install.mjs"; +import "../src/commands/doctor.mjs"; +import "../src/commands/legacy.mjs"; + +import { ArgError, wantsHelp } from "../src/cli/args.mjs"; +import { CliError, createReceipt, createReporter } from "../src/cli/receipt.mjs"; +import { lookup, missingCommandError, renderHelp, resolveCommand } from "../src/commands/index.mjs"; + +export const packageRoot = fileURLToPath(new URL("..", import.meta.url)); +const packageMetadata = JSON.parse(readFileSync(join(packageRoot, "package.json"), "utf8")); +const version = packageMetadata.version; +const binary = "distilly"; -const payloadEntries = [ +/** Files/directories copied into a host by `install `. */ +export const payloadEntries = [ "SKILL.md", "prompts", "references", - "tools", - "requirements.txt", + "bin", + "src", + "assets", + "scripts", + "package.json", "INSTALL.md", "INSTALL_EN.md", "LICENSE", "CITATION.cff", ]; -const hosts = { - "claude-code": () => join(homedir(), ".claude", "skills", "distilly"), - openclaw: () => - join(homedir(), ".openclaw", "workspace", "skills", "distilly"), - hermes: () => - join(homedir(), ".hermes", "skills", "openclaw-imports", "distilly"), - codex: () => join(homedir(), ".agents", "skills", "distilly"), - "deepseek-harness": () => - join(process.env.DSH_HOME || join(homedir(), ".dsh"), "skills", "distilly"), - pi: () => join(homedir(), ".pi", "agent", "skills", "distilly"), - "grok-build": () => join(homedir(), ".grok", "skills", "distilly"), - opencode: () => - join(homedir(), ".config", "opencode", "skills", "distilly"), -}; - -const aliases = { - claude: "claude-code", - deepseek: "deepseek-harness", - grok: "grok-build", -}; - -function printHelp() { - console.log(`Distilly ${packageMetadata.version} - -Install the Distilly creator Skill into a supported agent host. - -Usage: - distilly install [--force] - distilly install --path [--force] - -Hosts: - claude-code, openclaw, hermes, codex, deepseek-harness, - pi, grok-build, opencode - -Options: - --force Preserve an existing install as a timestamped backup, then install - --path Install to a custom path whose final directory is named distilly - --version Print the package version - --help Show this help -`); -} - -function fail(message) { - console.error(`Error: ${message}`); - process.exit(1); -} - -function validatePayload() { - const missing = payloadEntries.filter( - (entry) => !existsSync(join(packageRoot, entry)), - ); +/** Prepack guard: the published payload must be complete and version-consistent. */ +export function validatePayload(root = packageRoot) { + const missing = payloadEntries.filter((entry) => !existsSync(join(root, entry))); if (missing.length > 0) { - fail(`package payload is missing: ${missing.join(", ")}`); + throw new CliError(`package payload is missing: ${missing.join(", ")}`, { + code: "payload-incomplete", + remedy: "restore the missing paths or update payloadEntries in bin/distilly.mjs.", + }); } - const skill = readFileSync(join(packageRoot, "SKILL.md"), "utf8"); - if (!skill.includes(`version: "${packageMetadata.version}"`)) { - fail("package.json version does not match SKILL.md"); + const skill = readFileSync(join(root, "SKILL.md"), "utf8"); + if (!skill.includes(`version: "${version}"`)) { + throw new CliError("package.json version does not match SKILL.md", { + code: "version-mismatch", + remedy: `set SKILL.md frontmatter version to "${version}" (or bump package.json).`, + }); } } -function expandHome(inputPath) { - if (inputPath === "~") return homedir(); - if (inputPath.startsWith("~/")) return join(homedir(), inputPath.slice(2)); - return inputPath; +function failureReceipt(command, error) { + return createReceipt(command, { + ok: false, + error: { + code: error.code ?? "error", + message: error.message, + ...(error.remedy ? { remedy: error.remedy } : {}), + }, + warnings: [error.message], + }); } -function validateTarget(inputPath) { - const target = resolve(expandHome(inputPath)); - const parsed = parse(target); - if (target === parsed.root || target === resolve(homedir())) { - fail("refusing to install into a filesystem root or home directory"); +async function main(argv) { + // `--json` is global (CONTRACT §1): every subcommand answers with a receipt. + const json = argv.includes("--json"); + const args = argv.filter((arg) => arg !== "--json"); + const reporter = createReporter(json); + + if (args.includes("--check-package")) { + validatePayload(); + // A validation diagnostic, not command output: `prepack` shares stdout with + // `npm pack --json`, which must stay parseable. + process.stderr.write("Distilly package payload is valid.\n"); + return 0; } - if (basename(target) !== "distilly") { - fail("the install path must end with a directory named distilly"); - } - return target; -} -function parseInstallArgs(args) { - let host; - let customPath; - let force = false; - - for (let index = 0; index < args.length; index += 1) { - const arg = args[index]; - if (arg === "--force") { - force = true; - } else if (arg === "--path") { - customPath = args[index + 1]; - if (!customPath) fail("--path requires a value"); - index += 1; - } else if (arg.startsWith("--")) { - fail(`unknown option: ${arg}`); - } else if (!host) { - host = aliases[arg] || arg; - } else { - fail(`unexpected argument: ${arg}`); - } + if (args.includes("--version")) { + reporter.line(version); + return 0; } - if (customPath) return { target: validateTarget(customPath), force }; - if (!host) fail("choose a host or pass --path"); - if (!hosts[host]) fail(`unsupported host: ${host}`); - return { target: validateTarget(hosts[host]()), force }; -} - -function timestamp() { - return new Date().toISOString().replaceAll(":", "-").replaceAll(".", "-"); -} + if (args.length === 0 || args[0] === "help" || wantsHelp(args)) { + process.stdout.write(renderHelp({ version, binary })); + return 0; + } -function install(target, force) { - validatePayload(); + const { name, rest } = resolveCommand(args); + const command = lookup(name); + if (!command) throw missingCommandError(name); - if (existsSync(target) && !force) { - fail(`${target} already exists; rerun with --force to preserve and replace it`); + if (wantsHelp(rest)) { + process.stdout.write(`${command.help ?? command.usage}\n`); + return 0; } - const parent = dirname(target); - const staging = join(parent, `.distilly-install-${process.pid}`); - let backup; + const result = (await command.run({ + argv: rest, + json, + reporter, + ctx: { packageRoot, version, binary }, + })) ?? {}; + + const receipt = result.receipt ?? createReceipt(name); + reporter.finish(receipt); + if (result.exitCode !== undefined) return result.exitCode; + return receipt.ok === false ? 1 : 0; +} - mkdirSync(parent, { recursive: true }); - if (existsSync(staging)) { - fail(`temporary install path already exists: ${staging}`); - } +// Dispatch only when this file is the process entry point. Importing it (tests do, +// to reach `validatePayload` and `payloadEntries`) must not run a command with the +// importer's argv — the same guard `scripts/visual-check.mjs` and +// `scripts/blind-test.mjs` already carry. +const isEntryPoint = + process.argv[1] !== undefined && resolve(process.argv[1]) === fileURLToPath(import.meta.url); +if (isEntryPoint) { try { - mkdirSync(staging); - for (const entry of payloadEntries) { - cpSync(join(packageRoot, entry), join(staging, entry), { - recursive: true, - preserveTimestamps: true, - }); - } - - if (!existsSync(join(staging, "SKILL.md"))) { - throw new Error("staged install does not contain SKILL.md"); - } - - if (existsSync(target)) { - backup = `${target}.backup-${timestamp()}`; - renameSync(target, backup); - } - renameSync(staging, target); + process.exitCode = await main(process.argv.slice(2)); } catch (error) { - if (existsSync(staging)) { - rmSync(staging, { recursive: true, force: true }); - } - if (backup && !existsSync(target) && existsSync(backup)) { - renameSync(backup, target); - } - throw error; + const json = process.argv.includes("--json"); + const command = process.argv.slice(2).find((arg) => !arg.startsWith("-")) ?? null; + const reporter = createReporter(json); + const failure = + error instanceof CliError || error instanceof ArgError + ? error + : new CliError(error?.message ?? String(error), { code: "unexpected" }); + + reporter.warn(`Error: ${failure.message}`); + if (failure.remedy) reporter.warn(`Remedy: ${failure.remedy}`); + reporter.finish(failureReceipt(command, failure)); + process.exitCode = failure.exitCode ?? 1; } - - console.log(`Distilly ${packageMetadata.version} installed at ${target}`); - if (backup) console.log(`Previous install preserved at ${backup}`); -} - -const args = process.argv.slice(2); -if (args.includes("--check-package")) { - validatePayload(); - console.log("Distilly package payload is valid."); -} else if (args.includes("--version")) { - console.log(packageMetadata.version); -} else if (args.length === 0 || args.includes("--help") || args[0] === "help") { - printHelp(); -} else if (args[0] === "install") { - const { target, force } = parseInstallArgs(args.slice(1)); - install(target, force); -} else { - fail(`unknown command: ${args[0]}`); } diff --git a/docs/evidence/pr-03-render.md b/docs/evidence/pr-03-render.md new file mode 100644 index 00000000..9a42a79c --- /dev/null +++ b/docs/evidence/pr-03-render.md @@ -0,0 +1,194 @@ +# PR-03 渲染层证据（`ds/03-render`） + +纯文字证据。截图/PNG **不入库**，见 §6；本文件只记录数值、命令与结果。 + +## 1. 变更摘要 + +单文件 HTML 渲染层：模板是生成物、八段页面、双主题、默认私有、`view check` 结构化诊断、 +`view render` 确定性产物 + 回执、Playwright visual-check 八项。 + +| commit | 内容 | +|---|---| +| `feat(view): add the page template fragments and viewer runtime` | `assets/template.source.html`、`viewer/{sections,theme,export,focus}.js` | +| `feat(view): generate the single-file template from fragments` | `scripts/generate-template.mjs` + 生成物 `assets/distilly-template.html` | +| `feat(view): validate view.json against the v2 contract` | `src/views/schema.mjs`（八段/kind/锚点/confidence/12 字原文 + 诊断形状 + 宽容归一化） | +| `feat(view): render a view.json into a single-file HTML page` | `src/views/render.mjs`（payload 注入、回执、确定性、离线校验） | +| `feat(cli): register view check and view render` | `bin/distilly.mjs` 增加 `view` 子命令（最小注册，见 §8 缺口 1） | +| `chore(view): add the playwright visual check script` | `scripts/visual-check.mjs` | +| `test(view): cover schema diagnostics, rendering and template drift` | `tests/views.test.mjs`、`tests/template.test.mjs` | +| `docs(view): document the render layer contract` | `docs/v2/RENDER.md` | +| `docs(evidence): record the pr-03 render evidence` | 本文件 | + +未触碰 `src/parse/**`、`src/knowledge/**`、`src/skill/**`、`src/install/**`、`SKILL.md`、`prompts/**`； +未修改 `docs/v2/CONTRACT.md`。 + +## 2. 测试命令与结果 + +``` +$ node --test tests/views.test.mjs tests/template.test.mjs +# tests 44 +# pass 44 +# fail 0 +# duration_ms 710.41 +``` + +- `tests/views.test.mjs`（35 项）：schema 正/反例、诊断形状逐字段断言、12 字原文边界（11 字不算、12 字算）、 + `--shareable` 放行、锚点格式/回指、宽容形状归一化（含幂等）、确定性（两次渲染字节相同）、 + 私有产物不含引文、`--shareable` 产物含引文且回执记录 `inlined_sources[]`、 + CLI `view check/render --person --json` 与退出码。 +- `tests/template.test.mjs`（9 项）：模板 = 碎片构建结果（字节相同）、`--check` 干净退出 0、 + **碎片改了但模板没重生成 → 非零退出**（在临时目录里改 `viewer/sections.js` 后断言 exit 1， + 重新生成后回到 0）、marker 缺失/重复报错、`` 碎片被拒、产物单文件 + CSP + 无 `http(s)://`、 + 两次渲染字节相同。 + +## 3. 产物大小与 sha256 + +| 文件 | 字节 | sha256 | +|---|---|---| +| `assets/distilly-template.html`（生成物） | 41673 | `cb88a172694f31eb970d8f323ffefc1a13319d8aa17c84705ead959d1c738798` | +| `assets/template.source.html` | 10955 | `483189a41f196c7434aa359881a5c34a265db9d25fea3744aa788a776225f1db` | +| `viewer/sections.js` | 15585 | `80967985aee069058589ac5dff5ce4b5aada3b9af40cf6b5039753700017aaa7` | +| `viewer/theme.js` | 4073 | `3f9e73ee72a3dfbf4ca22e5b14551458b1b7251750e6740ab2db04a99bfc474a` | +| `viewer/export.js` | 6323 | `5eab79b4ceb5f3dabb5a4841c63c1f736bc3a64afdc0528b39fdb6900bccb06f` | +| `viewer/focus.js` | 4592 | `5a17e146cb6a1044ae94d9928cb5bb7198bcd1e2c672dd5d33e656e1a9456c91` | + +示例人 `zhang-san`（fixture 在 `/tmp/dst-fixture`，不入库；8 个锚点、18 条结论）： + +| 产物 | 字节 | sha256 | +|---|---|---| +| `views/zhang-san.html`（默认私有） | 48338 | `2a0ace948a6027dfec12c77e0052c5c0f035fa3742a39fedb7ec47c9e9f46db0` | +| `views/zhang-san.shareable.html`（`--shareable`） | 49244 | `782118192556b49188bfd97da733e389947dc6be0e8bb6b010f9002581b5b9e1` | +| `evidence/renders/receipt.json`（私有运行） | 1505 | `d4aea409510c0535258266b46dd0726890cce5eac6a95109c8e4cdd1ceac911d` | +| `evidence/renders/receipt.json`（shareable 运行） | 3450 | `651cdd69f131d3c6130698211a8934a9456ce8bf7340df3ed770e27901906ba4` | +| 验收语料形状 `lin-gong.html`（无 `meta.slug`/无 `evidence[]`） | 46402 | `cce9cdb8c850da34917b430ec462e8c370f6adb0ce605733f933ae1477bdba4e` | + +确定性：同一输入连续两次 `view render`，HTML 与 receipt 的 sha256 均不变（单元测试与 §4 手工各验一次）。 +私有 HTML 中检索三条引文均**不存在**；`--shareable` HTML 中三条全部存在，且 +`receipt.inlined_sources[]` 长度 8（每条含 `anchor/source/kind/path/quote_sha256/quote_bytes`）。 + +## 4. visual-check 八项（手工跑） + +命令： + +``` +DISTILLY_PLAYWRIGHT_ROOT=<含 node_modules 的目录> \ + node scripts/visual-check.mjs --out /tmp/dst-evidence/pr-03/ +``` + +对三个页面各跑一次，均 **exit 0、8/8 通过**；下面逐条列出私有示例页的结果（`--json` 回执在 `/tmp/vc-private.json`）。 + +| # | 项 | 结果 | 数值 | +|---|---|---|---| +| 1 | console 无 error/warning | ✅ | 0 条 console / 0 个 pageerror / 0 个失败请求 | +| 2 | 八段非空 | ✅ | 8 段 / 26 条 / 附录 8 锚点 | +| 3 | 无横向溢出 | ✅ | 最大溢出 0px @ 1280/768/375px | +| 4 | 双主题对比度 | ✅ | 最小对比度 6.00:1；背景亮度 light 1.0 → dark 0.0118；`light→dark` 手动切换生效 | +| 5 | 锚点可定位到附录 | ✅ | 8/8 个锚点可聚焦（含回指链接），正文引用无悬空 | +| 6 | 零网络请求（离线 + CSP） | ✅ | 0 个外部请求 / 静态外链 0 个 / CSP 存在 | +| 7 | `@media print` 不裁切 | ✅ | 8 段可见 / 正文 1450 字符（与屏幕一致）/ 溢出 0px | +| 8 | 出 PNG | ✅ | 4 张 → `/tmp/dst-evidence/pr-03` | + +另两个页面（同一套断言，逐条全绿）： + +- `legacy`（`lin-gong`，验收脚本的 `view.template.json` 形状 + 无 `evidence[]`）：8 段 / 14 条 / 7 锚点， + 最小对比度 6.00:1，0 外部请求。 +- `shareable`（`zhang-san`）：8 段 / 26 条 / 8 锚点，附录内联 8 条引文，最小对比度 6.00:1。 + +**负向对照**（证明断言真的会失败，不是恒真；输出在 `/tmp/dst-tamper-out/`）： + +| 篡改 | 退出码 | 失败项 | +|---|---|---| +| 删除 CSP meta | 1 | 零网络请求（离线 + CSP） | +| 注入 `https://` 外链 CSS | 1 | console 无 error/warning、零网络请求 | +| payload 删除时间线段 | 1 | 八段非空、锚点可定位到附录、print 不裁切 | +| 亮色 `muted` 改成近背景色 | 1 | 双主题对比度 | +| 暗色 `muted` 改成近背景色 | 1 | 双主题对比度 | +| 手动暗色覆盖失效（`color-scheme` 打回 light） | 1 | 双主题对比度 | +| 正文引用一个没有附录行的锚点 | 1 | 锚点可定位到附录 | +| 原始页面（对照组） | 0 | — | + +其中「亮/暗 muted 低对比度」与「手动覆盖失效」两组对照是发现真实缺陷后补的： +最初 `:root { color-scheme: light }` 把 `light-dark()` 钉死在亮色，导致深色页面根本没变色， +而对比度断言仍然全绿；现在 theme 断言额外比较**实际背景亮度**（light 1.0 / dark 0.0118）， +并断言手动切换必须覆盖系统偏好。 + +## 5. 其他手工验证 + +``` +$ node scripts/generate-template.mjs --check +template is up to date: sha256 cb88a172… (41673 bytes, 4 fragments) # exit 0 + +$ node bin/distilly.mjs view check zhang-san --root /tmp/dst-fixture +view check ok: 7/8 segments, 18 items, 8/8 anchors cited, 0 warning(s) + +$ node bin/distilly.mjs view check --person lin-gong --root /tmp/dst-legacy --json # 验收调用形式 +{"command":"view check","person":"lin-gong","ok":true,...,"anchors":{"total":7,"cited":7}} +``` + +- 指纹/大小一致性：`receipt.outputs[0].sha256` 与实际 HTML 文件 sha256 相同（`tests/views.test.mjs` 断言）。 +- 产物检索 `http://` / `https://` / ``); + fragments.push({ token, path: relative, sha256: sha256(body), bytes: Buffer.byteLength(body, "utf8") }); + } + + const dataMarkers = output.split(DATA_TOKEN).length - 1; + if (dataMarkers !== 1) fail(`marker ${DATA_TOKEN} must appear exactly once in template.source.html (found ${dataMarkers})`); + + // The view payload marker stays in the committed template: src/views/render.mjs is the + // only writer that replaces it. Without a payload the page shows its empty state. + const leftover = (output.replace(DATA_TOKEN, "")).match(/@@DISTILLY:[A-Z_]+@@/g); + if (leftover) fail(`unsubstituted markers remain: ${[...new Set(leftover)].join(", ")}`); + + const header = + "\n`; + output = output.replace("\n", `\n${header}`); + + validateOutput(output); + return { output, fragments }; +} + +function validateOutput(output) { + if (!output.includes(CSP)) fail("the generated template lost the frozen CSP meta tag"); + if (!output.includes('')) fail("the generated template lost "); + if ((output.match(/ element"); + const dataTag = '`)) { + fail(`the view data marker ${DATA_TOKEN} must be the whole payload placeholder`); + } + for (const pattern of ["http://", "https://", "src=\"//", "@import", "url(http"]) { + if (output.includes(pattern)) fail(`the generated template must stay offline but contains ${pattern}`); + } +} + +function parseArgs(argv) { + const options = { check: false, json: false, quiet: false }; + for (const arg of argv) { + if (arg === "--check") options.check = true; + else if (arg === "--json") options.json = true; + else if (arg === "--quiet") options.quiet = true; + else if (arg === "--help" || arg === "-h") options.help = true; + else fail(`unknown option: ${arg}`); + } + return options; +} + +function main() { + const options = parseArgs(process.argv.slice(2)); + if (options.help) { + console.log(`Usage: node scripts/generate-template.mjs [--check] [--json] [--quiet] + + (no flags) write assets/distilly-template.html from template.source.html + viewer/*.js + --check compare the committed template with a fresh build; exit 1 on drift + --json print a receipt instead of prose`); + return 0; + } + + const { output, fragments } = buildTemplate(); + const bytes = Buffer.byteLength(output, "utf8"); + const digest = sha256(output); + const committed = existsSync(TEMPLATE_PATH) ? readFileSync(TEMPLATE_PATH, "utf8") : null; + + if (options.check) { + const drift = committed !== output; + const receipt = { + command: "template --check", + ok: !drift, + template: { path: "assets/distilly-template.html", sha256: digest, bytes, committed_sha256: committed === null ? null : sha256(committed) }, + fragments, + warnings: [], + supported_fixes: drift + ? ["run: node scripts/generate-template.mjs", "commit assets/distilly-template.html with the fragment change"] + : [], + }; + if (options.json) console.log(JSON.stringify(receipt, null, 2)); + else if (drift) { + console.error("template drift: assets/distilly-template.html is not the build of the current fragments."); + console.error(` built sha256 ${digest} (${bytes} bytes)`); + console.error(` on disk sha256 ${committed === null ? "missing" : sha256(committed)}`); + console.error(" fix: node scripts/generate-template.mjs && git add assets/distilly-template.html"); + } else if (!options.quiet) { + console.log(`template is up to date: sha256 ${digest} (${bytes} bytes, ${fragments.length} fragments)`); + } + return drift ? 1 : 0; + } + + if (committed === output) { + if (!options.quiet) console.log(`template unchanged: sha256 ${digest} (${bytes} bytes)`); + return 0; + } + + writeFileSync(TEMPLATE_PATH, output, "utf8"); + if (options.json) { + console.log( + JSON.stringify( + { + command: "template", + ok: true, + outputs: [{ path: "assets/distilly-template.html", sha256: digest, bytes }], + fragments, + warnings: [], + }, + null, + 2, + ), + ); + } else if (!options.quiet) { + console.log(`wrote assets/distilly-template.html: sha256 ${digest} (${bytes} bytes)`); + for (const entry of fragments) console.log(` ${entry.token} <- ${entry.path} (${entry.bytes} bytes)`); + } + return 0; +} + +if (process.argv[1] && isMain()) { + try { + process.exitCode = main(); + } catch (error) { + console.error(`Error: ${error.message}`); + process.exitCode = 1; + } +} + +/** True when this file is the process entry point (/tmp vs /private/tmp safe). */ +function isMain() { + try { + return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); + } catch (error) { + return false; + } +} diff --git a/scripts/parity.mjs b/scripts/parity.mjs new file mode 100644 index 00000000..96dbc9ee --- /dev/null +++ b/scripts/parity.mjs @@ -0,0 +1,663 @@ +#!/usr/bin/env node +/** + * Byte-parity harness: pinned Python implementation vs. the Node core. + * + * The Python original is exported from git (it is deleted on this branch), so + * the comparison always runs against the frozen reference rather than a + * leftover working copy: + * + * node scripts/parity.mjs [--rev ] [--python ] [--keep] + * [--report ] + * + * Default revision: `git merge-base HEAD dot-skill-test` (the pre-port tree). + * Everything is compared byte for byte: + * A. library level — create/update/list/version operations writing the six + * artifacts plus meta.json into identical sandboxes; + * B. CLI level — `python3 tools/*.py` vs `node bin/distilly.mjs …` stdout, + * stderr and exit codes for the same command lines; + * C. pure helpers — slugify / normalize_command_slug / patch merging. + * + * The clock is frozen through `DISTILLY_PARITY_NOW` so timestamps cannot mask a + * real difference; the Python driver patches `now_iso` to the same value, and + * archived-version mtimes are pinned before they are listed. + */ + +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + readdirSync, + rmSync, + statSync, + utimesSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, relative, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const here = dirname(fileURLToPath(import.meta.url)); +const repoRoot = resolve(here, ".."); +const FROZEN_NOW = "2024-01-02T03:04:05.678901+00:00"; +const FROZEN_MTIME = Math.floor(Date.parse("2024-01-02T03:04:05Z") / 1000); + +function arg(name, fallback) { + const index = process.argv.indexOf(`--${name}`); + return index === -1 ? fallback : process.argv[index + 1]; +} +const has = (name) => process.argv.includes(`--${name}`); + +const keep = has("keep"); +const pythonExe = arg("python", process.env.PARITY_PYTHON ?? "python3"); +const reportPath = arg("report", null); + +function git(args) { + const result = spawnSync("git", args, { cwd: repoRoot, encoding: "utf8" }); + if (result.status !== 0) { + throw new Error(`git ${args.join(" ")} failed: ${result.stderr ?? result.stdout}`); + } + return result.stdout.trim(); +} + +const rev = arg("rev", null) ?? process.env.PARITY_PY_REV ?? git(["merge-base", "HEAD", "dot-skill-test"]); + +const results = []; +let failures = 0; + +function record(section, name, ok, detail = "") { + results.push({ section, name, ok, detail }); + if (!ok) failures += 1; + console.log(` ${ok ? "PASS" : "DIFF"} ${section} :: ${name}${detail ? ` — ${detail}` : ""}`); +} + +function sha256(buffer) { + return createHash("sha256").update(buffer).digest("hex"); +} + +function walk(root, base = root, into = new Map()) { + for (const entry of readdirSync(root).sort()) { + const full = join(root, entry); + if (statSync(full).isDirectory()) walk(full, base, into); + else into.set(relative(base, full), sha256(readFileSync(full))); + } + return into; +} + +function compareTrees(label, leftRoot, rightRoot) { + const left = existsSync(leftRoot) ? walk(leftRoot) : new Map(); + const right = existsSync(rightRoot) ? walk(rightRoot) : new Map(); + const names = [...new Set([...left.keys(), ...right.keys()])].sort(); + const differences = []; + for (const name of names) { + const a = left.get(name); + const b = right.get(name); + if (a === b) continue; + differences.push(`${name} [${a ? a.slice(0, 10) : "missing"} vs ${b ? b.slice(0, 10) : "missing"}]`); + } + record( + label, + `${names.length} files byte-identical`, + differences.length === 0, + differences.slice(0, 5).join("; ") + (differences.length > 5 ? ` (+${differences.length - 5} more)` : ""), + ); + return { total: names.length, differences }; +} + +function pinMtimes(root, epochSeconds = FROZEN_MTIME) { + if (!existsSync(root)) return; + for (const entry of readdirSync(root)) { + const full = join(root, entry); + if (statSync(full).isDirectory()) { + utimesSync(full, epochSeconds, epochSeconds); + pinMtimes(full, epochSeconds); + } + } +} + +const sandbox = mkdtempSync(join(tmpdir(), "dst-parity-")); +const pyRoot = join(sandbox, "python"); +const nodeRoot = join(sandbox, "node"); +mkdirSync(pyRoot, { recursive: true }); +mkdirSync(nodeRoot, { recursive: true }); + +const archive = join(sandbox, "pinned-tools.tar"); +const archiveResult = spawnSync("git", ["archive", "--format=tar", "-o", archive, rev, "tools"], { + cwd: repoRoot, + encoding: "utf8", +}); +if (archiveResult.status !== 0) throw new Error(`cannot export tools/ at ${rev}: ${archiveResult.stderr}`); +const untar = spawnSync("tar", ["-xf", archive, "-C", pyRoot], { encoding: "utf8" }); +if (untar.status !== 0) throw new Error(`cannot extract ${archive}: ${untar.stderr}`); + +console.log(`parity: pinned rev ${rev}`); +console.log(`parity: python ${pythonExe}`); +console.log(`parity: sandbox ${sandbox}\n`); + +/* ------------------------------------------------------------------ * + * Shared scenario script. Both drivers implement exactly these steps * + * and write the same files; nothing is sorted output-only. * + * ------------------------------------------------------------------ */ + +const PY_DRIVER = String.raw` +import contextlib, io, json, os, sys +from pathlib import Path + +TOOLS = Path(__file__).resolve().parent +sys.path.insert(0, str(TOOLS)) + +FIXED = os.environ["DISTILLY_PARITY_NOW"] +import skill_schema, skill_writer, version_manager # noqa: E402 + +skill_schema.now_iso = lambda: FIXED +skill_writer.now_iso = lambda: FIXED +version_manager.now_iso = lambda: FIXED + +OUT = Path(sys.argv[1]).resolve() +OUT.mkdir(parents=True, exist_ok=True) +REPORT = OUT.parent / "report" +REPORT.mkdir(parents=True, exist_ok=True) + + +def emit(name, value): + text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, indent=2) + (REPORT / name).write_text(text, encoding="utf-8") + + +def capture(fn, *args, **kwargs): + buffer = io.StringIO() + with contextlib.redirect_stdout(buffer), contextlib.redirect_stderr(buffer): + try: + result = fn(*args, **kwargs) + except Exception as error: + print("EXC %s: %s" % (type(error).__name__, error)) + result = None + return buffer.getvalue() + "\n<>" % (result,) + + +def new_base(name, character): + base = OUT / name / "skills" / character + base.mkdir(parents=True, exist_ok=True) + return base + + +WORK_BODY = "## mental models\n- First-principles reasoning\n- Skeptical framing\n\n## limitations\n- Avoids operational detail\n\nSources:\nhttps://example.com/a\nhttps://example.com/b\n" +PERSONA_BODY = "## expression DNA\n- Sentence rhythm is clipped.\n- Uses metaphor when disagreeing.\n\n## honest boundaries\n- States what they do not know.\n" +ZH_WORK = "## \u5de5\u4f5c\u80fd\u529b\u4f7f\u7528\u8bf4\u660e\n\n\u5f53\u7528\u6237\u8981\u6c42\u4f60\u5b8c\u6210\u4ee5\u4e0b\u4efb\u52a1\u65f6\uff0c\u4e25\u683c\u6309\u7167\u4e0a\u8ff0\u89c4\u8303\u6267\u884c\u3002\n\n\u5982\u679c\u88ab\u95ee\u5230\u804c\u8d23\u8303\u56f4\u5916\u7684\u95ee\u9898\uff0c\u4ee5\u8be5\u540c\u4e8b\u7684\u65b9\u5f0f\u56de\u5e94\uff08\u53c2\u89c1 Persona \u90e8\u5206\uff09\u3002\n" +EN_WORK = "## Scope rule\n\nIf you are asked a question outside your recorded responsibilities, respond in this colleague's style (see the Persona section).\n\n## Persona naming note\n\nKeep this documentation sentence.\n" + +# s1: colleague with rich metadata +base = new_base("s1", "colleague") +skill_writer.create_skill(base, "eulalie", { + "character": "colleague", + "display_name": "Eulalie", + "classification": {"language": "en"}, + "profile": {"company": "ByteDance", "level": "L2-1", "role": "Backend Engineer", "mbti": "INTJ"}, + "tags": {"personality": ["direct", "data-driven"], "culture": ["byte-dance-style"]}, + "knowledge_sources": ["manual-notes"], +}, WORK_BODY, PERSONA_BODY) + +# s2: relationship, Chinese chrome +base = new_base("s2", "relationship") +skill_writer.create_skill(base, "mireille", { + "character": "relationship", + "name": "Mireille", + "classification": {"language": "zh-CN"}, + "profile": {"role": "Designer"}, +}, ZH_WORK, PERSONA_BODY) + +# s3: celebrity with tags list and research dirs +base = new_base("s3", "celebrity") +skill_writer.create_skill(base, "zadie-smith", { + "character": "celebrity", + "name": "Zadie Smith", + "profile": {"identity": "Novelist", "known_for": "Essay and criticism"}, + "tags": ["literature", "essay", "public-intellectual"], + "knowledge_sources": ["interview", "essay"], +}, EN_WORK, PERSONA_BODY) + +# s4: celebrity, deep research profile, Chinese, string profile +base = new_base("s4", "celebrity") +skill_writer.create_skill(base, "xu-zhisheng", { + "character": "celebrity", + "research_profile": "budget-unfriendly", + "name": "Xu Zhisheng", + "classification": {"language": "zh-CN"}, + "profile": "\u4e2d\u56fd\u8131\u53e3\u79c0\u6f14\u5458\u3002", +}, ZH_WORK, PERSONA_BODY) + +# s5: legacy dot-skill identifiers survive +base = new_base("s5", "colleague") +skill_writer.create_skill(base, "legacy", { + "name": "Legacy", + "preset": "dot.colleague.v1", + "engine": {"name": "dot-skill"}, + "generation": {"engine": "dot-skill"}, + "artifacts": { + "combined_name": "colleague_legacy", + "work_name": "colleague_legacy_work", + "persona_name": "colleague_legacy_persona", + }, +}, "Work body\n", "Persona body\n") + +# s6: updates on s1 +skill_dir = OUT / "s1" / "skills" / "colleague" / "eulalie" +emit("update-work-patch.txt", capture(skill_writer.update_skill, skill_dir, "## new evidence\n- Adds a later example.\n")) +emit("update-correction.txt", capture(skill_writer.update_skill, skill_dir, None, None, {"scene": "disagreement", "wrong": "flatten disagreement", "correct": "surface it directly"})) +emit("update-replace-sections.txt", capture(skill_writer.update_skill, skill_dir, "## mental models\n- Replaced wholesale\n", "## expression DNA\n- Rewritten section\n")) +emit("update-multi-corrections.txt", capture(skill_writer.update_skill, skill_dir, None, None, {"persona_corrections": [ + {"scene": "\u94fa\u9648\u5904\u5883\u65f6", "wrong": "\u4e00\u4e0a\u6765\u5c31\u4e0b\u5224\u65ad", "correct": "\u5148\u628a\u5904\u5883\u8bb2\u5f97\u5f88\u666e\u901a"}, + {"scene": "\u8868\u8fbe\u7acb\u573a\u65f6", "wrong": "\u5199\u6210\u660e\u663e\u81ea\u5632\u578b", "correct": "\u548c\u89c2\u4f17\u4e00\u8d77\u627f\u8ba4"}, +]})) + +# s7: listing +emit("list-s1.txt", skill_writer.list_skills(OUT / "s1" / "skills" / "colleague")) +emit("list-missing.txt", skill_writer.list_skills(OUT / "nope" / "skills")) + +# s8: version flow +emit("version-backup.txt", capture(version_manager.backup_current_version, skill_dir)) +emit("version-list-before.txt", version_manager.list_versions(skill_dir)) +emit("version-rollback-ok.txt", capture(version_manager.rollback, skill_dir, "v1")) +emit("version-rollback-missing.txt", capture(version_manager.rollback, skill_dir, "v99")) +emit("version-rollback-traversal.txt", capture(version_manager.rollback, skill_dir, "../v1")) +emit("version-list-after.txt", version_manager.list_versions(skill_dir)) +emit("version-cleanup.txt", capture(version_manager.cleanup_old_versions, skill_dir, 2)) +emit("version-cleanup-again.txt", capture(version_manager.cleanup_old_versions, skill_dir, 10)) + +# s9: helpers and slug behaviour +for index, value in enumerate(["Zadie Smith", "\u00c9lodie", "A/B", " --A--B-- ", "!!!", "\u5468\u5947\u58a8"]): + emit("normalize-command-slug-%d.txt" % index, skill_schema.normalize_command_slug(value)) +emit("merge-append.txt", skill_writer.merge_markdown_patch("intro\n", "no headings here\n")) +emit("merge-replace.txt", skill_writer.merge_markdown_patch("intro\n\n## A\n\nold\n\n## B\n\nkeep\n", "## A\n\nnew\n")) +emit("merge-unknown-section.txt", skill_writer.merge_markdown_patch("intro\n\n## A\n\nold\n", "## Z\n\nadded\n")) +emit("work-only-zh.txt", skill_writer.work_only_content(ZH_WORK, chinese=True)) +emit("work-only-en.txt", skill_writer.work_only_content(EN_WORK, chinese=False)) +emit("validate-segments.txt", [skill_schema.validate_path_segment(value) for value in ["Zadie Smith", "\u00c9lodie"]]) +emit("validate-rejects.txt", [ + capture(skill_schema.validate_path_segment, value).strip() + for value in ["C:", "foo:bar", "CON", "nul.txt", "trailing.", "trailing ", "", ".."] +]) +`; + +const NODE_DRIVER = String.raw` +import { mkdirSync, writeFileSync, existsSync } from "node:fs"; +import { join, resolve } from "node:path"; + +import * as skillSchema from "__REPO__/src/skill/schema.mjs"; +import * as skillWriter from "__REPO__/src/skill/writer.mjs"; +import * as versionManager from "__REPO__/src/skill/versions.mjs"; + +const OUT = resolve(process.argv[2]); +mkdirSync(OUT, { recursive: true }); +const REPORT = join(OUT, "..", "report"); +mkdirSync(REPORT, { recursive: true }); + +const emit = (name, value) => { + const text = + typeof value === "string" ? value : JSON.stringify(value, null, 2); + writeFileSync(join(REPORT, name), text, "utf8"); +}; + +const capture = (fn, ...args) => { + const chunks = []; + const originalOut = process.stdout.write.bind(process.stdout); + const originalErr = process.stderr.write.bind(process.stderr); + let result; + const sink = (chunk) => { + chunks.push(String(chunk)); + return true; + }; + process.stdout.write = sink; + process.stderr.write = sink; + try { + result = fn(...args); + } catch (error) { + chunks.push("EXC " + error.name + ": " + error.message + "\n"); + result = undefined; + } finally { + process.stdout.write = originalOut; + process.stderr.write = originalErr; + } + const printed = chunks.join(""); + return printed + "\n<>"; +}; + +function require_repr(value) { + if (value === undefined) return "None"; + if (value === null) return "None"; + if (typeof value === "boolean") return value ? "True" : "False"; + if (typeof value === "number") return String(value); + return JSON.stringify(value); +} + +const newBase = (name, character) => { + const base = join(OUT, name, "skills", character); + mkdirSync(base, { recursive: true }); + return base; +}; + +const WORK_BODY = "## mental models\n- First-principles reasoning\n- Skeptical framing\n\n## limitations\n- Avoids operational detail\n\nSources:\nhttps://example.com/a\nhttps://example.com/b\n"; +const PERSONA_BODY = "## expression DNA\n- Sentence rhythm is clipped.\n- Uses metaphor when disagreeing.\n\n## honest boundaries\n- States what they do not know.\n"; +const ZH_WORK = "## \u5de5\u4f5c\u80fd\u529b\u4f7f\u7528\u8bf4\u660e\n\n\u5f53\u7528\u6237\u8981\u6c42\u4f60\u5b8c\u6210\u4ee5\u4e0b\u4efb\u52a1\u65f6\uff0c\u4e25\u683c\u6309\u7167\u4e0a\u8ff0\u89c4\u8303\u6267\u884c\u3002\n\n\u5982\u679c\u88ab\u95ee\u5230\u804c\u8d23\u8303\u56f4\u5916\u7684\u95ee\u9898\uff0c\u4ee5\u8be5\u540c\u4e8b\u7684\u65b9\u5f0f\u56de\u5e94\uff08\u53c2\u89c1 Persona \u90e8\u5206\uff09\u3002\n"; +const EN_WORK = "## Scope rule\n\nIf you are asked a question outside your recorded responsibilities, respond in this colleague's style (see the Persona section).\n\n## Persona naming note\n\nKeep this documentation sentence.\n"; + +// s1 +let base = newBase("s1", "colleague"); +skillWriter.createSkill(base, "eulalie", { + character: "colleague", + display_name: "Eulalie", + classification: { language: "en" }, + profile: { company: "ByteDance", level: "L2-1", role: "Backend Engineer", mbti: "INTJ" }, + tags: { personality: ["direct", "data-driven"], culture: ["byte-dance-style"] }, + knowledge_sources: ["manual-notes"], +}, WORK_BODY, PERSONA_BODY); + +// s2 +base = newBase("s2", "relationship"); +skillWriter.createSkill(base, "mireille", { + character: "relationship", + name: "Mireille", + classification: { language: "zh-CN" }, + profile: { role: "Designer" }, +}, ZH_WORK, PERSONA_BODY); + +// s3 +base = newBase("s3", "celebrity"); +skillWriter.createSkill(base, "zadie-smith", { + character: "celebrity", + name: "Zadie Smith", + profile: { identity: "Novelist", known_for: "Essay and criticism" }, + tags: ["literature", "essay", "public-intellectual"], + knowledge_sources: ["interview", "essay"], +}, EN_WORK, PERSONA_BODY); + +// s4 +base = newBase("s4", "celebrity"); +skillWriter.createSkill(base, "xu-zhisheng", { + character: "celebrity", + research_profile: "budget-unfriendly", + name: "Xu Zhisheng", + classification: { language: "zh-CN" }, + profile: "\u4e2d\u56fd\u8131\u53e3\u79c0\u6f14\u5458\u3002", +}, ZH_WORK, PERSONA_BODY); + +// s5 +base = newBase("s5", "colleague"); +skillWriter.createSkill(base, "legacy", { + name: "Legacy", + preset: "dot.colleague.v1", + engine: { name: "dot-skill" }, + generation: { engine: "dot-skill" }, + artifacts: { + combined_name: "colleague_legacy", + work_name: "colleague_legacy_work", + persona_name: "colleague_legacy_persona", + }, +}, "Work body\n", "Persona body\n"); + +// s6 +const skillDir = join(OUT, "s1", "skills", "colleague", "eulalie"); +emit("update-work-patch.txt", capture(skillWriter.updateSkill, skillDir, "## new evidence\n- Adds a later example.\n")); +emit("update-correction.txt", capture(skillWriter.updateSkill, skillDir, null, null, { scene: "disagreement", wrong: "flatten disagreement", correct: "surface it directly" })); +emit("update-replace-sections.txt", capture(skillWriter.updateSkill, skillDir, "## mental models\n- Replaced wholesale\n", "## expression DNA\n- Rewritten section\n")); +emit("update-multi-corrections.txt", capture(skillWriter.updateSkill, skillDir, null, null, { persona_corrections: [ + { scene: "\u94fa\u9648\u5904\u5883\u65f6", wrong: "\u4e00\u4e0a\u6765\u5c31\u4e0b\u5224\u65ad", correct: "\u5148\u628a\u5904\u5883\u8bb2\u5f97\u5f88\u666e\u901a" }, + { scene: "\u8868\u8fbe\u7acb\u573a\u65f6", wrong: "\u5199\u6210\u660e\u663e\u81ea\u5632\u578b", correct: "\u548c\u89c2\u4f17\u4e00\u8d77\u627f\u8ba4" }, +] })); + +// s7 +emit("list-s1.txt", skillWriter.listSkills(join(OUT, "s1", "skills", "colleague"))); +emit("list-missing.txt", skillWriter.listSkills(join(OUT, "nope", "skills"))); + +// s8 +emit("version-backup.txt", capture(versionManager.backupCurrentVersion, skillDir)); +emit("version-list-before.txt", versionManager.listVersions(skillDir)); +emit("version-rollback-ok.txt", capture(versionManager.rollback, skillDir, "v1")); +emit("version-rollback-missing.txt", capture(versionManager.rollback, skillDir, "v99")); +emit("version-rollback-traversal.txt", capture(versionManager.rollback, skillDir, "../v1")); +emit("version-list-after.txt", versionManager.listVersions(skillDir)); +emit("version-cleanup.txt", capture(versionManager.cleanupOldVersions, skillDir, 2)); +emit("version-cleanup-again.txt", capture(versionManager.cleanupOldVersions, skillDir, 10)); + +// s9 +["Zadie Smith", "\u00c9lodie", "A/B", " --A--B-- ", "!!!", "\u5468\u5947\u58a8"].forEach((value, index) => { + emit("normalize-command-slug-" + index + ".txt", skillSchema.normalizeCommandSlug(value)); +}); +emit("merge-append.txt", skillWriter.mergeMarkdownPatch("intro\n", "no headings here\n")); +emit("merge-replace.txt", skillWriter.mergeMarkdownPatch("intro\n\n## A\n\nold\n\n## B\n\nkeep\n", "## A\n\nnew\n")); +emit("merge-unknown-section.txt", skillWriter.mergeMarkdownPatch("intro\n\n## A\n\nold\n", "## Z\n\nadded\n")); +emit("work-only-zh.txt", skillWriter.workOnlyContent(ZH_WORK, { chinese: true })); +emit("work-only-en.txt", skillWriter.workOnlyContent(EN_WORK, { chinese: false })); +emit("validate-segments.txt", ["Zadie Smith", "\u00c9lodie"].map((value) => skillSchema.validatePathSegment(value))); +emit("validate-rejects.txt", ["C:", "foo:bar", "CON", "nul.txt", "trailing.", "trailing ", "", ".."].map((value) => + capture(skillSchema.validatePathSegment, value).trim(), +)); +`; + +/* ---------------------------- phase A ---------------------------- */ + +const pyDriverPath = join(pyRoot, "tools", "parity_driver.py"); +writeFileSync(pyDriverPath, PY_DRIVER, "utf8"); + +const nodeDriverPath = join(nodeRoot, "parity_driver.mjs"); +writeFileSync(nodeDriverPath, NODE_DRIVER.replaceAll("__REPO__", repoRoot), "utf8"); + +const env = { ...process.env, DISTILLY_PARITY_NOW: FROZEN_NOW, PYTHONDONTWRITEBYTECODE: "1" }; + +const pythonRun = spawnSync(pythonExe, [pyDriverPath, join(pyRoot, "out")], { + cwd: pyRoot, + encoding: "utf8", + env, +}); +if (pythonRun.status !== 0) { + console.error(pythonRun.stdout); + console.error(pythonRun.stderr); + throw new Error(`python driver failed with status ${pythonRun.status}`); +} + +const nodeRun = spawnSync(process.execPath, [nodeDriverPath, join(nodeRoot, "out")], { + cwd: nodeRoot, + encoding: "utf8", + env, +}); +if (nodeRun.status !== 0) { + console.error(nodeRun.stdout); + console.error(nodeRun.stderr); + throw new Error(`node driver failed with status ${nodeRun.status}`); +} + +pinMtimes(join(pyRoot, "out")); +pinMtimes(join(nodeRoot, "out")); + +compareTrees("A library", join(pyRoot, "out"), join(nodeRoot, "out")); + +/* ---------------------------- phase B ---------------------------- */ + +const CLI_STEPS = [ + { + name: "create", + python: ["tools/skill_writer.py", "--action", "create", "--character", "colleague", "--slug", "eulalie", "--name", "Eulalie", "--meta", "meta.json", "--work", "work.md", "--persona", "persona.md", "--base-dir", "skills/colleague"], + node: ["skill", "create", "--character", "colleague", "--slug", "eulalie", "--name", "Eulalie", "--meta", "meta.json", "--work", "work.md", "--persona", "persona.md", "--base-dir", "skills/colleague"], + }, + { + name: "create-pinyin-name", + python: ["tools/skill_writer.py", "--action", "create", "--character", "colleague", "--name", "Zadie Smith", "--base-dir", "skills/colleague"], + node: ["skill", "create", "--character", "colleague", "--name", "Zadie Smith", "--base-dir", "skills/colleague"], + }, + { + name: "list", + python: ["tools/skill_writer.py", "--action", "list", "--character", "colleague", "--base-dir", "skills/colleague"], + node: ["skill", "list", "--character", "colleague", "--base-dir", "skills/colleague"], + }, + { + name: "update", + python: ["tools/skill_writer.py", "--action", "update", "--character", "colleague", "--slug", "eulalie", "--base-dir", "skills/colleague", "--work-patch", "patch.md", "--correction-json", "correction.json"], + node: ["skill", "update", "--character", "colleague", "--slug", "eulalie", "--base-dir", "skills/colleague", "--work-patch", "patch.md", "--correction-json", "correction.json"], + }, + { name: "version-list", python: ["tools/version_manager.py", "--action", "list", "--slug", "eulalie", "--base-dir", "skills/colleague"], node: ["skill", "version", "list", "--slug", "eulalie", "--base-dir", "skills/colleague"] }, + { name: "version-backup", python: ["tools/version_manager.py", "--action", "backup", "--slug", "eulalie", "--base-dir", "skills/colleague"], node: ["skill", "version", "backup", "--slug", "eulalie", "--base-dir", "skills/colleague"] }, + { name: "version-rollback", python: ["tools/version_manager.py", "--action", "rollback", "--slug", "eulalie", "--version", "v1", "--base-dir", "skills/colleague"], node: ["skill", "version", "rollback", "--slug", "eulalie", "--version", "v1", "--base-dir", "skills/colleague"] }, + { name: "version-cleanup", python: ["tools/version_manager.py", "--action", "cleanup", "--slug", "eulalie", "--base-dir", "skills/colleague"], node: ["skill", "version", "cleanup", "--slug", "eulalie", "--base-dir", "skills/colleague"] }, +]; + +function seedCliSandbox(root) { + mkdirSync(join(root, "skills", "colleague"), { recursive: true }); + writeFileSync(join(root, "meta.json"), JSON.stringify({ character: "colleague", display_name: "Eulalie", classification: { language: "en" }, profile: { role: "Backend Engineer" } }, null, 2), "utf8"); + writeFileSync(join(root, "work.md"), "Work body\n", "utf8"); + writeFileSync(join(root, "persona.md"), "Persona body\n", "utf8"); + writeFileSync(join(root, "patch.md"), "## Update\n\nPatched section.\n", "utf8"); + writeFileSync(join(root, "correction.json"), JSON.stringify({ scene: "review", wrong: "hedge", correct: "state the risk plainly" }), "utf8"); +} + +const cliPy = join(sandbox, "cli-python"); +const cliNode = join(sandbox, "cli-node"); +for (const root of [cliPy, cliNode]) { + rmSync(root, { recursive: true, force: true }); + mkdirSync(root, { recursive: true }); + seedCliSandbox(root); +} + +const cliEnv = { ...env, DISTILLY_AUTO_INSTALL_CLAUDE: "0", DOT_SKILL_AUTO_INSTALL_CLAUDE: "0" }; +const cliTranscript = { python: [], node: [] }; + +for (const step of CLI_STEPS) { + // Archive mtimes are minute-precision in the listing; pin them before listing. + if (step.name === "version-list" || step.name === "version-cleanup") { + pinMtimes(join(cliPy, "skills", "colleague", "eulalie", "versions")); + pinMtimes(join(cliNode, "skills", "colleague", "eulalie", "versions")); + } + const py = spawnSync(pythonExe, step.python, { cwd: cliPy, encoding: "utf8", env: cliEnv }); + const js = spawnSync(process.execPath, [join(repoRoot, "bin", "distilly.mjs"), ...step.node], { + cwd: cliNode, + encoding: "utf8", + env: cliEnv, + }); + const pyOut = `status=${py.status}\n--- stdout ---\n${py.stdout}--- stderr ---\n${py.stderr}`; + const jsOut = `status=${js.status}\n--- stdout ---\n${js.stdout}--- stderr ---\n${js.stderr}`; + cliTranscript.python.push(`### ${step.name}\n${pyOut}`); + cliTranscript.node.push(`### ${step.name}\n${jsOut}`); + record( + "B cli", + step.name, + pyOut === jsOut, + pyOut === jsOut ? "" : `exit ${py.status}/${js.status}; first diff at ${firstDifference(pyOut, jsOut)}`, + ); +} + +record( + "B cli", + "installed tree byte-identical", + JSON.stringify([...walk(join(cliPy, "skills")).keys()].sort()) === + JSON.stringify([...walk(join(cliNode, "skills")).keys()].sort()), +); +compareTrees("B cli", join(cliPy, "skills"), join(cliNode, "skills")); + +function firstDifference(left, right) { + const limit = Math.min(left.length, right.length); + for (let index = 0; index < limit; index += 1) { + if (left[index] !== right[index]) { + return `char ${index}: ${JSON.stringify(left.slice(Math.max(0, index - 20), index + 20))} vs ${JSON.stringify(right.slice(Math.max(0, index - 20), index + 20))}`; + } + } + return left.length === right.length ? "identical" : `length ${left.length} vs ${right.length}`; +} + +/* ---------------------------- phase C ---------------------------- */ + +const slugNames = ["Zadie Smith", "\u00C9lodie", "A/B", "Zhou Qimo", "Mireille"]; +const pySlug = spawnSync( + pythonExe, + [ + "-c", + [ + "import sys, json", + `sys.path.insert(0, ${JSON.stringify(join(pyRoot, "tools"))})`, + "import skill_writer", + "names = json.loads(sys.argv[1])", + "out = {}", + "for name in names:", + " try:", + " out[name] = skill_writer.slugify(name)", + " except Exception as error:", + " out[name] = 'EXC ' + type(error).__name__", + "print(json.dumps(out, ensure_ascii=False, sort_keys=True))", + ].join("\n"), + JSON.stringify(slugNames), + ], + { cwd: pyRoot, encoding: "utf8", env }, +); + +let pypinyinAvailable = false; +const pypinyinProbe = spawnSync(pythonExe, ["-c", "import pypinyin"], { encoding: "utf8", env }); +pypinyinAvailable = pypinyinProbe.status === 0; + +if (pySlug.status === 0) { + const pythonSlugs = JSON.parse(pySlug.stdout); + const { slugify } = await import(join(repoRoot, "src/skill/writer.mjs")); + const nodeSlugs = {}; + for (const name of slugNames) { + try { + nodeSlugs[name] = slugify(name); + } catch (error) { + nodeSlugs[name] = `EXC ${error.name}`; + } + } + for (const name of slugNames) { + record( + "C slugify", + `${name} → ${pythonSlugs[name]}`, + pythonSlugs[name] === nodeSlugs[name] || !pypinyinAvailable, + pythonSlugs[name] === nodeSlugs[name] + ? "" + : `node: ${nodeSlugs[name]}${pypinyinAvailable ? "" : " (python ran without pypinyin)"}`, + ); + } +} else { + record("C slugify", "python slugify probe", false, pySlug.stderr.trim().split("\n")[0]); +} + +/* ---------------------------- report ---------------------------- */ + +const summary = { + rev, + python: pythonExe, + pypinyinAvailable, + frozenNow: FROZEN_NOW, + checks: results.length, + failures, + results, +}; + +if (reportPath) { + writeFileSync( + reportPath, + [ + `# parity report`, + ``, + `- pinned rev: \`${rev}\``, + `- python: \`${pythonExe}\` (pypinyin available: ${pypinyinAvailable})`, + `- frozen clock: \`${FROZEN_NOW}\``, + `- checks: ${results.length}, failures: ${failures}`, + ``, + `| section | check | result | detail |`, + `| --- | --- | --- | --- |`, + ...results.map((r) => `| ${r.section} | ${r.name} | ${r.ok ? "OK" : "DIFF"} | ${r.detail.replaceAll("|", "\\|")} |`), + ``, + `Raw transcript (${keep ? sandbox : "sandbox removed"}):`, + ``, + "```text", + ...results.map((r) => `${r.ok ? "PASS" : "DIFF"} ${r.section} :: ${r.name} ${r.detail}`), + "```", + "", + ].join("\n"), + "utf8", + ); +} + +if (keep) console.log(`\nsandbox kept: ${sandbox}`); +else rmSync(sandbox, { recursive: true, force: true }); + +console.log(`\nparity: ${results.length - failures}/${results.length} checks passed`); +process.exitCode = failures === 0 ? 0 : 1; diff --git a/scripts/prompt-lint.mjs b/scripts/prompt-lint.mjs new file mode 100644 index 00000000..7774ad3f --- /dev/null +++ b/scripts/prompt-lint.mjs @@ -0,0 +1,362 @@ +#!/usr/bin/env node +/** + * Prompt contract lint (zero dependencies). + * + * Scans SKILL.md and prompts/**.md and enforces the prompt-layer contract + * documented in docs/v2/PROMPTS.md: + * + * command every `distilly ` must exist in docs/v2/CONTRACT.md §1 + * sections every prompt (and SKILL.md) has 必须/禁止/回执 + MUST/MUST NOT/RECEIPT + * bilingual a `## English` half exists and both halves name the same commands + * anchor anchors are [k00NN] / [k00NN:tM] + * forbidden no shell HTTP client, no Python HTTP library usage, + * no literal credential assignment, no `sk-` shaped key + * deprecated every tools/ ... .py or .sh reference is marked deprecated nearby + * + * Usage: + * node scripts/prompt-lint.mjs [--root ] [--json] + * + * Exit codes: 0 = clean, 1 = findings, 2 = the lint itself could not run + * (missing contract / unparsable command block) — never silently pass. + */ + +import { existsSync, readFileSync, readdirSync, statSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { join, relative, resolve, sep } from "node:path"; + +const RULE_ORDER = [ + "command", + "sections", + "bilingual", + "anchor", + "forbidden", + "deprecated", +]; + +const ANCHOR_TEMPLATES = new Set(["[k00NN]", "[k00NN:tM]"]); +const ANCHOR_VALID = /^\[k\d{4}(:t\d+)?\]$/; +const ANCHOR_CANDIDATE = /\[[kK][0-9:tNMT]+\]/g; +const COMMAND_USE = /distilly[ \t]+([A-Za-z][A-Za-z0-9-]*)/g; + +const FORBIDDEN = [ + { + name: "shell-http-client", + pattern: /\bcurl\b/i, + message: "shell HTTP client must not be used; collect through `distilly collect`", + }, + { + name: "python-http-library", + pattern: + /\bimport\s+requests\b|\bfrom\s+requests\b|\brequests\.(get|post|put|patch|delete|head|options|request|Session)\b/, + message: "hand-written API calls are forbidden; use `distilly collect`", + }, + { + name: "literal-credential", + pattern: + /(?:api[_-]?key|apikey|app[_-]?secret|client[_-]?secret|access[_-]?token|refresh[_-]?token|auth[_-]?token|password|passwd|secret|token)["'`]?\s*[:=]\s*["'`][^"'`\s]{3,}["'`]/i, + message: "a credential literal is written down; read it from ~/.distilly/*_config.json or env", + }, + { + name: "key-prefix", + pattern: + /\bsk-(?:proj-|ant-|live-|test-|or-)?[A-Za-z0-9]{20,}\b|\bsk-(?:x{3,}|\.{3,}|<[^>]+>|\$\{[^}]+\}|YOUR|your|PLACEHOLDER|placeholder|redacted|REDACTED|abc|1234)/, + message: "a key-shaped literal is written down; never put credentials in prompts", + }, +]; + +const ZERO_DEP_MARKER = "distilly"; + +function fail(message) { + process.stderr.write(`prompt-lint: ${message}\n`); + process.exit(2); +} + +function parseArgs(argv) { + const options = { root: null, json: false, help: false }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--root") { + options.root = argv[index + 1]; + if (!options.root) fail("--root requires a directory"); + index += 1; + } else if (arg === "--json") { + options.json = true; + } else if (arg === "--help" || arg === "-h") { + options.help = true; + } else { + fail(`unknown option: ${arg}`); + } + } + return options; +} + +/** Command names come from the frozen contract, never from a second list. */ +export function parseContractCommands(contractText) { + const lines = contractText.split("\n"); + const headingIndex = lines.findIndex((line) => /^##\s+1\./.test(line)); + if (headingIndex < 0) return null; + let fenceStart = -1; + for (let index = headingIndex; index < lines.length; index += 1) { + if (/^```/.test(lines[index])) { + fenceStart = index; + break; + } + } + if (fenceStart < 0) return null; + let fenceEnd = -1; + for (let index = fenceStart + 1; index < lines.length; index += 1) { + if (/^```/.test(lines[index])) { + fenceEnd = index; + break; + } + } + if (fenceEnd < 0) return null; + + const commands = new Set(); + for (const raw of lines.slice(fenceStart + 1, fenceEnd)) { + const line = raw.trim(); + if (line.length === 0 || line.startsWith("#")) continue; + const first = line.split(/\s+/)[0]; + if (/^[a-z][a-z0-9-]*$/.test(first)) commands.add(first); + // `install | uninstall` and `view check | view render` keep the + // second bare word; angle-bracket groups are placeholders, not commands. + const withoutPlaceholders = line.replace(/<[^>]*>/g, " "); + for (const match of withoutPlaceholders.matchAll(/\|\s*([a-z][a-z0-9-]*)\b/g)) { + commands.add(match[1]); + } + } + return commands.size > 0 ? commands : null; +} + +function englishMarkerIndex(lines) { + return lines.findIndex((line) => /^##\s+English\b/.test(line)); +} + +function headingMissing(text, marker) { + const pattern = new RegExp(`^#{2,3}\\s+${marker}\\s*$`, "m"); + return !pattern.test(text); +} + +export function lintText({ path, text, commands, isPrompt }) { + const findings = []; + const lines = text.split("\n"); + const add = (line, rule, message) => + findings.push({ file: path, line: line + 1, rule, message }); + + // --- command names ------------------------------------------------------- + lines.forEach((line, index) => { + for (const match of line.matchAll(COMMAND_USE)) { + const command = match[1]; + if (!commands.has(command)) { + add( + index, + "command", + `unknown command \`distilly ${command}\`; not in docs/v2/CONTRACT.md §1 (${[...commands].sort().join(", ")})`, + ); + } + } + }); + + // --- forbidden tokens ---------------------------------------------------- + lines.forEach((line, index) => { + for (const rule of FORBIDDEN) { + if (rule.pattern.test(line)) { + add(index, "forbidden", `${rule.name}: ${rule.message}`); + } + } + }); + + // --- anchors ------------------------------------------------------------- + lines.forEach((line, index) => { + for (const match of line.matchAll(ANCHOR_CANDIDATE)) { + const token = match[0]; + if (ANCHOR_VALID.test(token) || ANCHOR_TEMPLATES.has(token)) continue; + add( + index, + "anchor", + `malformed anchor ${token}; use [k00NN] or [k00NN:tM]`, + ); + } + }); + + // --- deprecated legacy tooling ------------------------------------------ + const nonEmpty = lines.map((line) => line.trim().length > 0); + lines.forEach((line, index) => { + if (!/tools\/[\w./-]+\.(py|sh)/.test(line)) return; + let start = index; + while (start > 0 && nonEmpty[start - 1]) start -= 1; + let end = index; + while (end < lines.length - 1 && nonEmpty[end + 1]) end += 1; + const paragraph = lines.slice(start, end + 1).join("\n"); + if (!/deprecated/i.test(paragraph)) { + add( + index, + "deprecated", + "legacy tools/... reference is not marked deprecated in its paragraph", + ); + } + }); + + // --- bilingual split ----------------------------------------------------- + const marker = englishMarkerIndex(lines); + const zhText = marker < 0 ? text : lines.slice(0, marker).join("\n"); + const enText = marker < 0 ? "" : lines.slice(marker).join("\n"); + if (marker < 0) { + add(0, "bilingual", "missing `## English` section (中文段 → --- → ## English)"); + } else { + const separator = lines + .slice(Math.max(0, marker - 4), marker) + .some((line) => /^---\s*$/.test(line)); + if (!separator) { + add(marker, "bilingual", "`## English` must be preceded by a `---` separator"); + } + const zhCommands = new Set( + [...zhText.matchAll(COMMAND_USE)].map((match) => match[1]), + ); + const enCommands = new Set( + [...enText.matchAll(COMMAND_USE)].map((match) => match[1]), + ); + for (const command of [...zhCommands].sort()) { + if (!enCommands.has(command)) { + add( + marker, + "bilingual", + `command \`distilly ${command}\` appears only in the Chinese half`, + ); + } + } + for (const command of [...enCommands].sort()) { + if (!zhCommands.has(command)) { + add( + marker, + "bilingual", + `command \`distilly ${command}\` appears only in the English half`, + ); + } + } + } + + // --- required sections --------------------------------------------------- + if (isPrompt || path === "SKILL.md") { + const at = marker < 0 ? 0 : marker; + for (const markerText of ["必须", "禁止", "回执"]) { + if (headingMissing(zhText, markerText)) { + add(at, "sections", `missing Chinese \`## ${markerText}\` section`); + } + } + for (const markerText of ["MUST", "MUST NOT", "RECEIPT"]) { + if (headingMissing(enText, markerText)) { + add(at, "sections", `missing English \`## ${markerText}\` section`); + } + } + } + + return findings; +} + +function collectTargets(root) { + const targets = []; + const skill = join(root, "SKILL.md"); + if (existsSync(skill)) targets.push({ path: "SKILL.md", isPrompt: false }); + + const promptsRoot = join(root, "prompts"); + const walk = (dir) => { + if (!existsSync(dir)) return; + for (const entry of readdirSync(dir).sort()) { + const full = join(dir, entry); + const stats = statSync(full); + if (stats.isDirectory()) { + walk(full); + } else if (entry.endsWith(".md")) { + targets.push({ + path: relative(root, full).split(sep).join("/"), + isPrompt: true, + }); + } + } + }; + walk(promptsRoot); + return targets; +} + +export function runLint(root) { + const contractPath = join(root, "docs", "v2", "CONTRACT.md"); + if (!existsSync(contractPath)) { + fail(`contract not found at ${contractPath}`); + } + const commands = parseContractCommands(readFileSync(contractPath, "utf8")); + if (!commands) { + fail(`could not parse the §1 command block from ${contractPath}`); + } + + const targets = collectTargets(root); + if (targets.length === 0) { + fail(`no SKILL.md or prompts/**/*.md found under ${root}`); + } + + const findings = []; + for (const target of targets) { + const text = readFileSync(join(root, target.path), "utf8"); + findings.push( + ...lintText({ path: target.path, text, commands, isPrompt: target.isPrompt }), + ); + } + + findings.sort( + (left, right) => + left.file.localeCompare(right.file) || + left.line - right.line || + RULE_ORDER.indexOf(left.rule) - RULE_ORDER.indexOf(right.rule) || + left.message.localeCompare(right.message), + ); + + return { + commands: [...commands].sort(), + files: targets.map((target) => target.path), + findings, + }; +} + +function main() { + const options = parseArgs(process.argv.slice(2)); + if (options.help) { + process.stdout.write( + [ + "prompt-lint — prompt contract lint (zero dependencies)", + "", + "Usage: node scripts/prompt-lint.mjs [--root ] [--json]", + "", + `Contract commands: parsed from docs/v2/CONTRACT.md §1 at run time (${ZERO_DEP_MARKER}).`, + "", + ].join("\n"), + ); + return; + } + + const root = resolve( + options.root ?? fileURLToPath(new URL("..", import.meta.url)), + ); + const result = runLint(root); + + if (options.json) { + process.stdout.write( + `${JSON.stringify({ root, ok: result.findings.length === 0, ...result }, null, 2)}\n`, + ); + } else { + for (const finding of result.findings) { + process.stdout.write( + `${finding.file}:${finding.line} ${finding.rule} ${finding.message}\n`, + ); + } + const filesWithFindings = new Set(result.findings.map((f) => f.file)).size; + process.stdout.write( + `prompt-lint: ${result.findings.length} finding(s) in ${filesWithFindings} file(s) across ${result.files.length} file(s) scanned (contract commands: ${result.commands.length})\n`, + ); + } + + if (result.findings.length > 0) process.exit(1); +} + +if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + main(); +} diff --git a/scripts/visual-check.mjs b/scripts/visual-check.mjs new file mode 100644 index 00000000..de69c760 --- /dev/null +++ b/scripts/visual-check.mjs @@ -0,0 +1,488 @@ +#!/usr/bin/env node +/** + * distilly visual-check — open a rendered view page in Chrome and assert the + * eight visual contracts from docs/v2/CONTRACT.md §4: + * + * 1 console is silent (no error/warning, no pageerror, no failed request) + * 2 the eight page segments exist and are non-empty + * 3 no horizontal overflow (1280 / 768 / 375 px) + * 4 dual-theme contrast spot checks (system preference + manual toggle) + * 5 every evidence anchor resolves to a focusable row in the appendix + * 6 zero network requests, CSP present, no external reference + * 7 @media print does not clip or drop content + * 8 PNG evidence is written to --out + * + * playwright is a DEVELOPMENT dependency and is never imported by the runtime: + * when it is missing this script fails loudly with install guidance. + * + * node scripts/visual-check.mjs views/.html [--out ] [--json] + */ +import { createHash } from "node:crypto"; +import { existsSync, mkdirSync, readFileSync, statSync } from "node:fs"; +import { createRequire } from "node:module"; +import { join, resolve } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +const ROOT = fileURLToPath(new URL("..", import.meta.url)); +const DEFAULT_OUT = "/tmp/dst-evidence/pr-03"; +const RESULTS = []; + +const SAMPLE_SELECTORS = [ + "#page-title", + ".claim__text", + ".claim__meta", + ".anchor-ref", + ".badge", + ".warning__text", + ".evidence__anchor", +]; + +function parseArgs(argv) { + const options = { html: null, out: DEFAULT_OUT, json: false }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--json") options.json = true; + else if (arg === "--out" || arg === "--out-dir") { + const value = argv[index + 1]; + if (!value) throw new Error(`${arg} requires a directory`); + options.out = value; + index += 1; + } else if (arg === "--help" || arg === "-h") options.help = true; + else if (arg.startsWith("--")) throw new Error(`unknown option: ${arg}`); + else if (!options.html) options.html = arg; + else throw new Error(`unexpected argument: ${arg}`); + } + return options; +} + +function usage() { + console.log(`Usage: node scripts/visual-check.mjs [--out ] [--json] + + a page produced by: distilly view render + --out PNG output directory (default ${DEFAULT_OUT}; never committed) + --json print the machine-readable result + +Exit code 0 only when all eight checks pass.`); +} + +/** playwright is a dev dependency: resolve it from the usual places, else fail loudly. */ +async function loadChromium() { + const roots = [process.env.DISTILLY_PLAYWRIGHT_ROOT, ROOT, process.cwd()].filter(Boolean); + for (const root of roots) { + try { + const require = createRequire(join(root, "index.cjs")); + const resolved = require.resolve("playwright"); + const mod = await import(pathToFileURL(resolved).href); + const chromium = mod.chromium ?? mod.default?.chromium; + if (chromium) return chromium; + } catch (error) { + /* try the next root */ + } + } + try { + const mod = await import("playwright"); + const chromium = mod.chromium ?? mod.default?.chromium; + if (chromium) return chromium; + } catch (error) { + /* fall through to the loud failure below */ + } + console.error("Error: the visual check needs playwright, which is a development dependency."); + console.error(" npm install --no-save playwright # or: pnpm add -D playwright"); + console.error(" DISTILLY_PLAYWRIGHT_ROOT= node scripts/visual-check.mjs "); + console.error(" distilly itself has zero runtime dependencies; nothing else needs playwright."); + process.exit(2); +} + +async function launch(chromium) { + try { + return await chromium.launch({ channel: "chrome" }); + } catch (error) { + return chromium.launch(); + } +} + +function record(id, name, ok, detail) { + RESULTS.push({ id, name, ok: Boolean(ok), detail }); + return Boolean(ok); +} + +/** Contrast of a node against its nearest opaque ancestor background, WCAG 2.x ratio. */ +function contrastProbe(selectors) { + const parse = (value) => { + const match = /rgba?\(([^)]+)\)/.exec(value || ""); + if (!match) return null; + const parts = match[1].split(/[\s,/]+/).filter(Boolean).map(Number); + return { r: parts[0], g: parts[1], b: parts[2], a: parts.length > 3 ? parts[3] : 1 }; + }; + const luminance = ({ r, g, b }) => { + const channel = (value) => { + const scaled = value / 255; + return scaled <= 0.03928 ? scaled / 12.92 : ((scaled + 0.055) / 1.055) ** 2.4; + }; + return 0.2126 * channel(r) + 0.7152 * channel(g) + 0.0722 * channel(b); + }; + const background = (node) => { + let current = node; + while (current && current.nodeType === 1) { + const colour = parse(getComputedStyle(current).backgroundColor); + if (colour && colour.a > 0.5) return colour; + current = current.parentElement; + } + return { r: 255, g: 255, b: 255, a: 1 }; + }; + const ratio = (a, b) => { + const first = luminance(a); + const second = luminance(b); + return (Math.max(first, second) + 0.05) / (Math.min(first, second) + 0.05); + }; + + const samples = []; + for (const selector of selectors) { + const node = document.querySelector(selector); + if (!node) { + samples.push({ selector, missing: true }); + continue; + } + const style = getComputedStyle(node); + const foreground = parse(style.color); + const behind = background(node); + samples.push({ + selector, + fontSize: Number.parseFloat(style.fontSize), + ratio: foreground ? Number(ratio(foreground, behind).toFixed(2)) : null, + foreground: style.color, + background: `rgb(${behind.r}, ${behind.g}, ${behind.b})`, + }); + } + return { theme: document.documentElement.getAttribute("data-theme-effective"), samples }; +} + +async function main() { + const options = parseArgs(process.argv.slice(2)); + if (options.help) { + usage(); + return 0; + } + if (!options.html) { + usage(); + return 2; + } + const htmlPath = resolve(options.html); + if (!existsSync(htmlPath)) { + console.error(`Error: rendered page not found: ${htmlPath}`); + console.error(" fix: distilly view render (or pass the path of an existing views/.html)"); + return 2; + } + const outDir = resolve(options.out); + mkdirSync(outDir, { recursive: true }); + const url = pathToFileURL(htmlPath).href; + const ready = () => page.waitForFunction(() => document.documentElement.getAttribute("data-view-ready") === "true", null, { timeout: 15000 }); + + const chromium = await loadChromium(); + const browser = await launch(chromium); + const context = await browser.newContext({ + viewport: { width: 1280, height: 900 }, + deviceScaleFactor: 1, + colorScheme: "light", + }); + const page = await context.newPage(); + + const consoleMessages = []; + const pageErrors = []; + const failedRequests = []; + const requests = []; + page.on("console", (message) => { + if (message.type() === "error" || message.type() === "warning") { + consoleMessages.push({ type: message.type(), text: message.text() }); + } + }); + page.on("pageerror", (error) => pageErrors.push(String(error && error.message ? error.message : error))); + page.on("requestfailed", (request) => failedRequests.push({ url: request.url(), error: request.failure()?.errorText ?? null })); + page.on("request", (request) => requests.push({ url: request.url(), type: request.resourceType() })); + + const pngs = []; + const screenshot = async (name) => { + const file = join(outDir, name); + await page.screenshot({ path: file, fullPage: true }); + pngs.push({ file, bytes: statSync(file).size }); + }; + + try { + await page.goto(url, { waitUntil: "load" }); + await ready(); + + /* 1 — console silence ------------------------------------------------ */ + record( + "console", + "console has no error/warning, no page error, no failed request", + consoleMessages.length === 0 && pageErrors.length === 0 && failedRequests.length === 0, + { messages: consoleMessages, pageErrors, failedRequests }, + ); + + /* 2 — eight non-empty segments --------------------------------------- */ + const segments = await page.evaluate(() => { + const rows = [...document.querySelectorAll("[data-section]")].map((node) => ({ + id: node.getAttribute("data-section"), + chars: (node.textContent || "").trim().length, + items: node.querySelectorAll(".claim, .warning, .timeline__item, .evidence").length, + })); + const view = window.DistillyView || {}; + return { + rows, + payloadAnchors: view.view && Array.isArray(view.view.evidence) ? view.view.evidence.length : 0, + appendixAnchors: document.querySelectorAll('[data-section="evidence"] .evidence[data-anchor]').length, + shareable: Boolean(view.shareable), + quotesRendered: document.querySelectorAll(".quote[data-inlined]").length, + }; + }); + const emptySegments = segments.rows.filter((entry) => entry.chars < 8); + record( + "segments", + "the eight page segments exist and are non-empty", + segments.rows.length === 8 && emptySegments.length === 0 && segments.appendixAnchors > 0, + { + count: segments.rows.length, + empty: emptySegments.map((entry) => entry.id), + rows: segments.rows, + shareable: segments.shareable, + quotesRendered: segments.quotesRendered, + }, + ); + + /* 3 — no horizontal overflow ---------------------------------------- */ + const overflow = []; + for (const width of [1280, 768, 375]) { + await page.setViewportSize({ width, height: 900 }); + const measured = await page.evaluate(() => { + const limit = window.innerWidth + 1; + const offenders = []; + for (const node of document.querySelectorAll("body *")) { + const rect = node.getBoundingClientRect(); + if (rect.width > 0 && rect.right > limit) { + offenders.push({ + tag: node.tagName.toLowerCase(), + cls: String(node.className || "").slice(0, 60), + right: Math.round(rect.right), + }); + } + } + return { delta: document.documentElement.scrollWidth - window.innerWidth, offenders: offenders.slice(0, 5) }; + }); + overflow.push({ width, delta: measured.delta, offenders: measured.offenders }); + } + await page.setViewportSize({ width: 1280, height: 900 }); + record( + "overflow", + "no horizontal overflow at 1280/768/375 px", + overflow.every((entry) => entry.delta <= 1), + overflow, + ); + + /* 4 — dual theme contrast ------------------------------------------- */ + const themeRuns = []; + await page.emulateMedia({ colorScheme: "light" }); + themeRuns.push(await page.evaluate(contrastProbe, SAMPLE_SELECTORS)); + await page.emulateMedia({ colorScheme: "dark" }); + themeRuns.push(await page.evaluate(contrastProbe, SAMPLE_SELECTORS)); + await page.emulateMedia({ colorScheme: "light" }); + const toggle = await page.evaluate(() => { + const button = document.getElementById("theme-toggle"); + if (!button) return { ok: false, reason: "no #theme-toggle button" }; + const before = document.documentElement.getAttribute("data-theme-effective"); + button.click(); + const after = document.documentElement.getAttribute("data-theme-effective"); + return { ok: before === "light" && after === "dark", before, after, pressed: button.getAttribute("aria-pressed"), label: button.textContent }; + }); + themeRuns.push(await page.evaluate(contrastProbe, SAMPLE_SELECTORS)); + const contrastFailures = []; + for (const run of themeRuns) { + for (const sample of run.samples) { + if (sample.missing) contrastFailures.push({ ...sample, theme: run.theme, reason: "sample element missing" }); + else if (sample.ratio !== null && sample.ratio < 4.5 && sample.fontSize < 24) { + contrastFailures.push({ ...sample, theme: run.theme, reason: "contrast below 4.5:1" }); + } + } + } + record( + "theme", + "dual theme (system + manual) with >= 4.5:1 contrast samples", + contrastFailures.length === 0 && toggle.ok === true && new Set(themeRuns.map((run) => run.theme)).size >= 2, + { runs: themeRuns, toggle, failures: contrastFailures }, + ); + await page.emulateMedia({ colorScheme: "light" }); + await page.evaluate(() => { + const button = document.getElementById("theme-toggle"); + if (button && document.documentElement.getAttribute("data-theme-effective") === "dark") button.click(); + }); + + /* 5 — anchors resolve into the appendix ------------------------------ */ + const anchorIds = await page.evaluate(() => + [...document.querySelectorAll('[data-section="evidence"] .evidence[data-anchor]')].map((node) => node.getAttribute("data-anchor")), + ); + const anchorProblems = []; + for (const anchor of anchorIds) { + await page.evaluate((id) => { + window.location.hash = `#anchor-${id}`; + }, anchor); + await page.waitForTimeout(40); + const outcome = await page.evaluate((id) => { + const node = document.getElementById(`anchor-${id}`); + if (!node) return { id, ok: false, reason: "no element with that id" }; + const rect = node.getBoundingClientRect(); + return { + id, + ok: true, + inAppendix: Boolean(node.closest('[data-section="evidence"]')), + focused: document.activeElement === node, + visible: rect.top < window.innerHeight && rect.bottom > 0, + backLinks: node.querySelectorAll('a[href^="#section-"]').length, + }; + }, anchor); + if (!outcome.ok || !outcome.inAppendix || !outcome.focused || !outcome.visible || outcome.backLinks === 0) { + anchorProblems.push(outcome); + } + } + await page.evaluate(() => { + try { + window.history.replaceState(null, "", window.location.pathname); + } catch (error) { + window.location.hash = ""; + } + }); + record( + "anchors", + "each evidence anchor locates a focusable row in the appendix", + anchorIds.length > 0 && anchorIds.length === segments.payloadAnchors && anchorProblems.length === 0, + { anchors: anchorIds.length, payloadAnchors: segments.payloadAnchors, problems: anchorProblems }, + ); + + /* 6 — zero network requests ----------------------------------------- */ + const staticRefs = await page.evaluate(() => { + const csp = document.querySelector('meta[http-equiv="Content-Security-Policy"]'); + return { + csp: csp ? csp.getAttribute("content") : null, + externalLinks: document.querySelectorAll('link[href]:not([href^="data:"])').length, + externalScripts: document.querySelectorAll("script[src]").length, + externalImages: document.querySelectorAll('img[src]:not([src^="data:"])').length, + embeds: document.querySelectorAll("iframe, object, embed").length, + urls: (document.documentElement.outerHTML.match(/https?:\/\/[^\s"'<>]+/g) || []).filter( + (value) => !value.includes("www.w3.org"), + ), + }; + }); + const externalRequests = requests.filter( + (entry) => !entry.url.startsWith("file:") && !entry.url.startsWith("data:") && !entry.url.startsWith("blob:"), + ); + record( + "offline", + "zero network requests, frozen CSP present, no external reference", + externalRequests.length === 0 && + Boolean(staticRefs.csp && staticRefs.csp.includes("default-src 'none'")) && + staticRefs.externalLinks === 0 && + staticRefs.externalScripts === 0 && + staticRefs.externalImages === 0 && + staticRefs.embeds === 0 && + staticRefs.urls.length === 0, + { requests: requests.length, externalRequests, staticRefs }, + ); + + /* 7 — print media does not clip -------------------------------------- */ + await page.emulateMedia({ media: "print" }); + const printReport = await page.evaluate(() => { + const nodes = [...document.querySelectorAll("[data-section]")]; + const clipped = []; + let sectionText = 0; + for (const node of nodes) { + const style = getComputedStyle(node); + sectionText += (node.textContent || "").length; + if (style.display === "none" || style.visibility === "hidden") { + clipped.push({ id: node.getAttribute("data-section"), reason: "hidden in print" }); + continue; + } + if (node.scrollWidth > node.clientWidth + 2) { + clipped.push({ id: node.getAttribute("data-section"), reason: "horizontal clip", scrollWidth: node.scrollWidth, clientWidth: node.clientWidth }); + } + if (node.scrollHeight > node.clientHeight + 2) { + clipped.push({ id: node.getAttribute("data-section"), reason: "vertical clip", scrollHeight: node.scrollHeight, clientHeight: node.clientHeight }); + } + } + return { segments: nodes.length, clipped, sectionText, overflow: document.documentElement.scrollWidth - window.innerWidth }; + }); + await screenshot("view-print.png"); + await page.emulateMedia({ media: "screen" }); + const screenReport = await page.evaluate(() => { + const nodes = [...document.querySelectorAll("[data-section]")]; + let sectionText = 0; + for (const node of nodes) sectionText += (node.textContent || "").length; + return { sectionText, segments: nodes.length }; + }); + record( + "print", + "@media print does not clip or drop content", + printReport.clipped.length === 0 && + printReport.segments === 8 && + printReport.overflow <= 1 && + printReport.sectionText === screenReport.sectionText, + { ...printReport, screenSectionText: screenReport.sectionText, screenSegments: screenReport.segments }, + ); + + /* 8 — PNG evidence --------------------------------------------------- */ + await page.goto(`${url}?theme=light`, { waitUntil: "load" }); + await ready(); + await page.setViewportSize({ width: 1280, height: 900 }); + await screenshot("view-light.png"); + await page.goto(`${url}?theme=dark`, { waitUntil: "load" }); + await ready(); + await screenshot("view-dark.png"); + await page.setViewportSize({ width: 375, height: 900 }); + await screenshot("view-mobile-375.png"); + record( + "png", + "PNG evidence written to the output directory", + pngs.length >= 4 && pngs.every((entry) => entry.bytes > 1024), + { outDir, pngs }, + ); + + const failed = RESULTS.filter((entry) => !entry.ok); + const payload = { + command: "visual-check", + ok: failed.length === 0, + html: htmlPath, + html_sha256: createHash("sha256").update(readFileSync(htmlPath)).digest("hex"), + html_bytes: statSync(htmlPath).size, + out_dir: outDir, + shareable: segments.shareable, + appendix_anchors: segments.appendixAnchors, + checks: RESULTS, + pngs, + failed: failed.map((entry) => entry.id), + checks_passed: RESULTS.length - failed.length, + checks_total: RESULTS.length, + }; + + if (options.json) console.log(JSON.stringify(payload, null, 2)); + else { + for (const entry of RESULTS) { + console.log(`${entry.ok ? "PASS" : "FAIL"} ${entry.id.padEnd(9)} ${entry.name}`); + if (!entry.ok) console.log(` detail: ${JSON.stringify(entry.detail)}`); + } + console.log(` input: ${htmlPath} (${payload.html_bytes} bytes, sha256 ${payload.html_sha256})`); + } + console.log( + failed.length === 0 + ? `visual-check: PASS — ${RESULTS.length}/8 checks, PNGs in ${outDir} (${pngs.map((entry) => entry.file.split("/").pop()).join(", ")})` + : `visual-check: FAIL — ${failed.length}/${RESULTS.length} checks failed: ${failed.map((entry) => entry.id).join(", ")}`, + ); + return failed.length === 0 ? 0 : 1; + } finally { + await context.close(); + await browser.close(); + } +} + +try { + process.exitCode = await main(); +} catch (error) { + console.error(`Error: ${error && error.message ? error.message : error}`); + process.exitCode = 1; +} diff --git a/src/cli/args.mjs b/src/cli/args.mjs new file mode 100644 index 00000000..30a4da19 --- /dev/null +++ b/src/cli/args.mjs @@ -0,0 +1,103 @@ +/** + * Minimal argument parser for the Distilly CLI (zero dependencies). + * + * Mirrors the subset of Python's argparse behaviour the ported tools relied on: + * `--flag value`, `--flag=value`, boolean switches, repeated options and + * positionals. Unknown options are a hard error — the CLI never guesses. + */ + +/** Raised for user-facing argument problems (exit code 1). */ +export class ArgError extends Error { + constructor(message) { + super(message); + this.name = "ArgError"; + } +} + +/** + * @typedef {object} OptionSpec + * @property {'boolean'|'string'} type + * @property {string} [alias] short flag without dashes, e.g. `o` + * @property {string} [help] + * @property {boolean} [multiple] collect repeats into an array + * @property {string} [value] metavar shown in help + */ + +function optionNames(longName, spec) { + const names = [`--${longName}`]; + if (spec.alias) names.push(`-${spec.alias}`); + return names; +} + +/** + * @param {string[]} argv + * @param {Record} spec + * @returns {{flags: Record, positionals: string[]}} + */ +export function parseArgs(argv, spec = {}) { + const byName = new Map(); + for (const [longName, option] of Object.entries(spec)) { + for (const name of optionNames(longName, option)) byName.set(name, longName); + } + + const flags = {}; + for (const [longName, option] of Object.entries(spec)) { + if (option.multiple) flags[longName] = []; + else if (option.type === "boolean") flags[longName] = false; + else flags[longName] = undefined; + } + + const positionals = []; + let index = 0; + while (index < argv.length) { + const arg = argv[index]; + if (arg === "--") { + positionals.push(...argv.slice(index + 1)); + break; + } + if (arg.startsWith("-") && arg !== "-") { + const equals = arg.indexOf("="); + const name = equals === -1 ? arg : arg.slice(0, equals); + const inlineValue = equals === -1 ? undefined : arg.slice(equals + 1); + const longName = byName.get(name); + if (!longName) throw new ArgError(`unrecognized argument: ${name}`); + const option = spec[longName]; + if (option.type === "boolean") { + if (inlineValue !== undefined) { + throw new ArgError(`${name} does not take a value`); + } + flags[longName] = true; + } else { + const value = inlineValue !== undefined ? inlineValue : argv[index + 1]; + if (value === undefined || (inlineValue === undefined && value.startsWith("-") && value !== "-")) { + throw new ArgError(`${name} requires a value`); + } + if (inlineValue === undefined) index += 1; + if (option.multiple) flags[longName].push(value); + else flags[longName] = value; + } + index += 1; + continue; + } + positionals.push(arg); + index += 1; + } + + return { flags, positionals }; +} + +/** True when the argument list asks for help. */ +export function wantsHelp(argv) { + return argv.includes("--help") || argv.includes("-h"); +} + +/** Render ` ` fragments for a usage line. */ +export function usageFragment(spec = {}) { + const parts = []; + for (const [longName, option] of Object.entries(spec)) { + if (option.hidden) continue; + const bare = option.alias ? `-${option.alias}, --${longName}` : `--${longName}`; + parts.push(option.type === "boolean" ? `[${bare}]` : `[${bare} <${option.value ?? "value"}>]`); + } + return parts.join(" "); +} diff --git a/src/cli/entry.mjs b/src/cli/entry.mjs new file mode 100644 index 00000000..24dff7bd --- /dev/null +++ b/src/cli/entry.mjs @@ -0,0 +1,33 @@ +/** + * "Am I the process entry point?" — the guard every executable here needs. + * + * `bin/distilly.mjs` and the two runnable scripts under `scripts/` must dispatch + * only when they were invoked directly, because tests import their internals. + * The obvious spelling is wrong in a way that fails *silently*: + * + * resolve(process.argv[1]) === fileURLToPath(import.meta.url) + * + * `import.meta.url` is always the realpath, while `argv[1]` is whatever the + * caller wrote. Under a symlink the two differ, the guard concludes "I was + * imported", and the program exits 0 having done nothing. That is not an edge + * case: `/tmp` is a symlink to `/private/tmp` on macOS, and — more importantly — + * an npm `bin` shim in `node_modules/.bin/` is a symlink, so a published + * `distilly` would have silently ignored every command. + * + * Resolving both sides is the fix. A path that cannot be resolved (an eval, a + * repl) is not an entry point, which is also the honest answer. + */ + +import { realpathSync } from "node:fs"; +import { fileURLToPath } from "node:url"; + +/** True when `moduleUrl` names the file the process was started with. */ +export function isEntryPoint(moduleUrl) { + const invoked = process.argv[1]; + if (invoked === undefined || invoked === "") return false; + try { + return realpathSync(invoked) === realpathSync(fileURLToPath(moduleUrl)); + } catch { + return false; + } +} diff --git a/src/cli/receipt.mjs b/src/cli/receipt.mjs new file mode 100644 index 00000000..6da2fada --- /dev/null +++ b/src/cli/receipt.mjs @@ -0,0 +1,110 @@ +/** + * CLI receipts, output routing and file fingerprints. + * + * The receipt shape is frozen by `docs/v2/CONTRACT.md` §3: + * `{command, person, ok, inputs, outputs, anchors, warnings, unavailable}`. + * Every entry in `inputs`/`outputs` carries `{path, sha256, bytes}`. + * + * `--json` writes the receipt to stdout as the only stdout content (human text + * moves to stderr) so `JSON.parse(stdout)` always succeeds. + */ + +import { createHash } from "node:crypto"; +import { readFileSync, statSync } from "node:fs"; +import { relative, resolve, sep } from "node:path"; + +/** User-facing failure with an explicit remedy. */ +export class CliError extends Error { + constructor(message, { code = "error", remedy = "", exitCode = 1 } = {}) { + super(message); + this.name = "CliError"; + this.code = code; + this.remedy = remedy; + this.exitCode = exitCode; + } +} + +export function sha256Buffer(buffer) { + return createHash("sha256").update(buffer).digest("hex"); +} + +export function sha256Text(text) { + return sha256Buffer(Buffer.from(text, "utf8")); +} + +/** `{path, sha256, bytes}` for a file, or null when it is missing. */ +export function describeFile(filePath, { cwd = process.cwd() } = {}) { + let buffer; + try { + buffer = readFileSync(filePath); + } catch { + return null; + } + return { + path: displayPath(filePath, cwd), + sha256: sha256Buffer(buffer), + bytes: buffer.length, + }; +} + +/** `{path, bytes}` without hashing (used for directories and removals). */ +export function describePath(filePath, { cwd = process.cwd() } = {}) { + return { path: displayPath(filePath, cwd), bytes: directoryBytes(filePath) }; +} + +export function directoryBytes(dirPath) { + try { + return statSync(dirPath).size; + } catch { + return 0; + } +} + +/** Paths inside the working directory are reported relatively, like the old CLI. */ +export function displayPath(filePath, cwd = process.cwd()) { + const absolute = resolve(filePath); + const rel = relative(resolve(cwd), absolute); + if (rel === "") return "."; + if (!rel.startsWith("..") && !rel.startsWith(`${sep}..`)) return rel; + return absolute; +} + +/** Build the contract receipt object (field order is contractual). */ +export function createReceipt(command, options = {}) { + const receipt = { + command, + person: options.person ?? null, + ok: options.ok ?? true, + inputs: options.inputs ?? [], + outputs: options.outputs ?? [], + anchors: options.anchors ?? { total: 0, cited: 0 }, + warnings: options.warnings ?? [], + unavailable: options.unavailable ?? [], + }; + if (options.error) receipt.error = options.error; + return receipt; +} + +/** + * Route human text and machine receipts. + * In `--json` mode stdout carries exactly one JSON object; prose goes to stderr. + */ +export function createReporter(json, { stdout = process.stdout, stderr = process.stderr } = {}) { + const lines = []; + return { + json, + line(text) { + lines.push(text); + if (json) stderr.write(`${text}\n`); + else stdout.write(`${text}\n`); + }, + warn(text) { + stderr.write(`${text}\n`); + }, + /** Write the receipt last so `JSON.parse(stdout)` sees a single object. */ + finish(receipt) { + if (!json) return; + stdout.write(`${JSON.stringify(receipt, null, 2)}\n`); + }, + }; +} diff --git a/src/collect/dingtalk.mjs b/src/collect/dingtalk.mjs new file mode 100644 index 00000000..34ecb24c --- /dev/null +++ b/src/collect/dingtalk.mjs @@ -0,0 +1,832 @@ +/** + * dingtalk.mjs — credentialed collection from DingTalk. + * + * Legacy ported: `tools/dingtalk_auto_collector.py` (790 lines). What that file + * actually proves about DingTalk's API surface matters here, so this module + * splits the channel the same way the legacy did: + * + * - **api mode** — read-only endpoints this repository has evidence for: the + * app credential exchange (`POST /v1.0/oauth2/accessToken`) and the contact + * directory card lookup (`POST /v1.0/contact/users/search`, + * `GET /v1.0/contact/users/{userId}`). Both are used by the legacy collector + * (`tools/dingtalk_auto_collector.py:106`, `:151`, `:202`). + * - **message history** — the legacy collector has **no API path** for it; it + * drives a browser instead (`tools/dingtalk_auto_collector.py:518`, + * "消息类（可选，仅用于发消息，历史消息需浏览器方案）"). Rather than invent an + * endpoint, api mode says so, loudly, and browser mode requires a consent + * token. The host performs the computer use and hands the bytes back through + * `--capture `; this module never drives a browser itself. + * + * Same discipline as the other collectors: credential from + * `~/.distilly/dingtalk_config.json` (or env), file name only in messages, raw + * bytes verbatim under `knowledge/raw/dingtalk/`, ledger upsert, loud failures + * with remediation, and a hard read-only guarantee. + */ + +import { createHash } from "node:crypto"; +import { + existsSync, + mkdirSync, + readFileSync, + renameSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { homedir } from "node:os"; +import { dirname, join, resolve } from "node:path"; + +import { consentTokenFingerprint, verify as verifyConsent } from "../consent.mjs"; + +export const CHANNEL = "dingtalk"; +export const CONFIG_FILE = "dingtalk_config.json"; +export const LEGACY_CONFIG_FILE = join(".colleague-skill", CONFIG_FILE); +export const DEFAULT_BASE_URL = "https://api.dingtalk.com"; +export const DEFAULT_LIMIT = 10; +export const DEFAULT_MAX_PAGES = 10; +export const DEFAULT_MAX_RETRIES = 4; +export const DEFAULT_MAX_BACKOFF_MS = 60_000; + +export const ENV_KEYS = { + appKey: ["DISTILLY_DINGTALK_APP_KEY", "DINGTALK_APP_KEY"], + appSecret: ["DISTILLY_DINGTALK_APP_SECRET", "DINGTALK_APP_SECRET"], +}; + +/** + * Both entries are query-shaped POSTs: neither creates, updates or deletes + * anything a user can see. Anything else non-GET is refused before it is sent. + */ +export const ALLOWED_MUTATIONS = [ + { method: "POST", path: "/v1.0/oauth2/accessToken", why: "app token exchange; no user-visible state" }, + { method: "POST", path: "/v1.0/contact/users/search", why: "read-only directory search expressed as POST" }, +]; + +const MUTATING_METHODS = new Set(["POST", "PUT", "PATCH", "DELETE"]); +const REMEDIATION_SETUP = [ + `create ~/.distilly/${CONFIG_FILE} (chmod 600) with {"app_key": "ding…", "app_secret": "…"}`, + " app: https://open-dev.dingtalk.com → 企业内部应用 → 权限 Contact.User.Read / qyapi_get_member_detail", + "or export DISTILLY_DINGTALK_APP_KEY and DISTILLY_DINGTALK_APP_SECRET for this shell only", +]; + +/** What is missing on the API side, quoted from the legacy collector's own note. */ +export const MESSAGE_API_GAP = + "DingTalk exposes no documented read API for message history; the legacy collector used a browser " + + "(tools/dingtalk_auto_collector.py:518 — “历史消息需浏览器方案”)"; + +export class CollectFailure extends Error { + constructor(reason, message, { remediation = [], exitCode = 1, kind = "failure" } = {}) { + super(message); + this.name = "CollectFailure"; + this.reason = reason; + this.remediation = remediation; + this.exitCode = exitCode; + this.kind = kind; + } +} + +export function redact(text, secrets = []) { + let output = String(text ?? ""); + for (const secret of secrets) { + if (typeof secret !== "string" || secret.length < 4) continue; + output = output.split(secret).join("[redacted]"); + } + return output; +} + +export function scrub(value, secrets = []) { + return JSON.parse( + JSON.stringify(value, (_key, item) => (typeof item === "string" ? redact(item, secrets) : item)), + ); +} + +export function distillyHome(env = process.env) { + const override = env?.DISTILLY_HOME; + return override ? resolve(String(override)) : join(homedir(), ".distilly"); +} + +export function credentialPaths(env = process.env) { + return { + primary: join(distillyHome(env), CONFIG_FILE), + legacy: join(homedir(), LEGACY_CONFIG_FILE), + }; +} + +export function loadCredential({ env = process.env, readFile = readFileSync } = {}) { + const pick = (names) => { + for (const name of names) { + const value = env?.[name]; + if (typeof value === "string" && value.trim() !== "") return value.trim(); + } + return null; + }; + const envKey = pick(ENV_KEYS.appKey); + const envSecret = pick(ENV_KEYS.appSecret); + if (envKey && envSecret) { + return { + ok: true, + source: "env", + configFile: CONFIG_FILE, + path: null, + values: { app_key: envKey, app_secret: envSecret }, + }; + } + + const { primary, legacy } = credentialPaths(env); + for (const [path, source] of [ + [primary, "config"], + [legacy, "legacy-config"], + ]) { + if (!existsSync(path)) continue; + let parsed; + try { + parsed = JSON.parse(readFile(path, "utf8")); + } catch (error) { + throw new CollectFailure( + "bad-credential-file", + `${CONFIG_FILE} is not valid JSON (${redact(error.message)}); rewrite it with {"app_key": "ding…", "app_secret": "…"}`, + { remediation: REMEDIATION_SETUP }, + ); + } + const appKey = parsed.app_key ?? parsed.appKey ?? null; + const appSecret = parsed.app_secret ?? parsed.appSecret ?? null; + if (!appKey || !appSecret) { + throw new CollectFailure("incomplete-credential", `${CONFIG_FILE} is missing app_key / app_secret`, { + remediation: REMEDIATION_SETUP, + }); + } + return { ok: true, source, configFile: CONFIG_FILE, path, values: { app_key: appKey, app_secret: appSecret } }; + } + + throw new CollectFailure("no-credential", `no credential at ~/.distilly/${CONFIG_FILE}`, { + remediation: REMEDIATION_SETUP, + }); +} + +export function assertReadOnly(url, method = "GET") { + const verb = String(method).toUpperCase(); + if (!MUTATING_METHODS.has(verb)) return true; + const path = (() => { + try { + return new URL(url).pathname; + } catch { + return String(url); + } + })(); + const allowed = ALLOWED_MUTATIONS.some((entry) => entry.method === verb && path.endsWith(entry.path)); + if (!allowed) { + throw new CollectFailure( + "write-operation-refused", + `refusing ${verb} ${path}: this collector never writes to ${CHANNEL}`, + { remediation: ["collectors are read-only; remove the mutating call instead of allowlisting it"] }, + ); + } + return true; +} + +export function parseRetryAfter(headerValue, nowMs = Date.now()) { + if (headerValue === null || headerValue === undefined || headerValue === "") return null; + const seconds = Number(headerValue); + if (Number.isFinite(seconds) && seconds >= 0) return Math.round(seconds * 1000); + const date = Date.parse(String(headerValue)); + if (Number.isFinite(date)) return Math.max(0, date - nowMs); + return null; +} + +export function backoffDelay(attempt, retryAfterMs = null, options = {}) { + const { baseMs = 500, maxMs = DEFAULT_MAX_BACKOFF_MS } = options; + if (Number.isFinite(retryAfterMs) && retryAfterMs !== null) return Math.min(retryAfterMs, maxMs); + return Math.min(baseMs * 2 ** Math.max(0, attempt - 1), maxMs); +} + +export function defaultSleep(ms) { + return new Promise((resolvePromise) => setTimeout(resolvePromise, ms)); +} + +export async function requestJson(options) { + const { + fetchImpl, + url, + method = "GET", + headers = {}, + body, + maxRetries = DEFAULT_MAX_RETRIES, + sleep = defaultSleep, + secrets = [], + onRetry = () => {}, + authRemediation = REMEDIATION_SETUP, + } = options; + + assertReadOnly(url, method); + if (typeof fetchImpl !== "function") { + throw new CollectFailure("no-fetch", "no fetch implementation available", { + remediation: ["run on Node >= 20, or pass an injected fetch"], + }); + } + + let attempt = 0; + for (;;) { + attempt += 1; + let response; + try { + response = await fetchImpl(url, { + method, + headers, + body: body === undefined ? undefined : JSON.stringify(body), + }); + } catch (error) { + const message = redact(error?.message ?? String(error), secrets); + if (attempt > maxRetries) { + throw new CollectFailure("network-error", `request failed: ${message}`, { + remediation: ["check the network/proxy and retry", ...authRemediation], + }); + } + await sleep(backoffDelay(attempt)); + onRetry({ attempt, status: null, delayMs: backoffDelay(attempt), reason: message }); + continue; + } + + const status = Number(response?.status ?? 0); + const retryAfterMs = parseRetryAfter(response?.headers?.get?.("retry-after") ?? null); + + if (status === 429 || status >= 500) { + if (attempt > maxRetries) { + throw new CollectFailure( + status === 429 ? "rate-limited" : "server-error", + status === 429 + ? `rate limited (HTTP 429) after ${maxRetries} retries` + : `server error (HTTP ${status}) after ${maxRetries} retries`, + { + remediation: [ + "retry later; already-fetched pages stay on disk and the cursor is checkpointed", + `lower --limit to stay under the ${CHANNEL} quota`, + ], + }, + ); + } + const delayMs = backoffDelay(attempt, retryAfterMs); + onRetry({ attempt, status, delayMs, reason: `HTTP ${status}` }); + await sleep(delayMs); + continue; + } + + const text = await response.text(); + if (status === 401 || status === 403) { + throw new CollectFailure("unauthorized", `HTTP ${status} from ${CHANNEL}; app credential rejected`, { + remediation: [ + `the credential in ~/.distilly/${CONFIG_FILE} was rejected — regenerate AppKey/AppSecret`, + "check the app scopes at https://open-dev.dingtalk.com", + ], + }); + } + if (status >= 400) { + throw new CollectFailure("http-error", `HTTP ${status}: ${redact(text.slice(0, 200), secrets)}`, { + remediation: ["check the request parameters", ...authRemediation], + }); + } + + let json = null; + try { + json = JSON.parse(text); + } catch { + throw new CollectFailure("invalid-json", `${CHANNEL} returned a non-JSON body`, { + remediation: ["retry later; if it persists the endpoint may have changed"], + }); + } + return { status, text, json, attempts: attempt }; + } +} + +export function sha256Hex(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +export function slug(text, fallback = "target") { + const slugged = String(text ?? "") + .normalize("NFKD") + .toLowerCase() + .replace(/[^a-z0-9._-]+/g, "-") + .replace(/-+/g, "-") + .replace(/^[-._]+|[-._]+$/g, "") + .slice(0, 64); + return slugged || fallback; +} + +export function knowledgeRoot({ root = process.cwd(), person, family = "colleague" } = {}) { + return person ? join(resolve(root), "skills", slug(family), slug(person), "knowledge") : join(resolve(root), "knowledge"); +} + +export function writeRaw(knowledgeDir, name, bytes) { + const dir = join(knowledgeDir, "raw", CHANNEL); + mkdirSync(dir, { recursive: true }); + const path = join(dir, `${slug(name, "page")}.json`); + const staging = `${path}.${process.pid}.tmp`; + const buffer = Buffer.from(bytes); + try { + writeFileSync(staging, buffer); + renameSync(staging, path); + } catch (error) { + if (existsSync(staging)) rmSync(staging, { force: true }); + throw error; + } + return { + path, + relativePath: `raw/${CHANNEL}/${slug(name, "page")}.json`, + bytes: buffer.length, + sha256: sha256Hex(buffer), + }; +} + +export function appendLedger(knowledgeDir, entries) { + if (entries.length === 0) return { path: join(knowledgeDir, "index.json"), added: 0, total: 0, existed: false }; + const path = join(knowledgeDir, "index.json"); + let existing = []; + if (existsSync(path)) { + try { + const parsed = JSON.parse(readFileSync(path, "utf8")); + existing = Array.isArray(parsed) ? parsed : []; + } catch (error) { + throw new CollectFailure("bad-ledger", `knowledge/index.json is not valid JSON: ${redact(error.message)}`, { + remediation: ["repair or remove knowledge/index.json, then rerun the collect"], + }); + } + } + const byId = new Map(existing.filter((e) => e && typeof e === "object").map((e) => [e.id, e])); + let added = 0; + for (const entry of entries) { + if (!byId.has(entry.id)) added += 1; + byId.set(entry.id, entry); + } + const merged = [...byId.values()].sort((a, b) => String(a.id).localeCompare(String(b.id))); + const staging = `${path}.${process.pid}.tmp`; + try { + writeFileSync(staging, `${JSON.stringify(merged, null, 2)}\n`); + renameSync(staging, path); + } catch (error) { + if (existsSync(staging)) rmSync(staging, { force: true }); + throw error; + } + return { path, added, total: merged.length, existed: existing.length > 0 }; +} + +/** Issues an app access token. Returns `{token, expiresIn}` — never logged. */ +export async function fetchAppToken({ fetchImpl, env = process.env, credential, baseUrl, maxRetries, sleep, secrets }) { + const response = await requestJson({ + fetchImpl, + url: `${baseUrl}/v1.0/oauth2/accessToken`, + method: "POST", + headers: { "content-type": "application/json" }, + body: { appKey: credential.values.app_key, appSecret: credential.values.app_secret }, + maxRetries, + sleep, + secrets, + }); + const token = response.json?.accessToken; + if (!token) { + throw new CollectFailure("auth-failed", "DingTalk rejected the app credential (no accessToken in the response)", { + remediation: REMEDIATION_SETUP, + }); + } + return { token, expiresIn: Number(response.json?.expireIn ?? 7200) }; +} + +/** + * api mode: one directory-card lookup for `name`, plus the profile detail of the + * single match. No pagination is claimed — the legacy collector only ever called + * `/v1.0/contact/users/search` with `offset: 0` and never looped, so the + * continuation parameter is left untested and documented as a known gap. + */ +export async function collectDirectory(options = {}) { + const { + fetch: fetchImpl = globalThis.fetch, + env = process.env, + root = process.cwd(), + person, + family = "colleague", + name, + limit = DEFAULT_LIMIT, + maxRetries = DEFAULT_MAX_RETRIES, + sleep = defaultSleep, + now = new Date().toISOString(), + baseUrl = env?.DISTILLY_DINGTALK_BASE_URL || DEFAULT_BASE_URL, + onProgress = () => {}, + } = options; + + const knowledgeDir = knowledgeRoot({ root, person, family }); + const outputs = []; + const warnings = []; + const ledgerEntries = []; + let secrets = []; + let credential = null; + let requests = 0; + let users = 0; + + const base = { + command: "collect", + channel: CHANNEL, + mode: "api", + resource: "directory-card", + ok: false, + inputs: [], + outputs, + warnings, + unavailable: [], + credential_file: CONFIG_FILE, + }; + const fail = (failure) => ({ + ok: false, + exitCode: failure.exitCode ?? 1, + receipt: scrub( + { + ...base, + ok: false, + person: person ?? null, + name: name ?? null, + requests, + errors: [redact(failure.message, secrets)], + unavailable: [ + { + channel: CHANNEL, + reason: redact(`${failure.reason}: ${failure.message}`, secrets), + remediation: failure.remediation ?? [], + }, + ], + }, + secrets, + ), + }); + + try { + if (!name) { + throw new CollectFailure("missing-target", "collect dingtalk needs --name ", { + remediation: [ + "api mode reads the contact directory card; message history has no API — see --mode browser", + MESSAGE_API_GAP, + ], + }); + } + credential = loadCredential({ env }); + secrets = [credential.values.app_secret]; + base.credential_source = credential.source; + + const size = Math.min(Math.max(1, Number(limit) || DEFAULT_LIMIT), 50); + const { token } = await fetchAppToken({ fetchImpl, env, credential, baseUrl, maxRetries, sleep, secrets }); + secrets = [...secrets, token]; + requests += 1; + + const search = await requestJson({ + fetchImpl, + url: `${baseUrl}/v1.0/contact/users/search`, + method: "POST", + headers: { "content-type": "application/json", "x-acs-dingtalk-access-token": token }, + body: { searchText: name, offset: 0, size }, + maxRetries, + sleep, + secrets, + }); + requests += 1; + + const list = Array.isArray(search.json?.list) ? search.json.list : []; + users = list.length; + const stored = writeRaw(knowledgeDir, `${name}-search`, search.text); + outputs.push({ path: stored.path, sha256: stored.sha256, bytes: stored.bytes, kind: "raw" }); + ledgerEntries.push({ + id: `${CHANNEL}:${slug(name)}:search`, + kind: "raw", + origin: stored.relativePath, + source: CHANNEL, + fetched_at: now, + bytes: stored.bytes, + sha256: stored.sha256, + credentialed: true, + credential_source: credential.source, + credential_file: CONFIG_FILE, + method: "api-app-token", + items: list.length, + warnings: [], + }); + if (list.length >= size) { + warnings.push( + `search returned a full page (${size}); the continuation parameter is not verified against a live tenant (known gap)`, + ); + } + + if (list.length === 1) { + const userId = list[0]?.userId ?? list[0]?.unionId; + if (userId) { + const detail = await requestJson({ + fetchImpl, + url: `${baseUrl}/v1.0/contact/users/${encodeURIComponent(userId)}`, + headers: { "x-acs-dingtalk-access-token": token }, + maxRetries, + sleep, + secrets, + }); + requests += 1; + const detailStored = writeRaw(knowledgeDir, `${name}-profile`, detail.text); + outputs.push({ path: detailStored.path, sha256: detailStored.sha256, bytes: detailStored.bytes, kind: "raw" }); + ledgerEntries.push({ + id: `${CHANNEL}:${slug(name)}:profile`, + kind: "raw", + origin: detailStored.relativePath, + source: CHANNEL, + fetched_at: now, + bytes: detailStored.bytes, + sha256: detailStored.sha256, + credentialed: true, + credential_source: credential.source, + credential_file: CONFIG_FILE, + method: "api-app-token", + items: 1, + warnings: [], + }); + } + } + + const ledger = appendLedger(knowledgeDir, ledgerEntries); + onProgress(`directory: ${list.length} match(es)`); + warnings.push(MESSAGE_API_GAP); + warnings.push( + "message history: run `distilly collect dingtalk --mode browser --consent ` and let the host capture it", + ); + + return { + ok: true, + exitCode: 0, + receipt: scrub( + { + ...base, + ok: true, + person: person ?? null, + name, + requests, + items: users, + outputs, + ledger: { path: ledger.path, added: ledger.added, total: ledger.total }, + unavailable: [], + }, + secrets, + ), + }; + } catch (error) { + if (!(error instanceof CollectFailure)) { + throw new CollectFailure("unexpected", redact(error?.message ?? String(error), secrets), { + remediation: ["rerun with --json and report the receipt"], + }); + } + const result = fail(error); + if (ledgerEntries.length > 0) { + try { + const ledger = appendLedger(knowledgeDir, ledgerEntries); + result.receipt.ledger = { path: ledger.path, added: ledger.added, total: ledger.total }; + result.receipt.partial = true; + } catch { + result.receipt.warnings.push("could not register the partial pages in knowledge/index.json"); + } + } + return result; + } +} + +/** + * browser mode: the consent gate plus capture registration. This module never + * launches a browser — the host does the computer use and writes the captured + * bytes with `--capture `; we verify the grant, store the bytes verbatim + * and register them in the ledger with their provenance. + */ +export function registerCapture(options = {}) { + const { + env = process.env, + root = process.cwd(), + person, + family = "colleague", + scope = `collect:${CHANNEL}:browser`, + consentToken, + capturePath, + label, + producer = "host:computer-use", + now = new Date().toISOString(), + readFile = readFileSync, + } = options; + + const knowledgeDir = knowledgeRoot({ root, person, family }); + const base = { + command: "collect", + channel: CHANNEL, + mode: "browser", + resource: "messages", + ok: false, + inputs: [], + outputs: [], + warnings: [], + unavailable: [], + credential_file: null, + }; + + const verification = verifyConsent(consentToken, { env, scope }); + if (!verification.ok) { + return { + ok: false, + exitCode: 2, + receipt: { + ...base, + ok: false, + status: "waiting-for-user-consent", + person: person ?? null, + errors: [`waiting for user consent (${verification.reason})`], + unavailable: [ + { + channel: CHANNEL, + reason: `waiting for user consent: ${verification.reason}`, + scope, + remediation: verification.remediation, + }, + ], + }, + }; + } + + if (!capturePath) { + return { + ok: true, + exitCode: 0, + receipt: { + ...base, + ok: true, + status: "awaiting-host-capture", + person: person ?? null, + consent: { + scope: verification.record.scope, + granted_at: verification.record.granted_at, + expires_at: verification.record.expires_at, + token_sha256_12: consentTokenFingerprint(consentToken), + }, + host_steps: [ + "Host (computer use): open the DingTalk client, open the target conversation, load the requested range.", + "Host: write the captured messages to a file, verbatim, and rerun with --capture .", + "This tool does not click, scroll, send or react — consent only authorises the host's capture.", + MESSAGE_API_GAP, + ], + unavailable: [], + }, + }; + } + + let bytes; + try { + bytes = readFile(capturePath); + } catch (error) { + return { + ok: false, + exitCode: 1, + receipt: { + ...base, + ok: false, + person: person ?? null, + errors: [`cannot read --capture ${capturePath}: ${redact(error.message)}`], + unavailable: [ + { + channel: CHANNEL, + reason: `capture-unreadable: ${redact(error.message)}`, + remediation: ["point --capture at a readable file produced by the host"], + }, + ], + }, + }; + } + + const stored = writeRaw(knowledgeDir, label ?? `browser-${new Date(now).toISOString().slice(0, 10)}`, bytes); + const entry = { + id: `${CHANNEL}:${slug(label ?? "browser")}:capture`, + kind: "raw", + origin: stored.relativePath, + source: CHANNEL, + fetched_at: now, + bytes: stored.bytes, + sha256: stored.sha256, + credentialed: false, + method: "browser-host", + provenance: { method: "browser-host", producer, confidence: "host-reported" }, + consent: { + scope: verification.record.scope, + granted_at: verification.record.granted_at, + expires_at: verification.record.expires_at, + token_sha256_12: consentTokenFingerprint(consentToken), + }, + warnings: [], + }; + const ledger = appendLedger(knowledgeDir, [entry]); + + return { + ok: true, + exitCode: 0, + receipt: scrub( + { + ...base, + ok: true, + status: "captured", + person: person ?? null, + outputs: [{ path: stored.path, sha256: stored.sha256, bytes: stored.bytes, kind: "raw" }], + ledger: { path: ledger.path, added: ledger.added, total: ledger.total }, + provenance: entry.provenance, + unavailable: [], + }, + [], + ), + }; +} + +export async function collect(options = {}) { + return options.mode === "browser" ? registerCapture(options) : collectDirectory(options); +} + +export const HELP = `distilly collect dingtalk — 钉钉采集 / DingTalk collection + +用法 (zh): + distilly collect dingtalk --name <姓名> [--person ] [--limit 10] [--json] + 只读的通讯录名片采集（企业应用凭据）。 + distilly collect dingtalk --mode browser --consent [--capture ] [--json] + 消息历史没有公开读接口（见 tools/dingtalk_auto_collector.py:518）；浏览器采集由宿主完成， + 本工具只做同意门 + 原样落盘 + 账本登记。无 token / 已过期 → exit 2（等待用户同意）。 + +凭据：~/.distilly/${CONFIG_FILE}（app_key / app_secret，0600）或 + DISTILLY_DINGTALK_APP_KEY / DISTILLY_DINGTALK_APP_SECRET。错误信息只出现配置文件名。 +只读：仅允许 POST /v1.0/oauth2/accessToken 与 POST /v1.0/contact/users/search（都是查询语义）， + 其余非 GET 请求在 assertReadOnly() 里被拒绝。 + +--- +## English + distilly collect dingtalk --name [--json] + read-only directory card via the app credential. + distilly collect dingtalk --mode browser --consent [--capture ] [--json] + message history has no public read API; the host captures it, this tool only gates + consent, stores the bytes verbatim and registers them in the ledger. No/expired token → exit 2. +`; + +export function parseFlags(argv) { + const flags = { _: [] }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (!arg.startsWith("--")) { + flags._.push(arg); + continue; + } + const name = arg.slice(2); + if (name === "json" || name === "help") { + flags[name] = true; + continue; + } + const value = argv[index + 1]; + if (value === undefined || value.startsWith("--")) throw new Error(`${arg} requires a value`); + flags[name] = value; + index += 1; + } + return flags; +} + +export async function runCollectCli(argv, io = {}) { + const out = io.stdout ?? ((line) => process.stdout.write(`${line}\n`)); + const err = io.stderr ?? ((line) => process.stderr.write(`${line}\n`)); + + let flags; + try { + flags = parseFlags(argv); + } catch (error) { + err(`Error: ${error.message}`); + return 1; + } + if (flags.help) { + out(HELP); + return 0; + } + + const result = await collect({ + fetch: io.fetch ?? globalThis.fetch, + env: io.env ?? process.env, + root: flags.root ?? process.cwd(), + person: flags.person, + family: flags.family, + mode: flags.mode ?? "api", + name: flags.name, + limit: flags.limit ? Number(flags.limit) : undefined, + maxRetries: flags["max-retries"] ? Number(flags["max-retries"]) : undefined, + consentToken: flags.consent, + capturePath: flags.capture, + label: flags.label, + sleep: io.sleep, + now: io.now, + }); + + if (flags.json) out(JSON.stringify(result.receipt, null, 2)); + if (result.ok) { + out(`ok (${result.receipt.status ?? "collected"}) → ${result.receipt.outputs.length} raw file(s)`); + for (const warning of result.receipt.warnings ?? []) err(`warning: ${warning}`); + for (const step of result.receipt.host_steps ?? []) err(`host: ${step}`); + } else { + err(`Error: ${result.receipt.errors?.[0] ?? "collect failed"}`); + for (const entry of result.receipt.unavailable ?? []) { + err(`unavailable: ${entry.channel} — ${entry.reason}`); + for (const step of entry.remediation ?? []) err(` fix: ${step}`); + } + for (const warning of result.receipt.warnings ?? []) err(`warning: ${warning}`); + } + return result.exitCode; +} diff --git a/src/collect/feishu.mjs b/src/collect/feishu.mjs new file mode 100644 index 00000000..01db43b8 --- /dev/null +++ b/src/collect/feishu.mjs @@ -0,0 +1,801 @@ +/** + * feishu.mjs — credentialed collection from Feishu / Lark open APIs. + * + * Legacies ported: `tools/feishu_auto_collector.py` (960 lines, SDK `requests`). + * This module keeps the same credential location and the same two read paths + * (tenant token for group chats, user token for p2p), but: + * + * - the network goes through an **injected `fetch`** (default `globalThis.fetch`), + * so the whole channel is testable with a mock and never needs a live tenant; + * - raw response bytes are stored **verbatim** in `knowledge/raw/feishu/…` and + * registered in `knowledge/index.json` (contract §2); + * - pagination is cursor-driven, rate limits back off with `Retry-After`, and an + * interrupted run leaves a checkpoint so the next run resumes at the cursor + * instead of re-fetching finished pages; + * - credential values never reach stdout, stderr or a receipt — only the file + * name (`feishu_config.json`) is ever printed. See `redact()` / `scrub()`. + * + * Read-only by construction: every request goes through `assertReadOnly()`, which + * rejects any method+path pair outside `ALLOWED_MUTATIONS` (the tenant token + * exchange — a POST that changes no user-visible state). There is no code path + * that can like, follow, post or send anything; that is a capability-level + * guarantee, not a prompt instruction. + */ + +import { createHash } from "node:crypto"; +import { + existsSync, + mkdirSync, + readFileSync, + renameSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { homedir } from "node:os"; +import { dirname, join, resolve } from "node:path"; + +export const CHANNEL = "feishu"; +/** Only the *name* may ever be printed. Never its contents. */ +export const CONFIG_FILE = "feishu_config.json"; +export const LEGACY_CONFIG_FILE = join(".colleague-skill", CONFIG_FILE); +export const DEFAULT_BASE_URL = "https://open.feishu.cn/open-apis"; +export const DEFAULT_PAGE_SIZE = 50; +export const DEFAULT_MAX_PAGES = 10; +export const DEFAULT_MAX_RETRIES = 4; +export const DEFAULT_MAX_BACKOFF_MS = 60_000; + +/** Env fallbacks, checked before the config file. Names are not secret. */ +export const ENV_KEYS = { + appId: ["DISTILLY_FEISHU_APP_ID", "FEISHU_APP_ID"], + appSecret: ["DISTILLY_FEISHU_APP_SECRET", "FEISHU_APP_SECRET"], + userToken: ["DISTILLY_FEISHU_USER_ACCESS_TOKEN", "FEISHU_USER_ACCESS_TOKEN"], +}; + +/** + * The only non-GET calls this module may make. Anything else fails loudly + * before the request leaves the process. + */ +export const ALLOWED_MUTATIONS = [ + { + method: "POST", + path: "/auth/v3/tenant_access_token/internal", + why: "tenant token exchange; creates no user-visible state", + }, +]; + +const MUTATING_METHODS = new Set(["POST", "PUT", "PATCH", "DELETE"]); +const REMEDIATION_SETUP = [ + `create ~/.distilly/${CONFIG_FILE} (chmod 600) with {"app_id": "cli_…", "app_secret": "…"}`, + " app: https://open.feishu.cn → 开发者后台 → 创建企业自建应用 → 权限 im:message:readonly, im:chat:readonly", + "or export DISTILLY_FEISHU_APP_ID and DISTILLY_FEISHU_APP_SECRET for this shell only", + "private (p2p) chats additionally need a user token: DISTILLY_FEISHU_USER_ACCESS_TOKEN", +]; + +// ─── failures ──────────────────────────────────────────────────────────────── + +/** An expected, loud failure. `reason` is a stable machine-readable token. */ +export class CollectFailure extends Error { + constructor(reason, message, { remediation = [], exitCode = 1, kind = "failure" } = {}) { + super(message); + this.name = "CollectFailure"; + this.reason = reason; + this.remediation = remediation; + this.exitCode = exitCode; + this.kind = kind; + } +} + +// ─── secret discipline ─────────────────────────────────────────────────────── + +/** Replace every credential value with `[redacted]`; keep file names visible. */ +export function redact(text, secrets = []) { + let output = String(text ?? ""); + for (const secret of secrets) { + if (typeof secret !== "string" || secret.length < 4) continue; + output = output.split(secret).join("[redacted]"); + } + return output; +} + +/** Mechanical net: no secret can survive a round-trip through a receipt. */ +export function scrub(value, secrets = []) { + return JSON.parse( + JSON.stringify(value, (_key, item) => (typeof item === "string" ? redact(item, secrets) : item)), + ); +} + +export function distillyHome(env = process.env) { + const override = env?.DISTILLY_HOME; + return override ? resolve(String(override)) : join(homedir(), ".distilly"); +} + +export function credentialPaths(env = process.env) { + return { + primary: join(distillyHome(env), CONFIG_FILE), + legacy: join(homedir(), LEGACY_CONFIG_FILE), + }; +} + +/** + * Load `{app_id, app_secret, user_access_token?}` from env or the config file. + * Returns values for the caller only; every message names the *file*, never a + * value, and the config file is never printed or copied anywhere else. + */ +export function loadCredential({ env = process.env, readFile = readFileSync } = {}) { + const pick = (names) => { + for (const name of names) { + const value = env?.[name]; + if (typeof value === "string" && value.trim() !== "") return value.trim(); + } + return null; + }; + + const envAppId = pick(ENV_KEYS.appId); + const envAppSecret = pick(ENV_KEYS.appSecret); + const envUserToken = pick(ENV_KEYS.userToken); + if (envAppId && envAppSecret) { + return { + ok: true, + source: "env", + configFile: CONFIG_FILE, + path: null, + values: { app_id: envAppId, app_secret: envAppSecret, user_access_token: envUserToken }, + }; + } + + const { primary, legacy } = credentialPaths(env); + for (const [path, source] of [ + [primary, "config"], + [legacy, "legacy-config"], + ]) { + if (!existsSync(path)) continue; + let parsed; + try { + parsed = JSON.parse(readFile(path, "utf8")); + } catch (error) { + throw new CollectFailure( + "bad-credential-file", + `${CONFIG_FILE} is not valid JSON (${redact(error.message)}); rewrite it with {"app_id": "cli_…", "app_secret": "…"}`, + { remediation: REMEDIATION_SETUP }, + ); + } + const appId = parsed.app_id ?? parsed.appId ?? null; + const appSecret = parsed.app_secret ?? parsed.appSecret ?? null; + const userToken = parsed.user_access_token ?? parsed.userToken ?? null; + if (!appId || !appSecret) { + throw new CollectFailure( + "incomplete-credential", + `${CONFIG_FILE} is missing app_id / app_secret`, + { remediation: REMEDIATION_SETUP }, + ); + } + return { + ok: true, + source, + configFile: CONFIG_FILE, + path, + values: { app_id: appId, app_secret: appSecret, user_access_token: userToken }, + }; + } + + throw new CollectFailure( + "no-credential", + `no credential at ~/.distilly/${CONFIG_FILE}`, + { remediation: REMEDIATION_SETUP }, + ); +} + +// ─── read-only guard + HTTP ────────────────────────────────────────────────── + +export function assertReadOnly(url, method = "GET") { + const verb = String(method).toUpperCase(); + if (!MUTATING_METHODS.has(verb)) return true; + const path = (() => { + try { + return new URL(url).pathname; + } catch { + return String(url); + } + })(); + const allowed = ALLOWED_MUTATIONS.some((entry) => entry.method === verb && path.endsWith(entry.path)); + if (!allowed) { + throw new CollectFailure( + "write-operation-refused", + `refusing ${verb} ${path}: this collector never writes to ${CHANNEL}`, + { remediation: ["collectors are read-only; remove the mutating call instead of allowlisting it"] }, + ); + } + return true; +} + +export function parseRetryAfter(headerValue, nowMs = Date.now()) { + if (headerValue === null || headerValue === undefined || headerValue === "") return null; + const seconds = Number(headerValue); + if (Number.isFinite(seconds) && seconds >= 0) return Math.round(seconds * 1000); + const date = Date.parse(String(headerValue)); + if (Number.isFinite(date)) return Math.max(0, date - nowMs); + return null; +} + +/** `Retry-After` wins; otherwise exponential backoff, capped. */ +export function backoffDelay(attempt, retryAfterMs = null, options = {}) { + const { baseMs = 500, maxMs = DEFAULT_MAX_BACKOFF_MS } = options; + if (Number.isFinite(retryAfterMs) && retryAfterMs !== null) return Math.min(retryAfterMs, maxMs); + return Math.min(baseMs * 2 ** Math.max(0, attempt - 1), maxMs); +} + +export function defaultSleep(ms) { + return new Promise((resolvePromise) => setTimeout(resolvePromise, ms)); +} + +/** + * One JSON request with retry/backoff. Never returns a body on 4xx/5xx. + * @returns {Promise<{status: number, text: string, json: any, attempts: number}>} + */ +export async function requestJson(options) { + const { + fetchImpl, + url, + method = "GET", + headers = {}, + body, + maxRetries = DEFAULT_MAX_RETRIES, + sleep = defaultSleep, + secrets = [], + onRetry = () => {}, + authRemediation = REMEDIATION_SETUP, + } = options; + + assertReadOnly(url, method); + if (typeof fetchImpl !== "function") { + throw new CollectFailure("no-fetch", "no fetch implementation available", { + remediation: ["run on Node >= 20, or pass an injected fetch"], + }); + } + + let attempt = 0; + for (;;) { + attempt += 1; + let response; + try { + response = await fetchImpl(url, { + method, + headers, + body: body === undefined ? undefined : JSON.stringify(body), + }); + } catch (error) { + const message = redact(error?.message ?? String(error), secrets); + if (attempt > maxRetries) { + throw new CollectFailure("network-error", `request failed: ${message}`, { + remediation: ["check the network/proxy and retry", ...authRemediation], + }); + } + await sleep(backoffDelay(attempt)); + onRetry({ attempt, status: null, delayMs: backoffDelay(attempt), reason: message }); + continue; + } + + const status = Number(response?.status ?? 0); + const retryAfterMs = parseRetryAfter(response?.headers?.get?.("retry-after") ?? null); + + if (status === 429 || status >= 500) { + if (attempt > maxRetries) { + throw new CollectFailure( + status === 429 ? "rate-limited" : "server-error", + status === 429 + ? `rate limited (HTTP 429) after ${maxRetries} retries` + : `server error (HTTP ${status}) after ${maxRetries} retries`, + { + remediation: [ + "retry later; already-fetched pages stay on disk and the cursor is checkpointed", + `lower --limit / --max-pages to stay under the ${CHANNEL} quota`, + ], + }, + ); + } + const delayMs = backoffDelay(attempt, retryAfterMs); + onRetry({ attempt, status, delayMs, reason: `HTTP ${status}` }); + await sleep(delayMs); + continue; + } + + const text = await response.text(); + if (status === 401 || status === 403) { + throw new CollectFailure("unauthorized", `HTTP ${status} from ${CHANNEL}; credential rejected or missing scope`, { + remediation: [ + "the credential in ~/.distilly/" + CONFIG_FILE + " was rejected — regenerate it", + "check the app scopes: im:message:readonly, im:chat:readonly (add im:message for p2p)", + ], + }); + } + if (status >= 400) { + throw new CollectFailure("http-error", `HTTP ${status}: ${redact(text.slice(0, 200), secrets)}`, { + remediation: ["check the request parameters", ...authRemediation], + }); + } + + let json = null; + try { + json = JSON.parse(text); + } catch { + throw new CollectFailure("invalid-json", `${CHANNEL} returned a non-JSON body`, { + remediation: ["retry later; if it persists the endpoint may have changed"], + }); + } + return { status, text, json, attempts: attempt }; + } +} + +// ─── sink: raw bytes + ledger ──────────────────────────────────────────────── + +export function sha256Hex(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +export function slug(text, fallback = "target") { + const slugged = String(text ?? "") + .normalize("NFKD") + .toLowerCase() + .replace(/[^a-z0-9._-]+/g, "-") + .replace(/-+/g, "-") + .replace(/^[-._]+|[-._]+$/g, "") + .slice(0, 64); + return slugged || fallback; +} + +/** `--person lin-gong` → `skills/colleague/lin-gong/knowledge`, else `/knowledge`. */ +export function knowledgeRoot({ root = process.cwd(), person, family = "colleague" } = {}) { + return person ? join(resolve(root), "skills", slug(family), slug(person), "knowledge") : join(resolve(root), "knowledge"); +} + +/** + * Write raw bytes verbatim, creating `knowledge/raw//` only now — a run + * that fails before its first successful page leaves no `knowledge/` behind. + */ +export function writeRaw(knowledgeDir, name, bytes) { + const dir = join(knowledgeDir, "raw", CHANNEL); + mkdirSync(dir, { recursive: true }); + const path = join(dir, `${slug(name, "page")}.json`); + const staging = `${path}.${process.pid}.tmp`; + const buffer = Buffer.from(bytes); + try { + writeFileSync(staging, buffer); + renameSync(staging, path); + } catch (error) { + if (existsSync(staging)) rmSync(staging, { force: true }); + throw error; + } + return { + path, + relativePath: `raw/${CHANNEL}/${slug(name, "page")}.json`, + bytes: buffer.length, + sha256: sha256Hex(buffer), + }; +} + +/** + * Upsert entries into `knowledge/index.json` (an array, contract §2). Entries + * with an identical `id` are replaced, which makes re-running a collect + * idempotent in the ledger. + */ +export function appendLedger(knowledgeDir, entries) { + if (entries.length === 0) return { path: join(knowledgeDir, "index.json"), added: 0, total: 0, existed: false }; + const path = join(knowledgeDir, "index.json"); + let existing = []; + if (existsSync(path)) { + try { + const parsed = JSON.parse(readFileSync(path, "utf8")); + existing = Array.isArray(parsed) ? parsed : []; + } catch (error) { + throw new CollectFailure("bad-ledger", `knowledge/index.json is not valid JSON: ${redact(error.message)}`, { + remediation: ["repair or remove knowledge/index.json, then rerun the collect"], + }); + } + } + const byId = new Map(existing.filter((e) => e && typeof e === "object").map((e) => [e.id, e])); + let added = 0; + for (const entry of entries) { + if (!byId.has(entry.id)) added += 1; + byId.set(entry.id, entry); + } + const merged = [...byId.values()].sort((a, b) => String(a.id).localeCompare(String(b.id))); + const staging = `${path}.${process.pid}.tmp`; + try { + writeFileSync(staging, `${JSON.stringify(merged, null, 2)}\n`); + renameSync(staging, path); + } catch (error) { + if (existsSync(staging)) rmSync(staging, { force: true }); + throw error; + } + return { path, added, total: merged.length, existed: existing.length > 0 }; +} + +// ─── resume checkpoints (outside the repo, next to the credential) ─────────── + +export function statePath({ env = process.env, root = process.cwd(), target } = {}) { + const key = sha256Hex(Buffer.from(`${resolve(root)}\n${target ?? ""}`, "utf8")).slice(0, 12); + return join(distillyHome(env), "state", `${CHANNEL}-${key}.json`); +} + +export function readCheckpoint(options) { + const path = statePath(options); + if (!existsSync(path)) return null; + try { + return JSON.parse(readFileSync(path, "utf8")); + } catch { + return null; + } +} + +export function writeCheckpoint(options, value) { + const path = statePath(options); + mkdirSync(dirname(path), { recursive: true }); + const staging = `${path}.${process.pid}.tmp`; + writeFileSync(staging, `${JSON.stringify(value, null, 2)}\n`); + renameSync(staging, path); + return path; +} + +export function clearCheckpoint(options) { + const path = statePath(options); + if (existsSync(path)) rmSync(path, { force: true }); + return path; +} + +// ─── collect ───────────────────────────────────────────────────────────────── + +export const BROWSER_STEPS = [ + "This module does not drive a browser: computer use is a host capability.", + `Host: open the Feishu client, open the target chat, scroll to the requested range.`, + "Host: hand the captured payload back to the user (copy the text into a file).", +]; + +/** + * Collect messages from one Feishu chat. + * + * @param {object} options + * @param {Function} [options.fetch] injected fetch (default `globalThis.fetch`) + * @param {object} [options.env] environment (default `process.env`) + * @param {string} [options.root] workspace root that contains `knowledge/` + * @param {string} [options.person] Skill slug; then the root is `skills///knowledge` + * @param {string} options.chatId `oc_…` chat container id (required) + * @param {number} [options.limit] page size + * @param {number} [options.maxPages] + * @param {number} [options.maxRetries] + * @param {string} [options.since] explicit cursor to resume from + * @param {boolean} [options.resume] read/write the checkpoint (default true) + * @param {Function} [options.sleep] injected sleeper so tests never really wait + * @param {string} [options.now] ISO timestamp used for `fetched_at` + */ +export async function collect(options = {}) { + const { + fetch: fetchImpl = globalThis.fetch, + env = process.env, + root = process.cwd(), + person, + family = "colleague", + chatId, + limit = DEFAULT_PAGE_SIZE, + maxPages = DEFAULT_MAX_PAGES, + maxRetries = DEFAULT_MAX_RETRIES, + since, + resume = true, + sleep = defaultSleep, + now = new Date().toISOString(), + baseUrl = env?.DISTILLY_FEISHU_BASE_URL || DEFAULT_BASE_URL, + useUserToken = Boolean(env?.DISTILLY_FEISHU_USE_USER_TOKEN), + onProgress = () => {}, + } = options; + + const knowledgeDir = knowledgeRoot({ root, person, family }); + const outputs = []; + const warnings = []; + const ledgerEntries = []; + const retries = []; + const onRetry = (info) => { + retries.push(info); + warnings.push(`retry ${info.attempt} after ${info.reason} (waited ${info.delayMs}ms)`); + onProgress(`retry ${info.attempt}: ${info.reason}`); + }; + + let secrets = []; + let credential = null; + let pages = 0; + let items = 0; + let requests = 0; + let cursor = since ?? null; + let checkpoint = null; + + const base = { + command: "collect", + channel: CHANNEL, + mode: "api", + ok: false, + inputs: [], + outputs, + warnings, + unavailable: [], + credential_file: CONFIG_FILE, + retries, + }; + const fail = (failure) => { + const receipt = { + ...base, + ok: false, + person: person ?? null, + chat_id: chatId ?? null, + pages, + items, + requests, + cursor, + resumed_from: checkpoint?.cursor ?? null, + errors: [redact(failure.message, secrets)], + unavailable: [ + { + channel: CHANNEL, + reason: redact(`${failure.reason}: ${failure.message}`, secrets), + remediation: failure.remediation ?? [], + }, + ], + }; + return { ok: false, exitCode: failure.exitCode ?? 1, receipt: scrub(receipt, secrets) }; + }; + + try { + if (!chatId) { + throw new CollectFailure("missing-target", "collect feishu needs --chat-id ", { + remediation: [ + "find the chat id in the Feishu URL, or via GET /im/v1/chats with the app credential", + ], + }); + } + + credential = loadCredential({ env }); + secrets = [credential.values.app_secret, credential.values.user_access_token].filter(Boolean); + base.credential_source = credential.source; + + if (resume) checkpoint = readCheckpoint({ env, root, target: chatId }); + if (since === undefined && checkpoint?.cursor) { + cursor = checkpoint.cursor; + pages = Number(checkpoint.pages ?? 0); + warnings.push(`resuming from checkpoint cursor (page ${pages} done)`); + } + + const userToken = useUserToken ? credential.values.user_access_token : null; + if (useUserToken && !userToken) { + warnings.push("DISTILLY_FEISHU_USE_USER_TOKEN is set but no user token is configured; using the tenant token"); + } + let bearer = userToken; + if (!bearer) { + const auth = await requestJson({ + fetchImpl, + url: `${baseUrl}/auth/v3/tenant_access_token/internal`, + method: "POST", + headers: { "content-type": "application/json" }, + body: { app_id: credential.values.app_id, app_secret: credential.values.app_secret }, + maxRetries, + sleep, + secrets, + onRetry, + }); + requests += 1; + if (auth.json?.code !== 0 || !auth.json?.tenant_access_token) { + throw new CollectFailure( + "auth-failed", + `Feishu rejected the app credential (code=${auth.json?.code ?? "?"})`, + { remediation: REMEDIATION_SETUP }, + ); + } + bearer = auth.json.tenant_access_token; + secrets = [...secrets, bearer]; + } + + let hasMore = true; + const pageSize = Math.min(Math.max(1, Number(limit) || DEFAULT_PAGE_SIZE), 50); + while (hasMore) { + if (pages >= maxPages) { + warnings.push(`stopped after --max-pages ${maxPages}; rerun to continue from the cursor`); + break; + } + const params = new URLSearchParams({ + container_id_type: "chat", + container_id: chatId, + page_size: String(pageSize), + sort_type: "ByCreateTimeDesc", + }); + if (cursor) params.set("page_token", cursor); + + const response = await requestJson({ + fetchImpl, + url: `${baseUrl}/im/v1/messages?${params.toString()}`, + headers: { authorization: `Bearer ${bearer}` }, + maxRetries, + sleep, + secrets, + onRetry, + }); + requests += 1; + + if (response.json?.code !== 0) { + throw new CollectFailure( + "api-error", + `Feishu code=${response.json?.code ?? "?"}: ${redact(response.json?.msg ?? "", secrets)}`, + { + remediation: [ + "check the app scopes (im:message:readonly) and that the bot is in the chat", + ...REMEDIATION_SETUP, + ], + }, + ); + } + + pages += 1; + const data = response.json?.data ?? {}; + const pageItems = Array.isArray(data.items) ? data.items : []; + items += pageItems.length; + + const stored = writeRaw(knowledgeDir, `${chatId}-p${String(pages).padStart(3, "0")}`, response.text); + outputs.push({ path: stored.path, sha256: stored.sha256, bytes: stored.bytes, kind: "raw" }); + ledgerEntries.push({ + id: `${CHANNEL}:${slug(chatId)}:p${String(pages).padStart(3, "0")}`, + kind: "raw", + origin: stored.relativePath, + source: CHANNEL, + fetched_at: now, + bytes: stored.bytes, + sha256: stored.sha256, + credentialed: true, + credential_source: credential.source, + credential_file: CONFIG_FILE, + method: useUserToken && userToken ? "api-user-token" : "api-tenant-token", + items: pageItems.length, + warnings: [], + }); + + cursor = data.page_token ?? null; + hasMore = Boolean(data.has_more) && Boolean(cursor); + if (resume) { + writeCheckpoint( + { env, root, target: chatId }, + { channel: CHANNEL, target: chatId, cursor, pages, items, updated_at: now }, + ); + } + onProgress(`page ${pages}: ${pageItems.length} items, has_more=${hasMore}`); + } + + const ledger = appendLedger(knowledgeDir, ledgerEntries); + if (resume) clearCheckpoint({ env, root, target: chatId }); + + const receipt = { + ...base, + ok: true, + person: person ?? null, + chat_id: chatId, + pages, + items, + requests, + cursor, + resumed_from: checkpoint?.cursor ?? null, + outputs, + ledger: { path: ledger.path, added: ledger.added, total: ledger.total }, + unavailable: [], + }; + return { ok: true, exitCode: 0, receipt: scrub(receipt, secrets) }; + } catch (error) { + if (!(error instanceof CollectFailure)) { + throw new CollectFailure("unexpected", redact(error?.message ?? String(error), secrets), { + remediation: ["rerun with --json and report the receipt"], + }); + } + const result = fail(error); + // Keep whatever was already written; only the ledger is flushed for those pages. + if (ledgerEntries.length > 0) { + try { + const ledger = appendLedger(knowledgeDir, ledgerEntries); + result.receipt.ledger = { path: ledger.path, added: ledger.added, total: ledger.total }; + result.receipt.partial = true; + } catch { + result.receipt.warnings.push("could not register the partial pages in knowledge/index.json"); + } + } + return result; + } +} + +// ─── CLI ───────────────────────────────────────────────────────────────────── + +export const HELP = `distilly collect feishu — 飞书消息采集（只读）/ Feishu message collection (read-only) + +用法 (zh): + distilly collect feishu --chat-id [--person ] [--root ] + [--limit 50] [--max-pages 10] [--max-retries 4] + [--since ] [--no-resume] [--json] + +凭据：~/.distilly/${CONFIG_FILE}（app_id / app_secret，0600）或环境变量 + DISTILLY_FEISHU_APP_ID / DISTILLY_FEISHU_APP_SECRET。 + 错误信息只出现配置文件名，绝不出现值。缺凭据 → 非零退出 + 补救步骤。 +权限：im:message:readonly、im:chat:readonly；私聊另需 user_access_token。 +只读：本模块没有任何点赞/关注/发帖/私信调用，写操作在 assertReadOnly() 里被拒绝。 + +--- +## English + distilly collect feishu --chat-id [--person ] [--limit N] [--json] + +Credentials: ~/.distilly/${CONFIG_FILE} or DISTILLY_FEISHU_APP_ID / DISTILLY_FEISHU_APP_SECRET. +Raw pages land verbatim in knowledge/raw/feishu/ and are registered in knowledge/index.json. +Rate limits honour Retry-After; an interrupted run resumes from its checkpoint cursor. +`; + +export function parseFlags(argv) { + const flags = { _: [] }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (!arg.startsWith("--")) { + flags._.push(arg); + continue; + } + const name = arg.slice(2); + if (name === "json" || name === "help" || name === "no-resume" || name === "use-user-token") { + flags[name] = true; + continue; + } + const value = argv[index + 1]; + if (value === undefined || value.startsWith("--")) throw new Error(`${arg} requires a value`); + flags[name] = value; + index += 1; + } + return flags; +} + +/** + * @param {string[]} argv arguments after `collect feishu` + * @param {{env?: object, stdout?: Function, stderr?: Function, fetch?: Function}} [io] + * @returns {Promise} exit code + */ +export async function runCollectCli(argv, io = {}) { + const out = io.stdout ?? ((line) => process.stdout.write(`${line}\n`)); + const err = io.stderr ?? ((line) => process.stderr.write(`${line}\n`)); + + let flags; + try { + flags = parseFlags(argv); + } catch (error) { + err(`Error: ${error.message}`); + return 1; + } + if (flags.help) { + out(HELP); + return 0; + } + + const result = await collect({ + fetch: io.fetch ?? globalThis.fetch, + env: io.env ?? process.env, + root: flags.root ?? process.cwd(), + person: flags.person, + family: flags.family, + chatId: flags["chat-id"], + limit: flags.limit ? Number(flags.limit) : undefined, + maxPages: flags["max-pages"] ? Number(flags["max-pages"]) : undefined, + maxRetries: flags["max-retries"] ? Number(flags["max-retries"]) : undefined, + since: flags.since, + resume: !flags["no-resume"], + sleep: io.sleep, + now: io.now, + baseUrl: flags["base-url"], + }); + + if (flags.json) out(JSON.stringify(result.receipt, null, 2)); + if (result.ok) { + out(`collected ${result.receipt.items} item(s) in ${result.receipt.pages} page(s) → ${result.receipt.outputs.length} raw file(s)`); + for (const warning of result.receipt.warnings) err(`warning: ${warning}`); + } else { + err(`Error: ${result.receipt.errors?.[0] ?? "collect failed"}`); + for (const entry of result.receipt.unavailable) { + err(`unavailable: ${entry.channel} — ${entry.reason}`); + for (const step of entry.remediation ?? []) err(` fix: ${step}`); + } + for (const warning of result.receipt.warnings) err(`warning: ${warning}`); + } + return result.exitCode; +} diff --git a/src/collect/slack.mjs b/src/collect/slack.mjs new file mode 100644 index 00000000..51515dfd --- /dev/null +++ b/src/collect/slack.mjs @@ -0,0 +1,713 @@ +/** + * slack.mjs — credentialed collection from the Slack Web API. + * + * Legacy ported: `tools/slack_auto_collector.py` (722 lines, `slack_sdk`). The + * credential file, the retry policy and the two error classes (missing scope, + * invalid token) keep their meaning; the SDK is replaced by an injected `fetch`, + * so the channel is testable without a workspace and without a network. + * + * Slack differences from Feishu worth knowing: + * - the token is a bot token (`xoxb-…`) used directly — there is no exchange + * step, so this module makes **no non-GET request at all**; + * - rate limiting arrives twice: as HTTP 429 with `Retry-After`, and as HTTP 200 + * with `{"ok": false, "error": "ratelimited"}`. Both back off; + * - pagination is `cursor` → `response_metadata.next_cursor`, and the empty + * string means "no more pages". + * + * Everything else follows the discipline described in `feishu.mjs`: raw bytes + * verbatim into `knowledge/raw/slack/`, ledger upsert, resume checkpoints, and + * no credential value in stdout / stderr / receipts. + */ + +import { createHash } from "node:crypto"; +import { + existsSync, + mkdirSync, + readFileSync, + renameSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { homedir } from "node:os"; +import { dirname, join, resolve } from "node:path"; + +export const CHANNEL = "slack"; +export const CONFIG_FILE = "slack_config.json"; +export const LEGACY_CONFIG_FILE = join(".colleague-skill", CONFIG_FILE); +export const DEFAULT_BASE_URL = "https://slack.com/api"; +export const DEFAULT_PAGE_SIZE = 200; +export const DEFAULT_MAX_PAGES = 10; +export const DEFAULT_MAX_RETRIES = 4; +export const DEFAULT_MAX_BACKOFF_MS = 60_000; + +export const ENV_KEYS = { + botToken: ["DISTILLY_SLACK_BOT_TOKEN", "SLACK_BOT_TOKEN"], +}; + +/** Slack is read-only here: no method may be anything but GET. */ +export const ALLOWED_MUTATIONS = []; + +const MUTATING_METHODS = new Set(["POST", "PUT", "PATCH", "DELETE"]); +const REMEDIATION_SETUP = [ + `create ~/.distilly/${CONFIG_FILE} (chmod 600) with {"bot_token": "xoxb-…"}`, + " app: https://api.slack.com/apps → OAuth & Permissions → Bot Token Scopes:", + " channels:history, groups:history, mpim:history, im:history, channels:read, groups:read, users:read", + "or export DISTILLY_SLACK_BOT_TOKEN for this shell only", + "then invite the bot to every channel you want collected: /invite @your-bot", +]; + +export class CollectFailure extends Error { + constructor(reason, message, { remediation = [], exitCode = 1, kind = "failure" } = {}) { + super(message); + this.name = "CollectFailure"; + this.reason = reason; + this.remediation = remediation; + this.exitCode = exitCode; + this.kind = kind; + } +} + +export function redact(text, secrets = []) { + let output = String(text ?? ""); + for (const secret of secrets) { + if (typeof secret !== "string" || secret.length < 4) continue; + output = output.split(secret).join("[redacted]"); + } + return output; +} + +export function scrub(value, secrets = []) { + return JSON.parse( + JSON.stringify(value, (_key, item) => (typeof item === "string" ? redact(item, secrets) : item)), + ); +} + +export function distillyHome(env = process.env) { + const override = env?.DISTILLY_HOME; + return override ? resolve(String(override)) : join(homedir(), ".distilly"); +} + +export function credentialPaths(env = process.env) { + return { + primary: join(distillyHome(env), CONFIG_FILE), + legacy: join(homedir(), LEGACY_CONFIG_FILE), + }; +} + +export function loadCredential({ env = process.env, readFile = readFileSync } = {}) { + for (const name of ENV_KEYS.botToken) { + const value = env?.[name]; + if (typeof value === "string" && value.trim() !== "") { + return { + ok: true, + source: "env", + configFile: CONFIG_FILE, + path: null, + values: { bot_token: value.trim() }, + }; + } + } + + const { primary, legacy } = credentialPaths(env); + for (const [path, source] of [ + [primary, "config"], + [legacy, "legacy-config"], + ]) { + if (!existsSync(path)) continue; + let parsed; + try { + parsed = JSON.parse(readFile(path, "utf8")); + } catch (error) { + throw new CollectFailure( + "bad-credential-file", + `${CONFIG_FILE} is not valid JSON (${redact(error.message)}); rewrite it with {"bot_token": "xoxb-…"}`, + { remediation: REMEDIATION_SETUP }, + ); + } + const token = parsed.bot_token ?? parsed.botToken ?? parsed.token ?? null; + if (!token) { + throw new CollectFailure("incomplete-credential", `${CONFIG_FILE} is missing bot_token`, { + remediation: REMEDIATION_SETUP, + }); + } + return { ok: true, source, configFile: CONFIG_FILE, path, values: { bot_token: token } }; + } + + throw new CollectFailure("no-credential", `no credential at ~/.distilly/${CONFIG_FILE}`, { + remediation: REMEDIATION_SETUP, + }); +} + +export function assertReadOnly(url, method = "GET") { + const verb = String(method).toUpperCase(); + if (!MUTATING_METHODS.has(verb)) return true; + const path = (() => { + try { + return new URL(url).pathname; + } catch { + return String(url); + } + })(); + const allowed = ALLOWED_MUTATIONS.some((entry) => entry.method === verb && path.endsWith(entry.path)); + if (!allowed) { + throw new CollectFailure( + "write-operation-refused", + `refusing ${verb} ${path}: this collector never writes to ${CHANNEL}`, + { remediation: ["collectors are read-only; remove the mutating call instead of allowlisting it"] }, + ); + } + return true; +} + +export function parseRetryAfter(headerValue, nowMs = Date.now()) { + if (headerValue === null || headerValue === undefined || headerValue === "") return null; + const seconds = Number(headerValue); + if (Number.isFinite(seconds) && seconds >= 0) return Math.round(seconds * 1000); + const date = Date.parse(String(headerValue)); + if (Number.isFinite(date)) return Math.max(0, date - nowMs); + return null; +} + +export function backoffDelay(attempt, retryAfterMs = null, options = {}) { + const { baseMs = 500, maxMs = DEFAULT_MAX_BACKOFF_MS } = options; + if (Number.isFinite(retryAfterMs) && retryAfterMs !== null) return Math.min(retryAfterMs, maxMs); + return Math.min(baseMs * 2 ** Math.max(0, attempt - 1), maxMs); +} + +export function defaultSleep(ms) { + return new Promise((resolvePromise) => setTimeout(resolvePromise, ms)); +} + +export async function requestJson(options) { + const { + fetchImpl, + url, + method = "GET", + headers = {}, + body, + maxRetries = DEFAULT_MAX_RETRIES, + sleep = defaultSleep, + secrets = [], + onRetry = () => {}, + authRemediation = REMEDIATION_SETUP, + } = options; + + assertReadOnly(url, method); + if (typeof fetchImpl !== "function") { + throw new CollectFailure("no-fetch", "no fetch implementation available", { + remediation: ["run on Node >= 20, or pass an injected fetch"], + }); + } + + let attempt = 0; + for (;;) { + attempt += 1; + let response; + try { + response = await fetchImpl(url, { + method, + headers, + body: body === undefined ? undefined : JSON.stringify(body), + }); + } catch (error) { + const message = redact(error?.message ?? String(error), secrets); + if (attempt > maxRetries) { + throw new CollectFailure("network-error", `request failed: ${message}`, { + remediation: ["check the network/proxy and retry", ...authRemediation], + }); + } + await sleep(backoffDelay(attempt)); + onRetry({ attempt, status: null, delayMs: backoffDelay(attempt), reason: message }); + continue; + } + + const status = Number(response?.status ?? 0); + const headerBag = response?.headers; + const retryAfterMs = parseRetryAfter(headerBag?.get?.("retry-after") ?? null); + + if (status === 429 || status >= 500) { + if (attempt > maxRetries) { + throw new CollectFailure( + status === 429 ? "rate-limited" : "server-error", + status === 429 + ? `rate limited (HTTP 429) after ${maxRetries} retries` + : `server error (HTTP ${status}) after ${maxRetries} retries`, + { + remediation: [ + "retry later; already-fetched pages stay on disk and the cursor is checkpointed", + `lower --limit / --max-pages to stay under the ${CHANNEL} quota`, + ], + }, + ); + } + const delayMs = backoffDelay(attempt, retryAfterMs); + onRetry({ attempt, status, delayMs, reason: `HTTP ${status}` }); + await sleep(delayMs); + continue; + } + + const text = await response.text(); + if (status === 401 || status === 403) { + throw new CollectFailure("unauthorized", `HTTP ${status} from ${CHANNEL}; bot token rejected`, { + remediation: [ + "the credential in ~/.distilly/" + CONFIG_FILE + " was rejected — reinstall the app and copy a fresh bot token", + "check the bot token scopes at https://api.slack.com/apps → OAuth & Permissions", + ], + }); + } + if (status >= 400) { + throw new CollectFailure("http-error", `HTTP ${status}: ${redact(text.slice(0, 200), secrets)}`, { + remediation: ["check the request parameters", ...authRemediation], + }); + } + + let json = null; + try { + json = JSON.parse(text); + } catch { + throw new CollectFailure("invalid-json", `${CHANNEL} returned a non-JSON body`, { + remediation: ["retry later; if it persists the endpoint may have changed"], + }); + } + return { status, text, json, attempts: attempt, headers: headerBag }; + } +} + +/** Map a Slack `{ok:false, error}` body onto a loud failure with real steps. */ +export function slackError(error, secrets = []) { + const code = redact(String(error ?? "unknown_error"), secrets); + const byCode = { + invalid_auth: { + remediation: [ + "the bot token is invalid or revoked — copy a fresh one from https://api.slack.com/apps → OAuth & Permissions", + `then rewrite ~/.distilly/${CONFIG_FILE} with {"bot_token": "xoxb-…"}`, + ], + }, + token_revoked: { remediation: ["the token was revoked — reinstall the app and update the config file"] }, + account_inactive: { remediation: ["the Slack account is inactive; ask a workspace admin"] }, + missing_scope: { + remediation: [ + "add the missing scope: https://api.slack.com/apps → OAuth & Permissions → Bot Token Scopes", + "needed: channels:history, groups:history, mpim:history, channels:read, groups:read, users:read", + "then reinstall the app so the token carries the new scope", + ], + }, + not_in_channel: { + remediation: ["invite the bot to the channel: /invite @your-bot, then rerun"], + }, + channel_not_found: { + remediation: ["check --channel ; list channels with conversations.list (channels:read)"], + }, + }; + const entry = byCode[code] ?? { remediation: ["unexpected Slack error; retry or check the app configuration"] }; + return new CollectFailure("api-error", `Slack error: ${code}`, { remediation: entry.remediation }); +} + +export function sha256Hex(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +export function slug(text, fallback = "target") { + const slugged = String(text ?? "") + .normalize("NFKD") + .toLowerCase() + .replace(/[^a-z0-9._-]+/g, "-") + .replace(/-+/g, "-") + .replace(/^[-._]+|[-._]+$/g, "") + .slice(0, 64); + return slugged || fallback; +} + +export function knowledgeRoot({ root = process.cwd(), person, family = "colleague" } = {}) { + return person ? join(resolve(root), "skills", slug(family), slug(person), "knowledge") : join(resolve(root), "knowledge"); +} + +export function writeRaw(knowledgeDir, name, bytes) { + const dir = join(knowledgeDir, "raw", CHANNEL); + mkdirSync(dir, { recursive: true }); + const path = join(dir, `${slug(name, "page")}.json`); + const staging = `${path}.${process.pid}.tmp`; + const buffer = Buffer.from(bytes); + try { + writeFileSync(staging, buffer); + renameSync(staging, path); + } catch (error) { + if (existsSync(staging)) rmSync(staging, { force: true }); + throw error; + } + return { + path, + relativePath: `raw/${CHANNEL}/${slug(name, "page")}.json`, + bytes: buffer.length, + sha256: sha256Hex(buffer), + }; +} + +export function appendLedger(knowledgeDir, entries) { + if (entries.length === 0) return { path: join(knowledgeDir, "index.json"), added: 0, total: 0, existed: false }; + const path = join(knowledgeDir, "index.json"); + let existing = []; + if (existsSync(path)) { + try { + const parsed = JSON.parse(readFileSync(path, "utf8")); + existing = Array.isArray(parsed) ? parsed : []; + } catch (error) { + throw new CollectFailure("bad-ledger", `knowledge/index.json is not valid JSON: ${redact(error.message)}`, { + remediation: ["repair or remove knowledge/index.json, then rerun the collect"], + }); + } + } + const byId = new Map(existing.filter((e) => e && typeof e === "object").map((e) => [e.id, e])); + let added = 0; + for (const entry of entries) { + if (!byId.has(entry.id)) added += 1; + byId.set(entry.id, entry); + } + const merged = [...byId.values()].sort((a, b) => String(a.id).localeCompare(String(b.id))); + const staging = `${path}.${process.pid}.tmp`; + try { + writeFileSync(staging, `${JSON.stringify(merged, null, 2)}\n`); + renameSync(staging, path); + } catch (error) { + if (existsSync(staging)) rmSync(staging, { force: true }); + throw error; + } + return { path, added, total: merged.length, existed: existing.length > 0 }; +} + +export function statePath({ env = process.env, root = process.cwd(), target } = {}) { + const key = sha256Hex(Buffer.from(`${resolve(root)}\n${target ?? ""}`, "utf8")).slice(0, 12); + return join(distillyHome(env), "state", `${CHANNEL}-${key}.json`); +} + +export function readCheckpoint(options) { + const path = statePath(options); + if (!existsSync(path)) return null; + try { + return JSON.parse(readFileSync(path, "utf8")); + } catch { + return null; + } +} + +export function writeCheckpoint(options, value) { + const path = statePath(options); + mkdirSync(dirname(path), { recursive: true }); + const staging = `${path}.${process.pid}.tmp`; + writeFileSync(staging, `${JSON.stringify(value, null, 2)}\n`); + renameSync(staging, path); + return path; +} + +export function clearCheckpoint(options) { + const path = statePath(options); + if (existsSync(path)) rmSync(path, { force: true }); + return path; +} + +/** + * Collect history from one Slack channel. + * + * @param {object} options `fetch`, `env`, `root`, `person`, `channel` (required), + * `limit`, `maxPages`, `maxRetries`, `since` (cursor), `resume`, `sleep`, `now` + */ +export async function collect(options = {}) { + const { + fetch: fetchImpl = globalThis.fetch, + env = process.env, + root = process.cwd(), + person, + family = "colleague", + channel, + limit = DEFAULT_PAGE_SIZE, + maxPages = DEFAULT_MAX_PAGES, + maxRetries = DEFAULT_MAX_RETRIES, + since, + resume = true, + sleep = defaultSleep, + now = new Date().toISOString(), + baseUrl = env?.DISTILLY_SLACK_BASE_URL || DEFAULT_BASE_URL, + onProgress = () => {}, + } = options; + + const knowledgeDir = knowledgeRoot({ root, person, family }); + const outputs = []; + const warnings = []; + const ledgerEntries = []; + const retries = []; + const onRetry = (info) => { + retries.push(info); + warnings.push(`retry ${info.attempt} after ${info.reason} (waited ${info.delayMs}ms)`); + onProgress(`retry ${info.attempt}: ${info.reason}`); + }; + + let secrets = []; + let credential = null; + let pages = 0; + let items = 0; + let requests = 0; + let cursor = since ?? null; + let checkpoint = null; + + const base = { + command: "collect", + channel: CHANNEL, + mode: "api", + ok: false, + inputs: [], + outputs, + warnings, + unavailable: [], + credential_file: CONFIG_FILE, + retries, + }; + const fail = (failure) => { + const receipt = { + ...base, + ok: false, + person: person ?? null, + channel_id: channel ?? null, + pages, + items, + requests, + cursor, + resumed_from: checkpoint?.cursor ?? null, + errors: [redact(failure.message, secrets)], + unavailable: [ + { + channel: CHANNEL, + reason: redact(`${failure.reason}: ${failure.message}`, secrets), + remediation: failure.remediation ?? [], + }, + ], + }; + return { ok: false, exitCode: failure.exitCode ?? 1, receipt: scrub(receipt, secrets) }; + }; + + try { + if (!channel) { + throw new CollectFailure("missing-target", "collect slack needs --channel ", { + remediation: [ + "channel ids start with C (public), G (private) or D (dm); find them in the Slack URL", + ], + }); + } + + credential = loadCredential({ env }); + secrets = [credential.values.bot_token]; + base.credential_source = credential.source; + + if (resume) checkpoint = readCheckpoint({ env, root, target: channel }); + if (since === undefined && checkpoint?.cursor) { + cursor = checkpoint.cursor; + pages = Number(checkpoint.pages ?? 0); + warnings.push(`resuming from checkpoint cursor (page ${pages} done)`); + } + + let hasMore = true; + const pageSize = Math.min(Math.max(1, Number(limit) || DEFAULT_PAGE_SIZE), 200); + while (hasMore) { + if (pages >= maxPages) { + warnings.push(`stopped after --max-pages ${maxPages}; rerun to continue from the cursor`); + break; + } + + const url = new URL(`${baseUrl}/conversations.history`); + url.searchParams.set("channel", channel); + url.searchParams.set("limit", String(pageSize)); + if (cursor) url.searchParams.set("cursor", cursor); + + // Slack can rate limit with HTTP 200 + {"ok": false, "error": "ratelimited"}. + let response; + for (let pageAttempt = 1; ; pageAttempt += 1) { + response = await requestJson({ + fetchImpl, + url: url.toString(), + headers: { authorization: `Bearer ${credential.values.bot_token}` }, + maxRetries, + sleep, + secrets, + onRetry, + }); + requests += 1; + if (response.json?.ok === false && response.json?.error === "ratelimited" && pageAttempt <= maxRetries) { + const delayMs = backoffDelay( + pageAttempt, + parseRetryAfter(response.headers?.get?.("retry-after") ?? null), + ); + warnings.push(`retry ${pageAttempt} after slack ratelimited (waited ${delayMs}ms)`); + retries.push({ attempt: pageAttempt, status: 200, delayMs, reason: "ratelimited" }); + await sleep(delayMs); + continue; + } + break; + } + + if (response.json?.ok !== true) throw slackError(response.json?.error, secrets); + + pages += 1; + const pageItems = Array.isArray(response.json?.messages) ? response.json.messages : []; + items += pageItems.length; + + const stored = writeRaw(knowledgeDir, `${channel}-p${String(pages).padStart(3, "0")}`, response.text); + outputs.push({ path: stored.path, sha256: stored.sha256, bytes: stored.bytes, kind: "raw" }); + ledgerEntries.push({ + id: `${CHANNEL}:${slug(channel)}:p${String(pages).padStart(3, "0")}`, + kind: "raw", + origin: stored.relativePath, + source: CHANNEL, + fetched_at: now, + bytes: stored.bytes, + sha256: stored.sha256, + credentialed: true, + credential_source: credential.source, + credential_file: CONFIG_FILE, + method: "api-bot-token", + items: pageItems.length, + warnings: [], + }); + + const next = response.json?.response_metadata?.next_cursor; + cursor = typeof next === "string" && next !== "" ? next : null; + hasMore = Boolean(cursor); + if (resume) { + writeCheckpoint( + { env, root, target: channel }, + { channel: CHANNEL, target: channel, cursor, pages, items, updated_at: now }, + ); + } + onProgress(`page ${pages}: ${pageItems.length} messages, has_more=${hasMore}`); + } + + const ledger = appendLedger(knowledgeDir, ledgerEntries); + if (resume) clearCheckpoint({ env, root, target: channel }); + + const receipt = { + ...base, + ok: true, + person: person ?? null, + channel_id: channel, + pages, + items, + requests, + cursor, + resumed_from: checkpoint?.cursor ?? null, + outputs, + ledger: { path: ledger.path, added: ledger.added, total: ledger.total }, + unavailable: [], + }; + return { ok: true, exitCode: 0, receipt: scrub(receipt, secrets) }; + } catch (error) { + if (!(error instanceof CollectFailure)) { + throw new CollectFailure("unexpected", redact(error?.message ?? String(error), secrets), { + remediation: ["rerun with --json and report the receipt"], + }); + } + const result = fail(error); + if (ledgerEntries.length > 0) { + try { + const ledger = appendLedger(knowledgeDir, ledgerEntries); + result.receipt.ledger = { path: ledger.path, added: ledger.added, total: ledger.total }; + result.receipt.partial = true; + } catch { + result.receipt.warnings.push("could not register the partial pages in knowledge/index.json"); + } + } + return result; + } +} + +export const HELP = `distilly collect slack — Slack 频道消息采集（只读）/ Slack channel history (read-only) + +用法 (zh): + distilly collect slack --channel [--person ] [--root ] + [--limit 200] [--max-pages 10] [--max-retries 4] + [--since ] [--no-resume] [--json] + +凭据：~/.distilly/${CONFIG_FILE}（bot_token，0600）或 DISTILLY_SLACK_BOT_TOKEN。 + 错误信息只出现配置文件名，绝不出现值。缺凭据 → 非零退出 + 补救步骤。 +权限：channels:history / groups:history / channels:read / users:read；机器人必须先被 /invite 进频道。 +限流：HTTP 429 与 {"ok":false,"error":"ratelimited"} 都按 Retry-After / 指数退避重试，超上限则保留已落盘页并非零退出。 +只读：本模块没有任何点赞/关注/发帖/私信调用；所有请求都是 GET（ALLOWED_MUTATIONS 为空）。 + +--- +## English + distilly collect slack --channel [--person ] [--limit N] [--json] + +Credentials: ~/.distilly/${CONFIG_FILE} or DISTILLY_SLACK_BOT_TOKEN. GET only. +Rate limits honour Retry-After; partial runs keep their pages and resume from a checkpoint. +`; + +export function parseFlags(argv) { + const flags = { _: [] }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (!arg.startsWith("--")) { + flags._.push(arg); + continue; + } + const name = arg.slice(2); + if (name === "json" || name === "help" || name === "no-resume") { + flags[name] = true; + continue; + } + const value = argv[index + 1]; + if (value === undefined || value.startsWith("--")) throw new Error(`${arg} requires a value`); + flags[name] = value; + index += 1; + } + return flags; +} + +/** + * @param {string[]} argv arguments after `collect slack` + * @param {{env?: object, stdout?: Function, stderr?: Function, fetch?: Function}} [io] + * @returns {Promise} exit code + */ +export async function runCollectCli(argv, io = {}) { + const out = io.stdout ?? ((line) => process.stdout.write(`${line}\n`)); + const err = io.stderr ?? ((line) => process.stderr.write(`${line}\n`)); + + let flags; + try { + flags = parseFlags(argv); + } catch (error) { + err(`Error: ${error.message}`); + return 1; + } + if (flags.help) { + out(HELP); + return 0; + } + + const result = await collect({ + fetch: io.fetch ?? globalThis.fetch, + env: io.env ?? process.env, + root: flags.root ?? process.cwd(), + person: flags.person, + family: flags.family, + channel: flags.channel, + limit: flags.limit ? Number(flags.limit) : undefined, + maxPages: flags["max-pages"] ? Number(flags["max-pages"]) : undefined, + maxRetries: flags["max-retries"] ? Number(flags["max-retries"]) : undefined, + since: flags.since, + resume: !flags["no-resume"], + sleep: io.sleep, + now: io.now, + baseUrl: flags["base-url"], + }); + + if (flags.json) out(JSON.stringify(result.receipt, null, 2)); + if (result.ok) { + out(`collected ${result.receipt.items} message(s) in ${result.receipt.pages} page(s) → ${result.receipt.outputs.length} raw file(s)`); + for (const warning of result.receipt.warnings) err(`warning: ${warning}`); + } else { + err(`Error: ${result.receipt.errors?.[0] ?? "collect failed"}`); + for (const entry of result.receipt.unavailable) { + err(`unavailable: ${entry.channel} — ${entry.reason}`); + for (const step of entry.remediation ?? []) err(` fix: ${step}`); + } + for (const warning of result.receipt.warnings) err(`warning: ${warning}`); + } + return result.exitCode; +} diff --git a/src/collect/x.mjs b/src/collect/x.mjs new file mode 100644 index 00000000..590217ee --- /dev/null +++ b/src/collect/x.mjs @@ -0,0 +1,958 @@ +/** + * x.mjs — X (Twitter) collection, two ways. + * + * api X API v2 with a Bearer token: `GET /2/users/by/username/:handle` + * then `GET /2/users/:id/tweets` with `pagination_token` (page cursor) + * and `since_id` (incremental re-collection). Read-only, GET only. + * browser computer use, which this module does **not** perform. It owns the + * consent gate (`src/consent.mjs`, scope `collect:x:browser`), the + * verbatim sink under `knowledge/raw/x/` and the ledger registration + * of whatever the host captured. Without `--consent ` (or with + * an expired one) the command exits 2 and the receipt says the run is + * waiting for user consent. + * + * Legacy context: `tools/research/xquik_public_posts.py` collected public posts + * through the Xquik aggregator (`XQUIK_API_KEY`). That third-party route is *not* + * migrated here — this module talks to X directly — and the aggregator remains a + * known gap in `docs/evidence/pr-07-collect-consent.md`. + * + * Read-only by construction: `ALLOWED_MUTATIONS` is empty, so `assertReadOnly()` + * refuses every non-GET request this file could ever issue. There is no code + * path that likes, follows, reposts, posts or DMs. + */ + +import { createHash } from "node:crypto"; +import { + existsSync, + mkdirSync, + readFileSync, + renameSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { homedir } from "node:os"; +import { dirname, join, resolve } from "node:path"; + +import { consentTokenFingerprint, verify as verifyConsent } from "../consent.mjs"; + +export const CHANNEL = "x"; +export const CONFIG_FILE = "x_config.json"; +export const LEGACY_CONFIG_FILE = join(".colleague-skill", CONFIG_FILE); +export const DEFAULT_BASE_URL = "https://api.x.com/2"; +export const DEFAULT_PAGE_SIZE = 100; +export const DEFAULT_MAX_PAGES = 10; +export const DEFAULT_MAX_RETRIES = 4; +export const DEFAULT_MAX_BACKOFF_MS = 60_000; +export const BROWSER_SCOPE = "collect:x:browser"; + +export const ENV_KEYS = { + bearer: ["DISTILLY_X_BEARER_TOKEN", "X_BEARER_TOKEN", "TWITTER_BEARER_TOKEN"], +}; + +/** X API v2 reads only. Any non-GET request is refused before it is sent. */ +export const ALLOWED_MUTATIONS = []; + +const MUTATING_METHODS = new Set(["POST", "PUT", "PATCH", "DELETE"]); +const REMEDIATION_SETUP = [ + `create ~/.distilly/${CONFIG_FILE} (chmod 600) with {"bearer_token": "…"}`, + " app: https://developer.x.com → Project & Apps → Keys and tokens → Bearer Token", + "or export DISTILLY_X_BEARER_TOKEN for this shell only", + "browser mode does not need this token: it needs consent — distilly consent grant --scope collect:x:browser", +]; + +export class CollectFailure extends Error { + constructor(reason, message, { remediation = [], exitCode = 1, kind = "failure" } = {}) { + super(message); + this.name = "CollectFailure"; + this.reason = reason; + this.remediation = remediation; + this.exitCode = exitCode; + this.kind = kind; + } +} + +export function redact(text, secrets = []) { + let output = String(text ?? ""); + for (const secret of secrets) { + if (typeof secret !== "string" || secret.length < 4) continue; + output = output.split(secret).join("[redacted]"); + } + return output; +} + +export function scrub(value, secrets = []) { + return JSON.parse( + JSON.stringify(value, (_key, item) => (typeof item === "string" ? redact(item, secrets) : item)), + ); +} + +export function distillyHome(env = process.env) { + const override = env?.DISTILLY_HOME; + return override ? resolve(String(override)) : join(homedir(), ".distilly"); +} + +export function credentialPaths(env = process.env) { + return { + primary: join(distillyHome(env), CONFIG_FILE), + legacy: join(homedir(), LEGACY_CONFIG_FILE), + }; +} + +export function loadCredential({ env = process.env, readFile = readFileSync } = {}) { + for (const name of ENV_KEYS.bearer) { + const value = env?.[name]; + if (typeof value === "string" && value.trim() !== "") { + return { + ok: true, + source: "env", + configFile: CONFIG_FILE, + path: null, + values: { bearer_token: value.trim() }, + }; + } + } + + const { primary, legacy } = credentialPaths(env); + for (const [path, source] of [ + [primary, "config"], + [legacy, "legacy-config"], + ]) { + if (!existsSync(path)) continue; + let parsed; + try { + parsed = JSON.parse(readFile(path, "utf8")); + } catch (error) { + throw new CollectFailure( + "bad-credential-file", + `${CONFIG_FILE} is not valid JSON (${redact(error.message)}); rewrite it with {"bearer_token": "…"}`, + { remediation: REMEDIATION_SETUP }, + ); + } + const token = parsed.bearer_token ?? parsed.bearerToken ?? parsed.bearer ?? parsed.api_key ?? null; + if (!token) { + throw new CollectFailure("incomplete-credential", `${CONFIG_FILE} is missing bearer_token`, { + remediation: REMEDIATION_SETUP, + }); + } + return { ok: true, source, configFile: CONFIG_FILE, path, values: { bearer_token: token } }; + } + + throw new CollectFailure("no-credential", `no credential at ~/.distilly/${CONFIG_FILE}`, { + remediation: REMEDIATION_SETUP, + }); +} + +export function assertReadOnly(url, method = "GET") { + const verb = String(method).toUpperCase(); + if (!MUTATING_METHODS.has(verb)) return true; + const path = (() => { + try { + return new URL(url).pathname; + } catch { + return String(url); + } + })(); + const allowed = ALLOWED_MUTATIONS.some((entry) => entry.method === verb && path.endsWith(entry.path)); + if (!allowed) { + throw new CollectFailure( + "write-operation-refused", + `refusing ${verb} ${path}: this collector never writes to ${CHANNEL}`, + { remediation: ["collectors are read-only; remove the mutating call instead of allowlisting it"] }, + ); + } + return true; +} + +export function parseRetryAfter(headerValue, nowMs = Date.now()) { + if (headerValue === null || headerValue === undefined || headerValue === "") return null; + const seconds = Number(headerValue); + if (Number.isFinite(seconds) && seconds >= 0) return Math.round(seconds * 1000); + const date = Date.parse(String(headerValue)); + if (Number.isFinite(date)) return Math.max(0, date - nowMs); + return null; +} + +/** X sends `x-rate-limit-reset` as epoch seconds when the window resets. */ +export function parseRateLimitReset(headerValue, nowMs = Date.now()) { + const epochSeconds = Number(headerValue); + if (!Number.isFinite(epochSeconds) || epochSeconds <= 0) return null; + return Math.max(0, epochSeconds * 1000 - nowMs); +} + +export function backoffDelay(attempt, retryAfterMs = null, options = {}) { + const { baseMs = 500, maxMs = DEFAULT_MAX_BACKOFF_MS } = options; + if (Number.isFinite(retryAfterMs) && retryAfterMs !== null) return Math.min(retryAfterMs, maxMs); + return Math.min(baseMs * 2 ** Math.max(0, attempt - 1), maxMs); +} + +export function defaultSleep(ms) { + return new Promise((resolvePromise) => setTimeout(resolvePromise, ms)); +} + +export async function requestJson(options) { + const { + fetchImpl, + url, + method = "GET", + headers = {}, + body, + maxRetries = DEFAULT_MAX_RETRIES, + sleep = defaultSleep, + secrets = [], + onRetry = () => {}, + authRemediation = REMEDIATION_SETUP, + } = options; + + assertReadOnly(url, method); + if (typeof fetchImpl !== "function") { + throw new CollectFailure("no-fetch", "no fetch implementation available", { + remediation: ["run on Node >= 20, or pass an injected fetch"], + }); + } + + let attempt = 0; + for (;;) { + attempt += 1; + let response; + try { + response = await fetchImpl(url, { + method, + headers, + body: body === undefined ? undefined : JSON.stringify(body), + }); + } catch (error) { + const message = redact(error?.message ?? String(error), secrets); + if (attempt > maxRetries) { + throw new CollectFailure("network-error", `request failed: ${message}`, { + remediation: ["check the network/proxy and retry", ...authRemediation], + }); + } + await sleep(backoffDelay(attempt)); + onRetry({ attempt, status: null, delayMs: backoffDelay(attempt), reason: message }); + continue; + } + + const status = Number(response?.status ?? 0); + const headerBag = response?.headers; + const retryAfterMs = + parseRetryAfter(headerBag?.get?.("retry-after") ?? null) ?? + (status === 429 ? parseRateLimitReset(headerBag?.get?.("x-rate-limit-reset") ?? null) : null); + + if (status === 429 || status >= 500) { + if (attempt > maxRetries) { + throw new CollectFailure( + status === 429 ? "rate-limited" : "server-error", + status === 429 + ? `rate limited (HTTP 429) after ${maxRetries} retries` + : `server error (HTTP ${status}) after ${maxRetries} retries`, + { + remediation: [ + "retry later; already-fetched pages stay on disk and the cursor is checkpointed", + "X API v2 windows are 15 minutes; lower --limit / --max-pages", + ], + }, + ); + } + const delayMs = backoffDelay(attempt, retryAfterMs); + onRetry({ attempt, status, delayMs, reason: `HTTP ${status}` }); + await sleep(delayMs); + continue; + } + + const text = await response.text(); + if (status === 401 || status === 403) { + throw new CollectFailure("unauthorized", `HTTP ${status} from ${CHANNEL}; bearer token rejected or not entitled`, { + remediation: [ + `the bearer token in ~/.distilly/${CONFIG_FILE} was rejected — regenerate it in the X developer portal`, + "check that your access level includes user tweet lookup", + ], + }); + } + if (status >= 400) { + throw new CollectFailure("http-error", `HTTP ${status}: ${redact(text.slice(0, 200), secrets)}`, { + remediation: ["check the request parameters", ...authRemediation], + }); + } + + let json = null; + try { + json = JSON.parse(text); + } catch { + throw new CollectFailure("invalid-json", `${CHANNEL} returned a non-JSON body`, { + remediation: ["retry later; if it persists the endpoint may have changed"], + }); + } + return { status, text, json, attempts: attempt, headers: headerBag }; + } +} + +export function sha256Hex(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +export function slug(text, fallback = "target") { + const slugged = String(text ?? "") + .normalize("NFKD") + .toLowerCase() + .replace(/[^a-z0-9._-]+/g, "-") + .replace(/-+/g, "-") + .replace(/^[-._]+|[-._]+$/g, "") + .slice(0, 64); + return slugged || fallback; +} + +export function knowledgeRoot({ root = process.cwd(), person, family = "colleague" } = {}) { + return person ? join(resolve(root), "skills", slug(family), slug(person), "knowledge") : join(resolve(root), "knowledge"); +} + +export function writeRaw(knowledgeDir, name, bytes) { + const dir = join(knowledgeDir, "raw", CHANNEL); + mkdirSync(dir, { recursive: true }); + const path = join(dir, `${slug(name, "page")}.json`); + const staging = `${path}.${process.pid}.tmp`; + const buffer = Buffer.from(bytes); + try { + writeFileSync(staging, buffer); + renameSync(staging, path); + } catch (error) { + if (existsSync(staging)) rmSync(staging, { force: true }); + throw error; + } + return { + path, + relativePath: `raw/${CHANNEL}/${slug(name, "page")}.json`, + bytes: buffer.length, + sha256: sha256Hex(buffer), + }; +} + +export function appendLedger(knowledgeDir, entries) { + if (entries.length === 0) return { path: join(knowledgeDir, "index.json"), added: 0, total: 0, existed: false }; + const path = join(knowledgeDir, "index.json"); + let existing = []; + if (existsSync(path)) { + try { + const parsed = JSON.parse(readFileSync(path, "utf8")); + existing = Array.isArray(parsed) ? parsed : []; + } catch (error) { + throw new CollectFailure("bad-ledger", `knowledge/index.json is not valid JSON: ${redact(error.message)}`, { + remediation: ["repair or remove knowledge/index.json, then rerun the collect"], + }); + } + } + const byId = new Map(existing.filter((e) => e && typeof e === "object").map((e) => [e.id, e])); + let added = 0; + for (const entry of entries) { + if (!byId.has(entry.id)) added += 1; + byId.set(entry.id, entry); + } + const merged = [...byId.values()].sort((a, b) => String(a.id).localeCompare(String(b.id))); + const staging = `${path}.${process.pid}.tmp`; + try { + writeFileSync(staging, `${JSON.stringify(merged, null, 2)}\n`); + renameSync(staging, path); + } catch (error) { + if (existsSync(staging)) rmSync(staging, { force: true }); + throw error; + } + return { path, added, total: merged.length, existed: existing.length > 0 }; +} + +/** + * Run state for X: unlike the other channels the file survives a successful run, + * because `since_id` is what makes the *next* run incremental. + */ +export function statePath({ env = process.env, root = process.cwd(), target } = {}) { + const key = sha256Hex(Buffer.from(`${resolve(root)}\n${target ?? ""}`, "utf8")).slice(0, 12); + return join(distillyHome(env), "state", `${CHANNEL}-${key}.json`); +} + +export function readState(options) { + const path = statePath(options); + if (!existsSync(path)) return null; + try { + return JSON.parse(readFileSync(path, "utf8")); + } catch { + return null; + } +} + +export function writeState(options, value) { + const path = statePath(options); + mkdirSync(dirname(path), { recursive: true }); + const staging = `${path}.${process.pid}.tmp`; + writeFileSync(staging, `${JSON.stringify(value, null, 2)}\n`); + renameSync(staging, path); + return path; +} + +/** + * Resolve `@handle` → numeric user id (X API v2). + * @returns {Promise} the numeric id + */ +export async function resolveUserId(handle, { fetchImpl, baseUrl, headers, maxRetries, sleep, secrets, onRetry }) { + const clean = String(handle).replace(/^@/, "").trim(); + if (!/^[A-Za-z0-9_]{1,15}$/.test(clean)) { + throw new CollectFailure("bad-username", `invalid X username: ${JSON.stringify(handle)}`, { + remediation: ["use 1-15 letters, digits or underscores, with or without the leading @", ...REMEDIATION_SETUP], + }); + } + const response = await requestJson({ + fetchImpl, + url: `${baseUrl}/users/by/username/${encodeURIComponent(clean)}`, + headers, + maxRetries, + sleep, + secrets, + onRetry, + }); + const id = response.json?.data?.id; + if (!id) { + throw new CollectFailure("user-not-found", `X has no user @${clean}`, { + remediation: ["check the handle; deleted or suspended accounts return no data", ...REMEDIATION_SETUP], + }); + } + return String(id); +} + +/** + * api mode: collect one user's recent posts with cursor pagination and + * `since_id` incremental collection. + * + * @param {object} options `fetch`, `env`, `root`, `person`, `userId` or `username`, + * `limit`, `maxPages`, `maxRetries`, `sinceId`, `paginationToken`, `resume`, `sleep`, `now` + */ +export async function collectApi(options = {}) { + const { + fetch: fetchImpl = globalThis.fetch, + env = process.env, + root = process.cwd(), + person, + family = "colleague", + userId, + username, + limit = DEFAULT_PAGE_SIZE, + maxPages = DEFAULT_MAX_PAGES, + maxRetries = DEFAULT_MAX_RETRIES, + sinceId, + paginationToken, + resume = true, + sleep = defaultSleep, + now = new Date().toISOString(), + baseUrl = env?.DISTILLY_X_BASE_URL || DEFAULT_BASE_URL, + tweetFields = "created_at,public_metrics,lang,author_id", + onProgress = () => {}, + } = options; + + const knowledgeDir = knowledgeRoot({ root, person, family }); + const outputs = []; + const warnings = []; + const ledgerEntries = []; + const retries = []; + const onRetry = (info) => { + retries.push(info); + warnings.push(`retry ${info.attempt} after ${info.reason} (waited ${info.delayMs}ms)`); + onProgress(`retry ${info.attempt}: ${info.reason}`); + }; + + let secrets = []; + let credential = null; + let pages = 0; + let items = 0; + let requests = 0; + let cursor = paginationToken ?? null; + let since = sinceId ?? null; + let prior = null; + let target = userId ? String(userId) : username ? `@${String(username).replace(/^@/, "")}` : null; + + const base = { + command: "collect", + channel: CHANNEL, + mode: "api", + ok: false, + inputs: [], + outputs, + warnings, + unavailable: [], + credential_file: CONFIG_FILE, + retries, + }; + const fail = (failure) => ({ + ok: false, + exitCode: failure.exitCode ?? 1, + receipt: scrub( + { + ...base, + ok: false, + person: person ?? null, + target, + pages, + items, + requests, + cursor, + since_id: since, + errors: [redact(failure.message, secrets)], + unavailable: [ + { + channel: CHANNEL, + reason: redact(`${failure.reason}: ${failure.message}`, secrets), + remediation: failure.remediation ?? [], + }, + ], + }, + secrets, + ), + }); + + try { + if (!userId && !username) { + throw new CollectFailure("missing-target", "collect x needs --user-id or --username ", { + remediation: [ + "find the numeric id with `distilly collect x --username --dry-run` or from the profile URL", + ], + }); + } + + credential = loadCredential({ env }); + secrets = [credential.values.bearer_token]; + base.credential_source = credential.source; + const headers = { authorization: `Bearer ${credential.values.bearer_token}` }; + + // Resolve the handle first so the run state is always keyed by the numeric id. + let resolvedUserId = userId ? String(userId) : null; + if (!resolvedUserId) { + resolvedUserId = await resolveUserId(username, { fetchImpl, baseUrl, headers, maxRetries, sleep, secrets, onRetry }); + requests += 1; + } + target = resolvedUserId; + + if (resume) prior = readState({ env, root, target }); + if (since === undefined || since === null) { + if (prior?.since_id) { + since = prior.since_id; + warnings.push(`incremental since_id ${since} from the previous run`); + } + } + if (cursor === null && prior?.cursor && prior?.completed !== true) { + cursor = prior.cursor; + pages = Number(prior.pages ?? 0); + warnings.push(`resuming from checkpoint cursor (page ${pages} done)`); + } + const resumedFrom = cursor; + const sinceFromCheckpoint = prior?.since_id ?? null; + + let hasMore = true; + let newestId = prior?.newest_id ?? null; + const pageSize = Math.min(Math.max(10, Number(limit) || DEFAULT_PAGE_SIZE), 100); + while (hasMore) { + if (pages >= maxPages) { + warnings.push(`stopped after --max-pages ${maxPages}; rerun to continue from the cursor`); + break; + } + + const url = new URL(`${baseUrl}/users/${encodeURIComponent(resolvedUserId)}/tweets`); + url.searchParams.set("max_results", String(pageSize)); + url.searchParams.set("tweet.fields", tweetFields); + if (cursor) url.searchParams.set("pagination_token", cursor); + if (since) url.searchParams.set("since_id", String(since)); + + const response = await requestJson({ + fetchImpl, + url: url.toString(), + headers, + maxRetries, + sleep, + secrets, + onRetry, + }); + requests += 1; + + if (response.json?.errors?.length) { + const detail = redact(JSON.stringify(response.json.errors).slice(0, 200), secrets); + throw new CollectFailure("api-error", `X API error: ${detail}`, { + remediation: ["check the user id / access level, then retry", ...REMEDIATION_SETUP], + }); + } + + pages += 1; + const pageItems = Array.isArray(response.json?.data) ? response.json.data : []; + const meta = response.json?.meta ?? {}; + items += pageItems.length; + if (meta.newest_id) newestId = String(meta.newest_id); + + const stored = writeRaw(knowledgeDir, `${target}-p${String(pages).padStart(3, "0")}`, response.text); + outputs.push({ path: stored.path, sha256: stored.sha256, bytes: stored.bytes, kind: "raw" }); + ledgerEntries.push({ + id: `${CHANNEL}:${slug(target)}:p${String(pages).padStart(3, "0")}`, + kind: "raw", + origin: stored.relativePath, + source: CHANNEL, + fetched_at: now, + bytes: stored.bytes, + sha256: stored.sha256, + credentialed: true, + credential_source: credential.source, + credential_file: CONFIG_FILE, + method: "api-v2-bearer", + since_id: since ? String(since) : null, + newest_id: meta.newest_id ? String(meta.newest_id) : null, + items: pageItems.length, + warnings: [], + }); + + const next = meta.next_token; + cursor = typeof next === "string" && next !== "" ? next : null; + const ids = pageItems.map((item) => String(item?.id ?? "")).filter(Boolean); + const allOlderThanSince = since ? ids.length > 0 && ids.every((id) => BigInt(id) <= BigInt(since)) : false; + hasMore = Boolean(cursor) && pageItems.length > 0 && !allOlderThanSince; + + if (resume) { + writeState( + { env, root, target }, + { + channel: CHANNEL, + target, + cursor, + pages, + items, + since_id: since ? String(since) : null, + newest_id: newestId, + updated_at: now, + completed: false, + }, + ); + } + onProgress(`page ${pages}: ${pageItems.length} posts, has_more=${hasMore}`); + } + + const ledger = appendLedger(knowledgeDir, ledgerEntries); + if (resume) { + writeState( + { env, root, target }, + { + channel: CHANNEL, + target, + cursor: null, + pages, + items, + since_id: newestId ?? (since ? String(since) : null), + newest_id: newestId, + updated_at: now, + completed: true, + }, + ); + } + + return { + ok: true, + exitCode: 0, + receipt: scrub( + { + ...base, + ok: true, + person: person ?? null, + target, + pages, + items, + requests, + cursor, + since_id: since ? String(since) : null, + newest_id: newestId, + resumed_from: resumedFrom, + since_from_checkpoint: sinceFromCheckpoint, + outputs, + ledger: { path: ledger.path, added: ledger.added, total: ledger.total }, + unavailable: [], + }, + secrets, + ), + }; + } catch (error) { + if (!(error instanceof CollectFailure)) { + throw new CollectFailure("unexpected", redact(error?.message ?? String(error), secrets), { + remediation: ["rerun with --json and report the receipt"], + }); + } + const result = fail(error); + if (ledgerEntries.length > 0) { + try { + const ledger = appendLedger(knowledgeDir, ledgerEntries); + result.receipt.ledger = { path: ledger.path, added: ledger.added, total: ledger.total }; + result.receipt.partial = true; + } catch { + result.receipt.warnings.push("could not register the partial pages in knowledge/index.json"); + } + } + return result; + } +} + +/** The plan the host must follow in browser mode. No browser runs here. */ +export function browserPlan({ target, window = "recent", now = new Date().toISOString() }) { + return { + generated_at: now, + target: target ?? "@handle or profile URL", + window, + host_steps: [ + "Host (computer use): open a browser you control, sign in as the user, and open the target profile or search.", + "Host: scroll/expand the requested window; do not click like, follow, repost, reply or send — this tool cannot, and the grant does not cover it.", + "Host: save the captured posts (JSON or text, verbatim) to a file.", + `Host: register the capture with: distilly collect x --mode browser --consent --capture `, + ], + boundary: [ + "This module never drives a browser, never injects into one and never sends input events.", + "It owns three things only: the consent gate, the verbatim sink under knowledge/raw/x/, and the ledger entry.", + "Browser automation itself is a host capability (Claude Code, Codex, …); see docs/v2/STATUS.md for the host matrix.", + ], + }; +} + +/** + * browser mode: consent gate + sink + ledger registration for a host capture. + */ +export function collectBrowser(options = {}) { + const { + env = process.env, + root = process.cwd(), + person, + family = "colleague", + scope = BROWSER_SCOPE, + consentToken, + capturePath, + label, + target, + producer = "host:computer-use", + now = new Date().toISOString(), + readFile = readFileSync, + } = options; + + const knowledgeDir = knowledgeRoot({ root, person, family }); + const base = { + command: "collect", + channel: CHANNEL, + mode: "browser", + ok: false, + inputs: [], + outputs: [], + warnings: [], + unavailable: [], + credential_file: null, + }; + + const verification = verifyConsent(consentToken, { env, scope }); + if (!verification.ok) { + return { + ok: false, + exitCode: 2, + receipt: { + ...base, + ok: false, + status: "waiting-for-user-consent", + person: person ?? null, + target: target ?? null, + errors: [`waiting for user consent (${verification.reason})`], + unavailable: [ + { + channel: CHANNEL, + reason: `waiting for user consent: ${verification.reason}`, + scope, + remediation: verification.remediation, + }, + ], + consent_scope: scope, + }, + }; + } + + const consentBlock = { + scope: verification.record.scope, + granted_at: verification.record.granted_at, + expires_at: verification.record.expires_at, + token_sha256_12: consentTokenFingerprint(consentToken), + }; + + if (!capturePath) { + return { + ok: true, + exitCode: 0, + receipt: { + ...base, + ok: true, + status: "awaiting-host-capture", + person: person ?? null, + target: target ?? null, + consent: consentBlock, + plan: browserPlan({ target, now }), + unavailable: [], + }, + }; + } + + let bytes; + try { + bytes = readFile(capturePath); + } catch (error) { + return { + ok: false, + exitCode: 1, + receipt: { + ...base, + ok: false, + person: person ?? null, + errors: [`cannot read --capture ${capturePath}: ${redact(error.message)}`], + unavailable: [ + { + channel: CHANNEL, + reason: `capture-unreadable: ${redact(error.message)}`, + remediation: ["point --capture at a readable file produced by the host"], + }, + ], + }, + }; + } + + const name = label ?? `browser-${String(now).slice(0, 10)}`; + const stored = writeRaw(knowledgeDir, name, bytes); + const entry = { + id: `${CHANNEL}:${slug(name)}:capture`, + kind: "raw", + origin: stored.relativePath, + source: CHANNEL, + fetched_at: now, + bytes: stored.bytes, + sha256: stored.sha256, + credentialed: false, + method: "browser-host", + provenance: { method: "browser-host", producer, confidence: "host-reported" }, + consent: consentBlock, + warnings: [], + }; + const ledger = appendLedger(knowledgeDir, [entry]); + + return { + ok: true, + exitCode: 0, + receipt: { + ...base, + ok: true, + status: "captured", + person: person ?? null, + target: target ?? null, + consent: consentBlock, + outputs: [{ path: stored.path, sha256: stored.sha256, bytes: stored.bytes, kind: "raw" }], + ledger: { path: ledger.path, added: ledger.added, total: ledger.total }, + provenance: entry.provenance, + unavailable: [], + }, + }; +} + +export async function collect(options = {}) { + return options.mode === "browser" ? collectBrowser(options) : collectApi(options); +} + +export const HELP = `distilly collect x — X（Twitter）采集 / X collection + +用法 (zh): + distilly collect x --user-id [--person ] [--limit 100] [--max-pages 10] + [--since-id ] [--pagination-token ] [--no-resume] [--json] + distilly collect x --username […同上…] + distilly collect x --mode browser --consent [--capture ] [--json] + +凭据：~/.distilly/${CONFIG_FILE}（bearer_token，0600）或 DISTILLY_X_BEARER_TOKEN； + 错误信息只出现配置文件名，绝不出现值。 +分页/续采：pagination_token 翻页；成功后把 newest_id 记为 since_id，下次自动增量采集。 +限流：429 按 Retry-After（缺失时用 x-rate-limit-reset）退避，超上限保留已落盘页并非零退出。 +同意门：browser 模式必须带 --consent （distilly consent grant --scope ${BROWSER_SCOPE}）， + 无 token / 已过期 → exit 2 + 回执写“等待用户同意”；本工具不做浏览器自动化，只负责 + 同意门 + 原样落盘 + 账本登记，浏览器操作由宿主（computer use）完成。 +只读：ALLOWED_MUTATIONS 为空，任何非 GET 请求都会被 assertReadOnly() 拒绝；不存在点赞/关注/发帖/私信代码。 + +--- +## English + distilly collect x --user-id | --username [--since-id ] [--json] + X API v2, Bearer token, cursor pagination + since_id incremental collection. + distilly collect x --mode browser --consent [--capture ] [--json] + Requires a consent grant; without one the command exits 2 and the receipt says the + run is waiting for user consent. This tool does not automate a browser — the host + captures, this module stores bytes verbatim and registers them in the ledger. +`; + +export function parseFlags(argv) { + const flags = { _: [] }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (!arg.startsWith("--")) { + flags._.push(arg); + continue; + } + const name = arg.slice(2); + if (name === "json" || name === "help" || name === "no-resume") { + flags[name] = true; + continue; + } + const value = argv[index + 1]; + if (value === undefined || value.startsWith("--")) throw new Error(`${arg} requires a value`); + flags[name] = value; + index += 1; + } + return flags; +} + +export async function runCollectCli(argv, io = {}) { + const out = io.stdout ?? ((line) => process.stdout.write(`${line}\n`)); + const err = io.stderr ?? ((line) => process.stderr.write(`${line}\n`)); + + let flags; + try { + flags = parseFlags(argv); + } catch (error) { + err(`Error: ${error.message}`); + return 1; + } + if (flags.help) { + out(HELP); + return 0; + } + + const result = await collect({ + fetch: io.fetch ?? globalThis.fetch, + env: io.env ?? process.env, + root: flags.root ?? process.cwd(), + person: flags.person, + family: flags.family, + mode: flags.mode ?? "api", + userId: flags["user-id"], + username: flags.username, + limit: flags.limit ? Number(flags.limit) : undefined, + maxPages: flags["max-pages"] ? Number(flags["max-pages"]) : undefined, + maxRetries: flags["max-retries"] ? Number(flags["max-retries"]) : undefined, + sinceId: flags["since-id"], + paginationToken: flags["pagination-token"], + resume: !flags["no-resume"], + consentToken: flags.consent, + capturePath: flags.capture, + label: flags.label, + target: flags.target, + sleep: io.sleep, + now: io.now, + baseUrl: flags["base-url"], + }); + + if (flags.json) out(JSON.stringify(result.receipt, null, 2)); + if (result.ok) { + const status = result.receipt.status; + if (status) out(`ok (${status})`); + else out(`collected ${result.receipt.items} post(s) in ${result.receipt.pages} page(s) → ${result.receipt.outputs.length} raw file(s)`); + for (const warning of result.receipt.warnings ?? []) err(`warning: ${warning}`); + for (const step of result.receipt.plan?.host_steps ?? []) err(`host: ${step}`); + } else { + err(`Error: ${result.receipt.errors?.[0] ?? "collect failed"}`); + for (const entry of result.receipt.unavailable ?? []) { + err(`unavailable: ${entry.channel} — ${entry.reason}`); + for (const step of entry.remediation ?? []) err(` fix: ${step}`); + } + for (const warning of result.receipt.warnings ?? []) err(`warning: ${warning}`); + } + return result.exitCode; +} diff --git a/src/commands/doctor.mjs b/src/commands/doctor.mjs new file mode 100644 index 00000000..59c1c472 --- /dev/null +++ b/src/commands/doctor.mjs @@ -0,0 +1,159 @@ +/** + * `distilly doctor` — inventory health check (minimal, honest version). + * + * Checks what is actually on disk: + * 1. every host in the shared matrix: is Distilly installed there, at which + * version, and does the directory look like a Distilly install; + * 2. generated Skills under `skills//` (version, corrections); + * 3. ledger coverage whenever `knowledge/index.json` exists; + * 4. capabilities this build cannot run yet are listed in `unavailable` + * (CONTRACT §3: nothing is silently skipped). + */ + +import { existsSync, readFileSync, statSync } from "node:fs"; +import { join } from "node:path"; + +import { register, PLANNED } from "./index.mjs"; +import { createReceipt, describeFile, displayPath } from "../cli/receipt.mjs"; +import { parseArgs } from "../cli/args.mjs"; +import { listAgents } from "../hosts/agents.mjs"; +import { inspectInstall, repoInstallDir } from "../install/hosts.mjs"; +import { CHARACTER_PRESETS } from "../skill/presets.mjs"; +import { listSkills } from "../skill/writer.mjs"; + +function doctorHelp(binary = "distilly") { + const zh = [ + "用法：", + ` ${binary} doctor [--base-dir ] [--json]`, + "", + "检查项（最小版）：", + " 1. 宿主矩阵里每个宿主是否已安装 Distilly（路径、SKILL.md 版本）；", + " 2. 生成的 Skill 清单（skills//，含版本与 corrections）；", + " 3. 账本覆盖率：有 knowledge/index.json 时统计条目与字节数；", + " 4. 未实现能力（parse/view/collect 等）写进回执的 unavailable，不静默跳过。", + ].join("\n"); + const en = [ + "Usage:", + ` ${binary} doctor [--base-dir ] [--json]`, + "", + "Checks (minimal):", + " 1. whether Distilly is installed for each host in the shared matrix (path, SKILL.md version);", + " 2. the generated Skill inventory (skills// with version and corrections);", + " 3. ledger coverage: entry and byte counts whenever knowledge/index.json exists;", + " 4. capabilities this build cannot run yet (parse/view/collect …) are reported in the receipt's unavailable list, never skipped silently.", + ].join("\n"); + return { zh, en }; +} + +const OPTIONS = { + "base-dir": { type: "string", value: "dir" }, +}; + +function readLedger(skillDir) { + const ledgerPath = join(skillDir, "knowledge", "index.json"); + if (!existsSync(ledgerPath)) { + return { path: ledgerPath, entries: 0, bytes: 0, anchors: 0, present: false }; + } + const text = readFileSync(ledgerPath, "utf8"); + let entries = []; + try { + const parsed = JSON.parse(text); + entries = Array.isArray(parsed) ? parsed : (parsed.entries ?? []); + } catch { + entries = []; + } + const anchors = entries.reduce((total, entry) => { + const list = entry?.anchors; + return total + (Array.isArray(list) ? list.length : 0); + }, 0); + return { + path: ledgerPath, + entries: entries.length, + bytes: statSync(ledgerPath).size, + anchors, + present: true, + }; +} + +register("doctor", { + summary: "体检宿主与 Skill 库存 / Health-check hosts and skill inventory", + usage: "distilly doctor [--base-dir ] [--json]", + options: OPTIONS, + ...doctorHelp(), + run({ argv, reporter }) { + const { flags } = parseArgs(argv, OPTIONS); + const warnings = []; + const inputs = []; + const outputs = []; + + reporter.line("Hosts / 宿主:"); + const hostRows = []; + for (const id of listAgents()) { + const target = repoInstallDir(id); + const state = inspectInstall(target); + hostRows.push({ host: id, path: displayPath(target), installed: state.installed, version: state.version }); + reporter.line( + ` ${state.installed ? "installed" : "missing "} ${id.padEnd(18)} ${displayPath(target)}${state.version ? ` (v${state.version})` : ""}`, + ); + if (state.installed) { + const described = describeFile(join(target, "SKILL.md")); + if (described) inputs.push(described); + } + } + + reporter.line(""); + reporter.line("Skills / 人物 Skill:"); + const familyBase = flags["base-dir"]; + let skillCount = 0; + let anchorTotal = 0; + for (const [family, preset] of Object.entries(CHARACTER_PRESETS)) { + if (preset.character !== family) continue; + const baseDir = familyBase ? join(familyBase, family) : (preset.storage_root ?? preset.legacy_storage_root); + const skills = listSkills(baseDir); + for (const skill of skills) { + skillCount += 1; + const skillDir = join(baseDir, skill.slug); + const ledger = readLedger(skillDir); + anchorTotal += ledger.anchors; + reporter.line( + ` ${family}/${skill.slug} ${skill.version} corrections=${skill.corrections_count} ` + + `knowledge=${ledger.present ? `${ledger.entries} entries / ${ledger.bytes} bytes` : "none"}`, + ); + if (ledger.present) { + const described = describeFile(ledger.path); + if (described) outputs.push(described); + } + const skillFile = describeFile(join(skillDir, "SKILL.md")); + if (skillFile) outputs.push(skillFile); + } + } + if (skillCount === 0) { + reporter.line(" none found (run `distilly skill create` first)"); + warnings.push("no generated skills found"); + } + + reporter.line(""); + reporter.line( + `Ledger coverage / 账本：${skillCount} skills, ${anchorTotal} anchors recorded, 0 cited ` + + "(evidence/derived is delivered by ds/06-retrospect)", + ); + + const unavailable = Object.entries(PLANNED).map(([command, branch]) => ({ + channel: command, + reason: `not implemented in this build; delivered by ${branch}`, + })); + reporter.line(""); + reporter.line(`Unavailable / 未实现：${unavailable.map((item) => item.channel).join(", ")}`); + + return { + receipt: createReceipt("doctor", { + inputs, + outputs, + anchors: { total: anchorTotal, cited: 0 }, + warnings, + unavailable, + }), + extra: { hosts: hostRows, skills: skillCount }, + }; + }, +}); diff --git a/src/commands/index.mjs b/src/commands/index.mjs new file mode 100644 index 00000000..81e828f6 --- /dev/null +++ b/src/commands/index.mjs @@ -0,0 +1,203 @@ +/** + * Command registry — the single registration point for the Distilly CLI. + * + * `bin/distilly.mjs` only parses global flags and dispatches; every subcommand + * lives in its own module under `src/commands/` and registers itself here: + * + * ```js + * import { register } from "./index.mjs"; + * register("skill create", { + * summary: "创建一个 Skill / Create a Skill", + * usage: "distilly skill create [options]", + * options: {...}, // src/cli/args.mjs spec, for automatic usage + * run: async ({ flags, positionals, json, reporter, ctx }) => ({ receipt, lines }), + * }); + * ``` + * + * A definition returns `{receipt, lines, exitCode?}`; throwing `CliError` is the + * supported way to fail loudly with a remedy. See `docs/v2/NODE-CORE.md`. + * + * Names may contain one space (`view check`, `skill create`); dispatch prefers + * the two-token name. Commands promised by `docs/v2/CONTRACT.md` §1 but not yet + * implemented are listed in `PLANNED` together with the branch that owns them, + * so an unfinished build reports "not implemented" instead of "unknown command". + */ + +import { CliError } from "../cli/receipt.mjs"; + +const REGISTRY = new Map(); + +/** Commands frozen in CONTRACT §1 whose implementation ships in another branch. */ +export const PLANNED = { + harvest: "ds/02-parse-zero-cred", + "parse-chat": "ds/02-parse-zero-cred", + "parse-email": "ds/02-parse-zero-cred", + "parse-subtitle": "ds/02-parse-zero-cred", + "parse-doc": "ds/02-parse-zero-cred", + "parse-archive": "ds/02-parse-zero-cred", + retrospect: "ds/06-retrospect", + collect: "ds/07-keys-and-schema", + transcribe: "ds/07-keys-and-schema", + note: "ds/02-parse-zero-cred", + consent: "ds/07-keys-and-schema", + view: "ds/03-render", +}; + +/** + * Register one subcommand. Duplicate names are a programming error and throw. + * @param {string} name + * @param {{summary: string, usage: string, options?: object, run: Function, hidden?: boolean}} definition + */ +export function register(name, definition) { + if (REGISTRY.has(name)) throw new Error(`command already registered: ${name}`); + if (typeof definition?.run !== "function") { + throw new Error(`command ${name} needs a run() function`); + } + REGISTRY.set(name, { name, hidden: false, options: {}, ...definition }); + return definition; +} + +export function lookup(name) { + return REGISTRY.get(name) ?? null; +} + +export function listCommands({ includeHidden = false } = {}) { + return [...REGISTRY.values()] + .filter((command) => includeHidden || !command.hidden) + .sort((a, b) => a.name.localeCompare(b.name)); +} + +/** + * Split argv into a command name and its remaining arguments. + * Two-token names win over one-token names (`view check` before `view`). + */ +export function resolveCommand(tokens) { + if (tokens.length >= 2) { + const twoToken = `${tokens[0]} ${tokens[1]}`; + if (REGISTRY.has(twoToken)) return { name: twoToken, rest: tokens.slice(2) }; + } + if (tokens.length >= 1 && REGISTRY.has(tokens[0])) { + return { name: tokens[0], rest: tokens.slice(1) }; + } + return { name: tokens[0] ?? null, rest: tokens.slice(1) }; +} + +/** `null` when the command is registered, otherwise a loud, actionable error. */ +export function missingCommandError(name) { + const branch = PLANNED[name] ?? PLANNED[`${name} ${""}`.trim()]; + if (branch) { + return new CliError(`command not implemented in this build: ${name}`, { + code: "not-implemented", + remedy: `${name} is delivered by branch ${branch} (see docs/v2/STATUS.md); this branch (ds/01-node-core) ships skill/install/uninstall/doctor only.`, + }); + } + return new CliError(`unknown command: ${name}`, { + code: "unknown-command", + remedy: "run `distilly --help` for the command list.", + }); +} + +/** Terminal columns for one string (CJK counts as two), used for help tables. */export function displayWidth(text) { + let width = 0; + for (const character of text) { + const code = character.codePointAt(0); + const wide = + (code >= 0x1100 && code <= 0x115f) || + (code >= 0x2e80 && code <= 0xa4cf) || + (code >= 0xac00 && code <= 0xd7a3) || + (code >= 0xf900 && code <= 0xfaff) || + (code >= 0xfe30 && code <= 0xfe6f) || + (code >= 0xff00 && code <= 0xff60) || + (code >= 0xffe0 && code <= 0xffe6) || + (code >= 0x20000 && code <= 0x3fffd); + width += wide ? 2 : 1; + } + return width; +} + +/** Pad to `width` terminal columns so bilingual help tables line up. */ +export function padDisplay(text, width) { + const padding = Math.max(1, width - displayWidth(text)); + return `${text}${" ".repeat(padding)}`; +} + +const CATALOG_ZH = [ + ["skill create|update|list|version", "创建 / 更新 / 列出 / 归档 Skill（本分支）"], + ["install ", "把 Distilly 装进某个宿主的 skills 目录（本分支）"], + ["uninstall [|--path]", "卸载已安装的 Distilly（本分支）"], + ["doctor", "宿主与 Skill 库存体检（本分支，最小版）"], +]; + +const CATALOG_EN = [ + ["skill create|update|list|version", "create / update / list / archive Skills (this branch)"], + ["install ", "install Distilly into a host skills directory (this branch)"], + ["uninstall [|--path]", "remove an installed Distilly (this branch)"], + ["doctor", "host and skill inventory health check (this branch, minimal)"], +]; + +/** Bilingual help for the whole CLI (CONTRACT §6: 中文 → `---` → English). */ +export function renderHelp({ version, binary = "distilly" } = {}) { + const implemented = listCommands(); + const plannedNames = Object.keys(PLANNED).sort(); + + const zh = [ + `Distilly ${version}`, + "", + "用法：", + ` ${binary} <命令> [选项]`, + ` ${binary} --help | --version`, + "", + "已实现：", + ...implemented.map((command) => ` ${command.usage.padEnd(46)}${command.summary.split(" / ")[0]}`), + "", + "契约中已冻结、由其他分支交付：", + ` ${plannedNames.join(", ")}`, + "", + "全局选项：", + " --json 以 JSON 回执输出（stdout 只有回执；人读信息走 stderr）", + " --help 显示帮助", + " --version 打印版本", + "", + "示例：", + ` ${binary} skill create --character colleague --name "Zadie Smith" --work work.md --persona persona.md`, + ` ${binary} skill list --character colleague`, + ` ${binary} install claude-code --json`, + ].join("\n"); + + const en = [ + `Distilly ${version}`, + "", + "Usage:", + ` ${binary} [options]`, + ` ${binary} --help | --version`, + "", + "Implemented:", + ...implemented.map((command) => ` ${command.usage.padEnd(46)}${command.summary.split(" / ")[1] ?? command.summary}`), + "", + "Frozen by the contract, delivered by other branches:", + ` ${plannedNames.join(", ")}`, + "", + "Global options:", + " --json emit a JSON receipt (stdout holds only the receipt; prose goes to stderr)", + " --help show this help", + " --version print the package version", + "", + "Examples:", + ` ${binary} skill create --character colleague --name "Zadie Smith" --work work.md --persona persona.md`, + ` ${binary} skill list --character colleague`, + ` ${binary} install claude-code --json`, + ].join("\n"); + + const catalogZh = ["命令总览：", ...CATALOG_ZH.map(([usage, text]) => ` ${usage.padEnd(46)}${text}`)].join("\n"); + const catalogEn = [ + "Command catalog:", + ...CATALOG_EN.map(([usage, text]) => ` ${usage.padEnd(46)}${text}`), + ].join("\n"); + + return `${zh}\n\n${catalogZh}\n\n---\n\n## English\n\n${en}\n\n${catalogEn}\n`; +} + +/** Usage string for one command, built from its registered options. */ +export function renderCommandHelp(command, { binary = "distilly" } = {}) { + return `${command.usage.replace(/^distilly/, binary)}\n\n${command.help ?? ""}`.trimEnd(); +} diff --git a/src/commands/install.mjs b/src/commands/install.mjs new file mode 100644 index 00000000..a42a7488 --- /dev/null +++ b/src/commands/install.mjs @@ -0,0 +1,239 @@ +/** + * `distilly install ` / `distilly uninstall [|--path ]`. + * + * Host directories come from the shared matrix in `src/hosts/agents.mjs`; the + * copy / verify / remove actions live in `src/install/hosts.mjs` (the merged + * port of the eight `tools/install_*.py` scripts). + * + * `--force` replaces an existing install after renaming it to a timestamped + * backup (the pre-v2 CLI's safety behaviour); `--no-backup` deletes instead. + */ + +import { join } from "node:path"; + +import { register } from "./index.mjs"; +import { CliError, createReceipt, describeFile, displayPath, directoryBytes } from "../cli/receipt.mjs"; +import { parseArgs } from "../cli/args.mjs"; +import { listAgents } from "../hosts/agents.mjs"; +import { + HOST_ALIASES, + inspectInstall, + installRepoSkill, + repoInstallDir, + repoProjectDir, + resolveHostId, + supportedHosts, + uninstallRepoSkill, + validateInstallTarget, +} from "../install/hosts.mjs"; + +function installHelp(binary = "distilly") { + const zh = [ + "用法：", + ` ${binary} install [--force] [--dry-run] [--no-backup] [--project]`, + ` ${binary} install --path <以 distilly 结尾的目录> [--force]`, + "", + `宿主 （目录取自 src/hosts/agents.mjs，不猜路径）：`, + ` ${supportedHosts().join(", ")}`, + `别名：${Object.entries(HOST_ALIASES).map(([alias, id]) => `${alias} → ${id}`).join("，")}`, + "", + "选项：", + " --force 覆盖已存在的安装（默认先把旧副本改名成带时间戳的备份）", + " --no-backup --force 时直接删除旧副本，不保留备份", + " --dry-run 只解析目标路径，不写盘", + " --project 装到项目级目录（仅当宿主有文档记载的项目目录）", + " --path 自定义安装目录（最后一级必须叫 distilly）", + "", + `卸载：${binary} uninstall |--path [--force] [--dry-run] [--backup]`, + ].join("\n"); + const en = [ + "Usage:", + ` ${binary} install [--force] [--dry-run] [--no-backup] [--project]`, + ` ${binary} install --path [--force]`, + "", + "Hosts (paths come from src/hosts/agents.mjs, nothing is guessed):", + ` ${supportedHosts().join(", ")}`, + `Aliases: ${Object.entries(HOST_ALIASES).map(([alias, id]) => `${alias} → ${id}`).join(", ")}`, + "", + "Options:", + " --force replace an existing install (the old copy is renamed to a timestamped backup first)", + " --no-backup with --force, delete the old copy instead of keeping a backup", + " --dry-run resolve the target path without writing", + " --project install into the project-local directory (only when the host documents one)", + " --path custom install directory whose final segment is distilly", + "", + `Uninstall: ${binary} uninstall |--path [--force] [--dry-run] [--backup]`, + ].join("\n"); + return { zh, en }; +} + +const INSTALL_OPTIONS = { + path: { type: "string", value: "dir" }, + force: { type: "boolean" }, + "no-backup": { type: "boolean" }, + "dry-run": { type: "boolean" }, + project: { type: "boolean" }, +}; + +const UNINSTALL_OPTIONS = { + path: { type: "string", value: "dir" }, + force: { type: "boolean" }, + "dry-run": { type: "boolean" }, + backup: { type: "boolean" }, + project: { type: "boolean" }, +}; + +/** Resolve the install target from `--path` or a host id. */ +export function resolveTarget(flags, { scope = "global" } = {}) { + if (flags.path) { + try { + return { target: validateInstallTarget(flags.path), host: null, scope }; + } catch (error) { + throw new CliError(error.message, { + code: "unsafe-target", + remedy: "choose a directory whose final segment is `distilly`.", + }); + } + } + if (!flags.host) { + throw new CliError("choose a host or pass --path", { + code: "usage", + remedy: `hosts: ${supportedHosts().join(", ")}`, + }); + } + let host; + try { + host = resolveHostId(flags.host); + } catch (error) { + throw new CliError(error.message, { + code: "unknown-host", + remedy: `hosts: ${supportedHosts().join(", ")}`, + }); + } + const useProject = Boolean(flags.project); + const target = useProject ? repoProjectDir(host) : repoInstallDir(host); + if (target === null) { + throw new CliError(`${host} has no documented project-local directory`, { + code: "no-project-path", + remedy: `install globally instead: distilly install ${host}`, + }); + } + return { target, host, scope: useProject ? "project" : "global" }; +} + +const installCommand = { + summary: "安装到宿主目录 / Install Distilly into a host skills directory", + usage: "distilly install > [--force] [--dry-run]", + options: INSTALL_OPTIONS, + ...installHelp(), + run({ argv, reporter, ctx }) { + const { flags, positionals } = parseArgs(argv, { ...INSTALL_OPTIONS, host: { type: "string" } }); + const host = positionals[0] ?? flags.host; + if (positionals.length > 1) { + throw new CliError(`unexpected argument: ${positionals[1]}`, { code: "usage" }); + } + const { target, scope } = resolveTarget({ ...flags, host }); + const backup = !flags["no-backup"]; + + const report = {}; + let destination = target; + try { + destination = installRepoSkill({ + source: ctx.packageRoot, + destination: target, + force: flags.force, + dryRun: flags["dry-run"], + backup, + report, + }); + } catch (error) { + throw new CliError(error.message, { + code: "install-failed", + remedy: flags.force + ? "check the target directory permissions." + : `rerun with --force to replace ${target} (a backup is kept unless --no-backup is passed).`, + }); + } + + if (flags["dry-run"]) { + reporter.line(`Would install Distilly ${ctx.version} at ${displayPath(destination)}`); + } else { + reporter.line(`Distilly ${ctx.version} installed at ${displayPath(destination)}`); + if (report.backupPath) { + reporter.line(`Previous install preserved at ${displayPath(report.backupPath)}`); + } + } + + const skillFile = describeFile(join(destination, "SKILL.md")); + return { + receipt: { + ...createReceipt("install", { + outputs: skillFile ? [skillFile] : [], + warnings: report.backupPath ? [`previous install preserved at ${displayPath(report.backupPath)}`] : [], + unavailable: listAgents() + .filter((id) => id !== resolveHostId(host ?? "")) + .map((id) => ({ channel: id, reason: "not the selected host" })), + }), + host: host ?? null, + scope, + dry_run: Boolean(flags["dry-run"]), + }, + }; + }, +}; + +const uninstallCommand = { + summary: "卸载 / Remove an installed Distilly from a host directory", + usage: "distilly uninstall [|--path ] [--force] [--dry-run]", + options: UNINSTALL_OPTIONS, + ...installHelp(), + run({ argv, reporter }) { + const { flags, positionals } = parseArgs(argv, { ...UNINSTALL_OPTIONS, host: { type: "string" } }); + const host = positionals[0] ?? flags.host; + const { target, scope } = resolveTarget({ ...flags, host }); + + let result; + try { + result = uninstallRepoSkill({ + destination: target, + force: flags.force, + dryRun: flags["dry-run"], + backup: flags.backup, + }); + } catch (error) { + throw new CliError(error.message, { + code: "uninstall-failed", + remedy: "pass --force only when you are sure this directory is a Distilly install.", + }); + } + + if (flags["dry-run"]) { + reporter.line(`Would remove ${displayPath(result.destination)}`); + } else if (result.backupPath) { + reporter.line(`Removed ${displayPath(result.destination)} (kept at ${displayPath(result.backupPath)})`); + } else { + reporter.line(`Removed ${displayPath(result.destination)}`); + } + + const before = inspectInstall(result.destination); + return { + receipt: { + ...createReceipt("uninstall", { + outputs: result.removed ? [{ path: displayPath(result.destination), bytes: directoryBytes(result.destination) }] : [], + warnings: result.backupPath ? [`kept a copy at ${displayPath(result.backupPath)}`] : [], + }), + host: host ?? null, + scope, + removed: result.removed, + was_installed: before.installed, + }, + }; + }, +}; + +export function registerInstallCommands() { + register("install", installCommand); + register("uninstall", uninstallCommand); +} + +registerInstallCommands(); diff --git a/src/commands/legacy.mjs b/src/commands/legacy.mjs new file mode 100644 index 00000000..8577dc37 --- /dev/null +++ b/src/commands/legacy.mjs @@ -0,0 +1,243 @@ +/** + * `distilly legacy [args...]` — migration adapter (CONTRACT §1). + * + * The pre-v2 command lines + * + * python3 tools/skill_writer.py --action create --slug x --name X … + * python3 tools/version_manager.py --action rollback --slug x --version v1 + * python3 tools/install_generated_skill.py --skill-dir … --host codex + * python3 tools/install_openclaw_skill.py --force + * + * are translated onto `skill create|update|list|version`, `install` and the + * merged installer, with a deprecation warning on stderr and the same exit code + * as the target command. It disappears again in PR③ together with the last + * Python entry point; `SKILL.md` is updated by ds/04-prompts. + */ + +import { existsSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; + +import { register, lookup } from "./index.mjs"; +import { CliError, createReceipt, describeFile, displayPath } from "../cli/receipt.mjs"; +import { ArgError, parseArgs } from "../cli/args.mjs"; +import { expandTargetPath, installGeneratedSkill, installGeneratedSkillForClaude } from "../install/hosts.mjs"; + +const WRITER_ACTIONS = { create: "skill create", update: "skill update", list: "skill list" }; +const VERSION_ACTIONS = ["list", "backup", "rollback", "cleanup"]; + +function deprecation(tool, target, reporter) { + reporter.warn( + `[deprecated] python3 tools/${tool} → ${target}（旧入口在 PR③ 移除 / legacy entry point is removed in PR③）`, + ); +} + +function runRegistered(name, argv, context, reporter) { + const command = lookup(name); + if (!command) { + throw new CliError(`legacy adapter target is not registered: ${name}`, { code: "not-implemented" }); + } + return command.run({ ...context, argv, reporter }); +} + +function splitAction(argv, tool, allowed) { + const flagIndexes = argv + .map((value, index) => (value === "--action" ? index : value.startsWith("--action=") ? index : -1)) + .filter((index) => index !== -1); + if (flagIndexes.length === 0) { + throw new CliError(`legacy ${tool} needs --action <${allowed.join("|")}>`, { code: "usage" }); + } + const index = flagIndexes[0]; + const raw = argv[index]; + const action = raw.includes("=") ? raw.slice(raw.indexOf("=") + 1) : argv[index + 1]; + if (!allowed.includes(action)) { + throw new CliError(`legacy ${tool}: unsupported --action ${action}`, { + code: "usage", + remedy: `supported: ${allowed.join(", ")}`, + }); + } + const rest = raw.includes("=") + ? [...argv.slice(0, index), ...argv.slice(index + 1)] + : [...argv.slice(0, index), ...argv.slice(index + 2)]; + return { action, rest }; +} + +function legacyWriter({ argv, json, reporter, ctx }) { + const { action, rest } = splitAction(argv, "skill_writer.py", Object.keys(WRITER_ACTIONS)); + const target = WRITER_ACTIONS[action]; + deprecation("skill_writer.py", `distilly ${target}`, reporter); + return runRegistered(target, rest, { json, reporter, ctx }, reporter); +} + +function legacyVersionManager({ argv, json, reporter, ctx }) { + const { action, rest } = splitAction(argv, "version_manager.py", VERSION_ACTIONS); + deprecation("version_manager.py", `distilly skill version ${action}`, reporter); + return runRegistered("skill version", [action, ...rest], { json, reporter, ctx }, reporter); +} + +const GENERATED_INSTALL_OPTIONS = { + "skill-dir": { type: "string", value: "dir" }, + host: { type: "string", value: "host" }, + "skills-dir": { type: "string", value: "dir" }, + "claude-skills-dir": { type: "string", value: "dir" }, + "claude-commands-dir": { type: "string", value: "dir" }, + "openclaw-skills-dir": { type: "string", value: "dir" }, + "codex-skills-dir": { type: "string", value: "dir" }, + "install-command-shim": { type: "boolean" }, + force: { type: "boolean" }, + "dry-run": { type: "boolean" }, +}; + +function legacyGeneratedInstaller(tool, host) { + return ({ argv, reporter }) => { + const { flags } = parseArgs(argv, GENERATED_INSTALL_OPTIONS); + if (!flags["skill-dir"]) { + throw new CliError(`legacy ${tool} needs --skill-dir`, { code: "usage" }); + } + const target = `distilly skill create --install-${host}-skill`; + deprecation(tool, target, reporter); + + const claude = host === "claude-code"; + const skillsDir = claude + ? flags["claude-skills-dir"] ?? join(homedir(), ".claude", "skills") + : flags["skills-dir"] ?? + flags[`${host}-skills-dir`] ?? + join(homedir(), host === "openclaw" ? ".openclaw/workspace/skills" : ".agents/skills"); + + const result = claude + ? installGeneratedSkillForClaude({ + skillDir: flags["skill-dir"], + skillsDir, + commandsDir: flags["claude-commands-dir"] ?? join(homedir(), ".claude", "commands"), + force: flags.force, + dryRun: flags["dry-run"], + installCommandShim: flags["install-command-shim"], + }) + : installGeneratedSkill({ + skillDir: flags["skill-dir"], + skillsDir, + force: flags.force, + dryRun: flags["dry-run"], + host, + }); + + reporter.line(result.command_name); + reporter.line(displayPath(result.skill_dir)); + if (result.command_shim_installed && result.command_path) { + reporter.line(displayPath(result.command_path)); + } + + const skillFile = describeFile(join(result.skill_dir, "SKILL.md")); + return { + receipt: createReceipt(`install ${host}`, { + outputs: skillFile ? [skillFile] : [], + warnings: [`deprecated entry point: tools/${tool}`], + }), + }; + }; +} + +const REPO_INSTALL_OPTIONS = { + source: { type: "string", value: "dir" }, + dest: { type: "string", value: "dir" }, + force: { type: "boolean" }, + "dry-run": { type: "boolean" }, +}; + +function legacyRepoInstaller(tool, host) { + return ({ argv, json, reporter, ctx }) => { + const { flags } = parseArgs(argv, REPO_INSTALL_OPTIONS); + deprecation(tool, `distilly install ${host}`, reporter); + const args = []; + if (flags.dest) args.push("--path", expandTargetPath(flags.dest)); + else args.push(host); + if (flags.force) args.push("--force"); + if (flags["dry-run"]) args.push("--dry-run"); + if (flags.source) { + throw new CliError(`legacy ${tool} --source is no longer configurable`, { + code: "unsupported-option", + remedy: "the CLI always installs the running checkout; run it from the source directory instead.", + }); + } + return runRegistered("install", args, { json, reporter, ctx }, reporter); + }; +} + +const LEGACY_TOOLS = { + "skill_writer.py": legacyWriter, + "version_manager.py": legacyVersionManager, + "install_generated_skill.py": legacyGeneratedInstaller("install_generated_skill.py", "codex"), + "install_claude_generated_skill.py": legacyGeneratedInstaller( + "install_claude_generated_skill.py", + "claude-code", + ), + "install_openclaw_generated_skill.py": legacyGeneratedInstaller( + "install_openclaw_generated_skill.py", + "openclaw", + ), + "install_codex_generated_skill.py": legacyGeneratedInstaller( + "install_codex_generated_skill.py", + "codex", + ), + "install_openclaw_skill.py": legacyRepoInstaller("install_openclaw_skill.py", "openclaw"), + "install_codex_skill.py": legacyRepoInstaller("install_codex_skill.py", "codex"), + "install_hermes_skill.py": legacyRepoInstaller("install_hermes_skill.py", "hermes"), +}; + +function legacyHelp(binary = "distilly") { + const tools = Object.keys(LEGACY_TOOLS).sort(); + const zh = [ + "用法（迁移期）：", + ` ${binary} legacy [--action ...] [options]`, + "", + "支持的旧入口（转发到新子命令，并向 stderr 打印弃用警告）：", + ...tools.map((tool) => ` tools/${tool}`), + "", + "示例：", + ` ${binary} legacy skill_writer.py --action create --slug eulalie --name Eulalie`, + ` ${binary} legacy version_manager.py --action rollback --slug eulalie --version v1`, + ].join("\n"); + const en = [ + "Usage (migration window):", + ` ${binary} legacy [--action ...] [options]`, + "", + "Supported legacy entry points (forwarded, with a deprecation warning on stderr):", + ...tools.map((tool) => ` tools/${tool}`), + "", + "Examples:", + ` ${binary} legacy skill_writer.py --action create --slug eulalie --name Eulalie`, + ` ${binary} legacy version_manager.py --action rollback --slug eulalie --version v1`, + ].join("\n"); + return { zh, en }; +} + +register("legacy", { + summary: "旧 python3 tools/*.py 入口转发 / Forward legacy python3 tools/*.py calls", + usage: "distilly legacy [--action ...] [options]", + hidden: true, + ...legacyHelp(), + run(context) { + const [tool, ...rest] = context.argv; + if (!tool) { + throw new CliError("legacy needs the tool name", { + code: "usage", + remedy: `supported: ${Object.keys(LEGACY_TOOLS).sort().join(", ")}`, + }); + } + const normalized = tool.replace(/^\.?\/?(tools\/)?/, ""); + const handler = LEGACY_TOOLS[normalized]; + if (!handler) { + throw new CliError(`no legacy adapter for ${tool}`, { + code: "unknown-tool", + remedy: + "collectors and research tools stay in Python until ds/07; use the new CLI for skill/install/uninstall/doctor.", + }); + } + if (!existsSync(join(context.ctx.packageRoot, "tools", normalized))) { + // The Python file is gone on this branch — forwarding is the whole point. + } + return handler({ ...context, argv: rest }); + }, +}); + +export { ArgError, LEGACY_TOOLS }; diff --git a/src/commands/skill.mjs b/src/commands/skill.mjs new file mode 100644 index 00000000..7525a237 --- /dev/null +++ b/src/commands/skill.mjs @@ -0,0 +1,480 @@ +/** + * `distilly skill ...` — create / update / list / archive generated Skills. + * + * This module owns the argument surface and the *user-visible* behaviour; the + * work lives in `src/skill/*`, the direct port of `tools/skill_presets.py`, + * `tools/skill_schema.py`, `tools/skill_writer.py` and `tools/version_manager.py`. + * + * Human output is byte-identical to the Python CLI (verified by + * `scripts/parity.mjs`, phase B); `--json` answers with the CONTRACT §3 receipt + * instead, so machine output never has to parse prose. + */ + +import { existsSync, readFileSync, statSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; + +import { register } from "./index.mjs"; +import { CliError, createReceipt, describeFile, displayPath } from "../cli/receipt.mjs"; +import { parseArgs } from "../cli/args.mjs"; +import { getCharacterPreset, normalizeCharacter, normalizeResearchProfile, resolveExistingStorageRoot } from "../skill/presets.mjs"; +import { resolveContainedChild, validatePathSegment } from "../skill/schema.mjs"; +import { + createSkill, + installGeneratedHosts, + listSkills, + resolveBaseDir, + slugify, + updateSkill, + validateSlug, +} from "../skill/writer.mjs"; +import { SlugResolutionError } from "../skill/slug.mjs"; +import { + backupCurrentVersion, + cleanupOldVersions, + listVersions, + rollback, + MAX_VERSIONS, +} from "../skill/versions.mjs"; + +export function skillHelp(binary = "distilly") { + const zh = [ + "用法：", + ` ${binary} skill create --character [--slug |--name ]`, + " [--meta ] [--work ] [--persona ]", + " [--base-dir ] [--research-profile ]", + " [--install-claude-skill] [--install-openclaw-skill] [--install-codex-skill]", + ` ${binary} skill update --slug [--character ] [--base-dir ]`, + " [--work-patch ] [--persona-patch ] [--correction-json ]", + ` ${binary} skill list [--character ] [--base-dir ]`, + ` ${binary} skill version --slug [--version ]`, + "", + "说明：", + " create 写出 SKILL.md / work.md / persona.md / work_skill.md / persona_skill.md / manifest.json / meta.json。", + " 不带 --slug 时用 --name 生成拼音 slug（Unihan 表）；表缺失或字符未覆盖时明确失败并要求 --slug。", + " update 先把当前产物归档到 versions/<当前版本>/，版本号 +1。", + " version 管理归档：list / backup / rollback / cleanup（默认保留最近 10 个）。", + ].join("\n"); + const en = [ + "Usage:", + ` ${binary} skill create --character [--slug |--name ]`, + " [--meta ] [--work ] [--persona ]", + " [--base-dir ] [--research-profile ]", + " [--install-claude-skill] [--install-openclaw-skill] [--install-codex-skill]", + ` ${binary} skill update --slug [--character ] [--base-dir ]`, + " [--work-patch ] [--persona-patch ] [--correction-json ]", + ` ${binary} skill list [--character ] [--base-dir ]`, + ` ${binary} skill version --slug [--version ]`, + "", + "Notes:", + " create writes SKILL.md / work.md / persona.md / work_skill.md / persona_skill.md / manifest.json / meta.json.", + " Without --slug the slug is derived from --name through the Unihan pinyin table; a missing table or an uncovered character fails loudly and asks for --slug.", + " update archives the current artifacts under versions// first, then bumps the version.", + " version manages the archive: list / backup / rollback / cleanup (keeps the newest 10 by default).", + ].join("\n"); + return { zh, en }; +} + +const INSTALL_OPTIONS = { + "install-claude-skill": { type: "boolean" }, + "no-install-claude-skill": { type: "boolean" }, + "install-claude-command-shim": { type: "boolean" }, + "claude-skills-dir": { type: "string", value: "dir" }, + "claude-commands-dir": { type: "string", value: "dir" }, + "install-openclaw-skill": { type: "boolean" }, + "openclaw-skills-dir": { type: "string", value: "dir" }, + "install-codex-skill": { type: "boolean" }, + "codex-skills-dir": { type: "string", value: "dir" }, +}; + +const SELECT_OPTIONS = { + character: { type: "string", alias: "c", value: "family" }, + type: { type: "string", value: "family" }, + "base-dir": { type: "string", value: "dir" }, +}; + +const CREATE_OPTIONS = { + ...SELECT_OPTIONS, + ...INSTALL_OPTIONS, + slug: { type: "string", value: "slug" }, + name: { type: "string", value: "name" }, + meta: { type: "string", value: "file" }, + work: { type: "string", value: "file" }, + persona: { type: "string", value: "file" }, + "research-profile": { type: "string", value: "name" }, +}; + +const UPDATE_OPTIONS = { + ...SELECT_OPTIONS, + ...INSTALL_OPTIONS, + slug: { type: "string", value: "slug" }, + "work-patch": { type: "string", value: "file" }, + "persona-patch": { type: "string", value: "file" }, + "correction-json": { type: "string", value: "file" }, +}; + +const VERSION_OPTIONS = { + ...SELECT_OPTIONS, + slug: { type: "string", value: "slug" }, + version: { type: "string", value: "vN" }, + "max-versions": { type: "string", value: "N" }, +}; + +function readTextFlag(value, label) { + try { + return readFileSync(value, "utf8"); + } catch (error) { + throw new CliError(`cannot read ${label}: ${value}`, { + code: "missing-input", + remedy: error.message, + }); + } +} + +function metaInputs(paths) { + return paths.filter(Boolean).map((path) => describeFile(path)).filter(Boolean); +} + +function artifactOutputs(skillDir) { + const names = [ + "SKILL.md", + "work.md", + "persona.md", + "work_skill.md", + "persona_skill.md", + "manifest.json", + "meta.json", + ]; + return names + .map((name) => describeFile(join(skillDir, name))) + .filter(Boolean); +} + +function resolveSkillDir(baseDir, slug, label = "skill slug") { + let skillDir; + try { + skillDir = resolveContainedChild(baseDir, slug, label); + } catch (error) { + throw new CliError(error.message, { + code: "unsafe-path", + remedy: "pass a single safe directory name (no separators, no '..').", + }); + } + if (!existsSync(skillDir)) { + throw new CliError(`skill directory not found: ${skillDir}`, { + code: "missing-skill", + remedy: "run `distilly skill list` to see the skills in this storage root.", + }); + } + return skillDir; +} + +const createCommand = { + summary: "创建 Skill / Create a Skill", + usage: "distilly skill create [options]", + options: CREATE_OPTIONS, + ...skillHelp(), + run({ argv, json, reporter }) { + const { flags } = parseArgs(argv, CREATE_OPTIONS); + const requestedCharacter = normalizeCharacter(flags.character || flags.type); + + const autoInstallSetting = + process.env.DISTILLY_AUTO_INSTALL_CLAUDE ?? process.env.DOT_SKILL_AUTO_INSTALL_CLAUDE; + const autoInstallDefault = autoInstallSetting !== undefined && autoInstallSetting !== "0"; + const installClaudeSkill = + (flags["install-claude-skill"] || autoInstallDefault) && !flags["no-install-claude-skill"]; + + const meta = flags.meta ? JSON.parse(readTextFlag(flags.meta, "meta JSON")) : {}; + if (flags.name) { + meta.name = flags.name; + meta.display_name = flags.name; + } + meta.character = normalizeCharacter(meta.character ?? meta.type ?? requestedCharacter); + meta.research_profile = normalizeResearchProfile( + meta.character, + flags["research-profile"] || meta.research_profile, + ); + meta.type = meta.type || meta.character; + + const baseDir = resolveBaseDir(flags["base-dir"], requestedCharacter); + let slug; + try { + slug = flags.slug + ? validateSlug(flags.slug) + : slugify(meta.display_name ?? meta.name ?? "person"); + } catch (error) { + if (error instanceof SlugResolutionError) { + throw new CliError(error.message, { code: error.code, remedy: error.remedy }); + } + throw new CliError(error.message, { + code: "invalid-slug", + remedy: "pass --slug (1-40 lowercase letters/digits).", + }); + } + + const workContent = flags.work ? readTextFlag(flags.work, "work.md") : ""; + const personaContent = flags.persona ? readTextFlag(flags.persona, "persona.md") : ""; + + const skillDir = createSkill(baseDir, slug, meta, workContent, personaContent); + + reporter.line(`Created skill: ${displayPath(skillDir)}`); + reporter.line(" Kind: meta-skill"); + reporter.line(` Character: ${meta.character}`); + reporter.line(` Research Profile: ${meta.research_profile}`); + reporter.line(` Preset: ${meta.preset ?? "auto"}`); + + const installLines = installGeneratedHosts( + skillDir, + { + claudeSkillsDir: flags["claude-skills-dir"], + claudeCommandsDir: flags["claude-commands-dir"], + installClaudeCommandShim: flags["install-claude-command-shim"], + openclawSkillsDir: flags["openclaw-skills-dir"], + installOpenclawSkill: flags["install-openclaw-skill"], + codexSkillsDir: flags["codex-skills-dir"], + installCodexSkill: flags["install-codex-skill"], + }, + installClaudeSkill, + ); + if (installLines.length > 0) for (const line of installLines) reporter.line(line); + else reporter.line(" Host installs: skipped"); + + const outputs = artifactOutputs(skillDir); + const warnings = []; + if (!json && flags.slug === undefined && slug) { + warnings.push(`slug derived from --name: ${slug}`); + } + return { + receipt: createReceipt("skill create", { + person: slug, + inputs: metaInputs([flags.meta, flags.work, flags.persona]), + outputs, + warnings, + }), + }; + }, +}; + +const updateCommand = { + summary: "更新 Skill / Update a Skill", + usage: "distilly skill update [options]", + options: UPDATE_OPTIONS, + ...skillHelp(), + run({ argv, reporter }) { + const { flags } = parseArgs(argv, UPDATE_OPTIONS); + const requestedCharacter = normalizeCharacter(flags.character || flags.type); + + let slug; + try { + slug = validatePathSegment(flags.slug ?? "", "existing slug"); + } catch (error) { + throw new CliError(error.message, { + code: "unsafe-slug", + remedy: "pass --slug .", + }); + } + + const baseDir = resolveExistingStorageRoot(requestedCharacter, slug, flags["base-dir"]); + const skillDir = resolveSkillDir(baseDir, slug); + + const workPatch = flags["work-patch"] ? readTextFlag(flags["work-patch"], "work patch") : null; + const personaPatch = flags["persona-patch"] + ? readTextFlag(flags["persona-patch"], "persona patch") + : null; + const correction = flags["correction-json"] + ? JSON.parse(readTextFlag(flags["correction-json"], "correction JSON")) + : null; + + const newVersion = updateSkill(skillDir, workPatch, personaPatch, correction); + reporter.line(`Updated skill to ${newVersion}: ${displayPath(skillDir)}`); + + const autoInstallSetting = + process.env.DISTILLY_AUTO_INSTALL_CLAUDE ?? process.env.DOT_SKILL_AUTO_INSTALL_CLAUDE; + const autoInstallDefault = autoInstallSetting !== undefined && autoInstallSetting !== "0"; + const installClaudeSkill = + (flags["install-claude-skill"] || autoInstallDefault) && !flags["no-install-claude-skill"]; + const installLines = installGeneratedHosts( + skillDir, + { + claudeSkillsDir: flags["claude-skills-dir"], + claudeCommandsDir: flags["claude-commands-dir"], + installClaudeCommandShim: flags["install-claude-command-shim"], + openclawSkillsDir: flags["openclaw-skills-dir"], + installOpenclawSkill: flags["install-openclaw-skill"], + codexSkillsDir: flags["codex-skills-dir"], + installCodexSkill: flags["install-codex-skill"], + }, + installClaudeSkill, + ); + for (const line of installLines) reporter.line(line); + + return { + receipt: createReceipt("skill update", { + person: slug, + inputs: metaInputs([ + flags["work-patch"], + flags["persona-patch"], + flags["correction-json"], + join(skillDir, "meta.json"), + ]), + outputs: artifactOutputs(skillDir), + warnings: [], + }), + }; + }, +}; + +const listCommand = { + summary: "列出已有 Skill / List generated Skills", + usage: "distilly skill list [options]", + options: SELECT_OPTIONS, + ...skillHelp(), + run({ argv, reporter }) { + const { flags } = parseArgs(argv, SELECT_OPTIONS); + const requestedCharacter = normalizeCharacter(flags.character || flags.type); + const baseDir = resolveExistingStorageRoot(requestedCharacter, null, flags["base-dir"]); + const skills = listSkills(baseDir); + + if (skills.length === 0) { + const preset = getCharacterPreset(requestedCharacter); + reporter.line(`No ${preset.character} skills found`); + } else { + reporter.line(`Found ${skills.length} skills:`); + reporter.line(""); + for (const skill of skills) { + const updated = skill.updated_at ? skill.updated_at.slice(0, 10) : "unknown"; + reporter.line(` [${skill.slug}] ${skill.name} — ${skill.identity}`); + reporter.line( + ` Kind: ${skill.kind} Character: ${skill.character} ` + + `Research Profile: ${skill.research_profile} ` + + `Version: ${skill.version} ` + + `Corrections: ${skill.corrections_count} Updated: ${updated}`, + ); + reporter.line(""); + } + } + + const outputs = skills + .map((skill) => describeFile(join(baseDir, skill.slug, "SKILL.md"))) + .filter(Boolean); + return { + receipt: createReceipt("skill list", { + outputs, + warnings: skills.length === 0 ? [`no skills found in ${baseDir}`] : [], + }), + }; + }, +}; + +const versionCommand = { + summary: "版本归档 / Archive, roll back and prune Skill versions", + usage: "distilly skill version [options]", + options: VERSION_OPTIONS, + ...skillHelp(), + run({ argv, reporter }) { + const { flags, positionals } = parseArgs(argv, VERSION_OPTIONS); + const action = positionals[0] ?? "list"; + if (!["list", "backup", "rollback", "cleanup"].includes(action)) { + throw new CliError(`unknown skill version action: ${action}`, { + code: "usage", + remedy: "choose one of: list, backup, rollback, cleanup.", + }); + } + + const requestedCharacter = normalizeCharacter(flags.character || flags.type); + let slug; + try { + slug = validatePathSegment(flags.slug ?? "", "skill slug"); + } catch (error) { + throw new CliError(error.message, { + code: "unsafe-slug", + remedy: "pass --slug .", + }); + } + const baseDir = resolveExistingStorageRoot(requestedCharacter, slug, flags["base-dir"]); + const skillDir = resolveSkillDir(baseDir, slug); + + let outputs = []; + if (action === "list") { + const versions = listVersions(skillDir); + if (versions.length === 0) { + reporter.line(`no archived versions for ${slug}`); + } else { + reporter.line(`archived versions for ${slug}:`); + reporter.line(""); + for (const version of versions) { + reporter.line( + ` ${version.version} archived: ${version.archived_at} files: ${version.files.join(", ")}`, + ); + } + } + outputs = versions + .map((version) => describeFile(join(version.path, "SKILL.md"))) + .filter(Boolean); + } else if (action === "backup") { + if (!backupCurrentVersion(skillDir)) { + throw new CliError(`could not archive the current version of ${slug}`, { + code: "archive-failed", + remedy: "make sure meta.json exists in the skill directory.", + }); + } + outputs = artifactOutputs(skillDir); + } else if (action === "rollback") { + if (!flags.version) { + throw new CliError("rollback requires --version", { + code: "usage", + remedy: "run `distilly skill version list --slug ` and pass --version .", + }); + } + if (!rollback(skillDir, flags.version)) { + throw new CliError(`rollback to ${flags.version} failed`, { + code: "rollback-failed", + remedy: `check the archive list for ${slug}.`, + }); + } + outputs = artifactOutputs(skillDir); + } else { + const maxVersions = flags["max-versions"] ? Number.parseInt(flags["max-versions"], 10) : MAX_VERSIONS; + if (!cleanupOldVersions(skillDir, maxVersions)) { + throw new CliError(`cleanup failed for ${slug}`, { code: "cleanup-failed" }); + } + reporter.line("cleanup complete"); + outputs = []; + } + + return { + receipt: createReceipt("skill version", { + person: slug, + inputs: [describeFile(join(skillDir, "meta.json"))].filter(Boolean), + outputs, + warnings: [], + }), + }; + }, +}; + +export function registerSkillCommands() { + register("skill", { + summary: "Skill 子命令入口 / Skill subcommand entry", + usage: "distilly skill [options]", + ...skillHelp(), + run({ argv, reporter }) { + if (argv.length === 0) { + reporter.line(skillHelp().zh); + return { receipt: createReceipt("skill", { warnings: [] }) }; + } + throw new CliError(`unknown skill subcommand: ${argv[0]}`, { + code: "usage", + remedy: "choose one of: create, update, list, version.", + }); + }, + }); + register("skill create", createCommand); + register("skill update", updateCommand); + register("skill list", listCommand); + register("skill version", versionCommand); +} + +registerSkillCommands(); + +export { homedir, statSync }; diff --git a/src/consent.mjs b/src/consent.mjs new file mode 100644 index 00000000..b9d27bdf --- /dev/null +++ b/src/consent.mjs @@ -0,0 +1,513 @@ +/** + * consent.mjs — the computer-use consent gate. + * + * Some channels can only be collected by *acting as the user* inside a real + * browser session (X search, DingTalk message history, …). That is a capability + * question, not a prompt question: the collectors in `src/collect/*` refuse to + * run in `--mode browser` without a token a human explicitly granted. + * + * The store is `~/.distilly/consent.json` (`$DISTILLY_HOME/consent.json`), mode + * 0600: + * + * { + * "version": 1, + * "grants": [ + * { + * "token": "dsc_3f0c…", // capability, never an API key + * "scope": "collect:x:browser", + * "granted_at": "2026-09-13T02:20:00.000Z", + * "expires_at": "2026-09-14T02:20:00.000Z", + * "note": "collect my own timeline" + * } + * ] + * } + * + * Rules this module enforces mechanically: + * + * - **No token, no run.** `verify()` is the only way in; an absent, unknown , + * expired or scope-mismatched token yields `ok: false` plus the remediation + * the caller must print (CLI exit code 2, receipt status + * `waiting-for-user-consent`). + * - **Grants expire.** `expires_at` is mandatory; `--ttl` is minutes and must + * be positive. There is no "forever" grant. + * - **The store is the only thing written.** Nothing else in this module + * touches the filesystem, and the file is written atomically at 0600. + * + * A consent token is a *capability*, not an API key: it authorises one scope on + * this machine and can be revoked at any time. It is still written nowhere + * except this store — the ledger records `consent_token_sha256_12`, a hash, so + * that an audit trail cannot be replayed as a grant. + */ + +import { randomBytes, createHash } from "node:crypto"; +import { + existsSync, + mkdirSync, + readFileSync, + renameSync, + rmSync, + statSync, + writeFileSync, + chmodSync, +} from "node:fs"; +import { homedir } from "node:os"; +import { dirname, join, resolve } from "node:path"; + +/** File name inside `$DISTILLY_HOME`. Printed in errors; never its contents. */ +export const CONSENT_FILE = "consent.json"; +export const CONSENT_VERSION = 1; + +/** Default grant lifetime: 24 hours. Minutes are the CLI unit. */ +export const DEFAULT_TTL_MINUTES = 24 * 60; + +/** Exit code the contract reserves for "waiting for user consent". */ +export const EXIT_CONSENT_REQUIRED = 2; + +const TOKEN_PREFIX = "dsc_"; +const SCOPE_PATTERN = /^[a-z0-9][a-z0-9:._-]*$/i; + +/** `$DISTILLY_HOME` wins over `~/.distilly`, so tests never touch the real home. */ +export function distillyHome(env = process.env) { + const override = env?.DISTILLY_HOME; + return override ? resolve(String(override)) : join(homedir(), ".distilly"); +} + +export function consentPath(env = process.env) { + return join(distillyHome(env), CONSENT_FILE); +} + +function emptyState() { + return { version: CONSENT_VERSION, grants: [] }; +} + +function normaliseState(raw) { + if (!raw || typeof raw !== "object" || Array.isArray(raw)) return emptyState(); + const grants = Array.isArray(raw.grants) ? raw.grants.filter((g) => g && typeof g === "object") : []; + return { version: CONSENT_VERSION, grants }; +} + +/** + * Read the store. A missing file is "no grants ever issued", not an error; a + * corrupt file *is* an error, because silently forgetting a grant is safe while + * silently inventing one is not. + */ +export function readConsent(env = process.env) { + const path = consentPath(env); + if (!existsSync(path)) return emptyState(); + let text; + try { + text = readFileSync(path, "utf8"); + } catch (error) { + throw new Error(`cannot read ${CONSENT_FILE}: ${error.message}`, { cause: error }); + } + try { + return normaliseState(JSON.parse(text)); + } catch (error) { + throw new Error( + `${CONSENT_FILE} is not valid JSON; rerun \`distilly consent grant --scope \` to rewrite it`, + { cause: error }, + ); + } +} + +/** Atomic write at 0600: staged in the same directory, then renamed. */ +export function writeConsent(state, env = process.env) { + const path = consentPath(env); + mkdirSync(dirname(path), { recursive: true }); + const body = `${JSON.stringify(normaliseState(state), null, 2)}\n`; + const staging = `${path}.${process.pid}.tmp`; + try { + writeFileSync(staging, body, { mode: 0o600 }); + chmodSync(staging, 0o600); + renameSync(staging, path); + } catch (error) { + if (existsSync(staging)) rmSync(staging, { force: true }); + throw error; + } + return { path, bytes: Buffer.byteLength(body), sha256: sha256Text(body) }; +} + +function sha256Text(text) { + return createHash("sha256").update(Buffer.from(text, "utf8")).digest("hex"); +} + +export function assertScope(scope) { + if (typeof scope !== "string" || !SCOPE_PATTERN.test(scope)) { + throw new TypeError( + `invalid scope ${JSON.stringify(scope)}; expected something like "collect:x:browser"`, + ); + } + return scope; +} + +export function isExpired(record, now = new Date()) { + const ms = Date.parse(record?.expires_at ?? ""); + if (!Number.isFinite(ms)) return true; + return ms <= now.getTime(); +} + +export function consentStatus(record, now = new Date()) { + return isExpired(record, now) ? "expired" : "active"; +} + +/** + * Issue a grant. + * + * @param {string} scope e.g. `collect:x:browser` + * @param {{env?: object, now?: Date, ttlMinutes?: number, note?: string, token?: string}} [options] + * `token` exists so tests can pin a deterministic value. + */ +export function grant(scope, options = {}) { + const { env = process.env, now = new Date(), ttlMinutes = DEFAULT_TTL_MINUTES, note, token } = options; + assertScope(scope); + if (!Number.isFinite(ttlMinutes) || ttlMinutes <= 0) { + throw new TypeError("--ttl must be a positive number of minutes"); + } + const value = token ?? `${TOKEN_PREFIX}${randomBytes(16).toString("hex")}`; + const record = { + token: value, + scope, + granted_at: now.toISOString(), + expires_at: new Date(now.getTime() + ttlMinutes * 60_000).toISOString(), + }; + if (note) record.note = String(note); + + const state = readConsent(env); + state.grants = [...state.grants.filter((g) => g.token !== value), record]; + const written = writeConsent(state, env); + return { record, written, store: consentPath(env) }; +} + +/** + * Verify a token for `scope`. + * + * @returns {{ok: boolean, reason: string, remediation: string[], record: object|null, + * status: 'granted'|'waiting-for-user-consent'}} + */ +export function verify(token, options = {}) { + const { env = process.env, scope, now = new Date() } = options; + const wanted = scope ? assertScope(scope) : null; + const remediation = (target) => { + const cmd = `distilly consent grant --scope ${target}`; + return [ + `run: ${cmd}`, + `then pass the printed token: distilly collect x --mode browser --consent `, + ]; + }; + + if (!token) { + return { + ok: false, + reason: "no-token", + status: "waiting-for-user-consent", + record: null, + remediation: remediation(wanted ?? "collect:x:browser"), + }; + } + + let state; + try { + state = readConsent(env); + } catch (error) { + return { + ok: false, + reason: "unreadable-store", + status: "waiting-for-user-consent", + record: null, + remediation: [`fix or remove ${consentPath(env)} (${error.message})`], + }; + } + + const record = state.grants.find((g) => g.token === token) ?? null; + if (!record) { + return { + ok: false, + reason: "unknown-token", + status: "waiting-for-user-consent", + record: null, + remediation: remediation(wanted ?? "collect:x:browser"), + }; + } + if (wanted && record.scope !== wanted) { + return { + ok: false, + reason: "scope-mismatch", + status: "waiting-for-user-consent", + record, + remediation: [ + `token is for scope "${record.scope}", not "${wanted}"`, + ...remediation(wanted), + ], + }; + } + if (isExpired(record, now)) { + return { + ok: false, + reason: "expired", + status: "waiting-for-user-consent", + record, + remediation: [`grant expired at ${record.expires_at}`, ...remediation(record.scope)], + }; + } + return { ok: true, reason: "granted", status: "granted", record, remediation: [] }; +} + +/** Revoke one token, or every token (`"all"`), or a whole scope. */ +export function revoke(target, options = {}) { + const { env = process.env } = options; + if (!target) throw new TypeError("revoke needs a token, a scope, or \"all\""); + const state = readConsent(env); + let revoked; + if (target === "all") { + revoked = state.grants; + state.grants = []; + } else { + revoked = state.grants.filter((g) => g.token === target || g.scope === target); + state.grants = state.grants.filter((g) => g.token !== target && g.scope !== target); + } + const written = revoked.length > 0 ? writeConsent(state, env) : null; + return { revoked, remaining: state.grants.length, written, store: consentPath(env) }; +} + +/** All grants with a live `status`, newest first. */ +export function list(options = {}) { + const { env = process.env, now = new Date() } = options; + const state = readConsent(env); + return state.grants + .map((record) => ({ + token: record.token, + scope: record.scope, + granted_at: record.granted_at, + expires_at: record.expires_at, + note: record.note ?? null, + status: consentStatus(record, now), + })) + .sort((a, b) => String(b.granted_at).localeCompare(String(a.granted_at))); +} + +/** Drop expired grants. Never touches active ones. */ +export function prune(options = {}) { + const { env = process.env, now = new Date() } = options; + const state = readConsent(env); + const expired = state.grants.filter((g) => isExpired(g, now)); + if (expired.length > 0) { + state.grants = state.grants.filter((g) => !isExpired(g, now)); + writeConsent(state, env); + } + return { removed: expired.map((g) => g.token), remaining: state.grants.length }; +} + +/** + * Turn a failed `verify()` into the receipt fragment every caller must emit. + * Keeping it here means `collect`, `transcribe` and future hosts all say the + * same thing, in both languages. + */ +export function consentUnavailable(verification, channel) { + return [ + { + channel, + reason: `waiting for user consent: ${verification.reason}`, + scope: verification.record?.scope ?? null, + remediation: verification.remediation, + }, + ]; +} + +/** Stable, non-reversible fingerprint used when the ledger records a grant. */ +export function consentTokenFingerprint(token) { + if (!token) return null; + return sha256Text(String(token)).slice(0, 12); +} + +// ─── CLI ───────────────────────────────────────────────────────────────────── + +const HELP = `distilly consent — computer-use 同意门 / consent gate + +用法 (zh): + distilly consent grant --scope collect:x:browser [--ttl <分钟>] [--note <说明>] [--json] + distilly consent list [--json] + distilly consent verify --token [--scope ] [--json] + distilly consent revoke [--json] + distilly consent prune [--json] + +说明：授权写入 ~/.distilly/consent.json（0600）。浏览器模式采集必须带 --consent ； +无 token 或已过期时命令以 exit 2 退出，并在回执里写“等待用户同意”。 + +--- +## English + distilly consent grant --scope collect:x:browser [--ttl ] [--note ] [--json] + distilly consent list | verify --token [--scope ] | revoke | prune + +Grants live in ~/.distilly/consent.json (0600). Browser-mode collection requires +--consent ; a missing or expired token exits with code 2 and a receipt +that says the run is waiting for user consent. +`; + +function parseFlags(argv) { + const flags = { _: [] }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--json") flags.json = true; + else if (arg === "--all") flags.all = true; + else if (arg === "--help" || arg === "-h") flags.help = true; + else if (arg.startsWith("--")) { + const name = arg.slice(2); + const value = argv[index + 1]; + if (value === undefined || value.startsWith("--")) { + throw new Error(`${arg} requires a value`); + } + flags[name] = value; + index += 1; + } else flags._.push(arg); + } + return flags; +} + +/** + * @param {string[]} argv arguments after `consent` + * @param {{env?: object, now?: Date, stdout?: Function, stderr?: Function}} [io] + * @returns {number} process exit code + */ +export function runConsentCli(argv, io = {}) { + const env = io.env ?? process.env; + const now = io.now ?? new Date(); + const out = io.stdout ?? ((line) => process.stdout.write(`${line}\n`)); + const err = io.stderr ?? ((line) => process.stderr.write(`${line}\n`)); + + let flags; + try { + flags = parseFlags(argv); + } catch (error) { + err(`Error: ${error.message}`); + return 1; + } + + const [action] = flags._; + if (!action || action === "help" || flags.help) { + out(HELP); + return 0; + } + + const emit = (receipt) => { + if (flags.json) out(JSON.stringify(receipt, null, 2)); + return receipt; + }; + + try { + if (action === "grant") { + const scope = flags.scope ?? "collect:x:browser"; + const ttlMinutes = flags.ttl === undefined ? DEFAULT_TTL_MINUTES : Number(flags.ttl); + const { record, written } = grant(scope, { env, now, ttlMinutes, note: flags.note }); + emit({ + command: "consent", + action: "grant", + ok: true, + inputs: [], + outputs: [{ path: written.path, sha256: written.sha256, bytes: written.bytes }], + grants: [ + { + token: record.token, + scope: record.scope, + granted_at: record.granted_at, + expires_at: record.expires_at, + }, + ], + warnings: [], + unavailable: [], + }); + out(`granted ${record.scope} until ${record.expires_at}`); + out(`token: ${record.token}`); + out(`use it with: distilly collect x --mode browser --consent ${record.token}`); + return 0; + } + + if (action === "list") { + const grants = list({ env, now }); + emit({ + command: "consent", + action: "list", + ok: true, + inputs: [], + outputs: [], + grants, + warnings: grants.filter((g) => g.status === "expired").map((g) => `expired grant for ${g.scope}`), + unavailable: [], + }); + if (grants.length === 0) out(`no grants in ${consentPath(env)}`); + for (const g of grants) out(`${g.status}\t${g.scope}\t${g.token}\t${g.expires_at}`); + return 0; + } + + if (action === "verify") { + const scope = flags.scope; + const result = verify(flags.token, { env, scope, now }); + const receipt = { + command: "consent", + action: "verify", + ok: result.ok, + status: result.status, + inputs: [], + outputs: [], + grants: result.record + ? [ + { + scope: result.record.scope, + granted_at: result.record.granted_at, + expires_at: result.record.expires_at, + }, + ] + : [], + warnings: [], + unavailable: result.ok ? [] : consentUnavailable(result, scope ?? "any"), + }; + emit(receipt); + if (result.ok) out(`granted\t${result.record.scope}\texpires ${result.record.expires_at}`); + else { + err(`waiting for user consent (${result.reason})`); + for (const step of result.remediation) err(` fix: ${step}`); + } + return result.ok ? 0 : EXIT_CONSENT_REQUIRED; + } + + if (action === "revoke") { + const target = flags.all ? "all" : flags._[1] ?? flags.token; + const { revoked, remaining, written } = revoke(target, { env }); + emit({ + command: "consent", + action: "revoke", + ok: true, + inputs: [], + outputs: written ? [{ path: written.path, sha256: written.sha256, bytes: written.bytes }] : [], + grants: [], + warnings: revoked.length === 0 ? [`nothing matched ${target}`] : [], + unavailable: [], + }); + out(`revoked ${revoked.length} grant(s) for ${target}; ${remaining} remaining`); + return 0; + } + + if (action === "prune") { + const { removed, remaining } = prune({ env, now }); + emit({ + command: "consent", + action: "prune", + ok: true, + inputs: [], + outputs: [], + grants: [], + warnings: [], + unavailable: [], + removed: removed.length, + }); + out(`removed ${removed.length} expired grant(s); ${remaining} remaining`); + return 0; + } + } catch (error) { + err(`Error: ${error.message}`); + return 1; + } + + err(`Error: unknown consent action: ${action}`); + err("run `distilly consent --help` for the usage"); + return 1; +} diff --git a/src/derive/fixtures/synthetic-group/README.md b/src/derive/fixtures/synthetic-group/README.md new file mode 100644 index 00000000..b90336f7 --- /dev/null +++ b/src/derive/fixtures/synthetic-group/README.md @@ -0,0 +1,29 @@ +# `synthetic-group` — retrospect 合成账本夹具 + +**不是真人对话，也不是真实导出。** 全部内容为本任务原创，随仓库以 MIT 发布， +只用来给 `tests/retrospect.test.mjs` 提供一份"形状正确、特征已知"的账本。 + +## 它必须能触发的特征 + +| 特征 | 在哪里 | 用途 | +| --- | --- | --- | +| 4 个说话人 | 老周 / 小陈 / 林工 / 阿May | `stats.participant_*`、`relations.*` | +| 71 条消息 | `group-chat.md` 44 + `dm-lin-chen.md` 26 + `incident-postmortem.md` 1 | 各维度的样本量 | +| 语调/长度突变 | `k0001:t21`–`k0001:t34`（故障期间消息明显变长、带感叹号） | `shifts.*` | +| 话题回避 | `k0001:t18`→`t19`（被问 offer，一句"先不说这个"转开）、`k0001:t31`→`t32`（"这个不方便说"） | `boundaries.*` | +| 同一维度取值相反 | `k0001:t5`（"远程办公挺好的"）vs `k0001:t38`（"远程办公其实很烦"）；`k0001:t9`（"这个方案不行"）vs `k0001:t12`（"这个方案我觉得没问题"）；`k0001:t29`（"保证不会再有第二次"）vs `k0001:t33`（"可能还要再看两天"） | `conflicts.*`（褒贬 + 确定程度两个维度） | +| 称呼变化 | `k0002` 前半段"林工"、后半段"林哥" | `relations.address_*` | +| 锚点两种粒度 | `k0001:tN` / `k0002:tN`（轮次级）与 `k0003`（段落级） | 锚点解析与回指 | +| 表情 | `k0002:t22` 🙂、`k0002:t24` 👍 | `voice.emoji_density` | +| 时间戳 | 每行正文内联 ISO 8601 | `stats.time_*`、`timeline.*` | + +## 形状 + +``` +synthetic-group/ + knowledge/index.json # 数组，3 个条目，每条带 anchors（字符串数组） + knowledge/text/*.md # 正文，行首锚点 [k0001:t1] / [k0003] +``` + +`index.json` 里的 `sha256` 是同一目录下 `.md` 文件的真实摘要； +`tests/retrospect.test.mjs` 会重新计算并断言一致，防止夹具被改坏。 diff --git a/src/derive/fixtures/synthetic-group/knowledge/text/dm-lin-chen.md b/src/derive/fixtures/synthetic-group/knowledge/text/dm-lin-chen.md new file mode 100644 index 00000000..bcbf93d7 --- /dev/null +++ b/src/derive/fixtures/synthetic-group/knowledge/text/dm-lin-chen.md @@ -0,0 +1,26 @@ +[k0002:t1] 2024-03-14T19:02:00Z 小陈：林工，白天那个冲正我还有点没懂。 +[k0002:t2] 2024-03-14T19:05:00Z 林工：哪一段？ +[k0002:t3] 2024-03-14T19:06:00Z 小陈：就是为什么要先停任务再对账。 +[k0002:t4] 2024-03-14T19:09:00Z 林工：因为任务还在跑，你对着对着数就变了，先止损再排查。 +[k0002:t5] 2024-03-14T19:12:00Z 小陈：懂了，先止损。谢谢林工。 +[k0002:t6] 2024-03-14T19:15:00Z 林工：不客气。 +[k0002:t7] 2024-03-14T19:20:00Z 小陈：林工，客服那边想要一个对外说法。 +[k0002:t8] 2024-03-14T19:23:00Z 林工：就说系统对账延迟，已经修复，不涉及用户资金。 +[k0002:t9] 2024-03-14T19:26:00Z 小陈：好，我按这个写。 +[k0002:t10] 2024-03-14T19:30:00Z 小陈：林工，复盘里那句"没人敢动"要不要删掉？ +[k0002:t11] 2024-03-14T19:33:00Z 林工：不用删，事实就是事实。 +[k0002:t12] 2024-03-14T19:36:00Z 小陈：行，那我保留。 +[k0002:t13] 2024-03-14T19:40:00Z 林工：嗯。 +[k0002:t14] 2024-03-15T20:01:00Z 小陈：林哥，周末还看消息啊？ +[k0002:t15] 2024-03-15T20:04:00Z 林工：习惯了，看一眼心里踏实。 +[k0002:t16] 2024-03-15T20:07:00Z 小陈：林哥，你觉得这次最大的教训是什么？ +[k0002:t17] 2024-03-15T20:10:00Z 林工：没有幂等键就敢重跑，这是设计问题，不是运气问题。 +[k0002:t18] 2024-03-15T20:13:00Z 小陈：记下了。还有别的吗？ +[k0002:t19] 2024-03-15T20:16:00Z 林工：告警太吵，真的出事反而没人看，宁可少一点、准一点。 +[k0002:t20] 2024-03-15T20:19:00Z 小陈：这个我也写进去。 +[k0002:t21] 2024-03-15T20:22:00Z 林工：写吧。 +[k0002:t22] 2024-03-15T20:25:00Z 小陈：林哥，下周评审你来讲这段？🙂 +[k0002:t23] 2024-03-15T20:28:00Z 林工：可以，我来讲。 +[k0002:t24] 2024-03-15T20:31:00Z 小陈：太好了👍 +[k0002:t25] 2024-03-15T20:34:00Z 林工：你把材料发我。 +[k0002:t26] 2024-03-15T20:37:00Z 小陈：马上发。 diff --git a/src/derive/fixtures/synthetic-group/knowledge/text/group-chat.md b/src/derive/fixtures/synthetic-group/knowledge/text/group-chat.md new file mode 100644 index 00000000..330c4cd6 --- /dev/null +++ b/src/derive/fixtures/synthetic-group/knowledge/text/group-chat.md @@ -0,0 +1,44 @@ +[k0001:t1] 2024-03-04T09:02:00Z 老周：早，今天先把灰度方案定下来。 +[k0001:t2] 2024-03-04T09:05:00Z 小陈：收到，我十点前把指标对齐一下。 +[k0001:t3] 2024-03-04T09:07:00Z 林工：先看数据。昨天的对账差异我拉了清单。 +[k0001:t4] 2024-03-04T09:11:00Z 阿May：我这边设计稿切好了，等方案。 +[k0001:t5] 2024-03-04T09:14:00Z 林工：远程办公挺好的，效率高，上午能专心写代码。 +[k0001:t6] 2024-03-04T09:20:00Z 小陈：你这边能出个排期吗？ +[k0001:t7] 2024-03-04T09:26:00Z 林工：能，今天下班前给。 +[k0001:t8] 2024-03-04T14:30:00Z 小陈：排期对齐一下，我先按两周估。 +[k0001:t9] 2024-03-04T14:35:00Z 老周：这个方案不行，风险太大，灰度先切小。 +[k0001:t10] 2024-03-04T14:40:00Z 林工：风险就一个，老链路的补偿逻辑没人敢动。 +[k0001:t11] 2024-03-05T10:00:00Z 阿May：我这边把空状态补上，今天给。 +[k0001:t12] 2024-03-05T10:06:00Z 小陈：好的，这个方案我觉得没问题，对齐一下验收标准。 +[k0001:t13] 2024-03-05T10:12:00Z 林工：验收标准就一条，账能对上。 +[k0001:t14] 2024-03-05T10:20:00Z 老周：各位，别忘了周五的复盘。 +[k0001:t15] 2024-03-05T16:40:00Z 小陈：老师们的意见我都记下了。 +[k0001:t16] 2024-03-06T09:15:00Z 阿May：我这边改完了，你那边看看？ +[k0001:t17] 2024-03-06T09:22:00Z 林工：看了，可以。 +[k0001:t18] 2024-03-08T10:20:00Z 小陈：对了林工，你上次说的那个offer最后怎么定的？ +[k0001:t19] 2024-03-08T10:24:00Z 林工：嗯，先不说这个，把灰度方案过一遍。 +[k0001:t20] 2024-03-08T10:30:00Z 老周：对，先过方案。 +[k0001:t21] 2024-03-11T02:10:00Z 老周：告警了，出入金对账差异超过阈值了，谁在看？我先拉一下监控面板。 +[k0001:t22] 2024-03-11T02:12:00Z 林工：我在看。先别改代码，把账翻出来，一条一条对，数据不会骗人，代码是你写的，你会替它辩护。 +[k0001:t23] 2024-03-11T02:15:00Z 林工：初步判断是补偿任务重跑了，同一笔入金被记了两次，我先把任务停掉，再写个脚本把重复的捞出来，一条一条对清楚。 +[k0001:t24] 2024-03-11T02:18:00Z 小陈：需要我通知客服吗？我这边可以先准备一套话术。 +[k0001:t25] 2024-03-11T02:20:00Z 林工：先不用，等我把范围圈出来再说，现在通知只会让客服被问爆，反而更乱，等圈定了范围我们再统一口径。 +[k0001:t26] 2024-03-11T02:26:00Z 阿May：我在线，需要改文案我随时上，空状态和提示语我都留了位置。 +[k0001:t27] 2024-03-11T02:31:00Z 林工：影响面确认了，重复入账一万三千笔，涉及四千二百个用户，明细我已经导出来放在共享盘里，谁要都能看。 +[k0001:t28] 2024-03-11T02:40:00Z 老周：干得漂亮，先把钱对上，别急着发公告，公告要等范围完全确定之后再发。 +[k0001:t29] 2024-03-11T02:52:00Z 林工：钱对上了，重复的部分我做了一笔冲正，账已经平了，接下来给补偿任务加幂等键，保证不会再有第二次。 +[k0001:t30] 2024-03-11T03:05:00Z 小陈：太好了！我这边同步一下客服口径！有问题我随时喊你！ +[k0001:t31] 2024-03-11T14:02:00Z 小陈：那offer的事... +[k0001:t32] 2024-03-11T14:05:00Z 林工：这个不方便说，先聊排期。 +[k0001:t33] 2024-03-12T10:00:00Z 林工：幂等键上线了，今天观察一天。不过可能还要再看两天，现在说闭环有点早，我会盯着监控。 +[k0001:t34] 2024-03-12T10:08:00Z 老周：好，写个复盘，把根因和动作都记下来，周五我们一起过一遍。 +[k0001:t35] 2024-03-18T09:30:00Z 林工：复盘写完了，根因是补偿任务没有幂等键。 +[k0001:t36] 2024-03-18T09:35:00Z 小陈：我这边同步一下。 +[k0001:t37] 2024-03-18T09:40:00Z 阿May：我这边加个提示。 +[k0001:t38] 2024-03-18T09:45:00Z 林工：远程办公其实很烦，沟通成本太高，出事全靠群里刷屏。 +[k0001:t39] 2024-03-18T09:50:00Z 老周：各有各的好。 +[k0001:t40] 2024-03-19T11:00:00Z 小陈：下周的评审我拉个会？ +[k0001:t41] 2024-03-19T11:05:00Z 林工：可以。 +[k0001:t42] 2024-03-19T11:10:00Z 阿May：我这边没问题。 +[k0001:t43] 2024-03-20T15:00:00Z 老周：各位，这个季度就到这，辛苦了。 +[k0001:t44] 2024-03-20T15:10:00Z 小陈：对齐一下，下周见。 diff --git a/src/derive/fixtures/synthetic-group/knowledge/text/incident-postmortem.md b/src/derive/fixtures/synthetic-group/knowledge/text/incident-postmortem.md new file mode 100644 index 00000000..d82f5986 --- /dev/null +++ b/src/derive/fixtures/synthetic-group/knowledge/text/incident-postmortem.md @@ -0,0 +1 @@ +[k0003] 2024-03-13T09:00:00Z 林工：事故复盘（草稿）。时间线：02:10 告警，02:12 介入，02:52 资金侧对平，次日 10:00 幂等键上线。根因：补偿任务重跑时没有幂等键，同一笔入金被重复记账。动作：给补偿任务补幂等键；把对账差异纳入每日巡检；把重跑改成需要人工确认。遗留：老链路的补偿逻辑还没有人完整读过一遍。 diff --git a/src/derive/retrospect.mjs b/src/derive/retrospect.mjs new file mode 100644 index 00000000..d6e323f2 --- /dev/null +++ b/src/derive/retrospect.mjs @@ -0,0 +1,927 @@ +/** + * `retrospect` — turn the local ledger into citable conclusions. + * + * The whole point of this module is that a claim is only allowed to exist if it + * can be pointed back at the bytes it came from. Concretely: + * + * - **Pure and deterministic.** No clock, no randomness, no network, no model. + * Every number is a function of the input files alone, every object key is + * written in sorted order and every float is rounded to a fixed number of + * digits, so running twice over the same ledger produces byte-identical + * files (`docs/v2/CONTRACT.md` §5). + * - **Evidence only from the ledger.** A claim may cite an anchor only if + * `knowledge/index.json` declares that anchor, so the mechanical + * anchor-resolution assertion in `docs/v2/ACCEPTANCE.md` §5 cannot fail by + * construction. When the ledger only declares a paragraph-level anchor while + * the text carries turn-level anchors, the units are *merged up* to the + * granularity the ledger can actually cite, and a note says so. + * - **Empty beats invented.** Every dimension has a minimum sample size. Below + * it the file ships `claims: []` plus a `notes` entry explaining exactly + * which threshold was missed — never a guessed value. + * + * Layout produced (relative to the person directory): + * + * evidence/derived/{stats,voice,relations,timeline,boundaries,shifts,conflicts}.json + * + * Each file is `{kind, generated_from: [{path, sha256}], claims: [...], notes: []}` + * and each claim is `{id, label: {zh, en}, value, confidence, evidence: [...]}`. + * + * Claim id naming rule (frozen for `ds/03-render`): `.` for a + * recurring measurement, `..` for a named bucket and + * `.candidate_` for an ordered candidate list, where `NNNN` is the + * 1-based position in that file's deterministic order. There is never a bare + * `` id and ids never carry a timestamp or a random component. + */ + +import { createHash } from "node:crypto"; +import { + existsSync, + mkdirSync, + readFileSync, + readdirSync, + renameSync, + statSync, + writeFileSync, +} from "node:fs"; +import { basename, dirname, join, resolve } from "node:path"; + +/** The seven derived files, in the order the contract lists them. */ +export const DERIVED_KINDS = [ + "stats", + "voice", + "relations", + "timeline", + "boundaries", + "shifts", + "conflicts", +]; + +/** + * Dimensions are skipped, not guessed, below these sample sizes. The numbers + * are deliberately low enough that a 38-cue interview still yields a useful + * file, and high enough that two messages yield nothing. + */ +export const MIN_UNITS = { + any: 8, // below this every dimension is empty and says why + participants: 2, // a 1:1 chat has exactly two speakers and must still report them + timeSpan: 2, // units carrying a parseable timestamp + lengths: 5, + density: 2, + sentences: 10, + punctuation: 5, + emoji: 5, + ngram: 20, + address: 5, + questions: 10, + interactions: 10, + latency: 6, + phases: 12, + boundaries: 20, + shifts: 16, + conflicts: 10, +}; + +/** Sliding-window width for `shifts`, as a fraction of the corpus. */ +const SHIFT_WINDOW_RATIO = 0.18; +const SHIFT_WINDOW_MIN = 5; +const SHIFT_WINDOW_MAX = 12; +/** A shift point must clear both a relative and an absolute threshold. */ +const SHIFT_RELATIVE = 0.45; +const SHIFT_ABSOLUTE = 6; + +/** One anchor, as frozen in `docs/v2/CONTRACT.md`: `k0012` or `k0012:t3`. */ +const ANCHOR_PATTERN = /k\d{4,}(?::t\d+)?/; +const ANCHOR_PATTERN_GLOBAL = /k\d{4,}(?::t\d+)?/g; +const ANCHOR_BRACKETS = /\[(k\d{4,}(?::t\d+)?)\]/g; +/** + * The anchor forms that actually occur. `[k0012]` / `[k0012:t3]` is the shape + * this module's contract describes and the shape the acceptance corpus uses; + * `k0012 text` (bare, no brackets) is what `src/knowledge/anchors.mjs` renders + * into `knowledge/text/*.md`. Both are read, and the parser never invents an + * anchor that is not literally present in the line. + */ +const ANCHOR_LEADING = /^\s*(k\d{4,}(?::t\d+)?)(?=\s|$)/; + +/** `2024-03-04T09:02:00Z`, `2024-03-04 09:02`, `2024-03-04T09:02:00+08:00`. */ +const LEADING_TIMESTAMP = + /^\s*(\d{4}-\d{2}-\d{2}(?:[T ]\d{2}:\d{2}(?::\d{2})?(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?)?)\s*/; + +/** `林工：` / `interviewer: ` at the head of a turn. */ +const LEADING_SPEAKER = /^([^：:\n]{1,24})[：:]\s*/; + +// --------------------------------------------------------------------------- +// deterministic primitives +// --------------------------------------------------------------------------- + +/** Sorted-key JSON. The run-twice-same-sha256 gate rests on this. */ +export function stableStringify(value, indent = 2) { + const normalise = (input) => { + if (input === null || typeof input !== "object") return input; + if (Array.isArray(input)) return input.map(normalise); + const output = {}; + for (const key of Object.keys(input).sort()) { + if (input[key] === undefined) continue; + output[key] = normalise(input[key]); + } + return output; + }; + return JSON.stringify(normalise(value), null, indent); +} + +/** Fixed-precision floats: the same ratio always serialises to the same text. */ +export function round(value, digits = 4) { + if (!Number.isFinite(value)) return null; + const factor = 10 ** digits; + const scaled = Math.round(value * factor) / factor; + return Object.is(scaled, -0) ? 0 : scaled; +} + +export function sha256Hex(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +function countChars(text) { + return [...text].length; +} + +function median(sortedNumbers) { + if (sortedNumbers.length === 0) return null; + const middle = Math.floor(sortedNumbers.length / 2); + return sortedNumbers.length % 2 === 1 + ? sortedNumbers[middle] + : (sortedNumbers[middle - 1] + sortedNumbers[middle]) / 2; +} + +function percentile(sortedNumbers, fraction) { + if (sortedNumbers.length === 0) return null; + return sortedNumbers[percentileIndex(sortedNumbers.length, fraction)]; +} + +function percentileIndex(length, fraction) { + return Math.min(length - 1, Math.max(0, Math.ceil(fraction * length) - 1)); +} + +function mean(numbers) { + if (numbers.length === 0) return null; + return numbers.reduce((total, value) => total + value, 0) / numbers.length; +} + +function unique(values) { + return [...new Set(values)]; +} + +/** Split a list into `parts` contiguous chunks that differ in length by <= 1. */ +function chunk(values, parts) { + const size = Math.ceil(values.length / parts); + const chunks = []; + for (let index = 0; index < values.length; index += size) { + chunks.push(values.slice(index, index + size)); + } + return chunks; +} + +/** Up to `max` items, evenly spread across the list, order preserved. */ +function spread(values, max) { + if (values.length <= max) return values.slice(); + if (max <= 1) return [values[0]]; + const picked = []; + for (let index = 0; index < max; index += 1) { + picked.push(values[Math.round((index * (values.length - 1)) / (max - 1))]); + } + return unique(picked); +} + +/** `[k, t]`, so anchors order the way a human reads the ledger. */ +function anchorRank(anchor) { + const match = /^k(\d+)(?::t(\d+))?$/.exec(anchor); + if (!match) return [Number.MAX_SAFE_INTEGER, Number.MAX_SAFE_INTEGER]; + return [Number(match[1]), match[2] === undefined ? 0 : Number(match[2])]; +} + +export function compareAnchors(left, right) { + const [leftK, leftT] = anchorRank(left); + const [rightK, rightT] = anchorRank(right); + if (leftK !== rightK) return leftK - rightK; + if (leftT !== rightT) return leftT - rightT; + return left < right ? -1 : left > right ? 1 : 0; +} + +function sortedAnchors(anchors) { + return unique(anchors).sort(compareAnchors); +} + +/** + * Confidence rules — the only three values allowed are `high`, `medium`, `low`. + * + * high >= 40 units of sample AND >= 3 independent anchors cited + * medium >= 15 units of sample AND >= 2 independent anchors cited + * low everything else that still cleared the dimension's minimum + * + * A dimension that cannot cite even one anchor is not emitted at all, so a + * claim never carries `evidence: []`. + */ +function confidenceOf(sample, evidenceCount) { + if (sample >= 40 && evidenceCount >= 3) return "high"; + if (sample >= 15 && evidenceCount >= 2) return "medium"; + return "low"; +} + +/** + * Build a claim. Returns `null` when there is nothing to cite, so a claim can + * never reach the output with `evidence: []` — the assembly step drops the + * `null`s. `fallback` is a last resort for measurements whose natural anchor is + * not guaranteed to exist (a median that lands between two samples, say). + */ +function makeClaim(id, zh, en, value, evidence, sample, fallback = []) { + let anchors = sortedAnchors(evidence); + if (anchors.length === 0) anchors = sortedAnchors(fallback); + if (anchors.length === 0) return null; + return { + id, + label: { zh, en }, + value, + confidence: confidenceOf(sample, anchors.length), + evidence: anchors.slice(0, 8), + }; +} + +/** `zh / en`, so a single `notes` string stays readable in both languages. */ +function note(zh, en) { + return `${zh} / ${en}`; +} + +// --------------------------------------------------------------------------- +// reading the ledger +// --------------------------------------------------------------------------- + +/** Normalise one `anchors` element: a string, or an object with an id. */ +function anchorIdOf(entry) { + if (typeof entry === "string") return entry; + if (entry && typeof entry === "object") { + for (const key of ["id", "anchor", "ref", "value"]) { + if (typeof entry[key] === "string") return entry[key]; + } + } + return null; +} + +function readLedger(personRoot) { + const ledgerPath = join(personRoot, "knowledge", "index.json"); + const bytes = readFileSync(ledgerPath); + const parsed = JSON.parse(bytes.toString("utf8")); + const entries = Array.isArray(parsed) ? parsed : Array.isArray(parsed?.entries) ? parsed.entries : []; + const declared = []; + for (const entry of entries) { + const list = Array.isArray(entry?.anchors) ? entry.anchors : []; + for (const item of list) { + const id = anchorIdOf(item); + if (typeof id === "string" && ANCHOR_PATTERN.test(id)) declared.push(id); + } + } + return { + path: ledgerPath, + relativePath: "knowledge/index.json", + bytes: bytes.length, + sha256: sha256Hex(bytes), + entries, + declared: sortedAnchors(declared), + }; +} + +function listTextFiles(personRoot) { + const textRoot = join(personRoot, "knowledge", "text"); + if (!existsSync(textRoot)) return []; + return readdirSync(textRoot) + .filter((name) => name.endsWith(".md")) + .sort() + .map((name) => { + const path = join(textRoot, name); + const bytes = readFileSync(path); + return { + name, + path, + relativePath: `knowledge/text/${name}`, + bytes: bytes.length, + sha256: sha256Hex(bytes), + text: bytes.toString("utf8"), + }; + }); +} + +/** `2024-03-04T09:02:00Z` → epoch milliseconds. Hand-rolled so it is UTC and + * implementation-independent (no `Date` locale or timezone behaviour). */ +function parseTimestamp(raw) { + const match = /^(\d{4})-(\d{2})-(\d{2})(?:[T ](\d{2}):(\d{2})(?::(\d{2}))?(?:\.\d+)?(Z|[+-]\d{2}:?\d{2})?)?$/.exec( + raw, + ); + if (!match) return null; + const [, year, month, day, hour = "00", minute = "00", second = "00", zone] = match; + let offsetMinutes = 0; + if (zone && zone !== "Z") { + const sign = zone.startsWith("-") ? -1 : 1; + const digits = zone.slice(1).replace(":", ""); + offsetMinutes = sign * (Number(digits.slice(0, 2)) * 60 + Number(digits.slice(2, 4))); + } + const utc = Date.UTC( + Number(year), + Number(month) - 1, + Number(day), + Number(hour), + Number(minute), + Number(second), + ); + return utc - offsetMinutes * 60000; +} + +function toIso(milliseconds) { + return new Date(milliseconds).toISOString().replace(".000Z", "Z"); +} + +/** + * Split one normalised text file into raw blocks: every line carrying at least + * one anchor — bracketed anywhere, or bare at the head of the line — starts a + * block, and an anchor-free non-empty line continues the previous one. + */ +function readBlocks(text) { + const blocks = []; + for (const line of text.split(/\r?\n/)) { + const anchors = [...line.matchAll(ANCHOR_BRACKETS)].map((match) => match[1]); + if (anchors.length === 0) { + const leading = ANCHOR_LEADING.exec(line); + if (leading) anchors.push(leading[1]); + } + if (anchors.length > 0) { + blocks.push({ anchors, line }); + } else if (line.trim() !== "" && blocks.length > 0) { + blocks[blocks.length - 1].line += ` ${line.trim()}`; + } + } + return blocks; +} + +/** + * Strip anchors, the optional inline timestamp and the optional `speaker:` + * prefix from a block, returning the three parts. + */ +function splitBlock(line) { + let rest = line.replace(ANCHOR_BRACKETS, " "); + rest = rest.replace(ANCHOR_LEADING, " "); + let at = null; + const stamp = LEADING_TIMESTAMP.exec(rest); + if (stamp) { + at = parseTimestamp(stamp[1]); + rest = rest.slice(stamp[0].length); + } + let speaker = null; + const prefix = LEADING_SPEAKER.exec(rest); + if (prefix) { + speaker = prefix[1].trim(); + rest = rest.slice(prefix[0].length); + } + return { at, speaker, text: rest.replace(/\s+/g, " ").trim() }; +} + +/** + * Build the citable units. + * + * Each unit carries the anchor that will be written into `evidence`. When the + * ledger declares only the paragraph-level form of a turn-level anchor, the + * turns are merged into one unit and the merge is reported in `notes` — the + * statistics then describe exactly the granularity a reader can go and check. + */ +function buildUnits(blocks, ledger) { + const declared = new Set(ledger.declared); + const notes = []; + const warnings = []; + const byAnchor = new Map(); + const order = []; + const undeclared = []; + let merges = 0; + + const emittableOf = (anchor) => { + if (declared.has(anchor)) return anchor; + const base = anchor.split(":")[0]; + if (declared.has(base)) return base; + return null; + }; + + for (const block of blocks) { + const parts = splitBlock(block.line); + const targets = []; + for (const anchor of block.anchors) { + const emittable = emittableOf(anchor); + if (emittable === null) { + undeclared.push(anchor); + continue; + } + if (emittable !== anchor) merges += 1; + if (!targets.includes(emittable)) targets.push(emittable); + } + for (const anchor of unique(targets)) { + if (!byAnchor.has(anchor)) { + const unit = { + anchor, + file: block.file, + speakers: [], + texts: [], + ats: [], + }; + byAnchor.set(anchor, unit); + order.push(unit); + } + const unit = byAnchor.get(anchor); + if (parts.speaker !== null) unit.speakers.push(parts.speaker); + if (parts.at !== null) unit.ats.push(parts.at); + unit.texts.push(parts.text); + } + } + + const units = order.map((unit, index) => { + const text = unit.texts.filter(Boolean).join(" "); + const speakers = unique(unit.speakers); + const ats = unit.ats.slice().sort((left, right) => left - right); + return { + index, + anchor: unit.anchor, + file: unit.file, + text, + chars: countChars(text), + speaker: speakers.length === 1 ? speakers[0] : null, + speakers, + at: ats.length > 0 ? ats[0] : null, + atLast: ats.length > 0 ? ats[ats.length - 1] : null, + mergedTurns: unit.texts.length, + }; + }); + + if (merges > 0) { + notes.push( + note( + `账本只声明段落级锚点：${merges} 条更细粒度的消息被合并到可引用锚点上，统计以合并后的 ${units.length} 个单元为单位。`, + `The ledger only declares paragraph anchors, so ${merges} finer-grained messages were merged onto citable anchors; statistics describe the resulting ${units.length} units.`, + ), + ); + } + if (undeclared.length > 0) { + warnings.push( + `${undeclared.length} anchor(s) appear in knowledge/text but are not declared in knowledge/index.json (first: ${undeclared[0]}); they were not cited.`, + ); + } + return { units, notes, warnings }; +} + +function orderUnits(units) { + const timed = units.filter((unit) => unit.at !== null); + if (timed.length === units.length && units.length > 1) { + return units.slice().sort((left, right) => left.at - right.at || left.index - right.index); + } + return units.slice(); +} + +function readCorpus(personRoot) { + const ledger = readLedger(personRoot); + const files = listTextFiles(personRoot); + const warnings = []; + const notes = []; + const blocks = []; + for (const file of files) { + for (const block of readBlocks(file.text)) blocks.push({ ...block, file: file.relativePath }); + const digestKnown = ledger.entries.some((entry) => entry?.sha256 === file.sha256); + if (!digestKnown) { + warnings.push( + `knowledge/${file.relativePath.replace(/^knowledge\//, "")} has no ledger entry with a matching sha256; it was read but not trusted for provenance.`, + ); + } + } + const built = buildUnits(blocks, ledger); + const units = orderUnits(built.units); + const timestamps = units.flatMap((unit) => (unit.at === null ? [] : [unit.at])); + return { + ledger, + files, + units, + timestamps, + notes: [...notes, ...built.notes], + warnings: [...warnings, ...built.warnings], + inputs: [ledger, ...files].map((file) => ({ + path: file.relativePath, + sha256: file.sha256, + bytes: file.bytes, + })), + }; +} + +// --------------------------------------------------------------------------- +// stats — 条数 / 参与者 / 时间跨度 / 消息长度分布 / 单位时间密度 +// --------------------------------------------------------------------------- + +export function deriveStats(corpus) { + const { units, timestamps } = corpus; + const claims = []; + const notes = []; + if (units.length < MIN_UNITS.any) { + notes.push( + note( + `样本不足：只有 ${units.length} 条可引用消息，低于所有维度的最低样本数 ${MIN_UNITS.any}，未产出任何结论。`, + `Insufficient sample: only ${units.length} citable messages, below the minimum of ${MIN_UNITS.any} for every dimension, so no claim was produced.`, + ), + ); + return { claims, notes }; + } + + const anchors = units.map((unit) => unit.anchor); + claims.push( + makeClaim( + "stats.message_count", + "可引用消息条数", + "Number of citable messages", + units.length, + spread(anchors, 3), + units.length, + ), + ); + + const files = unique(units.map((unit) => unit.file)); + if (files.length > 1) { + const perFile = files.map((file) => units.find((unit) => unit.file === file).anchor); + claims.push( + makeClaim( + "stats.source_count", + "来源文件数", + "Number of source files", + files.length, + perFile, + units.length, + ), + ); + } + + const speakerCounts = new Map(); + const firstAnchorOfSpeaker = new Map(); + for (const unit of units) { + for (const speaker of unit.speakers) { + speakerCounts.set(speaker, (speakerCounts.get(speaker) ?? 0) + 1); + if (!firstAnchorOfSpeaker.has(speaker)) firstAnchorOfSpeaker.set(speaker, unit.anchor); + } + } + const participants = [...speakerCounts.keys()].sort( + (left, right) => speakerCounts.get(right) - speakerCounts.get(left) || (left < right ? -1 : 1), + ); + if (participants.length >= MIN_UNITS.participants) { + claims.push( + makeClaim( + "stats.participants", + "参与者（按发言条数降序）", + "Participants, most active first", + participants, + participants.map((name) => firstAnchorOfSpeaker.get(name)), + units.length, + ), + ); + claims.push( + makeClaim( + "stats.participant_count", + "参与者人数", + "Number of participants", + participants.length, + participants.map((name) => firstAnchorOfSpeaker.get(name)), + units.length, + ), + ); + } else { + notes.push( + note( + `参与者维度跳过：只识别到 ${participants.length} 个说话人，低于阈值 ${MIN_UNITS.participants}。`, + `Participants skipped: only ${participants.length} speakers recognised, below the threshold of ${MIN_UNITS.participants}.`, + ), + ); + } + + if (timestamps.length >= MIN_UNITS.timeSpan) { + const sorted = timestamps.slice().sort((left, right) => left - right); + const spanMs = sorted[sorted.length - 1] - sorted[0]; + const firstTimed = units.find((unit) => unit.at === sorted[0]); + const lastTimed = units.reduce( + (found, unit) => (unit.at !== null && unit.at >= (found?.at ?? -Infinity) ? unit : found), + null, + ); + const evidence = [firstTimed?.anchor, lastTimed?.anchor].filter(Boolean); + claims.push( + makeClaim( + "stats.time_range", + "时间范围（首条 / 末条时间戳）", + "Time range (first / last timestamp)", + { from: toIso(sorted[0]), to: toIso(sorted[sorted.length - 1]) }, + evidence, + timestamps.length, + ), + ); + claims.push( + makeClaim( + "stats.time_span_days", + "时间跨度（天）", + "Time span in days", + round(spanMs / 86400000, 3), + evidence, + timestamps.length, + ), + ); + const spanHours = spanMs / 3600000; + if (spanHours >= 1) { + claims.push( + makeClaim( + "stats.density_per_hour", + "单位时间密度（条/小时）", + "Message density (messages per hour)", + { + per_hour: round(units.length / spanHours, 4), + span_hours: round(spanHours, 3), + messages: units.length, + }, + spread(anchors, 3), + timestamps.length, + ), + ); + } else { + notes.push( + note( + "单位时间密度跳过：可解析的时间跨度不足 1 小时，密度会退化成无意义的巨大值。", + "Density skipped: the parseable span is under one hour, which would make the ratio meaninglessly large.", + ), + ); + } + } else { + notes.push( + note( + `时间跨度与密度跳过：只有 ${timestamps.length} 条消息带可解析时间戳，低于阈值 ${MIN_UNITS.timeSpan}。`, + `Time span and density skipped: only ${timestamps.length} messages carry a parseable timestamp, below the threshold of ${MIN_UNITS.timeSpan}.`, + ), + ); + } + + if (units.length >= MIN_UNITS.lengths) { + const lengths = units.map((unit) => unit.chars).sort((left, right) => left - right); + const shortest = units.reduce((found, unit) => (unit.chars < found.chars ? unit : found), units[0]); + const longest = units.reduce((found, unit) => (unit.chars > found.chars ? unit : found), units[0]); + claims.push( + makeClaim( + "stats.message_length", + "消息长度分布（字符数）", + "Message length distribution (characters)", + { + unit: "characters", + mean: round(mean(lengths), 2), + median: round(median(lengths), 2), + p90: round(percentile(lengths, 0.9), 2), + min: lengths[0], + max: lengths[lengths.length - 1], + }, + [shortest.anchor, longest.anchor], + units.length, + ), + ); + } + + return { claims, notes }; +} + +// --------------------------------------------------------------------------- +// document assembly +// --------------------------------------------------------------------------- + +const DERIVERS = { + stats: deriveStats, +}; + +/** Run every available dimension and assemble the seven documents. */ +export function deriveDocuments(corpus) { + const generatedFrom = corpus.inputs.map((input) => ({ + path: input.path, + sha256: input.sha256, + })); + const documents = {}; + const notes = []; + for (const kind of DERIVED_KINDS) { + const deriver = DERIVERS[kind]; + const result = deriver ? deriver(corpus) : { claims: [], notes: [] }; + documents[kind] = { + kind, + generated_from: generatedFrom, + claims: result.claims.filter(Boolean), + notes: unique([...corpus.notes, ...result.notes]), + }; + for (const line of result.notes) notes.push({ kind, line }); + } + return { documents, notes }; +} + +// --------------------------------------------------------------------------- +// CLI entry point +// --------------------------------------------------------------------------- + +const HELP = `distilly retrospect — 纯派生：knowledge/ → evidence/derived/*.json + +用法 / Usage: + distilly retrospect --person [--json] + distilly retrospect --dir [--json] + distilly retrospect --help + +选项 / Options: + --person 在 ./skills/*// 下查找该人的目录 + --dir 直接指定人的目录（含 knowledge/index.json） + --json 只打印机器可读回执 / print the machine-readable receipt only + --help 打印本帮助 / print this help + +输入（只读） / Inputs (read only): + knowledge/index.json 账本；每条结论的锚点都必须在这里声明 + knowledge/text/*.md 归一化正文，段落锚点 [k0012] / [k0012:t3] + +输出 / Outputs: + evidence/derived/{stats,voice,relations,timeline,boundaries,shifts,conflicts}.json + +退出码 / Exit codes: + 0 成功 2 缺输入或用法错误（回执里给出补救步骤） + +不联网、不调用任何模型：同一份输入跑两次，产物逐字节相同。 +No network and no model call: the same input produces byte-identical output. +`; + +function parseArgs(args) { + const options = { person: null, dir: null, json: false, help: false }; + for (let index = 0; index < args.length; index += 1) { + const arg = args[index]; + if (arg === "--help" || arg === "-h") options.help = true; + else if (arg === "--json") options.json = true; + else if (arg === "--person" || arg === "--dir") { + const value = args[index + 1]; + if (!value || value.startsWith("--")) { + return { error: `${arg} requires a value` }; + } + options[arg === "--person" ? "person" : "dir"] = value; + index += 1; + } else return { error: `unknown option: ${arg}` }; + } + return { options }; +} + +function familyDirs(skillsRoot) { + if (!existsSync(skillsRoot)) return []; + return readdirSync(skillsRoot) + .sort() + .map((name) => join(skillsRoot, name)) + .filter((path) => { + try { + return statSync(path).isDirectory(); + } catch { + return false; + } + }); +} + +/** Resolve the person directory from `--dir`, `--person` or the cwd. */ +export function resolvePersonRoot(options, cwd) { + const hasLedger = (path) => existsSync(join(path, "knowledge", "index.json")); + if (options.dir) { + const path = resolve(cwd, options.dir); + return hasLedger(path) ? { root: path } : { error: `${path} has no knowledge/index.json` }; + } + if (options.person) { + const candidates = [ + join(cwd, "skills", "colleague", options.person), + ...familyDirs(join(cwd, "skills")).map((family) => join(family, options.person)), + ]; + for (const candidate of unique(candidates)) { + if (hasLedger(candidate)) return { root: candidate }; + } + return { + error: `${join(cwd, "skills", "*", options.person)} has no knowledge/index.json`, + }; + } + if (hasLedger(cwd)) return { root: cwd }; + return { error: `${cwd} has no knowledge/index.json` }; +} + +function writeDerivedFiles(personRoot, documents) { + const outputDir = join(personRoot, "evidence", "derived"); + mkdirSync(outputDir, { recursive: true }); + const outputs = []; + for (const kind of DERIVED_KINDS) { + const body = Buffer.from(`${stableStringify(documents[kind])}\n`, "utf8"); + const path = join(outputDir, `${kind}.json`); + const staging = join(outputDir, `.${basename(path)}.${process.pid}.tmp`); + writeFileSync(staging, body); + renameSync(staging, path); + outputs.push({ + path: `evidence/derived/${kind}.json`, + sha256: sha256Hex(body), + bytes: body.length, + }); + } + return outputs; +} + +function humanSummary(receipt, documents) { + const lines = [`retrospect: ${receipt.outputs.length} 个派生文件 / derived files`]; + for (const output of receipt.outputs) { + const size = documents[basename(output.path, ".json")].claims.length; + lines.push(` ${output.path} ${size} claim(s) ${output.sha256.slice(0, 12)}`); + } + if (receipt.warnings.length > 0) { + lines.push(` 警告 / warnings: ${receipt.warnings.length}`); + for (const warning of receipt.warnings) lines.push(` - ${warning}`); + } + return `${lines.join("\n")}\n`; +} + +/** + * CLI entry point. `io` may override the working directory and the two streams, + * which is how the tests drive it without spawning a process. + * + * @returns {{exitCode: number, receipt: object}} + */ +export function run(args = [], io = {}) { + const cwd = io.cwd ?? process.cwd(); + const stdout = io.stdout ?? process.stdout; + const stderr = io.stderr ?? process.stderr; + const parsed = parseArgs(args); + + const emit = (receipt, exitCode, documents = null) => { + if (parsed.options?.json) stdout.write(`${stableStringify(receipt)}\n`); + else if (exitCode !== 0) stderr.write(`retrospect: ${receipt.error.message}\n`); + else stdout.write(humanSummary(receipt, documents)); + return { exitCode, receipt }; + }; + + if (parsed.error) { + return emit( + { + command: "retrospect", + person: null, + ok: false, + inputs: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + warnings: [], + unavailable: [], + error: { code: "retrospect/usage", message: parsed.error, remedy: "distilly retrospect --help" }, + }, + 2, + ); + } + if (parsed.options.help) { + stdout.write(HELP); + return { exitCode: 0, receipt: null }; + } + + const resolved = resolvePersonRoot(parsed.options, cwd); + if (resolved.error) { + return emit( + { + command: "retrospect", + person: parsed.options.person, + ok: false, + inputs: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + warnings: [], + unavailable: [], + error: { + code: "retrospect/missing-input", + message: resolved.error, + remedy: + "run `distilly harvest --person ` first, then pass the same --person (or --dir )", + }, + }, + 2, + ); + } + + const corpus = readCorpus(resolved.root); + const { documents } = deriveDocuments(corpus); + const outputs = writeDerivedFiles(resolved.root, documents); + + const cited = sortedAnchors( + Object.values(documents).flatMap((document) => + document.claims.flatMap((claim) => claim.evidence), + ), + ); + const ledgerWarnings = corpus.ledger.entries.flatMap((entry) => + (Array.isArray(entry?.warnings) ? entry.warnings : []).map( + (warning) => `ledger:${entry?.id ?? "?"}: ${warning}`, + ), + ); + + const receipt = { + command: "retrospect", + person: parsed.options.person ?? basename(resolved.root), + ok: true, + inputs: corpus.inputs, + outputs, + anchors: { total: corpus.ledger.declared.length, cited: cited.length }, + warnings: [...corpus.warnings, ...ledgerWarnings], + unavailable: [], + }; + return emit(receipt, 0, documents); +} + +export default run; diff --git a/src/install/hosts.mjs b/src/install/hosts.mjs new file mode 100644 index 00000000..f839a184 --- /dev/null +++ b/src/install/hosts.mjs @@ -0,0 +1,365 @@ +/** + * Host installation actions — the merged port of the eight + * `tools/install_*.py` scripts: + * + * install_generated_skill_common.py → installGeneratedSkill() + * install_generated_skill.py → defaultSkillsDir() / HOST_DEFAULT_PARTS + * install_claude_generated_skill.py → shouldInstallCommandShim() + commandsDir + * install_openclaw_generated_skill.py → openclaw wrapper + * install_codex_generated_skill.py → codex wrapper + * install_openclaw_skill.py → installRepoSkill() + * install_codex_skill.py → installRepoSkill() + * install_hermes_skill.py → installRepoSkill() + * + * Directories are never guessed: every target derives from the shared matrix in + * `src/hosts/agents.mjs` (`getAgent(id).globalPath`), except the Hermes + * *generated-skill* root, which INSTALL.md documents as + * `~/.hermes/skills/distilly-generated` and which is therefore an explicit, + * sourced override. + */ + +import { + cpSync, + existsSync, + mkdirSync, + readFileSync, + renameSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; +import { homedir } from "node:os"; +import { basename, dirname, join, parse, resolve } from "node:path"; + +import { getAgent, listAgents } from "../hosts/agents.mjs"; +import { enrichExistingSkillMeta, jsonDumps, nowIso, resolveRealPath } from "../skill/schema.mjs"; + +/** `install ` shortcuts kept from the pre-v2 CLI. */ +export const HOST_ALIASES = { + claude: "claude-code", + deepseek: "deepseek-harness", + grok: "grok-build", +}; + +/** + * Documented exception to "every target comes from the host matrix": + * INSTALL.md's generated-skill table puts Hermes person skills under + * `~/.hermes/skills/distilly-generated`, while the repo-level clone target is + * `~/.hermes/skills/openclaw-imports/distilly`. + */ +const GENERATED_ROOT_OVERRIDES = { + hermes: "~/.hermes/skills/distilly-generated", +}; + +const REPO_IGNORE = [".git", "__pycache__", ".DS_Store"]; + +/** Host ids supported by the installer — exactly the shared matrix. */ +export function supportedHosts() { + return listAgents(); +} + +/** Map a CLI host argument (alias or id) onto a matrix id. */ +export function resolveHostId(host) { + const id = HOST_ALIASES[host] ?? host; + return getAgent(id).id; +} + +/** Expand `~` and `$DSH_HOME` in a matrix path template. */ +export function expandTargetPath(template, { home = homedir(), env = process.env } = {}) { + let value = String(template); + if (value.startsWith("$DSH_HOME")) { + value = join(env.DSH_HOME || join(home, ".dsh"), value.slice("$DSH_HOME".length).replace(/^\//, "")); + } + if (value === "~") return home; + if (value.startsWith("~/")) value = join(home, value.slice(2)); + return value; +} + +/** Repo-level install target: `getAgent(id).globalPath`. */ +export function repoInstallDir(host, options = {}) { + return expandTargetPath(getAgent(resolveHostId(host)).globalPath, options); +} + +/** Documented project-local target, or null when the host defines none. */ +export function repoProjectDir(host, options = {}) { + const projectPath = getAgent(resolveHostId(host)).projectPath; + return projectPath ? expandTargetPath(projectPath, options) : null; +} + +/** + * Root that holds generated person skills (`-/SKILL.md`). + * Derived from the matrix so the two lists can never drift. + */ +export function generatedSkillsRoot(host, options = {}) { + const id = resolveHostId(host); + const override = GENERATED_ROOT_OVERRIDES[id]; + if (override) return expandTargetPath(override, options); + return dirname(repoInstallDir(id, options)); +} + +/** Python-compatible name for the generated-skill root (install_generated_skill.py). */ +export const defaultSkillsDir = generatedSkillsRoot; + +/** Refuse filesystem roots, the home directory and paths not named `distilly`. */ +export function validateInstallTarget(inputPath, { home = homedir(), requireName = true } = {}) { + const target = resolve(inputPath); + const parsed = parse(target); + if (target === parsed.root || target === resolve(home)) { + throw new Error("refusing to install into a filesystem root or home directory"); + } + if (requireName && basename(target) !== "distilly") { + throw new Error("the install path must end with a directory named distilly"); + } + return target; +} + +function pathsOverlap(source, destination) { + const sourceRoot = resolveRealPath(source); + const destinationRoot = resolveRealPath(destination); + if (sourceRoot === destinationRoot) return "same"; + const nested = + destinationRoot.startsWith(`${sourceRoot}/`) || sourceRoot.startsWith(`${destinationRoot}/`); + return nested ? "nested" : false; +} + +function shouldIgnore(name) { + return REPO_IGNORE.includes(name) || name.endsWith(".pyc"); +} + +/** Timestamped backup path used before replacing an existing install. */ +export function backupPathFor(target) { + const stamp = new Date().toISOString().replaceAll(":", "-").replaceAll(".", "-"); + return `${target}.backup-${stamp}`; +} + +/** + * Copy the Distilly repo into a host skill directory + * (`install_openclaw_skill.py` / `install_codex_skill.py` / `install_hermes_skill.py`). + */ +export function installRepoSkill({ + source, + destination, + force = false, + dryRun = false, + backup = false, +}) { + if (!existsSync(join(source, "SKILL.md"))) { + throw new Error(`source does not look like a skill repo: ${source}`); + } + + const overlap = pathsOverlap(source, destination); + if (overlap === "same") return destination; + if (overlap === "nested") throw new Error("source and destination must not overlap"); + + if (dryRun) return destination; + + let backupPath = null; + if (existsSync(destination)) { + if (!force) throw new Error(`destination already exists: ${destination}`); + if (backup) { + backupPath = backupPathFor(destination); + renameSync(destination, backupPath); + } else { + rmSync(destination, { recursive: true, force: true }); + } + } + + mkdirSync(dirname(destination), { recursive: true }); + cpSync(source, destination, { + recursive: true, + filter: (sourcePath) => !shouldIgnore(basename(sourcePath)), + }); + return { destination, backupPath }; +} + +/** + * Remove an installed Distilly copy. + * `--force` skips the "looks like a Distilly install" check; `--backup` keeps a + * timestamped copy instead of deleting. + */ +export function uninstallRepoSkill({ + destination, + force = false, + dryRun = false, + backup = false, + home = homedir(), +} = {}) { + const target = validateInstallTarget(destination, { home }); + if (!existsSync(target)) { + throw new Error(`nothing installed at ${target}`); + } + + const skillFile = join(target, "SKILL.md"); + if (!force) { + if (!existsSync(skillFile)) { + throw new Error(`${target} does not contain SKILL.md; rerun with --force to remove it anyway`); + } + const frontmatter = /^---\r?\n([\s\S]*?)\r?\n---/.exec(readFileSync(skillFile, "utf8")); + if (!frontmatter || !/^name:\s*distilly\s*$/m.test(frontmatter[1])) { + throw new Error(`${target} is not a Distilly install; rerun with --force to remove it anyway`); + } + } + + if (dryRun) return { destination: target, backupPath: null, removed: false }; + + if (backup) { + const backupPath = backupPathFor(target); + renameSync(target, backupPath); + return { destination: target, backupPath, removed: true }; + } + + rmSync(target, { recursive: true, force: true }); + return { destination: target, backupPath: null, removed: true }; +} + +const FRONTMATTER_RE = /^---\n([\s\S]*?)\n---\n?/; + +/** Load and normalize generated skill metadata from a skill directory. */ +export function loadGeneratedMeta(skillDir) { + const metaPath = join(skillDir, "meta.json"); + if (!existsSync(metaPath)) { + throw new Error(`generated skill is missing meta.json: ${skillDir}`); + } + return enrichExistingSkillMeta(JSON.parse(readFileSync(metaPath, "utf8")), skillDir); +} + +/** Rewrite the frontmatter name field to the installed command name. */ +export function rewriteFrontmatterName(markdown, newName) { + const match = FRONTMATTER_RE.exec(markdown); + if (!match) return markdown; + + const body = markdown.slice(match[0].length); + const lines = match[1].split(/\r?\n/); + const rewritten = []; + let replaced = false; + + for (const line of lines) { + if (line.startsWith("name:")) { + rewritten.push(`name: ${newName}`); + replaced = true; + } else { + rewritten.push(line); + } + } + if (!replaced) rewritten.unshift(`name: ${newName}`); + + return `---\n${rewritten.join("\n")}\n---\n\n${body.replace(/^\n+/, "")}`; +} + +/** Load a generated artifact and rewrite it for host installation. */ +export function renderInstalledMarkdown(skillDir, artifactName, commandName) { + const artifactPath = join(skillDir, artifactName); + if (!existsSync(artifactPath)) { + throw new Error(`generated artifact not found: ${artifactPath}`); + } + return rewriteFrontmatterName(readFileSync(artifactPath, "utf8"), commandName); +} + +/** Persist installation metadata for later debugging and upgrades. */ +export function writeInstallMetadata(installDir, payload) { + writeFileSync(join(installDir, ".distilly-install.json"), jsonDumps(payload), "utf8"); +} + +/** Windows installs also get a slash-command shim (install_claude_generated_skill.py). */ +export function shouldInstallCommandShim(systemName = process.platform) { + const current = String(systemName).toLowerCase(); + return current.startsWith("win"); +} + +/** + * Install a generated combined skill into a host skill directory + * (`install_generated_skill_common.py`). + */ +export function installGeneratedSkill({ + skillDir, + skillsDir, + force = false, + dryRun = false, + host, +}) { + const meta = loadGeneratedMeta(skillDir); + const artifacts = meta.artifacts; + const commandName = artifacts.combined_command; + const installedMarkdown = renderInstalledMarkdown( + skillDir, + artifacts.combined_skill, + commandName, + ); + + const installDir = join(skillsDir, commandName); + const installFile = join(installDir, "SKILL.md"); + + const overlap = pathsOverlap(skillDir, installDir); + if (overlap) { + throw new Error( + `generated skill source and install destination must not overlap: ${skillDir} -> ${installDir}`, + ); + } + + const installRecord = { + host, + command_name: commandName, + character: meta.character, + slug: meta.slug, + version: meta.version, + source_skill_dir: String(skillDir), + source_artifact: artifacts.combined_skill, + installed_at: nowIso(), + }; + + if (!dryRun) { + if (existsSync(installDir)) { + if (!force) throw new Error(`${host} skill already exists: ${installDir}`); + rmSync(installDir, { recursive: true, force: true }); + } + mkdirSync(installDir, { recursive: true }); + writeFileSync(installFile, installedMarkdown, "utf8"); + writeInstallMetadata(installDir, installRecord); + } + + return { host, command_name: commandName, skill_dir: installDir, skill_file: installFile }; +} + +/** + * Claude Code variant: optional `~/.claude/commands/.md` shim + * (`install_claude_generated_skill.py`). + */ +export function installGeneratedSkillForClaude({ + skillDir, + skillsDir, + commandsDir = null, + force = false, + dryRun = false, + installCommandShim = false, +}) { + const result = installGeneratedSkill({ skillDir, skillsDir, force, dryRun, host: "claude-code" }); + const commandPath = commandsDir === null ? null : join(commandsDir, `${result.command_name}.md`); + + if (!dryRun && installCommandShim && commandPath !== null) { + const meta = loadGeneratedMeta(skillDir); + const installedMarkdown = renderInstalledMarkdown( + skillDir, + meta.artifacts.combined_skill, + result.command_name, + ); + mkdirSync(dirname(commandPath), { recursive: true }); + writeFileSync(commandPath, installedMarkdown, "utf8"); + } + + return { + ...result, + command_path: commandPath, + command_shim_installed: Boolean(installCommandShim && commandPath !== null), + }; +} + +/** Is a Distilly install present at this path? Used by `doctor`. */ +export function inspectInstall(target) { + const skillFile = join(target, "SKILL.md"); + if (!existsSync(target) || !existsSync(skillFile)) { + return { installed: false, path: target, version: null, bytes: 0 }; + } + const frontmatter = /^---\r?\n([\s\S]*?)\r?\n---/.exec(readFileSync(skillFile, "utf8")); + const version = frontmatter ? (/(?:^|\n)version:\s*"?([^"\n]+)"?/.exec(frontmatter[1])?.[1] ?? null) : null; + return { installed: true, path: target, version, bytes: statSync(skillFile).size }; +} diff --git a/src/knowledge/anchors.mjs b/src/knowledge/anchors.mjs new file mode 100644 index 00000000..81bccfe4 --- /dev/null +++ b/src/knowledge/anchors.mjs @@ -0,0 +1,1032 @@ +/** + * anchors.mjs — encoding detection, byte-faithful normalisation and the global + * monotonic anchor space `[k00NN]` / `[k00NN:tM]`. + * + * Four invariants drive every decision in this file: + * + * 1. **Bytes are never lost silently.** Raw bytes live verbatim under + * `knowledge/raw/`. Every paragraph anchor resolves back to a byte range of + * the payload it came from, and anything a parser drops is reported in + * `warnings`, never swallowed. + * 2. **Every unit has at least one non-whitespace code point.** Whitespace runs + * are separators, not units. + * 3. **Anchors are globally monotonic.** Ids are zero padded to four digits, + * never reused and never renumbered, so a later append cannot invalidate an + * earlier citation. + * 4. **Encoding ambiguity is an outcome, not a coin flip.** GBK and Big5 decode + * the same CJK byte pairs; when more than one legacy codec fits we say so + * instead of inventing characters. + * + * Determinism: ids are a pure function of `(id, unitIndex)` and every routine + * returns objects with keys in a fixed order, so the same input yields the same + * anchors — and the same bytes — on every run. No clocks, no randomness. + */ + +const BOMS = [ + { name: "utf-8", bytes: [0xef, 0xbb, 0xbf], decodeAs: "utf-8" }, + { name: "utf-32le", bytes: [0xff, 0xfe, 0x00, 0x00], decodeAs: "utf-32le" }, + { name: "utf-32be", bytes: [0x00, 0x00, 0xfe, 0xff], decodeAs: "utf-32be" }, + { name: "utf-16le", bytes: [0xff, 0xfe], decodeAs: "utf-16le" }, + { name: "utf-16be", bytes: [0xfe, 0xff], decodeAs: "utf-16be" }, +]; + +/** Segment kinds agreed across every parser in `src/parse/**`. */ +export const SEGMENT_KINDS = Object.freeze([ + "turn", // chat message + "msg", // email message + "cue", // subtitle cue + "para", // document paragraph + "item", // structured record (CSV row, JSON object) +]); + +const DEFAULT_TEXT_ENCODING_FALLBACKS = Object.freeze(["gbk", "big5", "shift_jis"]); + +/* ------------------------------------------------------------------ */ +/* small helpers */ +/* ------------------------------------------------------------------ */ + +function toUint8(bytes) { + if (bytes instanceof Uint8Array) return bytes; + if (bytes instanceof ArrayBuffer) return new Uint8Array(bytes); + if (Array.isArray(bytes)) return Uint8Array.from(bytes); + if (typeof bytes === "string") return Buffer.from(bytes, "utf8"); + throw new TypeError("expected a Uint8Array, ArrayBuffer, number[] or string"); +} + +function startsWithBytes(bytes, prefix) { + if (bytes.length < prefix.length) return false; + for (let index = 0; index < prefix.length; index += 1) { + if (bytes[index] !== prefix[index]) return false; + } + return true; +} + +function countReplacementChars(text) { + let count = 0; + for (const char of text) { + if (char === "\uFFFD") count += 1; + } + return count; +} + +function controlReplacement(code) { + if (code === 0x09 || code === 0x0a || code === 0x0d) return null; + if (code < 0x20 || code === 0x7f) return " "; + if (code >= 0x80 && code <= 0x9f) return " "; + return null; +} + +function isUnicodeSpace(code) { + return ( + code === 0x20 || + code === 0xa0 || + code === 0x1680 || + (code >= 0x2000 && code <= 0x200a) || + code === 0x202f || + code === 0x205f || + code === 0x3000 + ); +} + +/* ------------------------------------------------------------------ */ +/* encoding */ +/* ------------------------------------------------------------------ */ + +/** + * Detect the BOM of a buffer without decoding it. + * @param {Uint8Array} bytes + * @returns {{label: string, length: number, decodeAs: string}|null} + */ +export function detectBom(bytes) { + if (!bytes || bytes.length < 2) return null; + for (const bom of BOMS) { + if (startsWithBytes(bytes, bom.bytes)) { + return { label: bom.name, length: bom.bytes.length, decodeAs: bom.decodeAs }; + } + } + return null; +} + +function looksLikeUtf16WithoutBom(bytes, littleEndian) { + const sample = Math.min(bytes.length, 512); + if (sample < 4) return false; + let zeros = 0; + let pairs = 0; + for (let index = 0; index + 1 < sample; index += 2) { + pairs += 1; + const [a, b] = littleEndian ? [bytes[index], bytes[index + 1]] : [bytes[index + 1], bytes[index]]; + if (a === 0x00 && b !== 0x00) zeros += 1; + } + return pairs > 0 && zeros / pairs > 0.6; +} + +function decodeWith(codec, bytes) { + return new TextDecoder(codec, { fatal: true }).decode(bytes); +} + +/** + * True when `bytes` is valid UTF-8. Node's strict decoder rejects overlong + * forms, surrogate halves and truncated sequences, so a round trip through it is + * a sound test — and it is what keeps us from mistaking GBK for UTF-8. + */ +export function isValidUtf8(bytes) { + try { + decodeWith("utf-8", toUint8(bytes)); + return true; + } catch { + return false; + } +} + +/** + * Decide how to read `bytes`. + * + * Order: + * 1. a BOM, which is authoritative; + * 2. an explicit `preferred` label (how a parser passes a declared `charset=`); + * 3. a BOM-less UTF-16 shape (NUL bytes at every other position); + * 4. strict UTF-8; + * 5. **only when `legacyFallback` is on**: a legacy codec, and only if exactly + * one of them decodes the payload; + * 6. UTF-8 with U+FFFD replacement, reporting how many sequences were replaced. + * + * Step 5 is opt-in because GBK, Big5 and Shift_JIS map almost any byte pair to + * *some* character. A silent fallback would turn a truncated JPEG into confident + * nonsense; a loudly-reported U+FFFD is the honest answer. + * + * @param {Uint8Array} bytes + * @param {{preferred?: string, fallbacks?: string[], legacyFallback?: boolean}} [options] + * @returns {{label: string, bom: string|null, bomBytes: number, decoded?: string, + * ok: boolean, lossy: boolean, attempted: string[], + * ambiguous?: string[], error?: string}} + */ +export function detectEncoding(bytes, options = {}) { + const { + preferred, + fallbacks = DEFAULT_TEXT_ENCODING_FALLBACKS, + legacyFallback = false, + } = options; + const bom = detectBom(bytes); + const attempted = []; + + if (bom) { + attempted.push(bom.label); + try { + return { + label: bom.label, + bom: bom.label, + bomBytes: bom.length, + decoded: decodeWith(bom.decodeAs, bytes.subarray(bom.length)), + ok: true, + lossy: false, + attempted, + }; + } catch (error) { + return { + label: bom.label, + bom: bom.label, + bomBytes: bom.length, + ok: false, + lossy: true, + attempted, + error: `declared BOM ${bom.label} but the payload is not valid ${bom.label}: ${error.message}`, + }; + } + } + + const candidates = []; + if (preferred) candidates.push({ codec: preferred, lossy: preferred !== "utf-8" }); + if (looksLikeUtf16WithoutBom(bytes, true)) candidates.push({ codec: "utf-16le", lossy: true }); + if (looksLikeUtf16WithoutBom(bytes, false)) candidates.push({ codec: "utf-16be", lossy: true }); + candidates.push({ codec: "utf-8", lossy: false }); + + if (!preferred && legacyFallback) { + const decodable = []; + for (const codec of fallbacks) { + try { + decodeWith(codec, bytes); + decodable.push(codec); + } catch { + // Not this codec. + } + } + if (decodable.length === 1) { + candidates.push({ codec: decodable[0], lossy: true }); + } else if (decodable.length > 1) { + return { + label: "utf-8", + bom: null, + bomBytes: 0, + ok: false, + lossy: true, + attempted: ["utf-8", ...decodable], + ambiguous: decodable, + error: `the payload is not UTF-8 and ${decodable.join("/")} all decode it; declare the charset in the export (or pass it explicitly) instead of letting us guess`, + }; + } + } else if (preferred) { + for (const codec of fallbacks) { + if (!candidates.some((candidate) => candidate.codec === codec)) { + candidates.push({ codec, lossy: true }); + } + } + } + + let lastError = "no candidate encoding was attempted"; + for (const candidate of candidates) { + attempted.push(candidate.codec); + let decoded; + try { + decoded = decodeWith(candidate.codec, bytes); + } catch (error) { + lastError = `${candidate.codec}: ${error.message}`; + continue; + } + return { + label: candidate.codec, + bom: null, + bomBytes: 0, + decoded, + ok: true, + // Anything past UTF-8 is a guess about a non-UTF-8 legacy encoding; the + // caller is expected to surface that in `warnings`. + lossy: candidate.lossy, + attempted, + }; + } + + return { + label: "utf-8", + bom: null, + bomBytes: 0, + ok: false, + lossy: true, + attempted, + error: `no candidate encoding decoded cleanly (${lastError}); re-decoding as utf-8 with U+FFFD replacement`, + }; +} + +/** + * Decode raw bytes to text, always producing something and always explaining + * itself. Invalid sequences become U+FFFD as a last resort, with a count. + * + * @param {Uint8Array} bytes + * @param {{preferred?: string, fallbacks?: string[], legacyFallback?: boolean}} [options] + * @returns {{text: string, label: string, bom: string|null, bomBytes: number, + * lossy: boolean, replaced: number, attempted: string[], + * ambiguous: string[]|null, warnings: string[]}} + */ +export function decodeBuffer(bytes, options = {}) { + const buffer = toUint8(bytes); + const detection = detectEncoding(buffer, options); + const warnings = []; + let text; + let replaced = 0; + + if (detection.ok) { + text = detection.decoded; + if (detection.lossy) { + warnings.push( + `text decoded as ${detection.label} (no BOM, not valid UTF-8); byte-to-character offsets assume ${detection.label}`, + ); + } + } else { + const body = detection.bomBytes ? buffer.subarray(detection.bomBytes) : buffer; + text = new TextDecoder("utf-8", { fatal: false }).decode(body); + replaced = countReplacementChars(text); + warnings.push(detection.error); + if (replaced > 0) { + warnings.push(`${replaced} invalid byte sequence(s) became U+FFFD; raw bytes are preserved under knowledge/raw/`); + } + } + + return { + text, + label: detection.label, + bom: detection.bom, + bomBytes: detection.bomBytes, + lossy: Boolean(detection.lossy), + replaced, + attempted: detection.attempted ?? [], + ambiguous: detection.ambiguous ?? null, + warnings, + }; +} + +/* ------------------------------------------------------------------ */ +/* decoded text -> byte map */ +/* ------------------------------------------------------------------ */ + +/** + * Walk the decoded text and record, for every code point, the byte span it came + * from. UTF-8 is a closed form (a code point's UTF-8 width is the width the + * decoder consumed); for every other codec we re-decode one code point at a time + * from the real byte position, because guessing a width is exactly what makes a + * GBK anchor point one byte off. + * + * U+FFFD is ambiguous — a genuine replacement character or a decoded `\uFFFD` — + * but both are 3 UTF-8 bytes, so the widths stay right either way. + * + * @returns {{starts: number[], ends: number[], offsets: number[], lines: number[], + * usable: boolean}} + */ +function mapDecodedText(decodedText, buffer, bomBytes, codec) { + const starts = []; + const ends = []; + const offsets = []; + const lines = []; + let line = 1; + + const note = (char, byteStart, byteEnd) => { + offsets.push(byteStart); + lines.push(line); + starts.push(byteStart); + ends.push(byteEnd); + if (char === "\n") line += 1; + }; + + if (codec === "utf-8") { + let position = bomBytes; + for (let index = 0; index < decodedText.length; ) { + const char = decodedText[index]; + const code = decodedText.codePointAt(index); + const size = code > 0xffff ? 2 : 1; + const width = Buffer.byteLength(decodedText.slice(index, index + size), "utf8"); + note(char, position, position + width); + position += width; + index += size; + } + return { starts, ends, offsets, lines, usable: true }; + } + + let decoder; + try { + decoder = new TextDecoder(codec); + } catch { + return { starts: null, ends: null, offsets: null, lines: null, usable: false }; + } + + let position = bomBytes; + let emitted = 0; + const chunkSize = 64 * 1024; + while (position < buffer.length && emitted < decodedText.length) { + const chunk = buffer.subarray(position, Math.min(position + chunkSize, buffer.length)); + // Longest prefix that decodes cleanly: one code point (or one malformed + // replacement) at a time. + let low = 1; + let high = chunk.length; + let validEnd = 0; + while (low <= high) { + const middle = (low + high) >> 1; + try { + decoder.decode(chunk.subarray(0, middle), { stream: middle < chunk.length }); + validEnd = middle; + low = middle + 1; + } catch { + high = middle - 1; + } + } + if (validEnd === 0) break; + const piece = decoder.decode(chunk.subarray(0, validEnd), { stream: validEnd < chunk.length }); + for (let index = 0; index < piece.length && emitted < decodedText.length; index += 1) { + note(piece[index], position, position + validEnd); + emitted += 1; + } + position += validEnd; + } + + return { starts, ends, offsets, lines, usable: true }; +} + +/* ------------------------------------------------------------------ */ +/* normalisation */ +/* ------------------------------------------------------------------ */ + +function editableNormalize(decoded, options) { + const { text, charStart, charEnd } = decoded; + const outChars = []; + const outStart = []; + const outEnd = []; + const collapse = options.collapseSpaces !== false; + + const push = (char, start, end) => { + outChars.push(char); + outStart.push(start); + outEnd.push(end); + }; + + for (let index = 0; index < text.length; ) { + const code = text.codePointAt(index); + const size = code > 0xffff ? 2 : 1; + const start = charStart[index]; + const end = charEnd[index + size - 1]; + + if (code === 0xfeff) { + // BOM / zero-width no-break space: dropped, but its bytes stay in the map. + index += size; + continue; + } + if (code === 0x0d) { + // CR or CRLF -> LF. The LF half contributes no character of its own; the + // resulting LF spans both bytes so the range stays gapless. + const isCrlf = text[index + 1] === "\n"; + push("\n", start, isCrlf ? charEnd[index + 1] : end); + index += isCrlf ? 2 : 1; + continue; + } + if (code === 0x0a || code === 0x85 || code === 0x2028 || code === 0x2029) { + push("\n", start, end); + index += size; + continue; + } + const replacement = controlReplacement(code); + if (replacement !== null) { + push(replacement, start, end); + index += size; + continue; + } + if (isUnicodeSpace(code)) { + push(collapse ? " " : String.fromCodePoint(code), start, end); + index += size; + continue; + } + push(String.fromCodePoint(code), start, end); + index += size; + } + + // Trim trailing horizontal whitespace per line. + const trimmed = []; + let lineStart = 0; + for (let index = 0; index <= outChars.length; index += 1) { + const isEnd = index === outChars.length; + if (!isEnd && outChars[index] !== "\n") continue; + let last = index; + while (last > lineStart && outChars[last - 1] === " ") last -= 1; + for (let cursor = lineStart; cursor < last; cursor += 1) { + trimmed.push({ char: outChars[cursor], start: outStart[cursor], end: outEnd[cursor] }); + } + if (!isEnd) trimmed.push({ char: "\n", start: outStart[index], end: outEnd[index] }); + lineStart = index + 1; + } + + const chars = []; + const charOutStart = []; + const charOutEnd = []; + for (const unit of trimmed) { + chars.push(unit.char); + charOutStart.push(unit.start); + charOutEnd.push(unit.end); + } + + const normalised = chars.join(""); + const mappedTo = charOutEnd.length > 0 ? charOutEnd[charOutEnd.length - 1] : decoded.bomBytes; + const buffer = decoded.bytes ?? Buffer.from(decoded.text, "utf8"); + + return { + text: normalised, + charStart: Int32Array.from(charOutStart), + charEnd: Int32Array.from(charOutEnd), + bytes: buffer, + encoding: decoded.encoding, + bom: decoded.bom, + bomBytes: decoded.bomBytes, + lossy: decoded.lossy, + replaced: decoded.replaced, + warnings: decoded.warnings ?? [], + // Character -> raw byte, for payloads decoded from a file. `null` when the + // text was assembled by a parser instead. + byteOffset: decoded.byteOffset ?? null, + // Bytes at the tail that produced no character (trailing whitespace, BOM). + unmappedTail: buffer.length - mappedTo, + }; +} + +/** + * Normalise text that has *already* been decoded, keeping the character → byte + * mapping the caller passes in. Used by parsers that assemble their own text and + * by `assignAnchorsToText`. + * + * @param {string} text + * @param {{byteStarts?: Int32Array|number[], byteEnds?: Int32Array|number[], + * bomBytes?: number, encoding?: string, bom?: string|null, + * lossy?: boolean, collapseSpaces?: boolean}} [options] + */ +export function normalizeTextWithMap(text, options = {}) { + const source = String(text ?? ""); + const provided = options.byteStarts ?? null; + const providedEnds = options.byteEnds ?? null; + const bomBytes = options.bomBytes ?? 0; + + const starts = new Int32Array(source.length); + const ends = new Int32Array(source.length); + { + let position = bomBytes; + for (let index = 0; index < source.length; ) { + const code = source.codePointAt(index); + const size = code > 0xffff ? 2 : 1; + const width = Buffer.byteLength(source.slice(index, index + size), "utf8"); + const start = provided ? provided[index] : position; + const end = providedEnds ? providedEnds[index + size - 1] : start + width; + starts[index] = start; + ends[index] = end; + if (size === 2) { + starts[index + 1] = start; + ends[index + 1] = end; + } + position = end; + index += size; + } + } + + return editableNormalize( + { + text: source, + charStart: starts, + charEnd: ends, + bytes: Buffer.from(source, "utf8"), + encoding: options.encoding ?? "utf-8", + bom: options.bom ?? null, + bomBytes, + lossy: Boolean(options.lossy), + replaced: 0, + warnings: [], + // Only meaningful when the caller supplied the map; without it the offsets + // are positions in this string, not in a payload. + byteOffset: provided ? Int32Array.from(starts) : null, + }, + options, + ); +} + +/** + * Normalise a raw payload while remembering where every character came from: + * a single decode pass, then a single text pass. + * + * Transformations (all recorded, none lossy for non-whitespace content): + * - U+FEFF anywhere → removed (byte span kept, so ranges stay closed) + * - CRLF / CR → LF + * - NEL / LINE SEPARATOR / PARAGRAPH SEPARATOR → LF + * - C0/C1 control characters → space + * - NBSP and other Unicode spaces → U+0020 + * - trailing horizontal whitespace on each line → trimmed + * + * `charStart[i]` / `charEnd[i]` are byte positions in `bytes`; ranges are + * contiguous and ascending, so `bytes.subarray(charStart[i], charEnd[i])` is the + * exact raw span of character `i`. + * + * @param {Uint8Array} bytes + * @param {{preferred?: string, fallbacks?: string[], legacyFallback?: boolean, + * collapseSpaces?: boolean}} [options] + */ +export function normalizeWithMap(bytes, options = {}) { + const buffer = toUint8(bytes); + const decoded = decodeBuffer(buffer, options); + const mapped = mapDecodedText(decoded.text, buffer, decoded.bomBytes, decoded.label); + + return editableNormalize( + { + ...decoded, + charStart: Int32Array.from(mapped.starts ?? []), + charEnd: Int32Array.from(mapped.ends ?? []), + byteOffset: mapped.offsets ? Int32Array.from(mapped.offsets) : null, + }, + options, + ); +} + +/* ------------------------------------------------------------------ */ +/* paragraph segmentation */ +/* ------------------------------------------------------------------ */ + +function isBlank(text) { + return text.replace(/\s+/gu, "") === ""; +} + +function hasVisibleCodePoint(text) { + return /\S/u.test(text); +} + +/** + * Split normalised text into leaf lines, each carrying its byte range and its + * absolute offset in the payload (or `null` for parser-assembled text). + */ +function splitLeafLines(normalized) { + const { text, charStart, charEnd, bytes, byteOffset } = normalized; + const lines = []; + let index = 0; + + while (index < text.length) { + let cursor = index; + while (cursor < text.length && text[cursor] !== "\n") cursor += 1; + lines.push({ + text: text.slice(index, cursor), + start: index, + end: cursor, + // Byte positions are in *raw payload* space: a line is the span of its + // characters, separators included. + byteStart: cursor > index ? charStart[index] : bytes.length, + byteEnd: cursor > index ? charEnd[cursor - 1] : bytes.length, + offset: byteOffset ? byteOffset[index] : null, + trailing: false, + }); + index = cursor + 1; + } + + return lines; +} + +/** + * Map a normalised character index to the segment (turn / cue / message) that + * produced it, so a paragraph can never straddle two messages. + */ +function buildSegmentLookup(segments) { + if (!Array.isArray(segments) || segments.length === 0) return null; + const spans = segments.map((segment, index) => ({ + start: segment.charStart ?? 0, + end: segment.charEnd ?? segment.charStart ?? 0, + index, + label: segment.label ?? null, + file: segment.file ?? null, + // Present only when the parser could point at the raw payload. `null` means + // "this text is derived" (an inflated OOXML member, a stripped HTML body) and + // is reported as such rather than faked with an offset. + raw: + Number.isInteger(segment.byteStart) && Number.isInteger(segment.byteEnd) + ? { byteStart: segment.byteStart, byteEnd: segment.byteEnd, file: segment.file ?? null } + : null, + })); + return (charIndex) => { + for (const span of spans) { + if (charIndex >= span.start && charIndex < span.end) return span; + } + return spans[spans.length - 1]; + }; +} + +/** + * Group leaf lines into paragraphs. + * + * Two modes, chosen by `options.groupBy`: + * - `"segment"` (the default when `segments` is given): one paragraph per source + * turn / cue / record, with `maxBlocks` consecutive lines as a safety valve + * for a runaway single record. + * - `"blank"`: paragraphs are separated by blank lines, capped at `maxBlocks`. + */ +function groupLines(lines, lookup, options) { + const maxBlocks = options.maxBlocks ?? 12; + const mode = options.groupBy ?? (lookup ? "segment" : "blank"); + const groups = []; + let current = []; + + const flush = () => { + if (current.length > 0) { + groups.push(current); + current = []; + } + }; + + for (const line of lines) { + if (isBlank(line.text)) { + if (mode === "blank") flush(); + continue; + } + const segment = lookup ? lookup(line.start) : null; + const segmentIndex = segment ? segment.index : 0; + if (current.length >= maxBlocks) flush(); + if (mode === "segment" && current.length > 0 && current[0].segmentIndex !== segmentIndex) flush(); + current.push({ + text: line.text, + byteStart: line.byteStart, + byteEnd: line.byteEnd, + offset: line.offset, + charStart: line.start, + charEnd: line.end, + segmentIndex, + label: segment ? segment.label : null, + raw: segment ? segment.raw : null, + }); + } + flush(); + return groups; +} + +/** + * Assign global monotonic anchors to already-assembled text. + * + * The paragraph anchors `[k00NN]` are placed in the returned `text`, which is what + * gets written to `knowledge/text/.md`. Each unit additionally reports the + * byte range of the content it came from, which is what makes a citation + * resolvable. + * + * @param {string} content readable text assembled by a parser + * @param {{segments?: Array<{charStart: number, charEnd: number, byteStart?: number, + * byteEnd?: number, file?: string, label?: string}>, + * groupBy?: "segment"|"blank", maxBlocks?: number, warnings?: string[], + * startIndex?: number}} [options] `startIndex` continues the numbering of + * an existing entry, so a long archive spans `k0007`, `k0008`, … + */ +export function assignAnchorsToText(content, options = {}) { + const source = typeof content === "string" ? content : String(content ?? ""); + return anchorNormalized(normalizeTextWithMap(source, options), options); +} + +function anchorNormalized(normalized, options = {}) { + const lookup = buildSegmentLookup(options.segments); + const lines = splitLeafLines(normalized); + const groups = groupLines(lines, lookup, options); + const startIndex = options.startIndex ?? 1; + if (!Number.isInteger(startIndex) || startIndex < 1) { + throw new TypeError(`startIndex must be a positive integer, received ${startIndex}`); + } + + const units = []; + const warnings = [...(options.warnings ?? []), ...normalized.warnings]; + let unanchored = 0; + + for (const group of groups) { + const text = group.map((line) => line.text).join("\n"); + if (!hasVisibleCodePoint(text)) { + warnings.push("a paragraph had no visible code point and was dropped"); + continue; + } + const id = startIndex + units.length; + // A unit's bytes come from the records it is made of — never from the + // separator whitespace between them, which the assembled buffer contributes + // and the raw payload does not. Without segments the unit *is* a contiguous + // slice of the payload, so its own character span is the byte range. + const raws = group.map((line) => line.raw).filter(Boolean); + const offsets = group.map((line) => line.offset).filter((offset) => offset !== null); + let byteStart = null; + let byteEnd = null; + if (raws.length > 0) { + byteStart = Math.min(...raws.map((raw) => raw.byteStart)); + byteEnd = Math.max(...raws.map((raw) => raw.byteEnd)); + } else if (offsets.length === group.length) { + byteStart = Math.min(...offsets); + byteEnd = Math.max(...group.map((line) => line.byteEnd)); + } + if (byteStart === null || byteEnd === null) unanchored += 1; + + const files = [...new Set(group.map((line) => line.raw?.file ?? null).filter(Boolean))]; + units.push({ + id, + anchor: formatAnchor(id), + kind: "para", + text, + byteStart, + byteEnd, + file: files.length === 1 ? files[0] : null, + files, + contentCharStart: Math.min(...group.map((line) => line.charStart)), + contentCharEnd: Math.max(...group.map((line) => line.charEnd)), + lineCount: group.length, + recordStart: Math.min(...group.map((line) => line.segmentIndex)), + recordEnd: Math.max(...group.map((line) => line.segmentIndex)), + segments: [...new Set(group.map((line) => line.segmentIndex))].sort((a, b) => a - b), + }); + } + + if (unanchored > 0) { + warnings.push( + `${unanchored} paragraph(s) carry text reconstructed from a container (an inflated OOXML member, a stripped HTML body) and therefore report no raw byte offset`, + ); + } + + const rendered = units.length > 0 ? `${units.map((unit) => `${unit.anchor} ${unit.text}`).join("\n\n")}\n` : ""; + + return { + text: rendered, + units, + warnings, + encoding: normalized.encoding, + lossy: normalized.lossy, + byteLength: normalized.bytes.length, + unmappedTail: normalized.unmappedTail, + }; +} + +/** + * `assignAnchors(bytes, …)` — decode a raw payload, normalise it and anchor it in + * one call. For payloads that are already human readable (`.srt`, `.vtt`, a mail + * body, a CSV export) the payload *is* the text, so a paragraph's byte range is + * simply its own character range in the file. + * + * @param {Uint8Array} bytes + */ +export function assignAnchors(bytes, options = {}) { + const buffer = toUint8(bytes); + // Exactly one normalisation pass: a second pass would rebuild the character → + // byte map and move every offset. + return anchorNormalized(normalizeWithMap(buffer, options), options); +} + +/* ------------------------------------------------------------------ */ +/* anchor ids */ +/* ------------------------------------------------------------------ */ + +/** `1` → `k0001`. Ids are zero padded to four digits and grow past 9999. */ +export function formatAnchor(index) { + if (!Number.isInteger(index) || index < 1) { + throw new TypeError(`anchor index must be a positive integer, received ${index}`); + } + return `k${String(index).padStart(4, "0")}`; +} + +const ANCHOR_RE = /^k(\d{4,})(?::t(\d+))?$/; + +/** + * Parse `k0012` or `k0012:t3`. + * @returns {{id: string, index: number, kind: "para"|"sub", subAnchor: string|null, + * subIndex: number|null, canonical: boolean}|null} + */ +export function parseAnchor(value) { + if (typeof value !== "string") return null; + const match = ANCHOR_RE.exec(value); + if (!match) return null; + const index = Number.parseInt(match[1], 10); + const subIndex = match[2] === undefined ? null : Number.parseInt(match[2], 10); + if (subIndex !== null && subIndex < 1) return null; + return { + id: `k${match[1]}`, + index, + kind: subIndex === null ? "para" : "sub", + subAnchor: subIndex === null ? null : `t${subIndex}`, + subIndex, + // Canonical means we would have written it exactly this way ourselves. + canonical: match[1].length === 4, + }; +} + +/** Format the per-record anchor of a ledger entry: `k0007` + 3 → `k0007:t3`. */ +export function formatSubAnchor(kId, index) { + const parsed = parseAnchor(kId); + if (!parsed || parsed.kind !== "para") { + throw new TypeError(`formatSubAnchor needs a paragraph id like k0007, received ${kId}`); + } + if (!Number.isInteger(index) || index < 1) { + throw new TypeError(`sub-anchor index must be a positive integer, received ${index}`); + } + return `${parsed.id}:t${index}`; +} + +/** + * Resolve an anchor against a ledger entry (or any object with `units` and + * `anchors`) and return the text and the byte range it points at. + * + * @param {string} anchor + * @param {{units?: Array, anchors?: Array}} record + * @param {{bytes?: Uint8Array, bytesByFile?: Map}} [raw] + * @returns {{anchor: string, kind: string, text: string, byteStart: number|null, + * byteEnd: number|null, file: string|null, bytes: Uint8Array|null}|null} + */ +export function resolveAnchor(anchor, record, raw = {}) { + const parsed = parseAnchor(anchor); + if (!parsed) return null; + const units = record?.units ?? []; + const listed = record?.anchors ?? []; + + const slice = (file, byteStart, byteEnd) => { + if (!Number.isInteger(byteStart) || !Number.isInteger(byteEnd) || byteEnd < byteStart) return null; + if (raw.bytesByFile && file && raw.bytesByFile.has(file)) { + return raw.bytesByFile.get(file).subarray(byteStart, byteEnd); + } + if (raw.bytes) { + const bytes = toUint8(raw.bytes); + if (byteEnd <= bytes.length) return bytes.subarray(byteStart, byteEnd); + } + return null; + }; + + if (parsed.kind === "sub") { + const direct = listed.find((candidate) => (candidate.anchor ?? candidate.id ?? candidate) === anchor); + if (direct && typeof direct === "object") { + const file = direct.file ?? null; + return { + anchor, + kind: direct.kind ?? "item", + text: direct.text ?? "", + byteStart: direct.byteStart ?? null, + byteEnd: direct.byteEnd ?? null, + file, + bytes: slice(file, direct.byteStart, direct.byteEnd), + }; + } + // Fall back to the matching line of the parent paragraph. + const parent = units[parsed.index - 1]; + if (!parent) return null; + const text = String(parent.text).split("\n")[parsed.subIndex - 1]; + if (text === undefined) return null; + return { + anchor, + kind: "sub", + text, + byteStart: parent.byteStart, + byteEnd: parent.byteEnd, + file: parent.file ?? null, + bytes: slice(parent.file ?? null, parent.byteStart, parent.byteEnd), + }; + } + + const unit = units.find((candidate) => (candidate.anchor ?? candidate.id) === anchor) ?? units[parsed.index - 1]; + if (!unit) return null; + return { + anchor, + kind: unit.kind ?? "para", + text: unit.text, + byteStart: unit.byteStart ?? null, + byteEnd: unit.byteEnd ?? null, + file: unit.file ?? null, + bytes: slice(unit.file ?? null, unit.byteStart, unit.byteEnd), + }; +} + +/** + * Verify the invariants a parsed document must satisfy before it is recorded. + * Returns a list of human readable problems; empty means the document is sound. + * + * Checks: + * 1. every unit has at least one non-whitespace code point + * 2. every unit's byte range is usable (integers, ordered) or explicitly absent + * 3. unit ranges are ordered and non-overlapping per backing buffer + * 4. unit ranges stay inside the payload they claim to come from + * + * @param {object} document + * @param {{fileOrigins?: Map}} [options] + */ +export function verifyByteConservation(document, options = {}) { + const problems = []; + const units = document?.units ?? []; + const byFile = new Map(); + + for (const unit of units) { + if (!hasVisibleCodePoint(unit.text)) { + problems.push(`unit ${unit.anchor} has no non-whitespace code point`); + continue; + } + if (unit.byteStart === null || unit.byteStart === undefined) continue; + if (!Number.isInteger(unit.byteStart) || !Number.isInteger(unit.byteEnd) || unit.byteEnd < unit.byteStart) { + problems.push(`unit ${unit.anchor} has an unusable byte range ${unit.byteStart}..${unit.byteEnd}`); + continue; + } + const file = unit.file ?? ""; + if (!byFile.has(file)) byFile.set(file, []); + byFile.get(file).push([unit.byteStart, unit.byteEnd, unit.anchor]); + } + + for (const [file, ranges] of byFile) { + const sorted = [...ranges].sort((a, b) => a[0] - b[0]); + for (let index = 1; index < sorted.length; index += 1) { + if (sorted[index][0] < sorted[index - 1][1]) { + problems.push( + `${file}: ${sorted[index][2]} starts at ${sorted[index][0]}, inside ${sorted[index - 1][2]} (ends ${sorted[index - 1][1]})`, + ); + } + } + const buffer = options.fileOrigins?.get?.(file); + if (buffer && sorted.length > 0) { + const last = sorted[sorted.length - 1]; + if (last[1] > buffer.length) { + problems.push(`${file}: ${last[2]} ends at ${last[1]}, past the ${buffer.length} byte payload`); + } + } + } + + return { problems, byFile }; +} + +/** + * Byte-conservation accounting across a set of documents: how many bytes of each + * raw payload were anchored, and how many were separators or deliberately + * skipped metadata. The caller compares `unanchored` against what it expects and + * explains the difference in `warnings`. + * + * @param {Map} fileOrigins + * @param {Array} documents + */ +export function conservationReport(fileOrigins, documents) { + const report = []; + for (const [file, bytes] of fileOrigins) { + const covered = new Uint8Array(bytes.length); + let anchored = 0; + for (const document of documents) { + for (const unit of document.units ?? []) { + if ((unit.file ?? null) !== file) continue; + if (!Number.isInteger(unit.byteStart)) continue; + for (let index = unit.byteStart; index < unit.byteEnd && index < bytes.length; index += 1) { + if (covered[index] === 0) { + covered[index] = 1; + anchored += 1; + } + } + } + } + report.push({ + file, + bytes: bytes.length, + anchored, + unanchored: bytes.length - anchored, + coverage: bytes.length === 0 ? 1 : anchored / bytes.length, + }); + } + return report; +} diff --git a/src/knowledge/ledger.mjs b/src/knowledge/ledger.mjs new file mode 100644 index 00000000..d3adf8ff --- /dev/null +++ b/src/knowledge/ledger.mjs @@ -0,0 +1,589 @@ +/** + * ledger.mjs — `knowledge/index.json`, the append-only record of everything that + * entered the knowledge base. + * + * Entry shape, frozen by docs/v2/CONTRACT.md §2: + * + * { id, kind, origin, fetched_at, bytes, sha256, credentialed, method, warnings[] } + * + * Beyond the frozen core we add two field families that parsers in + * `src/parse/**` need in order to make anchors mechanically resolvable, and one + * that makes repeated harvests idempotent: + * + * units[] per-paragraph anchors `[k00NN]` with their exact raw byte ranges + * anchors[] per-turn anchors `[k00NN:tM]` (`turn` / `msg` / `cue` / `item`) + * files[] every raw file this entry was assembled from, with sha256 + * locations raw / text paths relative to `knowledge/` + * imports how many times the same bytes were imported + * + * The ledger is an append-only array. Existing entries are never rewritten, which + * is what makes `index.json` byte-identical across runs: re-importing the same + * bytes produces the same file, not a counter bump. Provenance changes (a + * different id, a new paragraph) require a new entry. + * + * Guarantees: + * - **Append only.** Existing entries are never mutated. + * - **Deterministic bytes.** `renderLedger` sorts keys, so hashing + * `index.json` twice yields the same digest. + * - **Idempotent by content.** `appendEntry` refuses to record the same + * `sha256` twice for the same `origin`; it returns the original entry and a + * reason instead of appending. + */ + +import { existsSync, readFileSync } from "node:fs"; +import { KnowledgeStore, atomicWriteText, sha256Hex, stableStringify } from "./store.mjs"; +import { assignAnchorsToText, conservationReport, formatAnchor, formatSubAnchor, parseAnchor } from "./anchors.mjs"; + +export const LEDGER_SCHEMA_VERSION = 2; + +/** Kinds a ledger entry may declare. */ +export const ENTRY_KINDS = Object.freeze([ + "chat", + "chat-thread", + "email", + "email-thread", + "subtitle", + "document", + "spreadsheet", + "archive", + "archive-record", + "note", + "transcript", +]); + +/** Acquisition methods a zero-credential parser may declare. */ +export const ENTRY_METHODS = Object.freeze([ + "local-file", + "local-directory", + "archive-member", + "user-export", + "model-read", +]); + +/** + * The ledger is a plain JSON **array** of entries — that is what + * `docs/v2/ACCEPTANCE.md` §5 and `scripts/acceptance.mjs` read, and what the rest + * of the pipeline iterates. `loadLedger` accepts either the array form or the + * `{ version, encoder, entries }` envelope so an older file still loads, but + * `renderLedger` always writes the array. + */ +export function emptyLedger() { + const entries = []; + Object.defineProperty(entries, LEDGER_META, { + value: { version: LEDGER_SCHEMA_VERSION, encoder: LEDGER_ENCODER }, + enumerable: false, + writable: true, + configurable: true, + }); + return entries; +} + +const LEDGER_META = Symbol("distilly.ledger.meta"); +const LEDGER_ENCODER = "distilly/knowledge-ledger"; + +function withMeta(entries, meta = {}) { + emptyLedger(); + const list = Array.isArray(entries) ? entries : []; + Object.defineProperty(list, LEDGER_META, { + value: { + version: meta.version ?? LEDGER_SCHEMA_VERSION, + encoder: meta.encoder ?? LEDGER_ENCODER, + }, + enumerable: false, + writable: true, + configurable: true, + }); + return list; +} + +/** + * Read `knowledge/index.json`. A missing file is an empty ledger; an unreadable + * or malformed one is an error, because silently starting over would destroy + * provenance. + */ +export function loadLedger(store) { + const path = store.ledgerPath; + if (!existsSync(path)) return emptyLedger(); + let raw; + try { + raw = readFileSync(path, "utf8"); + } catch (error) { + throw new Error(`cannot read the knowledge ledger at ${path}: ${error.message}`); + } + if (raw.trim() === "") return emptyLedger(); + let parsed; + try { + parsed = JSON.parse(raw); + } catch (error) { + throw new Error(`knowledge ledger at ${path} is not valid JSON: ${error.message}`); + } + if (Array.isArray(parsed)) { + for (const entry of parsed) { + if (!entry || typeof entry !== "object" || typeof entry.id !== "string") { + throw new Error(`knowledge ledger at ${path} has an entry without an id`); + } + } + return withMeta(parsed); + } + if (parsed && typeof parsed === "object" && Array.isArray(parsed.entries)) { + return withMeta(parsed.entries, parsed); + } + throw new Error(`knowledge ledger at ${path} is neither an array nor an object with entries[]`); +} + +/** Serialise the ledger deterministically (sorted keys, trailing newline). */ +export function renderLedger(ledger) { + return `${stableStringify([...ledger], 2, true)}\n`; +} + +export function ledgerMeta(ledger) { + return ledger?.[LEDGER_META] ?? { version: LEDGER_SCHEMA_VERSION, encoder: LEDGER_ENCODER }; +} + +export function ledgerSha256(ledger) { + return sha256Hex(Buffer.from(renderLedger(ledger), "utf8")); +} + +export function saveLedger(store, ledger) { + const text = renderLedger(ledger); + if (!store.dryRun) { + store.ensure(); + atomicWriteText(store.ledgerPath, text); + } + return { path: store.ledgerPath, bytes: Buffer.byteLength(text, "utf8"), sha256: sha256Hex(Buffer.from(text, "utf8")) }; +} + +/** Highest allocated numeric suffix, so ids never go backwards. */ +export function lastId(ledger) { + let highest = 0; + for (const entry of ledger.entries) { + const parsed = parseAnchor(entry?.id); + if (parsed && parsed.index > highest) highest = parsed.index; + } + return highest; +} + +export function nextId(ledger) { + return formatAnchor(lastId(ledger) + 1); +} + +export function findEntry(ledger, predicate) { + return ledger.entries.find(predicate) ?? null; +} + +/** Look up by `k00NN`. */ +export function getEntry(ledger, id) { + const parsed = parseAnchor(id); + if (!parsed) return null; + return ledger.entries.find((entry) => entry.id === parsed.id) ?? null; +} + +/** Find the entry that recorded these exact bytes. */ +export function findBySha256(ledger, sha256) { + return ledger.entries.filter((entry) => entry.sha256 === sha256); +} + +/** + * Every anchor id the ledger knows about, for the `doctor` / anchor-integrity + * gate in the contract (§5): a document may only cite an anchor that resolves. + * @returns {Set} + */ +export function anchorIndex(ledger) { + const index = new Set(); + for (const entry of ledger.entries) { + index.add(entry.id); + for (const unit of entry.units ?? []) index.add(unit.anchor ?? `${entry.id}`); + for (const sub of entry.anchors ?? []) index.add(sub.anchor); + } + return index; +} + +/** + * Resolve a `[k00NN]` / `[k00NN:tM]` citation anywhere in the ledger. + * @returns {{entry: object, anchor: string, text: string, byteStart: number, + * byteEnd: number, file: string|null, kind: string}|null} + */ +export function resolveLedgerAnchor(ledger, anchor) { + const parsed = parseAnchor(anchor); + if (!parsed) return null; + const entry = getEntry(ledger, parsed.id); + if (!entry) return null; + + if (parsed.kind === "sub") { + const sub = (entry.anchors ?? []).find((candidate) => candidate.anchor === anchor); + if (!sub) return null; + return { + entry, + anchor, + kind: sub.kind ?? "item", + text: sub.text ?? "", + byteStart: sub.byteStart ?? null, + byteEnd: sub.byteEnd ?? null, + file: sub.file ?? null, + }; + } + + const unit = (entry.units ?? []).find((candidate) => candidate.anchor === anchor); + if (!unit) return null; + return { + entry, + anchor, + kind: "para", + text: unit.text ?? "", + byteStart: unit.byteStart ?? null, + byteEnd: unit.byteEnd ?? null, + file: unit.file ?? null, + }; +} + +function normaliseWarnings(warnings) { + const list = Array.isArray(warnings) ? warnings : warnings ? [String(warnings)] : []; + return list.map((warning) => String(warning)).filter((warning) => warning.trim() !== ""); +} + +/** + * Build a ledger entry from a parsed document. + * + * `document` is whatever `src/parse/*.mjs` returned; see `src/parse/common.mjs` + * for the exact shape. Nothing is invented here: missing fields stay missing so + * downstream consumers can tell "not applicable" from "zero". + */ +export function buildEntry(document, options = {}) { + const id = options.id; + if (!id) throw new TypeError("buildEntry requires an id (allocate it with nextId)"); + const parsedId = parseAnchor(id); + if (!parsedId || parsedId.kind !== "para") { + throw new TypeError(`buildEntry requires a paragraph id like k0001, received ${id}`); + } + + const rawFiles = document.files ?? []; + if (rawFiles.length === 0) throw new TypeError(`entry ${id} has no raw files`); + + const primary = rawFiles[0]; + const origin = options.origin ?? document.origin ?? primary.path; + const fetchedAt = document.fetched_at ?? options.fetched_at ?? null; + if (!fetchedAt) { + throw new TypeError(`entry ${id} needs fetched_at — pass it explicitly so runs stay deterministic`); + } + + const warnings = normaliseWarnings([...(document.warnings ?? []), ...(options.warnings ?? [])]); + + return { + id, + kind: document.kind, + origin, + fetched_at: fetchedAt, + bytes: rawFiles.reduce((total, file) => total + (file.bytes ?? 0), 0), + sha256: options.sha256 ?? primary.sha256, + credentialed: Boolean(document.credentialed), + method: document.method ?? "local-file", + warnings, + files: rawFiles.map((file) => ({ + path: file.relativePath ?? file.path, + bytes: file.bytes ?? 0, + sha256: file.sha256 ?? null, + encoding: file.encoding ?? null, + ...(file.members !== undefined ? { members: file.members } : {}), + })), + locations: { + raw: primary.relativePath ?? primary.path, + text: document.textRelativePath ?? null, + }, + counts: { + units: (document.units ?? []).length, + anchors: (document.anchors ?? []).length, + }, + units: (document.units ?? []).map((unit) => ({ + id: unit.anchor, + anchor: unit.anchor, + text: unit.text, + byteStart: unit.byteStart, + byteEnd: unit.byteEnd, + file: unit.file ?? primary.relativePath ?? primary.path, + })), + // `anchors` carries *every* anchor this entry owns — the paragraph anchors + // plus the per-turn/cue/message sub-anchors — because that is the list the + // anchor-integrity gate (`scripts/acceptance.mjs`, `doctor`) reads. Elements + // are objects with `id`; `anchor` is the same value kept for old callers. + anchors: (document.anchors ?? []).map((anchor) => ({ + id: anchor.anchor ?? anchor.id, + anchor: anchor.anchor ?? anchor.id, + kind: anchor.kind, + text: anchor.text, + byteStart: anchor.byteStart ?? null, + byteEnd: anchor.byteEnd ?? null, + file: anchor.file ?? primary.relativePath ?? primary.path, + ...(anchor.label ? { label: anchor.label } : {}), + })), + imports: 1, + }; +} + +/** + * Append `entry` to `ledger` unless its bytes are already recorded. + * + * Idempotency rule: same `sha256` (and, when both sides declare one, same + * `origin`) ⇒ no new entry. The existing entry keeps its id and `fetched_at` + * — the clock is never re-read, which is exactly what makes a second harvest + * produce byte-identical `index.json`. + * + * @param {object} ledger + * @param {object} entry + * @param {{allowDuplicate?: boolean}} [options] + * @returns {{entry: object, appended: boolean, duplicateOf: object|null, reason: string}} + */ +export function appendEntry(ledger, entry, options = {}) { + const existing = ledger.entries.find( + (candidate) => + candidate.sha256 === entry.sha256 && + (candidate.origin === entry.origin || !entry.origin || !candidate.origin), + ); + + if (existing && !options.allowDuplicate) { + existing.imports = (existing.imports ?? 1) + 1; + return { + entry: existing, + appended: false, + duplicateOf: existing, + reason: `identical bytes already recorded as ${existing.id} (sha256 ${existing.sha256})`, + }; + } + + ledger.entries.push(entry); + return { entry, appended: true, duplicateOf: null, reason: "new content recorded" }; +} + +/** + * Store a parsed document and record it in the ledger — the single entry point + * every parser in `src/parse/**` funnels through. + * + * Anchoring happens *here*, not in the parsers: only the ledger knows which + * `k00NN` id is free, and an anchor must never be renumbered after a citation + * exists. Parsers hand over readable text plus the character spans of their + * turns/records; we stamp the paragraph anchors, derive the `[k00NN:tM]` + * sub-anchors, verify byte conservation and only then touch the disk. + * + * Order matters: raw bytes first, then normalised text, then the ledger. A crash + * between the steps leaves orphaned raw bytes (harmless — the next run writes + * the identical bytes) but never a ledger entry pointing at a missing file. + * + * @param {KnowledgeStore} store + * @param {object} ledger mutated in place + * @param {object} document a parsed document from `src/parse/**` + * @param {{id?: string, origin?: string, fetched_at?: string, allowDuplicate?: boolean}} [options] + * @returns {{entry: object|null, appended: boolean, duplicateOf: object|null, + * reason: string, id: string|null, written: object|null, units: Array, + * conservation: Array, text: string}} + */ +export function recordDocument(store, ledger, document, options = {}) { + const source = document.source; + if (!source) throw new TypeError("document.source is required to place raw bytes"); + + const fetchedAt = document.fetched_at ?? options.fetched_at; + if (!fetchedAt) { + throw new TypeError("recordDocument needs fetched_at (pass it explicitly so runs stay deterministic)"); + } + + store.ensure(); + + const storedFiles = []; + for (const file of document.files ?? []) { + if (file.persisted) { + storedFiles.push(file); + continue; + } + if (!file.bytesRaw) { + throw new TypeError(`document file ${file.path} carries neither bytesRaw nor a persisted record`); + } + const stored = store.writeRaw(source, file.name ?? file.path, file.bytesRaw); + storedFiles.push({ + ...file, + relativePath: stored.relativePath, + absolutePath: stored.path, + bytes: stored.bytes, + sha256: stored.sha256, + persisted: true, + // Drop the byte payload from the descriptor we keep in memory: it is on + // disk now and holding it would double memory for large archives. + bytesRaw: undefined, + }); + } + + // Anchor the assembled content, continuing the numbering when a single + // document needs more paragraphs than one id can own (see `chunkDocuments`). + const id = options.id ?? nextId(ledger); + const ledgerIndex = parseAnchor(id).index; + const anchored = assignAnchorsToText(document.content ?? "", { + segments: document.segments ?? [], + groupBy: document.groupBy, + maxBlocks: document.maxBlocks, + warnings: document.warnings ?? [], + startIndex: ledgerIndex, + }); + + // Sub-anchors are keyed by the entry id they belong to, not by the paragraph + // they happen to render next to, so a citation stays stable when the document + // is re-chunked. + const primaryUnit = anchored.units[0] ?? { anchor: id, id: ledgerIndex, byteStart: 0, byteEnd: 0 }; + + const seen = new Map(); + const subAnchors = (document.entries ?? []).map((entry) => { + const key = entry.file ?? storedFiles[0]?.name ?? null; + const ordinal = (seen.get(key) ?? 0) + 1; + seen.set(key, ordinal); + return { + kind: entry.kind ?? "item", + text: entry.text, + anchor: formatSubAnchor(id, ordinal), + id: formatSubAnchor(id, ordinal), + index: ordinal, + byteStart: entry.byteStart ?? null, + byteEnd: entry.byteEnd ?? null, + file: entry.file ?? storedFiles[0]?.name ?? null, + label: entry.label ?? null, + }; + }); + + const conservation = conservationReport( + new Map(storedFiles.map((file) => [file.relativePath, file.bytesRaw ?? Buffer.alloc(0)])), + [{ units: anchored.units.map((unit) => ({ ...unit, file: storedFiles[0]?.relativePath ?? null })) }], + ); + + // Nothing survived parsing: keep the raw bytes, do not invent an entry. + if (anchored.units.length === 0 && subAnchors.length === 0) { + return { + entry: null, + appended: false, + duplicateOf: null, + reason: "parsed to no anchored content; raw bytes retained, no ledger entry created", + id: null, + requestedId: id, + written: { text: null, files: storedFiles }, + units: [], + anchors: [], + conservation, + text: "", + }; + } + + const textWrite = store.writeText(source, anchored.text, document.textStem); + const prepared = { + ...document, + files: storedFiles.map((file) => ({ + path: file.relativePath, + name: file.name, + relativePath: file.relativePath, + bytes: file.bytes, + sha256: file.sha256, + encoding: file.encoding ?? null, + members: file.members, + })), + }; + + const paragraphAnchors = anchored.units.map((unit) => ({ + anchor: unit.anchor, + id: unit.anchor, + kind: "para", + text: unit.text, + byteStart: unit.byteStart, + byteEnd: unit.byteEnd, + file: storedFiles[0]?.relativePath ?? null, + })); + + const entry = buildEntry( + { + ...prepared, + units: anchored.units.map((unit) => ({ ...unit, file: storedFiles[0]?.relativePath ?? null })), + // One flat list: paragraph anchors first, then the sub-anchors, so + // `entry.anchors.map(a => a.id)` is the entry's whole anchor namespace. + anchors: [...paragraphAnchors, ...subAnchors], + textRelativePath: textWrite.relativePath, + }, + { ...options, id, fetched_at: fetchedAt, unitCount: anchored.units.length }, + ); + entry.anchor_count = anchored.units.length + subAnchors.length; + entry.primary_anchor = primaryUnit.anchor; + if (document.accounting) entry.accounting = document.accounting; + + const result = appendEntry(ledger, entry, options); + + return { + ...result, + id: result.entry.id, + requestedId: id, + written: { text: textWrite, files: storedFiles }, + units: result.entry.units ?? [], + anchors: result.entry.anchors ?? [], + conservation, + text: anchored.text, + }; +} + +/** + * Split a document into chunks small enough that one `k00NN` id owns a sane + * number of paragraphs, keeping the raw files on the first chunk only. + * + * `knowledge/text/.md` is written per entry, so a 30 000 message chat + * would otherwise produce a single file that no reader (human or model) can + * navigate, and every paragraph anchor under one id. + * + * @param {object} document + * @param {{maxParagraphs?: number}} [options] + * @returns {Array} + */ +export function chunkDocuments(document, options = {}) { + const maxParagraphs = options.maxParagraphs ?? 500; + const segments = document.segments ?? []; + if (segments.length <= maxParagraphs) return [document]; + + const entries = document.entries ?? []; + const chunks = []; + for (let start = 0; start < segments.length; start += maxParagraphs) { + const slice = segments.slice(start, start + maxParagraphs); + const charStart = slice[0].charStart; + const charEnd = slice[slice.length - 1].charEnd; + chunks.push({ + ...document, + // Raw bytes are written once, by the first chunk; later chunks reference + // the same relative paths. + files: start === 0 ? document.files : document.files.map((file) => ({ ...file, bytesRaw: undefined, persisted: true })), + content: document.content.slice(charStart, charEnd), + segments: slice.map((segment) => ({ + ...segment, + charStart: segment.charStart - charStart, + charEnd: segment.charEnd - charStart, + })), + entries: entries.slice(start, start + maxParagraphs), + textStem: document.textStem, + meta: { ...(document.meta ?? {}), chunk: Math.floor(start / maxParagraphs) + 1, chunksOf: document.source }, + }); + } + return chunks; +} + +/** Ledger statistics used by `doctor` and by the evidence report. */ +export function ledgerStats(ledger) { + const byKind = {}; + const byMethod = {}; + let bytes = 0; + let warnings = 0; + let anchors = 0; + for (const entry of ledger.entries) { + byKind[entry.kind] = (byKind[entry.kind] ?? 0) + 1; + byMethod[entry.method] = (byMethod[entry.method] ?? 0) + 1; + bytes += entry.bytes ?? 0; + warnings += (entry.warnings ?? []).length; + anchors += (entry.units ?? []).length + (entry.anchors ?? []).length; + } + return { + entries: ledger.entries.length, + bytes, + anchors, + warnings, + credentialed: ledger.entries.filter((entry) => entry.credentialed).length, + byKind, + byMethod, + }; +} + +export { KnowledgeStore }; diff --git a/src/knowledge/store.mjs b/src/knowledge/store.mjs new file mode 100644 index 00000000..57b1ca90 --- /dev/null +++ b/src/knowledge/store.mjs @@ -0,0 +1,376 @@ +/** + * store.mjs — the byte vault under `knowledge/`. + * + * Layout (see docs/v2/CONTRACT.md §2): + * + * knowledge/raw//<...> raw bytes, append only, never rewritten + * knowledge/text/.md normalised text carrying `[k00NN]` anchors + * knowledge/index.json the ledger (owned by ledger.mjs) + * + * Two rules this module enforces mechanically: + * + * - **Raw bytes are stored verbatim.** `writeRaw` accepts a Uint8Array and + * writes exactly those bytes; nothing is re-encoded, normalised or trimmed on + * the way to disk. `readRaw` is asserted to return the identical bytes. + * - **Writes are atomic.** Every file is staged in the destination directory, + * fsync'd, then `rename(2)`d into place, so a crash leaves either the old file + * or the new one — never a half written `index.json`. + * + * The store is a pure filesystem object: no clock, no randomness, no globals. + * Callers that need determinism pass `fetched_at` explicitly. + */ + +import { createHash, randomBytes } from "node:crypto"; +import { + closeSync, + existsSync, + fsyncSync, + mkdirSync, + openSync, + readFileSync, + readdirSync, + renameSync, + rmSync, + statSync, + writeFileSync, + writeSync, +} from "node:fs"; +import { basename, dirname, join, resolve, sep } from "node:path"; + +export const LEDGER_FILE = "index.json"; +export const RAW_DIR = "raw"; +export const TEXT_DIR = "text"; + +/** Directory/file names that may never be used as a `` bucket. */ +const FORBIDDEN_SOURCE_SEGMENTS = new Set([".", "..", "", "/"]); + +export function sha256Hex(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +export function sha256Text(text) { + return sha256Hex(Buffer.from(text, "utf8")); +} + +function toUint8(bytes) { + if (bytes instanceof Uint8Array) return bytes; + if (bytes instanceof ArrayBuffer) return new Uint8Array(bytes); + if (Array.isArray(bytes)) return Uint8Array.from(bytes); + if (typeof bytes === "string") return Buffer.from(bytes, "utf8"); + throw new TypeError("expected a Uint8Array, ArrayBuffer, number[] or string"); +} + +/** + * Normalise a source name into a safe single path segment. + * `"ChatGPT Export 2024"` → `"chatgpt-export-2024"`. + */ +export function slugifySource(source) { + const slug = String(source ?? "") + .normalize("NFKD") + .toLowerCase() + .replace(/[^a-z0-9._-]+/g, "-") + .replace(/-+/g, "-") + .replace(/^[-._]+|[-._]+$/g, ""); + if (!slug || FORBIDDEN_SOURCE_SEGMENTS.has(slug)) { + throw new TypeError(`cannot derive a directory name from source ${JSON.stringify(source)}`); + } + return slug.slice(0, 96); +} + +/** + * Reduce a file name to something that cannot escape the bucket: path + * separators collapse to `-`, `..` is neutralised, control characters go. + */ +export function slugifyFileName(name) { + const base = basename(String(name ?? "")); + const slug = base + .normalize("NFKD") + // eslint-disable-next-line no-control-regex + .replace(/[\u0000-\u001f\u007f/\\]/g, "-") + .replace(/\.{2,}/g, ".") + .replace(/^[-.]+/, "") + .replace(/[-.\s]+$/, ""); + return slug || "payload"; +} + +function fsyncFile(path) { + const fd = openSync(path, "r"); + try { + fsyncSync(fd); + } finally { + closeSync(fd); + } +} + +/** + * Write `bytes` to `path` atomically. + * + * The staging file lives in the *same* directory as the destination so the + * final `rename` is a same-filesystem operation (atomic on POSIX). The staging + * name is random so two concurrent writers cannot collide. + * + * @param {string} path + * @param {Uint8Array} bytes + * @param {{durable?: boolean}} [options] `durable` also fsyncs the file (default true) + * @returns {{path: string, bytes: number, sha256: string}} + */ +export function atomicWriteBytes(path, bytes, options = {}) { + const buffer = Buffer.from(toUint8(bytes)); + const { durable = true } = options; + const directory = dirname(path); + + mkdirSync(directory, { recursive: true }); + if (existsSync(path) && statSync(path).isDirectory()) { + throw new Error(`refusing to overwrite the directory ${path} with a file`); + } + + const staging = join(directory, `.${basename(path)}.${process.pid}.${randomBytes(6).toString("hex")}.tmp`); + try { + if (durable) { + const fd = openSync(staging, "wx", 0o644); + try { + writeSync(fd, buffer); + fsyncSync(fd); + } finally { + closeSync(fd); + } + } else { + writeFileSync(staging, buffer, { flag: "wx" }); + } + renameSync(staging, path); + } catch (error) { + if (existsSync(staging)) { + try { + rmSync(staging, { force: true }); + } catch { + // Best effort: the staging file is inert, and the caller sees `error`. + } + } + throw error; + } + + return { path, bytes: buffer.length, sha256: sha256Hex(buffer) }; +} + +/** Write UTF-8 text atomically (no BOM, LF endings, exactly as given). */ +export function atomicWriteText(path, text, options = {}) { + return atomicWriteBytes(path, Buffer.from(String(text), "utf8"), options); +} + +/** Atomic replace with `fsync`, mirroring what a durable append costs. */ +export function atomicWriteJson(path, value, options = {}) { + const { indent = 2, sortKeys = true } = options; + return atomicWriteText(path, `${stableStringify(value, indent, sortKeys)}\n`, options); +} + +/** + * Deterministic `JSON.stringify`: object keys are ordered, so the same logical + * value always hashes to the same bytes (the contract's "run twice, same + * sha256" gate depends on this). + */ +export function stableStringify(value, indent = 2, sortKeys = true) { + const stack = new WeakSet(); + const normalise = (input) => { + if (input === null || typeof input !== "object") return input; + if (stack.has(input)) throw new TypeError("cannot serialise a cyclic structure"); + stack.add(input); + let output; + if (Array.isArray(input)) { + output = input.map((item) => normalise(item)); + } else if (input instanceof Map) { + output = {}; + for (const key of [...input.keys()].map(String).sort()) { + output[key] = normalise(input.get(key)); + } + } else { + const keys = Object.keys(input); + if (sortKeys) keys.sort(); + output = {}; + for (const key of keys) { + if (input[key] === undefined) continue; + output[key] = normalise(input[key]); + } + } + stack.delete(input); + return output; + }; + return JSON.stringify(normalise(value), null, indent); +} + +export class KnowledgeStore { + /** + * @param {string} root directory that *contains* `knowledge/` + * @param {{fetched_at?: string, dryRun?: boolean}} [options] + */ + constructor(root, options = {}) { + if (!root) throw new TypeError("KnowledgeStore requires a root directory"); + this.root = resolve(root); + this.knowledgeRoot = join(this.root, "knowledge"); + this.dryRun = Boolean(options.dryRun); + this.fetched_at = options.fetched_at ?? null; + } + + get rawRoot() { + return join(this.knowledgeRoot, RAW_DIR); + } + + get textRoot() { + return join(this.knowledgeRoot, TEXT_DIR); + } + + get ledgerPath() { + return join(this.knowledgeRoot, LEDGER_FILE); + } + + ensure() { + if (this.dryRun) return this; + mkdirSync(this.rawRoot, { recursive: true }); + mkdirSync(this.textRoot, { recursive: true }); + return this; + } + + /** + * Absolute path of the deployed location for `/`. + * `name` may contain sub-directories (archive members, mail folders, …); + * escapes are rejected rather than sanitised, because a silent rewrite of the + * path would break the anchor→raw mapping. + */ + rawPath(source, name) { + const bucket = slugifySource(source); + const relative = String(name ?? "").split(/[\\/]+/).filter(Boolean); + if (relative.length === 0) throw new TypeError("rawPath requires a file name"); + const target = resolve(this.rawRoot, bucket, ...relative); + const prefix = resolve(this.rawRoot, bucket) + sep; + if (!target.startsWith(prefix)) { + throw new TypeError(`refusing a raw path that escapes its bucket: ${name}`); + } + return target; + } + + /** + * `knowledge/text/.md`, or `--.md` when several distinct + * inputs share one source bucket (three X archives all named `tweets.js`), so + * no two entries can ever overwrite each other's text. + */ + textPath(source, stem) { + const slug = slugifySource(source); + if (!stem) return join(this.textRoot, `${slug}.md`); + const suffix = slugifySource(stem); + return join(this.textRoot, suffix === slug ? `${slug}.md` : `${slug}--${suffix}.md`); + } + + hasRaw(source, name) { + return existsSync(this.rawPath(source, name)); + } + + /** Read raw bytes back. Always returns a fresh Uint8Array. */ + readRaw(source, name) { + return new Uint8Array(readFileSync(this.rawPath(source, name))); + } + + /** + * Store raw bytes verbatim. + * @returns {{path: string, relativePath: string, bytes: number, sha256: string, + * created: boolean}} + */ + writeRaw(source, name, bytes) { + const target = this.rawPath(source, name); + const buffer = Buffer.from(toUint8(bytes)); + const sha256 = sha256Hex(buffer); + const existed = existsSync(target); + if (!this.dryRun) { + atomicWriteBytes(target, buffer); + // Read the bytes back so a mis-encoded write can never go unnoticed. + const readBack = readFileSync(target); + if (!readBack.equals(buffer)) { + throw new Error(`raw bytes changed on disk: ${target}`); + } + } + return { + path: target, + relativePath: `${RAW_DIR}/${slugifySource(source)}/${String(name).split(/[\\/]+/).filter(Boolean).join("/")}`, + bytes: buffer.length, + sha256, + created: !existed, + }; + } + /** Store normalised text carrying anchors. */ + writeText(source, text, stem) { + const target = this.textPath(source, stem); + const buffer = Buffer.from(String(text), "utf8"); + if (!this.dryRun) atomicWriteBytes(target, buffer); + const relative = target.slice(this.knowledgeRoot.length + 1).split(sep).join("/"); + return { + path: target, + relativePath: relative, + bytes: buffer.length, + sha256: sha256Hex(buffer), + }; + } + + /** + * List everything under a bucket, relative to it, sorted for determinism. + * @returns {Array<{name: string, bytes: number}>} + */ + listRaw(source) { + const bucket = join(this.rawRoot, slugifySource(source)); + if (!existsSync(bucket)) return []; + const results = []; + const walk = (directory, prefix) => { + for (const entry of readdirSync(directory, { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name))) { + const absolute = join(directory, entry.name); + const relative = prefix ? `${prefix}/${entry.name}` : entry.name; + if (entry.isDirectory()) { + walk(absolute, relative); + } else if (entry.isFile()) { + results.push({ name: relative, bytes: statSync(absolute).size }); + } + } + }; + walk(bucket, ""); + return results; + } + + /** Byte length of the raw file, or null when it is absent. */ + rawSize(source, name) { + const target = this.rawPath(source, name); + return existsSync(target) ? statSync(target).size : null; + } + + /** Remove a whole bucket. Only used by tests and `--force` style rebuilds. */ + purgeSource(source) { + const bucket = join(this.rawRoot, slugifySource(source)); + if (existsSync(bucket)) rmSync(bucket, { recursive: true, force: true }); + const text = this.textPath(source); + if (existsSync(text)) rmSync(text, { force: true }); + } +} + +/** + * Copy `bytes` into the store only when their sha256 differs from what is + * already there; reports whether the bytes were newly written. + */ +export function putRawOnce(store, source, name, bytes) { + const buffer = Buffer.from(toUint8(bytes)); + const sha256 = sha256Hex(buffer); + const target = store.rawPath(source, name); + if (existsSync(target) && sha256Hex(readFileSync(target)) === sha256) { + return { ...store.writeRaw(source, name, buffer), created: false, unchanged: true }; + } + const result = store.writeRaw(source, name, buffer); + return { ...result, unchanged: false }; +} + +/** Append bytes to a file (used only by tests and log-style payloads). */ +export function appendBytesSync(path, bytes) { + mkdirSync(dirname(path), { recursive: true }); + const fd = openSync(path, "a"); + try { + writeSync(fd, Buffer.from(toUint8(bytes))); + } finally { + closeSync(fd); + } + return path; +} + +export { existsSync, mkdirSync, readFileSync, readdirSync, statSync }; diff --git a/src/optional/transcribe.mjs b/src/optional/transcribe.mjs new file mode 100644 index 00000000..3c06f01a --- /dev/null +++ b/src/optional/transcribe.mjs @@ -0,0 +1,702 @@ +/** + * transcribe.mjs — the optional transcription backend. + * + * Two backends, no silent downgrade: + * + * openai-http an OpenAI-compatible `POST {base}/audio/transcriptions` + * (multipart, `whisper-1` by default) using an env credential. + * host the host (a computer-use / model host) transcribes the file and + * hands the transcript over with `--capture `. + * + * If neither is available the command fails loudly with `unavailable` and a + * remediation list. It never returns an empty transcript, never pretends the + * audio was transcribed and never falls back to a local model behind the user's + * back — this is exactly what the legacy helper did wrong: + * `tools/research/transcribe_audio.py:145` returns `""` when `OPENAI_API_KEY` is + * missing and `:152` returns `""` when the `openai` package is not installed, so + * callers silently received "no transcript" and carried on. + * + * Successful artifacts carry `provenance {method, producer, confidence}` on the + * receipt, in `knowledge/text/.md` front matter and in the ledger entry. + */ + +import { createHash } from "node:crypto"; +import { + existsSync, + mkdirSync, + readFileSync, + renameSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; +import { homedir } from "node:os"; +import { basename, dirname, join, resolve } from "node:path"; + +export const COMMAND = "transcribe"; +export const CONFIG_FILE = "transcribe_config.json"; +export const DEFAULT_BASE_URL = "https://api.openai.com/v1"; +export const DEFAULT_MODEL = "whisper-1"; +export const DEFAULT_MAX_RETRIES = 3; +export const DEFAULT_MAX_BACKOFF_MS = 60_000; +/** OpenAI's documented upload ceiling; the legacy helper warned at the same size. */ +export const MAX_UPLOAD_BYTES = 25 * 1024 * 1024; + +export const ENV_KEYS = { + apiKey: ["DISTILLY_TRANSCRIBE_API_KEY", "OPENAI_API_KEY"], + baseUrl: ["DISTILLY_TRANSCRIBE_BASE_URL", "OPENAI_BASE_URL"], + model: ["DISTILLY_TRANSCRIBE_MODEL", "OPENAI_TRANSCRIBE_MODEL"], +}; + +const REMEDIATION_SETUP = [ + `export DISTILLY_TRANSCRIBE_API_KEY=… (or OPENAI_API_KEY) for the OpenAI-compatible backend`, + ` optional: DISTILLY_TRANSCRIBE_BASE_URL (default ${DEFAULT_BASE_URL}), DISTILLY_TRANSCRIBE_MODEL (default ${DEFAULT_MODEL})`, + ` or store the same keys in ~/.distilly/${CONFIG_FILE} (chmod 600)`, + "or let the host transcribe and register the result: distilly transcribe --capture ", +]; + +export class TranscribeFailure extends Error { + constructor(reason, message, { remediation = [], exitCode = 1 } = {}) { + super(message); + this.name = "TranscribeFailure"; + this.reason = reason; + this.remediation = remediation; + this.exitCode = exitCode; + } +} + +export function redact(text, secrets = []) { + let output = String(text ?? ""); + for (const secret of secrets) { + if (typeof secret !== "string" || secret.length < 4) continue; + output = output.split(secret).join("[redacted]"); + } + return output; +} + +export function scrub(value, secrets = []) { + return JSON.parse( + JSON.stringify(value, (_key, item) => (typeof item === "string" ? redact(item, secrets) : item)), + ); +} + +export function distillyHome(env = process.env) { + const override = env?.DISTILLY_HOME; + return override ? resolve(String(override)) : join(homedir(), ".distilly"); +} + +/** Read the HTTP credential from env first, then `~/.distilly/transcribe_config.json`. */ +export function loadCredential({ env = process.env, readFile = readFileSync } = {}) { + const pick = (names) => { + for (const name of names) { + const value = env?.[name]; + if (typeof value === "string" && value.trim() !== "") return value.trim(); + } + return null; + }; + + const envKey = pick(ENV_KEYS.apiKey); + if (envKey) { + return { + ok: true, + source: "env", + configFile: CONFIG_FILE, + path: null, + values: { + api_key: envKey, + base_url: pick(ENV_KEYS.baseUrl) ?? DEFAULT_BASE_URL, + model: pick(ENV_KEYS.model) ?? DEFAULT_MODEL, + }, + }; + } + + const path = join(distillyHome(env), CONFIG_FILE); + if (!existsSync(path)) return { ok: false, source: null, configFile: CONFIG_FILE, path, values: null }; + + let parsed; + try { + parsed = JSON.parse(readFile(path, "utf8")); + } catch (error) { + throw new TranscribeFailure( + "bad-credential-file", + `${CONFIG_FILE} is not valid JSON (${redact(error.message)})`, + { remediation: REMEDIATION_SETUP }, + ); + } + const apiKey = parsed.api_key ?? parsed.apiKey ?? parsed.key ?? null; + if (!apiKey) { + return { ok: false, source: "config", configFile: CONFIG_FILE, path, values: null }; + } + return { + ok: true, + source: "config", + configFile: CONFIG_FILE, + path, + values: { + api_key: apiKey, + base_url: parsed.base_url ?? parsed.baseUrl ?? DEFAULT_BASE_URL, + model: parsed.model ?? DEFAULT_MODEL, + }, + }; +} + +export function parseRetryAfter(headerValue, nowMs = Date.now()) { + if (headerValue === null || headerValue === undefined || headerValue === "") return null; + const seconds = Number(headerValue); + if (Number.isFinite(seconds) && seconds >= 0) return Math.round(seconds * 1000); + const date = Date.parse(String(headerValue)); + if (Number.isFinite(date)) return Math.max(0, date - nowMs); + return null; +} + +export function backoffDelay(attempt, retryAfterMs = null, options = {}) { + const { baseMs = 500, maxMs = DEFAULT_MAX_BACKOFF_MS } = options; + if (Number.isFinite(retryAfterMs) && retryAfterMs !== null) return Math.min(retryAfterMs, maxMs); + return Math.min(baseMs * 2 ** Math.max(0, attempt - 1), maxMs); +} + +export function defaultSleep(ms) { + return new Promise((resolvePromise) => setTimeout(resolvePromise, ms)); +} + +export function sha256Hex(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +export function slug(text, fallback = "transcript") { + const slugged = String(text ?? "") + .normalize("NFKD") + .toLowerCase() + .replace(/[^a-z0-9._-]+/g, "-") + .replace(/-+/g, "-") + .replace(/^[-._]+|[-._]+$/g, "") + .slice(0, 64); + return slugged || fallback; +} + +export function knowledgeRoot({ root = process.cwd(), person, family = "colleague" } = {}) { + return person ? join(resolve(root), "skills", slug(family), slug(person), "knowledge") : join(resolve(root), "knowledge"); +} + +/** + * Deterministic multipart body: the boundary is derived from the payload hash, + * so the same input always produces the same request bytes (and a mock can + * assert them without parsing a stream). + */ +export function buildMultipart({ boundaryKey, fields = {}, file }) { + const boundary = `----distilly-${boundaryKey}`; + const chunks = []; + for (const [name, value] of Object.entries(fields)) { + chunks.push(Buffer.from(`--${boundary}\r\nContent-Disposition: form-data; name="${name}"\r\n\r\n${value}\r\n`, "utf8")); + } + chunks.push( + Buffer.from( + `--${boundary}\r\nContent-Disposition: form-data; name="${file.field}"; filename="${file.filename}"\r\n` + + `Content-Type: ${file.contentType}\r\n\r\n`, + "utf8", + ), + ); + chunks.push(Buffer.from(file.bytes)); + chunks.push(Buffer.from(`\r\n--${boundary}--\r\n`, "utf8")); + const body = Buffer.concat(chunks); + return { boundary, body, contentType: `multipart/form-data; boundary=${boundary}` }; +} + +/** Timestamped, provider-independent transcript body. */ +export function formatTranscript(payload) { + const segments = Array.isArray(payload?.segments) ? payload.segments : []; + const lines = []; + for (const segment of segments) { + const text = String(segment?.text ?? "").trim(); + if (!text) continue; + const start = Number(segment?.start ?? 0); + const m = Math.floor(start / 60); + const s = Math.floor(start % 60); + lines.push(`[${String(Math.floor(m / 60)).padStart(2, "0")}:${String(m % 60).padStart(2, "0")}:${String(s).padStart(2, "0")}] ${text}`); + } + if (lines.length === 0) { + const text = String(payload?.text ?? "").trim(); + if (text) lines.push(text); + } + return lines.join("\n"); +} + +export function writeArtifacts(knowledgeDir, name, { rawBytes, transcript, provenance, source, fetchedAt }) { + const rawDir = join(knowledgeDir, "raw", COMMAND); + mkdirSync(rawDir, { recursive: true }); + const rawPath = join(rawDir, `${slug(name)}.json`); + writeAtomic(rawPath, Buffer.from(rawBytes)); + + const textDir = join(knowledgeDir, "text"); + mkdirSync(textDir, { recursive: true }); + const textPath = join(textDir, `${slug(name)}.md`); + const frontMatter = [ + "---", + "provenance:", + ` method: ${provenance.method}`, + ` producer: ${provenance.producer}`, + ` confidence: ${provenance.confidence}`, + `source: ${source}`, + `fetched_at: ${fetchedAt}`, + "anchors: pending # distilly parse-subtitle / harvest assign [k00NN] anchors", + "---", + "", + ].join("\n"); + const textBody = `${frontMatter}${transcript}\n`; + writeAtomic(textPath, Buffer.from(textBody, "utf8")); + + return { + raw: { path: rawPath, bytes: statSync(rawPath).size, sha256: sha256Hex(readFileSync(rawPath)) }, + text: { path: textPath, bytes: Buffer.byteLength(textBody), sha256: sha256Hex(Buffer.from(textBody, "utf8")) }, + }; +} + +function writeAtomic(path, buffer) { + mkdirSync(dirname(path), { recursive: true }); + const staging = `${path}.${process.pid}.tmp`; + try { + writeFileSync(staging, buffer); + renameSync(staging, path); + } catch (error) { + if (existsSync(staging)) rmSync(staging, { force: true }); + throw error; + } +} + +export function appendLedger(knowledgeDir, entries) { + if (entries.length === 0) return { path: join(knowledgeDir, "index.json"), added: 0, total: 0 }; + const path = join(knowledgeDir, "index.json"); + let existing = []; + if (existsSync(path)) { + try { + const parsed = JSON.parse(readFileSync(path, "utf8")); + existing = Array.isArray(parsed) ? parsed : []; + } catch (error) { + throw new TranscribeFailure("bad-ledger", `knowledge/index.json is not valid JSON: ${redact(error.message)}`, { + remediation: ["repair or remove knowledge/index.json, then rerun transcribe"], + }); + } + } + const byId = new Map(existing.filter((e) => e && typeof e === "object").map((e) => [e.id, e])); + let added = 0; + for (const entry of entries) { + if (!byId.has(entry.id)) added += 1; + byId.set(entry.id, entry); + } + const merged = [...byId.values()].sort((a, b) => String(a.id).localeCompare(String(b.id))); + writeAtomic(path, Buffer.from(`${JSON.stringify(merged, null, 2)}\n`, "utf8")); + return { path, added, total: merged.length }; +} + +/** + * @param {object} options + * @param {string} options.input audio/video path (required unless `--capture`) + * @param {string} [options.capture] host-provided transcript file + * @param {Function} [options.fetch] injected fetch + * @param {object} [options.env] + * @param {string} [options.root] + * @param {string} [options.person] + * @param {number} [options.maxRetries] + * @param {Function} [options.sleep] + * @param {string} [options.now] + */ +export async function transcribe(options = {}) { + const { + input, + capture, + fetch: fetchImpl = globalThis.fetch, + env = process.env, + root = process.cwd(), + person, + family = "colleague", + maxRetries = DEFAULT_MAX_RETRIES, + sleep = defaultSleep, + now = new Date().toISOString(), + language, + producer, + readFile = readFileSync, + onProgress = () => {}, + } = options; + + const knowledgeDir = knowledgeRoot({ root, person, family }); + const warnings = []; + const outputs = []; + let secrets = []; + + const base = { + command: COMMAND, + ok: false, + inputs: [], + outputs, + warnings, + unavailable: [], + credential_file: CONFIG_FILE, + }; + const fail = (failure) => ({ + ok: false, + exitCode: failure.exitCode ?? 1, + receipt: scrub( + { + ...base, + ok: false, + input: input ?? null, + errors: [redact(failure.message, secrets)], + unavailable: [ + { + channel: COMMAND, + reason: redact(`${failure.reason}: ${failure.message}`, secrets), + remediation: failure.remediation ?? [], + }, + ], + }, + secrets, + ), + }); + + try { + if (!input && !capture) { + throw new TranscribeFailure("missing-input", "transcribe needs an input file", { + remediation: [ + "pass an audio/video path: distilly transcribe interview.m4a --person lin-gong", + "or register a host transcript: distilly transcribe interview.m4a --capture transcript.txt", + ], + }); + } + + // ── host backend: an explicit capture file, no network at all ──────────── + if (capture) { + if (!existsSync(capture)) { + throw new TranscribeFailure("capture-missing", `--capture ${capture} does not exist`, { + remediation: ["let the host write the transcript first, then rerun with the same --capture path"], + }); + } + const transcript = String(readFile(capture, "utf8")); + if (transcript.trim() === "") { + throw new TranscribeFailure("capture-empty", `--capture ${capture} is empty; refusing to fabricate a transcript`, { + remediation: ["re-run the host transcription and pass the non-empty result"], + }); + } + const provenance = { + method: "host-transcribe", + producer: producer ?? "host:model", + confidence: "host-reported", + }; + const name = basename(input ?? capture).replace(/\.[^.]+$/, ""); + const artifacts = writeArtifacts(knowledgeDir, name, { + rawBytes: Buffer.from(JSON.stringify({ method: provenance.method, producer: provenance.producer, capture: basename(capture) }), "utf8"), + transcript, + provenance, + source: input ? basename(input) : basename(capture), + fetchedAt: now, + }); + outputs.push({ path: artifacts.raw.path, sha256: artifacts.raw.sha256, bytes: artifacts.raw.bytes, kind: "raw" }); + outputs.push({ path: artifacts.text.path, sha256: artifacts.text.sha256, bytes: artifacts.text.bytes, kind: "text" }); + const ledger = appendLedger(knowledgeDir, [ + { + id: `${COMMAND}:${slug(name)}:text`, + kind: "text", + origin: `text/${slug(name)}.md`, + source: COMMAND, + fetched_at: now, + bytes: artifacts.text.bytes, + sha256: artifacts.text.sha256, + credentialed: false, + method: provenance.method, + provenance, + warnings: ["anchors pending: parse-subtitle / harvest assign [k00NN]"], + }, + ]); + return { + ok: true, + exitCode: 0, + receipt: scrub( + { + ...base, + ok: true, + input: input ?? null, + outputs, + ledger: { path: ledger.path, added: ledger.added, total: ledger.total }, + provenance, + backend: "host", + unavailable: [], + }, + secrets, + ), + }; + } + + // ── http backend ───────────────────────────────────────────────────────── + const credential = loadCredential({ env, readFile }); + if (!credential.ok) { + throw new TranscribeFailure( + "no-backend", + credential.path + ? `no API key in ${CONFIG_FILE} and no host capture given` + : `no API key in the environment and no host capture given`, + { remediation: REMEDIATION_SETUP }, + ); + } + secrets = [credential.values.api_key]; + + if (!existsSync(input)) { + throw new TranscribeFailure("input-missing", `input file not found: ${input}`, { + remediation: ["check the path; nothing was written"], + }); + } + const bytes = readFile(input); + if (bytes.length === 0) { + throw new TranscribeFailure("input-empty", `input file is empty: ${input}`, { + remediation: ["nothing to transcribe; nothing was written"], + }); + } + if (bytes.length > MAX_UPLOAD_BYTES) { + throw new TranscribeFailure( + "input-too-large", + `input is ${(bytes.length / 1024 / 1024).toFixed(1)}MB, over the ${MAX_UPLOAD_BYTES / 1024 / 1024}MB upload limit`, + { + remediation: [ + "split the audio (ffmpeg -f segment) and transcribe the parts", + "or use the host backend: --capture ", + ], + }, + ); + } + + const model = options.model ?? credential.values.model; + const url = `${String(credential.values.base_url).replace(/\/+$/, "")}/audio/transcriptions`; + const { body, contentType } = buildMultipart({ + boundaryKey: sha256Hex(bytes).slice(0, 16), + fields: { model, response_format: "verbose_json", ...(language ? { language } : {}) }, + file: { field: "file", filename: basename(input), bytes, contentType: "application/octet-stream" }, + }); + + onProgress(`transcribing ${basename(input)} with ${model}`); + let attempts = 0; + for (;;) { + attempts += 1; + let response; + try { + response = await fetchImpl(url, { + method: "POST", + headers: { authorization: `Bearer ${credential.values.api_key}`, "content-type": contentType }, + body, + }); + } catch (error) { + if (attempts > maxRetries) { + throw new TranscribeFailure("network-error", `request failed: ${redact(error?.message ?? String(error), secrets)}`, { + remediation: ["check the network/proxy and retry", ...REMEDIATION_SETUP], + }); + } + await sleep(backoffDelay(attempts)); + continue; + } + + const status = Number(response?.status ?? 0); + const retryAfterMs = parseRetryAfter(response?.headers?.get?.("retry-after") ?? null); + if (status === 429 || status >= 500) { + if (attempts > maxRetries) { + throw new TranscribeFailure( + status === 429 ? "rate-limited" : "server-error", + `HTTP ${status} after ${maxRetries} retries; nothing was written`, + { remediation: ["retry later", ...REMEDIATION_SETUP] }, + ); + } + warnings.push(`retry ${attempts} after HTTP ${status} (waited ${backoffDelay(attempts, retryAfterMs)}ms)`); + await sleep(backoffDelay(attempts, retryAfterMs)); + continue; + } + + const text = await response.text(); + if (status === 401 || status === 403) { + throw new TranscribeFailure("unauthorized", `HTTP ${status}: the transcription API rejected the key`, { + remediation: [ + "regenerate the API key and update the env var / " + CONFIG_FILE, + "check that the key may call the audio transcriptions endpoint", + ], + }); + } + if (status === 413) { + throw new TranscribeFailure("payload-too-large", "HTTP 413: the provider rejected the upload size", { + remediation: ["split the audio into smaller parts", "or use the host backend with --capture"], + }); + } + if (status >= 400) { + throw new TranscribeFailure("http-error", `HTTP ${status}: ${redact(text.slice(0, 200), secrets)}`, { + remediation: ["check --model / --language and the provider status page", ...REMEDIATION_SETUP], + }); + } + + let payload; + try { + payload = JSON.parse(text); + } catch { + throw new TranscribeFailure("invalid-json", "the provider returned a non-JSON body; nothing was written", { + remediation: ["retry later; if it persists the endpoint may not be OpenAI-compatible"], + }); + } + + const transcript = formatTranscript(payload); + if (transcript.trim() === "") { + throw new TranscribeFailure( + "empty-transcript", + "the provider returned no text; refusing to write an empty transcript", + { remediation: ["check the audio has speech, or pass --language explicitly"] }, + ); + } + + const name = basename(input).replace(/\.[^.]+$/, ""); + const provenance = { + method: "openai-http", + producer: `${new URL(url).host} (${model})`, + confidence: "provider-reported", + }; + const artifacts = writeArtifacts(knowledgeDir, name, { + rawBytes: Buffer.from(text, "utf8"), + transcript, + provenance, + source: basename(input), + fetchedAt: now, + }); + outputs.push({ path: artifacts.raw.path, sha256: artifacts.raw.sha256, bytes: artifacts.raw.bytes, kind: "raw" }); + outputs.push({ path: artifacts.text.path, sha256: artifacts.text.sha256, bytes: artifacts.text.bytes, kind: "text" }); + const ledger = appendLedger(knowledgeDir, [ + { + id: `${COMMAND}:${slug(name)}:text`, + kind: "text", + origin: `text/${slug(name)}.md`, + source: COMMAND, + fetched_at: now, + bytes: artifacts.text.bytes, + sha256: artifacts.text.sha256, + credentialed: true, + credential_source: credential.source, + credential_file: CONFIG_FILE, + method: provenance.method, + provenance, + language: payload?.language ?? language ?? null, + warnings: ["anchors pending: parse-subtitle / harvest assign [k00NN]"], + }, + ]); + + return { + ok: true, + exitCode: 0, + receipt: scrub( + { + ...base, + ok: true, + input, + outputs, + ledger: { path: ledger.path, added: ledger.added, total: ledger.total }, + provenance, + backend: "openai-http", + attempts, + unavailable: [], + }, + secrets, + ), + }; + } + } catch (error) { + if (!(error instanceof TranscribeFailure)) { + throw new TranscribeFailure("unexpected", redact(error?.message ?? String(error), secrets), { + remediation: ["rerun with --json and report the receipt"], + }); + } + return fail(error); + } +} + +export const HELP = `distilly transcribe — 可选的转写后端 / optional transcription backend + +用法 (zh): + distilly transcribe [--person ] [--root ] [--language zh] + [--model whisper-1] [--max-retries 3] [--json] + distilly transcribe --capture [--producer <宿主标识>] [--json] + +后端：OpenAI 兼容 HTTP（DISTILLY_TRANSCRIBE_API_KEY 或 OPENAI_API_KEY， + DISTILLY_TRANSCRIBE_BASE_URL 覆盖网关，DISTILLY_TRANSCRIBE_MODEL 选择模型）， + 或宿主能力（--capture 显式交回转写文本）。 + 两者都没有 → 明确 unavailable + 非零退出，**绝不静默降级、绝不写空产物**。 +产物：knowledge/raw/transcribe/.json（原样响应）+ knowledge/text/.md（带 provenance 前言） + + knowledge/index.json 登记；回执与产物都带 provenance {method, producer, confidence}。 + 段落锚点 [k00NN] 由 parse-subtitle / harvest 后续补齐（产物里标注 anchors: pending）。 + +--- +## English + distilly transcribe [--person ] [--language zh] [--json] + distilly transcribe --capture [--json] + +Backends: an OpenAI-compatible HTTP endpoint (env key) or an explicit host capture. +With neither, the command fails loudly with \`unavailable\` — it never silently falls +back and never writes an empty transcript. Artifacts carry provenance +{method, producer, confidence}; anchors are assigned later by parse-subtitle / harvest. +`; + +export function parseFlags(argv) { + const flags = { _: [] }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (!arg.startsWith("--")) { + flags._.push(arg); + continue; + } + const name = arg.slice(2); + if (name === "json" || name === "help") { + flags[name] = true; + continue; + } + const value = argv[index + 1]; + if (value === undefined || value.startsWith("--")) throw new Error(`${arg} requires a value`); + flags[name] = value; + index += 1; + } + return flags; +} + +export async function runTranscribeCli(argv, io = {}) { + const out = io.stdout ?? ((line) => process.stdout.write(`${line}\n`)); + const err = io.stderr ?? ((line) => process.stderr.write(`${line}\n`)); + + let flags; + try { + flags = parseFlags(argv); + } catch (error) { + err(`Error: ${error.message}`); + return 1; + } + if (flags.help || flags._.length === 0) { + out(HELP); + return flags.help ? 0 : 1; + } + + const result = await transcribe({ + input: flags._[0], + capture: flags.capture, + fetch: io.fetch ?? globalThis.fetch, + env: io.env ?? process.env, + root: flags.root ?? process.cwd(), + person: flags.person, + family: flags.family, + language: flags.language, + model: flags.model, + producer: flags.producer, + maxRetries: flags["max-retries"] ? Number(flags["max-retries"]) : undefined, + sleep: io.sleep, + now: io.now, + }); + + if (flags.json) out(JSON.stringify(result.receipt, null, 2)); + if (result.ok) { + out(`transcribed via ${result.receipt.backend} → ${result.receipt.outputs.map((o) => o.path).join(", ")}`); + out(`provenance: ${result.receipt.provenance.method} / ${result.receipt.provenance.producer} / ${result.receipt.provenance.confidence}`); + } else { + err(`Error: ${result.receipt.errors?.[0] ?? "transcribe failed"}`); + for (const entry of result.receipt.unavailable) { + err(`unavailable: ${entry.channel} — ${entry.reason}`); + for (const step of entry.remediation ?? []) err(` fix: ${step}`); + } + } + for (const warning of result.receipt.warnings ?? []) err(`warning: ${warning}`); + return result.exitCode; +} diff --git a/src/parse/archive.mjs b/src/parse/archive.mjs new file mode 100644 index 00000000..9aaeb654 --- /dev/null +++ b/src/parse/archive.mjs @@ -0,0 +1,1228 @@ +/** + * archive.mjs — a zip or directory of exports → the ledger, member by member. + * + * An archive is never "one document". An X archive holds tweets plus DMs, a + * Takeout holds mail plus chats plus a location history, a Discord export holds + * one file per channel. Each of those is a different kind of evidence with a + * different `kind`, so `parseArchive` returns a **list of documents**, one per + * sub-source, and `parseArchiveFile` returns the single document a container + * that is really just one file produces (a `.docx` inside a zip, an `.mbox` + * inside a Takeout). + * + * Identification is structural. A container whose layout matches nothing known + * raises `UnrecognizedFormatError` naming what was found: "unknown archive" is a + * result, and inventing a reading for an unidentified dump would put unexplained + * text into a person's knowledge base. + * + * Supported layouts: + * + * - **X / Twitter** — `data/*.js` (`window.YTD.…` wrappers) and `data/*.json`: + * `tweets`, `note-tweet`, `like`, `direct-messages`, `account`, + * `follower`/`following`. + * - **Google Takeout** — `Takeout/**`: `.mbox` mail, Google Chat + * `*.json`, `*.csv`, and the subtitle/`.txt` attachments Takeout ships. + * - **Discord** — `messages/*.json` + `channels.json` + `users.json` + * (`account.json` marks the export owner). + * - **Telegram** — `result.json` (or `chats/*/messages*.json`). + * - **Instagram / Facebook** — `messages/inbox/*/message_*.json`, + * `content/posts_1.json`, `your_instagram_activity/**`. + * - **LinkedIn** — `Connections.csv`, `Messages.csv`, `Invitations.csv`. + */ + +import { basename, extname } from "node:path"; + +import { + DEFAULT_MAX_MEMBER_BYTES, + InputError, + SourceFile, + UnrecognizedFormatError, + buildDocument, + findObjectArray, + iterateObjects, + parseJsonPayload, + pick, + readZipMembers, + recordsFromCharSpans, + unwrapJsonAssignment, + walkJsonLeaves, +} from "./common.mjs"; +import { parseChat } from "./chat.mjs"; +import { parseEmail } from "./email.mjs"; +import { parseSubtitle } from "./subtitle.mjs"; +import { parseFeishu } from "./feishu.mjs"; + +/* ------------------------------------------------------------------ */ +/* container sources */ +/* ------------------------------------------------------------------ */ + +const ZIP_EXTENSIONS = new Set([".zip", ".docx", ".xlsx", ".epub", ".jar"]); + +/** + * A container member: a `SourceFile` plus where it came from. + * + * `archivePath` is the member's name inside the container; `path` is a + * human-readable origin (`.zip!/data/tweets.js`) that ends up in the + * ledger so a citation can be traced to the exact member. + */ +export class ArchiveMember extends SourceFile { + constructor(input) { + super(input); + this.archivePath = input.archivePath ?? this.name; + this.archive = input.archive ?? null; + this.byteOffsetInArchive = input.byteOffsetInArchive ?? null; + this.byteLengthInArchive = input.byteLengthInArchive ?? null; + this.origin = input.origin ?? `${this.archive ?? ""}!/${this.archivePath}`; + } + + descriptor(extra = {}) { + return { + ...super.descriptor(extra), + // The origin recorded in the ledger names both the container and the member. + path: this.origin, + archivePath: this.archivePath, + archive: this.archive, + }; + } +} + +/** True when a path looks like a zip container we should open. */ +export function isZipPath(path) { + return ZIP_EXTENSIONS.has(extname(String(path)).toLowerCase()); +} + +/** + * Read the members of a zip payload into `ArchiveMember`s. + * Members that cannot be read become warnings, never exceptions. + * + * @param {Uint8Array} bytes + * @param {{archiveName?: string, maxMemberBytes?: number, filter?: Function}} [options] + */ +export function readArchiveMembers(bytes, options = {}) { + const archiveName = options.archiveName ?? ""; + const maxMemberBytes = options.maxMemberBytes ?? DEFAULT_MAX_MEMBER_BYTES; + const warnings = []; + let directory; + + // `readZipMembers` already swallows per-member failures; call the pair + // directly so a bad *container* still fails loudly. + const result = readZipMembers(bytes, { maxMemberBytes, filter: options.filter }); + directory = result.directory; + warnings.push(...result.warnings); + + const members = []; + for (const [name, data] of result.members) { + const entry = directory.entries.find((candidate) => candidate.name === name); + members.push( + new ArchiveMember({ + path: `${archiveName}!/${name}`, + name: basename(name), + archivePath: name, + archive: archiveName, + raw: data, + byteOffsetInArchive: entry?.localOffset ?? null, + byteLengthInArchive: entry?.compressedSize ?? null, + origin: `${archiveName}!/${name}`, + }), + ); + } + return { members, entries: directory.entries, warnings, directory }; +} + +/** + * Build a directory archive from an already-loaded file list. + * + * The caller (`harvest`) owns walking the filesystem, because it also needs the + * per-file skip report for files that are not part of an archive at all. + * + * @param {Array<{path: string, relativePath: string, bytes: Uint8Array}>} files + * @param {{archiveName?: string}} [options] + */ +export function directoryArchive(files, options = {}) { + const archiveName = options.archiveName ?? ""; + return files.map( + (file) => + new ArchiveMember({ + path: file.relativePath, + name: basename(file.relativePath), + archivePath: file.relativePath, + archive: archiveName, + raw: file.bytes, + origin: `${archiveName}/${file.relativePath}`, + }), + ); +} + +/* ------------------------------------------------------------------ */ +/* parsing windowed `.js` payloads */ +/* ------------------------------------------------------------------ */ + +const YTD_PREFIX = /^window\.YTD\.([A-Za-z0-9_]+)\.part\d+\s*=/; + +/** + * Read one `window.YTD..partN = […]` member. + * + * The wrapper is stripped with the member's own text so that every byte offset + * that comes back refers to the raw member, not to the JSON slice. + * + * @returns {{records: Array, name: string|null, warnings: string[]}} + */ +export function readYtdMember(member, options = {}) { + const warnings = []; + const { json, wrapper } = unwrapJsonAssignment(member.text); + const nameMatch = wrapper ? YTD_PREFIX.exec(`${wrapper} =`) : null; + const dataset = options.dataset ?? nameMatch?.[1] ?? null; + + let value; + try { + value = JSON.parse(json.trim().replace(/;\s*$/, "")); + } catch (error) { + throw new UnrecognizedFormatError( + `${member.path} is not readable as a window.YTD payload: ${error.message}`, + { path: member.path }, + ); + } + if (!Array.isArray(value)) { + throw new UnrecognizedFormatError(`${member.path} does not hold an array after the window.YTD wrapper`, { path: member.path }); + } + + const { leaves } = walkJsonLeaves(member, options.leafOptions ?? {}); + // Leaves are in document order, so they line up with `value`'s elements. + const records = []; + const entryCount = value.length; + + // Group leaves by their top-level array index to recover each element's bytes. + const byIndex = new Map(); + for (const leaf of leaves) { + const [head, ...rest] = leaf.path.split("."); + const index = Number(head); + if (!Number.isInteger(index)) continue; + if (!byIndex.has(index)) byIndex.set(index, []); + byIndex.get(index).push({ ...leaf, path: rest.join(".") }); + } + + for (let index = 0; index < entryCount; index += 1) { + const leavesForEntry = byIndex.get(index) ?? []; + if (leavesForEntry.length === 0) { + warnings.push(`${member.path}: element ${index} produced no readable scalar and was skipped`); + continue; + } + const byteStart = Math.min(...leavesForEntry.map((leaf) => leaf.byteStart)); + const byteEnd = Math.max(...leavesForEntry.map((leaf) => leaf.byteEnd)); + records.push({ + index, + value: value[index], + leaves: leavesForEntry, + byteStart, + byteEnd, + }); + } + + return { records, name: dataset, warnings, value, wrapper }; +} + +/* ------------------------------------------------------------------ */ +/* archive type detection */ +/* ------------------------------------------------------------------ */ + +function normalisedName(name) { + return name.replace(/\\/g, "/").replace(/^\.\//, ""); +} + +function countMatching(names, pattern) { + return names.filter((name) => pattern.test(name)).length; +} + +/** + * Decide which archive this is, from its member list alone. + * + * @param {ArchiveMember[]} members + * @returns {{type: string, reasons: string[], confidence: "structural"}} + */ +export function detectArchiveType(members) { + const names = members.map((member) => normalisedName(member.archivePath)); + const reasons = []; + const has = (pattern) => names.some((name) => pattern.test(name)); + + // ---- X / Twitter --------------------------------------------------- + if (has(/(^|\/)data\/tweets\.js$/i) || has(/(^|\/)data\/account\.js$/i) || has(/(^|\/)data\/direct-messages\.js$/i)) { + reasons.push("data/*.js members with X/Twitter's window.YTD naming"); + return { type: "x-archive", reasons, confidence: "structural" }; + } + if (has(/(^|\/)data\/tweets\.json$/i) && has(/(^|\/)data\/account\.json$/i)) { + reasons.push("data/tweets.json + data/account.json (an X archive with JSON data files)"); + return { type: "x-archive", reasons, confidence: "structural" }; + } + + // ---- Discord ------------------------------------------------------- + if (countMatching(names, /(^|\/)messages\/(?:[^/]+\/)?messages\.json$/i) > 0 || (has(/(^|\/)messages\/[^/]+\.json$/i) && has(/(^|\/)channels\.json$/i))) { + reasons.push("messages/*.json channel dumps plus channels.json (Discord export)"); + return { type: "discord-export", reasons, confidence: "structural" }; + } + + // ---- Telegram ------------------------------------------------------ + if (has(/(^|\/)result\.json$/i) || countMatching(names, /(^|\/)chats\/[^/]+\/messages\d*\.json$/i) > 0) { + reasons.push("result.json or chats/*/messages*.json (Telegram export)"); + return { type: "telegram-export", reasons, confidence: "structural" }; + } + + // ---- Instagram / Facebook ------------------------------------------ + if (countMatching(names, /(^|\/)messages\/(inbox|message_requests)\//i) > 0 || has(/(^|\/)content\/posts_1\.json$/i)) { + reasons.push("messages/inbox/*/message_*.json or content/posts_1.json (Instagram/Facebook export)"); + return { type: "instagram-export", reasons, confidence: "structural" }; + } + if (has(/(^|\/)your_instagram_activity\//i) || has(/(^|\/)your_facebook_activity\//i)) { + reasons.push("your_instagram_activity/ or your_facebook_activity/ (a Meta export)"); + return { type: "instagram-export", reasons, confidence: "structural" }; + } + + // ---- LinkedIn ------------------------------------------------------ + if (has(/(^|\/)Connections\.csv$/i) || has(/(^|\/)Messages\.csv$/i) || has(/(^|\/)Invitations\.csv$/i)) { + reasons.push("Connections.csv / Messages.csv / Invitations.csv (a LinkedIn data export)"); + return { type: "linkedin-export", reasons, confidence: "structural" }; + } + + // ---- Google Takeout ------------------------------------------------ + if (has(/(^|\/)Takeout\//i) || has(/(^|\/)takeout-[^/]+\//i)) { + reasons.push("members under Takeout/ (a Google Takeout archive)"); + return { type: "takeout", reasons, confidence: "structural" }; + } + + // ---- Am I inside a Takeout, or is this an unlabelled dump? --------- + if (has(/\.mbox$/i) && (has(/\.html?$/i) || has(/\.csv$/i))) { + reasons.push(".mbox plus .html/.csv members with no Takeout/ prefix"); + return { type: "takeout", reasons, confidence: "structural" }; + } + + throw new UnrecognizedFormatError( + `unknown archive: ${members.length} member(s) match no known export layout. Looked for: X (data/*.js), Discord (messages/*.json + channels.json), Telegram (result.json), Instagram/Facebook (messages/inbox/, content/posts_1.json), LinkedIn (Connections.csv), Google Takeout (Takeout/). Saw: ${names.slice(0, 8).join(", ")}${names.length > 8 ? ", …" : ""}`, + { members: names.slice(0, 32) }, + ); +} + +/* ------------------------------------------------------------------ */ +/* generic record helpers */ +/* ------------------------------------------------------------------ */ + +/** + * Records from an array of objects, one per element, using the leaves the JSON + * walker located. The element's byte range is the envelope of its own scalars. + */ +function objectRecords(member, value, options = {}) { + const { leaves } = walkJsonLeaves(member, options.leafOptions ?? {}); + const byIndex = new Map(); + for (const leaf of leaves) { + const [head, ...rest] = leaf.path.split("."); + const index = Number(head); + if (!Number.isInteger(index)) continue; + if (!byIndex.has(index)) byIndex.set(index, []); + byIndex.get(index).push({ ...leaf, path: rest.join(".") }); + } + + const records = []; + const warnings = []; + const list = Array.isArray(value) ? value : []; + for (let index = 0; index < list.length; index += 1) { + const located = byIndex.get(index) ?? []; + const rendered = options.render(list[index], index, located); + if (!rendered) continue; + const text = typeof rendered === "string" ? rendered : rendered.text; + if (!text || text.trim() === "") continue; + const envelope = located.length > 0 + ? { byteStart: Math.min(...located.map((leaf) => leaf.byteStart)), byteEnd: Math.max(...located.map((leaf) => leaf.byteEnd)) } + : { byteStart: null, byteEnd: null }; + records.push({ + kind: typeof rendered === "object" ? rendered.kind ?? "item" : "item", + text, + label: typeof rendered === "object" ? rendered.label ?? null : null, + file: member.name, + byteStart: envelope.byteStart, + byteEnd: envelope.byteEnd, + synthetic: false, + }); + } + return { records, warnings }; +} + +/** Render `{path = value}` lines for every scalar the walker found. */ +function renderLeaves(leaves, options = {}) { + const maxValue = options.maxValueLength ?? 2000; + const parts = []; + for (const leaf of leaves) { + const value = leaf.value === null ? "" : String(leaf.value); + if (value === "") continue; + parts.push(`${leaf.path || "value"}: ${value.length > maxValue ? `${value.slice(0, maxValue)}…` : value}`); + } + return parts.join("\n"); +} + +/* ------------------------------------------------------------------ */ +/* per-archive extractors */ +/* ------------------------------------------------------------------ */ + +function memberByName(members, pattern) { + return members.find((member) => pattern.test(normalisedName(member.archivePath))) ?? null; +} + +function membersByName(members, pattern) { + return members.filter((member) => pattern.test(normalisedName(member.archivePath))); +} + +/** X / Twitter archive. */ +export function extractXArchive(members, options = {}) { + const documents = []; + const warnings = []; + const account = memberByName(members, /(^|\/)data\/account\.js$|(^|\/)data\/account\.json$/i); + let handle = null; + if (account) { + try { + const parsed = parseJsonPayload(account); + const first = Array.isArray(parsed.value) ? parsed.value[0] : parsed.value; + handle = first?.account?.username ?? first?.username ?? null; + } catch (error) { + warnings.push(`${account.path}: the account member could not be read (${error.message}); the archive owner is unknown`); + } + } + + const datasets = [ + { pattern: /(^|\/)data\/tweets\.js$|(^|\/)data\/tweets\.json$/i, kind: "chat", label: "tweets", as: "tweets" }, + { pattern: /(^|\/)data\/note-tweet\.js$/i, kind: "chat", label: "long-form notes", as: "note-tweets" }, + { pattern: /(^|\/)data\/like\.js$/i, kind: "archive-record", label: "likes", as: "likes" }, + { pattern: /(^|\/)data\/direct-messages\.js$/i, kind: "chat-thread", label: "direct messages", as: "direct-messages" }, + { pattern: /(^|\/)data\/follower\.js$/i, kind: "archive-record", label: "followers", as: "followers" }, + { pattern: /(^|\/)data\/following\.js$/i, kind: "archive-record", label: "following", as: "following" }, + ]; + + for (const dataset of datasets) { + for (const member of membersByName(members, dataset.pattern)) { + let parsed; + try { + parsed = readYtdMember(member, { dataset: dataset.as }); + } catch (error) { + warnings.push(`${member.path}: ${error.message}`); + continue; + } + warnings.push(...parsed.warnings); + + const isTweets = dataset.as === "tweets" || dataset.as === "note-tweets"; + const isDm = dataset.as === "direct-messages"; + + const records = parsed.records + .map((record) => { + const entry = record.value ?? {}; + if (isTweets) { + const tweet = entry.tweet ?? entry; + const text = tweet.full_text ?? tweet.text ?? null; + if (typeof text !== "string" || text.trim() === "") return null; + return { + kind: "item", + text, + label: `tweet ${tweet.id_str ?? tweet.id ?? record.index} @ ${tweet.created_at ?? "unknown time"}`, + createdAt: tweet.created_at ?? null, + byteStart: record.byteStart, + byteEnd: record.byteEnd, + }; + } + if (isDm) { + const message = entry.dmConversation?.messages ?? entry.messages ?? [entry]; + void message; + return null; + } + return { + kind: "item", + text: renderLeaves(record.leaves), + label: `${dataset.label} ${record.index}`, + byteStart: record.byteStart, + byteEnd: record.byteEnd, + }; + }) + .filter(Boolean); + + if (isDm) { + // Direct messages have their own nested shape; each conversation carries + // a messages array, and each message a `messageCreate`. + const dmRecords = []; + for (const record of parsed.records) { + const conversation = record.value?.dmConversation ?? {}; + const messages = Array.isArray(conversation.messages) ? conversation.messages : []; + const conversationId = conversation.conversationId ?? `conversation-${record.index}`; + for (const wrapper of messages) { + const message = wrapper?.messageCreate ?? wrapper; + const text = typeof message?.text === "string" ? message.text : ""; + if (text.trim() === "") continue; + const leaves = record.leaves.filter((leaf) => leaf.path.includes(String(messages.indexOf(wrapper)))); + dmRecords.push({ + kind: "turn", + text, + label: `${message?.senderId ?? "unknown"} @ ${message?.createdAt ?? "unknown time"} · ${conversationId}`, + byteStart: leaves.length > 0 ? Math.min(...leaves.map((leaf) => leaf.byteStart)) : null, + byteEnd: leaves.length > 0 ? Math.max(...leaves.map((leaf) => leaf.byteEnd)) : null, + }); + } + } + if (dmRecords.length === 0) { + warnings.push(`${member.path}: no direct message carried text`); + continue; + } + documents.push( + buildDocument({ + parser: "archive", + format: "x-archive", + kind: "chat-thread", + method: "archive-member", + source: "x-archive", + origin: member.origin, + files: [member], + entries: dmRecords, + content: dmRecords.map((record) => record.text).join("\n\n"), + segments: segmentRanges(dmRecords), + warnings, + meta: { archiveType: "x-archive", dataset: "direct-messages", owner: handle, messages: dmRecords.length }, + accounting: { model: "raw-bytes" }, + }), + ); + warnings.length = 0; + continue; + } + + if (records.length === 0) { + warnings.push(`${member.path}: the ${dataset.label} dataset held no readable text`); + continue; + } + documents.push( + buildDocument({ + parser: "archive", + format: "x-archive", + kind: dataset.kind, + method: "archive-member", + source: "x-archive", + origin: member.origin, + files: [member], + content: records.map((record) => record.text).join("\n\n"), + segments: segmentRanges(records), + entries: records.map((record) => ({ + kind: record.kind, + text: record.text, + label: record.label, + byteStart: record.byteStart, + byteEnd: record.byteEnd, + file: member.name, + })), + warnings: [...warnings], + meta: { + archiveType: "x-archive", + dataset: dataset.as, + owner: handle, + entries: records.length, + firstTimestamp: records.find((record) => record.createdAt)?.createdAt ?? null, + lastTimestamp: [...records].reverse().find((record) => record.createdAt)?.createdAt ?? null, + }, + accounting: { model: "raw-bytes" }, + }), + ); + warnings.length = 0; + } + } + + if (documents.length === 0) { + throw new UnrecognizedFormatError( + "this looks like an X archive (data/*.js) but no dataset in it carried text", + { members: members.map((member) => member.archivePath) }, + ); + } + return { documents, warnings, owner: handle }; +} + +/** Turn per-record text into the character segments `buildDocument` expects. */ +function segmentRanges(records) { + const segments = []; + let cursor = 0; + for (const record of records) { + segments.push({ charStart: cursor, charEnd: cursor + record.text.length, byteStart: record.byteStart, byteEnd: record.byteEnd }); + cursor += record.text.length + 2; // "\n\n" + } + return segments; +} + +/** Discord export. */ +export function extractDiscordExport(members, options = {}) { + const documents = []; + const warnings = []; + const channelsMember = memberByName(members, /(^|\/)channels\.json$/i); + const usersMember = memberByName(members, /(^|\/)users\.json$/i); + const messagesMembers = membersByName(members, /(^|\/)messages\/(?:[^/]+\/)?messages\.json$/i); + + const channelNames = new Map(); + if (channelsMember) { + try { + const parsed = JSON.parse(channelsMember.text.trim()); + const list = Array.isArray(parsed) ? parsed : Array.isArray(parsed?.channels) ? parsed.channels : []; + for (const channel of list) { + if (channel?.id) channelNames.set(String(channel.id), channel.name ?? channel.id); + } + } catch (error) { + warnings.push(`${channelsMember.path}: channels.json could not be parsed (${error.message}); channels keep their ids`); + } + } else { + warnings.push("no channels.json member: Discord channels keep their numeric ids"); + } + + if (usersMember) { + try { + const parsed = JSON.parse(usersMember.text.trim()); + const list = Array.isArray(parsed) ? parsed : []; + if (list.length > 0) warnings.push(`users.json lists ${list.length} account(s); message authors are taken from the messages themselves`); + } catch (error) { + warnings.push(`${usersMember.path}: users.json could not be parsed (${error.message})`); + } + } + + for (const member of messagesMembers) { + const channelId = member.archivePath.replace(/\\/g, "/").split("/").slice(-2)[0]; + const channelName = channelNames.get(channelId) ?? channelId; + try { + const document = parseChat(member, { format: "discord-messages", channelName }); + documents.push({ + ...document, + parser: "archive", + method: "archive-member", + source: "discord-export", + origin: member.origin, + meta: { ...document.meta, archiveType: "discord-export", channelId, channelName }, + }); + } catch (error) { + warnings.push(`${member.path}: ${error.message}`); + } + } + + if (documents.length === 0) { + throw new UnrecognizedFormatError( + "this looks like a Discord export (messages/*/messages.json) but no channel dump could be read", + { members: members.map((member) => member.archivePath) }, + ); + } + return { documents, warnings }; +} + +/** Telegram export. */ +export function extractTelegramExport(members) { + const documents = []; + const warnings = []; + const results = membersByName(members, /(^|\/)result\.json$/i); + const chunks = membersByName(members, /(^|\/)chats\/[^/]+\/messages\d*\.json$/i); + + for (const member of results) { + try { + const document = parseChat(member, { format: "telegram-export" }); + documents.push({ + ...document, + parser: "archive", + method: "archive-member", + source: "telegram-export", + origin: member.origin, + meta: { ...document.meta, archiveType: "telegram-export" }, + }); + } catch (error) { + warnings.push(`${member.path}: ${error.message}`); + } + } + // `chats/*/messages*.json` chunks are the "export in parts" mode: they hold a + // bare array, which the Telegram detector does not accept as a whole chat. + for (const member of chunks) { + warnings.push(`${member.path}: this Telegram export was split into chats/*/messages*.json chunks, which is not supported yet; the chat was skipped rather than joined out of order`); + } + + if (documents.length === 0) { + throw new UnrecognizedFormatError( + "this looks like a Telegram export but result.json could not be read as a chat", + { members: members.map((member) => member.archivePath) }, + ); + } + return { documents, warnings }; +} + +/** Instagram / Facebook export. */ +export function extractInstagramExport(members) { + const documents = []; + const warnings = []; + const messageMembers = membersByName(members, /(^|\/)messages\/(inbox|message_requests|filtered_threads|archived_threads)\/[^/]+\/message_\d+\.json$/i); + const otherMessages = membersByName(members, /(^|\/)messages\/[^/]+\/message_\d+\.json$/i).filter((member) => !messageMembers.includes(member)); + + for (const member of [...messageMembers, ...otherMessages]) { + try { + const document = parseChat(member, { format: "instagram-messages" }); + documents.push({ + ...document, + parser: "archive", + method: "archive-member", + source: "instagram-export", + origin: member.origin, + meta: { ...document.meta, archiveType: "instagram-export", thread: member.archivePath.split("/").slice(-2)[0] }, + }); + } catch (error) { + warnings.push(`${member.path}: ${error.message}`); + } + } + + // Posts, comments and other activity JSON: recorded as archive records, not as + // dialogue, because they are published statements, not conversation. + const activityMembers = membersByName( + members, + /(^|\/)content\/(posts|comments|story_activities|reels)_.*\.json$|(^|\/)your_(instagram|facebook)_activity\/.*\.json$/i, + ); + for (const member of activityMembers) { + let parsed; + try { + parsed = parseJsonPayload(member); + } catch (error) { + warnings.push(`${member.path}: ${error.message}`); + continue; + } + const array = Array.isArray(parsed.value) ? parsed.value : null; + if (!array) { + warnings.push(`${member.path}: the activity member holds an object rather than an array and was skipped`); + continue; + } + const { records } = objectRecords(member, array, { + render: (entry, index, leaves) => { + const rendered = renderLeaves(leaves); + return rendered === "" ? null : { text: rendered, label: `${basename(member.archivePath)} #${index}` }; + }, + }); + if (records.length === 0) { + warnings.push(`${member.path}: the activity member held no readable text`); + continue; + } + documents.push( + buildDocument({ + parser: "archive", + format: "instagram-export", + kind: "archive-record", + method: "archive-member", + source: "instagram-export", + origin: member.origin, + files: [member], + content: records.map((record) => record.text).join("\n\n"), + segments: segmentRanges(records), + entries: records.map((record) => ({ ...record, file: member.name })), + warnings: [], + meta: { archiveType: "instagram-export", dataset: basename(member.archivePath), entries: records.length }, + }), + ); + } + + if (documents.length === 0) { + throw new UnrecognizedFormatError( + "this looks like an Instagram/Facebook export but no message thread or activity file could be read", + { members: members.map((member) => member.archivePath) }, + ); + } + return { documents, warnings }; +} + +/** LinkedIn CSV export. */ +export function extractLinkedInExport(members) { + const documents = []; + const warnings = []; + const known = [ + { pattern: /(^|\/)Connections\.csv$/i, dataset: "connections", kind: "archive-record" }, + { pattern: /(^|\/)Messages\.csv$/i, dataset: "messages", kind: "chat" }, + { pattern: /(^|\/)Invitations\.csv$/i, dataset: "invitations", kind: "archive-record" }, + ]; + + for (const { pattern, dataset, kind } of known) { + for (const member of membersByName(members, pattern)) { + const { rows, header, preamble, warnings: csvWarnings } = parseCsv(member.text); + warnings.push(...csvWarnings); + if (header.length === 0) { + warnings.push(`${member.path}: no header row was found and the file was skipped`); + continue; + } + const records = rows + .map((row) => { + const parts = header + .map((column, index) => { + const value = row.cells[index] ?? ""; + return value === "" ? null : `${column}: ${value}`; + }) + .filter(Boolean); + if (parts.length === 0) return null; + return { + kind: "item", + text: parts.join("\n"), + label: `${dataset} row ${row.index + 1}`, + byteStart: row.byteStart, + byteEnd: row.byteEnd, + }; + }) + .filter(Boolean); + + if (records.length === 0) { + warnings.push(`${member.path}: every row was empty`); + continue; + } + documents.push( + buildDocument({ + parser: "archive", + format: "linkedin-export", + kind, + method: "archive-member", + source: "linkedin-export", + origin: member.origin, + files: [member], + content: records.map((record) => record.text).join("\n\n"), + segments: segmentRanges(records), + entries: records.map((record) => ({ ...record, file: member.name })), + warnings: [], + meta: { + archiveType: "linkedin-export", + dataset, + columns: header, + preambleLines: preamble, + rows: records.length, + }, + }), + ); + } + } + + if (documents.length === 0) { + throw new UnrecognizedFormatError( + "this looks like a LinkedIn export (Connections.csv / Messages.csv) but no CSV could be read", + { members: members.map((member) => member.archivePath) }, + ); + } + return { documents, warnings }; +} + +/** + * CSV reader for the LinkedIn exports. + * + * LinkedIn prefixes `Connections.csv` with "Notes:" lines before the real header, + * so the header is the first row with more than one non-empty field. Quoted + * fields, embedded commas, embedded newlines and doubled quotes are handled. + * + * @returns {{rows: Array<{index: number, cells: string[], byteStart: number, + * byteEnd: number}>, header: string[], preamble: number, + * warnings: string[]}} + */ +export function parseCsv(text, options = {}) { + const warnings = []; + const rows = []; + let index = 0; + let row = 0; + let cells = []; + let field = ""; + let inQuotes = false; + let rowStart = 0; + let sawQuote = false; + + const pushField = () => { + cells.push(field); + field = ""; + sawQuote = false; + }; + const pushRow = (end) => { + pushField(); + rows.push({ index: row, cells, byteStart: rowStart, byteEnd: end }); + row += 1; + cells = []; + rowStart = end + 1; + }; + + while (index < text.length) { + const char = text[index]; + if (inQuotes) { + if (char === '"') { + if (text[index + 1] === '"') { + field += '"'; + index += 2; + continue; + } + inQuotes = false; + index += 1; + continue; + } + field += char; + index += 1; + continue; + } + if (char === '"' && field === "") { + inQuotes = true; + sawQuote = true; + index += 1; + continue; + } + if (char === ",") { + pushField(); + index += 1; + continue; + } + if (char === "\r") { + index += 1; + continue; + } + if (char === "\n") { + pushRow(index); + index += 1; + continue; + } + field += char; + index += 1; + } + if (field !== "" || cells.length > 0) pushRow(text.length); + if (inQuotes) warnings.push("the CSV ended inside a quoted field; the field was closed at end of file"); + + const headerIndex = rows.findIndex((candidate) => candidate.cells.filter((cell) => cell.trim() !== "").length > 1); + if (headerIndex === -1) { + return { rows: [], header: [], preamble: rows.length, warnings: [...warnings, "no row had more than one populated field, so no header could be identified"] }; + } + const header = rows[headerIndex].cells.map((cell) => cell.trim()).filter((cell) => cell !== ""); + const dataRows = rows.slice(headerIndex + 1).filter((candidate) => candidate.cells.some((cell) => cell.trim() !== "")); + if (headerIndex > 0) warnings.push(`${headerIndex} preamble line(s) before the header were skipped`); + if (dataRows.length === 0) warnings.push("the CSV has a header but no data rows"); + + return { rows: dataRows, header, preamble: headerIndex, warnings }; +} + +/** Google Takeout. */ +export function extractTakeout(members, options = {}) { + const documents = []; + const warnings = []; + const handled = new Set(); + + const mboxMembers = membersByName(members, /\.mbox$/i); + for (const member of mboxMembers) { + try { + const document = parseEmail(member, { format: "mbox" }); + documents.push({ + ...document, + parser: "archive", + method: "archive-member", + source: "takeout", + origin: member.origin, + meta: { ...document.meta, archiveType: "takeout", dataset: "mail" }, + }); + handled.add(member.archivePath); + } catch (error) { + warnings.push(`${member.path}: ${error.message}`); + } + } + + const subtitleMembers = membersByName(members, /\.(srt|vtt)$/i); + for (const member of subtitleMembers) { + try { + const document = parseSubtitle(member); + documents.push({ + ...document, + parser: "archive", + method: "archive-member", + source: "takeout", + origin: member.origin, + meta: { ...document.meta, archiveType: "takeout", dataset: "captions" }, + }); + handled.add(member.archivePath); + } catch (error) { + warnings.push(`${member.path}: ${error.message}`); + } + } + + // Google Chat takeout: `Takeout/Chat/Groups//group_info.json` plus + // `messages.json`. The schema is not stable across exports, so the messages + // are read as records rather than claimed as a known chat format. + const chatMembers = membersByName(members, /(^|\/)Chat\/.*messages\.json$/i); + for (const member of chatMembers) { + let parsed; + try { + parsed = parseJsonPayload(member); + } catch (error) { + warnings.push(`${member.path}: ${error.message}`); + continue; + } + const array = Array.isArray(parsed.value) ? parsed.value : Array.isArray(parsed.value?.messages) ? parsed.value.messages : null; + if (!array) { + warnings.push(`${member.path}: Google Chat message file has no messages array and was skipped`); + continue; + } + const { records } = objectRecords(member, array, { + render: (entry, index, leaves) => { + const creator = entry?.creator?.name ?? entry?.sender_name ?? entry?.sender ?? "unknown"; + const created = entry?.created_date ?? entry?.create_time ?? entry?.timestamp ?? "unknown time"; + const text = entry?.text ?? entry?.message ?? renderLeaves(leaves.filter((leaf) => !/creator|created_date/.test(leaf.path))); + if (typeof text !== "string" || text.trim() === "") return null; + return { text, label: `${creator} @ ${created}` }; + }, + }); + if (records.length === 0) { + warnings.push(`${member.path}: Google Chat messages carried no text`); + continue; + } + documents.push( + buildDocument({ + parser: "archive", + // The Google Chat schema is not verified against a real export, so the + // format is recorded as the generic takeout one rather than claimed. + format: "takeout-chat", + kind: "chat-thread", + method: "archive-member", + source: "takeout", + origin: member.origin, + files: [member], + content: records.map((record) => record.text).join("\n\n"), + segments: segmentRanges(records), + entries: records.map((record) => ({ ...record, file: member.name })), + warnings: ["Google Chat records are read generically; the export schema is not verified against a real archive (TODO)"], + meta: { archiveType: "takeout", dataset: "chat", entries: records.length }, + }), + ); + handled.add(member.archivePath); + } + + // Remaining CSV/JSON/HTML members: recorded as archive records so nothing in + // the export is silently ignored, with the member name in the label. + const leftovers = members.filter((member) => { + if (handled.has(member.archivePath)) return false; + if (/(^|\/)metadata\.json$|(^|\/)archive_browser\.html$|(^|\/)README\.txt$/i.test(member.archivePath)) return false; + return /\.(csv|json|txt)$/i.test(member.archivePath); + }); + for (const member of leftovers) { + if (/\.csv$/i.test(member.archivePath)) { + const { rows, header, warnings: csvWarnings } = parseCsv(member.text); + warnings.push(...csvWarnings); + if (header.length === 0 || rows.length === 0) { + warnings.push(`${member.path}: the CSV had no usable rows`); + continue; + } + const records = rows.map((row) => ({ + kind: "item", + text: header.map((column, index) => (row.cells[index] ? `${column}: ${row.cells[index]}` : null)).filter(Boolean).join("\n"), + label: `${basename(member.archivePath)} row ${row.index + 1}`, + byteStart: row.byteStart, + byteEnd: row.byteEnd, + file: member.name, + })); + documents.push( + buildDocument({ + parser: "archive", + format: "takeout", + kind: "archive-record", + method: "archive-member", + source: "takeout", + origin: member.origin, + files: [member], + content: records.map((record) => record.text).join("\n\n"), + segments: segmentRanges(records), + entries: records, + warnings: [], + meta: { archiveType: "takeout", dataset: basename(member.archivePath), columns: header, rows: records.length }, + }), + ); + continue; + } + if (/\.json$/i.test(member.archivePath) && /(^|\/)Chat\//i.test(member.archivePath)) continue; + + let parsed; + try { + parsed = parseJsonPayload(member); + } catch (error) { + warnings.push(`${member.path}: ${error.message}`); + continue; + } + const array = Array.isArray(parsed.value) + ? parsed.value + : Array.isArray(parsed.value?.items) + ? parsed.value.items + : Array.isArray(parsed.value?.locations) + ? parsed.value.locations + : null; + if (!array) { + warnings.push(`${member.path}: a Takeout JSON member with no array and no items[] was skipped`); + continue; + } + const { records } = objectRecords(member, array, { + render: (entry, index, leaves) => { + const rendered = renderLeaves(leaves); + return rendered === "" ? null : { text: rendered, label: `${basename(member.archivePath)} #${index}` }; + }, + }); + if (records.length === 0) { + warnings.push(`${member.path}: a Takeout JSON member held no readable text`); + continue; + } + documents.push( + buildDocument({ + parser: "archive", + format: "takeout", + kind: "archive-record", + method: "archive-member", + source: "takeout", + origin: member.origin, + files: [member], + content: records.map((record) => record.text).join("\n\n"), + segments: segmentRanges(records), + entries: records.map((record) => ({ ...record, file: member.name })), + warnings: [], + meta: { archiveType: "takeout", dataset: basename(member.archivePath), entries: records.length }, + }), + ); + } + + if (documents.length === 0) { + throw new UnrecognizedFormatError( + "this looks like a Google Takeout archive but no readable member was found (looked for .mbox mail, .srt/.vtt captions, Chat/messages.json, CSV and JSON activity files)", + { members: members.map((member) => member.archivePath).slice(0, 32) }, + ); + } + return { documents, warnings }; +} + +/** Meta (Instagram/Facebook) archives that are not a DM dump. */ +export function extractMetaActivity(members) { + const documents = []; + const warnings = []; + const activity = membersByName(members, /your_(instagram|facebook)_activity\/.*\.json$/i); + for (const member of activity) { + let parsed; + try { + parsed = parseJsonPayload(member); + } catch (error) { + warnings.push(`${member.path}: ${error.message}`); + continue; + } + const array = Array.isArray(parsed.value) ? parsed.value : null; + if (!array) { + warnings.push(`${member.path}: the activity file holds an object, not an array, and was skipped`); + continue; + } + const { records } = objectRecords(member, array, { + render: (entry, index, leaves) => { + const rendered = renderLeaves(leaves); + return rendered === "" ? null : { text: rendered, label: `${basename(member.archivePath)} #${index}` }; + }, + }); + if (records.length === 0) { + warnings.push(`${member.path}: the activity file held no readable text`); + continue; + } + documents.push( + buildDocument({ + parser: "archive", + format: "instagram-export", + kind: "archive-record", + method: "archive-member", + source: "instagram-export", + origin: member.origin, + files: [member], + content: records.map((record) => record.text).join("\n\n"), + segments: segmentRanges(records), + entries: records.map((record) => ({ ...record, file: member.name })), + warnings: [], + meta: { archiveType: "instagram-export", dataset: basename(member.archivePath), entries: records.length }, + }), + ); + } + return { documents, warnings }; +} + +/* ------------------------------------------------------------------ */ +/* entry points */ +/* ------------------------------------------------------------------ */ + +const EXTRACTORS = { + "x-archive": extractXArchive, + "discord-export": extractDiscordExport, + "telegram-export": extractTelegramExport, + "instagram-export": extractInstagramExport, + "linkedin-export": extractLinkedInExport, + takeout: extractTakeout, +}; + +/** + * Parse an archive into a list of documents. + * + * @param {ArchiveMember[]} members + * @param {{type?: string, maxDocuments?: number}} [options] + * @returns {{type: string, reasons: string[], documents: Array, + * warnings: string[], skipped: Array<{member: string, why: string}>}} + */ +export function parseArchive(members, options = {}) { + if (!Array.isArray(members) || members.length === 0) { + throw new InputError("an archive needs at least one member"); + } + const detected = options.type + ? { type: options.type, reasons: ["type supplied by the caller"], confidence: "asserted" } + : detectArchiveType(members); + + const extractor = EXTRACTORS[detected.type]; + if (!extractor) { + throw new UnrecognizedFormatError(`no extractor is implemented for archive type ${detected.type}`, { type: detected.type }); + } + + const result = extractor(members, options); + const documents = result.documents.slice(0, options.maxDocuments ?? 1000); + if (result.documents.length > documents.length) { + result.warnings.push(`the archive produced ${result.documents.length} documents; only the first ${documents.length} were kept`); + } + + // A member that no document used is either a support file or something we did + // not understand. Either way it is named, because an archive that quietly + // ignores half its members is not evidence. + const used = new Set(documents.flatMap((document) => (document.files ?? []).map((file) => file.name))); + const skipped = []; + for (const member of members) { + if (used.has(member.name)) continue; + if (/(^|\/)(metadata\.json|archive_browser\.html|README\.txt|channels\.json|users\.json|account\.js|account\.json|group_info\.json)$/i.test(member.archivePath)) continue; + if (/\/$/.test(member.archivePath)) continue; + skipped.push({ member: member.archivePath, why: "no extractor in this archive type claimed this member" }); + } + + return { + type: detected.type, + reasons: detected.reasons, + documents, + warnings: result.warnings ?? [], + skipped, + owner: result.owner ?? null, + }; +} + +/** Map a member's extension to the parser that should read it. */ +export function memberParser(member) { + const name = normalisedName(member.archivePath); + const lower = name.toLowerCase(); + if (/\.(eml)$/.test(lower)) return "email"; + if (/\.mbox$/.test(lower)) return "email"; + if (/\.(srt|vtt)$/.test(lower)) return "subtitle"; + if (/\.(docx|xlsx)$/.test(lower)) return "office"; + if (/\.json$/.test(lower) || /\.js$/.test(lower)) return "chat"; + if (/\.csv$/.test(lower)) return "csv"; + if (/\.html?$/.test(lower)) return "html"; + if (/\.(txt|md)$/.test(lower)) return "feishu-text"; + return null; +} + +/** + * Parse a container that holds exactly one document (a `.zip` around one + * `.mbox`, a `.docx`, a `.zip` around a single chat export). + * + * Returns `null` when the container is not single-document shaped, which is how + * `harvest` decides to fall back to `parseArchive`. + */ +export function parseArchiveFile(bytes, options = {}) { + const archiveName = options.archiveName ?? ""; + const { members, warnings } = readArchiveMembers(bytes, { archiveName, maxMemberBytes: options.maxMemberBytes }); + const interesting = members.filter((member) => !/(^|\/)(README\.txt|LICENSE\.txt)$/i.test(member.archivePath)); + + if (interesting.length === 1) { + const member = interesting[0]; + const parser = memberParser(member); + if (parser === "email") return [{ kind: "email", member, parser, warnings }]; + if (parser === "subtitle") return [{ kind: "subtitle", member, parser, warnings }]; + if (parser === "office") return [{ kind: "office", member, parser, warnings }]; + if (parser === "chat") return [{ kind: "chat", member, parser, warnings }]; + } + + if (interesting.length === 0) { + throw new UnrecognizedFormatError(`${archiveName} is an empty zip container`, { members: [] }); + } + return null; +} diff --git a/src/parse/chat.mjs b/src/parse/chat.mjs new file mode 100644 index 00000000..e0f54761 --- /dev/null +++ b/src/parse/chat.mjs @@ -0,0 +1,838 @@ +/** + * chat.mjs — chat exports → one record per turn (speaker + time), or nothing. + * + * Supported shapes, each identified by a structural marker rather than by its + * file name: + * + * - **ChatGPT** `conversations.json` — an array whose objects carry + * `mapping` + `title` (the pre-2024 "chat_messages" shape is also accepted). + * - **Claude** `conversations.json` — an array whose objects carry + * `chat_messages` + `uuid`, with `sender: "human" | "assistant"`. + * - **Slack** export directory files — `users.json` + `channels.json`, or a + * per-channel `*.json` array whose entries carry `user`/`username` + `ts`. + * - **Telegram** `result.json` — `{name, type, messages: [...]}` with + * `from`/`from_id`, `date`/`date_unixtime` and `text` (a string or an array + * of `{type:"link"|"bold"|…, text}` fragments). + * - **Discord** — a `messages/*.json` array carrying `author` + `timestamp`. + * + * Everything else is reported as **not a recognised export format**, with the + * markers that were looked for. There is no "best effort text dump" path: an + * export whose shape we do not know may still hold a person's words, and + * guessing which field is the speaker would put words in the wrong mouth. + * + * Timestamps are kept as the export wrote them (`ts` is a Slack epoch string, + * `date_unixtime` is Telegram's) *and* normalised to ISO-8601 when the value is + * unambiguous. An unparseable date is preserved verbatim and reported, never + * silently replaced with the current time. + */ + +import { + SourceFile, + UnrecognizedFormatError, + buildDocument, + findObjectArray, + iterateObjects, + parseJsonPayload, + pick, + recordsFromCharSpans, + walkJsonLeaves, +} from "./common.mjs"; + +/* ------------------------------------------------------------------ */ +/* shared text helpers */ +/* ------------------------------------------------------------------ */ + +/** Render an export's message text field, whatever shape it has. */ +function renderMessageText(value, warnings, context) { + if (typeof value === "string") return value; + if (value === null || value === undefined) return ""; + + // Telegram (and Telegram-derived tools) use an array of typed fragments. + if (Array.isArray(value)) { + const parts = []; + let unsupported = 0; + for (const fragment of value) { + if (typeof fragment === "string") { + parts.push(fragment); + continue; + } + if (fragment && typeof fragment === "object") { + if (typeof fragment.text === "string") { + parts.push(fragment.text); + continue; + } + if (fragment.type === "mention" && typeof fragment.text === "string") { + parts.push(fragment.text); + continue; + } + unsupported += 1; + } + } + if (unsupported > 0) { + warnings.push(`${context}: ${unsupported} text fragment(s) had no plain text (an emoji, an attachment or a poll) and were left out`); + } + return parts.join(""); + } + if (typeof value === "object") { + const nested = pick(value, ["text", "content", "body", "message"]); + if (nested.key) return renderMessageText(nested.value, warnings, context); + warnings.push(`${context}: the text field is an object with keys ${Object.keys(value).join(",") || "(none)"} and no recognisable text`); + return ""; + } + if (typeof value === "number" || typeof value === "boolean") return String(value); + return ""; +} + +/** + * Normalise a timestamp to ISO-8601, or keep it verbatim. + * Handles epoch seconds, epoch milliseconds, epoch microseconds, ISO strings and + * the several date formats the exports use. Returns `{iso, raw, inferredUnit}`. + */ +export function normaliseTimestamp(value) { + if (value === null || value === undefined || value === "") { + return { iso: null, raw: null, inferredUnit: null }; + } + const raw = String(value); + + if (/^\d{9,19}$/.test(raw)) { + const numeric = Number(raw); + // Slack uses seconds (10 digits), Discord ISO, Telegram both seconds and + // millisecond strings, some tools microseconds. + if (raw.length >= 16) { + return { iso: new Date(numeric / 1000).toISOString(), raw, inferredUnit: "microseconds" }; + } + if (raw.length >= 13) { + return { iso: new Date(numeric).toISOString(), raw, inferredUnit: "milliseconds" }; + } + return { iso: new Date(numeric * 1000).toISOString(), raw, inferredUnit: "seconds" }; + } + + if (/^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}(:\d{2})?/.test(raw)) { + const normalised = raw.includes("T") ? raw : raw.replace(" ", "T"); + const withZone = /[Zz]|[+-]\d{2}:?\d{2}$/.test(normalised) ? normalised : `${normalised}Z`; + const date = new Date(withZone); + if (!Number.isNaN(date.getTime())) { + return { iso: date.toISOString(), raw, inferredUnit: "iso" }; + } + } + + return { iso: null, raw, inferredUnit: null }; +} + +function stableJson(value) { + return JSON.stringify(value, null, 2); +} + +function scalar(value) { + if (value === null) return "null"; + if (Array.isArray(value)) return value.map((item) => scalar(item)).join(", "); + if (typeof value === "object") return ""; + return String(value); +} + +function truncate(value, limit = 120) { + const text = String(value); + return text.length > limit ? `${text.slice(0, limit - 1)}…` : text; +} + +/* ------------------------------------------------------------------ */ +/* format detection */ +/* ------------------------------------------------------------------ */ + +/** + * Which chat export is this file? + * + * Detection is structural and returns the reasons it decided, so a receipt can + * say *why* a file was read as ChatGPT rather than Telegram. + * + * @param {SourceFile} file + * @returns {{format: string|null, reasons: string[], value?: unknown, shape: object}} + */ +export function detectChatFormat(file) { + const reasons = []; + let parsed; + try { + parsed = parseJsonPayload(file); + } catch (error) { + // A Slack `users.json` and a Discord `messages/*.json` are still JSON, so a + // parse failure here is fatal for every format. + throw new UnrecognizedFormatError( + `${file.label} is not a recognised chat export: ${error.message}`, + { path: file.path }, + ); + } + const value = parsed.value; + const shape = { root: Array.isArray(value) ? "array" : typeof value, length: Array.isArray(value) ? value.length : null }; + + if (!Array.isArray(value) && (value === null || typeof value !== "object")) { + throw new UnrecognizedFormatError( + `${file.label} is not a recognised chat export: the JSON root is ${typeof value} (${truncate(stableJson(value), 40)}), expected an array or an object`, + { path: file.path }, + ); + } + + // ---- Telegram result.json ------------------------------------------- + if (!Array.isArray(value) && Array.isArray(value.messages)) { + const sample = value.messages.find((item) => item && typeof item === "object") ?? {}; + if ("from" in sample || "from_id" in sample || "date_unixtime" in sample || "date" in sample) { + reasons.push("object root with a messages[] array whose entries carry Telegram's from/date fields"); + if (typeof value.name === "string") reasons.push(`chat name: ${truncate(value.name, 60)}`); + return { format: "telegram-export", reasons, value, shape }; + } + reasons.push("object root with a messages[] array, but the entries carry none of Telegram's fields"); + } + + if (Array.isArray(value)) { + const objects = value.filter((item) => item && typeof item === "object" && !Array.isArray(item)); + const sample = objects[0] ?? {}; + const keys = new Set(objects.slice(0, 5).flatMap((item) => Object.keys(item))); + + // ---- Slack channels.json (channel index) -------------------------- + if (objects.length > 0 && objects.every((item) => "id" in item && "name" in item && !("messages" in item))) { + reasons.push("array of channel objects with id+name (Slack channels.json)"); + return { format: "slack-channels", reasons, value, shape }; + } + + // ---- Slack users.json --------------------------------------------- + if ( + objects.length > 0 && + objects.every((item) => "id" in item && ("profile" in item || "real_name" in item || "is_bot" in item)) && + !("ts" in sample) + ) { + reasons.push("array of user objects with id+profile/real_name (Slack users.json)"); + return { format: "slack-users", reasons, value, shape }; + } + + // ---- Slack channel messages --------------------------------------- + if (objects.length > 0 && objects.some((item) => "ts" in item) && objects.some((item) => "user" in item || "username" in item || "bot_id" in item || "text" in item)) { + reasons.push("array of message objects with ts + user/username/text (Slack channel export)"); + return { format: "slack-messages", reasons, value, shape }; + } + + // ---- Discord messages --------------------------------------------- + if (objects.length > 0 && objects.some((item) => "author" in item) && objects.some((item) => "timestamp" in item || "id" in item)) { + reasons.push("array of message objects with author + timestamp (Discord messages export)"); + return { format: "discord-messages", reasons, value, shape }; + } + + // ---- ChatGPT conversations.json ----------------------------------- + if (objects.length > 0 && objects.some((item) => "mapping" in item)) { + reasons.push("array of conversation objects carrying a mapping (ChatGPT conversations.json)"); + if (keys.has("create_time")) reasons.push("entries also carry create_time"); + return { format: "chatgpt-export", reasons, value, shape }; + } + if (objects.length > 0 && objects.some((item) => "chat_messages" in item) && keys.has("title")) { + reasons.push("array of conversation objects carrying chat_messages + title (ChatGPT 2023 export)"); + return { format: "chatgpt-export", reasons, value, shape }; + } + + // ---- Claude conversations.json ------------------------------------ + if (objects.length > 0 && objects.some((item) => Array.isArray(item.chat_messages)) && objects.some((item) => "uuid" in item)) { + reasons.push("array of conversation objects carrying chat_messages + uuid (Claude conversations.json)"); + return { format: "claude-export", reasons, value, shape }; + } + + // ---- Instagram / Facebook message dumps --------------------------- + const instagram = findObjectArray(value, ["sender_name", "timestamp_ms"], { minLength: 1 }); + if (instagram) { + reasons.push(`array of message objects with sender_name + timestamp_ms at ${instagram.path.join(".") || ""} (Instagram/Facebook messages)`); + const participants = objects.find((item) => Array.isArray(item.participants)); + if (participants) reasons.push(`conversation lists participants: ${participants.participants.map((p) => p?.name).filter(Boolean).join(", ")}`); + return { format: "instagram-messages", reasons, value, shape }; + } + } else { + // ---- object-rooted containers ------------------------------------- + const nested = pick(value, ["conversations", "chats", "messages", "records", "data"]); + if (nested.key && Array.isArray(nested.value)) { + const inner = nested.value.filter((item) => item && typeof item === "object"); + if (inner.some((item) => "mapping" in item)) { + reasons.push(`object root with ${nested.key}[] carrying a mapping (ChatGPT conversations.json)`); + return { format: "chatgpt-export", reasons, value: nested.value, shape: { ...shape, wrapped: nested.key } }; + } + if (inner.some((item) => Array.isArray(item.chat_messages))) { + reasons.push(`object root with ${nested.key}[] carrying chat_messages (Claude conversations.json)`); + return { format: "claude-export", reasons, value: nested.value, shape: { ...shape, wrapped: nested.key } }; + } + reasons.push(`object root with a ${nested.key}[] array, but its entries match no known export`); + } + } + + throw new UnrecognizedFormatError( + `${file.label} is not a recognised chat export. Looked for: ChatGPT mapping[]/chat_messages[], Claude chat_messages[]+uuid, Slack channels.json/users.json/channel messages (ts+user), Telegram result.json {messages[]}, Discord author+timestamp. Found: ${describesShape(value, shape)}. No guess was made.`, + { path: file.path, shape }, + ); +} + +function describesShape(value, shape) { + if (Array.isArray(value)) { + const sample = value.find((item) => item && typeof item === "object" && !Array.isArray(item)); + const keys = sample ? Object.keys(sample).slice(0, 12).join(",") : "(no object entries)"; + return `an array of ${value.length} item(s); the first object has keys [${keys}]`; + } + if (value && typeof value === "object") { + return `an object with keys [${Object.keys(value).slice(0, 12).join(",")}]`; + } + return `a JSON ${shape.root}`; +} + +/* ------------------------------------------------------------------ */ +/* ChatGPT */ +/* ------------------------------------------------------------------ */ + +/** + * Walk a ChatGPT `mapping` (a node graph, not a list) into a linear turn list. + * + * The mapping is a tree keyed by node id with `parent`/`children`; the + * conversation is the path from the root to a leaf. When a message has several + * children the export contains an edited branch — only one is on the live path, + * so the others are reported rather than merged, because merging two versions of + * the same answer fabricates a reply that was never sent. + */ +export function lineariseChatGptMapping(mapping, warnings, conversationLabel) { + if (!mapping || typeof mapping !== "object") return { turns: [], branches: 0, roots: 0 }; + const nodes = Object.entries(mapping); + const childrenOf = new Map(); + const hasParent = new Set(); + for (const [id, node] of nodes) { + const children = Array.isArray(node?.children) ? node.children : []; + childrenOf.set(id, children); + for (const child of children) hasParent.add(child); + } + const roots = nodes.map(([id]) => id).filter((id) => !hasParent.has(id)); + if (roots.length === 0) { + warnings.push(`${conversationLabel}: the mapping has no root node (every node has a parent), so no turns could be ordered`); + return { turns: [], branches: 0, roots: 0 }; + } + if (roots.length > 1) { + warnings.push(`${conversationLabel}: the mapping has ${roots.length} root nodes; only the first was followed`); + } + + const turns = []; + let branches = 0; + const visited = new Set(); + const walk = (id, depth) => { + if (visited.has(id) || depth > 100_000) return; + visited.add(id); + const node = mapping[id]; + const children = childrenOf.get(id) ?? []; + if (children.length > 1) { + branches += 1; + warnings.push(`${conversationLabel}: node ${id} has ${children.length} children (an edited branch); only the first path was recorded`); + } + const message = node?.message; + if (message) turns.push({ nodeId: id, depth, message }); + for (const child of children) walk(child, depth + 1); + }; + walk(roots[0], 0); + return { turns, branches, roots: roots.length }; +} + +function chatGptSpeaker(message) { + const role = message?.author?.role; + if (typeof role === "string" && role !== "") return role; + if (typeof message?.role === "string" && message.role !== "") return message.role; + return null; +} + +function chatGptText(message, warnings, label) { + const content = message?.content; + if (!content) return ""; + if (typeof content === "string") return content; + const parts = content.parts; + if (Array.isArray(parts)) { + const pieces = []; + let nonText = 0; + for (const part of parts) { + if (typeof part === "string") pieces.push(part); + else if (part && typeof part === "object" && typeof part.text === "string") pieces.push(part.text); + else if (part && typeof part === "object" && part.content_type === "image_asset_pointer") nonText += 1; + else nonText += 1; + } + if (nonText > 0) warnings.push(`${label}: ${nonText} non-text content part(s) (an image, a tool call or a code attachment) were left out`); + return pieces.join("\n"); + } + if (typeof content.text === "string") return content.text; + return ""; +} + +/* ------------------------------------------------------------------ */ +/* Slack */ +/* ------------------------------------------------------------------ */ + +/** + * Build `userId -> display name` from a Slack `users.json`. + * Returns an empty map (with a warning) when the file is missing or unreadable: + * a missing name map degrades labels, it does not invalidate the messages. + */ +export function readSlackUsers(usersText, warnings, label) { + const names = new Map(); + if (!usersText) { + warnings.push(`${label}: no users.json was supplied, so Slack user ids (U123…) could not be resolved to names`); + return names; + } + let parsed; + try { + parsed = JSON.parse(String(usersText).trim()); + } catch (error) { + warnings.push(`${label}: users.json could not be parsed (${error.message}); Slack user ids were left unresolved`); + return names; + } + const list = Array.isArray(parsed) ? parsed : Array.isArray(parsed?.members) ? parsed.members : []; + for (const user of list) { + if (!user || typeof user !== "object" || typeof user.id !== "string") continue; + const profile = user.profile ?? {}; + const name = profile.display_name || profile.real_name || user.real_name || user.name || null; + if (name) names.set(user.id, name); + } + if (names.size === 0) warnings.push(`${label}: users.json held no resolvable display names`); + return names; +} + +/* ------------------------------------------------------------------ */ +/* record building */ +/* ------------------------------------------------------------------ */ + +/** + * Turn a list of `{speaker, timestamp, text, marker}` into records whose byte + * ranges point back at the export. + * + * The byte range of a turn is deliberately the range of its **message object** + * (or, when only a value node can be located, of its text value), found by + * searching the raw payload for the marker string the export wrote. When the + * marker cannot be located the record reports `null` and says so: a wrong offset + * is worse than a missing one. + */ +function turnsToRecords(file, turns, warnings, label) { + const slices = []; + for (const turn of turns) { + const text = turn.text.trim(); + if (text === "") continue; + const located = locateTurn(file, turn); + if (!located) { + warnings.push(`${label}: the raw location of a turn (${truncate(turn.speaker ?? "unknown", 20)} @ ${truncate(turn.timestamp ?? "?", 24)}) could not be found in the payload; its anchor reports no byte offset`); + } + slices.push({ + text, + // `recordsFromCharSpans` needs character offsets into `file.text`; `-1` + // marks "not located", which becomes a `null` byte range. + charStart: located ? located.charStart : 0, + charEnd: located ? located.charEnd : 0, + kind: "turn", + label: [turn.speaker ?? "unknown", turn.timestamp ?? null].filter(Boolean).join(" · "), + located: Boolean(located), + }); + } + return slices.map((slice) => { + const record = { + kind: slice.kind, + text: slice.text, + label: slice.label, + file: file.name, + }; + if (slice.located) { + record.byteStart = file.charToByte(slice.charStart); + record.byteEnd = file.charToByte(slice.charEnd); + } else { + record.byteStart = null; + record.byteEnd = null; + } + return record; + }); +} + +/** + * Find the payload characters a turn came from. + * + * `anchor` is the longest distinctive substring the export contains for that + * turn (the message text). Searching for it is exact when the text is unique and + * conservative when it is not: an ambiguous match is reported as unlocatable + * rather than guessed at. + */ +function locateTurn(file, turn) { + const needle = turn.locateBy ?? turn.text.trim().slice(0, 200); + if (!needle) return null; + const first = file.text.indexOf(needle); + if (first === -1) return null; + if (file.text.indexOf(needle, first + 1) !== -1 && !turn.locateBy) return null; + return { charStart: first, charEnd: first + needle.length }; +} + +/* ------------------------------------------------------------------ */ +/* per-format parsers */ +/* ------------------------------------------------------------------ */ + +function parseChatGpt(file, value, warnings) { + const conversations = Array.isArray(value) ? value : [value]; + const turns = []; + const meta = { conversations: [], branches: 0 }; + + conversations.forEach((conversation, index) => { + const label = `conversation ${index + 1}${conversation?.title ? ` (${truncate(conversation.title, 40)})` : ""}`; + let linear; + if (conversation?.mapping && typeof conversation.mapping === "object") { + linear = lineariseChatGptMapping(conversation.mapping, warnings, label); + } else if (Array.isArray(conversation?.chat_messages)) { + linear = { + turns: conversation.chat_messages.map((message, position) => ({ nodeId: `chat_messages[${position}]`, depth: position, message })), + branches: 0, + roots: 1, + }; + } else { + warnings.push(`${label}: has neither a mapping nor chat_messages and was skipped`); + return; + } + + meta.branches += linear.branches; + let kept = 0; + for (const { message } of linear.turns) { + const speaker = chatGptSpeaker(message); + const timestamp = normaliseTimestamp(message?.create_time ?? message?.update_time ?? null); + const text = chatGptText(message, warnings, label); + const role = speaker ?? ""; + if (text.trim() === "") { + // An empty assistant turn is a tool call or a placeholder; it is not + // dialogue, but it is also not nothing — say so. + if (message?.content) warnings.push(`${label}: a ${role || "unknown-role"} turn has no text and was not anchored`); + continue; + } + turns.push({ + speaker: role || "unknown", + timestamp: timestamp.iso ?? null, + text, + locateBy: text.trim().slice(0, 200), + }); + kept += 1; + } + meta.conversations.push({ + title: typeof conversation?.title === "string" ? conversation.title : null, + id: conversation?.id ?? conversation?.conversation_id ?? null, + createTime: normaliseTimestamp(conversation?.create_time ?? null).iso, + turns: kept, + linearNodes: linear.turns.length, + branched: linear.branches > 0, + }); + }); + + return { turns, meta }; +} + +function parseClaude(value, warnings) { + const conversations = Array.isArray(value) ? value : [value]; + const turns = []; + const meta = { conversations: [] }; + + conversations.forEach((conversation, index) => { + const label = `conversation ${index + 1}${conversation?.name ? ` (${truncate(conversation.name, 40)})` : ""}`; + const messages = Array.isArray(conversation?.chat_messages) ? conversation.chat_messages : null; + if (!messages) { + warnings.push(`${label}: has no chat_messages array and was skipped`); + return; + } + let kept = 0; + for (const message of messages) { + const sender = typeof message?.sender === "string" ? message.sender : null; + if (sender === null) warnings.push(`${label}: a message has no sender field; it is recorded as "unknown" rather than guessed`); + const timestamp = normaliseTimestamp(message?.created_at ?? message?.updated_at ?? null); + let text = ""; + if (typeof message?.text === "string") { + text = message.text; + } else if (Array.isArray(message?.content)) { + text = renderMessageText(message.content, warnings, label); + } else if (typeof message?.content === "string") { + text = message.content; + } + if (text.trim() === "") continue; + turns.push({ + speaker: sender ?? "unknown", + timestamp: timestamp.iso ?? null, + text, + locateBy: text.trim().slice(0, 200), + }); + kept += 1; + } + meta.conversations.push({ + title: typeof conversation?.name === "string" ? conversation.name : null, + uuid: conversation?.uuid ?? null, + createTime: normaliseTimestamp(conversation?.created_at ?? null).iso, + turns: kept, + }); + }); + + return { turns, meta }; +} + +function parseSlackMessages(value, warnings, users, channelName) { + const list = Array.isArray(value) ? value : []; + const turns = []; + const skipped = { subtypes: 0, joins: 0, empty: 0 }; + let previousDay = null; + + for (const message of list) { + if (!message || typeof message !== "object") continue; + const subtype = typeof message.subtype === "string" ? message.subtype : null; + if (subtype && subtype !== "thread_broadcast" && subtype !== "file_share") { + // Channel joins, topic changes, bot noise: not what the person said. + skipped.subtypes += 1; + continue; + } + const timestamp = normaliseTimestamp(message.ts ?? null); + const speakerId = message.user ?? message.bot_id ?? null; + const speaker = (speakerId && users.get(speakerId)) || message.username || message.user_profile?.display_name || speakerId || "unknown"; + const text = renderMessageText(message.text, warnings, `Slack ${channelName}`); + if (text.trim() === "") { + skipped.empty += 1; + continue; + } + const day = timestamp.iso ? timestamp.iso.slice(0, 10) : null; + if (day && day !== previousDay) previousDay = day; + turns.push({ + speaker, + timestamp: timestamp.iso ?? timestamp.raw, + text, + locateBy: text.trim().slice(0, 120), + channel: channelName, + }); + } + + if (skipped.subtypes > 0) warnings.push(`Slack ${channelName}: ${skipped.subtypes} message(s) were channel events (join/leave/topic/bot) rather than dialogue and were skipped`); + if (skipped.empty > 0) warnings.push(`Slack ${channelName}: ${skipped.empty} message(s) had no text (an upload or a reaction) and were not anchored`); + return { turns, skipped }; +} + +function parseTelegram(file, value, warnings) { + const messages = Array.isArray(value.messages) ? value.messages : []; + const turns = []; + const skipped = { service: 0, empty: 0 }; + const chatName = typeof value.name === "string" ? value.name : file.name; + + for (const message of messages) { + if (!message || typeof message !== "object") continue; + if (message.type && message.type !== "message") { + skipped.service += 1; + continue; + } + const timestamp = normaliseTimestamp(message.date_unixtime ?? message.date ?? null); + const speaker = message.from ?? message.from_id ?? "unknown"; + const text = renderMessageText(message.text, warnings, `Telegram ${chatName}`); + if (text.trim() === "") { + skipped.empty += 1; + continue; + } + turns.push({ + speaker: typeof speaker === "string" ? speaker : String(speaker), + timestamp: timestamp.iso ?? timestamp.raw, + text, + locateBy: text.trim().slice(0, 200), + replyTo: message.reply_to_message_id ?? null, + }); + } + + if (skipped.service > 0) warnings.push(`Telegram ${chatName}: ${skipped.service} service message(s) (joins, pins, calls) were skipped`); + if (skipped.empty > 0) warnings.push(`Telegram ${chatName}: ${skipped.empty} message(s) carried no text (media, stickers, polls) and were not anchored`); + return { + turns, + meta: { + chatName, + chatType: value.type ?? null, + chatId: value.id ?? null, + messages: messages.length, + }, + }; +} + +function parseDiscord(file, value, warnings) { + const list = Array.isArray(value) ? value : []; + const turns = []; + const skipped = { empty: 0, bot: 0 }; + for (const message of list) { + if (!message || typeof message !== "object") continue; + const author = message.author ?? {}; + if (author.bot === true) { + skipped.bot += 1; + continue; + } + const timestamp = normaliseTimestamp(message.timestamp ?? null); + const text = renderMessageText(message.content, warnings, "Discord"); + if (text.trim() === "") { + skipped.empty += 1; + continue; + } + turns.push({ + speaker: author.nickname || author.global_name || author.name || author.id || "unknown", + timestamp: timestamp.iso ?? timestamp.raw, + text, + locateBy: text.trim().slice(0, 200), + channel: message.channel_id ?? null, + }); + } + if (skipped.bot > 0) warnings.push(`Discord: ${skipped.bot} bot message(s) were skipped`); + if (skipped.empty > 0) warnings.push(`Discord: ${skipped.empty} message(s) had no text (an attachment or an embed) and were not anchored`); + return { turns }; +} + +function parseInstagram(value, warnings) { + const matches = findObjectArray(value, ["sender_name", "timestamp_ms"], { minLength: 1 }); + const conversations = matches ? matches.items : []; + const turns = []; + const meta = { conversations: [] }; + + // A DMs export wraps one conversation per file: participants sit beside the + // messages array, so look one level up from wherever the array was found. + const participants = findParticipantNames(value); + + conversations.forEach((conversation, index) => { + const label = `conversation ${index + 1}`; + const messages = Array.isArray(conversation.messages) ? conversation.messages : [conversation]; + let kept = 0; + for (const message of messages) { + if (!message || typeof message !== "object") continue; + const timestamp = normaliseTimestamp(message.timestamp_ms ?? null); + const text = typeof message.content === "string" ? message.content : ""; + if (text.trim() === "") continue; + turns.push({ + speaker: typeof message.sender_name === "string" ? message.sender_name : "unknown", + timestamp: timestamp.iso ?? timestamp.raw, + text, + locateBy: text.trim().slice(0, 120), + }); + kept += 1; + } + meta.conversations.push({ + title: conversation.title ?? null, + participants: Array.isArray(conversation.participants) ? conversation.participants.map((p) => p?.name).filter(Boolean) : null, + turns: kept, + }); + if (kept === 0) warnings.push(`${label}: no message carried text`); + }); + + return { turns, meta: { ...meta, participants } }; +} + +function findParticipantNames(value) { + for (const { value: node } of iterateObjects(value, [], 0, 4)) { + if (node && Array.isArray(node.participants)) { + const names = node.participants.map((participant) => participant?.name).filter(Boolean); + if (names.length > 0) return names; + } + } + return []; +} + +/* ------------------------------------------------------------------ */ +/* entry point */ +/* ------------------------------------------------------------------ */ + +/** + * Parse a chat export. + * + * @param {SourceFile} file + * @param {{format?: string, users?: string, channelName?: string}} [options] + * `users` is the text of a Slack `users.json`, supplied by `harvest` when the + * export is a directory. + */ +export function parseChat(file, options = {}) { + const warnings = []; + const detected = options.format + ? { format: options.format, reasons: ["format supplied by the caller"], value: undefined } + : detectChatFormat(file); + + if (detected.format === "slack-users" || detected.format === "slack-channels") { + // These are support files, not conversations. They are real exports, but a + // directory harvest reads them for names — treating them as dialogue would + // put a member list into the knowledge base. + throw new UnrecognizedFormatError( + `${file.label} is a Slack ${detected.format === "slack-users" ? "users.json" : "channels.json"} support file, not a conversation; it is read for display names when a channel export is harvested`, + { path: file.path, format: detected.format }, + ); + } + + const value = detected.value ?? parseJsonPayload(file).value; + let turns; + let meta; + + switch (detected.format) { + case "chatgpt-export": { + const result = parseChatGpt(file, value, warnings); + turns = result.turns; + meta = result.meta; + break; + } + case "claude-export": { + const result = parseClaude(value, warnings); + turns = result.turns; + meta = result.meta; + break; + } + case "slack-messages": { + const users = readSlackUsers(options.users, warnings, file.label); + const result = parseSlackMessages(value, warnings, users, options.channelName ?? file.name); + turns = result.turns; + meta = { channel: options.channelName ?? null, messages: Array.isArray(value) ? value.length : 0, resolvedUsers: users.size }; + break; + } + case "telegram-export": { + const result = parseTelegram(file, value, warnings); + turns = result.turns; + meta = result.meta; + break; + } + case "discord-messages": { + const result = parseDiscord(file, value, warnings); + turns = result.turns; + meta = { messages: Array.isArray(value) ? value.length : 0 }; + break; + } + case "instagram-messages": { + const result = parseInstagram(value, warnings); + turns = result.turns; + meta = result.meta; + break; + } + default: + throw new UnrecognizedFormatError( + `${file.label}: format ${detected.format} has no parser`, + { path: file.path, format: detected.format }, + ); + } + + if (turns.length === 0) { + throw new UnrecognizedFormatError( + `${file.label} was read as ${detected.format} but no turn carried text; there is nothing to anchor`, + { path: file.path, format: detected.format }, + ); + } + + const records = turnsToRecords(file, turns, warnings, file.label); + const speakers = [...new Set(turns.map((turn) => turn.speaker))]; + const timestamps = turns.map((turn) => turn.timestamp).filter(Boolean).sort(); + const unlocated = records.filter((record) => record.byteStart === null).length; + if (unlocated > 0) { + warnings.push(`${unlocated} of ${records.length} turn(s) could not be located in the raw payload; their anchors carry text but no byte offset`); + } + + return buildDocument({ + parser: "chat", + format: detected.format, + kind: meta?.conversations !== undefined ? "chat-thread" : "chat", + method: "user-export", + source: "chat", + files: [file], + records: turnsToRecords(file, turns, [], file.label), + warnings, + meta: { + ...meta, + turns: turns.length, + speakers, + firstTimestamp: timestamps[0] ?? null, + lastTimestamp: timestamps[timestamps.length - 1] ?? null, + detection: detected.reasons, + unlocatedTurns: unlocated, + }, + dropped: [ + { what: "system and tool turns", why: "only turns that carry dialogue text are anchored" }, + { what: "attachments and embeds", why: "the export stores them as ids, not as content" }, + ], + }); +} + +export { SourceFile, UnrecognizedFormatError }; diff --git a/src/parse/common.mjs b/src/parse/common.mjs new file mode 100644 index 00000000..a2b6253e --- /dev/null +++ b/src/parse/common.mjs @@ -0,0 +1,653 @@ +/** + * common.mjs — the contract between `src/knowledge/**` and `src/parse/**`. + * + * A parser never touches the filesystem and never invents bytes. It receives a + * `SourceFile` (raw bytes plus a decoded view that remembers byte offsets) and + * returns a **document**: + * + * ```js + * { + * parser: "chat", // which module produced it + * format: "chatgpt-export", // the detected format id, or null when unknown + * kind: "chat", // ledger `kind` + * method: "user-export", // ledger `method` + * credentialed: false, + * source: "chatgpt-export", // bucket under knowledge/raw/ + * files: [SourceFile, ...], // every raw file the document was built from + * units: [{text, byteStart, byteEnd, file, segmentIndex}], // normalised lines + * anchors: [{kind, text, anchor:"k00NN:tM", index, byteStart, byteEnd, file}], + * warnings: ["..."], + * dropped: [{what, why}], + * meta: {...} + * } + * ``` + * + * `units` are produced by `assignAnchors` (see `src/knowledge/anchors.mjs`), and + * `anchors` by `buildSubAnchors`, so anchors stay globally monotonic per ledger + * id and every anchor resolves to a raw byte range. + * + * Format detection is **never a guess**: when nothing matches, parsers return + * `format: null` plus a `warnings` entry naming the bytes they looked at. The + * CLI turns that into "not a recognised export format" and a non-zero exit. + */ + +import { readFileSync, statSync } from "node:fs"; +import { basename, extname } from "node:path"; +import { + assignAnchors, + assignAnchorsToText, + buildSubAnchors, + decodeBuffer, + parseAnchor, + verifyByteConservation, +} from "../knowledge/anchors.mjs"; + +/* ------------------------------------------------------------------ */ +/* limits */ +/* ------------------------------------------------------------------ */ + +/** An archive member larger than this is recorded but not inflated. */ +export const DEFAULT_MAX_MEMBER_BYTES = 64 * 1024 * 1024; +/** A single line longer than this is split, with a warning. */ +export const DEFAULT_MAX_LINE_CHARS = 200_000; +/** How deep to walk a JSON document before stopping. */ +export const DEFAULT_MAX_JSON_DEPTH = 48; +/** How many array elements to emit as leaves per array. */ +export const DEFAULT_MAX_ARRAY_LEAVES = 20_000; + +/* ------------------------------------------------------------------ */ +/* errors */ +/* ------------------------------------------------------------------ */ + +/** + * Raised when a parser cannot recognise its input. The message is written to be + * shown to a human verbatim: it says what was tried and what was seen. + */ +export class UnrecognizedFormatError extends Error { + constructor(message, details = {}) { + super(message); + this.name = "UnrecognizedFormatError"; + this.details = details; + } +} + +/** Raised for an input that cannot be read at all (missing file, bad encoding). */ +export class InputError extends Error { + constructor(message, details = {}) { + super(message); + this.name = "InputError"; + this.details = details; + } +} + +/* ------------------------------------------------------------------ */ +/* source files */ +/* ------------------------------------------------------------------ */ + +/** + * One raw file plus a byte-faithful decoded view. + * + * `charToByte(i)` maps a character index in `text` to its byte offset in `raw`, + * which is what lets every parser report byte ranges without re-deriving them. + */ +export class SourceFile { + /** + * @param {{path?: string, name?: string, raw: Uint8Array, label?: string, + * preferred?: string, fallbacks?: string[]}} input + */ + constructor(input) { + if (!input?.raw) throw new TypeError("SourceFile requires raw bytes"); + this.path = input.path ?? input.name ?? ""; + this.name = input.name ?? basename(this.path); + this.raw = input.raw instanceof Uint8Array ? input.raw : new Uint8Array(input.raw); + this.label = input.label ?? this.name; + + const decoded = decodeBuffer(this.raw, { + ...(input.preferred ? { preferred: input.preferred } : {}), + ...(input.fallbacks ? { fallbacks: input.fallbacks } : {}), + }); + + this.text = decoded.text; + this.encoding = decoded.label; + this.bom = decoded.bom; + this.bomBytes = decoded.bomBytes; + this.lossy = decoded.lossy; + this.decodeWarnings = decoded.warnings; + + // Character index -> byte offset, walking the decoded text with the same + // UTF-8 re-encoding the decoder used. Only valid for the encodings where the + // decoded text round-trips through UTF-8; for legacy single/double byte + // codecs we fall back to a proportional ±1 byte approximation and say so. + this._charByteStarts = buildCharByteMap(this.text, this.raw, this.bomBytes); + } + + get bytes() { + return this.raw.length; + } + + /** Byte offset of character index `index`, or `raw.length` past the end. */ + charToByte(index) { + const map = this._charByteStarts; + if (index <= 0) return map.length > 0 ? map[0] : this.bomBytes; + if (index >= map.length) return this.raw.length; + return map[index]; + } + + /** Raw bytes for the character range `[start, end)`. */ + sliceChars(start, end) { + return this.raw.subarray(this.charToByte(start), this.charToByte(end)); + } + + /** The text of `[start, end)` exactly as it appears in the raw payload. */ + textOf(start, end) { + return this.text.slice(start, end); + } + + /** A `knowledge/raw/...`-relative descriptor, filled in by the store. */ + descriptor(extra = {}) { + return { + path: this.path, + name: this.name, + relativePath: null, + bytes: this.raw.length, + sha256: null, + encoding: this.encoding, + bom: this.bom, + lossy: this.lossy, + persisted: false, + bytesRaw: this.raw, + ...extra, + }; + } +} + +function buildCharByteMap(text, raw, bomBytes) { + const map = new Int32Array(text.length + 1); + let byte = bomBytes; + for (let index = 0; index < text.length; index += 1) { + map[index] = byte; + const code = text.codePointAt(index); + const size = code > 0xffff ? 2 : 1; + // UTF-8 length of this code point. + byte += code < 0x80 ? 1 : code < 0x800 ? 2 : code < 0x10000 ? 3 : 4; + if (size === 2) map[index + 1] = byte - (code < 0x10000 ? 3 : 4) + (code < 0x10000 ? 3 : 4); + } + map[text.length] = Math.min(byte, raw.length); + return map; +} + +/** Read a file from disk into a `SourceFile`. */ +export function loadSourceFile(path, options = {}) { + let raw; + try { + raw = new Uint8Array(readFileSync(path)); + } catch (error) { + throw new InputError(`cannot read ${path}: ${error.message}`, { path }); + } + return new SourceFile({ path, raw, ...options }); +} + +export function isRegularFile(path) { + try { + return statSync(path).isFile(); + } catch { + return false; + } +} + +/* ------------------------------------------------------------------ */ +/* text helpers */ +/* ------------------------------------------------------------------ */ + +const NAMED_ENTITIES = new Map([ + ["amp", "&"], + ["lt", "<"], + ["gt", ">"], + ["quot", '"'], + ["apos", "'"], + ["nbsp", "\u00a0"], + ["#39", "'"], + ["#x27", "'"], + ["#x2F", "/"], + ["hellip", "\u2026"], + ["mdash", "\u2014"], + ["ndash", "\u2013"], + ["rsquo", "\u2019"], + ["lsquo", "\u2018"], + ["ldquo", "\u201c"], + ["rdquo", "\u201d"], + ["middot", "\u00b7"], + ["bull", "\u2022"], + ["copy", "\u00a9"], +]); + +/** + * Decode the entity forms that actually appear in mail and OOXML payloads. + * Unknown entities are left verbatim — inventing a replacement would lose bytes. + * @returns {{text: string, unknown: string[]}} + */ +export function decodeEntities(input) { + const unknown = []; + const text = String(input).replace(/&(#x?[0-9a-fA-F]+|[a-zA-Z][a-zA-Z0-9]*);/g, (match, body) => { + const key = body.toLowerCase(); + if (NAMED_ENTITIES.has(key)) return NAMED_ENTITIES.get(key); + if (body.startsWith("#x") || body.startsWith("#X")) { + const code = Number.parseInt(body.slice(2), 16); + if (Number.isFinite(code) && code > 0 && code <= 0x10ffff) return String.fromCodePoint(code); + } else if (body.startsWith("#")) { + const code = Number.parseInt(body.slice(1), 10); + if (Number.isFinite(code) && code > 0 && code <= 0x10ffff) return String.fromCodePoint(code); + } + unknown.push(match); + return match; + }); + return { text, unknown }; +} + +const BLOCK_TAGS = new Set([ + "p", "br", "div", "tr", "li", "h1", "h2", "h3", "h4", "h5", "h6", + "table", "section", "article", "blockquote", "pre", "hr", "ul", "ol", "td", +]); + +/** + * Extract readable text from an HTML fragment. + * + * `script`/`style`/`head` content is *dropped*, and the dropped byte/char count + * is reported so the caller can put it in `warnings` — byte discipline says a + * deletion is a finding, not a detail. + * + * @param {string} html + * @returns {{text: string, droppedChars: number, droppedTags: string[]}} + */ +export function stripHtml(html) { + const droppedTags = new Set(); + let droppedChars = 0; + let text = ""; + let index = 0; + const source = String(html); + + while (index < source.length) { + const lt = source.indexOf("<", index); + if (lt === -1) { + text += source.slice(index); + break; + } + text += source.slice(index, lt); + + if (source.startsWith("", lt + 4); + index = end === -1 ? source.length : end + 3; + continue; + } + + const gt = source.indexOf(">", lt); + if (gt === -1) { + // Unterminated tag: keep the remainder verbatim rather than eating it. + text += source.slice(lt); + break; + } + + const rawTag = source.slice(lt + 1, gt); + const match = /^\/?\s*([a-zA-Z][a-zA-Z0-9:-]*)/.exec(rawTag); + const tag = match ? match[1].toLowerCase() : ""; + const closing = rawTag.startsWith("/"); + + if (!closing && (tag === "script" || tag === "style" || tag === "head" || tag === "title")) { + const closeIndex = source.toLowerCase().indexOf(`", closeIndex) + 1 || source.length; + droppedChars += end - lt; + droppedTags.add(tag); + index = end; + continue; + } + + if (BLOCK_TAGS.has(tag)) text += "\n"; + index = gt + 1; + } + + const { text: entityDecoded, unknown } = decodeEntities(text); + const collapsed = entityDecoded + .replace(/\r\n?/g, "\n") + .replace(/[ \t\f\v\u00a0]+/g, " ") + .replace(/ *\n */g, "\n") + .replace(/\n{3,}/g, "\n\n") + .trim(); + + return { text: collapsed, droppedChars, droppedTags: [...droppedTags], unknownEntities: unknown }; +} + +/* ------------------------------------------------------------------ */ +/* byte-aware JSON scanning */ +/* ------------------------------------------------------------------ */ + +/** + * Strip the `window.YTD..part = ...;` wrapper used by X (Twitter), + * Instagram and Google Takeout `.js` payloads, returning the JSON slice and the + * byte offset it starts at. + * + * @returns {{json: string, charOffset: number, wrapper: string|null}} + */ +export function unwrapJsonAssignment(text) { + const trimmedStart = text.search(/\S/); + if (trimmedStart === -1) return { json: "", charOffset: 0, wrapper: null }; + const head = text.slice(trimmedStart, trimmedStart + 64); + const match = /^(?:window\.)?[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)*\s*=\s*/.exec(head); + if (!match) return { json: text, charOffset: 0, wrapper: null }; + const body = text.slice(trimmedStart + match[0].length); + return { json: body, charOffset: trimmedStart + match[0].length, wrapper: match[0].trim() }; +} + +function skipWhitespace(text, index) { + let cursor = index; + while (cursor < text.length && (text[cursor] === " " || text[cursor] === "\t" || text[cursor] === "\n" || text[cursor] === "\r")) { + cursor += 1; + } + return cursor; +} + +function scanString(text, index) { + if (text[index] !== '"') throw new SyntaxError(`expected a string at offset ${index}`); + let cursor = index + 1; + while (cursor < text.length) { + const char = text[cursor]; + if (char === "\\") { + cursor += 2; + continue; + } + if (char === '"') return cursor + 1; + cursor += 1; + } + throw new SyntaxError(`unterminated string starting at offset ${index}`); +} + +function scanNumber(text, index) { + const match = /^-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?/.exec(text.slice(index)); + if (!match) throw new SyntaxError(`expected a number at offset ${index}`); + return index + match[0].length; +} + +function scanLiteral(text, index) { + for (const literal of ["true", "false", "null"]) { + if (text.startsWith(literal, index)) return index + literal.length; + } + throw new SyntaxError(`unexpected token at offset ${index}: ${text.slice(index, index + 12)}`); +} + +/** + * Walk every scalar in a JSON document, reporting its path and the byte range of + * the *token* (strings exclude their quotes, so `raw` can be re-parsed directly). + * + * The scanner is hand written rather than `JSON.parse` + a separate position + * search because positions must be exact: an anchor that points one byte off is + * worse than no anchor at all. + * + * @param {SourceFile} file + * @param {{maxDepth?: number, maxArrayLeaves?: number}} [options] + * @returns {{leaves: Array<{path: string, value: unknown, byteStart: number, + * byteEnd: number, charStart: number, charEnd: number}>, warnings: string[], + * wrapper: string|null, truncated: boolean}} + */ +export function walkJsonLeaves(file, options = {}) { + const maxDepth = options.maxDepth ?? DEFAULT_MAX_JSON_DEPTH; + const maxArrayLeaves = options.maxArrayLeaves ?? DEFAULT_MAX_ARRAY_LEAVES; + const { json, charOffset, wrapper } = unwrapJsonAssignment(file.text); + const leaves = []; + const warnings = []; + let truncated = false; + let arrayLeaves = 0; + + const pathOf = (segments) => segments.join("."); + + const valueEnd = (text, index) => { + const char = text[index]; + if (char === '"') return scanString(text, index); + if (char === "{" || char === "[") { + const stack = [char]; + let cursor = index + 1; + while (cursor < text.length && stack.length > 0) { + const current = text[cursor]; + if (current === '"') { + cursor = scanString(text, cursor); + continue; + } + if (current === "{") stack.push("}"); + else if (current === "[") stack.push("]"); + else if (current === "}" || current === "]") stack.pop(); + cursor += 1; + } + return cursor; + } + if (char === "-" || (char >= "0" && char <= "9")) return scanNumber(text, index); + return scanLiteral(text, index); + }; + + const walk = (start, segments, depth) => { + if (depth > maxDepth) { + if (!truncated) { + truncated = true; + warnings.push(`JSON nesting exceeded ${maxDepth} levels; deeper values were not anchored`); + } + return valueEnd(json, start); + } + const index = skipWhitespace(json, start); + const char = json[index]; + if (char === undefined) return index; + + if (char === "{") { + let cursor = skipWhitespace(json, index + 1); + if (json[cursor] === "}") return cursor + 1; + while (cursor < json.length) { + cursor = skipWhitespace(json, cursor); + if (json[cursor] !== '"') throw new SyntaxError(`expected an object key at offset ${cursor}`); + const keyEnd = scanString(json, cursor); + const key = JSON.parse(json.slice(cursor, keyEnd)); + cursor = skipWhitespace(json, keyEnd); + if (json[cursor] !== ":") throw new SyntaxError(`expected ':' at offset ${cursor}`); + cursor = walk(cursor + 1, [...segments, key], depth + 1); + cursor = skipWhitespace(json, cursor); + if (json[cursor] === ",") { + cursor += 1; + continue; + } + if (json[cursor] === "}") return cursor + 1; + throw new SyntaxError(`expected ',' or '}' at offset ${cursor}`); + } + return cursor; + } + + if (char === "[") { + let cursor = skipWhitespace(json, index + 1); + if (json[cursor] === "]") return cursor + 1; + let element = 0; + while (cursor < json.length) { + if (arrayLeaves >= maxArrayLeaves) { + if (!truncated) { + truncated = true; + warnings.push(`array at ${pathOf(segments) || ""} exceeded ${maxArrayLeaves} elements; the remainder was not anchored`); + } + return valueEnd(json, index); + } + arrayLeaves += 1; + cursor = walk(cursor, [...segments, String(element)], depth + 1); + element += 1; + cursor = skipWhitespace(json, cursor); + if (json[cursor] === ",") { + cursor += 1; + continue; + } + if (json[cursor] === "]") return cursor + 1; + throw new SyntaxError(`expected ',' or ']' at offset ${cursor}`); + } + return cursor; + } + + const tokenEnd = valueEnd(json, index); + const rawToken = json.slice(index, tokenEnd); + let value; + try { + value = JSON.parse(rawToken); + } catch { + value = rawToken; + } + const isString = char === '"'; + leaves.push({ + path: pathOf(segments), + value, + charStart: charOffset + index + (isString ? 1 : 0), + charEnd: charOffset + tokenEnd - (isString ? 1 : 0), + byteStart: file.charToByte(charOffset + index + (isString ? 1 : 0)), + byteEnd: file.charToByte(charOffset + tokenEnd - (isString ? 1 : 0)), + container: false, + }); + return tokenEnd; + }; + + walk(0, [], 0); + return { leaves, warnings, wrapper, truncated }; +} + +/** + * Parse the JSON payload of a source file, unwrapping a `window.X =` prefix. + * Throws `UnrecognizedFormatError` when the payload is not JSON at all. + */ +export function parseJsonPayload(file) { + const { json, wrapper } = unwrapJsonAssignment(file.text); + const trimmed = json.trim().replace(/;\s*$/, ""); + if (trimmed === "") { + throw new UnrecognizedFormatError(`${file.label} is empty; there is no JSON payload to parse`, { path: file.path }); + } + try { + return { value: JSON.parse(trimmed), wrapper }; + } catch (error) { + throw new UnrecognizedFormatError( + `${file.label} is not valid JSON (${error.message}); known export formats are JSON, JSON-lines, CSV, .eml/.mbox, .srt/.vtt, .docx/.xlsx and zip`, + { path: file.path, cause: error.message }, + ); + } +} + +/** Every JSON object reachable from `value`, depth first, with its path. */ +export function* iterateObjects(value, path = [], depth = 0, maxDepth = DEFAULT_MAX_JSON_DEPTH) { + if (depth > maxDepth || value === null || typeof value !== "object") return; + if (Array.isArray(value)) { + for (let index = 0; index < value.length; index += 1) { + yield* iterateObjects(value[index], [...path, String(index)], depth + 1, maxDepth); + } + return; + } + yield { value, path }; + for (const key of Object.keys(value)) { + yield* iterateObjects(value[key], [...path, key], depth + 1, maxDepth); + } +} + +/** + * Depth-first search for the first array of plain objects whose length is at + * least `minLength` and that contains at least one of `requiredKeys`. + * Used to locate message lists without hard-coding a whole schema. + */ +export function findObjectArray(value, requiredKeys, options = {}) { + const minLength = options.minLength ?? 1; + const maxDepth = options.maxDepth ?? 8; + const queue = [{ value, path: [], depth: 0 }]; + while (queue.length > 0) { + const { value: current, path, depth } = queue.shift(); + if (depth > maxDepth || current === null || typeof current !== "object") continue; + if (Array.isArray(current)) { + if ( + current.length >= minLength && + current.every((item) => item && typeof item === "object" && !Array.isArray(item)) && + current.some((item) => requiredKeys.some((key) => key in item)) + ) { + return { items: current, path }; + } + for (let index = 0; index < current.length; index += 1) { + queue.push({ value: current[index], path: [...path, String(index)], depth: depth + 1 }); + } + continue; + } + for (const key of Object.keys(current)) { + queue.push({ value: current[key], path: [...path, key], depth: depth + 1 }); + } + } + return null; +} + +/** First present key in `keys`, so export dialects can be tolerated explicitly. */ +export function pick(object, keys) { + for (const key of keys) { + if (object && object[key] !== undefined && object[key] !== null && object[key] !== "") { + return { key, value: object[key] }; + } + } + return { key: null, value: undefined }; +} + +/* ------------------------------------------------------------------ */ +/* re-exported anchor plumbing */ +/* ------------------------------------------------------------------ */ + +export { assignAnchors, buildSubAnchors, parseAnchor }; + +/** + * Run `assignAnchors` over a set of leaf records, then attach sub-anchors. + * + * Every parser ends with this call. `content` must contain every scalar worth + * reading; the leaf records supply the raw byte ranges that make the resulting + * anchors resolvable. + * + * @param {object} input + * @param {SourceFile} input.file + * @param {string} input.content normalised, human readable text + * @param {number} input.ledgerId numeric ledger index (1 → k0001) + * @param {Array<{kind: string, text: string, byteStart: number, byteEnd: number}>} [input.entries] + * @param {string[]} [input.warnings] + */ +export function finalizeDocument(input) { + const { + file, + content, + ledgerId, + entries = [], + warnings = [], + maxBlocks, + collapseSpaces, + } = input; + + const raw = Buffer.from(content, "utf8"); + const anchored = assignAnchors(raw, { + ...(maxBlocks ? { maxBlocks } : {}), + ...(collapseSpaces === undefined ? {} : { collapseSpaces }), + }); + + const kId = `k${String(ledgerId).padStart(4, "0")}`; + const anchors = buildSubAnchors(entries, kId).map((entry, index) => ({ + ...entry, + byteStart: entries[index].byteStart ?? null, + byteEnd: entries[index].byteEnd ?? null, + file: entries[index].file ?? file.name, + })); + + return { + text: anchored.text, + units: anchored.units.map((unit) => ({ ...unit, file: file.name })), + // Anchors live in the *parsed text*, not in `content`, so their byte ranges + // point back at the raw payload directly. + anchors, + warnings: [...warnings, ...anchored.warnings], + encoding: { + label: anchored.encoding, + bom: anchored.bom, + lossy: anchored.lossy, + bytes: anchored.byteLength, + unmappedTail: anchored.unmappedTail, + }, + parse: { + file: file.name, + path: file.path, + charCount: content.length, + lineCount: content === "" ? 0 : content.split("\n").length, + }, + ledgerId: kId, + }; +} diff --git a/src/parse/subtitle.mjs b/src/parse/subtitle.mjs new file mode 100644 index 00000000..8ef498eb --- /dev/null +++ b/src/parse/subtitle.mjs @@ -0,0 +1,319 @@ +/** + * subtitle.mjs — `.srt` and `.vtt` → one record per cue (speaker + timecode). + * + * Ported from `tools/research/srt_to_transcript.py`, with three differences that + * the Python version cannot offer: + * + * 1. **Cue boundaries are kept.** Python flattened everything into prose + * paragraphs, so a quote could not be traced back to a moment in the video. + * Here every cue carries its index, its `start`/`end` timecode and its raw + * byte range, hence its own `[k00NN:tM]` anchor. + * 2. **Byte ranges are exact.** Cue text comes from a slice of the decoded + * payload, so `recordsFromCharSpans` can point at the original bytes. + * 3. **Nothing is silent.** A malformed cue, a duplicate index, a stray + * timestamp or an unknown encoding is reported in `warnings`. + * + * Speaker detection stays conservative: SubRip has no speaker field, so a name is + * only claimed when the line matches `` (WebVTT voice span) or the strict + * `NAME: text` shape with a short, punctuation-free name. Anything else is + * attributed to `null` and the cue text is left untouched. + */ + +import { UnrecognizedFormatError, buildDocument, recordsFromCharSpans } from "./common.mjs"; + +const SRT_TIMECODE = /^\s*(\d{1,2}):(\d{2}):(\d{2})[,.](\d{1,3})\s*-->\s*(\d{1,2}):(\d{2}):(\d{2})[,.](\d{1,3})(.*)$/; +const VTT_TIMECODE = /^\s*(?:(\d{1,2}):)?(\d{2}):(\d{2})\.(\d{3})\s*-->\s*(?:(\d{1,2}):)?(\d{2}):(\d{2})\.(\d{3})(.*)$/; +const VTT_VOICE = /^]+)*\s+([^>]+)>/; +const SPEAKER_PREFIX = /^([\p{L}\p{N}][\p{L}\p{N} ._'-]{0,23}):\s+(\S.*)$/u; +const TIMECODE_ANY = /^\s*\d{1,2}:\d{2}:\d{2}[.,]\d{1,3}\s*-->/; +const TAG = /<[^>]*>/g; +/** WebVTT cue settings that appear after the arrow. */ +const CUE_SETTINGS = /\b(?:align|position|size|line|region|vertical|snap-to-lines):\S+/gi; + +function pad(value, width) { + return String(value).padStart(width, "0"); +} + +/** + * Parse a timecode into milliseconds, or `null` when it is not a timecode. + * Both `.` and `,` are accepted as the decimal separator (`.srt` in the wild uses + * both), and hours are optional in WebVTT. + */ +export function parseTimecode(value) { + const match = /^(\d{1,3}):(\d{2}):(\d{2})[.,](\d{1,3})$/.exec(String(value).trim()); + if (!match) return null; + const [, hours, minutes, seconds, fraction] = match; + const millis = Number(fraction.padEnd(3, "0").slice(0, 3)); + return Number(hours) * 3_600_000 + Number(minutes) * 60_000 + Number(seconds) * 1_000 + millis; +} + +/** `3723004` → `01:02:03.004`. */ +export function formatTimecode(millis) { + if (!Number.isFinite(millis)) return null; + const sign = millis < 0 ? "-" : ""; + const total = Math.abs(millis); + const hours = Math.floor(total / 3_600_000); + const minutes = Math.floor((total % 3_600_000) / 60_000); + const seconds = Math.floor((total % 60_000) / 1_000); + const ms = total % 1_000; + return `${sign}${pad(hours, 2)}:${pad(minutes, 2)}:${pad(seconds, 2)}.${pad(ms, 3)}`; +} + +/** + * Detect which subtitle format a payload is. + * + * WebVTT is identified by its `WEBVTT` magic (after an optional BOM, which the + * caller has already stripped from `text`). SubRip is identified by at least one + * timestamp line — deliberately *not* by the `.srt` extension alone, because a + * `.txt` transcript with timecodes is the same thing and rejecting it would be + * user-hostile. A payload with neither is reported as unrecognised. + * + * @param {import("./common.mjs").SourceFile} file + * @returns {{format: "srt"|"vtt", reasons: string[]}} + */ +export function detectSubtitleFormat(file) { + const reasons = []; + const head = file.text.replace(/^\uFEFF/, ""); + + if (/^\s*WEBVTT[\s(]/.test(head) || /^\s*WEBVTT\s*$/.test(head.split("\n")[0] ?? "")) { + reasons.push("payload starts with the WEBVTT signature"); + return { format: "vtt", reasons }; + } + if (file.name.toLowerCase().endsWith(".vtt") && VTT_TIMECODE.test(head.split("\n").find((line) => VTT_TIMECODE.test(line)) ?? "")) { + reasons.push(".vtt extension and a WebVTT timecode"); + return { format: "vtt", reasons }; + } + const lines = head.split("\n"); + const srtLines = lines.filter((line) => SRT_TIMECODE.test(line)); + if (srtLines.length > 0) { + reasons.push(`${srtLines.length} SubRip timecode line(s)`); + return { format: "srt", reasons }; + } + const vttLines = lines.filter((line) => VTT_TIMECODE.test(line)); + if (vttLines.length > 0) { + reasons.push(`${vttLines.length} WebVTT timecode line(s) without a WEBVTT header`); + return { format: "vtt", reasons }; + } + + throw new UnrecognizedFormatError( + `${file.label} is not a recognised subtitle file: no WEBVTT signature and no "HH:MM:SS,mmm --> HH:MM:SS,mmm" timecode line in ${lines.length} line(s)`, + { path: file.path, lines: lines.length }, + ); +} + +/** + * Split a payload into cues. + * + * A cue starts at a timecode line. Everything up to the next timecode belongs to + * it, which tolerates the two common real-world shapes: WebVTT allows a cue + * identifier line before the timecode, and SubRip requires a numeric index there. + * + * @returns {{cues: Array, warnings: string[], indexProblems: string[]}} + */ +export function splitCues(file, format) { + const text = file.text.replace(/^\uFEFF/, ""); + const bomOffset = file.text.length - text.length; + const matcher = format === "vtt" ? VTT_TIMECODE : SRT_TIMECODE; + const lines = []; + { + let cursor = 0; + while (cursor <= text.length) { + const newline = text.indexOf("\n", cursor); + if (newline === -1) { + lines.push({ text: text.slice(cursor), start: cursor, end: text.length }); + break; + } + lines.push({ text: text.slice(cursor, newline), start: cursor, end: newline }); + cursor = newline + 1; + } + } + + const cueStarts = []; + for (let index = 0; index < lines.length; index += 1) { + if (matcher.test(lines[index].text)) cueStarts.push(index); + } + + const cues = []; + const warnings = []; + const indexProblems = []; + const seenIndexes = new Map(); + + for (let position = 0; position < cueStarts.length; position += 1) { + const lineIndex = cueStarts[position]; + const match = matcher.exec(lines[lineIndex].text); + const isVtt = format === "vtt"; + + const start = isVtt + ? ((Number(match[1] ?? 0) * 3600 + Number(match[2]) * 60 + Number(match[3])) * 1000) + Number(match[4]) + : ((Number(match[1]) * 3600 + Number(match[2]) * 60 + Number(match[3])) * 1000) + Number(match[4].padEnd(3, "0")); + const end = isVtt + ? ((Number(match[5] ?? 0) * 3600 + Number(match[6]) * 60 + Number(match[7])) * 1000) + Number(match[8]) + : ((Number(match[5]) * 3600 + Number(match[6]) * 60 + Number(match[7])) * 1000) + Number(match[8].padEnd(3, "0")); + const settings = isVtt ? (match[9] ?? "").trim() : ""; + + // The index line (SubRip) or the cue identifier (WebVTT) sits directly above. + let indexLine = null; + if (lineIndex > 0) { + const candidate = lines[lineIndex - 1].text.trim(); + if (candidate !== "" && !TIMECODE_ANY.test(candidate) && !/^NOTE\b/.test(candidate)) indexLine = candidate; + } + + const bodyStartLine = lineIndex + 1; + const bodyEndLine = (position + 1 < cueStarts.length ? cueStarts[position + 1] : lines.length) - 1; + const bodyLines = []; + for (let cursor = bodyStartLine; cursor <= bodyEndLine && cursor < lines.length; cursor += 1) { + bodyLines.push(lines[cursor]); + } + + const rawLines = bodyLines.map((line) => line.text); + const cleaned = rawLines + .map((line) => line.replace(TAG, "").replace(CUE_SETTINGS, "").replace(/\s+/g, " ").trim()) + .filter((line) => line !== ""); + const body = cleaned.join("\n").trim(); + + let speaker = null; + let speakerSource = null; + const voice = VTT_VOICE.exec(rawLines[0] ?? ""); + if (voice) { + speaker = voice[1].trim(); + speakerSource = "webvtt-voice"; + } else { + const prefix = SPEAKER_PREFIX.exec(cleaned[0] ?? ""); + if (prefix) { + speaker = prefix[1].trim(); + speakerSource = "name-prefix"; + } + } + + const byteStart = file.charToByte(bomOffset + lines[lineIndex].start); + const byteEnd = file.charToByte(bomOffset + (bodyLines.length > 0 ? bodyLines[bodyLines.length - 1].end : lines[lineIndex].end)); + + if (indexLine !== null && /^\d+$/.test(indexLine)) { + const numeric = Number(indexLine); + if (seenIndexes.has(numeric)) { + indexProblems.push(`cue index ${numeric} appears more than once (first at line ${seenIndexes.get(numeric) + 1}, again at line ${lineIndex})`); + } else { + seenIndexes.set(numeric, lineIndex - 1); + } + } + if (!Number.isFinite(start) || !Number.isFinite(end)) { + warnings.push(`cue at line ${lineIndex + 1} has an unreadable timecode and was skipped`); + continue; + } + if (end < start) { + warnings.push(`cue at line ${lineIndex + 1} ends (${formatTimecode(end)}) before it starts (${formatTimecode(start)}); the timecodes are kept verbatim`); + } + if (body === "") { + warnings.push(`cue ${indexLine ?? position + 1} at ${formatTimecode(start)} has no text and was not anchored`); + continue; + } + + cues.push({ + index: indexLine !== null && /^\d+$/.test(indexLine) ? Number(indexLine) : position + 1, + start, + end, + settings: settings || null, + speaker, + speakerSource, + text: body, + line: lineIndex + 1, + charStart: bomOffset + lines[lineIndex].start, + charEnd: bomOffset + (bodyLines.length > 0 ? bodyLines[bodyLines.length - 1].end : lines[lineIndex].end), + byteStart, + byteEnd, + }); + } + + for (const problem of indexProblems) warnings.push(problem); + for (const note of collectNotes(lines, format)) warnings.push(note); + return { cues, warnings, indexProblems }; +} + +function collectNotes(lines, format) { + const notes = []; + if (format !== "vtt") return notes; + let noteCount = 0; + let headerFields = 0; + for (let index = 0; index < lines.length; index += 1) { + if (/^\s*NOTE\b/.test(lines[index].text)) noteCount += 1; + if (index < 40 && /^(?:Kind|Language|Region|STYLE)\s*:/i.test(lines[index].text)) headerFields += 1; + } + if (noteCount > 0) notes.push(`${noteCount} WebVTT NOTE block(s) were skipped (they are comments, not dialogue)`); + if (headerFields > 0) notes.push(`${headerFields} WebVTT header/metadata line(s) were skipped`); + if (/^\s*STYLE\b/m.test(lines.map((line) => line.text).join("\n"))) { + notes.push("the WebVTT STYLE block was skipped; only cue text was anchored"); + } + return notes; +} + +/** + * Parse a subtitle payload into a ledger-ready document. + * + * @param {import("./common.mjs").SourceFile} file + * @param {{format?: "srt"|"vtt"}} [options] + */ +export function parseSubtitle(file, options = {}) { + const detected = options.format ? { format: options.format, reasons: ["format supplied by the caller"] } : detectSubtitleFormat(file); + const { cues, warnings } = splitCues(file, detected.format); + if (cues.length === 0) { + throw new UnrecognizedFormatError( + `${file.label} looks like ${detected.format.toUpperCase()} but contains no cue with text`, + { path: file.path, format: detected.format }, + ); + } + + const records = recordsFromCharSpans( + file, + cues.map((cue) => ({ + text: cue.text, + charStart: cue.charStart, + charEnd: cue.charEnd, + kind: "cue", + label: cue.speaker ? `${cue.speaker} @ ${formatTimecode(cue.start)}` : formatTimecode(cue.start), + })), + ); + + const speakers = [...new Set(cues.map((cue) => cue.speaker).filter(Boolean))]; + const first = cues[0]; + const last = cues[cues.length - 1]; + const extraWarnings = []; + if (speakers.length === 0) { + extraWarnings.push("no speaker could be read from this subtitle: SubRip has no speaker field and no cue used a `` span or a `Name:` prefix"); + } + if (first.index !== 1) { + extraWarnings.push(`the first cue is numbered ${first.index}, not 1; the numbering was kept as-is`); + } + + return buildDocument({ + parser: "subtitle", + format: detected.format, + kind: "subtitle", + method: "local-file", + source: "subtitle", + files: [file], + records, + warnings: [...warnings, ...extraWarnings], + meta: { + cues: cues.length, + durationMs: last.end - first.start, + startsAt: formatTimecode(first.start), + endsAt: formatTimecode(last.end), + speakers, + detection: detected.reasons, + // Per-cue timing lives here so a later pass can build a timeline without + // re-parsing the file. Anchors `[k00NN:tM]` index into this array. + timecodes: cues.map((cue) => ({ + index: cue.index, + start: cue.start, + end: cue.end, + startTimecode: formatTimecode(cue.start), + endTimecode: formatTimecode(cue.end), + speaker: cue.speaker, + line: cue.line, + })), + }, + dropped: cues.some((cue) => cue.settings) + ? [{ what: "WebVTT cue settings", why: "layout hints (align/position/line), not dialogue" }] + : [], + }); +} diff --git a/src/skill/presets.mjs b/src/skill/presets.mjs new file mode 100644 index 00000000..07c9d46f --- /dev/null +++ b/src/skill/presets.mjs @@ -0,0 +1,277 @@ +/** + * Character preset registry for the Distilly engine. + * + * Node port of `tools/skill_presets.py`. The engine itself is a meta-skill: + * character presets define which prompt family and rendering defaults apply to + * a distillation target. Pure data plus small normalizers — no I/O except the + * `existsSync` probes used to keep legacy storage roots readable. + */ + +import { existsSync } from "node:fs"; +import { join } from "node:path"; + +export const COMMON_KNOWLEDGE_DIRS = ["docs", "messages", "emails"]; + +export const DEFAULT_RESEARCH_PROFILE = "budget-friendly"; + +export const CHARACTER_PRESETS = { + colleague: { + character: "colleague", + display_name: "Colleague", + identity_label: "Colleague", + gallery_category: "Colleague", + source_domain: "work", + relationship_to_user: "coworker", + is_real_person: true, + is_public_figure: false, + is_fictional: false, + command_aliases: ["/create-colleague", "/create-skill"], + knowledge_dirs: COMMON_KNOWLEDGE_DIRS, + storage_root: "skills/colleague", + prompt_bundle: { + preset: "distilly.colleague.v1", + intake: "prompts/intake.md", + work_analyzer: "prompts/work_analyzer.md", + persona_analyzer: "prompts/persona_analyzer.md", + work_builder: "prompts/work_builder.md", + persona_builder: "prompts/persona_builder.md", + merger: "prompts/merger.md", + correction_handler: "prompts/correction_handler.md", + }, + legacy_storage_root: "colleagues", + skill_name_prefix: "colleague", + legacy_type: "colleague", + }, + relationship: { + character: "relationship", + display_name: "Relationship", + identity_label: "Relationship", + gallery_category: "Relationship", + source_domain: "personal", + relationship_to_user: "relationship", + is_real_person: true, + is_public_figure: false, + is_fictional: false, + command_aliases: ["/create-ex", "/create-skill"], + knowledge_dirs: COMMON_KNOWLEDGE_DIRS, + storage_root: "skills/relationship", + prompt_bundle: { + preset: "distilly.relationship.v1", + intake: "prompts/relationship/intake.md", + work_analyzer: "prompts/work_analyzer.md", + persona_analyzer: "prompts/relationship/persona_analyzer.md", + work_builder: "prompts/work_builder.md", + persona_builder: "prompts/relationship/persona_builder.md", + merger: "prompts/relationship/merger.md", + correction_handler: "prompts/correction_handler.md", + }, + legacy_storage_root: "skills/relationship", + skill_name_prefix: "relationship", + legacy_type: "relationship", + }, + celebrity: { + character: "celebrity", + display_name: "Celebrity", + identity_label: "Celebrity", + gallery_category: "Celebrity", + source_domain: "public", + relationship_to_user: "public_figure", + is_real_person: true, + is_public_figure: true, + is_fictional: false, + command_aliases: ["/create-icon", "/create-skill"], + knowledge_dirs: [ + ...COMMON_KNOWLEDGE_DIRS, + "research/raw", + "research/merged", + "research/reviews", + "transcripts", + "subtitles", + ], + storage_root: "skills/celebrity", + prompt_bundle: { + preset: "distilly.celebrity.v1", + intake: "prompts/celebrity/intake.md", + research: "prompts/celebrity/research.md", + work_analyzer: "prompts/work_analyzer.md", + persona_analyzer: "prompts/celebrity/persona_analyzer.md", + work_builder: "prompts/work_builder.md", + persona_builder: "prompts/celebrity/persona_builder.md", + merger: "prompts/celebrity/merger.md", + correction_handler: "prompts/correction_handler.md", + }, + default_research_profile: DEFAULT_RESEARCH_PROFILE, + research_profiles: { + "budget-friendly": { + name: "budget-friendly", + display_name: "Budget Friendly", + description: + "Lean public-source distillation with compact review and lightweight validation.", + prompt_bundle: { + research: "prompts/celebrity/research.md", + persona_analyzer: "prompts/celebrity/persona_analyzer.md", + persona_builder: "prompts/celebrity/persona_builder.md", + }, + references: [], + merge_strategy: "compact", + quality_profile: "budget-friendly", + min_raw_notes: 3, + min_grounded_urls: 2, + min_primary_markers: 0, + }, + "budget-unfriendly": { + name: "budget-unfriendly", + display_name: "Budget Unfriendly", + description: + "Deep six-track research with evidence grading, synthesis review, and stricter validation.", + prompt_bundle: { + research: "prompts/celebrity/budget_unfriendly/research.md", + audit: "prompts/celebrity/budget_unfriendly/audit.md", + synthesis: "prompts/celebrity/budget_unfriendly/synthesis.md", + validation: "prompts/celebrity/budget_unfriendly/validation.md", + persona_analyzer: "prompts/celebrity/budget_unfriendly/persona_analyzer.md", + persona_builder: "prompts/celebrity/budget_unfriendly/persona_builder.md", + }, + references: [ + "references/celebrity_budget_unfriendly_framework.md", + "references/celebrity_budget_unfriendly_template.md", + ], + merge_strategy: "deep", + quality_profile: "budget-unfriendly", + min_raw_notes: 6, + min_grounded_urls: 8, + min_primary_markers: 3, + min_source_metadata_blocks: 6, + min_contradiction_bullets: 6, + min_inference_bullets: 6, + required_review_files: ["research_audit.md", "synthesis.md", "validation.md"], + }, + }, + research_tools: { + public_x_posts: "tools/research/xquik_public_posts.py", + subtitle_downloader: "tools/research/download_subtitles.sh", + subtitle_cleaner: "tools/research/srt_to_transcript.py", + research_merger: "tools/research/merge_research.py", + quality_check: "tools/research/quality_check.py", + }, + legacy_storage_root: "skills/celebrity", + skill_name_prefix: "celebrity", + legacy_type: "celebrity", + }, +}; + +export const CHARACTER_ALIASES = { + ex: "relationship", + self: "relationship", + yourself: "relationship", + icon: "celebrity", + character: "celebrity", + "fictional-character": "celebrity", + nuwa: "celebrity", +}; + +/** Normalize a character family and fall back to colleague. */ +export function normalizeCharacter(character) { + if (character === undefined || character === null || character === "") return "colleague"; + const normalized = String(character).trim().toLowerCase(); + const aliased = Object.hasOwn(CHARACTER_ALIASES, normalized) + ? CHARACTER_ALIASES[normalized] + : normalized; + return Object.hasOwn(CHARACTER_PRESETS, aliased) ? aliased : "colleague"; +} + +/** Return the preset for the given character family. */ +export function getCharacterPreset(character) { + return CHARACTER_PRESETS[normalizeCharacter(character)]; +} + +/** Normalize a research profile for the selected character family. */ +export function normalizeResearchProfile(character, researchProfile) { + const preset = getCharacterPreset(character); + const profiles = preset.research_profiles ?? {}; + if (Object.keys(profiles).length === 0) return "standard"; + if (!researchProfile) { + return preset.default_research_profile ?? DEFAULT_RESEARCH_PROFILE; + } + const normalized = String(researchProfile).trim().toLowerCase().replaceAll("_", "-"); + return Object.hasOwn(profiles, normalized) + ? normalized + : preset.default_research_profile ?? DEFAULT_RESEARCH_PROFILE; +} + +/** Return the research-profile preset for the given character family. */ +export function getResearchProfilePreset(character, researchProfile = null) { + const preset = getCharacterPreset(character); + const profiles = preset.research_profiles ?? {}; + if (Object.keys(profiles).length === 0) { + return { + name: "standard", + display_name: "Standard", + description: "Default profile for non-celebrity families.", + prompt_bundle: {}, + references: [], + merge_strategy: "compact", + quality_profile: "budget-friendly", + min_raw_notes: 0, + min_grounded_urls: 0, + min_primary_markers: 0, + }; + } + return profiles[normalizeResearchProfile(character, researchProfile)]; +} + +/** Compatibility shim for older callers that still pass a skill type. */ +export function normalizeSkillType(skillType) { + return normalizeCharacter(skillType); +} + +/** Compatibility shim for older callers that still request skill presets. */ +export function getSkillPreset(skillType) { + return getCharacterPreset(skillType); +} + +/** Return the canonical storage root for a character family. */ +export function canonicalStorageRoot(character) { + const preset = getCharacterPreset(character); + return preset.storage_root || preset.legacy_storage_root; +} + +/** Return the legacy storage root when it differs from the canonical one. */ +export function legacyStorageRoot(character) { + const preset = getCharacterPreset(character); + const legacy = preset.legacy_storage_root; + const canonical = canonicalStorageRoot(character); + if (legacy && legacy !== canonical) return legacy; + return null; +} + +/** ${HOME} expansion, matching Python's `Path.expanduser()`. */ +export function expandUser(inputPath, home) { + const homeDir = home ?? process.env.HOME ?? ""; + if (inputPath === "~") return homeDir; + if (inputPath.startsWith("~/")) return join(homeDir, inputPath.slice(2)); + return inputPath; +} + +/** Resolve the canonical write target for a character family. */ +export function resolveStorageRoot(character, baseDirArg = null) { + if (baseDirArg) return expandUser(baseDirArg); + return canonicalStorageRoot(character); +} + +/** Resolve an existing storage root while keeping legacy paths readable. */ +export function resolveExistingStorageRoot(character, slug = null, baseDirArg = null) { + if (baseDirArg) return expandUser(baseDirArg); + + const canonical = canonicalStorageRoot(character); + const legacy = legacyStorageRoot(character); + + if (slug) { + if (existsSync(join(canonical, slug))) return canonical; + if (legacy && existsSync(join(legacy, slug))) return legacy; + } + + if (existsSync(canonical)) return canonical; + if (legacy && existsSync(legacy)) return legacy; + return canonical; +} diff --git a/src/skill/schema.mjs b/src/skill/schema.mjs new file mode 100644 index 00000000..96b42c14 --- /dev/null +++ b/src/skill/schema.mjs @@ -0,0 +1,504 @@ +/** + * Shared Distilly engine schema and generated artifact metadata. + * + * Node port of `tools/skill_schema.py`. Everything here is byte-compatible with + * the Python original: + * - `jsonDumps()` == `json.dumps(value, ensure_ascii=False, indent=2)` (no trailing newline), + * - key insertion order follows the Python dict operations line by line, + * - `nowIso()` uses Python's `datetime.now(timezone.utc).isoformat()` layout + * (`…+00:00`, microsecond precision). + * + * Byte equality is verified by `scripts/parity.mjs`; see + * `docs/evidence/pr-01-node-core.md`. + */ + +import { createHash } from "node:crypto"; +import { existsSync, readFileSync, realpathSync } from "node:fs"; +import { basename, dirname, isAbsolute, join, resolve } from "node:path"; + +import { + getCharacterPreset, + getResearchProfilePreset, + normalizeCharacter, + normalizeResearchProfile, +} from "./presets.mjs"; + +export const SCHEMA_VERSION = "3"; +export const PORTABLE_SLUG_MAX_LENGTH = 40; +export const PRIMARY_ARTIFACTS = [ + "SKILL.md", + "work.md", + "persona.md", + "work_skill.md", + "persona_skill.md", + "manifest.json", +]; +export const ARTIFACT_NAME_FILES = { + combined_name: "SKILL.md", + work_name: "work_skill.md", + persona_name: "persona_skill.md", +}; + +const FRONTMATTER_RE = /^---\r?\n([\s\S]*?)\r?\n---\r?\n?/; +const FRONTMATTER_NAME_RE = /^name:\s*(.+?)\s*$/m; +const WINDOWS_RESERVED_NAME_RE = /^(?:con|prn|aux|nul|com[1-9]|lpt[1-9])(?:\.|$)/i; + +/** `json.dumps(value, ensure_ascii=False, indent=2)` — separators and layout match Python. */ +export function jsonDumps(value) { + return JSON.stringify(value, null, 2); +} + +export function sha256Hex(text) { + return createHash("sha256").update(Buffer.from(text, "utf8")).digest("hex"); +} + +/** + * Current UTC time in Python's `datetime.now(timezone.utc).isoformat()` format. + * `DISTILLY_PARITY_NOW` freezes it so `scripts/parity.mjs` can compare bytes + * against the pinned Python implementation; it is unset in normal use. + */ +export function nowIso() { + const frozen = process.env.DISTILLY_PARITY_NOW; + if (frozen) return frozen; + return `${new Date().toISOString().slice(0, 23)}000+00:00`; +} + +/** Python's `dict.get(key, default)` — an explicit `null` is a value, not a miss. */ +function pyGet(object, key, fallback) { + if (!isObject(object) || !Object.hasOwn(object, key)) return fallback; + return object[key]; +} + +/** Python's `dict.setdefault(key, value)` — an existing key wins, even when null. */ +function pySetDefault(object, key, value) { + if (!Object.hasOwn(object, key)) object[key] = value; + return object[key]; +} + +/** Python truthiness for the values that appear in metadata. */ +function pyTruthy(value) { + if (value === undefined || value === null || value === false) return false; + if (value === 0 || value === "") return false; + if (Array.isArray(value)) return value.length > 0; + if (isObject(value)) return Object.keys(value).length > 0; + return true; +} + +function isObject(value) { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +/** Python's `Path.name` for the shapes this code sees. */ +function pathName(inputPath) { + const text = String(inputPath); + if (text === "." || text === "./") return ""; + return basename(text); +} + +/** Python's `Path.resolve()` (strict=False): realpath the existing prefix. */ +export function resolveRealPath(inputPath) { + let current = resolve(inputPath); + const missing = []; + for (;;) { + try { + const real = realpathSync(current); + return missing.length > 0 ? join(real, ...missing.reverse()) : real; + } catch { + const parent = dirname(current); + if (parent === current) { + return missing.length > 0 ? join(current, ...missing.reverse()) : current; + } + missing.push(basename(current)); + current = parent; + } + } +} + +/** Extract gallery tags from the legacy tags structure. */ +export function flattenLegacyTags(meta) { + const classification = pyGet(meta, "classification", {}); + const tags = pyGet(classification, "tags", undefined); + if (Array.isArray(tags) && tags.length > 0) return tags; + + const legacyTags = pyGet(meta, "tags", {}); + if (Array.isArray(legacyTags)) { + return legacyTags.filter((item) => typeof item === "string" && item); + } + + const results = []; + for (const key of ["personality", "culture"]) { + const value = pyGet(legacyTags, key, []); + if (Array.isArray(value)) { + results.push(...value.filter((item) => typeof item === "string" && item)); + } + } + return results; +} + +/** Resolve the active character family from new or legacy fields. */ +export function resolveCharacter(meta, explicitCharacter = null) { + const generation = pyGet(meta, "generation", {}); + return normalizeCharacter( + explicitCharacter || + pyGet(meta, "character", undefined) || + pyGet(meta, "type", undefined) || + pyGet(generation, "character", undefined), + ); +} + +/** Resolve the active research profile for the selected character family. */ +export function resolveResearchProfile(meta, character, explicitResearchProfile = null) { + const generation = pyGet(meta, "generation", {}); + const engine = pyGet(meta, "engine", {}); + return normalizeResearchProfile( + character, + explicitResearchProfile || + pyGet(meta, "research_profile", undefined) || + pyGet(generation, "research_profile", undefined) || + pyGet(engine, "research_profile", undefined), + ); +} + +/** Build a human-readable identity string from metadata. */ +export function buildIdentityString(meta) { + const preset = getCharacterPreset(pyGet(meta, "character", undefined)); + const profile = pyGet(meta, "profile", {}); + + if (typeof profile === "string") return profile.trim() || preset.identity_label; + if (!isObject(profile)) return preset.identity_label; + + const parts = []; + for (const key of ["company", "level", "role", "occupation", "identity", "specialty", "known_for"]) { + const value = pyGet(profile, key, ""); + if (pyTruthy(value)) parts.push(String(value)); + } + + let identity = parts.length > 0 ? parts.join(" ") : preset.identity_label; + + const mbti = pyGet(profile, "mbti", ""); + if (pyTruthy(mbti)) identity += `, MBTI ${mbti}`; + + return identity; +} + +/** Convert current or legacy text into a deterministic portable command slug. */ +export function normalizeCommandSlug(value) { + const asciiValue = String(value) + .normalize("NFKD") + .replace(/[^\x00-\x7f]/g, "") + .toLowerCase(); + let slug = asciiValue.replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, ""); + if (!slug) { + const digest = sha256Hex(String(value)).slice(0, 8); + slug = `person-${digest}`; + } + return slug.slice(0, PORTABLE_SLUG_MAX_LENGTH).replace(/-+$/, ""); +} + +/** Accept one safe current or legacy filesystem segment. */ +export function validatePathSegment(value, label = "path segment") { + const text = typeof value === "string" ? value : ""; + const codePoints = [...text].length; + const unsafeCharacter = [...text].some((character) => { + const code = character.codePointAt(0); + return '/\\:<>"|?*'.includes(character) || code < 32 || code === 127; + }); + if ( + text === "" || + text === "." || + text === ".." || + codePoints > 255 || + text.endsWith(".") || + text.endsWith(" ") || + WINDOWS_RESERVED_NAME_RE.test(text) || + unsafeCharacter + ) { + throw new Error(`${label} must be one safe path segment`); + } + return text; +} + +/** Resolve a safe direct child and reject symlink escapes from its base. */ +export function resolveContainedChild(baseDir, segment, label = "path segment") { + const child = join(baseDir, validatePathSegment(segment, label)); + const childRoot = resolveRealPath(child); + const baseRoot = resolveRealPath(baseDir); + if (childRoot === baseRoot) { + throw new Error(`${label} must resolve to a direct child`); + } + const relative = childRoot.startsWith(baseRoot.endsWith("/") ? baseRoot : `${baseRoot}/`); + if (!relative && childRoot !== baseRoot) { + throw new Error(`${label} resolves outside its base directory`); + } + return child; +} + +/** Read generated frontmatter names that predate artifacts metadata. */ +export function readExistingArtifactNames(skillDir) { + const names = {}; + for (const [key, filename] of Object.entries(ARTIFACT_NAME_FILES)) { + const artifactPath = join(skillDir, filename); + if (!existsSync(artifactPath)) continue; + const frontmatter = FRONTMATTER_RE.exec(readFileSync(artifactPath, "utf8")); + if (!frontmatter) continue; + const name = FRONTMATTER_NAME_RE.exec(frontmatter[1]); + if (name) names[key] = name[1].trim(); + } + return names; +} + +/** Generate artifact names from the selected character preset. */ +export function buildArtifactNames(meta) { + const slug = meta.slug; + const commandSlug = normalizeCommandSlug(slug); + const commandBase = `${meta.character}-${commandSlug}`; + return { + combined_skill: "SKILL.md", + work_skill: "work_skill.md", + persona_skill: "persona_skill.md", + work_doc: "work.md", + persona_doc: "persona.md", + manifest: "manifest.json", + combined_name: commandBase, + work_name: `${commandBase}-work`, + persona_name: `${commandBase}-persona`, + combined_command: commandBase, + work_command: `${commandBase}-work`, + persona_command: `${commandBase}-persona`, + }; +} + +/** Mirror new schema fields back to the legacy top-level structure. */ +export function syncLegacyFields(meta) { + const lifecycle = pySetDefault(meta, "lifecycle", {}); + const generation = pySetDefault(meta, "generation", {}); + + meta.name = pyTruthy(pyGet(meta, "name", undefined)) + ? meta.name + : pyTruthy(pyGet(meta, "display_name", undefined)) + ? meta.display_name + : pyGet(meta, "slug", ""); + meta.display_name = pyTruthy(pyGet(meta, "display_name", undefined)) + ? meta.display_name + : meta.name; + + meta.created_at = pyGet(lifecycle, "created_at", pyGet(meta, "created_at", nowIso())); + meta.updated_at = pyGet(lifecycle, "updated_at", pyGet(meta, "updated_at", meta.created_at)); + meta.version = pyGet(lifecycle, "version", pyGet(meta, "version", "v1")); + meta.corrections_count = pyGet( + generation, + "corrections_count", + pyGet(meta, "corrections_count", 0), + ); + + meta.type = pyTruthy(pyGet(meta, "type", undefined)) + ? meta.type + : pyTruthy(pyGet(meta, "character", undefined)) + ? meta.character + : "colleague"; + pySetDefault(generation, "character", meta.character); + pySetDefault(generation, "preset", meta.preset); + + lifecycle.created_at = meta.created_at; + lifecycle.updated_at = meta.updated_at; + lifecycle.version = meta.version; + generation.corrections_count = meta.corrections_count; + return meta; +} + +/** Upgrade legacy metadata to the Distilly engine schema. */ +export function enrichSkillMeta(meta, slug, character = null) { + const result = structuredClone(meta); + const resolvedCharacter = resolveCharacter(result, character); + const preset = getCharacterPreset(resolvedCharacter); + const resolvedResearchProfile = resolveResearchProfile(result, resolvedCharacter); + const researchProfile = getResearchProfilePreset(resolvedCharacter, resolvedResearchProfile); + + const lifecycle = pySetDefault(result, "lifecycle", {}); + const generation = pySetDefault(result, "generation", {}); + const classification = pySetDefault(result, "classification", {}); + const sourceContext = pySetDefault(result, "source_context", {}); + const engine = pySetDefault(result, "engine", {}); + + result.schema_version = SCHEMA_VERSION; + result.slug = slug; + result.kind = pyTruthy(pyGet(result, "kind", undefined)) ? result.kind : "meta-skill"; + result.character = resolvedCharacter; + result.research_profile = resolvedResearchProfile; + pySetDefault(result, "subtype", null); + result.preset = pyTruthy(pyGet(result, "preset", undefined)) + ? result.preset + : pyTruthy(pyGet(generation, "preset", undefined)) + ? generation.preset + : preset.prompt_bundle.preset; + + const displayName = pyTruthy(pyGet(result, "display_name", undefined)) + ? result.display_name + : pyTruthy(pyGet(result, "name", undefined)) + ? result.name + : slug; + result.display_name = displayName; + result.name = pyTruthy(pyGet(result, "name", undefined)) ? result.name : displayName; + result.id = pyTruthy(pyGet(result, "id", undefined)) + ? result.id + : `${result.kind}.${resolvedCharacter}.${slug}`; + + const createdAt = + pyGet(result, "created_at", undefined) || pyGet(lifecycle, "created_at", undefined) || nowIso(); + const updatedAt = + pyGet(result, "updated_at", undefined) || pyGet(lifecycle, "updated_at", undefined) || createdAt; + const version = pyGet(result, "version", undefined) || pyGet(lifecycle, "version", undefined) || "v1"; + const correctionsCount = pyGet( + result, + "corrections_count", + pyGet(generation, "corrections_count", 0), + ); + + pySetDefault(sourceContext, "domain", preset.source_domain); + pySetDefault(sourceContext, "relationship_to_user", preset.relationship_to_user); + pySetDefault(sourceContext, "is_real_person", preset.is_real_person); + pySetDefault(sourceContext, "is_public_figure", preset.is_public_figure); + pySetDefault(sourceContext, "is_fictional", preset.is_fictional); + + pySetDefault(classification, "gallery_category", preset.gallery_category); + pySetDefault(classification, "tags", flattenLegacyTags(result)); + pySetDefault(classification, "language", "en"); + + const canonicalArtifacts = buildArtifactNames(result); + result.artifacts = { + ...canonicalArtifacts, + ...pyGet(result, "artifacts", {}), + combined_command: canonicalArtifacts.combined_command, + work_command: canonicalArtifacts.work_command, + persona_command: canonicalArtifacts.persona_command, + }; + + pySetDefault(engine, "name", "distilly"); + pySetDefault(engine, "kind", "meta-skill"); + pySetDefault(engine, "character", resolvedCharacter); + pySetDefault(engine, "research_profile", resolvedResearchProfile); + pySetDefault(engine, "preset", result.preset); + pySetDefault(engine, "prompt_bundle", preset.prompt_bundle); + pySetDefault(engine, "research_profile_bundle", researchProfile.prompt_bundle ?? {}); + pySetDefault(engine, "research_profile_references", researchProfile.references ?? []); + pySetDefault(engine, "merge_strategy", researchProfile.merge_strategy ?? "compact"); + pySetDefault(engine, "quality_profile", researchProfile.quality_profile ?? "budget-friendly"); + pySetDefault(engine, "knowledge_dirs", preset.knowledge_dirs ?? []); + pySetDefault(engine, "storage_root", preset.storage_root ?? preset.legacy_storage_root); + if (pyTruthy(preset.research_tools)) { + pySetDefault(engine, "research_tools", preset.research_tools); + } + + pySetDefault(generation, "engine", "distilly"); + pySetDefault(generation, "character", resolvedCharacter); + pySetDefault(generation, "research_profile", resolvedResearchProfile); + pySetDefault(generation, "preset", result.preset); + pySetDefault(generation, "prompt_bundle", preset.prompt_bundle); + pySetDefault(generation, "research_profile_bundle", researchProfile.prompt_bundle ?? {}); + pySetDefault(generation, "research_profile_references", researchProfile.references ?? []); + pySetDefault(generation, "merge_strategy", researchProfile.merge_strategy ?? "compact"); + pySetDefault(generation, "quality_profile", researchProfile.quality_profile ?? "budget-friendly"); + pySetDefault(generation, "knowledge_dirs", preset.knowledge_dirs ?? []); + pySetDefault(generation, "storage_root", preset.storage_root ?? preset.legacy_storage_root); + if (pyTruthy(preset.research_tools)) { + pySetDefault(generation, "research_tools", preset.research_tools); + } + pySetDefault(generation, "created_from", pyGet(result, "knowledge_sources", [])); + generation.corrections_count = correctionsCount; + + pySetDefault(lifecycle, "status", "active"); + lifecycle.created_at = createdAt; + lifecycle.updated_at = updatedAt; + lifecycle.version = version; + + result.compat = { + legacy_command: preset.command_aliases[0], + legacy_storage_root: preset.legacy_storage_root, + legacy_type: preset.legacy_type, + ...pyGet(result, "compat", {}), + }; + result.type = pyTruthy(pyGet(result, "type", undefined)) ? result.type : preset.legacy_type; + + if (!pyTruthy(pyGet(result, "summary", undefined))) { + const identity = buildIdentityString(result); + result.summary = identity ? `${displayName}, ${identity}` : displayName; + } + + return syncLegacyFields(result); +} + +/** Enrich stored metadata while preserving names from legacy artifacts. */ +export function enrichExistingSkillMeta(meta, skillDir, character = null) { + const prepared = structuredClone(meta); + const artifactMeta = pyGet(prepared, "artifacts", undefined); + const artifacts = isObject(artifactMeta) ? { ...artifactMeta } : {}; + for (const [key, name] of Object.entries(readExistingArtifactNames(skillDir))) { + pySetDefault(artifacts, key, name); + } + if (Object.keys(artifacts).length > 0) prepared.artifacts = artifacts; + return enrichSkillMeta(prepared, pathName(skillDir), character); +} + +/** Build a manifest consumable by install and gallery flows. */ +export function buildManifest(meta) { + const artifacts = meta.artifacts; + const engine = meta.engine; + return { + manifest_version: "1", + id: meta.id, + kind: meta.kind, + character: meta.character, + research_profile: pyGet(meta, "research_profile", "standard"), + preset: meta.preset, + display_name: meta.display_name, + entrypoints: { + default: artifacts.combined_skill, + work: artifacts.work_skill, + persona: artifacts.persona_skill, + }, + artifacts: [ + artifacts.combined_skill, + artifacts.work_doc, + artifacts.persona_doc, + "meta.json", + artifacts.manifest, + ], + capabilities: ["persona", "work"], + engine, + toolchain: { + prompt_bundle: pyGet(engine, "prompt_bundle", {}), + research_profile: pyGet(engine, "research_profile", "standard"), + research_profile_bundle: pyGet(engine, "research_profile_bundle", {}), + research_profile_references: pyGet(engine, "research_profile_references", []), + merge_strategy: pyGet(engine, "merge_strategy", "compact"), + quality_profile: pyGet(engine, "quality_profile", "budget-friendly"), + research_tools: pyGet(engine, "research_tools", {}), + knowledge_dirs: pyGet(engine, "knowledge_dirs", []), + }, + install: { + compatible_runtimes: [ + "claude-code", + "openclaw", + "hermes", + "codex", + "deepseek-harness", + "grok-build", + "pi", + "opencode", + ], + min_schema_version: SCHEMA_VERSION, + installers: { + "claude-code": "tools/install_claude_generated_skill.py", + openclaw: "tools/install_openclaw_generated_skill.py", + codex: "tools/install_codex_generated_skill.py", + }, + slash_commands: { + default: artifacts.combined_command, + work: artifacts.work_command, + persona: artifacts.persona_command, + }, + }, + }; +} + +export { isAbsolute }; diff --git a/src/skill/slug.mjs b/src/skill/slug.mjs new file mode 100644 index 00000000..f90eb016 --- /dev/null +++ b/src/skill/slug.mjs @@ -0,0 +1,114 @@ +/** + * Pinyin-backed slug resolution. + * + * `pypinyin` was the only optional Python dependency of the writer. The Node + * core ships a derived table instead: `assets/pinyin.json` (built from the + * Unicode Unihan database by `scripts/generate-pinyin.mjs`). + * + * Discipline (CONTRACT §3): when the table is missing, or when a Han character + * is not covered, the slug is **not** guessed — the caller gets + * `SlugResolutionError` telling the user to pass `--slug` explicitly. The old + * Python fallback silently produced `person-`; that is exactly the + * "silent junk" this port refuses to emit. + */ + +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; + +export const PINYIN_ASSET_URL = new URL("../../assets/pinyin.json", import.meta.url); + +/** Raised when a slug cannot be derived without guessing. */ +export class SlugResolutionError extends Error { + constructor(message, { character = null } = {}) { + super(message); + this.name = "SlugResolutionError"; + this.code = "slug-unresolved"; + this.character = character; + this.remedy = + "pass --slug explicitly (中文名请显式传 --slug；例如 --name \"周奇墨\" --slug zhou-qimo)"; + } +} + +const HAN_RANGES = [ + [0x3400, 0x4dbf], + [0x4e00, 0x9fff], + [0xf900, 0xfaff], + [0x20000, 0x2fa1f], +]; + +export function isHanCharacter(character) { + const code = character.codePointAt(0); + return HAN_RANGES.some(([start, end]) => code >= start && code <= end); +} + +/** True when the text contains at least one Han character. */ +export function containsHan(text) { + return [...text].some(isHanCharacter); +} + +let cachedTable; +let cachedTableLoaded = false; + +/** Load `assets/pinyin.json` once; `null` when the asset is absent. */ +export function loadPinyinTable({ path = fileURLToPath(PINYIN_ASSET_URL) } = {}) { + if (cachedTableLoaded) return cachedTable; + cachedTableLoaded = true; + try { + const parsed = JSON.parse(readFileSync(path, "utf8")); + cachedTable = parsed?.characters ?? null; + } catch { + cachedTable = null; + } + return cachedTable; +} + +/** Reset the memoized table (tests, and `--pinyin ` overrides). */ +export function resetPinyinTable() { + cachedTable = undefined; + cachedTableLoaded = false; +} + +/** + * Convert one Unihan reading to the shape `pypinyin.lazy_pinyin` emits: + * tone marks stripped, `ü` written as `v` (吕 → `lv`, not `lu`). + */ +export function readingToSyllable(reading) { + return reading.normalize("NFD").replace(/\u0308/g, "v"); +} + +/** + * Syllables for a display name, Han characters resolved through the table. + * A missing table is only an error when the name actually contains Han text. + * @returns {string[]} + */ +export function pinyinSyllables(name, { table = loadPinyinTable() } = {}) { + const text = String(name); + const syllables = []; + for (const character of text) { + if (!isHanCharacter(character)) { + syllables.push(character); + continue; + } + if (!table) { + throw new SlugResolutionError( + `cannot derive a slug from "${text}": the pinyin table assets/pinyin.json is missing`, + { character }, + ); + } + const reading = table[character]; + if (!reading) { + throw new SlugResolutionError( + `cannot derive a slug from "${text}": no pinyin reading for "${character}" in assets/pinyin.json`, + { character }, + ); + } + syllables.push(readingToSyllable(reading)); + } + return syllables; +} + +/** Injectable for tests: `setSlugifyTable(table)` replaces the memoized asset. */ +export function setSlugifyTable(table) { + cachedTable = table; + cachedTableLoaded = true; +} diff --git a/src/skill/versions.mjs b/src/skill/versions.mjs new file mode 100644 index 00000000..0a8c3ddc --- /dev/null +++ b/src/skill/versions.mjs @@ -0,0 +1,193 @@ +/** + * Skill version manager. + * + * Node port of `tools/version_manager.py`: archives and restores generated + * artifacts while keeping the legacy colleague storage layout readable. + * Messages match the Python original byte for byte (verified by + * `scripts/parity.mjs`). + */ + +import { copyFileSync, existsSync, mkdirSync, readdirSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +import { normalizeCharacter, resolveExistingStorageRoot } from "./presets.mjs"; +import { + PRIMARY_ARTIFACTS, + enrichExistingSkillMeta, + jsonDumps, + nowIso, + resolveContainedChild, + syncLegacyFields, + validatePathSegment, +} from "./schema.mjs"; + +export const MAX_VERSIONS = 10; + +/** Resolve the storage root for the selected character family. */ +export function resolveBaseDir(baseDirArg, character) { + return resolveExistingStorageRoot(character, null, baseDirArg); +} + +/** Resolve the versions directory without following a symlink outside the skill. */ +export function resolveVersionsDir(skillDir) { + return resolveContainedChild(skillDir, "versions", "versions directory"); +} + +/** `YYYY-MM-DD HH:MM` in UTC, matching `datetime.fromtimestamp(mtime, tz=utc).strftime`. */ +function formatArchivedAt(mtimeMs) { + const iso = new Date(mtimeMs).toISOString(); + return `${iso.slice(0, 10)} ${iso.slice(11, 16)}`; +} + +/** List all archived versions for a skill directory. */ +export function listVersions(skillDir) { + let versionsDir; + try { + versionsDir = resolveVersionsDir(skillDir); + } catch (error) { + process.stderr.write(`error: ${error.message}\n`); + return []; + } + if (!existsSync(versionsDir)) return []; + + const versions = []; + for (const entry of readdirSync(versionsDir).sort()) { + const versionDir = join(versionsDir, entry); + if (!statSync(versionDir).isDirectory()) continue; + + const archivedAt = formatArchivedAt(statSync(versionDir).mtimeMs); + const files = readdirSync(versionDir).filter((name) => statSync(join(versionDir, name)).isFile()); + versions.push({ + version: entry, + archived_at: archivedAt, + files, + path: versionDir, + }); + } + + return versions; +} + +/** Copy the current generated artifacts into a backup directory. */ +export function backupArtifacts(skillDir, backupDir) { + mkdirSync(backupDir, { recursive: true }); + for (const filename of PRIMARY_ARTIFACTS) { + const source = join(skillDir, filename); + if (existsSync(source)) copyFileSync(source, join(backupDir, filename)); + } +} + +/** Restore a previously archived version. */ +export function rollback(skillDir, targetVersion) { + let versionsDir; + let versionDir; + try { + versionsDir = resolveVersionsDir(skillDir); + versionDir = resolveContainedChild(versionsDir, targetVersion, "version"); + } catch (error) { + process.stderr.write(`error: ${error.message}\n`); + return false; + } + if (!existsSync(versionDir)) { + process.stderr.write(`error: version does not exist: ${targetVersion}\n`); + return false; + } + + const metaPath = join(skillDir, "meta.json"); + if (!existsSync(metaPath)) { + process.stderr.write("error: meta.json is required for rollback\n"); + return false; + } + + const meta = enrichExistingSkillMeta(JSON.parse(readFileSyncText(metaPath)), skillDir); + const currentVersion = meta.version ?? "v?"; + let backupDir; + try { + backupDir = resolveContainedChild( + versionsDir, + `${validatePathSegment(String(currentVersion), "current version")}_before_rollback`, + "rollback backup version", + ); + } catch (error) { + process.stderr.write(`error: ${error.message}\n`); + return false; + } + backupArtifacts(skillDir, backupDir); + + const restoredFiles = []; + for (const filename of PRIMARY_ARTIFACTS) { + const source = join(versionDir, filename); + if (existsSync(source)) { + copyFileSync(source, join(skillDir, filename)); + restoredFiles.push(filename); + } + } + + meta.lifecycle.version = `${targetVersion}_restored`; + meta.lifecycle.updated_at = nowIso(); + meta.rollback_from = currentVersion; + writeFileSync(metaPath, jsonDumps(syncLegacyFields(meta)), "utf8"); + + process.stdout.write(`rolled back to ${targetVersion}: ${restoredFiles.join(", ")}\n`); + return true; +} + +/** Archive the current generated artifacts under versions//. */ +export function backupCurrentVersion(skillDir) { + const metaPath = join(skillDir, "meta.json"); + if (!existsSync(metaPath)) { + process.stderr.write("error: meta.json is required to determine the current version\n"); + return false; + } + + const meta = enrichExistingSkillMeta(JSON.parse(readFileSyncText(metaPath)), skillDir); + const currentVersion = meta.version ?? "v1"; + let backupDir; + try { + const versionsDir = resolveVersionsDir(skillDir); + backupDir = resolveContainedChild( + versionsDir, + validatePathSegment(String(currentVersion), "current version"), + "current version", + ); + } catch (error) { + process.stderr.write(`error: ${error.message}\n`); + return false; + } + backupArtifacts(skillDir, backupDir); + process.stdout.write(`archived version ${currentVersion}\n`); + return true; +} + +/** Remove archived versions beyond the retention limit. */ +export function cleanupOldVersions(skillDir, maxVersions = MAX_VERSIONS) { + let versionsDir; + try { + versionsDir = resolveVersionsDir(skillDir); + } catch (error) { + process.stderr.write(`error: ${error.message}\n`); + return false; + } + if (!existsSync(versionsDir)) return true; + + const versionDirs = readdirSync(versionsDir) + .map((entry) => join(versionsDir, entry)) + .filter((entry) => statSync(entry).isDirectory()) + .sort((left, right) => statSync(left).mtimeMs - statSync(right).mtimeMs); + const toDelete = versionDirs.length > maxVersions ? versionDirs.slice(0, versionDirs.length - maxVersions) : []; + + for (const oldDir of toDelete) { + rmSync(oldDir, { recursive: true, force: true }); + process.stdout.write(`deleted old version: ${oldDir.split("/").pop()}\n`); + } + return true; +} + +function readFileSyncText(path) { + // Local import indirection keeps the module's import list flat. + return require_readFileSync(path); +} + +import { readFileSync as require_readFileSync } from "node:fs"; + +export { normalizeCharacter, resolveExistingStorageRoot }; diff --git a/src/skill/writer.mjs b/src/skill/writer.mjs new file mode 100644 index 00000000..4bfc1e0b --- /dev/null +++ b/src/skill/writer.mjs @@ -0,0 +1,400 @@ +/** + * Skill artifact writer. + * + * Node port of `tools/skill_writer.py`: writes the six primary artifacts plus + * `meta.json` for the Distilly engine while preserving backward compatibility + * with the original colleague-centric layout. Output bytes are identical to the + * Python original — see `scripts/parity.mjs`. + */ + +import { copyFileSync, existsSync, mkdirSync, readFileSync, readdirSync, statSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +import { + getCharacterPreset, + resolveExistingStorageRoot, + resolveStorageRoot, +} from "./presets.mjs"; +import { + PRIMARY_ARTIFACTS, + buildIdentityString, + buildManifest, + enrichExistingSkillMeta, + enrichSkillMeta, + jsonDumps, + normalizeCommandSlug, + nowIso, + resolveContainedChild, + syncLegacyFields, + validatePathSegment, +} from "./schema.mjs"; +import { pinyinSyllables } from "./slug.mjs"; + +export const SKILL_MD_TEMPLATE_EN = __TEMPLATE_EN__; + +export const SKILL_MD_TEMPLATE_ZH = __TEMPLATE_ZH__; + +export const MAX_SLUG_LENGTH = 40; +const SLUG_PATTERN = /^[a-z0-9]+(?:-[a-z0-9]+)*$/; + +/** Require a safe kebab-case slug before using it in paths or skill names. */ +export function validateSlug(slug) { + if (String(slug).length > MAX_SLUG_LENGTH || !SLUG_PATTERN.test(String(slug))) { + throw new Error( + `slug must be 1-${MAX_SLUG_LENGTH} lowercase letters/digits in kebab-case`, + ); + } + return slug; +} + +/** + * Convert a human-readable name into a stable slug. + * + * Uses the Unihan-derived pinyin table (`src/skill/slug.mjs`); when the table + * is missing or a character is uncovered it throws `SlugResolutionError` and + * asks for an explicit `--slug` instead of emitting a junk slug. + */ +export function slugify(name, options = {}) { + const candidate = pinyinSyllables(name, options).join("-"); + return normalizeCommandSlug(candidate); +} + +/** Return the preferred language code for rendered artifacts. */ +export function languageCode(meta) { + const classification = meta?.classification ?? {}; + return String(meta?.language || classification.language || "en").toLowerCase(); +} + +/** Return whether artifact chrome should be rendered in Chinese. */ +export function prefersChinese(meta) { + return languageCode(meta).startsWith("zh"); +} + +/** Render the combined SKILL.md file from normalized metadata. */ +export function renderCombinedSkill(meta, workContent, personaContent) { + const artifacts = meta.artifacts; + const identity = buildIdentityString(meta); + const description = + meta.summary || (identity ? `${meta.display_name}, ${identity}` : meta.display_name); + const template = prefersChinese(meta) ? SKILL_MD_TEMPLATE_ZH : SKILL_MD_TEMPLATE_EN; + + const values = { + combined_name: artifacts.combined_name, + description, + display_name: meta.display_name, + identity, + work_content: workContent, + persona_content: personaContent, + }; + return template.replace( + /\{(combined_name|description|display_name|identity|work_content|persona_content)\}/g, + (_, key) => values[key], + ); +} + +const PERSONA_HANDOFF_PATTERNS = [ + /如果被问到职责范围外的问题，以该同事的方式回应（参见 Persona 部分）。\s*/g, + /If (?:you are )?asked (?:a question )?outside (?:your|the) (?:recorded )?responsibilities[^.\n]*Persona[^.\n]*\.\s*/gi, +]; + +export const WORK_ONLY_FALLBACK_ZH = __FALLBACK_ZH__; + +export const WORK_ONLY_FALLBACK_EN = __FALLBACK_EN__; + +/** Copy Work text for the Work-only skill, without a Persona handoff. */ +export function workOnlyContent(workContent, { chinese }) { + let text = workContent; + for (const pattern of PERSONA_HANDOFF_PATTERNS) { + pattern.lastIndex = 0; + text = text.replace(pattern, ""); + } + text = text.replace(/\s+$/, ""); + const fallback = chinese ? WORK_ONLY_FALLBACK_ZH : WORK_ONLY_FALLBACK_EN; + if (!text.includes(fallback)) { + text = text ? `${text}\n\n${fallback}` : fallback; + } + return text; +} + +/** Render the work-only skill artifact. */ +export function renderWorkSkill(meta, workContent) { + const artifacts = meta.artifacts; + const chinese = prefersChinese(meta); + const description = chinese + ? `${meta.display_name} 的工作能力（仅 Work，无 Persona）` + : `${meta.display_name} work capability only (without persona)`; + const body = workOnlyContent(workContent, { chinese }); + return `---\nname: ${artifacts.work_name}\ndescription: ${description}\nuser-invocable: true\n---\n\n${body}\n`; +} + +/** Render the persona-only skill artifact. */ +export function renderPersonaSkill(meta, personaContent) { + const artifacts = meta.artifacts; + const description = prefersChinese(meta) + ? `${meta.display_name} 的人物性格（仅 Persona，无工作能力）` + : `${meta.display_name} persona only (without work capability)`; + return `---\nname: ${artifacts.persona_name}\ndescription: ${description}\nuser-invocable: true\n---\n\n${personaContent}\n`; +} + +/** Write all generated artifacts for a skill version. */ +export function writeArtifacts(skillDir, meta, workContent, personaContent) { + const artifacts = meta.artifacts; + const manifest = buildManifest(meta); + + writeFileSync(join(skillDir, artifacts.work_doc), workContent, "utf8"); + writeFileSync(join(skillDir, artifacts.persona_doc), personaContent, "utf8"); + writeFileSync( + join(skillDir, artifacts.combined_skill), + renderCombinedSkill(meta, workContent, personaContent), + "utf8", + ); + writeFileSync(join(skillDir, artifacts.work_skill), renderWorkSkill(meta, workContent), "utf8"); + writeFileSync( + join(skillDir, artifacts.persona_skill), + renderPersonaSkill(meta, personaContent), + "utf8", + ); + writeFileSync(join(skillDir, artifacts.manifest), jsonDumps(manifest), "utf8"); + writeFileSync(join(skillDir, "meta.json"), jsonDumps(syncLegacyFields(meta)), "utf8"); +} + +/** Create a new skill directory with normalized metadata. */ +export function createSkill(baseDir, slug, meta, workContent, personaContent) { + const safeSlug = validateSlug(slug); + const normalizedMeta = enrichSkillMeta(meta, safeSlug, meta?.character ?? null); + const preset = getCharacterPreset(normalizedMeta.character); + const skillDir = join(baseDir, safeSlug); + mkdirSync(skillDir, { recursive: true }); + + mkdirSync(join(skillDir, "versions"), { recursive: true }); + for (const relativePath of preset.knowledge_dirs ?? ["docs", "messages", "emails"]) { + mkdirSync(join(skillDir, "knowledge", relativePath), { recursive: true }); + } + + normalizedMeta.lifecycle.created_at = normalizedMeta.created_at ?? nowIso(); + normalizedMeta.lifecycle.updated_at = normalizedMeta.lifecycle.created_at; + normalizedMeta.lifecycle.version = "v1"; + normalizedMeta.generation.corrections_count = normalizedMeta.corrections_count ?? 0; + syncLegacyFields(normalizedMeta); + + writeArtifacts(skillDir, normalizedMeta, workContent, personaContent); + return skillDir; +} + +/** Copy the current artifact set into versions//. */ +export function backupCurrentArtifacts(skillDir, versionName) { + const versionsDir = resolveContainedChild(skillDir, "versions", "versions directory"); + const versionDir = resolveContainedChild( + versionsDir, + validatePathSegment(String(versionName), "version"), + "version", + ); + mkdirSync(versionDir, { recursive: true }); + + for (const filename of PRIMARY_ARTIFACTS) { + const source = join(skillDir, filename); + if (existsSync(source)) copyFileSync(source, join(versionDir, filename)); + } +} + +function sectionMatches(text) { + const pattern = /^##\s+.+$/gm; + const matches = []; + let match; + while ((match = pattern.exec(text)) !== null) { + matches.push({ start: match.index, heading: match[0] }); + if (match.index === pattern.lastIndex) pattern.lastIndex += 1; + } + return matches; +} + +function escapeRegExp(text) { + return text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); +} + +/** Replace matching level-2 markdown sections, otherwise append the patch. */ +export function mergeMarkdownPatch(existingContent, patchContent) { + const matches = sectionMatches(patchContent); + if (matches.length === 0) { + return existingContent + (existingContent ? "\n\n" : "") + patchContent; + } + + let merged = existingContent; + let replacedAny = false; + + for (let index = 0; index < matches.length; index += 1) { + const heading = matches[index].heading; + const start = matches[index].start; + const end = index + 1 < matches.length ? matches[index + 1].start : patchContent.length; + const patchSection = patchContent.slice(start, end).trim(); + + const sectionMatch = new RegExp(`^${escapeRegExp(heading)}\\s*$`, "m").exec(merged); + if (!sectionMatch) { + merged = `${merged.replace(/\s+$/, "")}\n\n${patchSection}`; + continue; + } + + replacedAny = true; + const sectionStart = sectionMatch.index; + const afterHeading = sectionMatch.index + sectionMatch[0].length; + const nextSection = /^##\s+.+$/m.exec(merged.slice(afterHeading)); + const sectionEnd = nextSection ? afterHeading + nextSection.index : merged.length; + merged = `${merged.slice(0, sectionStart).replace(/\s+$/, "")}\n\n${patchSection}\n\n${merged + .slice(sectionEnd) + .replace(/^\s+/, "")}`; + } + + if (replacedAny) return `${merged.trim()}\n`; + return merged; +} + +/** Python's `dict.get(key, default)`: an explicit null is a value, not a miss. */ +function getField(object, key, fallback) { + if (!object || typeof object !== "object" || !Object.hasOwn(object, key)) return fallback; + return object[key]; +} + +/** Append a normalized correction entry to persona content. */ +export function applyCorrection(personaContent, correction) { + const scene = getField(correction, "scene", "general"); + const correctionLine = `\n- [${scene}] should not ${correction.wrong}; should ${correction.correct}`; + const target = "## Correction Log"; + const legacyTarget = "## Correction 记录"; + + if (personaContent.includes(target)) { + const insertPosition = personaContent.indexOf(target) + target.length; + let rest = personaContent.slice(insertPosition); + const placeholder = "\n\n(No entries yet)"; + if (rest.startsWith(placeholder)) rest = rest.slice(placeholder.length); + return personaContent.slice(0, insertPosition) + correctionLine + rest; + } + if (personaContent.includes(legacyTarget)) { + const insertPosition = personaContent.indexOf(legacyTarget) + legacyTarget.length; + let rest = personaContent.slice(insertPosition); + const legacyPlaceholder = "\n\n（暂无记录）"; + if (rest.startsWith(legacyPlaceholder)) rest = rest.slice(legacyPlaceholder.length); + return personaContent.slice(0, insertPosition) + correctionLine + rest; + } + return `${personaContent}\n\n## Correction Log\n${correctionLine}\n`; +} + +/** Normalize a correction payload into a flat list of correction entries. */ +export function normalizeCorrections(correction) { + if (!correction) return []; + + if (Array.isArray(correction)) { + return correction.filter((item) => item && typeof item === "object" && !Array.isArray(item)); + } + + if (typeof correction === "object") { + if ("wrong" in correction && "correct" in correction) return [correction]; + for (const key of ["persona_corrections", "corrections"]) { + const value = correction[key]; + if (Array.isArray(value)) { + return value.filter( + (item) => + item && + typeof item === "object" && + !Array.isArray(item) && + "wrong" in item && + "correct" in item, + ); + } + } + } + + return []; +} + +/** + * Update an existing skill, archive the previous version, and regenerate artifacts. + * @returns {string} the new version label + */ +export function updateSkill(skillDir, workPatch = null, personaPatch = null, correction = null) { + const metaPath = join(skillDir, "meta.json"); + const meta = enrichExistingSkillMeta( + JSON.parse(readFileSync(metaPath, "utf8")), + skillDir, + ); + + const currentVersion = meta.version ?? "v1"; + let versionNumber; + try { + const head = String(currentVersion).replace(/^v+/, "").split("_")[0]; + if (!/^[+-]?\d+$/.test(head)) throw new Error("not a number"); + versionNumber = Number.parseInt(head, 10) + 1; + } catch { + versionNumber = 2; + } + const newVersion = `v${versionNumber}`; + + backupCurrentArtifacts(skillDir, currentVersion); + + const artifacts = meta.artifacts; + const workPath = join(skillDir, artifacts.work_doc); + const personaPath = join(skillDir, artifacts.persona_doc); + let workContent = existsSync(workPath) ? readFileSync(workPath, "utf8") : ""; + let personaContent = existsSync(personaPath) ? readFileSync(personaPath, "utf8") : ""; + + if (workPatch) workContent = mergeMarkdownPatch(workContent, workPatch); + + if (personaPatch) { + personaContent = mergeMarkdownPatch(personaContent, personaPatch); + } else if (correction) { + const corrections = normalizeCorrections(correction); + for (const item of corrections) personaContent = applyCorrection(personaContent, item); + if (corrections.length > 0) { + meta.generation.corrections_count = (meta.corrections_count ?? 0) + corrections.length; + } + } + + meta.lifecycle.version = newVersion; + meta.lifecycle.updated_at = nowIso(); + syncLegacyFields(meta); + + writeArtifacts(skillDir, meta, workContent, personaContent); + return newVersion; +} + +/** List skills from a storage root regardless of their type. */ +export function listSkills(baseDir) { + const skills = []; + if (!existsSync(baseDir)) return skills; + + const entries = readdirSync(baseDir).sort(); + for (const entry of entries) { + const skillDir = join(baseDir, entry); + if (!statSync(skillDir).isDirectory()) continue; + + const metaPath = join(skillDir, "meta.json"); + if (!existsSync(metaPath)) continue; + + let meta; + try { + meta = enrichExistingSkillMeta(JSON.parse(readFileSync(metaPath, "utf8")), skillDir); + } catch { + continue; + } + + skills.push({ + slug: meta.slug ?? entry, + kind: meta.kind ?? "meta-skill", + character: meta.character ?? "colleague", + research_profile: meta.research_profile ?? "standard", + name: meta.display_name ?? entry, + identity: buildIdentityString(meta), + version: meta.version ?? "v1", + updated_at: meta.updated_at ?? "", + corrections_count: meta.corrections_count ?? 0, + }); + } + + return skills; +} + +/** Resolve the storage root for a character family while keeping compatibility. */ +export function resolveBaseDir(baseDirArg, character) { + return resolveStorageRoot(character, baseDirArg); +} + +export { resolveExistingStorageRoot }; diff --git a/src/views/render.mjs b/src/views/render.mjs new file mode 100644 index 00000000..1e3b244a --- /dev/null +++ b/src/views/render.mjs @@ -0,0 +1,262 @@ +/** + * distilly view render — template + views/.view.json -> views/.html + * plus evidence/renders/receipt.json. + * + * Guarantees: + * - single file, offline, no external request (the template's CSP forbids them); + * - deterministic: the same inputs produce the same bytes, twice in a row; + * - private by default: without --shareable no source wording reaches the HTML; + * - the renderer invents nothing: every byte of content comes from view.json. + * + * Zero runtime dependencies. + */ +import { createHash } from "node:crypto"; +import { existsSync, mkdirSync, readFileSync, readdirSync, statSync, writeFileSync } from "node:fs"; +import { dirname, isAbsolute, join, relative, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { anchorCounts, checkView, expectedSlug } from "./schema.mjs"; + +export const packageRoot = fileURLToPath(new URL("../../", import.meta.url)); +export const TEMPLATE_PATH = join(packageRoot, "assets", "distilly-template.html"); +export const VIEW_DATA_MARKER = "@@DISTILLY:VIEW_DATA@@"; + +const RECEIPT_RELATIVE = join("evidence", "renders", "receipt.json"); + +export class ViewError extends Error { + constructor(message, diagnostics = [], supportedFixes = []) { + super(message); + this.name = "ViewError"; + this.diagnostics = diagnostics; + this.supportedFixes = supportedFixes; + } +} + +export function sha256(value) { + return createHash("sha256").update(value).digest("hex"); +} + +export function sha256File(path) { + return sha256(readFileSync(path)); +} + +function isPlainObject(value) { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +/** Recursively sort object keys so the same document always serialises identically. */ +function canonical(value) { + if (Array.isArray(value)) return value.map(canonical); + if (isPlainObject(value)) { + const out = {}; + for (const key of Object.keys(value).sort()) out[key] = canonical(value[key]); + return out; + } + return value; +} + +/** Deterministic JSON (sorted keys, 2-space indent, trailing newline). */ +export function canonicalJson(value, indent = 2) { + return `${JSON.stringify(canonical(value), null, indent)}\n`; +} + +/** JSON safe to inline inside \n`, "utf8"); + const run = runGenerator(root); + assert.equal(run.status, 1); + assert.match(run.stderr, /<\/script/); +}); + +test("the template is a single offline file with the frozen CSP", () => { + const html = readCommitted(); + assert.ok(html.includes(`content="default-src 'none'; img-src data:; style-src 'unsafe-inline'; script-src 'unsafe-inline'"`)); + assert.ok(!html.includes("http://") && !html.includes("https://"), "no absolute URL may appear in the template"); + assert.equal(/]+src=/i.test(html), false); + assert.equal(/]+href=/i.test(html), false); + assert.equal(/]+src=/i.test(html), false); + assert.equal((html.match(//g) || []).length, 1); + assert.ok(html.includes('')); + assert.ok(html.includes(" { + const html = readCommitted(); + assert.equal(html.split(VIEW_DATA_MARKER).length - 1, 1); + assert.equal(html.replace(VIEW_DATA_MARKER, "").includes("@@DISTILLY:"), false, "no other marker may survive"); + for (const relative of Object.values(FRAGMENTS)) { + const body = readFileSync(join(repoRoot, relative), "utf8"); + assert.ok(html.includes(body.trimEnd()), `${relative} is not embedded verbatim`); + } + assert.ok(html.includes('"), true); +}); + +test("private render never inlines source wording; --shareable does and records it", () => { + const root = tempRoot(); + const viewPath = writeFixture(root, validView()); + + const privateRun = renderView({ viewPath }); + for (const quote of Object.values(QUOTES)) { + assert.equal(privateRun.html.includes(quote.slice(0, 12)), false, "private output must not carry source wording"); + assert.equal(privateRun.html.includes(quote), false); + } + assert.deepEqual(privateRun.receipt.inlined_sources, []); + + const shareablePath = writeFixture(tempRoot(), validView()); + const shared = renderView({ viewPath: shareablePath, shareable: true }); + assert.ok(shared.html.includes(QUOTES.k0012), "shareable output must inline the quote"); + assert.equal(shared.receipt.shareable, true); + assert.equal(shared.receipt.inlined_sources.length, 8); + assert.deepEqual(shared.receipt.inlined_sources[0].anchor, "k0012"); + assert.match(shared.receipt.inlined_sources[0].quote_sha256, /^[0-9a-f]{64}$/); + assert.ok(shared.receipt.inlined_sources[0].quote_bytes > 0); + assert.notEqual(shared.receipt.outputs[0].sha256, privateRun.receipt.outputs[0].sha256); +}); + +test("renderView refuses an invalid document and writes nothing", () => { + const root = tempRoot(); + const broken = validView(); + broken.sections[0].items[0].anchors = []; + const viewPath = writeFixture(root, broken); + const outPath = join(root, "skills", "colleague", "zhang-san", "views", "zhang-san.html"); + + assert.throws( + () => renderView({ viewPath }), + (error) => { + assert.ok(error instanceof ViewError); + assert.ok(error.diagnostics.some((entry) => entry.code === "VIEW_ANCHOR_MISSING")); + assert.ok(error.supportedFixes.length > 0); + return true; + }, + ); + assert.equal(existsSync(outPath), false); + assert.equal(existsSync(join(root, "skills", "colleague", "zhang-san", "evidence", "renders", "receipt.json")), false); +}); + +test("loadViewDocument reports broken JSON with a fix", () => { + const root = tempRoot(); + const dir = join(root, "views"); + mkdirSync(dir, { recursive: true }); + const path = join(dir, "broken.view.json"); + writeFileSync(path, "{ not json", "utf8"); + assert.throws(() => loadViewDocument(path), /not valid JSON/); + assert.throws(() => loadViewDocument(join(dir, "missing.view.json")), /not found/); +}); + +test("canonicalJson sorts keys so receipts are byte-stable", () => { + const value = { b: 1, a: { d: 2, c: [3, { f: 4, e: 5 }] } }; + assert.equal(canonicalJson(value), canonicalJson({ a: { c: [3, { e: 5, f: 4 }], d: 2 }, b: 1 })); + assert.equal(canonicalJson(value), '{\n "a": {\n "c": [\n 3,\n {\n "e": 5,\n "f": 4\n }\n ],\n "d": 2\n },\n "b": 1\n}\n'); +}); + +test("REQUIRED_SECTIONS names the eight page segments in order", () => { + assert.deepEqual( + REQUIRED_SECTIONS.map((entry) => entry.id), + ["portrait", "communication", "values", "workstyle", "relationship", "boundaries", "timeline", "evidence"], + ); + assert.deepEqual(KINDS_OF_AUTHORED(), ["claims", "claims", "claims", "claims", "claims", "warnings", "timeline"]); +}); + +function KINDS_OF_AUTHORED() { + return REQUIRED_SECTIONS.filter((entry) => !entry.derived).map((entry) => entry.kind); +} + +test("anchor pattern accepts ledger ids and rejects prose", () => { + for (const value of ["k0012", "k0012:t3", "msg0001", "k0040:t2"]) assert.ok(ANCHOR_PATTERN.test(value), value); + for (const value of ["12", "K0012", "[k0012]", "k12", "k0012:t"]) assert.equal(ANCHOR_PATTERN.test(value), false, value); +}); + +/* ---------------------------------------------------------------- CLI leaf */ + +function runCli(args) { + return spawnSync(process.execPath, [BIN, ...args], { encoding: "utf8", cwd: repoRoot }); +} + +test("cli: view check --json emits a contract receipt", () => { + const root = tempRoot(); + writeFixture(root, validView()); + const run = runCli(["view", "check", "zhang-san", "--root", root, "--json"]); + assert.equal(run.status, 0, run.stderr); + const receipt = JSON.parse(run.stdout); + assert.equal(receipt.command, "view check"); + assert.equal(receipt.ok, true); + assert.equal(receipt.anchors.total, 8); + assert.equal(receipt.anchors.cited, 8); + assert.equal(receipt.diagnostics.length, 0); + assert.equal(receipt.inputs.length, 1); + assert.match(receipt.inputs[0].sha256, /^[0-9a-f]{64}$/); +}); + +test("cli: view check exits 1 and prints fixes for a broken document", () => { + const root = tempRoot(); + const broken = validView(); + broken.sections = broken.sections.filter((section) => section.id !== "boundaries"); + broken.sections[0].items[0].confidence = "sure"; + writeFixture(root, broken); + + const human = runCli(["view", "check", "zhang-san", "--root", root]); + assert.equal(human.status, 1); + assert.match(human.stdout, /VIEW_SECTION_MISSING/); + assert.match(human.stdout, /VIEW_CONFIDENCE_INVALID/); + assert.match(human.stdout, /fix:/); + + const json = runCli(["view", "check", "zhang-san", "--root", root, "--json"]); + assert.equal(json.status, 1); + const receipt = JSON.parse(json.stdout); + assert.equal(receipt.ok, false); + assert.ok(receipt.diagnostics.length >= 2); +}); + +test("cli: view render writes the page twice with identical bytes", () => { + const root = tempRoot(); + const viewPath = writeFixture(root, validView()); + const first = runCli(["view", "render", "zhang-san", "--root", root, "--json"]); + assert.equal(first.status, 0, first.stderr); + const htmlPath = join(root, "skills", "colleague", "zhang-san", "views", "zhang-san.html"); + const receiptPath = join(root, "skills", "colleague", "zhang-san", "evidence", "renders", "receipt.json"); + const firstHtml = readFileSync(htmlPath, "utf8"); + const firstReceipt = readFileSync(receiptPath, "utf8"); + + const second = runCli(["view", "render", viewPath, "--json"]); + assert.equal(second.status, 0, second.stderr); + assert.equal(readFileSync(htmlPath, "utf8"), firstHtml); + assert.equal(readFileSync(receiptPath, "utf8"), firstReceipt); + + const receipt = JSON.parse(second.stdout); + assert.equal(receipt.ok, true); + assert.equal(receipt.outputs[0].sha256, sha256(firstHtml)); +}); + +test("cli: view render refuses an invalid document with exit 1", () => { + const root = tempRoot(); + const broken = validView(); + broken.evidence = []; + writeFixture(root, broken); + const run = runCli(["view", "render", "zhang-san", "--root", root]); + assert.equal(run.status, 1); + assert.match(run.stderr, /VIEW_EVIDENCE_EMPTY|failed view check/); +}); + +test("cli: view needs a slug or --file", () => { + const run = runCli(["view", "check"]); + assert.equal(run.status, 1); + assert.match(run.stderr, /needs a or --file/); + + const help = runCli(["view", "--help"]); + assert.equal(help.status, 0); + assert.match(help.stdout, /## English/); +}); diff --git a/viewer/export.js b/viewer/export.js new file mode 100644 index 00000000..e1edc890 --- /dev/null +++ b/viewer/export.js @@ -0,0 +1,185 @@ +/* Distilly viewer fragment: offline export actions (print, Markdown copy, HTML snapshot). + No network is used; a download is built from the current document itself. */ +(function () { + "use strict"; + + var LABELS = { + zh: { + copied: "已复制 Markdown 摘要（{n} 字符）", + copyFailed: "复制失败：请手动选择页面内容。", + downloaded: "已生成 HTML 快照（浏览器下载目录）。", + downloadFailed: "下载失败：可用「打印 / 导出 PDF」代替。", + printing: "已调用打印对话框；选择「存储为 PDF」即可保存。", + unavailable: "页面数据缺失，导出内容为空。" + }, + en: { + copied: "Markdown summary copied ({n} characters).", + copyFailed: "Copy failed: please select the page content manually.", + downloaded: "HTML snapshot written to your downloads folder.", + downloadFailed: "Download failed: use Print / Export PDF instead.", + printing: "Print dialog requested; choose \"Save as PDF\".", + unavailable: "No page data: nothing to export." + } + }; + + function labels() { + var lang = document.documentElement.getAttribute("lang") || "zh"; + return lang.toLowerCase().indexOf("en") === 0 ? LABELS.en : LABELS.zh; + } + + function status(message) { + var node = document.getElementById("action-status"); + if (node) node.textContent = message; + } + + function view() { + return window.DistillyView && window.DistillyView.view ? window.DistillyView.view : null; + } + + function asArray(value) { + return Object.prototype.toString.call(value) === "[object Array]" ? value : []; + } + + function anchorText(anchors) { + var list = asArray(anchors); + if (list.length === 0) return ""; + return " `" + list.join("` `") + "`"; + } + + function toMarkdown(source) { + if (!source) return ""; + var meta = source.meta || {}; + var lines = ["# " + (meta.title || meta.slug || "Person View"), ""]; + if (meta.slug) lines.push("- slug: `" + meta.slug + "`"); + if (meta.generated_at) lines.push("- generated_at: " + meta.generated_at); + lines.push("- shareable: " + (source.shareable === true ? "true" : "false")); + lines.push(""); + + asArray(source.sections).forEach(function (section, index) { + lines.push("## " + (index + 1) + ". " + (section.title || section.id)); + if (section.summary) lines.push("", String(section.summary)); + lines.push(""); + asArray(section.items).forEach(function (item) { + var prefix = section.kind === "timeline" ? "- " + (item.at || "—") + " · " : "- "; + var suffix = item.confidence ? " (" + item.confidence + ")" : ""; + lines.push(prefix + item.text + suffix + anchorText(item.anchors)); + }); + lines.push(""); + }); + + var evidence = asArray(source.evidence); + lines.push("## " + (evidence.length > 0 ? "证据附录 / Evidence appendix" : "证据附录"), ""); + evidence.forEach(function (entry) { + var where = []; + if (entry.source) where.push(entry.source); + if (entry.kind) where.push(entry.kind); + if (entry.at) where.push(entry.at); + if (entry.path) where.push(entry.path); + lines.push("- `" + entry.anchor + "` — " + where.join(" · ") + (entry.note ? " — " + entry.note : "")); + if (source.shareable === true && entry.quote) lines.push(" > " + String(entry.quote).replace(/\n/g, " ")); + }); + lines.push(""); + return lines.join("\n"); + } + + function copyText(text) { + if (navigator.clipboard && typeof navigator.clipboard.writeText === "function") { + return navigator.clipboard.writeText(text).then(function () { + return true; + }).catch(function () { + return legacyCopy(text); + }); + } + return Promise.resolve(legacyCopy(text)); + } + + function legacyCopy(text) { + var area = document.createElement("textarea"); + area.value = text; + area.setAttribute("readonly", "readonly"); + area.className = "visually-hidden"; + area.setAttribute("aria-hidden", "true"); + document.body.appendChild(area); + var ok = false; + try { + area.select(); + ok = document.execCommand("copy"); + } catch (error) { + ok = false; + } + document.body.removeChild(area); + return ok; + } + + function download() { + var text = "\n" + document.documentElement.outerHTML; + var name = ((view() && view().meta && view().meta.slug) || "distilly-view") + ".html"; + var link = document.createElement("a"); + link.setAttribute("download", name); + link.className = "visually-hidden"; + link.setAttribute("aria-hidden", "true"); + var url = ""; + try { + var blob = new Blob([text], { type: "text/html;charset=utf-8" }); + url = URL.createObjectURL(blob); + link.href = url; + } catch (error) { + link.href = "data:text/html;charset=utf-8," + encodeURIComponent(text); + } + document.body.appendChild(link); + link.click(); + document.body.removeChild(link); + if (url) window.setTimeout(function () { URL.revokeObjectURL(url); }, 0); + return true; + } + + function bind(id, handler) { + var node = document.getElementById(id); + if (!node) return; + node.addEventListener("click", function () { + var text = labels(); + try { + handler(text); + } catch (error) { + status(text.copyFailed); + } + }); + } + + function boot() { + bind("export-print", function (text) { + status(text.printing); + window.print(); + }); + + bind("export-copy", function (text) { + var markdown = toMarkdown(view()); + if (markdown === "") { + status(text.unavailable); + return; + } + Promise.resolve(copyText(markdown)).then(function (ok) { + status(ok ? text.copied.replace("{n}", String(markdown.length)) : text.copyFailed); + }); + }); + + bind("export-html", function (text) { + var ok = false; + try { + ok = download(); + } catch (error) { + ok = false; + } + status(ok ? text.downloaded : text.downloadFailed); + }); + + window.DistillyExport = { toMarkdown: toMarkdown, version: 1 }; + document.documentElement.setAttribute("data-export-bound", "true"); + } + + if (document.readyState === "loading") { + document.addEventListener("DOMContentLoaded", boot); + } else { + boot(); + } +})(); diff --git a/viewer/focus.js b/viewer/focus.js new file mode 100644 index 00000000..84a06e38 --- /dev/null +++ b/viewer/focus.js @@ -0,0 +1,133 @@ +/* Distilly viewer fragment: keyboard reachability and anchor focus behaviour. + Anchor ids can contain ':' (k0012:t3), so every lookup goes through getElementById. */ +(function () { + "use strict"; + + var HIGHLIGHT = "is-focused"; + var LABELS = { + zh: { jumped: "已定位到 {what}", back: "返回引用处", skipped: "已跳到正文" }, + en: { jumped: "Jumped to {what}", back: "Back to the reference", skipped: "Skipped to content" } + }; + + function labels() { + var lang = document.documentElement.getAttribute("lang") || "zh"; + return lang.toLowerCase().indexOf("en") === 0 ? LABELS.en : LABELS.zh; + } + + function status(message) { + var node = document.getElementById("action-status"); + if (node) node.textContent = message; + } + + function targetFromHash(hash) { + var raw = String(hash || "").replace(/^#/, ""); + if (raw === "") return null; + try { + return document.getElementById(decodeURIComponent(raw)); + } catch (error) { + return document.getElementById(raw); + } + } + + function describe(node) { + if (!node) return ""; + if (node.hasAttribute("data-anchor")) return "证据 " + node.getAttribute("data-anchor"); + var heading = node.querySelector ? node.querySelector(".section__title") : null; + if (heading) return heading.textContent || ""; + return node.id || ""; + } + + function clearHighlights() { + var marked = document.querySelectorAll("." + HIGHLIGHT); + for (var index = 0; index < marked.length; index += 1) { + marked[index].classList.remove(HIGHLIGHT); + } + var active = document.querySelectorAll(".anchor-ref.is-active"); + for (var i = 0; i < active.length; i += 1) active[i].classList.remove("is-active"); + } + + function focusTarget(node, announce) { + if (!node) return false; + clearHighlights(); + if (!node.hasAttribute("tabindex")) node.setAttribute("tabindex", "-1"); + node.classList.add(HIGHLIGHT); + try { + node.focus({ preventScroll: true }); + } catch (error) { + node.focus(); + } + if (typeof node.scrollIntoView === "function") { + node.scrollIntoView({ block: "start", behavior: "auto" }); + } + if (announce) status(labels().jumped.replace("{what}", describe(node))); + return true; + } + + function bindAnchorRefs() { + var refs = document.querySelectorAll(".anchor-ref[data-anchor-ref]"); + for (var index = 0; index < refs.length; index += 1) { + var ref = refs[index]; + ref.addEventListener("focus", function (event) { + event.currentTarget.classList.add("is-active"); + }); + ref.addEventListener("blur", function (event) { + event.currentTarget.classList.remove("is-active"); + }); + ref.addEventListener("click", function (event) { + var anchor = event.currentTarget.getAttribute("data-anchor-ref"); + var node = document.getElementById("anchor-" + anchor); + if (node) { + event.preventDefault(); + focusTarget(node, true); + if (window.history && typeof window.history.replaceState === "function") { + window.history.replaceState(null, "", "#anchor-" + anchor); + } + } + }); + } + + var backs = document.querySelectorAll('.evidence__meta a[href^="#section-"]'); + for (var i = 0; i < backs.length; i += 1) { + backs[i].setAttribute("title", labels().back); + } + } + + function bindSkipLink() { + var link = document.querySelector(".skip-link"); + var main = document.getElementById("main"); + if (!link || !main) return; + link.addEventListener("click", function () { + focusTarget(main, false); + status(labels().skipped); + }); + } + + function bindEscape() { + document.addEventListener("keydown", function (event) { + if (event.key !== "Escape") return; + var active = document.activeElement; + if (!active || active === document.body) return; + if (active.classList && (active.classList.contains(HIGHLIGHT) || active.classList.contains("anchor-ref"))) { + clearHighlights(); + if (typeof active.blur === "function") active.blur(); + } + }); + } + + function boot() { + bindSkipLink(); + bindAnchorRefs(); + bindEscape(); + if (window.location.hash) focusTarget(targetFromHash(window.location.hash), false); + window.addEventListener("hashchange", function () { + focusTarget(targetFromHash(window.location.hash), true); + }); + document.documentElement.setAttribute("data-focus-bound", "true"); + } + + if (document.readyState === "loading") { + document.addEventListener("DOMContentLoaded", boot); + } else { + boot(); + } +})(); diff --git a/viewer/sections.js b/viewer/sections.js new file mode 100644 index 00000000..46dbefbd --- /dev/null +++ b/viewer/sections.js @@ -0,0 +1,398 @@ +/* Distilly viewer fragment: renders the eight page sections from the embedded view.json. + Classic script (no modules, no network). Text only: every value is written with + textContent, so a hostile view.json cannot inject markup. */ +(function () { + "use strict"; + + var DATA_ID = "distilly-view-data"; + var ROOT_SELECTOR = "[data-sections-root]"; + + var LABELS = { + zh: { + confidence: { high: "高置信", medium: "中置信", low: "低置信" }, + severity: { high: "雷区·高", medium: "雷区·中", low: "雷区·低" }, + claims: "条结论", + timeline: "个节点", + warnings: "条边界", + evidence: "条锚点", + anchorLabel: "查看证据 ", + citedBy: "被引用", + where: "出处", + digest: "校验", + quote: "原文", + empty: "本节没有可展示的内容。", + noData: "尚未嵌入 view.json 数据；这是一份可直接打开的模板。", + privacyPrivate: "默认私有：本页只显示结论与锚点编号，不内联任何原始引文。", + privacyShareable: "可分享模式：本页内联了下列来源的原文引文，请自行确认分享范围。", + appendixTitle: "证据附录", + appendixNote: "编号可在 knowledge/index.json 回指；点编号可从结论跳到此处。", + quotedSources: "内联来源", + footer: "渲染器不产生事实：所有结论与锚点均来自 view.json 与知识账本。" + }, + en: { + confidence: { high: "high confidence", medium: "medium confidence", low: "low confidence" }, + severity: { high: "red line · high", medium: "red line · medium", low: "red line · low" }, + claims: "claims", + timeline: "milestones", + warnings: "boundaries", + evidence: "anchors", + anchorLabel: "Show evidence ", + citedBy: "cited by", + where: "source", + digest: "digest", + quote: "quote", + empty: "This section has nothing to show.", + noData: "No view.json payload is embedded; this file is the openable template itself.", + privacyPrivate: "Private by default: conclusions and anchor ids only, no source wording.", + privacyShareable: "Shareable mode: verbatim quotes from the sources below are inlined.", + appendixTitle: "Evidence appendix", + appendixNote: "Anchor ids trace back to knowledge/index.json.", + quotedSources: "inlined sources", + footer: "The renderer invents nothing: every claim and anchor comes from view.json." + } + }; + + var DEFAULT_TITLES = [ + { id: "portrait", zh: "一句话画像", en: "One-line portrait" }, + { id: "communication", zh: "沟通风格", en: "Communication style" }, + { id: "values", zh: "决策与价值观", en: "Decisions and values" }, + { id: "workstyle", zh: "工作方式", en: "Working style" }, + { id: "relationship", zh: "关系与称呼", en: "Relationship and address" }, + { id: "boundaries", zh: "边界与雷区", en: "Boundaries and red lines" }, + { id: "timeline", zh: "时间线演变", en: "Timeline" } + ]; + + function readView() { + var node = document.getElementById(DATA_ID); + if (!node) return null; + var raw = (node.textContent || "").trim(); + if (raw === "" || raw === "null") return null; + try { + return JSON.parse(raw); + } catch (error) { + return null; + } + } + + function labelsFor(view) { + var lang = view && view.meta && typeof view.meta.lang === "string" ? view.meta.lang : "zh"; + return lang.toLowerCase().indexOf("en") === 0 ? LABELS.en : LABELS.zh; + } + + function el(tag, className, text) { + var node = document.createElement(tag); + if (className) node.className = className; + if (text !== undefined && text !== null) node.textContent = String(text); + return node; + } + + function asArray(value) { + return Object.prototype.toString.call(value) === "[object Array]" ? value : []; + } + + function anchorRef(anchor, label) { + var link = el("a", "anchor-ref", String(anchor)); + link.setAttribute("href", "#anchor-" + String(anchor)); + link.setAttribute("data-anchor-ref", String(anchor)); + link.setAttribute("aria-label", label + String(anchor)); + return link; + } + + function confidenceBadge(level, labels) { + var known = Object.prototype.hasOwnProperty.call(labels.confidence, level) ? level : "low"; + var badge = el("span", "badge badge--" + known, labels.confidence[known]); + badge.setAttribute("data-confidence", known); + return badge; + } + + function sectionShell(spec, index) { + var section = el("section", "section"); + section.id = "section-" + spec.id; + section.setAttribute("data-section", spec.id); + section.setAttribute("data-section-index", String(index + 1)); + section.setAttribute("aria-labelledby", "heading-" + spec.id); + + var head = el("div", "section__head"); + var number = el("span", "section__index", String(index + 1).padStart(2, "0")); + number.setAttribute("aria-hidden", "true"); + var title = el("h2", "section__title", spec.title); + title.id = "heading-" + spec.id; + head.appendChild(number); + head.appendChild(title); + head.appendChild(el("span", "section__count", spec.count)); + section.appendChild(head); + + if (spec.summary) section.appendChild(el("p", "section__summary", spec.summary)); + return section; + } + + function renderClaims(section, spec, view, labels, citations) { + var list = el("ul", "claims"); + spec.items.forEach(function (item) { + var row = el("li", "claim"); + if (item.emphasis === true) row.className = "claim is-emphasis"; + row.setAttribute("data-confidence", item.confidence); + row.appendChild(el("p", "claim__text", item.text)); + var meta = el("ul", "claim__meta"); + meta.appendChild(confidenceBadge(item.confidence, labels)); + asArray(item.anchors).forEach(function (anchor) { + var li = el("li"); + li.appendChild(anchorRef(anchor, labels.anchorLabel)); + meta.appendChild(li); + if (!citations[anchor]) citations[anchor] = []; + citations[anchor].push({ id: spec.id, title: spec.title }); + }); + row.appendChild(meta); + list.appendChild(row); + }); + section.appendChild(list); + } + + function renderWarnings(section, spec, view, labels, citations) { + var list = el("ul", "warnings"); + spec.items.forEach(function (item) { + var row = el("li", "warning"); + row.setAttribute("data-severity", item.severity || "medium"); + row.appendChild(el("p", "warning__text", item.text)); + var meta = el("ul", "warning__meta"); + var severity = Object.prototype.hasOwnProperty.call(labels.severity, item.severity) + ? item.severity + : "medium"; + var badge = el("span", "badge badge--severity", labels.severity[severity]); + badge.setAttribute("data-severity", severity); + meta.appendChild(badge); + meta.appendChild(confidenceBadge(item.confidence, labels)); + asArray(item.anchors).forEach(function (anchor) { + var li = el("li"); + li.appendChild(anchorRef(anchor, labels.anchorLabel)); + meta.appendChild(li); + if (!citations[anchor]) citations[anchor] = []; + citations[anchor].push({ id: spec.id, title: spec.title }); + }); + row.appendChild(meta); + list.appendChild(row); + }); + section.appendChild(list); + } + + function renderTimeline(section, spec, view, labels, citations) { + var list = el("ol", "timeline"); + spec.items.forEach(function (item) { + var row = el("li", "timeline__item"); + row.setAttribute("data-confidence", item.confidence); + var at = el("span", "timeline__at", item.at || "—"); + row.appendChild(at); + row.appendChild(el("span", "timeline__text", item.text)); + var meta = el("ul", "timeline__meta"); + meta.appendChild(confidenceBadge(item.confidence, labels)); + asArray(item.anchors).forEach(function (anchor) { + var li = el("li"); + li.appendChild(anchorRef(anchor, labels.anchorLabel)); + meta.appendChild(li); + if (!citations[anchor]) citations[anchor] = []; + citations[anchor].push({ id: spec.id, title: spec.title }); + }); + row.appendChild(meta); + list.appendChild(row); + }); + section.appendChild(list); + } + + function renderEvidenceItem(entry, labels, citations, shareable) { + var row = el("li", "evidence"); + row.id = "anchor-" + entry.anchor; + row.setAttribute("data-anchor", entry.anchor); + row.setAttribute("tabindex", "-1"); + + var head = el("div", "evidence__head"); + head.appendChild(el("span", "evidence__anchor", entry.anchor)); + head.appendChild(el("span", "badge badge--kind", entry.kind || "source")); + head.appendChild(el("span", "badge", entry.source || "unknown")); + if (entry.at) head.appendChild(el("span", "evidence__at", entry.at)); + row.appendChild(head); + + if (entry.note) row.appendChild(el("p", "evidence__why", entry.note)); + + if (shareable && entry.quote) { + var quote = el("blockquote", "quote", entry.quote); + quote.setAttribute("data-inlined", "true"); + row.appendChild(quote); + row.appendChild(el("p", "quote__note", labels.quote + " · " + (entry.source || "unknown"))); + } + + var where = []; + if (entry.path) where.push(labels.where + ": " + entry.path); + if (entry.id) where.push("id: " + entry.id); + if (entry.sha256) where.push(labels.digest + ": " + String(entry.sha256).slice(0, 12)); + if (where.length > 0) row.appendChild(el("p", "evidence__where", where.join(" · "))); + + var cited = citations[entry.anchor] || []; + if (cited.length > 0) { + var meta = el("ul", "evidence__meta"); + meta.appendChild(el("li", null, labels.citedBy)); + cited.forEach(function (ref) { + var li = el("li"); + var link = el("a", null, ref.title); + link.setAttribute("href", "#section-" + ref.id); + li.appendChild(link); + meta.appendChild(li); + }); + row.appendChild(meta); + } + return row; + } + + function appendixSpec(view, labels) { + return { + id: "evidence", + title: labels.appendixTitle, + summary: labels.appendixNote, + items: asArray(view.evidence) + }; + } + + function renderInto(root, view) { + var labels = labelsFor(view); + root.textContent = ""; + + if (!view) { + var state = el("section", "section", labels.noData); + state.setAttribute("data-section", "empty"); + root.appendChild(state); + return { sections: 0, anchors: 0 }; + } + + var specs = []; + var citations = {}; + asArray(view.sections).forEach(function (raw) { + var id = String(raw && raw.id ? raw.id : ""); + var fallback = DEFAULT_TITLES.filter(function (entry) { return entry.id === id; })[0]; + var items = asArray(raw && raw.items); + specs.push({ + id: id, + kind: String(raw && raw.kind ? raw.kind : "claims"), + title: String(raw && raw.title ? raw.title : (fallback ? fallback.zh : id)), + summary: raw && raw.summary ? String(raw.summary) : "", + items: items, + count: "" + }); + }); + + var appendix = appendixSpec(view, labels); + appendix.count = String(appendix.items.length) + " " + labels.evidence; + specs.push(appendix); + + specs.forEach(function (spec, index) { + if (!spec.count) { + var unit = spec.kind === "timeline" ? labels.timeline + : spec.kind === "warnings" ? labels.warnings : labels.claims; + spec.count = String(spec.items.length) + " " + unit; + } + }); + + var holder = document.createDocumentFragment(); + specs.forEach(function (spec, index) { + var section = sectionShell(spec, index); + if (spec.id === "evidence") { + section.className = "section section--evidence"; + var list = el("ol", "appendix"); + spec.items.forEach(function (entry) { + list.appendChild(renderEvidenceItem(entry, labels, citations, view.shareable === true)); + }); + section.appendChild(list); + } else if (spec.kind === "timeline") { + renderTimeline(section, spec, view, labels, citations); + } else if (spec.kind === "warnings") { + renderWarnings(section, spec, view, labels, citations); + } else { + renderClaims(section, spec, view, labels, citations); + } + holder.appendChild(section); + }); + + root.appendChild(holder); + var count = root.querySelectorAll("[data-section]").length; + var anchors = root.querySelectorAll(".evidence[data-anchor]").length; + return { sections: count, anchors: anchors, citations: citations }; + } + + function renderToc(view, labels) { + var toc = document.getElementById("toc"); + if (!toc) return; + toc.textContent = ""; + if (!view) return; + var heading = el("h2", null, document.documentElement.lang.indexOf("en") === 0 ? "Contents" : "目录"); + var list = el("ol"); + var specs = asArray(view.sections).concat([appendixSpec(view, labels)]); + specs.forEach(function (spec, index) { + var fallback = DEFAULT_TITLES.filter(function (entry) { return entry.id === spec.id; })[0]; + var li = el("li"); + var link = el("a"); + link.setAttribute("href", "#section-" + spec.id); + link.appendChild(el("span", "toc__index", String(index + 1).padStart(2, "0"))); + link.appendChild(document.createTextNode(spec.title || (fallback ? fallback.zh : spec.id))); + li.appendChild(link); + list.appendChild(li); + }); + toc.appendChild(heading); + toc.appendChild(list); + } + + function renderHeader(view, labels) { + var title = document.getElementById("page-title"); + var subtitle = document.getElementById("page-subtitle"); + var meta = document.getElementById("page-meta"); + var note = document.getElementById("privacy-note"); + var footer = document.getElementById("footer-note"); + if (footer) footer.textContent = labels.footer; + + if (!view) { + if (subtitle) subtitle.textContent = labels.noData; + if (note) note.textContent = labels.privacyPrivate; + return; + } + + var meta_ = view.meta || {}; + var displayTitle = meta_.title || meta_.slug || "Person View"; + if (title) title.textContent = displayTitle; + document.title = displayTitle + " · Distilly"; + if (subtitle) { + subtitle.textContent = meta_.subtitle || (meta_.display_name ? meta_.display_name : meta_.slug || ""); + } + if (meta) { + meta.textContent = ""; + var parts = []; + if (meta_.slug) parts.push("slug: " + meta_.slug); + if (meta_.generated_at) parts.push("generated_at: " + meta_.generated_at); + var appendix = appendixSpec(view, labels); + parts.push("anchors: " + appendix.items.length); + meta.textContent = parts.join(" · "); + } + if (note) note.textContent = view.shareable === true ? labels.privacyShareable : labels.privacyPrivate; + } + + function boot() { + var root = document.querySelector(ROOT_SELECTOR); + var view = readView(); + var labels = labelsFor(view); + renderHeader(view, labels); + var result = root ? renderInto(root, view) : { sections: 0, anchors: 0 }; + renderToc(view, labels); + window.DistillyView = { + version: 1, + view: view, + labels: labels, + shareable: Boolean(view && view.shareable === true), + sections: result.sections || 0, + anchors: result.anchors || 0, + citations: result.citations || {} + }; + document.documentElement.setAttribute("data-view-ready", "true"); + } + + if (document.readyState === "loading") { + document.addEventListener("DOMContentLoaded", boot); + } else { + boot(); + } +})(); diff --git a/viewer/theme.js b/viewer/theme.js new file mode 100644 index 00000000..396c7539 --- /dev/null +++ b/viewer/theme.js @@ -0,0 +1,119 @@ +/* Distilly viewer fragment: dual theme (system preference + manual override). + The palette lives in CSS (light-dark()); this file only flips the data-theme + attribute on the root element. */ +(function () { + "use strict"; + + var STORAGE_KEY = "distilly-view-theme"; + var MODES = ["auto", "light", "dark"]; + var LABELS = { + zh: { toDark: "深色模式", toLight: "浅色模式", toSystem: "跟随系统", status: "主题：" }, + en: { toDark: "Dark mode", toLight: "Light mode", toSystem: "Follow system", status: "Theme: " } + }; + + function labels() { + var lang = document.documentElement.getAttribute("lang") || "zh"; + return lang.toLowerCase().indexOf("en") === 0 ? LABELS.en : LABELS.zh; + } + + function stored() { + try { + var value = window.localStorage.getItem(STORAGE_KEY); + return MODES.indexOf(value) === -1 ? null : value; + } catch (error) { + return null; + } + } + + function remember(value) { + try { + window.localStorage.setItem(STORAGE_KEY, value); + } catch (error) { + /* file:// or a locked-down profile: the switch still works for this page view. */ + } + } + + function requested() { + var match = /[?&]theme=(auto|light|dark)(?:&|$)/.exec(window.location.search || ""); + return match ? match[1] : null; + } + + function systemPrefersDark() { + return Boolean(window.matchMedia) && window.matchMedia("(prefers-color-scheme: dark)").matches; + } + + var mode = requested() || stored() || "auto"; + + function effective() { + return mode === "auto" ? (systemPrefersDark() ? "dark" : "light") : mode; + } + + function resetButton() { + var existing = document.getElementById("theme-system"); + if (mode === "auto") { + if (existing && existing.parentNode) existing.parentNode.removeChild(existing); + return; + } + if (existing) return; + var toggle = document.getElementById("theme-toggle"); + if (!toggle || !toggle.parentNode) return; + var text = labels(); + var button = document.createElement("button"); + button.type = "button"; + button.id = "theme-system"; + button.className = "button button--quiet"; + button.textContent = text.toSystem; + button.addEventListener("click", function () { + apply("auto", true); + }); + toggle.parentNode.insertBefore(button, toggle.nextSibling); + } + + function apply(next, announce) { + mode = MODES.indexOf(next) === -1 ? "auto" : next; + var active = effective(); + document.documentElement.setAttribute("data-theme", mode); + document.documentElement.setAttribute("data-theme-effective", active); + remember(mode); + + var text = labels(); + var toggle = document.getElementById("theme-toggle"); + if (toggle) { + var action = active === "dark" ? text.toLight : text.toDark; + toggle.textContent = action; + toggle.setAttribute("aria-pressed", active === "dark" ? "true" : "false"); + toggle.setAttribute("aria-label", action); + } + resetButton(); + if (announce) { + var status = document.getElementById("action-status"); + if (status) status.textContent = text.status + active + (mode === "auto" ? " (" + text.toSystem + ")" : ""); + } + document.documentElement.setAttribute("data-theme-ready", "true"); + } + + function boot() { + apply(mode, false); + var toggle = document.getElementById("theme-toggle"); + if (toggle) { + toggle.addEventListener("click", function () { + apply(effective() === "dark" ? "light" : "dark", true); + }); + } + if (window.matchMedia) { + var query = window.matchMedia("(prefers-color-scheme: dark)"); + var onChange = function () { + if (mode === "auto") apply("auto", false); + }; + if (typeof query.addEventListener === "function") query.addEventListener("change", onChange); + else if (typeof query.addListener === "function") query.addListener(onChange); + } + document.documentElement.setAttribute("data-theme-bound", "true"); + } + + if (document.readyState === "loading") { + document.addEventListener("DOMContentLoaded", boot); + } else { + boot(); + } +})(); From 05e88268fb92dbefe4fd2e750f9826687b24abca Mon Sep 17 00:00:00 2001 From: zhoutianyi Date: Tue, 15 Sep 2026 13:42:56 +0800 Subject: [PATCH 15/90] =?UTF-8?q?wip(recovery):=20=E6=8A=8A=20read=20?= =?UTF-8?q?=E8=A7=82=E6=B5=8B=E4=B8=8E=20bash=20heredoc=20=E5=B9=B6?= =?UTF-8?q?=E5=85=A5=E6=97=B6=E9=97=B4=E7=BA=BF=E9=87=8D=E5=BB=BA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 第一版只重放了 write/edit（80 个文件），拿不到「由 bash heredoc 建立、之后只用 sed/heredoc 改动」的那些文件。这一版改成**四类事件同一条时间线**： write / edit / heredoc(cat > file <<'EOF') / read(完整读取的观测值) 关键在于把 read 当作时间线上的**状态重置**：文件常被未记录成 write/edit 的 bash 改动推进过，之后的完整读取就是那一刻的真实状态，能纠正重放漂移。同时给 read 与 heredoc 也加了语法校验 —— 审计子代理读到过更早的版本（例如还是 Python 正则 `(?P…)` 的那一版），不加校验会让旧内容覆盖新内容。 配套两个抽取器的修正： - heredoc 的相对路径要按**这条命令的 cwd** 解析（`cd /tmp/dst && cat > src/x.mjs`）， 并按会话过滤掉同一 store 里其它项目的记录 - read 结果末尾的 `(End of file - total N lines)` 不是文件内容，要截掉 结果：distilly 文件 80 → 184；`node --test` 137 → 220 个测试（99 pass / 121 fail）。 还手工修了两处重建损伤：`archive.mjs` 块注释里的 `*/` 提前结束注释（原文是 `*\/`）， `feishu.mjs` 的 Python 命名捕获组 `(?P` → `(?`。 仍缺（下一步）： - `src/knowledge/anchors.mjs` 少了 `buildSubAnchors` 导出（17 个测试失败） - `bin/distilly.mjs` 少了 `payloadEntries`、`scripts/visual-check.mjs` 少了 `anchorProblem` - 5 个测试报 `__TEMPLATE_EN__ is not defined`：viewer 碎片需要模板生成器先跑 --- $H/.hermes/config.yaml | 10 + $R/shim-v/hermes | 6 + .github/workflows/ci.yml | 94 +++- README.md | 363 +++--------- bin/distilly.mjs | 232 ++++---- docs/evidence/pr-01-node-core.md | 53 ++ docs/evidence/pr-02-parse-zero-cred.md | 59 ++ .../pr-09-evidence-spine-blind-test.md | 120 ++++ docs/evidence/pr-10-blind-test-runs.md | 76 +++ docs/evidence/pr-11-attribution.md | 89 +++ docs/evidence/pr-12-feishu-routes-note.md | 84 +++ docs/evidence/pr-13-objective-audit.md | 53 ++ docs/evidence/pr-15-identity.md | 54 ++ docs/evidence/pr-16-discord-notion.md | 55 ++ docs/evidence/pr-17-reddit-gmail.md | 55 ++ docs/evidence/pr-18-blind-identity.md | 37 ++ docs/evidence/pr-19-release-migration.md | 61 ++ docs/v2/ACCEPTANCE.md | 3 +- docs/v2/BLIND-TEST-RUNBOOK.md | 113 ++++ docs/v2/CONTRACT.md | 1 + docs/v2/IDENTITY.md | 43 ++ docs/v2/MIGRATION.md | 1 + docs/v2/STATUS.md | 8 +- dsh-install-probe.mjs | 98 ++++ package.json | 7 +- prompts/correction_handler.md | 1 + prompts/intake.md | 1 + prompts/merger.md | 1 + prompts/persona_analyzer.md | 1 + prompts/persona_builder.md | 1 + prompts/work_analyzer.md | 1 + prompts/work_builder.md | 1 + scripts/acceptance.mjs | 1 + scripts/audit-objective.mjs | 244 ++++++++ scripts/blind-test.mjs | 525 ++++++++++++++++++ scripts/check_release.mjs | 337 ++++++----- scripts/split-corpus.mjs | 112 ++++ scripts/visual-check.mjs | 502 +---------------- src/cli/paths.mjs | 36 ++ src/collect/discord.mjs | 308 ++++++++++ src/collect/feishu-browser.mjs | 303 ++++++++++ src/collect/feishu-mcp.mjs | 460 +++++++++++++++ src/collect/gmail.mjs | 390 +++++++++++++ src/collect/kit.mjs | 309 +++++++++++ src/collect/notion.mjs | 418 ++++++++++++++ src/collect/reddit.mjs | 405 ++++++++++++++ src/commands/credentialed.mjs | 135 +++++ src/commands/harvest.mjs | 202 +++++++ src/commands/migrate.mjs | 110 ++++ src/commands/note.mjs | 202 +++++++ src/commands/parse-chat.mjs | 70 +++ src/commands/parse-email.mjs | 141 +++++ src/commands/parse-shared.mjs | 138 +++++ src/commands/parse-subtitle.mjs | 58 ++ src/commands/retrospect.mjs | 47 ++ src/commands/view.mjs | 163 ++++++ src/hosts/agents.mjs | 25 +- src/knowledge/identity.mjs | 114 ++++ src/knowledge/ledger.mjs | 153 +++-- src/parse/archive.mjs | 4 +- src/parse/email.mjs | 251 +++++++++ src/parse/feishu.mjs | 228 ++++++++ src/parse/office.mjs | 126 +++++ src/parse/subtitle.mjs | 107 ++-- src/skill/migrate.mjs | 118 ++++ tests/audit-objective.test.mjs | 45 ++ tests/blind-test.test.mjs | 193 +++++++ tests/collect-discord-notion.test.mjs | 266 +++++++++ tests/collect-reddit-gmail.test.mjs | 258 +++++++++ tests/command-parse.test.mjs | 115 ++++ tests/doctor-coverage.test.mjs | 100 ++++ tests/feishu-browser.test.mjs | 159 ++++++ tests/feishu-mcp.test.mjs | 238 ++++++++ .../parse/chat/chatgpt-conversations.json | 43 ++ .../parse/chat/claude-conversations.json | 12 + .../fixtures/parse/chat/discord-messages.json | 6 + .../parse/chat/instagram-message_1.json | 9 + tests/fixtures/parse/chat/slack-messages.json | 7 + tests/fixtures/parse/chat/slack-users.json | 5 + .../fixtures/parse/chat/telegram-result.json | 11 + tests/fixtures/parse/subtitle/interview.srt | 13 + tests/fixtures/parse/subtitle/talk.vtt | 18 + tests/fixtures/public-corpus/README.md | 2 +- .../expected/view.template.json | 2 +- tests/identity.test.mjs | 161 ++++++ tests/install-generated-skill.test.mjs | 41 ++ tests/knowledge-anchors.test.mjs | 241 ++++++++ tests/knowledge-ledger.test.mjs | 286 ++++++++++ tests/knowledge-store.test.mjs | 105 ++++ tests/note.test.mjs | 146 +++++ tests/parse-archive.test.mjs | 178 ++++++ tests/parse-chat.test.mjs | 309 +++++++++++ tests/parse-doc.test.mjs | 174 ++++++ tests/parse-email.test.mjs | 225 ++++++++ tests/parse-feishu.test.mjs | 147 +++++ tests/parse-subtitle.test.mjs | 212 +++++++ tests/release-check.test.mjs | 37 ++ tests/release_manifest.test.mjs | 105 ++++ tests/schema-migration.test.mjs | 165 ++++++ tests/test_cli_lifecycle.py | 1 + tests/test_skill_writer.py | 1 + tests/text-attribution.test.mjs | 165 ++++++ tests/visual-check-rule.test.mjs | 32 ++ tools/email_parser.py | 1 + tools/feishu_parser.py | 1 + tools/research/srt_to_transcript.py | 1 + tools/skill_presets.py | 1 + tools/skill_schema.py | 1 + tools/skill_writer.py | 1 + tools/version_manager.py | 1 + 110 files changed, 11551 insertions(+), 1213 deletions(-) create mode 100644 $H/.hermes/config.yaml create mode 100644 $R/shim-v/hermes create mode 100644 docs/evidence/pr-01-node-core.md create mode 100644 docs/evidence/pr-02-parse-zero-cred.md create mode 100644 docs/evidence/pr-09-evidence-spine-blind-test.md create mode 100644 docs/evidence/pr-10-blind-test-runs.md create mode 100644 docs/evidence/pr-11-attribution.md create mode 100644 docs/evidence/pr-12-feishu-routes-note.md create mode 100644 docs/evidence/pr-13-objective-audit.md create mode 100644 docs/evidence/pr-15-identity.md create mode 100644 docs/evidence/pr-16-discord-notion.md create mode 100644 docs/evidence/pr-17-reddit-gmail.md create mode 100644 docs/evidence/pr-18-blind-identity.md create mode 100644 docs/evidence/pr-19-release-migration.md create mode 100644 docs/v2/BLIND-TEST-RUNBOOK.md create mode 100644 docs/v2/IDENTITY.md create mode 100644 dsh-install-probe.mjs mode change 100755 => 100644 scripts/acceptance.mjs create mode 100644 scripts/audit-objective.mjs create mode 100644 scripts/blind-test.mjs create mode 100644 scripts/split-corpus.mjs create mode 100644 src/cli/paths.mjs create mode 100644 src/collect/discord.mjs create mode 100644 src/collect/feishu-browser.mjs create mode 100644 src/collect/feishu-mcp.mjs create mode 100644 src/collect/gmail.mjs create mode 100644 src/collect/kit.mjs create mode 100644 src/collect/notion.mjs create mode 100644 src/collect/reddit.mjs create mode 100644 src/commands/credentialed.mjs create mode 100644 src/commands/harvest.mjs create mode 100644 src/commands/migrate.mjs create mode 100644 src/commands/note.mjs create mode 100644 src/commands/parse-chat.mjs create mode 100644 src/commands/parse-email.mjs create mode 100644 src/commands/parse-shared.mjs create mode 100644 src/commands/parse-subtitle.mjs create mode 100644 src/commands/retrospect.mjs create mode 100644 src/commands/view.mjs create mode 100644 src/knowledge/identity.mjs create mode 100644 src/parse/email.mjs create mode 100644 src/parse/feishu.mjs create mode 100644 src/parse/office.mjs create mode 100644 src/skill/migrate.mjs create mode 100644 tests/audit-objective.test.mjs create mode 100644 tests/blind-test.test.mjs create mode 100644 tests/collect-discord-notion.test.mjs create mode 100644 tests/collect-reddit-gmail.test.mjs create mode 100644 tests/command-parse.test.mjs create mode 100644 tests/doctor-coverage.test.mjs create mode 100644 tests/feishu-browser.test.mjs create mode 100644 tests/feishu-mcp.test.mjs create mode 100644 tests/fixtures/parse/chat/chatgpt-conversations.json create mode 100644 tests/fixtures/parse/chat/claude-conversations.json create mode 100644 tests/fixtures/parse/chat/discord-messages.json create mode 100644 tests/fixtures/parse/chat/instagram-message_1.json create mode 100644 tests/fixtures/parse/chat/slack-messages.json create mode 100644 tests/fixtures/parse/chat/slack-users.json create mode 100644 tests/fixtures/parse/chat/telegram-result.json create mode 100644 tests/fixtures/parse/subtitle/interview.srt create mode 100644 tests/fixtures/parse/subtitle/talk.vtt create mode 100644 tests/identity.test.mjs create mode 100644 tests/knowledge-anchors.test.mjs create mode 100644 tests/knowledge-ledger.test.mjs create mode 100644 tests/knowledge-store.test.mjs create mode 100644 tests/note.test.mjs create mode 100644 tests/parse-archive.test.mjs create mode 100644 tests/parse-chat.test.mjs create mode 100644 tests/parse-doc.test.mjs create mode 100644 tests/parse-email.test.mjs create mode 100644 tests/parse-feishu.test.mjs create mode 100644 tests/parse-subtitle.test.mjs create mode 100644 tests/release-check.test.mjs create mode 100644 tests/release_manifest.test.mjs create mode 100644 tests/schema-migration.test.mjs create mode 100644 tests/text-attribution.test.mjs create mode 100644 tests/visual-check-rule.test.mjs mode change 100755 => 100644 tools/research/srt_to_transcript.py diff --git a/$H/.hermes/config.yaml b/$H/.hermes/config.yaml new file mode 100644 index 00000000..43f64cb9 --- /dev/null +++ b/$H/.hermes/config.yaml @@ -0,0 +1,10 @@ +_config_version: 33 +mcp_servers: + distilly: + command: $W + connect_timeout: 20.0 + enabled: true + bogus_key: 1 + tools: + resources: false + prompts: false diff --git a/$R/shim-v/hermes b/$R/shim-v/hermes new file mode 100644 index 00000000..7b7970f7 --- /dev/null +++ b/$R/shim-v/hermes @@ -0,0 +1,6 @@ +#!/bin/sh +if [ "$1" = "--version" ]; then + echo "Hermes Agent v0.19.1 (2026.8.1)" + exit 0 +fi +exec /tmp/hermes-venv/bin/hermes "$@" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a7d5565f..082269cd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,53 +2,89 @@ name: CI on: push: - branches: [dot-skill, main] + branches: [dot-skill-test, dot-skill, main] pull_request: - branches: [dot-skill, main] + branches: [dot-skill-test, dot-skill, main] jobs: test: - name: Python ${{ matrix.python-version }} + name: Node ${{ matrix.node-version }} runs-on: ubuntu-latest strategy: fail-fast: false matrix: - python-version: ["3.9", "3.11"] + node-version: ["20", "22"] steps: - uses: actions/checkout@v4 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 + - uses: actions/setup-node@v4 with: - python-version: ${{ matrix.python-version }} - cache: pip + node-version: ${{ matrix.node-version }} - - name: Install dependencies + - name: Syntax-check every module run: | - python -m pip install --upgrade pip - if [ -f requirements.txt ]; then pip install -r requirements.txt; fi + find bin src scripts tests -name '*.mjs' -print0 | xargs -0 -n1 node --check - - name: Compile all Python sources - run: python -m compileall -q tools + # `npm test`, not a bare `node --test`: Node's default discovery also + # matches `scripts/blind-test.mjs` (`**/*-test.mjs`), spawns it as a test + # file and records its usage error as a failing test. The script's own + # guard makes that invocation harmless now, but the suite is `tests/`, so + # the command says so — and this step is the same string as `npm test`. + - name: Unit tests + run: npm test - - name: Run unit tests - run: | - if [ -d tests ]; then - python -m unittest discover -s tests -p 'test_*.py' -v - else - echo "No tests/ directory yet — skipping unittest discover." - fi - - lint: - name: Ruff + - name: Prompt contract lint + run: node scripts/prompt-lint.mjs + + - name: Skill template freshness + run: node scripts/generate-template.mjs --check + + # Every demand of the v2 objective, mapped to the artefact that proves it. + # Acceptance itself runs in the next job, so this is scope-only. + - name: Objective audit + run: node scripts/audit-objective.mjs --skip-acceptance + + # Release hygiene: versions, schema marker, carried directories, gates — + # and the packed artifact itself (`npm pack` + run the extracted bin). + - name: Release check + run: node scripts/check_release.mjs + + acceptance: + name: Acceptance (public corpus) runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: actions/setup-python@v5 + + - uses: actions/setup-node@v4 + with: + node-version: "22" + + # The visual-check phase drives a real browser; playwright is a dev + # dependency and never ships with the package. + - name: Install Playwright + run: | + npm install --no-save playwright@1.62.1 + npx playwright install --with-deps chromium + + - name: End-to-end acceptance on the bundled corpus + run: node scripts/acceptance.mjs --evidence "$RUNNER_TEMP/evidence" + + # The second public corpus is multi-source (two chat exports + a document) + # and dated, so the same phases run on a corpus the derivation was designed for. + - name: End-to-end acceptance on the multi-source corpus + run: node scripts/acceptance.mjs --corpus tests/fixtures/public-corpus/synthetic-multisource --evidence "$RUNNER_TEMP/evidence-multisource" + + # Scope audit *with* the acceptance run it normally performs. + - name: Objective audit (includes acceptance) + run: DISTILLY_PLAYWRIGHT_ROOT="$PWD" node scripts/audit-objective.mjs + + - uses: actions/upload-artifact@v4 + if: always() with: - python-version: "3.11" - - name: Install ruff - run: pip install ruff - - name: Run ruff (non-blocking for now) - run: ruff check tools/ || true + name: acceptance-evidence + path: | + ${{ runner.temp }}/evidence + ${{ runner.temp }}/evidence-multisource + if-no-files-found: ignore + diff --git a/README.md b/README.md index b529f9c3..ad46d853 100644 --- a/README.md +++ b/README.md @@ -4,336 +4,139 @@
-# 🧬 Distilly +# Distilly -**Formerly: Colleague Skill / colleague-skill.** +### Distill how they think into Person Profiles for Agents. -### Distill a person's experience, judgment, voice, and ways of working into a reusable Person Profile for AI agents and compatible bots. - -**Messages · documents · interviews · public sources → Distilly → Person Profile → Agent / Bot** +**Colleague Skill / colleague-skill (original name)** [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) -[![Python 3.9+](https://img.shields.io/badge/Python-3.9%2B-blue.svg)](https://python.org) -[![AgentSkills](https://img.shields.io/badge/AgentSkills-Standard-green)](https://agentskills.io) -[![Stars](https://img.shields.io/github/stars/titanwings/colleague-skill?style=social)](https://github.com/titanwings/distilly/stargazers) - -[![Discord](https://img.shields.io/badge/Discord-Join%20Community-5865F2?logo=discord&logoColor=white)](https://discord.gg/NVX66RxWZv) - -
- - - -
- -🧑‍💼  Your colleague quit, your mentor graduated, your teammate transferred — taking their whole playbook and context with them?
-💞  Your family, old friends, partner drifting apart — and you want to hold on to the way it felt to be with them?
-🌟  Your favorite author, idol, thinker you'll never meet — but you want to know what they'd say about your question? - -
- -### ✨ One project, many kinds of people. - -
- -Distilly is the person-modeling layer for agents. It turns the materials you provide into a portable, source-grounded Person Profile built from observable experience, decision patterns, expression, and ways of working; it does not claim to clone the person behind them. - -Colleagues · partners · family · old friends · idols · public figures · fictional characters — even yourself - -**Source material + your description → a source-grounded Person Profile → your Agent or compatible Bot** - -> A Person Profile is the reusable output. The current release packages each profile as an Agent Skill so supported hosts can install and invoke it. The canonical creator Skill is named `distilly`; install it in a `distilly` directory. The former name above remains for search continuity and project history. - -
- -[🆕 What Distilly does](#-what-distilly-does-today) · [📦 Data Sources](#-supported-data-sources) · [⚡ Install](#-install) · [🚀 Usage](#-usage) · [✨ Demo](#-demo) · [📝 Citation](#-citation) · [💬 Discord](https://discord.gg/NVX66RxWZv) - -[**Chinese**](docs/lang/README_ZH.md) · [**Spanish**](docs/lang/README_ES.md) · [**German**](docs/lang/README_DE.md) · [**Japanese**](docs/lang/README_JA.md) · [**Russian**](docs/lang/README_RU.md) · [**Portuguese**](docs/lang/README_PT.md) · [**Korean**](docs/lang/README_KO.md) - - - ---- - -
- -### 🎉 2026.08.13 Milestone — **the project has passed 20K ⭐!** - -Massive thanks to everyone who starred — we'll keep shipping, keep distilling. - -
- -> 🧬 **2026.08.24 Update** — The creator is now named **Distilly** end to end and documents native local Skill discovery for Claude Code, Hermes, OpenClaw, Codex, DeepSeek Harness, Pi, Grok Build, and OpenCode. Grok Bot is listed separately as a saved-Skill workflow preview. - -> 📝 **2026.06.01 Update** — **[The COLLEAGUE.SKILL technical report](https://arxiv.org/pdf/2605.31264) is now available**. The most rewarding part was not simply publishing a paper, but seeing the community grow the gallery to 215 skills contributed by 165 people, with more than 100,000 stars across the skill cards. The paper's Acknowledgements explicitly recognize every community contributor. - -> 🗺️ **2026.04.13** — **The Distilly Roadmap is live!** What began as Colleague Skill is growing beyond colleagues: distill people into Skills that Agents can reuse. 👉 **[Full Roadmap](ROADMAP.md)** · **[💬 Discord](https://discord.gg/NVX66RxWZv)** - -> 🌐 **2026.04.07** — Community gallery is live! Any skill / meta-skill can drive traffic directly to your own GitHub repo. No middleman. 👉 **[titanwings.github.io/colleague-skill-site](https://titanwings.github.io/colleague-skill-site/)** - -
- -Created by [@titanwings](https://github.com/titanwings) +[![Node.js](https://img.shields.io/badge/Node.js-22.19%2B-339933.svg)](https://nodejs.org/) +[![Codex Preview](https://img.shields.io/badge/Codex-Developer%20Preview-black)](https://github.com/titanwings/distilly/tree/distilly-plugin)
---- - -## 🆕 What Distilly does today - -### 1️⃣ From Colleague Skill to Distilly - -The project is no longer limited to the colleague scenario. Its `distilly` creator builds source-grounded Person Profiles for three person families with one workflow, then packages each profile as an Agent Skill. - -### 2️⃣ Three character families - - - - - - - - - - - - - - - - - - - - - -
🧑‍💼 colleague💞 relationship🌟 celebrity
Coworkers · mentors · teammates · up/downstream partnersExes · partners · parents · friends · close familyPublic figures · creators · public voices · fictional characters
Builds a Work Skill + Persona from material-derived technical standards, workflows, expression, and workplace behavior. Supports Lark / DingTalk / Slack collection.Organizes material-derived expression patterns, emotional triggers, conflict patterns, and repair patterns into a reusable Persona Skill.Ships with a six-dimension research toolchain (subtitles → transcript cleanup → research merge → quality check) for organizing observable decisions, expression, and mental models.
- -Each family has its own source-collection strategy, analysis dimensions, and Person Profile structure. - -### 3️⃣ More Agent hosts - -The old version only ran in Claude Code. Distilly now supports native local Skill discovery across eight agent hosts. - - - - - - - - - - - - - - -
Claude CodeHermes AgentOpenClawCodex
DeepSeek HarnessPi coding agentGrok BuildOpenCode
- -**Grok Bot preview:** Grok Bot supports saved/private Skills, but its official docs do not describe direct local `SKILL.md` imports. Distilly's workflow can be migrated manually into a saved Skill; direct repo installation is not yet verified. +Distilly is a local-first product for turning a person's source material, working habits, judgment, and voice into a versioned **Person Profile for Agents**. The profile can be recalled temporarily during a run or explicitly installed as a long-lived host Skill. The storage authority stays local; no additional model API key is required. -Each generated Person Profile is packaged as an Agent Skill and can be installed into any supported host. +This `distilly-plugin` branch carries the unreleased `0.1.0-preview.1` Developer Preview; the repository's default branch is `dot-skill`, the separate legacy implementation, so a bare clone lands on that line instead of this one. Codex, OpenClaw `2026.3.24`, and Hermes `v0.9.0` each have an immutable real-host transport-capacity fixture. The OpenClaw and Hermes measurements use a deterministic synthetic fixture server through the real host executable, model, and MCP transport; they do not by themselves certify packaged restart or the full product lifecycle. Setup remains fail-closed for any unrecorded host version or changed release tuple. This branch is not a tagged release or an npm package yet. ---- +[Chinese](docs/lang/README_ZH.md) · [Español](docs/lang/README_ES.md) · [Deutsch](docs/lang/README_DE.md) · [日本語](docs/lang/README_JA.md) · [한국어](docs/lang/README_KO.md) · [Português](docs/lang/README_PT.md) · [Русский](docs/lang/README_RU.md) -## 📦 Supported Data Sources +## Install the Developer Preview -| Logo | Source | Messages | Docs / Wiki | Notes | -|:----:|--------|:--------:|:-----------:|-------| -| Lark | Lark (auto) | ✅ API | ✅ | Just enter a name, fully automatic | -| DingTalk | DingTalk (auto) | ⚠️ Browser | ✅ | DingTalk API doesn't support message history | -| Slack | Slack (auto) | ✅ API | — | Requires admin to install Bot; free plan limited to 90 days | -| X | Public X posts | ✅ API | — | Optional, bounded celebrity research candidates through metered third-party service Xquik | -| WeChat | WeChat chat history | ✅ SQLite | — | Export first with WeChatMsg or PyWxDump | -| 📄 | PDF / Images / Screenshots | — | ✅ | Manual upload | -| Lark | Lark JSON export | ✅ | ✅ | Manual upload | -| ✉️ | Email `.eml` / `.mbox` | ✅ | — | Manual upload | -| 📝 | Markdown / direct paste | ✅ | ✅ | Manual input | +### For an agent ---- +Give your coding agent the following task and let it run the commands in a fresh checkout: -## ⚡ Install +> Install the Distilly Developer Preview from the `distilly-plugin` branch, build it with Node 22.19+ (or Node 24), run `distilly setup --host codex`, run `distilly doctor --host codex`, and report the result. Do not modify another branch. -### 🤖 For Agents +The exact checkout and setup commands are shown below so the agent can verify every step. -Open any supported local Agent host and send: +### For a human -> Install Distilly from `https://github.com/titanwings/distilly`, then verify that this host can discover it. - -The Agent installs Distilly as a Skill named `distilly` in the correct host directory. - -### 👤 For Humans - -Clone Distilly into the Skills directory used by your host: +Requirements: Node.js `22.19+` or `24`, pnpm `10.32+`, and a locally installed Codex CLI. From a terminal: ```bash -git clone https://github.com/titanwings/distilly +git clone --branch distilly-plugin https://github.com/titanwings/distilly.git +cd distilly +corepack enable +pnpm install --frozen-lockfile +pnpm run build +node packages/cli/lib/bin.js setup --host codex +node packages/cli/lib/bin.js doctor --host codex ``` -Host paths, migration, Windows, generated-profile installation, and credential setup are in the **[Install Guide](INSTALL_EN.md)**. - ---- - -## 🚀 Usage - -In your Agent, say: - -> Use Distilly to create a Person Profile for ``. - -Then: - -1. Choose `colleague`, `relationship`, or `celebrity`. -2. Add a description or source material. Every field is optional. -3. Review the result and let Distilly generate the profile. - -The result is packaged as an Agent Skill named `{character}-{slug}`. For the optional celebrity research pipeline and advanced tools, see the **[Install Guide](INSTALL_EN.md)**. - ---- +Restart Codex after setup. The launcher registers the self-contained Plugin and its five MCP tools. To remove the host integration while keeping all local people, profiles, and source data: -## ✨ Demo - -One from each family. - -
- -### 🌟 celebrity — distilling Andrej Karpathy - -> Six-dimension research: works · interviews · decisions · expression DNA · external evaluations · timeline
-> Case adapted from community work [alchaincyf/karpathy-skill](https://github.com/alchaincyf/karpathy-skill) - -``` -User ❯ Everyone's talking about Agents now — what do you think the 2026 AI Agent looks like? - -celebrity.skill ❯ Everyone's doing demos, nobody's doing eval. - The Agent bottleneck was never model intelligence — it's scaffolding: - a strict evaluation harness, fine-grained error replay, - controllable tool-calling budgets, clean failure-mode analysis. - Most products are still stuck at "wrap an API + a few tools + a system prompt." - That's a demo, not an agent. - In 2026 I'm not betting on the strongest model — I'm betting on the best - scaffolding and the strictest eval. +```bash +node packages/cli/lib/bin.js uninstall --host codex ``` -
- -### 🧑‍💼 colleague — distilling a ByteDance L2-1 backend - -> Input: `ByteDance L2-1 backend engineer, INTJ, blame-shifter, ByteDance-style` +To install one approved profile as a persistent Skill after a profile has been created, use its exact subject id: +```bash +node packages/cli/lib/bin.js install subject_<32 lowercase hex characters> --host codex ``` -User ❯ Can you review this API design? -colleague.skill ❯ Hold on — what's the impact? You haven't explained the context. - (after reading) N+1 query, fix it. Use the standard - {code, message, data} response format. That's the spec, - don't ask why. +## Host compatibility and explicit Legacy fallback -User ❯ This bug was introduced by you, right? +Codex uses the native Plugin preview above. The Preview also includes compatibility bindings for OpenClaw and Hermes: -colleague.skill ❯ Does the timeline match? That feature touched multiple places, - there were other changes too. -``` +- **OpenClaw** loads the Claude-compatible bundle from `~/.openclaw/extensions/distilly` and its real `.mcp.json`. Check discovery with `openclaw plugins inspect distilly --json`. +- **Hermes** installs the canonical Skill at `~/.hermes/skills/distilly`, a managed wrapper at `~/.distilly/bin/distilly-hermes`, and an MCP entry in `~/.hermes/config.yaml`. `resources` and `prompts` are disabled so the exposed surface remains five tools; check it with `hermes mcp test distilly`. -
+The CLI recognizes both hosts and enables setup when the installed version matches the recorded real-host transport fixture. The current net budgets, measured in isolated clean sessions with `openai-codex/gpt-5.4`, are 65,536 serialized bytes for OpenClaw and 49,752 for Hermes (the same conservative byte/token accounting used by the Codex fixture). These are transport/value lower bounds for the recorded probe, not a guarantee of remaining context in every model or user session. Any unrecorded version, release digest, tool descriptor, or serializer tuple returns `host_unsupported` before writing an unverified integration. There is no automatic switch to the legacy implementation. -### 💞 relationship — distilling someone you have a crush on +Until a host has a verified Plugin binding, you can explicitly choose the maintained `dot-skill` branch as a **Legacy Skill compatibility mode**: -> Upload half a year of chat logs + "sensitive, quiet but stubborn, will actually reply seriously when it matters" +> Install Distilly in Legacy Skill compatibility mode from the `dot-skill` branch into this host's normal Skills directory, using a clean checkout whose final directory is named `distilly`. Verify discovery and report the installed Git commit. Do not run Plugin setup or claim SQLite, five-tool MCP, Panel, or Plugin lifecycle support. -``` -User ❯ Did you think about me today? +For a manual install, replace `` with the complete final path in the [detailed install guide](INSTALL.md), including the last `distilly` component, and create its parent first: -relationship.skill ❯ ...I did, a little bit. Why are you asking? +```bash +git clone --single-branch --branch dot-skill --depth 1 \ + https://github.com/titanwings/distilly.git \ + +git -C rev-parse HEAD ``` -
- -📚 More real-world cases in the **[community gallery](https://titanwings.github.io/colleague-skill-site/)** — 100+ skills and counting - -
- ---- - -## 🔧 Features +This is an explicit, separate file-based implementation—not an automatic runtime fallback. It does not share a supported data model with the Plugin, and a failed Plugin preflight never switches modes. The compatibility promise currently covers local files and pasted text only. Do not enable legacy collectors while the Plugin uses the same home directory: current legacy collectors can write credential configuration into the same `~/.distilly/` namespace and remain outside the Preview's reviewed security boundary. Keep exactly one `distilly` active in any host discovery scope and verify which copy the host loaded. -### 🧱 Generated Skill Structure +## The first usable flow -Distilly's current creator uses **Persona** as the universal base, with family-specific modules layered on top: +On Codex, the complete flow below is verified. OpenClaw `2026.3.24` and Hermes `v0.9.0` have the same briefing transport path verified against their recorded capacity fixture; their packaged restart, long-lived Skill, and uninstall lifecycle checks remain separate. Restart the selected host and ask it to research and distill a person. Supply only the files, text, or public URLs you want included. Distilly then: -| Family | Persona Content | Additional Modules | -|--------|-----------------|-------------------| -| 🧑‍💼 **colleague** | 6-layer personality: hard rules → identity → expression → decisions → interpersonal → Correction | ➕ **Work Skill**: scope, workflow, output preferences, experience knowledge base | -| 💞 **relationship** | Expression DNA · emotional triggers · conflict pattern · repair pattern | — | -| 🌟 **celebrity** | Mental models · decision heuristics · expression DNA · external-evaluation contrast | ➕ Six-dimension research dossier (works / interviews / decisions / timeline...) | +1. resolves or creates the person; +2. imports the selected material with deterministic local parsers; +3. creates a pending research job and a complete evidence-bound briefing; +4. commits a versioned Person Profile; +5. returns the profile or a complete temporary prompt for the current run; +6. accepts an explicit correction and sends a candidate to review; +7. lets you promote, reject, or roll back the candidate in the local Panel; and +8. installs the approved profile as a self-contained host Skill when you ask it to. -> **Execution**: Receive task → Persona selects material-derived preferences and tone → Additional modules fill in execution detail → Produce a source-grounded response +The model-facing surface remains exactly five MCP tools: -### 🧬 Evolution +`distilly_get` · `distilly_ingest` · `distilly_pending` · `distilly_commit` · `distilly_correct` -- 📥 **Append files** → auto-analyze delta → merge into relevant sections, never overwrite existing conclusions -- 💬 **Conversation correction** → say "they wouldn't do that, they'd be xxx" → writes to the Correction layer, takes effect immediately -- 🕰️ **Version control** → auto-archive on every update, rollback to any previous version -- 🔬 **Celebrity research pipeline** → subtitles → transcript cleanup → six-dimension research → quality check +Distilly never silently truncates a complete briefing or profile prompt. If a verified host budget cannot carry the complete value, it reports a bounded capacity error with measurements and keeps the stored data unchanged. ---- +## Host status -## ⚠️ Notes +| Host | Native Plugin | Current compatibility route | +| --- | --- | --- | +| Codex | Fully verified in this release branch | Native Plugin | +| Claude Code | Binding included; exact host fixture still needed | Explicit `dot-skill` Legacy Skill | +| OpenClaw | Transport-capacity fixture recorded for `2026.3.24` (65,536-byte net budget); lifecycle pending | Claude-compatible bundle + discovery smoke | +| Hermes | Transport-capacity fixture recorded for `v0.9.0` (49,752-byte net budget); lifecycle pending | Managed Skill + MCP configuration | +| DeepSeek Harness (DSH) | Community binding planned | Explicit `dot-skill` Legacy Skill | +| Pi agent | Community binding planned | Explicit `dot-skill` Legacy Skill | +| Grok Build | Community binding planned | Explicit `dot-skill` Legacy Skill | +| OpenCode | Community binding planned | Explicit `dot-skill` Legacy Skill | +| Grok Bot | Community binding planned | Manual saved/private Skill only; local repository import is not claimed | -**Source material quality = Person Profile quality** — and quality sources differ across families: +Host compatibility is a binding concern. Legacy Skill discovery is useful continuity, but it does not make a host a verified Plugin target. -| Family | Source priority (high → low) | -|--------|------------------------------| -| 🧑‍💼 **colleague** | Their **own long-form writing** (design docs / review comments) **›** **decision-making replies** **›** casual group chat | -| 💞 **relationship** | Complete chat history **›** letters / social posts / diaries **›** third-party descriptions | -| 🌟 **celebrity** | First-person books / blogs / long interviews **›** decision records (launches, commits, Q&A) **›** verified first-person short-form posts **›** third-party commentary | +## Local material formats -- **colleague** Lark-compatible auto-collection: requires adding the App bot to relevant group chats -- **relationship**: longer time spans are better; material covering both conflict and repair is ideal -- **celebrity**: avoid feeding only second-hand interpretations -- This is still a demo version — please file issues if you find bugs! +The first Preview accepts explicit local `TXT`, `Markdown`, `JSON`, and `SRT/VTT` files. It also accepts pasted text and public URLs through the host's visible research flow. Files are read only from the paths or sources the user supplies; symlinked selected files and duplicate file names are rejected. PDF, email, provider exports, and hosted connectors are follow-up work. ---- +## 📣 2026-09 update: help expand coding-agent Plugins -## 📄 Technical Report +Codex, OpenClaw, and Hermes now have real host/version capacity fixtures. We need community support to provide the same evidence for **Claude Code, DeepSeek Harness (DSH), Pi agent, Grok Build, OpenCode, and Grok Bot**, then to build and validate their coding-agent Plugin packages. I will actively review those contributions and keep the public contracts, release digests, and host behavior aligned. -> **[COLLEAGUE.SKILL: Automated AI Skill Generation via Expert Knowledge Distillation](https://arxiv.org/pdf/2605.31264)** ([arXiv](https://arxiv.org/abs/2605.31264) · [arXiv PDF](https://arxiv.org/pdf/2605.31264)) -> -> This is the paper for **COLLEAGUE.SKILL / colleague-skill**, Distilly's predecessor. It covers the Work Skill + Persona two-layer architecture, multi-source data collection, and Skill generation mechanics — the theoretical foundation for today's `colleague` family. Separate papers on the relationship / celebrity family extensions are planned. +See the full call for contributors in [UPDATES.md](UPDATES.md) and the current priorities in [ROADMAP.md](ROADMAP.md). ---- +## Project documents -## 📝 Citation +- [Detailed Preview installation](INSTALL.md) +- [Changelog](CHANGELOG.md) +- [Architecture and shipped-state map](docs/architecture.md) +- [Testing contract](docs/testing.md) +- [Development workflow](docs/development.md) +- [Design corpus](docs/design/README.md) +- [Release manifest](plugins/release-manifest.json) +- [Contributing](CONTRIBUTING.md) -If you use **Distilly** or **COLLEAGUE.SKILL** in your research or applications, please cite the technical report: +Distilly is released under the [MIT License](LICENSE). Created by [@titanwings](https://github.com/titanwings). -```bibtex -@misc{zhou2026colleagueskill, - title = {COLLEAGUE.SKILL: Automated AI Skill Generation via Expert Knowledge Distillation}, - author = {Tianyi Zhou and Dongrui Liu and Leitao Yuan and Jing Shao and Xia Hu}, - year = {2026}, - eprint = {2605.31264}, - archivePrefix = {arXiv}, - primaryClass = {cs.AI}, - url = {https://arxiv.org/abs/2605.31264} -} -``` - -You can also use the machine-readable citation metadata in [CITATION.cff](CITATION.cff). - ---- - -## ⭐ Star History - - - - - - Star History Chart - - - ---- - -
- -**MIT License** © [titanwings](https://github.com/titanwings) - -
diff --git a/bin/distilly.mjs b/bin/distilly.mjs index a5bc9be0..0ce75ee7 100644 --- a/bin/distilly.mjs +++ b/bin/distilly.mjs @@ -1,149 +1,107 @@ #!/usr/bin/env node +// 验收脚本自测用的最小实现（不属于仓库，只在 /tmp 下临时目录里）。 +import { createHash } from 'node:crypto'; +import { mkdir, readFile, readdir, writeFile } from 'node:fs/promises'; +import path from 'node:path'; + /** - * Distilly entry point. - * - * Contract: `docs/v2/CONTRACT.md` §1 — this is the only user-facing entry. It - * parses global flags, resolves a subcommand through the registry in - * `src/commands/index.mjs`, prints a bilingual help screen, and always answers - * with the receipt shape from §3 when `--json` is set. - * - * Adding a command: create `src/commands/.mjs`, call `register(...)` from - * it, and import that module below. See `docs/v2/NODE-CORE.md`. + * Credentialed commands (`collect` / `consent` / `transcribe`) are dispatched + * here, in their own block: the install path above is untouched, and each module + * owns its flags, receipts and exit codes (2 means "waiting for user consent"). */ - -import { existsSync, readFileSync } from "node:fs"; -import { join, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; - -import "../src/commands/skill.mjs"; -import "../src/commands/install.mjs"; -import "../src/commands/doctor.mjs"; -import "../src/commands/legacy.mjs"; - -import { ArgError, wantsHelp } from "../src/cli/args.mjs"; -import { CliError, createReceipt, createReporter } from "../src/cli/receipt.mjs"; -import { lookup, missingCommandError, renderHelp, resolveCommand } from "../src/commands/index.mjs"; - -export const packageRoot = fileURLToPath(new URL("..", import.meta.url)); -const packageMetadata = JSON.parse(readFileSync(join(packageRoot, "package.json"), "utf8")); -const version = packageMetadata.version; -const binary = "distilly"; - -/** Files/directories copied into a host by `install `. */ -export const payloadEntries = [ - "SKILL.md", - "prompts", - "references", - "bin", - "src", - "assets", - "scripts", - "package.json", - "INSTALL.md", - "INSTALL_EN.md", - "LICENSE", - "CITATION.cff", -]; - -/** Prepack guard: the published payload must be complete and version-consistent. */ -export function validatePayload(root = packageRoot) { - const missing = payloadEntries.filter((entry) => !existsSync(join(root, entry))); - if (missing.length > 0) { - throw new CliError(`package payload is missing: ${missing.join(", ")}`, { - code: "payload-incomplete", - remedy: "restore the missing paths or update payloadEntries in bin/distilly.mjs.", - }); +const collectChannels = { + feishu: () => import("../src/collect/feishu.mjs"), + slack: () => import("../src/collect/slack.mjs"), + dingtalk: () => import("../src/collect/dingtalk.mjs"), + x: () => import("../src/collect/x.mjs"), +}; + +async function runCredentialedCommand(commandArgs) { + const [command, ...rest] = commandArgs; + if (command === "consent") { + const { runConsentCli } = await import("../src/consent.mjs"); + return runConsentCli(rest); } - - const skill = readFileSync(join(root, "SKILL.md"), "utf8"); - if (!skill.includes(`version: "${version}"`)) { - throw new CliError("package.json version does not match SKILL.md", { - code: "version-mismatch", - remedy: `set SKILL.md frontmatter version to "${version}" (or bump package.json).`, - }); + if (command === "transcribe") { + const { runTranscribeCli } = await import("../src/optional/transcribe.mjs"); + return runTranscribeCli(rest); } -} - -function failureReceipt(command, error) { - return createReceipt(command, { - ok: false, - error: { - code: error.code ?? "error", - message: error.message, - ...(error.remedy ? { remedy: error.remedy } : {}), - }, - warnings: [error.message], - }); -} - -async function main(argv) { - // `--json` is global (CONTRACT §1): every subcommand answers with a receipt. - const json = argv.includes("--json"); - const args = argv.filter((arg) => arg !== "--json"); - const reporter = createReporter(json); - if (args.includes("--check-package")) { - validatePayload(); - // A validation diagnostic, not command output: `prepack` shares stdout with - // `npm pack --json`, which must stay parseable. - process.stderr.write("Distilly package payload is valid.\n"); - return 0; - } - - if (args.includes("--version")) { - reporter.line(version); - return 0; + const [channel, ...channelArgs] = rest; + const known = Object.keys(collectChannels).join("|"); + if (!channel || channel === "--help" || channel === "help") { + console.log(`Usage: distilly collect <${known}> [options] [--json]`); + console.log("Run `distilly collect --help` for the per-channel options."); + return channel ? 0 : 1; } + const load = collectChannels[channel]; + if (!load) fail(`unsupported channel: ${channel} (known: ${Object.keys(collectChannels).join(", ")})`); + const { runCollectCli } = await load(); + return runCollectCli(channelArgs); +} - if (args.length === 0 || args[0] === "help" || wantsHelp(args)) { - process.stdout.write(renderHelp({ version, binary })); - return 0; +const args = process.argv.slice(2); +const cmd = args[0]; +const opt = (name, fallback) => { const i = args.indexOf(`--${name}`); return i === -1 ? fallback : args[i + 1]; }; +const person = opt('person', 'lin-gong'); +const dir = path.join(process.cwd(), 'skills', 'colleague', person); +const sha = (b) => createHash('sha256').update(b).digest('hex'); +const receipt = (extra) => console.log(JSON.stringify({ command: `${cmd}`, person, ok: true, inputs: [], outputs: [], warnings: [], ...extra }, null, 2)); + +if (cmd === 'harvest') { + const src = args[1]; + const rawDir = path.join(dir, 'knowledge', 'raw'); + const textDir = path.join(dir, 'knowledge', 'text'); + await mkdir(rawDir, { recursive: true }); await mkdir(textDir, { recursive: true }); + const name = path.basename(src); + const bytes = await readFile(path.join(src, 'transcript.srt')); + const rawPath = path.join(rawDir, name); + await writeFile(rawPath, bytes); + const cues = bytes.toString('utf8').trim().split(/\n\n+/).map((c, i) => ({ n: i + 1, text: c.split('\n').slice(2).join(' ') })); + const anchors = []; + const md = cues.map((c, i) => { const a = `k${String(i + 1).padStart(4, '0')}`; anchors.push(a); return `[${a}] ${c.text}`; }).join('\n\n'); + const textPath = path.join(textDir, `${name}.md`); + await writeFile(textPath, md, 'utf8'); + const ledgerPath = path.join(dir, 'knowledge', 'index.json'); + let ledger = []; + try { ledger = JSON.parse(await readFile(ledgerPath, 'utf8')); } catch { /* new */ } + const digest = sha(bytes); + if (!ledger.some((e) => e.sha256 === digest)) { + ledger.push({ id: 'k-src-1', kind: 'subtitle', origin: name, fetched_at: '2026-09-13T00:00:00Z', bytes: bytes.length, sha256: digest, credentialed: false, method: 'local', warnings: [], anchors }); + await writeFile(ledgerPath, JSON.stringify(ledger, null, 2)); } - - const { name, rest } = resolveCommand(args); - const command = lookup(name); - if (!command) throw missingCommandError(name); - - if (wantsHelp(rest)) { - process.stdout.write(`${command.help ?? command.usage}\n`); - return 0; + receipt({ inputs: [{ path: rawPath, sha256: digest, bytes: bytes.length }], outputs: [{ path: textPath, sha256: sha(Buffer.from(md)), bytes: Buffer.byteLength(md) }], anchors: { total: anchors.length } }); +} else if (cmd === 'retrospect') { + const ledger = JSON.parse(await readFile(path.join(dir, 'knowledge', 'index.json'), 'utf8')); + const anchors = ledger.flatMap((e) => e.anchors ?? []); + const derDir = path.join(dir, 'evidence', 'derived'); + await mkdir(derDir, { recursive: true }); + const kinds = ['stats', 'voice', 'relations', 'timeline', 'boundaries', 'shifts', 'conflicts']; + const outputs = []; + for (const kind of kinds) { + const body = JSON.stringify({ kind, claims: [{ id: `${kind}.sample`, value: 1, confidence: 'high', evidence: anchors.slice(0, 2) }] }, null, 2); + const p = path.join(derDir, `${kind}.json`); + await writeFile(p, body); + outputs.push({ path: p, sha256: sha(Buffer.from(body)), bytes: Buffer.byteLength(body) }); } - - const result = (await command.run({ - argv: rest, - json, - reporter, - ctx: { packageRoot, version, binary }, - })) ?? {}; - - const receipt = result.receipt ?? createReceipt(name); - reporter.finish(receipt); - if (result.exitCode !== undefined) return result.exitCode; - return receipt.ok === false ? 1 : 0; -} - -// Dispatch only when this file is the process entry point. Importing it (tests do, -// to reach `validatePayload` and `payloadEntries`) must not run a command with the -// importer's argv — the same guard `scripts/visual-check.mjs` and -// `scripts/blind-test.mjs` already carry. -const isEntryPoint = - process.argv[1] !== undefined && resolve(process.argv[1]) === fileURLToPath(import.meta.url); - -if (isEntryPoint) { - try { - process.exitCode = await main(process.argv.slice(2)); - } catch (error) { - const json = process.argv.includes("--json"); - const command = process.argv.slice(2).find((arg) => !arg.startsWith("-")) ?? null; - const reporter = createReporter(json); - const failure = - error instanceof CliError || error instanceof ArgError - ? error - : new CliError(error?.message ?? String(error), { code: "unexpected" }); - - reporter.warn(`Error: ${failure.message}`); - if (failure.remedy) reporter.warn(`Remedy: ${failure.remedy}`); - reporter.finish(failureReceipt(command, failure)); - process.exitCode = failure.exitCode ?? 1; + receipt({ outputs }); +} else if (cmd === 'view') { + const sub = args[1]; + const viewPath = path.join(dir, 'views', `${person}.view.json`); + const view = JSON.parse(await readFile(viewPath, 'utf8')); + const ledger = JSON.parse(await readFile(path.join(dir, 'knowledge', 'index.json'), 'utf8')); + const known = new Set(ledger.flatMap((e) => e.anchors ?? [])); + if (sub === 'check') { + const bad = []; + for (const s of view.sections ?? []) for (const c of s.claims ?? []) for (const a of c.anchors ?? []) if (!known.has(a)) bad.push(a); + if (bad.length) { console.error(JSON.stringify({ ok: false, code: 'view/orphan-anchor', bad })); process.exit(1); } + receipt({}); + } else { + const htmlPath = path.join(dir, 'views', `${person}.html`); + const html = `${view.meta?.title ?? person}

${view.headline?.text ?? ''}

${(view.sections ?? []).map((s) => `

${s.title}

${(s.claims ?? []).map((c) => `

${c.text} ${(c.anchors ?? []).join(' ')}

`).join('')}
`).join('')}`; + await writeFile(htmlPath, html); + receipt({ outputs: [{ path: htmlPath, sha256: sha(Buffer.from(html)), bytes: Buffer.byteLength(html) }] }); } +} else { + console.error(`unknown command ${cmd}`); process.exit(2); } diff --git a/docs/evidence/pr-01-node-core.md b/docs/evidence/pr-01-node-core.md new file mode 100644 index 00000000..6797d69e --- /dev/null +++ b/docs/evidence/pr-01-node-core.md @@ -0,0 +1,53 @@ +# PR-01 · Node 单栈基座：入口 CLI + Skill 内核 + 安装器 + 拼音 + 测试移植 + +- 分支：`ds/01-node-core`（12 个提交，已本地合并进 `dot-skill-test`） +- 依赖：无。这一条是并行工作的基座：契约、命令注册表、验收脚本都由它落下来 +- 交付：38 个文件 / +11 050 行 + +## 1. 变更 + +| # | 提交 | 内容 | +| --- | --- | --- | +| 1 | `c383fca` | `src/skill/writer.mjs` + `src/skill/slug.mjs`：Python `skill_writer.py` 的 Node 移植，含拼音 slug | +| 2 | `3337c39` | `src/skill/versions.mjs`：版本归档（list / backup / rollback / cleanup），归档时间戳按 UTC 固定 | +| 3 | `579b159` | 归档列表确定性：同一天两次 `version list` 输出一致 | +| 4 | `f94675d` | `scripts/parity.mjs`：与迁移前 Python 的**逐字节 parity** 证据（按 rev 跑，不进日常门禁） | +| 5 | `441ceaf` | `src/commands/skill.mjs`：`skill create\|update\|list\|version` 接进注册表 | +| 6 | `9ebfa6e` | `src/install/hosts.mjs`：8 个宿主安装器合并成一个模块（路径矩阵单一出处） | +| 7 | `3436730` | 安装器测试移植到 `node --test`（claude / codex / openclaw / hermes） | +| 8 | `43c837e` | 已注册命令的 `--help` 打印双语两段 | +| 9 | `f6dcf87` | `listCommands()` 暴露命令名（供 doctor / prompt-lint / 审计共用） | +| 10 | `f97bc0c` | `install` / `uninstall` / `doctor` / `legacy` 适配器（旧 `python3 tools/*.py` 调用转发 + deprecation 警告） | +| 11 | `28b0c32` | `assets/pinyin.json`：从 Unihan 生成，去掉 `pypinyin` 运行时依赖 | +| 12 | `1da31aa` | 其余 Python 测试套件移植为 `node --test` | + +新增文件（节选）：`src/commands/{index,skill,install,doctor,legacy}.mjs`、`src/cli/{args,receipt}.mjs`、 +`src/skill/{writer,presets,schema,slug,versions}.mjs`、`src/install/hosts.mjs`、`src/hosts/agents.mjs`、 +`assets/pinyin.json`、`scripts/{generate-pinyin,parity,acceptance}.mjs`、`docs/v2/{CONTRACT,ACCEPTANCE,STATUS}.md`、 +`tests/{dispatcher,commands,cli-lifecycle,skill-writer,pinyin-slug,install-*}.test.mjs`、 +公开语料夹具 `tests/fixtures/public-corpus/synthetic-interview/**`。 + +## 2. 验收（当前树，可复算） + +```bash +node --test tests/dispatcher.test.mjs tests/commands.test.mjs tests/cli-lifecycle.test.mjs \ + tests/skill-writer.test.mjs tests/pinyin-slug.test.mjs tests/install-*.test.mjs +# 33 个测试文件、330 个 test() 块：node --test tests/*.test.mjs → 340 pass / 0 fail +node bin/distilly.mjs --help # 22 个命令名，全部有中英两段 +node bin/distilly.mjs doctor # 宿主矩阵 8 个宿主，逐个报告是否已安装 +node scripts/parity.mjs # 历史 parity 证据（需要旧 rev） +``` + +要点：**零运行时依赖**（`package.json` 无 dependencies）；入口唯一（`bin/distilly.mjs`）； +`--json` 在任何命令上只输出一个对象（`tests/dispatcher.test.mjs` 断言）； +命令注册表是唯一注册点，两段式命令名优先（`skill create` 赢过 `skill`）。 + +## 3. 回滚 + +- 逐提交可 revert；`src/commands/legacy.mjs` 单独 revert 会让旧 `tools/*.py` 调用直接报未知命令。 +- `assets/pinyin.json` 是生成物：`node scripts/generate-pinyin.mjs` 可重现（`--check` 防漂移）。 + +## 4. 已知缺口 + +- `scripts/parity.mjs` 需要一份迁移前的 rev 才能跑：parity 是历史证据，不是日常门禁。 +- 这一条只交付 skill/install/doctor 内核；`harvest` / `parse-*` / `view` / `collect` 由 #02/#03/#07 交付。 diff --git a/docs/evidence/pr-02-parse-zero-cred.md b/docs/evidence/pr-02-parse-zero-cred.md new file mode 100644 index 00000000..b962cd52 --- /dev/null +++ b/docs/evidence/pr-02-parse-zero-cred.md @@ -0,0 +1,59 @@ +# PR-02 · 零凭据解析：`knowledge/` 账本 + 锚点 + chat/subtitle/archive 解析 + +- 分支：`ds/02-parse-zero-cred`（6 个提交，已本地合并进 `dot-skill-test`） +- 依赖：契约（`docs/v2/CONTRACT.md`，由 #01 冻结） +- 交付：31 个文件 / +5 905 行 + +## 1. 变更 + +| # | 提交 | 内容 | +| --- | --- | --- | +| 1 | `05ff594` | 冻结 v2 契约：磁盘布局、回执形状、密钥纪律、computer-use 同意 | +| 2 | `6f95373` | `src/knowledge/store.mjs`：`knowledge/raw/` 字节保险库（逐字落盘 + 读回校验 + 路径逃逸拒绝） | +| 3 | `66bc96f` | `src/knowledge/{anchors,ledger}.mjs`：段落锚点分配、只增账本、`units`/`anchor_detail`、去重（sha256 + origin） | +| 4 | `34e20a3` | `src/parse/subtitle.mjs`：`.srt`/`.vtt` → 一条 cue 一个锚点单元（含说话人、时间码、字节区间） | +| 5 | `1fd53af` | `src/parse/common.mjs`：零依赖读共享 zip 容器（中央目录、CRC 校验、成员流式解压） | +| 6 | `6c42d88` | `src/parse/chat.mjs`：ChatGPT / Claude / Slack / Telegram / Discord / Instagram 导出，其余按名拒绝 | + +新增文件：`src/knowledge/{store,anchors,ledger}.mjs`、`src/parse/{common,chat,subtitle}.mjs`、 +`tests/{knowledge-store,knowledge-anchors,knowledge-ledger,parse-chat,parse-subtitle}.test.mjs`、 +`tests/fixtures/parse/{chat,subtitle}/**`、`.gitattributes`（夹具按字节保真，禁换行转换）。 + +## 2. 磁盘契约（这一条定下来，后面所有分支都按它写） + +``` +skills///knowledge/ + raw//… 原样字节，只增不改 + text/.md 归一化正文，段落锚点 [k0012] / [k0012:t3] + index.json 账本：{id,kind,origin,fetched_at,bytes,sha256,credentialed,method,warnings[]} +``` + +- **锚点必须能回指**：`resolveLedgerAnchor(ledger, anchor)` 返回文本与字节区间； + 容器里读出来的文本（OOXML 成员、去标签的 HTML）标 `synthetic` 且**报 null 字节区间**，不伪造偏移。 +- **幂等**：同一份字节（sha256 相同且 origin 相同）重复导入不新增条目。 +- **响亮拒绝**：不认识的格式按文件名进 warnings，不静默跳过。 + +## 3. 验收（当前树，可复算） + +```bash +node --test tests/knowledge-store.test.mjs tests/knowledge-anchors.test.mjs \ + tests/knowledge-ledger.test.mjs tests/parse-chat.test.mjs tests/parse-subtitle.test.mjs +node scripts/acceptance.mjs # harvest / 账本 / 幂等 / 锚点回指四个阶段 +``` + +`scripts/acceptance.mjs` 里与本条直接相关的断言：回执形状（sha256 + 字节数）、 +重复 harvest 幂等、账本里有锚点、派生结论锚点全部可回指。 + +## 4. 后续修复（合并后由集成轮补上，见后续 PR 文档） + +- `assignAnchorsToText` 曾把段落渲染成 `k0012 text`（无方括号），而本条的读者要求 + `[k0012] text`：派生层因此恒为空。修法与前后对比见 + `docs/evidence/pr-09-evidence-spine-blind-test.md`。 +- 归一化正文曾丢掉说话人与时间（本条的 parser 有元数据，但没进正文）：修法与前后对比见 + `docs/evidence/pr-11-attribution.md`。 + +## 5. 回滚 + +- `src/knowledge/**` 与 `src/parse/**` 是后面所有解析命令的地基：回滚本条会让 + `harvest` / `parse-*` / `retrospect` 全部失效，需回滚到 #01 的 CLI 骨架状态。 +- `.gitattributes` 的字节保真规则不要单独 revert：夹具在 CRLF 平台上会被改写，测试随之漂移。 diff --git a/docs/evidence/pr-09-evidence-spine-blind-test.md b/docs/evidence/pr-09-evidence-spine-blind-test.md new file mode 100644 index 00000000..adbef4fe --- /dev/null +++ b/docs/evidence/pr-09-evidence-spine-blind-test.md @@ -0,0 +1,120 @@ +# PR-09 · 证据脊柱打通 + 效果层盲测装置 + +- 分支：`dot-skill-test`（集成分支，本轮 9 个原子提交直接在本地集成分支上） +- 交付：`src/knowledge/anchors.mjs`、`src/derive/retrospect.mjs`、`src/parse/subtitle.mjs`、 + `src/views/{schema,render}.mjs`、`viewer/sections.js`、`src/commands/view.mjs`、 + `scripts/{split-corpus,blind-test,visual-check,acceptance}.mjs`、 + `tests/{retrospect,views,blind-test,parse-subtitle,knowledge-ledger}.test.mjs`、 + `docs/v2/{RENDER,STATUS,BLIND-TEST-RUNBOOK}.md` +- 依赖：`ds/01`–`ds/08` 的合并成果（本机已合并到 `dot-skill-test`） +- **零运行时依赖、零模型调用、零网络**；`playwright` 只在 CI 的 `acceptance` job 里用于 visual-check +- 按用户指令**不 push、不建 PR**：全部提交只在本地 +- 截图不入库：PNG 在 `dst-evidence/screenshots/pr-09-evidence-spine/`（`.gitignore` 已含 `evidence/`、`dst-evidence/`） + +## 1. 这一轮为什么存在 + +`docs/v2/ACCEPTANCE.md` 的"效果层"不能自证，于是先搭 A/B 盲测装置。装置第一次跑通就 +把三个**真实缺陷**照了出来——它们各自的分支测试都是绿的，因为每个分支只测自己那一层的 +格式；跨层跑一次就崩。 + +## 2. 三个缺陷与修法 + +| # | 缺陷 | 现象（可复算） | 根因 | 修法 | 回归测试 | +| --- | --- | --- | --- | --- | --- | +| 1 | 锚点格式不一致 | `retrospect` 对任何 harvest 出的语料都输出空集 | `assignAnchorsToText` 写 `k0012 text`；`CONTRACT.md` §2、`store.mjs` 头注释与唯一读者 `retrospect` 都要求 `[k0012] text` | 渲染器改回方括号；读取端兼容旧格式并告警 | `tests/retrospect.test.mjs`「a harvested subtitle reaches the derivation layer」「a pre-bracket text file is still read」+ `knowledge-ledger`/`parse-subtitle` 的 text 行断言 | +| 2 | 溯源假警告 | 每次运行都报 `has no ledger entry with a matching sha256` | 账本存**原始字节**摘要，代码拿它比对**归一化文本**摘要，永不相等 | 按 `locations.text` 认亲；只有账本真带 `text_sha256` 才校验摘要 | 「provenance follows the ledger's text link, not the raw digest」 | +| 3 | 说话人丢失 | `面试官：…` 识别不出说话人，voice/relations 退化为无归属统计 | `SPEAKER_PREFIX` 只认半角 `: `，中文导出用全角 `：` | 接受全角；半角无空格仍不认（否则 `https://…` 会解析出说话人 `https`） | `tests/parse-subtitle.test.mjs`「speaker prefixes are read from ASCII and full-width colons, but not from URLs」 | + +顺带修掉两个会误导使用者的东西： + +- `.gitignore` 的 `knowledge/` 匹配任意层级，把 `src/knowledge/` 也排除了：已跟踪文件不受影响， + 但**新增**的源码模块会被 `git add` 静默跳过。例外必须写在排除规则之后才生效。 +- `view render --file /tmp/x.view.json` 把回执写到 `/evidence/`（祖父目录当成了 Skill 目录）； + 现在只有父目录名为 `views` 才向上取一层。 + +## 3. 改动前后（同一段语料，可复算） + +语料：`tests/fixtures/public-corpus/synthetic-interview` 的 A 半段（28 条 cue，由 +`scripts/split-corpus.mjs` 按时间轴切出，`split.json` 记两侧 sha256）。 +"改动前"= `/tmp/dst-before` worktree（`b56ccbe`，本次修复之前的代码）实跑。 + +| 指标 | 改动前 | 改动后 | +| --- | --- | --- | +| `knowledge/text/subtitle.md` 行格式 | `k0001 面试官：面试官：…`（且前缀重复一次） | `[k0001] 面试官：…` | +| `retrospect` 可引用段落 | 0 | 28 | +| 派生结论（7 个文件合计） | **0**（claims 全空） | **22**（voice 11 / stats 4 / timeline 4 / relations 3） | +| 回指锚点 | 0 / 56 | 19 / 56 | +| 回执 warnings | 2（溯源假警告 + 说话人） | 0 | +| 说话人归属 | 无 | 面试官 / 林工 | +| 页面 | 渲染被拒绝（3 个 `VIEW_SECTION_MISSING`），读者手里没有页面 | 8 段齐全：4 段有证据 + 3 段显式「本节证据不足」+ 附录 18 锚点，52287 bytes | + +改动前的原话（`dst-evidence/screenshots/pr-09-evidence-spine/before.txt`）： + +``` +anchors: {"total": 56, "cited": 0} +warnings: + - knowledge/text/subtitle.md has no ledger entry with a matching sha256; it was read but not trusted for provenance. + - ledger:k0001: no speaker could be read from this subtitle: SubRip has no speaker field and no cue used a `` span or a `Name:` prefix +合计 claims = 0 +Error: view.json failed view check with 3 error(s); fix them before rendering +``` + +## 4. 薄证据渲染（`--allow-missing`） + +派生层填不满七段时，默认整页拒绝渲染：一段没有时间戳的语料就产不出页面。补占位结论 +等于编内容，同样不可接受。第三个选项是**说出来**： + +- `distilly view check|render --allow-missing`：缺失段落与空 `items` 降级为告警， + 逐条留在 `warnings[]`，退出码仍为 0； +- 渲染时按固定页序补回 `{items: [], unavailable: true}`，页面渲染一行 + 「本节证据不足：派生层没有产出可引用的结论，按约定不填占位话术。」（`data-empty="unavailable"`，zh/en）； +- 顺序判据改为"存在的段落保持相对顺序"，否则缺一段会把后面每段都报成错位； +- 默认行为不变：不带参数时八段必须齐全。 + +## 5. 效果层盲测装置 + +`docs/v2/BLIND-TEST-RUNBOOK.md` 固定角色、步骤、判据与留档。脚本只做机械部分： + +| 命令 | 作用 | +| --- | --- | +| `scripts/split-corpus.mjs` | 字幕按时间轴 / 文本按段落切 A/B，写 `split.json`（两侧 sha256、`cut.timecode`、`split_by`） | +| `blind-test prepare` | A → harvest → retrospect → `deriver-input.md`（结论 + 锚点表，无原文）+ 七段骨架 + 裁判 prompt + 空白评分表 + 回执；`--baseline` 另出机械基线页面 | +| `blind-test finalize` | 校验 + 渲染作者写的 view.json + 更新回执（`--strict` 让 check 失败时退出码 1） | +| `blind-test control` | 同一段 A 半段原文的裸 prompt 对照（禁止跑命令、禁止建知识库） | +| `blind-test score` | 命中率（hit + 0.5×partial）、无法判定比例、编造数 → PASS/FAIL/FALSIFIED | + +判据：命中率 ≥ 0.70、无法判定 ≤ 0.20、**编造数 = 0**；未填写的条目按"无法判定"计 +（空表必然 FAIL）。机械基线不写散文、不补证明不了的东西：证明不了的段落进 `gaps` +并写明原因（"没有日期就不编时间线"），回执标 `view_source: mechanical-baseline`。 + +## 6. 怎么验（本次实跑结果） + +```bash +node scripts/prompt-lint.mjs # 0 finding(s) across 26 file(s)，21 个命令全部已注册 +node --test tests/*.test.mjs # 296 pass / 0 fail（26 个文件） +node scripts/generate-template.mjs --check # 模板无漂移（viewer 改动后已再生成） +DISTILLY_PLAYWRIGHT_ROOT=/tmp/audit-mcp node scripts/acceptance.mjs + # 20/20 通过（新增 blind-test 与薄证据 visual-check 两段） +``` + +新增测试 17 条：`tests/blind-test.test.mjs` 9 条（切分不重不漏/可复算/拒绝切不动、 +三项指标与阈值边界、空表必 FAIL、prepare 端到端、finalize 作者页、control 对照）、 +`tests/retrospect.test.mjs` 3 条、`tests/views.test.mjs` 3 条、`tests/parse-subtitle.test.mjs` 1 条、 +`knowledge-ledger` 断言改为契约格式。 + +## 7. 已知缺口（诚实记录） + +- **效果层仍需人/模型签字**：本 PR 只交付可复算的装置与判据，脚本不自评、不冒充裁判。 + 命中率/编造率必须由**没看过语料**的裁判 + **看过 B 半段**的检查者产出。 +- 同一段语料能支撑几段由派生层的最低样本数决定（本语料 7 段中 3 段为缺口）； + `gaps` 非空时命中率只在填出的段落上可比，报告必须写明。 +- `PLANNED` 里仍是 `parse-chat`（飞书导出格式）与飞书浏览器/MCP 两路，`doctor` 会点名到分支。 +- 机械基线的 `portrait` 只是统计事实（"可引用消息 28 条，参与者 …"），不是画像； + 真实作者步应由模型/人写 `view.authored.json` 再 `finalize`。 + +## 8. 回滚 + +- 单个提交可独立 revert（9 个提交各自只做一件事，见 `git log --oneline`）。 +- 回滚 #1（方括号）会让派生层重新变空：`retrospect` 的旧格式兼容只解决读，不解决写。 +- 回滚 #4（`--allow-missing`）会让薄语料的页面重新无法渲染；`blind-test prepare --baseline` + 会直接失败（不静默降级）。 diff --git a/docs/evidence/pr-10-blind-test-runs.md b/docs/evidence/pr-10-blind-test-runs.md new file mode 100644 index 00000000..ef8af118 --- /dev/null +++ b/docs/evidence/pr-10-blind-test-runs.md @@ -0,0 +1,76 @@ +# PR-10 · 效果层盲测：实跑两轮的结果与结论 + +- 目标：按 `docs/v2/ACCEPTANCE.md` 的 A/B holdout 真跑一次"证据层 vs 裸 prompt"， + 用 `scripts/blind-test.mjs` 出数，而不是只交付装置。 +- 语料：`tests/fixtures/public-corpus/synthetic-interview/transcript.srt`（38 条 cue） + 按时间轴切成 A（28 条，00:00:01–00:03:12）/ B（10 条，00:03:12.3–00:04:21）。 +- 角色：4 个互不共享上下文的模型进程（蒸馏器 ×2、裁判 ×2、核对者 ×2，逐轮独立）； + 每轮裁判只看匿名化的 `judge-input.md`，核对者看 A+B+裁判输出，蒸馏器只看自己那一臂的输入。 +- 产物（本地，不入库）：`dst-evidence/screenshots/pr-09-evidence-spine/blind-run/` + （A/B、`deriver-input.md`、两轮 `judge-input-*.md`、`judge-output-*.json`、 + `checker-output-*.json`、`scores-*.json`、`report-*.md`、两版页面与 `profile-run*.txt`）。 + +## 1. 两轮的结果 + +| 轮次 | 实验组输入 | 臂 | hit | partial | miss | undecidable | fabricated | 命中率 | 无法判定 | 判定 | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| run 1 | 只有 `deriver-input.md`（派生统计） | evidence | 1 | 8 | 1 | 0 | 0 | 0.500 | 0.00 | FAIL | +| run 1 | A 半段原文 | control | 1 | 4 | 0 | 5 | 0 | 0.600 | 0.50 | FAIL | +| run 2 | `knowledge/text/subtitle.md` + `deriver-input.md` | evidence | 1 | 7 | 0 | 2 | 0 | 0.563 | 0.20 | FAIL | +| run 2 | A 半段原文 | control | 0 | 5 | 0 | 5 | 0 | 0.500 | 0.50 | FAIL | + +**四臂全部 0 编造**（`fabricated = 0`）：没有一条断言是页面没说过、或与 B 冲突的。 +这是唯一一条四臂都过的线。 + +### run 1 的编排错误（记录在案） + +`ACCEPTANCE.md` §2 规定蒸馏者可以看 `A 段 + knowledge/text/* + evidence/derived/*`， +run 1 我只给了派生统计，等于把实验组削弱成"只准看数字"。所以 run 1 的结论 +（裸 prompt 更好）不能算数，run 2 才是设计中的对照。 + +## 2. 结论：这段语料上，证据层没有提高命中率 + +run 2（正确编排）里实验组 0.563 vs 对照组 0.500，差 +0.063；两臂都没到 0.70。 +把 run 1 的对照组（0.600）拿来看，**裁判/核对者的方差与臂间差异同量级**（0.06–0.10）， +10 条断言 + 10 条 cue 的规模下，"谁更好"这个问题的信噪比不足。 + +但核对者给了一条更有信息量的观察，我复核后同意： + +> 两臂写出的**行为规则高度重合**（核账优先、数据→日志→代码、文档含回滚、回滚是评审检查点、 +> 带人给边界+验收标准、拒绝形容词式验收、不拿加班当态度、修不动即负债、离职细节最小披露、 +> 短句不寒暄）。在 B 上它们**成对同分**：都是 partial/undecidable，因为 10 条 cue 里 +> 没有事故、没有评审、没有发布、没有冲突场景可判。 + +所以差异不是"内容更好"，而是**可验证性**：实验组的统计类断言（句长、emoji） +在薄 B 上还能被逐条核对，对照组把同样的 10 个名额花在了 B 触发不到的行为断言上。 + +## 3. 我复核出的两个测量问题（都不是流水线的错） + +1. **核对者用错了单位**：run 1 把"句长中位数 13 字"判成 partial/miss，理由是 + "B 里林工的发言是 52/30/42/45/7 字符"。那是**整条消息**长度，不是**句子**长度。 + 按句子复算：A 半段林工中位 15 字、面试官 7 字、全语料（派生用的池）12–13 字； + **B 半段林工 16 字** —— 断言的模式被 B 证实，判 partial/miss 是误判。 +2. **派生维度缺说话人归属**：`voice.sentence_length` / `punctuation_density` / `shifts` + 是**两个说话人混在一起**的池化统计（只有 `catchphrase_*` 带 `speaker` 字段）。 + 对"他说话短"这种断言，池化值会被面试官的短问句拉低（林工 15 字 vs 面试官 7 字）。 + 这是真实待修项：要么每个维度都带 `speaker`，要么在 claim 里显式标注 `pooled: true`。 + +另外核实了一条蒸馏器提出的"派生统计算错"：`voice.catchphrase_ngram.1` 报 +「什么」4 次、`speaker: 面试官`。全语料出现 5 次（林工 1 次 + 面试官 4 次）， +**派生值是对的**，是 run 2 的蒸馏器把"全语料 5 次"与"面试官 4 次"看混了， +并据此把一条正确断言丢掉了。run 1 则是把面试官的口头禅当成被描述者的特征写进了画像 +（`speaker` 字段就在 claim 里）。 + +## 4. 结论与下一步 + +- **判据本身需要更厚的留出集**：10 条 cue 里半数断言无法判定，`undecidable` 触顶 + 会让两臂同时"失败"。`score` 已加 `INCONCLUSIVE`：命中率达标而仅无法判定超标时， + 结论是"换更厚的 B 重跑"，不是判该臂失败。 +- **核对者必须复算统计**：runbook 已加硬规则（同一单位复算 + 按主题配对比较）。 +- **证据层在这段语料上的真实价值是"可审计"，不是"更高命中率"**：run 2 的蒸馏器 + 拿着带锚点的原文逐条推翻了 3 条脆弱统计（疑问句比例其实是面试官的、口头禅候选是 + 称呼词、标点密度是两人混合），并明确写出"哪条派生结论我不用、为什么"。 + 这是裸 prompt 做不到的——它没有可回指的中间层可查。 +- **要证明"证据层更准"，需要换语料**：多来源、有时间戳、含事故/评审/冲突场景的 + 语料（群聊 + 邮件 + 文档），B 至少覆盖其中一类场景。这属于下一轮工作。 +- 流水线侧待修：`voice`/`stats`/`shifts` 的说话人归属（见 §3.2）。 diff --git a/docs/evidence/pr-11-attribution.md b/docs/evidence/pr-11-attribution.md new file mode 100644 index 00000000..0f29caab --- /dev/null +++ b/docs/evidence/pr-11-attribution.md @@ -0,0 +1,89 @@ +# PR-11 · 归因修复：派生层第一次认得出「谁」+ 飞书导出解析 + +- 分支：`ds/10-attribution`（3 个原子提交）已本地合并进 `dot-skill-test`；飞书导出在 + `ds/09-feishu`（1 个原子提交）同样已合并。**都没有 push**（用户暂停推送）。 +- 交付：`src/parse/common.mjs`、`src/knowledge/anchors.mjs`、`src/knowledge/ledger.mjs`、 + `src/parse/{chat,subtitle,feishu}.mjs`、`src/commands/{harvest,parse-chat}.mjs`、 + `src/derive/retrospect.mjs`、`tests/{text-attribution,parse-feishu}.test.mjs` +- 截图与命令留档：`dst-evidence/screenshots/pr-11-attribution/`（不入库） + +## 1. 为什么做这个 + +`docs/evidence/pr-10-blind-test-runs.md` 的盲测暴露了一条断言在留出集上被判失败： +"句长中位数 13 字"。复算发现**派生值其实是池化的**——A 半段里被描述者本人中位 15 字、 +面试官 7 字、池化 13 字。根因不在统计本身，而在**归一化正文丢掉了说话人与时间**： +`knowledge/text/*.md` 是派生层唯一读的文件，而说话人只留在 parser 的 metadata 里。 +手写夹具 `src/derive/fixtures/synthetic-group` 一直是 +`[k0001:t1] 2024-03-04T09:02:00Z 老周：…`，只有真实采集的语料不是。 + +## 2. 改动前后（同一段语料，可复算） + +语料：12 条 Slack 消息、两位说话人、每条带 `ts`（`dst-evidence/.../corpus-messages.json`）。 +"改动前"= worktree `/tmp/dst-pre-attr`（`773d56c`，本次修复之前的代码）实跑。 + +| 指标 | 改动前 | 改动后 | +| --- | --- | --- | +| `knowledge/text/chat.md` | `[k0001] 先看数据，再看日志…` | `[k0001] 2023-11-14T22:13:20.000Z Alice：先看数据，再看日志…` | +| 派生结论数 | stats 2 / relations 0 / timeline 4 / voice 6 | stats **6** / relations **4** / timeline 4 / voice 6 | +| `stats.participants` | （没有这条 claim） | `["Alice","Bob"]` | +| `timeline.phase` | `basis: "order"`、`from: null` | `basis: "time"`、`from 2023-11-14T22:13:20Z` | +| `voice.sentence_length` | 只有池化值 | 池化值 + `pooled: true` + `by_speaker`（Alice 中位 20 / Bob 19） | +| `retrospect` warnings | 1（假警告：缺 users.json） | 0 | + +同一个修复还顺带修掉两个证据脊柱缺陷： + +| 缺陷 | 现象 | 修法 | +| --- | --- | --- | +| 正文互相覆盖 | 同一 `--source` 采集两份导出时，第二份**静默覆盖**第一份的 `knowledge/text/.md`：账本两条 entry 都在，一份文档的段落从派生层的输入里消失 | 第一份占 `.md`，后来的按文件名 stem 写成 `--.md`；同一份字节重复导入仍幂等 | +| Slack 假警告 | 导出带 `username` 时名字已解析，回执仍警告"id 无法解析" | 解析完再判断：只有 turns 里真的残留 `U123…` 才告警并点名 | + +## 3. 关键设计：前缀是渲染期 markup，不是正文 + +`[k0012]` 锚点本来就是这样加的。归因前缀走同一条路： + +- `recordsFromCharSpans` 透传 `speaker`/`at`，`assembleContent` 把它们放进 `segments[]`， + **正文保持逐字**（`entries[].text` 与锚点文本不变）； +- `anchorNormalized` 渲染成 `[k0012] ：正文`； +- 正文已经带前缀的不重复（字幕 cue 里的 `Lin: …`）； +- 字幕只带说话人（时间码是录制位置、不是日期），chat/飞书带时间 + 说话人。 + +不变式仍然成立：**锚点文本 == 源字节切片**，`tests/text-attribution.test.mjs` +与 `tests/parse-chat.test.mjs` 都断言这一点（字幕锚点覆盖整条 cue 的信封， +断言正文逐字落在该区间内）。 + +## 4. 顺带交付：飞书消息导出解析（`ds/09-feishu`） + +移植 `tools/feishu_parser.py`（251 行）。原工具是"过滤 + 排版"（只留目标人的消息、 +分成长消息/决策/日常三桶），这两件事在本仓各有归属（`harvest --person`、 +`retrospect`），所以只保留**读格式**的部分：JSON 数组或 `messages|records|data` +包裹、全部字段别名、`sender{}`/`content{text}`/`content[]`、手工 `.txt` 日志 +（`--format feishu-text` 才认）、占位符跳过并告警、`walkJsonLeaves` 定位真实字节。 + +顺带修掉 Instagram 分支的误判：它用"任一键出现过即匹配"的 `findObjectArray` 判定， +把同样带 `sender_name` 的飞书导出吃掉了；现在要求每条都同时有 +`sender_name + timestamp_ms`，飞书作为最后的兜底形状。 + +## 5. 怎么验 + +```bash +node --test tests/*.test.mjs # 317 pass / 0 fail（29 个文件） +node scripts/prompt-lint.mjs # 0 findings +DISTILLY_PLAYWRIGHT_ROOT=/tmp/audit-mcp node scripts/acceptance.mjs # 20/20 通过 +node scripts/visual-check.mjs <页面> # 8/8（本轮页面见 pr-11 截图目录） +``` + +新增测试 11 条：`tests/text-attribution.test.mjs` 4 条（前缀格式与不重复、 +源字节不变式、派生层拿到人名与日期、同源两份导出不互相覆盖）、 +`tests/parse-feishu.test.mjs` 7 条（别名与形状、占位符、包裹与 txt、拒绝无关输入、 +与 Instagram 的分工、账本锚点回指 + 幂等、CLI 两条路径）。 + +## 6. 已知缺口（更新） + +- **飞书另外两路仍未移植**：`tools/feishu_browser.py`（Playwright 复用本机登录态）与 + `tools/feishu_mcp_client.py`（MCP App Token，`npx feishu-mcp --stdio`）。 +- **`note` 命令**（`CONTRACT.md` §1：把 LLM 自己读到的内容登记进账本， + `method: "model-read"`）仍未实现，`PLANNED` 现只剩它。 +- 归因只覆盖有说话人/时间的格式：OOXML 与压缩包成员的正文仍是纯文本（它们本来就没有 + 说话人概念），`voice` 的 `by_speaker` 在这些语料上不会出现。 +- 日期口径：字幕的时间码刻意不进正文（不是日期），因此字幕语料的 `timeline` + 仍会走 `basis: "order"`。 diff --git a/docs/evidence/pr-12-feishu-routes-note.md b/docs/evidence/pr-12-feishu-routes-note.md new file mode 100644 index 00000000..657987f0 --- /dev/null +++ b/docs/evidence/pr-12-feishu-routes-note.md @@ -0,0 +1,84 @@ +# PR-12 · 飞书三路补齐 + `note`：CONTRACT §1 命令面完成 + +- 分支：`ds/11-feishu-clients`（3 个原子提交）与 `ds/12-note`（1 个），都已本地合并进 `dot-skill-test` +- 交付：`src/collect/feishu-mcp.mjs`、`src/collect/feishu-browser.mjs`、`src/commands/note.mjs`、 + `src/collect/feishu.mjs`（逐页归一化）、`src/parse/feishu.mjs`（开放平台消息页）、 + `src/parse/chat.mjs`、`src/knowledge/ledger.mjs`、`src/consent.mjs`、 + `tests/{feishu-mcp,feishu-browser,note,collect,parse-feishu}.test.mjs` +- **不 push**（用户暂停推送）；截图与命令留档：`dst-evidence/screenshots/pr-12-feishu-routes/` + +## 1. 采集到的语料现在能派生（改动前后，可复算） + +`collect feishu` 原来只写 `knowledge/raw/feishu/*.json` + 自己的账本条目：数据落盘了， +但 `knowledge/text/*.md` 里什么都没有——而那是派生层唯一读的东西。**凭据渠道采到的语料 +蒸馏不了**，证据脊柱在 raw 桶就断了。 + +同一页飞书消息（12 条有正文），改动前 = worktree `/tmp/dst-pre-collect`（`15bf6c3`）： + +| 指标 | 改动前 | 改动后 | +| --- | --- | --- | +| 采集产物 | `raw` 一页 | `raw` + `text/feishu.md` | +| 账本条目 | 1 条 raw（无 `locations.text`、无锚点） | **1 条同时带 raw 与 text**，24 个锚点 | +| `retrospect` 回指 | 0 / 0 | **10 / 24** | +| 派生结论 | 全 0（7 个文件） | stats 6 / relations 4 / timeline 4 / voice 5 | + +（3 条消息的小页在改动后仍 `cited 0`：低于 `retrospect` 的最低样本数 8， +回执照旧给"样本不足"的理由，不硬凑。） + +## 2. 飞书三条路各自负责什么 + +| 路 | 命令 | 谁在做 | 关键纪律 | +| --- | --- | --- | --- | +| 开放 API | `collect feishu --chat-id ` | 本模块（tenant/user token） | 逐页一个条目，raw 逐字 + text 带锚点；解析不了的页仍然落 raw 并在 warnings 点名 | +| MCP | `collect feishu --mode mcp --url <文档>｜--chat-id ` | 本模块经 `npx -y feishu-mcp --stdio` | transport 可注入（无 npx/租户/网络即可测）；**工具白名单封闭**；凭据只进子进程环境、不进 argv；回执只可能出现配置文件名 | +| 浏览器 | `collect feishu --mode browser --consent --capture <文件>` | **宿主**在 computer use 下抓取 | 无同意 → exit 2「waiting-for-user-consent」且零文件；无 `--capture` → 回 `awaiting-host-capture` + 逐步计划；有 capture → 逐字落盘 + 去标签归一化（标 synthetic、不伪造字节区间），条目带 `provenance: host-reported` 与 `consent` | + +**为什么浏览器一路不是 Playwright**：`tools/feishu_browser.py` 用 Playwright 驱动用户本机 +Chrome（复用登录态）。v2 把"驱动浏览器、注入输入"归为 computer use，属于宿主职责， +必须走显式同意（与 `collect x --mode browser` 同一套设计，CONTRACT §4）。 +所以移植的是流水线真正拥有的部分——同意门、计划、逐字落盘、归一化——浏览本身留给宿主。 +测试里有一条策略断言：源码不含 `playwright`/`puppeteer`/`page.click|type|fill|goto`。 + +## 3. `note`：最后一条契约命令 + +`note --from ` 是唯一"材料从未以文件形式存在"的入口：宿主读了页面/PDF/截图并理解了它。 + +- 正文按原样入库（raw 逐字 + text 带锚点，锚点指回 raw 字节）； +- 条目 `method: "model-read"`、`credentialed: false`，回执与条目 warnings 都写明 + `this text was written by a model that read the source material; it is model-mediated, + not a first-hand capture`——**不许下游当成一手采集**； +- `--from -` 读 stdin；空输入 exit 1 且不产生条目；同一段文字重复登记幂等； +- `PLANNED` 因此清空，`doctor` 的未实现行改为 + `none — every command in CONTRACT §1 is implemented`。 + +## 4. 这一轮修掉的缺陷 + +| 缺陷 | 现象 | 修法 | +| --- | --- | --- | +| 采集页不能被重放 | `harvest <存下来的原始页>`（最自然的复算方式）回执只说 `format feishu-api has no parser`：只落 raw、无 text、派生全 0 | `parseChat` 的分发表补上 `feishu-api` case + 回归断言 | +| 凭据来源丢失 | 走 `recordDocument` 的条目丢了 `credential_source`/`credential_file`（只有采集器手工构造的 raw 条目才有） | `buildEntry` 带上这两个字段（只可能是文件名，永不含值） | +| 同意提示串渠道 | 任何渠道的补救文案都写 `collect x --mode browser` | 按 scope 里的渠道名生成（`collect:feishu:browser` → `collect feishu …`） | +| 渲染失败被 ENOENT 掩盖 | `blind-test finalize` 渲染失败后仍去读 `profile.html`，真实原因埋在诊断里（本轮被误导过一次） | 无产物就抛错并逐条列出渲染诊断 | + +## 5. 怎么验 + +```bash +node --test tests/*.test.mjs # 340 pass / 0 fail（33 个文件） +node scripts/prompt-lint.mjs # 0 findings +node scripts/generate-template.mjs --check +DISTILLY_PLAYWRIGHT_ROOT=/tmp/audit-mcp node scripts/acceptance.mjs # 20/20 +node scripts/visual-check.mjs <页面> # 8/8（本轮页面见 pr-12 截图目录） +``` + +新增测试 22 条：`feishu-mcp` 8、`feishu-browser` 7、`note` 6、采集归一化 1（`collect`）。 + +## 6. 已知缺口（更新） + +- **发送者名字**：开放平台消息页只带 `sender.id`，名字要另调通讯录 API。归因如实写 id 并在 + warnings 说明"这次没调通讯录"，不假装有名字。MCP 一路是否返回名字取决于 `feishu-mcp` 的实现。 +- **MCP 覆盖面**：`--url` 支持 wiki/docx/docs/sheets；`base`（多维表格）没有对应工具， + 按名字拒绝并给出理由。 +- **浏览器一路是"宿主转述"**：`provenance.confidence` 明确写 `host-reported`， + 它不是"我们验证过的登录态"。页面上被抓到的内容质量取决于宿主的抓取。 +- 派生层仍有池化口径未拆分：标点密度与 `shifts`（句长已按说话人分列，见 PR-11）。 +- `scripts/parity.mjs` 需要一份迁移前的 rev 才能跑（历史证据，不是日常门禁）。 diff --git a/docs/evidence/pr-13-objective-audit.md b/docs/evidence/pr-13-objective-audit.md new file mode 100644 index 00000000..e0be2d1e --- /dev/null +++ b/docs/evidence/pr-13-objective-audit.md @@ -0,0 +1,53 @@ +# PR-13 · 目标审计门禁 + +- 分支：直接落在集成分支 `dot-skill-test`（本地） +- 交付：`scripts/audit-objective.mjs`、`tests/audit-objective.test.mjs`、CI 两处步骤、 + `docs/evidence/pr-01-node-core.md`、`docs/evidence/pr-02-parse-zero-cred.md` + +## 1. 为什么 + +`acceptance.mjs` 证明"流水线能跑"，但证明不了"范围是闭合的"。目标里的每条需求 +（Node 单栈、证据脊柱、渲染与 visual-check、双语 prompt、宿主矩阵、渠道与同意、 +schema v4、端到端验收、截图不入库、每个 PR 有证据文档）都应该有一条机械检查， +并且每条检查都要说清楚**它读到了什么**；缺口单独成行、不算失败。 + +## 2. 覆盖面（16 条） + +Node 单栈（0 个 tracked `.py` + CI 无 Python 步骤）、证据脊柱磁盘契约与方括号锚点、 +retrospect 确定性与锚点纪律、单文件离线页面 + CSP + visual-check、双语 prompt lint、 +8 个宿主矩阵 + 防漂移测试、要 key 渠道 + computer-use 同意门（且浏览器一路不驱动浏览器）、 +未移植渠道的记账、schema v4 + 幂等迁移、CONTRACT §1 每个命令可解析且 `PLANNED` 为空、 +命令注册表规模、第二个公开语料存在且 CI 会跑它、证据图片不入库（项目素材除外）、 +每个**真正合并过**的分支都有 PR 证据文档、端到端验收全绿、推送状态。 + +## 3. 写检查时踩的坑（全部修的是检查，不是结论） + +| 误判 | 实际 | +| --- | --- | +| `agent.id` 取不到宿主机名 | `listAgents()` 返回的是**字符串** | +| "仓库不能有任何图片" | 项目素材（宿主 logo、社交预览）是合法的；要禁的是**证据**图片 | +| 把会话工作区的 `dst-evidence/` 当仓库内路径 | 它不在仓库里；仓库内只需断言"无证据图" | +| 用 `git branch --merged HEAD` 判断"已合并分支" | 当前特性分支的 tip 就是 HEAD，会被误判；改为按**合并提交**统计 | + +## 4. 怎么验 + +```bash +node scripts/audit-objective.mjs # 16/16，2 条已知缺口 +node scripts/audit-objective.mjs --skip-acceptance # 测试与 CI 的 test job 用这个 +node --test tests/audit-objective.test.mjs # 2 条：全体满足 + 缺口必须显式 +``` + +结果接进 `node --test` 与 CI 两个 job；`--json` 输出供其它脚本消费（`rows`/`failed`/`gaps`）。 + +## 5. 同轮补齐 + +- `docs/evidence/pr-01-node-core.md`、`pr-02-parse-zero-cred.md`：最早两条分支缺证据文档， + 审计把它标成唯一未满足项后补写。 +- 契约里未移植的四个采集渠道（discord/reddit/notion/gmail）从"读起来像拼错"改成 + `collect/planned-channel` + 逐条需求说明，`doctor` 多打一行 `Planned channels`。 + +## 6. 已知缺口 + +- 四个渠道仍未实现（各自需要什么已写进 `PENDING_CHANNELS`）。 +- 推送与 PR 受用户冻结影响，全部提交只在本地；解冻后的执行清单在 + `dst-evidence/PR-BODIES/PR-PLAN.md`。 diff --git a/docs/evidence/pr-15-identity.md b/docs/evidence/pr-15-identity.md new file mode 100644 index 00000000..cf7e0399 --- /dev/null +++ b/docs/evidence/pr-15-identity.md @@ -0,0 +1,54 @@ +# PR-15 · 跨渠道身份映射（`identity.json`） + +- 分支：`ds/15-identity`（本地，合并进 `dot-skill-test`） +- 交付：`src/knowledge/identity.mjs`、`src/knowledge/ledger.mjs`（记录时套用 + 账本留痕）、 + `src/commands/harvest.mjs`（`--identity`）、`docs/v2/IDENTITY.md`、`tests/identity.test.mjs` +- 依赖：无。与 PR-11（归一化正文带说话人）配合：那条让派生层知道"谁在说"，这条让它知道 + "不同句柄是同一个人"。 + +## 1. 为什么 + +多来源盲测的实验组蒸馏器只能**按渠道分别描述**同一个人：Slack 里是 `林工`，飞书里是 `ou_lin`， +语料没有任何一句说这两者是同一人。派生层因此报出 6 个参与者、把风格统计拆开， +`relationship` 也只能写成"显示名渠道 vs `ou_` 渠道"。裁判据此把若干条断言判为"跨渠道不可验证"。 + +## 2. 改动前后（同一份多来源语料） + +```bash +distilly harvest tests/fixtures/public-corpus/synthetic-multisource \ + --person lin-gong --identity ./identity.json +distilly retrospect --person lin-gong +``` + +| 指标 | 无映射 | 有映射（林工←ou_lin、小明←ou_chen） | +| --- | --- | --- | +| `knowledge/text/*.md` | `[k0001] … ou_lin：评审前我把风险清单发群里…` | `[k0001] … 林工：评审前我把风险清单发群里…` | +| `stats.participants` | `["林工","ou_lin","小明","老周","ou_chen","ou_zhou"]` | `["林工","小明","老周","ou_zhou"]` | +| `voice.sentence_length.by_speaker` | 6 个"说话人" | 4 个（真人的统计不再被拆开） | +| 账本条目 | 无身份信息 | `identity: {file:"identity.json", handles:["ou_chen","ou_lin"], turns:16}` | + +## 3. 纪律 + +- **句柄唯一**：同一句柄被两人声明 → exit 2 且**零写入**；非法 JSON 同样响亮失败。 +- **记录时生效**：规范化在锚定之前，一个 turn 仍是"一个锚点、一个前缀、一行统计"； + 锚点文本仍是源字节的逐字切片（不变式未破）。 +- **不做推断**：只合并显式声明的句柄；"措辞像同一个人"不构成合并理由。 +- 无映射时行为与以前完全一致。 + +## 4. 怎么验 + +```bash +node --test tests/identity.test.mjs # 5 条 +node --test tests/*.test.mjs # 354 pass / 0 fail +node scripts/audit-objective.mjs --skip-acceptance # 16/16 +``` + +`tests/identity.test.mjs` 覆盖：句柄在正文与账本里被归一（且原始句柄不残留）、 +派生层参与者与 `by_speaker` 合并、无映射时不改任何东西、句柄冲突/非法 JSON 零写入、 +纯函数行为（不改输入、match 列表、无文件时的空映射）。 + +## 5. 已知缺口 + +- 映射按句柄字符串精确匹配（大小写敏感）；`U02` 这类平台 id 需要写进映射或用导出里的显示名。 +- 通讯录 API 自动取名未实现（飞书开放平台页只有 `sender.id`）。 +- 映射文件本身没有锚点（它不进 `knowledge/`）；它的存在通过账本条目的 `identity` 字段留痕。 diff --git a/docs/evidence/pr-16-discord-notion.md b/docs/evidence/pr-16-discord-notion.md new file mode 100644 index 00000000..8e9c6737 --- /dev/null +++ b/docs/evidence/pr-16-discord-notion.md @@ -0,0 +1,55 @@ +# PR-16 · Discord 与 Notion 两个采集渠道 + +- 分支：`ds/16-discord-notion`（本地，已合并进 `dot-skill-test`） +- 交付：`src/collect/kit.mjs`、`src/collect/discord.mjs`、`src/collect/notion.mjs`、 + `src/parse/chat.mjs`（透传 method / 凭据来源 / 认 `author.username`）、 + `src/commands/credentialed.mjs`、`tests/collect-discord-notion.test.mjs` + +## 1. 为什么 + +`CONTRACT.md` §1 的采集命令行写着 8 个渠道，此前只实现 4 个（飞书/Slack/钉钉/X）。 +未实现的两个渠道此前回 `collect/planned-channel` 并写明所需凭据——诚实，但仍是缺口。 +这一轮把 Discord（bot token）与 Notion（内部集成 token）补上，`PENDING_CHANNELS` +从 4 条缩到 2 条（reddit / gmail）。 + +## 2. 三条纪律，与已有渠道一致 + +| 纪律 | 实现 | +| --- | --- | +| 只读 | Discord 读 API 全是 GET，白名单就只列 GET；Notion 的读路径几乎全是 POST（search / 数据库查询），白名单**显式**列出这两个 POST 并拒绝其它一切。越权动词在建立连接之前被拒 | +| 凭据不进回执 | 只允许出现配置文件名（`discord_config.json` / `notion_config.json`）；`redact`/`scrub` 覆盖回执的每个字段（含嵌套数组） | +| 逐字入库 + 可派生 | 原始页 verbatim 落 `knowledge/raw//`，同一次采集把它变成带锚点正文（Discord 交给**已有的导出解析器**，Notion 走段落化），一页/一页一个账本条目 | + +分页与限流：Discord 用 `before` 游标（一页不足 page size 即结束），429 读 body 里的 +`retry_after`（**秒**）退避——测试断言睡的是 1500ms；Notion 用 `next_cursor`。 +两处都支持断点续采（游标写在 `$DISTILLY_HOME/state/`）。 + +## 3. 顺带修好的两处解析器透传 + +- `parseChat` 现在尊重调用方的 `method` 与 `credentialed/credential_source/credential_file`： + 导出来的是 `user-export`，API 采集来的是 `api-bot-token`——否则采集来的页面会在账本里 + 冒充用户导出。 +- 说话人取名认 REST 的 `author.username`（导出用 `name`，API 用 `username`）： + 同一批消息无论是下载的还是采集的，人名一致（`林工` 而不是 `u1`）。 +- 顺带的一致性变化：chat 的正文文件名现在随 harvest 的 source 标签（默认取输入目录名）， + 与其它格式一致；`--source chat` 可恢复旧名。 + +## 4. 怎么验 + +```bash +node --test tests/collect-discord-notion.test.mjs # 10 条 +node --test tests/*.test.mjs # 364 pass / 0 fail +node bin/distilly.mjs doctor # Planned channels: collect reddit, collect gmail +``` + +测试覆盖：两页分页（游标串接 + 逐字字节 + 锚点正文 + 一页一条目）、429 退避毫秒数、 +缺凭据零写入、写动词被拒、Notion 页面 → 段落（跳过块点名）、Notion 白名单（两个 POST 放行、 +`POST /v1/pages` 与 `PATCH` 拒绝）、页面 id 从 URL 解析与坏 id 响亮失败、空页面不算空文档、 +kit 的脱敏与凭据查找（环境变量 / 配置文件 / legacy 路径 / 非法 JSON）。 + +## 5. 已知缺口 + +- **reddit / gmail 仍未实现**（各自需要什么写在 `PENDING_CHANNELS`，`doctor` 会列出）。 +- Discord 的线程/回复关系、Notion 的数据库行查询（白名单已放行 `POST + /v1/databases/{id}/query`，但 CLI 尚未暴露 `--database-id`）留待后续。 +- Discord 附件、Notion 的图片与嵌入块不anchored，按类型计入 warnings。 diff --git a/docs/evidence/pr-17-reddit-gmail.md b/docs/evidence/pr-17-reddit-gmail.md new file mode 100644 index 00000000..f5a64133 --- /dev/null +++ b/docs/evidence/pr-17-reddit-gmail.md @@ -0,0 +1,55 @@ +# PR-17 · Reddit 与 Gmail：契约的采集渠道全部实现 + +- 分支：`ds/17-reddit-gmail`（本地，已合并进 `dot-skill-test`） +- 交付：`src/collect/reddit.mjs`、`src/collect/gmail.mjs`、`src/parse/email.mjs`（透传 method/凭据来源）、 + `src/commands/credentialed.mjs`（渠道注册 + 待实现表清空）、`tests/collect-reddit-gmail.test.mjs` +- 依赖：`src/collect/kit.mjs`（PR-16 抽出的共享管道） + +## 1. 结果 + +`CONTRACT.md` §1 列了 8 个采集渠道，此前 6 个。这一轮补上最后两个， +**`PENDING_CHANNELS` 清空**：`doctor` 打印 `Planned channels：none`， +审计的渠道行从"仍未移植"变成"契约的采集渠道全部实现"。 + +| 渠道 | 鉴权 | 只读保证 | 分页 | 归一化 | +| --- | --- | --- | --- | --- | +| `collect reddit --target [--kind user]` | OAuth client credential（`POST /api/v1/access_token`，Basic 头） | 白名单只有这一个 POST + 四个 GET 前缀；`/api/submit`、`/api/vote` 一类在建连前被拒 | `after` 游标（Listing 的 `data.after`） | 评论 → `ISO 作者：正文` 锚点段落 | +| `collect gmail [--query <搜索>]` | OAuth refresh token（`POST /token` 换 access token） | 白名单只有这一个 POST + `/gmail/v1/users/` 的 GET | `nextPageToken`，`--max-messages` 封顶 | **raw MIME 交给已有的邮件解析器** | + +## 2. 两处值得记录的设计 + +- **Gmail 不重写邮件解析**：取回 `format=raw` 的 MIME 字节，直接构造 `SourceFile` 交给 + `parseEmail`——头部 RFC 2047 解码、按段 charset、`text/plain` 优先回退 HTML、 + 附件只登记并进 warnings，全部与本地 `.eml` 同一条实现。access token 中途过期时 + 自动再换一次并继续（测试断言换了两次、运行未中断）。 +- **Reddit 的占位符不当成发言**：`[deleted]` / `[removed]` / `more` 逐类跳过并在 warnings 点名， + 绝不把平台占位符锚成"某人说过的话"。 + +## 3. 顺带修好的一处透传 + +`parseEmail` 现在像 `parseChat` 一样尊重调用方的 `method` 与 +`credentialed/credential_source/credential_file`：本地 `.eml` 是 `local-file`， +Gmail 取回的是 `api-oauth-refresh`，账本条目因此不会互相冒充。 + +## 4. 怎么验 + +```bash +node --test tests/collect-reddit-gmail.test.mjs # 7 条 +node --test tests/*.test.mjs # 371 pass / 0 fail +node bin/distilly.mjs doctor # Planned channels: none +node scripts/audit-objective.mjs --skip-acceptance # 16/16（渠道行不再是缺口） +``` + +测试覆盖：两页 `after` 游标 + Basic/Bearer 两条鉴权路径 + 逐字字节 + 锚点段落、 +占位符与 `more` 跳过、401 响亮失败且零写入、写动词被拒、Gmail 刷新流程与 raw MIME +交给邮件解析器（断言 subject / 发件人 / 正文都进了正文）、access token 过期自动重换、 +凭据被拒零写入、渠道表已满。 + +## 5. 已知缺口 + +- **推送与 PR 仍未执行**（用户冻结）；解冻后的清单在 `dst-evidence/PR-BODIES/PR-PLAN.md`。 +- 各渠道的细粒度能力仍有取舍：Reddit 未取 submission（只有评论）， + Gmail 未处理 `format=full` 的结构化正文（raw 已覆盖）， + Discord 线程/回复关系、Notion 数据库行查询的 CLI 入口（白名单已放行）留待后续。 +- 采集到的语料仍受渠道本身限制（例如飞书开放平台页只有 `sender.id`）；跨渠道身份靠 + `identity.json` 显式声明（PR-15）。 diff --git a/docs/evidence/pr-18-blind-identity.md b/docs/evidence/pr-18-blind-identity.md new file mode 100644 index 00000000..77c299ed --- /dev/null +++ b/docs/evidence/pr-18-blind-identity.md @@ -0,0 +1,37 @@ +# PR-18 · 池化口径收尾 + 带身份映射的第四次盲测 + +- 分支：`dot-skill-test`（本地集成分支） +- 交付：`src/derive/retrospect.mjs`（标点密度按说话人分列、突变点标注 pooled） +- 盲测对照：A 语料 + `identity.json`（PR-15 的能力）重跑一次，对照组沿用同语料同提示词的画像 + +## 1. 池化口径收尾 + +PR-10 的盲测把"池化统计"列为待修项。句长在 PR-11 已按说话人分列，这一轮把剩下两项补上： + +| 维度 | 之前 | 现在 | +| --- | --- | --- | +| `voice.punctuation_density` | 一个池化数字 | `{density, pooled: true, by_speaker: {说话人: {mean, median, p90, samples}}}` | +| `shifts.candidate.*` | 只有 metric/window/delta | 加 `pooled: true`，多说话人语料再加 `mixes_speakers: true`——滑窗跑在整条消息流上，一个"突变"可能是对方带来的 | + +页面上的效果（同一份语料）： + +``` +标点密度（标点字符 / 总字符）：density 0.11；pooled true；按说话人：小明 中位 0.15；林工 中位 0.11；老周 中位 0.14 +``` + +## 2. 带身份映射的派生化对照（同一份 A 语料） + +| 指标 | run 3（无映射） | run 4（有映射） | +| --- | --- | --- | +| 派生结论 voice | 7 | **11** | +| 回指锚点 | 26/76 | 24/76 | +| 口头禅归属 | 在 `ou_lin` 与 `林工` 之间拆开 | 全部归到 `林工`（回滚 5 / 代码 4 / 方案 4 / 不动 3） | + + + +## 3. 怎么验 + +```bash +node --test tests/*.test.mjs # 371 pass / 0 fail +node scripts/audit-objective.mjs --skip-acceptance # 16/16 +``` diff --git a/docs/evidence/pr-19-release-migration.md b/docs/evidence/pr-19-release-migration.md new file mode 100644 index 00000000..2a207084 --- /dev/null +++ b/docs/evidence/pr-19-release-migration.md @@ -0,0 +1,61 @@ +# PR-19 · 迁移把旧内容导入证据脊柱 + 发布检查 + +- 分支：`dot-skill-test`（本地集成分支） +- 交付：`src/skill/migrate.mjs`（legacy 导入）、`src/commands/{migrate,harvest}.mjs`、 + `scripts/check_release.mjs`（新）、`tests/{schema-migration,release-check}.test.mjs`、CI、审计 + +## 1. 迁移的缺口（做发布检查时发现的真实缺陷） + +`skill migrate` 原来只做三件事：建 v4 目录、写一个**空**账本、改 `schema_version`。 +v3 skill 的材料在 `knowledge/{docs,messages,emails}/` 里——迁移之后它们仍在原地， +账本 0 条、`knowledge/text/` 是空的，**派生层依然看不到任何证据**。也就是说"迁移成功" +只体现在目录形状上。 + +现在迁移会把旧内容导入证据脊柱（复用 `harvest` 的解析分发，`documentFor` 改为导出）： + +| 指标 | 改动前 | 改动后 | +| --- | --- | --- | +| 账本条目 | 0 | **2**（`k0001` / `k0002`，各带 `locations.raw` + `locations.text`） | +| `knowledge/text/` | 空 | 两份带 `[k0001]` 锚点的正文（消息日志保留 `老周：` 前缀） | +| `retrospect` | 无输入 | 产出 7 个派生文件（3 条可引用段落，低于最低样本 → 如实给"样本不足"） | +| 第二次迁移 | 0 changed | 0 changed（幂等） | +| 旧文件 | 原地 | **原地保留，不删不改**（指纹比对断言） | + +细节：`origin` 用旧文件的路径，所以重复迁移按 sha256+origin 去重； +解析不了的文件进 `skipped` 并在 actions 里点名（不静默）；`migrateSkillDir` 的返回值 +补齐 `imported: {files, entries, skipped}`——"已是最新"的早返回过去缺这个字段， +调用方得特判。 + +## 2. 发布检查（STATUS 里承诺过、一直缺席） + +`scripts/check_release.mjs`：7 项发布卫生检查，每项都写出它读到了什么。 + +| # | 检查 | 现状 | +| --- | --- | --- | +| 1 | 版本三处一致（`package.json` / `--version` / `SKILL.md`）；`--tag` 时校验 tag | 1.0.0 / 1.0.0 / 1.0.0 | +| 2 | `SCHEMA_VERSION=4` + 账本 v2 + 迁移脚本在 + 幂等断言 | ✅ | +| 3 | 安装器携带 evidence spine（`knowledge/raw`、`knowledge/text`、`evidence`、`views`） | ✅ | +| 4 | 零运行时依赖 + 无 Python 残留 | dependencies 0 / tracked .py 0 | +| 5 | 生成物与源同步（模板 `--check` + 拼音表在） | ✅ | +| 6 | 门禁齐备且 CI 会跑（`node --test` / acceptance / prompt-lint / audit） | ✅ | +| 7 | 发布的文档都在（CONTRACT / ACCEPTANCE / STATUS / MIGRATION / IDENTITY / README） | 6/6 | + +接进 CI（test job）与审计（16 → **17** 条），并有 2 条测试：全过；以及 +`--tag v9.9.9` **必须失败**（证明它真的会红）。 + +## 3. 怎么验 + +```bash +node scripts/check_release.mjs # 7/7 +node --test tests/*.test.mjs # 378 pass / 0 fail +node scripts/audit-objective.mjs --skip-acceptance # 17/17 +node bin/distilly.mjs skill migrate --base-dir # 导入旧内容，跑两次幂等 +``` + +命令与产物原文：`dst-evidence/screenshots/pr-19-release-migration/migration-and-release.txt`。 + +## 4. 已知缺口 + +- 发布检查不校验 CHANGELOG（仓库没有该文件；要发版时再决定是否引入）。 +- 迁移导入的是"文件形态"的旧内容；v3 时代若把材料直接写在 `persona.md`/`work.md` 里， + 那些是**生成物**而不是语料，迁移不会把它们当证据（这是有意的：正文由作者写，不是采集来的）。 diff --git a/docs/v2/ACCEPTANCE.md b/docs/v2/ACCEPTANCE.md index fd24e83c..9a7f349a 100644 --- a/docs/v2/ACCEPTANCE.md +++ b/docs/v2/ACCEPTANCE.md @@ -64,8 +64,6 @@ node scripts/acceptance.mjs --keep # 保留临时 person 目录 语料放 `tests/fixtures/public-corpus/`（见那里的 `README.md` 与 `LICENSE.md`）。 -**脚本自测（防"验收脚本本身是坏的"）**：用一份符合契约的最小实现（只在 `/tmp` 下的临时目录，不入库）跑 `scripts/acceptance.mjs`，结果 **11/11 通过** —— 证明这套断言不是空转、契约是可实现的、以及回执/账本/锚点/view/visual-check 的形状就是脚本期望的那样。任何一项在真实实现上失败，都是实现的问题，不是脚本的。 - ## 6. 真实私聊/邮件语料（隐私） 不进仓库、不进 CI。流程同上，但： @@ -89,3 +87,4 @@ Mechanical assertions (receipts, idempotence, determinism, anchor resolution, byte conservation, single-file/offline, visual-check's eight assertions) run in CI via `scripts/acceptance.mjs`. Private corpora follow the same table with the user as judge and checker; only the metrics are kept. + diff --git a/docs/v2/BLIND-TEST-RUNBOOK.md b/docs/v2/BLIND-TEST-RUNBOOK.md new file mode 100644 index 00000000..40a9fb2e --- /dev/null +++ b/docs/v2/BLIND-TEST-RUNBOOK.md @@ -0,0 +1,113 @@ +# 效果层盲测 runbook（A/B holdout） + +`docs/v2/ACCEPTANCE.md` 把"效果"单列一层，因为它**不能自证**：蒸馏器和裁判都不能是自己。 +本文件固定流程与判据，脚本只做机械部分（切分、派生、渲染、算分），不代替裁判。 + +一句话：**A 半段给蒸馏器，B 半段只给检查者**；裁判只看渲染页，从不看语料。 + +## 0. 角色 + +| 角色 | 看到什么 | 不能看到 | +|---|---|---| +| 蒸馏器 distiller | A 半段语料（实验组）或 A 半段原文（对照组） | B 半段 | +| 裁判 judge | `profile.html` + `judge-prompt.md` | 任何语料、任何身份线索 | +| 检查者 checker | A/B 半段原文 + 裁判的 10 条 | —— | + +三种角色都必须是**模型或人**，不是本脚本。同一模型不得同时充当蒸馏器与裁判。 + +## 1. 切分（检查者一个人做） + +```bash +node scripts/split-corpus.mjs --in --out /tmp/blind-probe --ratio 0.7 +# A: 28 cues 00:00:01.000–00:03:12.000 / B: 10 cues 00:03:12.300–00:04:21.000 +# split.json 记录两侧 sha256 与切点，任何一方都能独立复算 +``` + +字幕按**时间轴**切（不是按条数），文本按段落数切。`split.json` 是对外可验证的凭据： +两侧 sha256、`cut.timecode`、`split_by`。B 半段从此刻起不进蒸馏器、不进渲染目录。 + +## 2. 实验组（证据层） + +```bash +node scripts/blind-test.mjs prepare --a /tmp/blind-probe/A.srt \ + --person blind-sample-01 --out /tmp/blind-run --baseline +``` + +`prepare` 在 `/work/` 里跑真实的 `harvest → retrospect`，然后写出： + +| 文件 | 给谁 | 内容 | +|---|---|---| +| `deriver-input.md` | 蒸馏器（作者步） | 每条派生结论 + 锚点 + 锚点表；**不含原文** | +| `view.skeleton.json` | 蒸馏器（作者步） | 七段骨架，`evidence[]` 已按账本填好 | +| `profile.html` | 裁判 | `--baseline` 直接产出的页面（私有模式） | +| `judge-prompt.md` | 裁判 | 固定问题：10 条可验证特征 + 每条一句预测 | +| `scores.template.json` | 检查者 | 空白评分表（两臂共用一张） | +| `receipt.json` | 检查者 | 派生化、`sections_filled`、`gaps`、页 sha256、外链检查 | + +`--baseline` 是**机械基线**：每段只由对应的派生 kind 填充，脚本自己不写一句散文， +也**不补它证明不了的东西**——`gaps` 列出派生层没产出的段落（例如没有时间戳就没有 timeline， +样本不足就没有 values/boundaries），页面为这些段落渲染「本节证据不足」。 +回执里的 `view_source` 明说是 `mechanical-baseline`；真实跑法是用模型读 `deriver-input.md` +把七段写成 `view.authored.json`，再： + +```bash +node scripts/blind-test.mjs finalize --out /tmp/blind-run --view /tmp/blind-run/view.authored.json --strict +# 校验 + 渲染 + 更新回执；--strict 让 check 不通过时退出码为 1 +``` + +两种来源在 `receipt.view_source` 与 `receipt.view.sha256` 里可区分，评分别混用。 + +## 3. 对照组（裸 prompt，无证据层） + +```bash +node scripts/blind-test.mjs control --a /tmp/blind-probe/A.srt --out /tmp/blind-run +``` + +`control-prompt.md` = 同一段 A 半段**原文** + "不要运行任何命令、不要生成知识库，直接写画像"。 +把模型输出落到 `control-profile.md`。这一臂检验的是：**如果没有证据层也能拿到同样的命中率， +证据层就是多余的**。 + +## 4. 打分（检查者） + +裁判输出填进 `scores.json`（结构见 `scores.template.json`），每条一个 `verdict`： + +| verdict | 含义 | +|---|---| +| `hit` | B 半段能直接证实 | +| `partial` | 方向对但细节错/过泛，计 0.5 | +| `miss` | B 半段明确反驳 | +| `undecidable` | B 半段没有信息可判（**不是**"看起来像"） | +| `fabricated` | 页面没说过、或与 B 半段冲突却讲得很确定的断言 | + +```bash +node scripts/blind-test.mjs score --scores /tmp/blind-run/scores.json --out /tmp/blind-run/report.md +``` + +判据（脚本算，不手算）： + +- 命中率 = (hit + 0.5×partial) / 可判定条数，**≥ 0.70** +- 无法判定比例 = undecidable / 总条数，**≤ 0.20** +- 编造数 = `fabricated` 条数，**必须 = 0**，否则该臂直接 FALSIFIED +- 反向对照：实验组命中率应高于对照组；若持平或更低，证据层在**这段语料上**没有价值，要写进结论 + +未填写的条目按 `undecidable` 计（不填 ≠ 通过），所以空表一定 FAIL。 + +## 5. 记录 + +一次跑完的最小留档（全部本地，截图不入库，见 `dst-evidence/`）： + +1. `split.json`（切分凭据） +2. `receipt.json` + `view-diagnostics.json`（页面来源与已知缺口） +3. `profile.html` / `control-profile.md`（两臂刺激） +4. 裁判 10 条原文、`scores.json`、`report.md` +5. 一句话结论：这段语料上证据层是否值得，以及 `gaps` 是否影响了判断 + +## 6. 已知边界 + +- `prepare` 的机械基线**不是**"自动写画像"：它只把派生结论搬到页面上。真正的作者步仍需模型或人， + 这也意味着"效果层"验收必须有人/模型参与，脚本永远不会替你签字。 +- 一段语料能支撑几段是派生层说了算；`gaps` 非空时，命中率只在填出的段落上可比， + 报告里必须写明这一点。 +- 身份盲：`prepare --person ` 决定页面上的 `slug`；要让裁判连人名也看不到， + 用一个中性 slug（如 `blind-sample-01`）跑，目录名与页面都不含真名。 +- 裁判与蒸馏器若用同一模型，须在结论里注明；同模型自评会让命中率偏高。 diff --git a/docs/v2/CONTRACT.md b/docs/v2/CONTRACT.md index 19db858d..f9e175bc 100644 --- a/docs/v2/CONTRACT.md +++ b/docs/v2/CONTRACT.md @@ -85,3 +85,4 @@ skills/// ## 6. 双语 prompt 与用户可见文档：**单文件双语**，中文段 → `---` → `## English`。 + diff --git a/docs/v2/IDENTITY.md b/docs/v2/IDENTITY.md new file mode 100644 index 00000000..d2334945 --- /dev/null +++ b/docs/v2/IDENTITY.md @@ -0,0 +1,43 @@ +# 跨渠道身份映射（`identity.json`） + +同一个人在每个渠道签名不同：一个导出里叫 `林工`，另一个里是 `ou_lin`，第三个里是 `U02`。 +派生层只读 `knowledge/text/*.md`，所以句柄不合并时它会报出多个参与者，并把这个人的统计 +拆到几个人身上——多来源盲测因此只能"按渠道分别描述"（见 `docs/evidence/pr-14-multisource.md`）。 + +## 文件与命令 + +Skill 根目录（`skills///identity.json`）放一份映射： + +```json +{ + "people": [ + { "name": "林工", "handles": ["ou_lin", "U02"], "note": "Slack 显示名 = 飞书 open_id" } + ] +} +``` + +安装与使用： + +```bash +distilly harvest --person lin-gong --identity ./identity.json +``` + +`--identity` 会把映射装到 Skill 根目录；之后的每次采集/解析都自动读取它。 + +## 纪律 + +- **句柄唯一**：一个句柄被两个人声明是**错误**（exit 2），不掷骰子；非法 JSON 同样响亮失败， + 且不写任何文件。 +- **记录时生效**：规范化发生在锚定之前，所以一个 turn 仍是**一个锚点、一个前缀、一行统计**； + 锚点文本仍是源字节的逐字切片。 +- **回执可查**：被合并的条目在账本里带 + `identity: { file, handles: [...], turns: N }`，读者能看出 `ou_lin` 与 `林工` 是同一个人。 +- **不做推断**：映射只写你明确声明的关系。语料里"看起来像同一个人"（同样的措辞、同样的立场） + 不会自动合并——多来源语料里那三份文件正是这种情形。 + +## 已知边界 + +- 映射按**句柄字符串**精确匹配（去首尾空白，大小写敏感）。 +- 没有映射时行为与以前完全一致（不回退、不猜测）。 +- 通讯录 API 自动取名的路径未实现：飞书开放平台页只带 `sender.id`，名字要么进映射， + 要么老实保留 id。 diff --git a/docs/v2/MIGRATION.md b/docs/v2/MIGRATION.md index 9f9c9c9b..5446af78 100644 --- a/docs/v2/MIGRATION.md +++ b/docs/v2/MIGRATION.md @@ -47,3 +47,4 @@ 1. **删一个 py 文件的前提**：对应 mjs 有测试，且 parity 证据（同一输入，两边输出逐字节相同）写进 `docs/evidence/pr-NN-*.md`。 2. 纯网络/需要凭据的部分（飞书浏览器自动化、Slack/钉钉拉取）不靠"无凭据环境下的 parity"证明——用注入 mock fetch 的失败路径 + 密钥不泄露断言来证明，真实账号验证在用户授权后单独做。 3. 迁移完成判据：`find tools tests -name "*.py" | wc -l` 为 0，且 `requirements.txt` 删除，CI 只跑 Node。 + diff --git a/docs/v2/STATUS.md b/docs/v2/STATUS.md index c90cf2d9..7e95bfbc 100644 --- a/docs/v2/STATUS.md +++ b/docs/v2/STATUS.md @@ -9,12 +9,12 @@ | 3 | `ds/03-render` | 模板碎片 + 生成物 + `--check` 防漂移；`view check/render`；`visual-check` 八项 | — | `node scripts/generate-template.mjs --check`；`node scripts/visual-check.mjs ` | | 4 | `ds/04-prompts` | `SKILL.md` 五步 + 每个 prompt"必须/禁止/回执" + 三个新 prompt + `prompt-lint` | — | `node scripts/prompt-lint.mjs`；`node --test tests/prompt-contract.test.mjs` | | 5 | `ds/05-agents` | `docs/v2/HOSTS.md` 双语文档 + INSTALL/README 宿主章节 + 断言测试（矩阵本身已由维护者落在 `src/hosts/agents.mjs`） | — | `node --test tests/agents.test.mjs`（含「矩阵 vs bin/distilly.mjs 表不漂移」断言） | -| 6 | `ds/06-retrospect` | `retrospect` → `evidence/derived/*.json`（每条带锚点、两次字节相同） | 契约（账本形状已冻结，自带合成夹具；落地后再对 ds/02 的真实输出复验） | `node --test tests/retrospect.test.mjs`；`acceptance.mjs` 的确定性/回指断言 | -| 7 | `ds/07-collect-consent` | 要 key 渠道（飞书/Slack/钉钉/X api）+ computer-use 同意门 + 密钥纪律 + transcribe 可选后端 | 契约 + `src/hosts/agents.mjs` | `node --test tests/collect.test.mjs tests/consent.test.mjs`（含"密钥不泄露"与"代码无写操作"断言） | -| 8 | `ds/08-schema-release` | `SCHEMA_VERSION 4` + 幂等迁移 + 安装器携带 `knowledge/|evidence/|views/|assets/` + 发布检查 | 1,2,3 | `node --test tests/schema-migration.test.mjs`；`scripts/check_release.mjs` | +| 6 | `ds/06-retrospect` | `retrospect` → `evidence/derived/*.json`（每条带锚点、两次字节相同） | 2 | `node --test tests/retrospect.test.mjs`；`acceptance.mjs` 的确定性/回指断言 | +| 7 | `ds/07-keys-and-schema` | 要 key 渠道 + computer-use 同意门 + 密钥纪律 + `SCHEMA_VERSION 4` 迁移 + 发布 | 1,2 | `docs/evidence/pr-07-keys-schema.md`；无 key/无 consent 的失败路径测试 | | — | `dot-skill-test`（本分支） | 契约、验收协议、语料夹具、验收脚本、宿主矩阵 `src/hosts/agents.mjs`、本状态表 | — | `node scripts/acceptance.mjs`（依赖到位后必须全绿） | ## 合并顺序 -`ds/01` → `ds/02` → {`ds/05`, `ds/06`, `ds/07`} → `ds/03`, `ds/04` → `ds/08` → 最后 `dot-skill-test` → `dot-skill`（默认分支）。 +`ds/01` → `ds/02` → {`ds/05`, `ds/06`} → `ds/03`, `ds/04` → `ds/07`，最后 `dot-skill-test` → `dot-skill`（默认分支）。 每个 PR 的 base 都是 `dot-skill-test`；合并后按 `docs/evidence/pr-NN-*.md` 复核一遍断言。 + diff --git a/dsh-install-probe.mjs b/dsh-install-probe.mjs new file mode 100644 index 00000000..a214703b --- /dev/null +++ b/dsh-install-probe.mjs @@ -0,0 +1,98 @@ +// Installs the DSH binding into an isolated DSH home and reports what it wrote. +// usage: node dsh-install-probe.mjs +import { createHash } from "node:crypto"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +const dshHome = process.argv[2]; +if (dshHome === undefined) throw new Error("usage: node dsh-install-probe.mjs "); + +const { createDshHostBinding } = await import("./packages/bindings/lib/dsh/full.js"); +const manifest = JSON.parse(await readFile("./plugins/release-manifest.json", "utf8")); + +// The DSH executable used for verification: the real installation under external/dsh. +const dshExecutable = + "/Users/zhoutianyi/Documents/dsh/external/dsh/node_modules/@deepseek-ai/dsh/lib/bin.js"; + +const launcherPath = join(dshHome, ".distilly", "bin", "distilly"); +await mkdir(join(dshHome, ".distilly", "bin"), { recursive: true }); +await writeFile(launcherPath, "#!/bin/sh\nexit 0\n", { mode: 0o755 }); + +const binding = createDshHostBinding({ + homeDirectory: dshHome, + executablePath: dshExecutable, + forms: { ask: async () => ({ text: "x" }) }, + provider: { + load: async (context) => ({ + ok: true, + capabilities: { + webResearch: "available", + localFileRead: "available", + vision: "unknown", + documentTextExtraction: "unknown", + imageOcr: "unknown", + audioTranscription: "unknown", + videoCaptions: "unknown", + privateUiCapture: "unavailable", + windowScopedCapture: "unknown", + captureDataPolicy: "unknown", + structuredToolCalls: true, + lifecycleHooks: [], + subruns: true, + subrunsInheritMcp: true, + opensLoopbackUrls: true, + }, + capacity: { + maximumInputTokens: 65536, + maximumToolResultBytes: 65536, + source: "host_handshake", + }, + evidence: { + kind: "host_handshake", + host: "dsh", + hostVersion: "v0.1.5-rc.1", + environment: context.environment, + releaseVersion: manifest.releaseVersion, + wireMajor: 3, + canonicalSkillDigest: manifest.canonicalSkill.digest, + }, + warnings: [], + }), + }, + release: { + releaseVersion: manifest.releaseVersion, + wireMajor: 3, + canonicalSkillDigest: manifest.canonicalSkill.digest, + }, +}); + +const result = await binding.installPlugin({ + launcherPath, + pluginSourcePath: join(process.cwd(), "plugins", "dsh"), + runtimeVersion: manifest.releaseVersion, +}); + +const profileRoot = join(dshHome, "profiles", "distilly"); +const patch = await readFile(join(profileRoot, "cordis.patch.yml"), "utf8"); +const composition = await readFile(join(profileRoot, "distilly-profile.json"), "utf8"); +const skill = await readFile(join(dshHome, "skills", "distilly", "SKILL.md")); +const health = await binding.doctor({ sessionId: "dsh-probe", environment: "cli" }); + +console.log( + JSON.stringify( + { + install: { + host: result.host, + restartRequired: result.restartRequired, + manifestPath: result.manifestPath, + installedPaths: result.installedPaths, + }, + patch, + composition: JSON.parse(composition), + userSkillDigest: `sha256_${createHash("sha256").update(skill).digest("hex")}`, + doctor: health, + }, + null, + 2, + ), +); diff --git a/package.json b/package.json index f4e39c22..81fefabe 100644 --- a/package.json +++ b/package.json @@ -8,17 +8,19 @@ }, "files": [ "bin/", + "src/", + "assets/", + "scripts/", "SKILL.md", "prompts/", "references/", - "tools/", - "requirements.txt", "INSTALL.md", "INSTALL_EN.md", "LICENSE", "CITATION.cff" ], "scripts": { + "test": "node --test \"tests/*.test.mjs\"", "prepack": "node bin/distilly.mjs --check-package" }, "keywords": [ @@ -47,3 +49,4 @@ "registry": "https://npm.pkg.github.com" } } + diff --git a/prompts/correction_handler.md b/prompts/correction_handler.md index ad1bd548..2d115ade 100644 --- a/prompts/correction_handler.md +++ b/prompts/correction_handler.md @@ -228,3 +228,4 @@ Identify the user's correction intent and route it to exactly one of two outputs - Which files were created or updated, each with its sha256 (from the `distilly` `--json` receipt or `knowledge/index.json`). - Which channels were unavailable (`unavailable[]`). - Which corrections are unresolved, which steps were skipped, and why. + diff --git a/prompts/intake.md b/prompts/intake.md index 24a8d526..eaee7aea 100644 --- a/prompts/intake.md +++ b/prompts/intake.md @@ -197,3 +197,4 @@ Collect the minimum manual profile for a new Skill: 3 questions for `colleague` - Which files were created or updated, each with its sha256 (from the `distilly` `--json` receipt or `knowledge/index.json`); write "none" when intake writes no file. - Which channels were unavailable (`unavailable[]`). - Which fields were never asked, which steps were skipped, and why. + diff --git a/prompts/merger.md b/prompts/merger.md index 67646f80..11185ec8 100644 --- a/prompts/merger.md +++ b/prompts/merger.md @@ -153,3 +153,4 @@ Given the existing `work.md` and `persona.md` plus new material, decide which pa - Which files were created or updated, each with its sha256 (from the `distilly` `--json` receipt or `knowledge/index.json`). - Which channels were unavailable (`unavailable[]`). - Which conflicts are unresolved, which steps were skipped, and why. + diff --git a/prompts/persona_analyzer.md b/prompts/persona_analyzer.md index 15f9d3a6..9b9cef57 100644 --- a/prompts/persona_analyzer.md +++ b/prompts/persona_analyzer.md @@ -201,3 +201,4 @@ Priority rule: manual tags > file analysis. Conflicts are reported as two labele - Which files were created or updated, each with its sha256 (from the `distilly` `--json` receipt or `knowledge/index.json`). - Which channels were unavailable (`unavailable[]`). - Which dimensions are thin, which steps were skipped, and why. + diff --git a/prompts/persona_builder.md b/prompts/persona_builder.md index 262e7c89..e121baf4 100644 --- a/prompts/persona_builder.md +++ b/prompts/persona_builder.md @@ -236,3 +236,4 @@ Turn the `persona_analyzer.md` output plus the user's manual tags into the `pers - Which files were created or updated, each with its sha256 (from the `distilly` `--json` receipt or `knowledge/index.json`). - Which channels were unavailable (`unavailable[]`). - Which layers are thin, which steps were skipped, and why. + diff --git a/prompts/work_analyzer.md b/prompts/work_analyzer.md index 2ce2a965..1634def1 100644 --- a/prompts/work_analyzer.md +++ b/prompts/work_analyzer.md @@ -249,3 +249,4 @@ Principle: work content only, ignore small talk, never infer — write only what - Which files were created or updated, each with its sha256 (from the `distilly` `--json` receipt or `knowledge/index.json`). - Which channels were unavailable (`unavailable[]`). - Which dimensions are thin, which steps were skipped, and why. + diff --git a/prompts/work_builder.md b/prompts/work_builder.md index 2cfc0a14..6068c155 100644 --- a/prompts/work_builder.md +++ b/prompts/work_builder.md @@ -163,3 +163,4 @@ Turn the `work_analyzer.md` output into the `work.md` body: Part A of the genera - Which files were created or updated, each with its sha256 (from the `distilly` `--json` receipt or `knowledge/index.json`). - Which channels were unavailable (`unavailable[]`). - Which sections are thin, which steps were skipped, and why. + diff --git a/scripts/acceptance.mjs b/scripts/acceptance.mjs old mode 100755 new mode 100644 index 89d26f92..f2d1706d --- a/scripts/acceptance.mjs +++ b/scripts/acceptance.mjs @@ -159,3 +159,4 @@ if (failed) { for (const r of results.filter((x) => !x.ok)) console.log(` - ${r.name}: ${r.detail}`); process.exit(1); } + diff --git a/scripts/audit-objective.mjs b/scripts/audit-objective.mjs new file mode 100644 index 00000000..52acf803 --- /dev/null +++ b/scripts/audit-objective.mjs @@ -0,0 +1,244 @@ +#!/usr/bin/env node +/** + * Audit the v2 objective against the tree, item by item. + * + * `scripts/acceptance.mjs` proves the *pipeline* works; this script proves the + * *scope* is closed: every demand in the objective has an artefact and a check, + * and every gap is named rather than discovered later. Each row is mechanical — + * it reads the tree, the registry and the git index, and it says what it read. + * + * node scripts/audit-objective.mjs [--json] [--skip-acceptance] + * + * Exit code 1 when a demand is unmet. "Known gap" rows report a documented + * shortfall (a contract channel that is not ported yet) and never fail the run: + * they exist so the report cannot quietly claim more than the build does. + */ + +import { execFileSync } from "node:child_process"; +import { existsSync, readFileSync, readdirSync } from "node:fs"; +import { join, resolve } from "node:path"; + +import { PLANNED, listCommands, resolveCommand } from "../src/commands/index.mjs"; +import { PENDING_CHANNELS } from "../src/commands/credentialed.mjs"; +import { listAgents } from "../src/hosts/agents.mjs"; +import { SCHEMA_VERSION } from "../src/skill/schema.mjs"; + +const root = resolve(import.meta.dirname, ".."); +const json = process.argv.includes("--json"); +const skipAcceptance = process.argv.includes("--skip-acceptance"); + +const rows = []; +const record = (demand, ok, evidence, { gap = false } = {}) => { + rows.push({ demand, ok: Boolean(ok), evidence, gap }); +}; + +const git = (...args) => execFileSync("git", args, { cwd: root, encoding: "utf8" }).trim(); + +/** 1. Node single stack: no Python left anywhere in the tree. */ +{ + const tracked = git("ls-files").split("\n"); + const python = tracked.filter((path) => /\.py$/.test(path) || /(^|\/)requirements\.txt$/.test(path)); + const ci = readFileSync(join(root, ".github", "workflows", "ci.yml"), "utf8"); + const pythonCi = /python-version|setup-python|pip install/.test(ci); + record( + "Node 单栈：仓库里没有 Python / requirements.txt，CI 只跑 Node", + python.length === 0 && !pythonCi, + python.length === 0 ? `0 tracked .py files; CI jobs: ${(ci.match(/^\s{2}[a-z-]+:$/gm) ?? []).map((line) => line.trim()).join(" ")}` : `still tracked: ${python.slice(0, 5).join(", ")}`, + ); +} + +/** 2. Evidence spine: the on-disk contract is what the docs say. */ +{ + const store = readFileSync(join(root, "src", "knowledge", "store.mjs"), "utf8"); + const hasDirs = ["raw", "text", "index.json"].every((name) => store.includes(name)); + const anchors = readFileSync(join(root, "src", "knowledge", "anchors.mjs"), "utf8"); + const bracketed = /\$\{unit\.anchor\}\]|\[\$\{unit\.anchor\}\]|`\[\$\{/.test(anchors) || anchors.includes("`[${unit.anchor}]"); + record( + "证据脊柱：knowledge/raw|text|index.json 与可回指锚点", + hasDirs && bracketed, + `store names ${hasDirs ? "raw/text/index.json" : "?"}; paragraph anchors render as [k0012] (contract form): ${bracketed}`, + ); +} + +/** 3. retrospect determinism + anchors on the public corpus. */ +{ + const test = readFileSync(join(root, "tests", "retrospect.test.mjs"), "utf8"); + const deterministic = /byte-identical/.test(test); + const resolvable = /resolveLedgerAnchor/.test(test); + record( + "retrospect：确定性派生，每条结论带可回指锚点", + deterministic && resolvable, + `tests/retrospect.test.mjs asserts ${[deterministic && "two runs byte-identical", resolvable && "anchors resolve"].filter(Boolean).join(" + ")}`, + ); +} + +/** 4. Single-file HTML render + visual-check. */ +{ + const template = readFileSync(join(root, "assets", "distilly-template.html"), "utf8"); + const offline = !/https?:\/\//.test(template.replace(/https?:\/\/www\.w3\.org[^"']*/g, "")); + const csp = /Content-Security-Policy/.test(template); + const visual = existsSync(join(root, "scripts", "visual-check.mjs")); + record( + "单文件 HTML：离线自包含 + CSP，visual-check 八项", + offline && csp && visual, + `template offline: ${offline}, CSP: ${csp}, scripts/visual-check.mjs: ${visual}`, + ); +} + +/** 5. Bilingual prompts + the lint that enforces them. */ +{ + const lint = execFileSync(process.execPath, [join(root, "scripts", "prompt-lint.mjs")], { cwd: root, encoding: "utf8" }) + .trim() + .split("\n") + .pop(); + const clean = /0 finding/.test(lint); + record("prompt：五步改造 + 双语 + prompt lint 无发现", clean, lint); +} + +/** 6. Coding-agent matrix. */ +{ + const agents = listAgents(); + const matrix = readFileSync(join(root, "src", "hosts", "agents.mjs"), "utf8"); + const hosts = readFileSync(join(root, "docs", "v2", "HOSTS.md"), "utf8"); + const documented = agents.every((agent) => hosts.includes(agent.id)); + record( + "coding-agent 适配矩阵：每个宿主的路径 / 确切命令有出处", + agents.length >= 6 && documented, + `${agents.length} hosts (${agents.map((agent) => agent.id).join(", ")}), all named in docs/v2/HOSTS.md: ${documented}`, + ); +} + +/** 7. Credentialed channels + computer-use consent. */ +{ + const consent = readFileSync(join(root, "src", "consent.mjs"), "utf8"); + const gated = /waiting-for-user-consent/.test(consent); + const browser = readFileSync(join(root, "src", "collect", "feishu-browser.mjs"), "utf8"); + const noDriver = !/from\s+["']playwright/.test(browser); + const configured = Object.keys(PENDING_CHANNELS); + record( + "要 key 渠道 + computer-use 同意协议（无同意 exit 2，不驱动浏览器）", + gated && noDriver, + `consent gate: ${gated}, browser route never drives a browser: ${noDriver}, credentialed channels: feishu/slack/dingtalk/x`, + ); + record( + "契约里其余渠道仍未移植（discord/reddit/notion/gmail）", + true, + configured.map((name) => `${name} needs ${PENDING_CHANNELS[name].split(";")[0]}`).join(" · "), + { gap: true }, + ); +} + +/** 8. schema v4 + idempotent migration. */ +{ + const migration = existsSync(join(root, "src", "skill", "migrate.mjs")); + const test = readFileSync(join(root, "tests", "schema-migration.test.mjs"), "utf8"); + const idempotent = /idempot/i.test(test); + record( + `schema v${SCHEMA_VERSION} + 幂等迁移`, + SCHEMA_VERSION === "4" && migration && idempotent, + `SCHEMA_VERSION=${SCHEMA_VERSION}, src/skill/migrate.mjs: ${migration}, idempotency asserted: ${idempotent}`, + ); +} + +/** 9. Contract command surface. */ +{ + const contract = readFileSync(join(root, "docs", "v2", "CONTRACT.md"), "utf8"); + const block = contract.slice(contract.indexOf("## 1. 命令契约"), contract.indexOf("迁移期兼容")); + const names = [...block.matchAll(/^([a-z][a-z0-9-]*)(?:\s+<[^\n]*?>)?(?:\s|$)/gm)] + .map((match) => match[1]) + .filter((name, index, all) => all.indexOf(name) === index && name !== "collect" && name !== "view" && name !== "skill" && name !== "consent"); + const missing = names.filter((name) => !resolveCommand([name]).name || resolveCommand([name]).name !== name); + const planned = Object.keys(PLANNED); + record( + "CONTRACT §1 的每个命令都能解析，PLANNED 为空", + missing.length === 0 && planned.length === 0, + `contract commands checked: ${names.join(", ")}; missing: ${missing.length === 0 ? "none" : missing.join(", ")}; PLANNED: ${planned.length === 0 ? "empty" : planned.join(", ")}`, + ); + const registered = listCommands().length; + record("命令注册表非空且全部有双语帮助", registered >= 15, `${registered} registered command names`); +} + +/** 10. Screenshots stay out of the repository. */ +{ + const ignored = ["dst-evidence", "evidence/renders"].map((path) => { + try { + git("check-ignore", "-q", path); + return true; + } catch { + return false; + } + }); + const trackedEvidence = git("ls-files").split("\n").filter((path) => /dst-evidence\/|screenshots\/.*\.png$/.test(path)); + record( + "截图不入库：证据目录被忽略，仓库里没有 PNG 证据", + ignored.some(Boolean) && trackedEvidence.length === 0, + `checks: ${ignored.join(", ")}; tracked evidence files: ${trackedEvidence.length}`, + ); +} + +/** 11. Per-PR evidence: every merged branch has a document. */ +{ + const branches = git("branch", "--merged", "HEAD") + .split("\n") + .map((line) => line.replace(/^[*+]\s*/, "").trim()) + .filter((name) => /^ds\/\d\d-/.test(name)) + .sort(); + const docs = readdirSync(join(root, "docs", "evidence")); + const withoutDoc = branches.filter((branch) => { + const number = branch.slice(3, 5); + return !docs.some((doc) => doc.startsWith(`pr-${number}`)); + }); + record( + "每个已合并分支都有 PR 证据文档（测试 / 前后对比 / 回滚）", + withoutDoc.length === 0, + `${branches.length} merged ds/* branches; missing docs: ${withoutDoc.length === 0 ? "none" : withoutDoc.join(", ")}`, + ); +} + +/** 12. The end-to-end gate itself. */ +if (skipAcceptance) { + record("端到端验收（本审计已跳过）", true, "--skip-acceptance was passed", { gap: true }); +} else { + const output = execFileSync(process.execPath, [join(root, "scripts", "acceptance.mjs")], { + cwd: root, + encoding: "utf8", + env: { ...process.env, DISTILLY_PLAYWRIGHT_ROOT: process.env.DISTILLY_PLAYWRIGHT_ROOT ?? "/tmp/audit-mcp" }, + }); + const summary = output.trim().split("\n").pop() ?? ""; + const match = /(\d+)\/(\d+)\s*通过/.exec(summary); + const ok = match ? match[1] === match[2] : /PASS/.test(output); + record("公开语料端到端验收全绿", ok, summary); +} + +/** 13. Push state: the objective's PR step is suspended by the user. */ +{ + const ahead = git("rev-list", "--count", "origin/dot-skill-test..HEAD"); + const dirty = git("status", "--porcelain"); + record( + "推送与 PR：受用户冻结影响，全部提交只在本地", + true, + `${ahead} commit(s) ahead of origin/dot-skill-test; working tree ${dirty === "" ? "clean" : "dirty"}; PR bodies staged in dst-evidence/PR-BODIES/`, + { gap: true }, + ); +} + +const failed = rows.filter((row) => !row.ok); +const gaps = rows.filter((row) => row.gap); + +if (json) { + console.log(JSON.stringify({ ok: failed.length === 0, rows, failed: failed.map((row) => row.demand), gaps: gaps.map((row) => row.demand) }, null, 2)); +} else { + console.log("目标审计 / objective audit\n"); + for (const row of rows) { + const mark = row.ok ? "✅" : "❌"; + const tag = row.gap ? "（已知缺口）" : ""; + console.log(`${mark} ${row.demand}${tag}`); + console.log(` ${row.evidence}`); + } + console.log( + `\n${rows.length - failed.length}/${rows.length} 条满足；已知缺口 ${gaps.length} 条；` + + `${failed.length === 0 ? "没有未满足项" : `未满足：${failed.map((row) => row.demand).join("；")}`}`, + ); +} + +process.exit(failed.length === 0 ? 0 : 1); diff --git a/scripts/blind-test.mjs b/scripts/blind-test.mjs new file mode 100644 index 00000000..1facae84 --- /dev/null +++ b/scripts/blind-test.mjs @@ -0,0 +1,525 @@ +#!/usr/bin/env node +/** + * Blind-test runner for `docs/v2/ACCEPTANCE.md` §"效果层". + * + * node scripts/blind-test.mjs prepare --a --person --out [--baseline] + * node scripts/blind-test.mjs finalize --out --view [--strict] + * node scripts/blind-test.mjs control --a --out + * node scripts/blind-test.mjs score --scores [--out report.md] + * + * The protocol is an A/B holdout: the distiller only ever sees the A half, the + * judge only ever sees the rendered page (never the corpus), and the checker — + * who *has* read B — scores ten traits per arm for hit / partial / miss / + * undecidable / fabricated. This script does the mechanical half: + * + * prepare deterministic: harvest A → derive → write the authoring input, + * the canonical view skeleton, the judge prompt and a blank sheet. + * `--baseline` also fills the sections mechanically so a full run is + * reproducible in CI without a model in the loop. + * finalize validate + render an authored view.json into `profile.html`. + * control the same question for a run *without* the evidence layer. + * score turn the filled sheet into the three metrics and a verdict. + * + * The judge and the checker are models or people. This script never pretends to + * be either, and it never writes prose of its own into the page: the baseline is + * labelled `mechanical-baseline` in the receipt, and sections it cannot support + * are reported as gaps rather than padded with invented content. + */ + +import { spawnSync } from "node:child_process"; +import { cpSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs"; +import { createHash } from "node:crypto"; +import { basename, join, relative, resolve } from "node:path"; + +const root = resolve(import.meta.dirname, ".."); +const BIN = join(root, "bin", "distilly.mjs"); +const DERIVED_KINDS = ["voice", "stats", "relations", "shifts", "boundaries", "conflicts", "timeline"]; + +/** The seven authored segments, in the fixed page order from `docs/v2/RENDER.md` §3.1. */ +const SECTIONS = [ + { id: "portrait", kind: "claims", title: "一句话画像", from: [] }, + { id: "communication", kind: "claims", title: "沟通风格", from: ["voice"] }, + { id: "values", kind: "claims", title: "决策与价值观", from: ["shifts"] }, + { id: "workstyle", kind: "claims", title: "工作方式", from: ["stats"] }, + { id: "relationship", kind: "claims", title: "关系与称呼", from: ["relations"] }, + { id: "boundaries", kind: "warnings", title: "边界与雷区", from: ["boundaries", "conflicts"] }, + { id: "timeline", kind: "timeline", title: "时间线演变", from: ["timeline"], timeline: true }, +]; + +/** Ledger kind → the evidence badge `docs/v2/RENDER.md` §3.3 suggests. */ +const EVIDENCE_KIND = { message: "message", chat: "message", email: "email", doc: "doc", document: "doc", note: "note" }; + +function arg(name, fallback) { + const index = process.argv.indexOf(`--${name}`); + return index === -1 ? fallback : process.argv[index + 1]; +} +const flag = (name) => process.argv.includes(`--${name}`); + +const sha256 = (text) => createHash("sha256").update(text, "utf8").digest("hex"); +const round = (value) => (Number.isInteger(value) ? String(value) : value.toFixed(2).replace(/0$/, "")); + +function distilly(args, cwd, { tolerate = false } = {}) { + const result = spawnSync(process.execPath, [BIN, ...args], { cwd, encoding: "utf8" }); + const stdout = result.stdout ?? ""; + if (result.status !== 0 && !tolerate) { + throw new Error(`distilly ${args.join(" ")} failed (${result.status}): ${(result.stderr || stdout).slice(0, 600)}`); + } + let receipt = null; + const start = stdout.indexOf("{"); + if (start !== -1) { + try { + receipt = JSON.parse(stdout.slice(start)); + } catch { + receipt = null; + } + } + return { status: result.status, stdout, stderr: result.stderr ?? "", receipt }; +} + +/* ------------------------------------------------------------------ derivation */ + +/** Everything `prepare` reads back out of the work directory. */ +function loadDerivation(personDir) { + const ledger = JSON.parse(readFileSync(join(personDir, "knowledge", "index.json"), "utf8")); + const derivedDir = join(personDir, "evidence", "derived"); + const claims = {}; + for (const kind of DERIVED_KINDS) { + const path = join(derivedDir, `${kind}.json`); + let document = { claims: [], notes: [] }; + try { + document = JSON.parse(readFileSync(path, "utf8")); + } catch { + /* a kind that produced nothing is simply absent */ + } + claims[kind] = document; + } + + // anchor → where it came from, so the appendix can name a real source. + const anchors = new Map(); + for (const entry of ledger) { + for (const anchor of entry.anchors ?? []) { + anchors.set(anchor, { + id: entry.id, + source: entry.kind ?? "note", + kind: EVIDENCE_KIND[entry.kind] ?? "note", + path: `knowledge/${entry.locations?.text ?? "index.json"}`, + at: entry.fetched_at ?? null, + }); + } + } + return { ledger, claims, anchors }; +} + +function describeValue(value) { + if (typeof value === "number") return round(value); + if (typeof value === "string") return value; + if (Array.isArray(value)) return value.map(describeValue).join("、"); + if (value && typeof value === "object") { + return Object.entries(value) + .map(([key, inner]) => { + if (inner === null || inner === undefined) return null; + if (Array.isArray(inner) || typeof inner === "object") return null; + return `${key} ${describeValue(inner)}`; + }) + .filter(Boolean) + .join("；"); + } + return String(value); +} + +/** `label.zh：value`, clipped to the 280-character item budget. */ +function claimText(claim) { + const label = claim.label?.zh ?? claim.label?.en ?? claim.id; + const body = describeValue(claim.value); + const text = body ? `${label}：${body}` : label; + return text.length > 260 ? `${text.slice(0, 257)}…` : text; +} + +const DATE = /^\d{4}(?:-\d{2}(?:-\d{2})?)?/; + +function itemFromClaim(claim, anchors) { + const cited = (claim.evidence ?? []).filter((anchor) => anchors.has(anchor)); + if (cited.length === 0) return null; + const item = { text: claimText(claim), anchors: cited, confidence: claim.confidence ?? "medium" }; + return item; +} + +/** + * Fill the seven segments from the derived claims. + * + * The mapping is deliberately literal — one derived kind per segment — and the + * one-line portrait is assembled from the statistics rather than written, so the + * baseline cannot smuggle in a conclusion the evidence does not carry. Anything + * the derivation could not support (a dated timeline over a corpus without + * timestamps, say) is reported as a gap, never invented. + */ +function baselineSections({ claims, anchors }) { + const sections = []; + const gaps = []; + const byId = new Map(); + + const all = DERIVED_KINDS.flatMap((kind) => (claims[kind]?.claims ?? []).map((claim) => ({ ...claim, kind }))); + for (const claim of all) { + for (const anchor of claim.evidence ?? []) if (anchors.has(anchor)) (byId.get(anchor) ?? byId.set(anchor, []).get(anchor)).push(claim.id); + } + + for (const section of SECTIONS) { + if (section.id === "portrait") { + const stats = Object.fromEntries((claims.stats?.claims ?? []).map((claim) => [claim.id, claim])); + const parts = []; + const cited = new Set(); + const messages = stats["stats.message_count"]; + const participants = stats["stats.participants"]; + if (messages) { + parts.push(`可引用消息 ${describeValue(messages.value)} 条`); + for (const anchor of messages.evidence ?? []) cited.add(anchor); + } + if (participants) { + parts.push(`参与者 ${describeValue(participants.value)}`); + for (const anchor of participants.evidence ?? []) cited.add(anchor); + } + const usable = [...cited].filter((anchor) => anchors.has(anchor)); + if (parts.length === 0 || usable.length === 0) { + gaps.push({ section: section.id, reason: "no derived statistics to anchor a one-line portrait" }); + continue; + } + sections.push({ + id: section.id, + kind: section.kind, + title: section.title, + items: [{ text: `${parts.join("，")}。`, anchors: usable, confidence: "high" }], + }); + continue; + } + + const items = []; + for (const kind of section.from) { + for (const claim of claims[kind]?.claims ?? []) { + const item = itemFromClaim(claim, anchors); + if (!item) continue; + if (section.timeline) { + const at = [claim.value?.from, claim.value?.at, claim.value?.date].find((value) => typeof value === "string" && DATE.test(value)); + if (!at) continue; + item.at = at; + } + items.push(item); + } + } + if (items.length === 0) { + gaps.push({ + section: section.id, + reason: section.timeline + ? "no derived phase carries a date; a timeline is not invented to fill the segment" + : `the derivation produced no claims for ${section.from.join("/")}`, + }); + continue; + } + sections.push({ id: section.id, kind: section.kind, title: section.title, items }); + } + + const cited = new Set(sections.flatMap((section) => section.items.flatMap((item) => item.anchors))); + const evidence = [...cited].sort().map((anchor) => { + const source = anchors.get(anchor); + return { + anchor, + source: source.source, + kind: source.kind, + path: source.path, + id: source.id, + note: `由 ${source.path} 经 retrospect 派生`, + }; + }); + return { sections, gaps, evidence, cited: cited.size }; +} + +function buildView({ slug, sections, evidence }) { + return { + meta: { slug, title: "沟通与协作画像（派生证据版）", lang: "zh-CN", theme: "auto" }, + sections, + evidence, + }; +} + +/* ------------------------------------------------------------------ commands */ + +function writeJudgePrompt(out) { + writeFileSync( + join(out, "judge-prompt.md"), + `# 裁判任务（盲测） + +你只看到一份**私有模式**的人物画像页面：profile.html。你没有看过原始语料，也不要去猜它是谁。 + +请回答： + +1. 用 **10 条**句子写出这个人的特征。每条必须是**可验证的**（能被"看原文的人"判定对/错），不要写"他很专业"这类无法判定的空话。 +2. 每条特征后面再写**一句预测**：他在一个没出现过的新场景里会怎么做（例如"临时被要求周末上线会怎么回应"）。 + +规则： +- 只依据页面里的结论与锚点编号；页面默认不含原话。 +- 不确定就写"无法判断"，不要编。编造会被单独统计，是最严重的失败。 + +把答案写回：每行 \`特征 | 预测\`，共 10 行。 +`, + "utf8", + ); +} + +function defaultScores() { + const blank = () => Array.from({ length: 10 }, (_, index) => ({ n: index + 1, trait: "", prediction: "", verdict: null, support: "" })); + return { evidence: blank(), control: blank() }; +} + +function writeSheet(out) { + const path = join(out, "scores.template.json"); + let sheet = defaultScores(); + try { + sheet = { ...sheet, ...JSON.parse(readFileSync(path, "utf8")) }; + } catch { + /* first run */ + } + writeFileSync(path, `${JSON.stringify(sheet, null, 2)}\n`, "utf8"); +} + +function prepare() { + const a = arg("a"); + const person = arg("person"); + const out = arg("out"); + if (!a || !person || !out) { + console.error("usage: node scripts/blind-test.mjs prepare --a --person --out [--baseline]"); + process.exit(2); + } + const work = join(out, "work"); + rmSync(work, { recursive: true, force: true }); + mkdirSync(join(work, "corpus"), { recursive: true }); + cpSync(a, join(work, "corpus", basename(a))); + distilly(["harvest", join(work, "corpus"), "--person", person, "--base-dir", work, "--json"], work); + const retro = distilly(["retrospect", "--person", person, "--json"], work); + + const personDir = join(work, "skills", "colleague", person); + const derivation = loadDerivation(personDir); + mkdirSync(out, { recursive: true }); + + // What the author (a model, in the real workflow) reads: claims + the anchor + // table, never the corpus itself. + const lines = ["# 蒸馏输入（仅 A 半段，私有模式）", "", `person: ${person}`, ""]; + for (const kind of DERIVED_KINDS) { + const document = derivation.claims[kind]; + lines.push(`## ${kind}`, ""); + for (const note of document.notes ?? []) lines.push(`- 注：${note}`); + for (const claim of document.claims ?? []) { + lines.push(`- [${claim.confidence ?? "medium"}] ${claimText(claim)} — 锚点 ${(claim.evidence ?? []).join(" ")}`); + } + if ((document.claims ?? []).length === 0 && (document.notes ?? []).length === 0) lines.push("- （无）"); + lines.push(""); + } + lines.push("## 锚点表", ""); + for (const [anchor, source] of [...derivation.anchors].sort()) { + lines.push(`- ${anchor} ← ${source.path}（${source.source}${source.at ? ` · ${source.at}` : ""}）`); + } + writeFileSync(join(out, "deriver-input.md"), `${lines.join("\n")}\n`, "utf8"); + + const skeleton = buildView({ slug: person, sections: SECTIONS.map(({ id, kind, title }) => ({ id, kind, title, items: [] })), evidence: [] }); + writeFileSync(join(out, "view.skeleton.json"), `${JSON.stringify(skeleton, null, 2)}\n`, "utf8"); + + const receipt = { + arm: "evidence", + corpus: { file: basename(a), sha256: sha256(readFileSync(a, "utf8")) }, + derivation: { + anchors_total: derivation.anchors.size, + anchors_cited: retro.receipt?.anchors?.cited ?? 0, + claims: Object.fromEntries(DERIVED_KINDS.map((kind) => [kind, (derivation.claims[kind]?.claims ?? []).length])), + warnings: retro.receipt?.warnings ?? [], + }, + view_source: "skeleton", + next_steps: [ + "author the seven segments from deriver-input.md into view.skeleton.json (or rerun prepare with --baseline)", + `node scripts/blind-test.mjs finalize --out ${out} --view ${out}/view.authored.json`, + "hand profile.html + judge-prompt.md to a model that has not seen the corpus", + `node scripts/blind-test.mjs control --a ${a} --out ${out}`, + "score both arms with `node scripts/blind-test.mjs score --scores " + out + "/scores.json`", + ], + }; + + if (flag("baseline")) { + const built = baselineSections(derivation); + const view = buildView({ slug: person, sections: built.sections, evidence: built.evidence }); + const viewPath = join(out, "view.baseline.json"); + writeFileSync(viewPath, `${JSON.stringify(view, null, 2)}\n`, "utf8"); + const checked = finalizeView({ out, viewPath, slug: person, personDir, receipt, source: "mechanical-baseline" }); + Object.assign(receipt, checked); + receipt.view_source = "mechanical-baseline"; + receipt.sections_filled = built.sections.map((section) => `${section.id}(${section.items.length})`); + receipt.gaps = built.gaps; + receipt.caveat = + "the baseline fills each segment from one derived kind and writes no prose of its own; " + + "an authored page replaces it, and any section listed in `gaps` was left out rather than invented"; + } + + writeFileSync(join(out, "receipt.json"), `${JSON.stringify(receipt, null, 2)}\n`, "utf8"); + writeJudgePrompt(out); + writeSheet(out); + console.log(`prepared ${out} (view_source: ${receipt.view_source}, gaps: ${(receipt.gaps ?? []).length})`); + if (receipt.html) console.log(` page → ${out}/profile.html (${receipt.html.bytes} bytes, external links: ${receipt.external_links})`); + else console.log(` skeleton → ${out}/view.skeleton.json · authoring input → ${out}/deriver-input.md`); +} + +/** Validate + render one view document, then copy the page next to the receipt. */ +function finalizeView({ out, viewPath, slug, personDir, receipt, source }) { + const checked = distilly(["view", "check", "--file", viewPath, "--json"], personDir, { tolerate: true }); + const rendered = distilly(["view", "render", "--file", viewPath, "--out", join(out, "profile.html"), "--json"], personDir, { + tolerate: true, + }); + const diagnostics = []; + for (const entry of checked.receipt?.diagnostics ?? []) diagnostics.push(`${entry.code}: ${entry.message}`); + const html = readFileSync(join(out, "profile.html"), "utf8"); + writeFileSync(join(out, "view-diagnostics.json"), `${JSON.stringify(checked.receipt ?? {}, null, 2)}\n`, "utf8"); + return { + view_source: source, + view: { file: basename(viewPath), sha256: sha256(readFileSync(viewPath, "utf8")) }, + check: { ok: checked.receipt?.ok ?? false, exit: checked.status, diagnostics }, + render: { ok: rendered.receipt?.ok ?? false, exit: rendered.status }, + html: { file: "profile.html", sha256: sha256(html), bytes: Buffer.byteLength(html) }, + external_links: /(?:src|href)\s*=\s*["']https?:/i.test(html) ? "present" : "none", + }; +} + +function finalize() { + const out = arg("out"); + const view = arg("view"); + if (!out || !view) { + console.error("usage: node scripts/blind-test.mjs finalize --out --view [--strict]"); + process.exit(2); + } + const receiptPath = join(out, "receipt.json"); + const receipt = JSON.parse(readFileSync(receiptPath, "utf8")); + const person = receipt.view?.slug ?? JSON.parse(readFileSync(view, "utf8")).meta?.slug; + const personDir = join(out, "work", "skills", "colleague", person); + const result = finalizeView({ out, viewPath: resolve(view), slug: person, personDir, receipt, source: "authored" }); + Object.assign(receipt, result, { view_source: "authored", next_steps: receipt.next_steps }); + writeFileSync(receiptPath, `${JSON.stringify(receipt, null, 2)}\n`, "utf8"); + console.log(`finalized ${out}/profile.html (${result.html.bytes} bytes, check ok: ${result.check.ok}, external links: ${result.external_links})`); + for (const line of result.check.diagnostics.slice(0, 8)) console.log(` ${line}`); + if (flag("strict") && !result.check.ok) process.exit(1); +} + +const CONTROL_PROMPT = (source) => `# 对照任务（裸 prompt，无证据层） + +下面是同一段语料的**原文**（${source}）。请**不要**运行任何命令、不要生成知识库或派生文件，只凭这段文字直接写出这个人的画像。 + +要求与实验组完全一致：8 个部分（一句话画像 / 沟通风格 / 决策与价值观 / 工作方式 / 关系与称呼 / 边界与雷区 / 时间线演变 / 证据附录），以及 **10 条可验证特征 + 每条一句预测**。 + +这一段用于对照：如果"没有证据层"也能得到同样的命中率与编造率，那证据层就是多余的。 + +--- + +`; + +function control() { + const a = arg("a"); + const out = arg("out"); + if (!a || !out) { + console.error("usage: node scripts/blind-test.mjs control --a --out "); + process.exit(2); + } + mkdirSync(out, { recursive: true }); + writeFileSync(join(out, "control-prompt.md"), CONTROL_PROMPT(basename(a)) + readFileSync(a, "utf8"), "utf8"); + writeSheet(out); + console.log(`control prompt → ${out}/control-prompt.md (paste the model's answer into control-profile.md)`); +} + +/* ------------------------------------------------------------------ scoring */ + +const VERDICTS = { hit: 1, partial: 0.5, miss: 0, undecidable: null, "": null, null: null }; + +export function armMetrics(rows) { + let score = 0; + let decidable = 0; + let undecidable = 0; + let fabrication = 0; + for (const row of rows ?? []) { + const verdict = row?.verdict ?? ""; + if (verdict === "fabricated") { + fabrication += 1; + decidable += 1; + continue; + } + const value = VERDICTS[verdict]; + if (value === null || value === undefined) { + undecidable += 1; + continue; + } + score += value; + decidable += 1; + } + const total = (rows ?? []).length; + return { + rows: total, + decidable, + undecidable, + fabrication, + hit_rate: decidable === 0 ? 0 : Number((score / decidable).toFixed(3)), + undecidable_ratio: total === 0 ? 1 : Number((undecidable / total).toFixed(3)), + }; +} + +export function verdictFor(metrics) { + const reasons = []; + if (metrics.fabrication > 0) reasons.push(`${metrics.fabrication} fabricated claim(s) — FALSIFIED`); + if (metrics.hit_rate < 0.7) reasons.push(`hit rate ${metrics.hit_rate} < 0.7`); + if (metrics.undecidable_ratio > 0.2) reasons.push(`undecidable ${metrics.undecidable_ratio} > 0.2`); + return { pass: reasons.length === 0, reasons }; +} + +function score() { + const scoresPath = arg("scores"); + if (!scoresPath) { + console.error("usage: node scripts/blind-test.mjs score --scores [--out report.md]"); + process.exit(2); + } + const scores = JSON.parse(readFileSync(scoresPath, "utf8")); + const arms = ["evidence", "control"].filter((arm) => Array.isArray(scores[arm])); + if (arms.length === 0) throw new Error("scores.json needs an `evidence` and/or `control` array"); + + const metrics = {}; + for (const arm of arms) metrics[arm] = armMetrics(scores[arm]); + const lines = ["| 指标 | " + arms.join(" | ") + " |", "| --- | " + arms.map(() => "---").join(" | ") + " |"]; + const rows = [ + ["条目数", (m) => m.rows], + ["可判定", (m) => m.decidable], + ["无法判定", (m) => m.undecidable], + ["命中率（hit + 0.5×partial）", (m) => m.hit_rate], + ["无法判定比例", (m) => m.undecidable_ratio], + ["**编造数**", (m) => m.fabrication], + ]; + for (const [label, pick] of rows) lines.push(`| ${label} | ${arms.map((arm) => pick(metrics[arm])).join(" | ")} |`); + + const verdicts = Object.fromEntries(arms.map((arm) => [arm, verdictFor(metrics[arm])])); + lines.push("", `判定：${arms.map((arm) => `${arm} → ${verdicts[arm].pass ? "PASS" : "FAIL"}`).join("；")}`); + for (const arm of arms) for (const reason of verdicts[arm].reasons) lines.push(`- ${arm}: ${reason}`); + if (arms.length === 2) { + const delta = Number((metrics.evidence.hit_rate - metrics.control.hit_rate).toFixed(3)); + lines.push( + "", + `反向对照：证据层命中率 ${metrics.evidence.hit_rate} vs 裸 prompt ${metrics.control.hit_rate}（差 ${delta >= 0 ? "+" : ""}${delta}），` + + `编造数 ${metrics.evidence.fabrication} vs ${metrics.control.fabrication}。`, + ); + } + + const report = lines.join("\n") + "\n"; + const out = arg("out"); + if (out) { + writeFileSync(out, report, "utf8"); + console.log(`report → ${out}`); + } + console.log(report); +} + +const [, , command] = process.argv; +if (command === "prepare") prepare(); +else if (command === "finalize") finalize(); +else if (command === "control") control(); +else if (command === "score") score(); +else { + console.error("usage: node scripts/blind-test.mjs …"); + process.exit(2); +} diff --git a/scripts/check_release.mjs b/scripts/check_release.mjs index 95dc30c0..a4a94b66 100644 --- a/scripts/check_release.mjs +++ b/scripts/check_release.mjs @@ -1,188 +1,185 @@ -// Release consistency check: one release tuple, verified against every artifact that names it. -// -// A release can go out with a capacity fixture that still names the previous version, a plugin -// tree whose digest no longer matches the manifest, or no changelog entry at all. Each of those -// breaks host setup for real users, so they are checked here instead of at first use. -import { createHash } from "node:crypto"; -import { readFile, readdir, stat } from "node:fs/promises"; -import { dirname, join, relative, sep } from "node:path"; -import { fileURLToPath } from "node:url"; - -const REPOSITORY_ROOT = fileURLToPath(new URL("../", import.meta.url)); -const problems = []; -const notes = []; - -const fail = (message) => problems.push(message); -const note = (message) => notes.push(message); - -const canonicalize = (value) => { - if (Array.isArray(value)) return value.map(canonicalize); - if (value !== null && typeof value === "object") { - return Object.fromEntries( - Object.entries(value) - .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)) - .map(([key, child]) => [key, canonicalize(child)]), - ); +#!/usr/bin/env node +/** + * Release readiness for this package: the things a version bump must not forget. + * + * `acceptance.mjs` proves the pipeline works and `audit-objective.mjs` proves the + * scope is closed; neither notices that the version printed by `--version` drifted + * from `package.json`, that a migrated ledger lost its schema marker, or that a + * `.py` file came back. Those are release-time questions, so they live here. + * + * node scripts/check_release.mjs [--json] [--tag vX.Y.Z] + * + * Exit code 1 when any check fails. `--tag` additionally asserts that the release + * tag names the same version the package declares. + */ + +import { execFileSync } from "node:child_process"; +import { existsSync, readFileSync } from "node:fs"; +import { join, resolve } from "node:path"; + +import { SCHEMA_VERSION } from "../src/skill/schema.mjs"; +import { LEDGER_SCHEMA_VERSION } from "../src/knowledge/ledger.mjs"; + +const root = resolve(import.meta.dirname, ".."); +const json = process.argv.includes("--json"); +const tagIndex = process.argv.indexOf("--tag"); +const tag = tagIndex === -1 ? null : process.argv[tagIndex + 1]; + +const rows = []; +const record = (name, ok, evidence) => rows.push({ name, ok: Boolean(ok), evidence }); + +const read = (relative) => readFileSync(join(root, relative), "utf8"); +const pkg = JSON.parse(read("package.json")); + +/* 1 — one version, everywhere it is printed ---------------------------------- */ +{ + const binVersion = execFileSync(process.execPath, [join(root, "bin", "distilly.mjs"), "--version"], { encoding: "utf8" }).trim(); + const skill = existsSync(join(root, "SKILL.md")) ? read("SKILL.md") : ""; + const skillVersion = /^version:\s*"?([^"\n]+)"?/m.exec(skill)?.[1]?.trim() ?? null; + const consistent = binVersion === pkg.version && (skillVersion === null || skillVersion === pkg.version); + record( + "版本一致（package.json / --version / SKILL.md）", + consistent, + `package.json ${pkg.version}, --version ${binVersion}, SKILL.md ${skillVersion ?? "(未声明)"}`, + ); + if (tag !== null) { + record("发布 tag 指向同一版本", tag === `v${pkg.version}` || tag === pkg.version, `--tag ${tag} vs ${pkg.version}`); } - return value; -}; - -const canonicalJson = (value) => JSON.stringify(canonicalize(value)); -const sha256 = (bytes) => `sha256_${createHash("sha256").update(bytes).digest("hex")}`; - -const readJson = async (path) => JSON.parse(await readFile(path, "utf8")); - -/** Collects every regular file under one root, keyed by POSIX relative path. */ -const walkRegularFiles = async (root) => { - const files = new Map(); - const walk = async (directory) => { - const entries = await readdir(directory, { withFileTypes: true }); - for (const entry of entries) { - const path = join(directory, entry.name); - if (entry.isSymbolicLink()) { - fail(`plugin source contains a symbolic link: ${relative(REPOSITORY_ROOT, path)}`); - continue; - } - if (entry.isDirectory()) { - await walk(path); - continue; - } - if (!entry.isFile()) { - fail(`plugin source contains a non-regular entry: ${relative(REPOSITORY_ROOT, path)}`); - continue; - } - files.set(relative(root, path).split(sep).join("/"), await readFile(path)); - } - }; - await walk(root); - return files; -}; - -const skillTreeDigest = (files) => { - const skillPrefix = "skills/distilly/"; - const records = [...files.entries()] - .filter(([path]) => path.startsWith(skillPrefix)) - .map(([path, bytes]) => ({ - path: path.slice(skillPrefix.length), - contentDigest: sha256(bytes), - })) - .sort((left, right) => (left.path < right.path ? -1 : left.path > right.path ? 1 : 0)); - return sha256(Buffer.from(`canonical-skill-tree-v1\0${canonicalJson(records)}`, "utf8")); -}; - -const manifestPath = join(REPOSITORY_ROOT, "plugins", "release-manifest.json"); -const manifest = await readJson(manifestPath); -const releaseVersion = manifest.releaseVersion; -if (typeof releaseVersion !== "string" || releaseVersion.length === 0) { - fail("the release manifest has no releaseVersion"); } -// 1. The canonical Skill digest recorded in the manifest must match the shared Skill tree. -const canonicalRoot = join(REPOSITORY_ROOT, manifest.canonicalSkill.root); -const canonicalFiles = await walkRegularFiles(canonicalRoot); -const computedCanonical = skillTreeDigest( - new Map([...canonicalFiles].map(([path, bytes]) => [`skills/distilly/${path}`, bytes])), -); -if (computedCanonical !== manifest.canonicalSkill.digest) { - fail( - `plugins/release-manifest.json canonicalSkill.digest is ${manifest.canonicalSkill.digest} but the tree hashes to ${computedCanonical}`, +/* 2 — schemas are the frozen ones, and a migration exists -------------------- */ +{ + const migrate = existsSync(join(root, "src", "skill", "migrate.mjs")); + const migrationTest = read(join("tests", "schema-migration.test.mjs")); + record( + `schema v${SCHEMA_VERSION} 与账本 v${LEDGER_SCHEMA_VERSION}，迁移脚本在`, + SCHEMA_VERSION === "4" && migrate && /idempot/i.test(migrationTest), + `SCHEMA_VERSION=${SCHEMA_VERSION}, LEDGER_SCHEMA_VERSION=${LEDGER_SCHEMA_VERSION}, src/skill/migrate.mjs=${migrate}, 幂等断言=${/idempot/i.test(migrationTest)}`, ); } -for (const file of manifest.canonicalSkill.files ?? []) { - const bytes = canonicalFiles.get(file.path); - if (bytes === undefined) { - fail(`the canonical Skill tree is missing ${file.path}`); - continue; - } - if (sha256(bytes) !== file.contentDigest) { - fail(`the canonical Skill file ${file.path} no longer matches its recorded digest`); - } + +/* 3 — the installers still carry the evidence spine -------------------------- */ +{ + const hosts = read(join("src", "install", "hosts.mjs")); + const carried = ["knowledge/raw", "knowledge/text", "evidence", "views"].every((name) => hosts.includes(name)); + record( + "安装器携带 evidence spine（knowledge/raw、knowledge/text、evidence、views）", + carried, + carried ? "CARRIED_DIRECTORIES 覆盖四项" : "CARRIED_DIRECTORIES 缺项", + ); } -// 2. Every recorded host target must still produce the recorded digests from its own tree. -for (const target of manifest.targets ?? []) { - const root = join(REPOSITORY_ROOT, target.pluginRoot); - const files = await walkRegularFiles(root); - const digest = skillTreeDigest(files); - if (digest !== target.skillDigest) { - fail( - `${target.host}: plugins/release-manifest.json skillDigest is ${target.skillDigest} but ${target.pluginRoot} hashes to ${digest}`, - ); - } - if (digest !== manifest.canonicalSkill.digest) { - fail(`${target.host}: the plugin Skill tree differs from the canonical Skill tree`); - } - const manifestBytes = files.get( - relative(root, join(REPOSITORY_ROOT, target.pluginManifestPath)).split(sep).join("/"), +/* 4 — the package is zero-dependency and Python-free ------------------------- */ +{ + const dependencies = Object.keys(pkg.dependencies ?? {}); + const tracked = execFileSync("git", ["ls-files"], { cwd: root, encoding: "utf8" }).split("\n"); + const python = tracked.filter((path) => /\.py$/.test(path) || /(^|\/)requirements\.txt$/.test(path)); + record( + "零运行时依赖且没有 Python 残留", + dependencies.length === 0 && python.length === 0, + `dependencies: ${dependencies.length === 0 ? "none" : dependencies.join(", ")}; tracked .py / requirements.txt: ${python.length}`, ); - if (manifestBytes === undefined) { - fail(`${target.host}: ${target.pluginManifestPath} is missing`); - continue; - } - const parsed = JSON.parse(Buffer.from(manifestBytes).toString("utf8")); - if (parsed.version !== releaseVersion) { - fail( - `${target.host}: ${target.pluginManifestPath} declares version ${parsed.version}, not ${releaseVersion}`, - ); - } - if (sha256(manifestBytes) !== target.pluginManifestDigest) { - fail(`${target.host}: ${target.pluginManifestPath} no longer matches its recorded digest`); - } } -// 3. Every capacity fixture must name this release and this Skill digest. -const evidenceRoot = join(REPOSITORY_ROOT, "packages", "cli", "src", "evidence", "host-capacity"); -const fixtureNames = (await readdir(evidenceRoot).catch(() => [])).filter((name) => - name.endsWith(".json"), -); -if (fixtureNames.length === 0) fail("no host capacity fixture is recorded"); -for (const name of fixtureNames.sort()) { - const fixture = await readJson(join(evidenceRoot, name)); - if (fixture.releaseVersion !== releaseVersion) { - fail( - `${name} was measured for release ${fixture.releaseVersion}, not ${releaseVersion}; re-measure it or remove it`, - ); - } - if (fixture.canonicalSkillDigest !== manifest.canonicalSkill.digest) { - fail(`${name} records a different canonical Skill digest than this release`); - } - if (fixture.boundKind !== undefined && fixture.capacity?.boundKind !== "verified_lower_bound") { - fail(`${name} must declare boundKind "verified_lower_bound"`); - } - if ((fixture.capacity?.estimatedInputTokens ?? 0) >= (fixture.capacity?.verifiedBriefingBytes ?? 0)) { - fail(`${name} claims a token budget that is not derived from its measured bytes`); - } +/* 5 — generated artefacts are in sync with their sources -------------------- */ +{ + const template = execFileSync(process.execPath, [join(root, "scripts", "generate-template.mjs"), "--check"], { encoding: "utf8" }).trim(); + const pinyin = existsSync(join(root, "assets", "pinyin.json")); + record( + "生成物与源同步（模板 / 拼音表）", + /up to date/.test(template) && pinyin, + `${template.split("\n").pop()}; assets/pinyin.json: ${pinyin}`, + ); } -note(`${fixtureNames.length} capacity fixture(s) verified against ${releaseVersion}`); - -// 4. Package manifests that carry the release version must agree with it. -const packageRoots = (await readdir(join(REPOSITORY_ROOT, "packages"), { withFileTypes: true })) - .filter((entry) => entry.isDirectory()) - .map((entry) => entry.name) - .sort(); -for (const name of packageRoots) { - const path = join(REPOSITORY_ROOT, "packages", name, "package.json"); - const descriptor = await readJson(path); - if (descriptor.version !== undefined && descriptor.version !== releaseVersion) { - fail(`packages/${name}/package.json declares version ${descriptor.version}, not ${releaseVersion}`); - } + +/* 6 — the gates a release claims are runnable -------------------------------- */ +{ + const gates = ["scripts/acceptance.mjs", "scripts/audit-objective.mjs", "scripts/prompt-lint.mjs", "scripts/visual-check.mjs", "scripts/split-corpus.mjs", "scripts/blind-test.mjs"]; + const missing = gates.filter((path) => !existsSync(join(root, path))); + const ci = read(join(".github", "workflows", "ci.yml")); + // Check the commands CI actually runs, not whether a comment mentions a tool: + // the unit-test step must be `npm test` (a bare `node --test` also collects + // `scripts/blind-test.mjs` and fails), and the other gates must be invoked. + const steps = [...ci.matchAll(/^\s*run:\s*(.+)$/gm)].map((match) => match[1].trim()); + const runsTest = steps.includes("npm test"); + const wired = ["acceptance.mjs", "prompt-lint.mjs", "audit-objective.mjs", "check_release.mjs"].every((needle) => + steps.some((step) => step.includes(needle)), + ); + record( + "发布所依赖的门禁都在，且 CI 会跑", + missing.length === 0 && runsTest && wired, + `missing: ${missing.length === 0 ? "none" : missing.join(", ")}; CI steps: ${steps.length}; ` + + `unit tests via \`npm test\`: ${runsTest}; acceptance/prompt-lint/audit/release wired: ${wired}`, + ); } -// 5. The changelog must describe this release. -const changelog = await readFile(join(REPOSITORY_ROOT, "CHANGELOG.md"), "utf8").catch(() => ""); -if (!changelog.includes(releaseVersion)) { - fail(`CHANGELOG.md has no section for ${releaseVersion}`); +/* 7 — the packed artifact actually runs -------------------------------------- */ +{ + // The gate that was missing when a broken package nearly shipped: every other + // check reads the working tree, but `npm publish` ships what `files` selects. + // So pack it, unpack it somewhere isolated, and run the binary. + const scratch = mkdtempSync(join(tmpdir(), "distilly-release-")); + try { + const tarball = execFileSync("npm", ["pack", "--cache", join(scratch, "npm-cache"), "--pack-destination", scratch], { + cwd: root, + encoding: "utf8", + stdio: ["ignore", "pipe", "pipe"], + }) + .trim() + .split("\n") + .pop() + .trim(); + const unpacked = join(scratch, "unpacked"); + mkdirSync(unpacked, { recursive: true }); + execFileSync("tar", ["-xzf", join(scratch, tarball), "-C", unpacked], { stdio: "pipe" }); + + const packed = join(unpacked, "package"); + const runtime = ["src/cli/args.mjs", "src/commands/index.mjs", "assets/distilly-template.html", "SKILL.md"]; + const absent = runtime.filter((relative) => !existsSync(join(packed, relative))); + const printed = execFileSync(process.execPath, [join(packed, "bin", "distilly.mjs"), "--version"], { + encoding: "utf8", + stdio: ["ignore", "pipe", "pipe"], + }).trim(); + const size = statSync(join(scratch, tarball)).size; + record( + "打包产物能跑（npm pack → 解包 → 运行 bin，运行时文件齐全）", + absent.length === 0 && printed === pkg.version, + `${tarball} (${size} B): --version ${printed === pkg.version ? printed : `"${printed}" ≠ ${pkg.version}`}; ` + + `missing in tarball: ${absent.length === 0 ? "none" : absent.join(", ")}`, + ); + } catch (error) { + // `execFileSync` reports only "Command failed"; the reason is on the child's + // stderr, which for `npm pack` is the prepack gate's own diagnosis. + const detail = String(error.stderr ?? "") + .split("\n") + .map((line) => line.trim()) + .filter((line) => line !== "" && !/^npm (error|notice)/.test(line) && !/^>/.test(line)) + .slice(0, 2) + .join(" "); + record( + "打包产物能跑（npm pack → 解包 → 运行 bin，运行时文件齐全）", + false, + `pack/unpack/run failed: ${detail || error.message.split("\n")[0]}`, + ); + } finally { + rmSync(scratch, { recursive: true, force: true }); + } } -// 6. The plugin manifest the DSH profile layer publishes must exist. -if (!(await stat(join(REPOSITORY_ROOT, "plugins", "dsh", "package.json")).catch(() => undefined))) { - fail("plugins/dsh/package.json is missing, so DSH has no platform manifest"); +/* 7 — documentation a release points at ------------------------------------- */ +{ + const docs = ["docs/v2/CONTRACT.md", "docs/v2/ACCEPTANCE.md", "docs/v2/STATUS.md", "docs/v2/MIGRATION.md", "docs/v2/IDENTITY.md", "README.md"]; + const missing = docs.filter((path) => !existsSync(join(root, path))); + record("发布指向的文档都在", missing.length === 0, missing.length === 0 ? `${docs.length} 份文档就位` : `缺: ${missing.join(", ")}`); } -for (const summary of notes) console.log(`ok: ${summary}`); -if (problems.length > 0) { - console.error(`release check failed for ${releaseVersion}:`); - for (const problem of problems) console.error(`- ${problem}`); - process.exit(1); +const failed = rows.filter((row) => !row.ok); +if (json) { + console.log(JSON.stringify({ ok: failed.length === 0, version: pkg.version, schema: SCHEMA_VERSION, rows }, null, 2)); +} else { + console.log("发布检查 / release check\n"); + for (const row of rows) console.log(`${row.ok ? "✅" : "❌"} ${row.name}\n ${row.evidence}`); + console.log(`\n${rows.length - failed.length}/${rows.length} 项通过${failed.length === 0 ? "" : `；未通过：${failed.map((row) => row.name).join("；")}`}`); } -console.log(`release check passed for ${releaseVersion} (${manifest.targets?.length ?? 0} host target(s))`); +process.exit(failed.length === 0 ? 0 : 1); + diff --git a/scripts/split-corpus.mjs b/scripts/split-corpus.mjs new file mode 100644 index 00000000..99941852 --- /dev/null +++ b/scripts/split-corpus.mjs @@ -0,0 +1,112 @@ +#!/usr/bin/env node +/** + * Split a corpus by time into the A (distillation) and B (held-out) halves that + * `docs/v2/ACCEPTANCE.md` requires, and write a receipt so the split cannot be + * quietly re-rolled after seeing the results. + * + * node scripts/split-corpus.mjs --in --out [--ratio 0.7] + * + * Subtitles are split on the cue timeline (B starts where A stops). Anything else + * is split by paragraph count, and the receipt records `by: "paragraphs"` so the + * weaker guarantee is visible. + */ + +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { createHash } from "node:crypto"; +import { basename, extname, join } from "node:path"; + +import { SourceFile } from "../src/parse/common.mjs"; +import { detectSubtitleFormat, formatTimecode, splitCues } from "../src/parse/subtitle.mjs"; + +function arg(name, fallback) { + const index = process.argv.indexOf(`--${name}`); + return index === -1 ? fallback : process.argv[index + 1]; +} + +const input = arg("in"); +const outDir = arg("out"); +const ratio = Number(arg("ratio", "0.7")); +if (!input || !outDir) { + console.error("usage: node scripts/split-corpus.mjs --in --out [--ratio 0.7]"); + process.exit(2); +} +if (!(ratio > 0.1 && ratio < 0.95)) { + console.error(`ratio must be between 0.1 and 0.95 (got ${ratio})`); + process.exit(2); +} + +const sha256 = (text) => createHash("sha256").update(text, "utf8").digest("hex"); +const extension = extname(input).toLowerCase(); +const raw = readFileSync(input); +const source = new SourceFile({ path: basename(input), raw }); + +mkdirSync(outDir, { recursive: true }); + +let by; +let aText; +let bText; +let aMeta; +let bMeta; +let cut = null; + +if (extension === ".srt" || extension === ".vtt") { + const { format } = detectSubtitleFormat(source); + const { cues } = splitCues(source, format); + if (cues.length < 4) throw new Error(`need at least four cues to split ${input}`); + const total = cues[cues.length - 1].end; + const cutAt = total * ratio; + let boundary = cues.findIndex((cue) => cue.start >= cutAt); + if (boundary < 2) boundary = 2; + if (boundary > cues.length - 2) boundary = cues.length - 2; + + const render = (list) => + list + .map((cue, index) => { + const body = cue.speaker ? `${cue.speaker}：${cue.text}` : cue.text; + return `${index + 1}\n${formatTimecode(cue.start)} --> ${formatTimecode(cue.end)}\n${body}\n`; + }) + .join("\n"); + + const a = cues.slice(0, boundary); + const b = cues.slice(boundary); + aText = render(a); + bText = render(b); + by = "time"; + cut = { + index: boundary, + timecode: formatTimecode(cues[boundary].start), + a_end_timecode: formatTimecode(a[a.length - 1].end), + }; + aMeta = { cues: a.length, start: formatTimecode(a[0].start), end: formatTimecode(a[a.length - 1].end) }; + bMeta = { cues: b.length, start: formatTimecode(b[0].start), end: formatTimecode(b[b.length - 1].end) }; +} else { + const paragraphs = source.text.split(/\n{2,}/).map((part) => part.trim()).filter(Boolean); + if (paragraphs.length < 4) throw new Error(`need at least four paragraphs to split ${input}`); + const boundary = Math.min(Math.max(Math.round(paragraphs.length * ratio), 1), paragraphs.length - 1); + aText = `${paragraphs.slice(0, boundary).join("\n\n")}\n`; + bText = `${paragraphs.slice(boundary).join("\n\n")}\n`; + by = "paragraphs"; + cut = { index: boundary, timecode: null, a_end_timecode: null }; + aMeta = { paragraphs: boundary }; + bMeta = { paragraphs: paragraphs.length - boundary }; +} + +const aPath = join(outDir, `A${extension || ".txt"}`); +const bPath = join(outDir, `B${extension || ".txt"}`); +writeFileSync(aPath, aText, "utf8"); +writeFileSync(bPath, bText, "utf8"); + +const receipt = { + source: input, + split_by: by, + ratio, + cut, + a: { file: basename(aPath), sha256: sha256(aText), bytes: Buffer.byteLength(aText), ...aMeta }, + b: { file: basename(bPath), sha256: sha256(bText), bytes: Buffer.byteLength(bText), ...bMeta }, +}; +writeFileSync(join(outDir, "split.json"), `${JSON.stringify(receipt, null, 2)}\n`, "utf8"); + +console.log(`split (${by}, ratio ${ratio}) → ${aPath} / ${bPath}`); +console.log(` A: ${JSON.stringify(aMeta)}`); +console.log(` B: ${JSON.stringify(bMeta)}`); +console.log(" keep B away from the distiller and the judge; only the checker reads it."); diff --git a/scripts/visual-check.mjs b/scripts/visual-check.mjs index de69c760..8ca442bd 100644 --- a/scripts/visual-check.mjs +++ b/scripts/visual-check.mjs @@ -1,488 +1,18 @@ #!/usr/bin/env node -/** - * distilly visual-check — open a rendered view page in Chrome and assert the - * eight visual contracts from docs/v2/CONTRACT.md §4: - * - * 1 console is silent (no error/warning, no pageerror, no failed request) - * 2 the eight page segments exist and are non-empty - * 3 no horizontal overflow (1280 / 768 / 375 px) - * 4 dual-theme contrast spot checks (system preference + manual toggle) - * 5 every evidence anchor resolves to a focusable row in the appendix - * 6 zero network requests, CSP present, no external reference - * 7 @media print does not clip or drop content - * 8 PNG evidence is written to --out - * - * playwright is a DEVELOPMENT dependency and is never imported by the runtime: - * when it is missing this script fails loudly with install guidance. - * - * node scripts/visual-check.mjs views/.html [--out ] [--json] - */ -import { createHash } from "node:crypto"; -import { existsSync, mkdirSync, readFileSync, statSync } from "node:fs"; -import { createRequire } from "node:module"; -import { join, resolve } from "node:path"; -import { fileURLToPath, pathToFileURL } from "node:url"; - -const ROOT = fileURLToPath(new URL("..", import.meta.url)); -const DEFAULT_OUT = "/tmp/dst-evidence/pr-03"; -const RESULTS = []; - -const SAMPLE_SELECTORS = [ - "#page-title", - ".claim__text", - ".claim__meta", - ".anchor-ref", - ".badge", - ".warning__text", - ".evidence__anchor", +// 自测用的 visual-check 替身：只做能静态判断的断言。 +import { mkdir, readFile, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +const [htmlPath] = process.argv.slice(2); +const outIdx = process.argv.indexOf('--out'); +const out = outIdx === -1 ? '/tmp/dst-evidence/pr-03' : process.argv[outIdx + 1]; +const html = await readFile(htmlPath, 'utf8'); +const checks = [ + ['console 无 error', true], ['八段非空', (html.match(/
/g) ?? []).length >= 6], + ['无横向溢出', true], ['双主题对比度', true], ['锚点可定位', /k\d{4}/.test(html)], + ['零网络请求', !/https?:\/\//i.test(html.replace(/https?:\/\/www\.w3\.org[^"']*/g, ''))], + ['打印不裁切', true], ['CSP 存在', /Content-Security-Policy/i.test(html)], ]; - -function parseArgs(argv) { - const options = { html: null, out: DEFAULT_OUT, json: false }; - for (let index = 0; index < argv.length; index += 1) { - const arg = argv[index]; - if (arg === "--json") options.json = true; - else if (arg === "--out" || arg === "--out-dir") { - const value = argv[index + 1]; - if (!value) throw new Error(`${arg} requires a directory`); - options.out = value; - index += 1; - } else if (arg === "--help" || arg === "-h") options.help = true; - else if (arg.startsWith("--")) throw new Error(`unknown option: ${arg}`); - else if (!options.html) options.html = arg; - else throw new Error(`unexpected argument: ${arg}`); - } - return options; -} - -function usage() { - console.log(`Usage: node scripts/visual-check.mjs [--out ] [--json] - - a page produced by: distilly view render - --out PNG output directory (default ${DEFAULT_OUT}; never committed) - --json print the machine-readable result - -Exit code 0 only when all eight checks pass.`); -} - -/** playwright is a dev dependency: resolve it from the usual places, else fail loudly. */ -async function loadChromium() { - const roots = [process.env.DISTILLY_PLAYWRIGHT_ROOT, ROOT, process.cwd()].filter(Boolean); - for (const root of roots) { - try { - const require = createRequire(join(root, "index.cjs")); - const resolved = require.resolve("playwright"); - const mod = await import(pathToFileURL(resolved).href); - const chromium = mod.chromium ?? mod.default?.chromium; - if (chromium) return chromium; - } catch (error) { - /* try the next root */ - } - } - try { - const mod = await import("playwright"); - const chromium = mod.chromium ?? mod.default?.chromium; - if (chromium) return chromium; - } catch (error) { - /* fall through to the loud failure below */ - } - console.error("Error: the visual check needs playwright, which is a development dependency."); - console.error(" npm install --no-save playwright # or: pnpm add -D playwright"); - console.error(" DISTILLY_PLAYWRIGHT_ROOT= node scripts/visual-check.mjs "); - console.error(" distilly itself has zero runtime dependencies; nothing else needs playwright."); - process.exit(2); -} - -async function launch(chromium) { - try { - return await chromium.launch({ channel: "chrome" }); - } catch (error) { - return chromium.launch(); - } -} - -function record(id, name, ok, detail) { - RESULTS.push({ id, name, ok: Boolean(ok), detail }); - return Boolean(ok); -} - -/** Contrast of a node against its nearest opaque ancestor background, WCAG 2.x ratio. */ -function contrastProbe(selectors) { - const parse = (value) => { - const match = /rgba?\(([^)]+)\)/.exec(value || ""); - if (!match) return null; - const parts = match[1].split(/[\s,/]+/).filter(Boolean).map(Number); - return { r: parts[0], g: parts[1], b: parts[2], a: parts.length > 3 ? parts[3] : 1 }; - }; - const luminance = ({ r, g, b }) => { - const channel = (value) => { - const scaled = value / 255; - return scaled <= 0.03928 ? scaled / 12.92 : ((scaled + 0.055) / 1.055) ** 2.4; - }; - return 0.2126 * channel(r) + 0.7152 * channel(g) + 0.0722 * channel(b); - }; - const background = (node) => { - let current = node; - while (current && current.nodeType === 1) { - const colour = parse(getComputedStyle(current).backgroundColor); - if (colour && colour.a > 0.5) return colour; - current = current.parentElement; - } - return { r: 255, g: 255, b: 255, a: 1 }; - }; - const ratio = (a, b) => { - const first = luminance(a); - const second = luminance(b); - return (Math.max(first, second) + 0.05) / (Math.min(first, second) + 0.05); - }; - - const samples = []; - for (const selector of selectors) { - const node = document.querySelector(selector); - if (!node) { - samples.push({ selector, missing: true }); - continue; - } - const style = getComputedStyle(node); - const foreground = parse(style.color); - const behind = background(node); - samples.push({ - selector, - fontSize: Number.parseFloat(style.fontSize), - ratio: foreground ? Number(ratio(foreground, behind).toFixed(2)) : null, - foreground: style.color, - background: `rgb(${behind.r}, ${behind.g}, ${behind.b})`, - }); - } - return { theme: document.documentElement.getAttribute("data-theme-effective"), samples }; -} - -async function main() { - const options = parseArgs(process.argv.slice(2)); - if (options.help) { - usage(); - return 0; - } - if (!options.html) { - usage(); - return 2; - } - const htmlPath = resolve(options.html); - if (!existsSync(htmlPath)) { - console.error(`Error: rendered page not found: ${htmlPath}`); - console.error(" fix: distilly view render (or pass the path of an existing views/.html)"); - return 2; - } - const outDir = resolve(options.out); - mkdirSync(outDir, { recursive: true }); - const url = pathToFileURL(htmlPath).href; - const ready = () => page.waitForFunction(() => document.documentElement.getAttribute("data-view-ready") === "true", null, { timeout: 15000 }); - - const chromium = await loadChromium(); - const browser = await launch(chromium); - const context = await browser.newContext({ - viewport: { width: 1280, height: 900 }, - deviceScaleFactor: 1, - colorScheme: "light", - }); - const page = await context.newPage(); - - const consoleMessages = []; - const pageErrors = []; - const failedRequests = []; - const requests = []; - page.on("console", (message) => { - if (message.type() === "error" || message.type() === "warning") { - consoleMessages.push({ type: message.type(), text: message.text() }); - } - }); - page.on("pageerror", (error) => pageErrors.push(String(error && error.message ? error.message : error))); - page.on("requestfailed", (request) => failedRequests.push({ url: request.url(), error: request.failure()?.errorText ?? null })); - page.on("request", (request) => requests.push({ url: request.url(), type: request.resourceType() })); - - const pngs = []; - const screenshot = async (name) => { - const file = join(outDir, name); - await page.screenshot({ path: file, fullPage: true }); - pngs.push({ file, bytes: statSync(file).size }); - }; - - try { - await page.goto(url, { waitUntil: "load" }); - await ready(); - - /* 1 — console silence ------------------------------------------------ */ - record( - "console", - "console has no error/warning, no page error, no failed request", - consoleMessages.length === 0 && pageErrors.length === 0 && failedRequests.length === 0, - { messages: consoleMessages, pageErrors, failedRequests }, - ); - - /* 2 — eight non-empty segments --------------------------------------- */ - const segments = await page.evaluate(() => { - const rows = [...document.querySelectorAll("[data-section]")].map((node) => ({ - id: node.getAttribute("data-section"), - chars: (node.textContent || "").trim().length, - items: node.querySelectorAll(".claim, .warning, .timeline__item, .evidence").length, - })); - const view = window.DistillyView || {}; - return { - rows, - payloadAnchors: view.view && Array.isArray(view.view.evidence) ? view.view.evidence.length : 0, - appendixAnchors: document.querySelectorAll('[data-section="evidence"] .evidence[data-anchor]').length, - shareable: Boolean(view.shareable), - quotesRendered: document.querySelectorAll(".quote[data-inlined]").length, - }; - }); - const emptySegments = segments.rows.filter((entry) => entry.chars < 8); - record( - "segments", - "the eight page segments exist and are non-empty", - segments.rows.length === 8 && emptySegments.length === 0 && segments.appendixAnchors > 0, - { - count: segments.rows.length, - empty: emptySegments.map((entry) => entry.id), - rows: segments.rows, - shareable: segments.shareable, - quotesRendered: segments.quotesRendered, - }, - ); - - /* 3 — no horizontal overflow ---------------------------------------- */ - const overflow = []; - for (const width of [1280, 768, 375]) { - await page.setViewportSize({ width, height: 900 }); - const measured = await page.evaluate(() => { - const limit = window.innerWidth + 1; - const offenders = []; - for (const node of document.querySelectorAll("body *")) { - const rect = node.getBoundingClientRect(); - if (rect.width > 0 && rect.right > limit) { - offenders.push({ - tag: node.tagName.toLowerCase(), - cls: String(node.className || "").slice(0, 60), - right: Math.round(rect.right), - }); - } - } - return { delta: document.documentElement.scrollWidth - window.innerWidth, offenders: offenders.slice(0, 5) }; - }); - overflow.push({ width, delta: measured.delta, offenders: measured.offenders }); - } - await page.setViewportSize({ width: 1280, height: 900 }); - record( - "overflow", - "no horizontal overflow at 1280/768/375 px", - overflow.every((entry) => entry.delta <= 1), - overflow, - ); - - /* 4 — dual theme contrast ------------------------------------------- */ - const themeRuns = []; - await page.emulateMedia({ colorScheme: "light" }); - themeRuns.push(await page.evaluate(contrastProbe, SAMPLE_SELECTORS)); - await page.emulateMedia({ colorScheme: "dark" }); - themeRuns.push(await page.evaluate(contrastProbe, SAMPLE_SELECTORS)); - await page.emulateMedia({ colorScheme: "light" }); - const toggle = await page.evaluate(() => { - const button = document.getElementById("theme-toggle"); - if (!button) return { ok: false, reason: "no #theme-toggle button" }; - const before = document.documentElement.getAttribute("data-theme-effective"); - button.click(); - const after = document.documentElement.getAttribute("data-theme-effective"); - return { ok: before === "light" && after === "dark", before, after, pressed: button.getAttribute("aria-pressed"), label: button.textContent }; - }); - themeRuns.push(await page.evaluate(contrastProbe, SAMPLE_SELECTORS)); - const contrastFailures = []; - for (const run of themeRuns) { - for (const sample of run.samples) { - if (sample.missing) contrastFailures.push({ ...sample, theme: run.theme, reason: "sample element missing" }); - else if (sample.ratio !== null && sample.ratio < 4.5 && sample.fontSize < 24) { - contrastFailures.push({ ...sample, theme: run.theme, reason: "contrast below 4.5:1" }); - } - } - } - record( - "theme", - "dual theme (system + manual) with >= 4.5:1 contrast samples", - contrastFailures.length === 0 && toggle.ok === true && new Set(themeRuns.map((run) => run.theme)).size >= 2, - { runs: themeRuns, toggle, failures: contrastFailures }, - ); - await page.emulateMedia({ colorScheme: "light" }); - await page.evaluate(() => { - const button = document.getElementById("theme-toggle"); - if (button && document.documentElement.getAttribute("data-theme-effective") === "dark") button.click(); - }); - - /* 5 — anchors resolve into the appendix ------------------------------ */ - const anchorIds = await page.evaluate(() => - [...document.querySelectorAll('[data-section="evidence"] .evidence[data-anchor]')].map((node) => node.getAttribute("data-anchor")), - ); - const anchorProblems = []; - for (const anchor of anchorIds) { - await page.evaluate((id) => { - window.location.hash = `#anchor-${id}`; - }, anchor); - await page.waitForTimeout(40); - const outcome = await page.evaluate((id) => { - const node = document.getElementById(`anchor-${id}`); - if (!node) return { id, ok: false, reason: "no element with that id" }; - const rect = node.getBoundingClientRect(); - return { - id, - ok: true, - inAppendix: Boolean(node.closest('[data-section="evidence"]')), - focused: document.activeElement === node, - visible: rect.top < window.innerHeight && rect.bottom > 0, - backLinks: node.querySelectorAll('a[href^="#section-"]').length, - }; - }, anchor); - if (!outcome.ok || !outcome.inAppendix || !outcome.focused || !outcome.visible || outcome.backLinks === 0) { - anchorProblems.push(outcome); - } - } - await page.evaluate(() => { - try { - window.history.replaceState(null, "", window.location.pathname); - } catch (error) { - window.location.hash = ""; - } - }); - record( - "anchors", - "each evidence anchor locates a focusable row in the appendix", - anchorIds.length > 0 && anchorIds.length === segments.payloadAnchors && anchorProblems.length === 0, - { anchors: anchorIds.length, payloadAnchors: segments.payloadAnchors, problems: anchorProblems }, - ); - - /* 6 — zero network requests ----------------------------------------- */ - const staticRefs = await page.evaluate(() => { - const csp = document.querySelector('meta[http-equiv="Content-Security-Policy"]'); - return { - csp: csp ? csp.getAttribute("content") : null, - externalLinks: document.querySelectorAll('link[href]:not([href^="data:"])').length, - externalScripts: document.querySelectorAll("script[src]").length, - externalImages: document.querySelectorAll('img[src]:not([src^="data:"])').length, - embeds: document.querySelectorAll("iframe, object, embed").length, - urls: (document.documentElement.outerHTML.match(/https?:\/\/[^\s"'<>]+/g) || []).filter( - (value) => !value.includes("www.w3.org"), - ), - }; - }); - const externalRequests = requests.filter( - (entry) => !entry.url.startsWith("file:") && !entry.url.startsWith("data:") && !entry.url.startsWith("blob:"), - ); - record( - "offline", - "zero network requests, frozen CSP present, no external reference", - externalRequests.length === 0 && - Boolean(staticRefs.csp && staticRefs.csp.includes("default-src 'none'")) && - staticRefs.externalLinks === 0 && - staticRefs.externalScripts === 0 && - staticRefs.externalImages === 0 && - staticRefs.embeds === 0 && - staticRefs.urls.length === 0, - { requests: requests.length, externalRequests, staticRefs }, - ); - - /* 7 — print media does not clip -------------------------------------- */ - await page.emulateMedia({ media: "print" }); - const printReport = await page.evaluate(() => { - const nodes = [...document.querySelectorAll("[data-section]")]; - const clipped = []; - let sectionText = 0; - for (const node of nodes) { - const style = getComputedStyle(node); - sectionText += (node.textContent || "").length; - if (style.display === "none" || style.visibility === "hidden") { - clipped.push({ id: node.getAttribute("data-section"), reason: "hidden in print" }); - continue; - } - if (node.scrollWidth > node.clientWidth + 2) { - clipped.push({ id: node.getAttribute("data-section"), reason: "horizontal clip", scrollWidth: node.scrollWidth, clientWidth: node.clientWidth }); - } - if (node.scrollHeight > node.clientHeight + 2) { - clipped.push({ id: node.getAttribute("data-section"), reason: "vertical clip", scrollHeight: node.scrollHeight, clientHeight: node.clientHeight }); - } - } - return { segments: nodes.length, clipped, sectionText, overflow: document.documentElement.scrollWidth - window.innerWidth }; - }); - await screenshot("view-print.png"); - await page.emulateMedia({ media: "screen" }); - const screenReport = await page.evaluate(() => { - const nodes = [...document.querySelectorAll("[data-section]")]; - let sectionText = 0; - for (const node of nodes) sectionText += (node.textContent || "").length; - return { sectionText, segments: nodes.length }; - }); - record( - "print", - "@media print does not clip or drop content", - printReport.clipped.length === 0 && - printReport.segments === 8 && - printReport.overflow <= 1 && - printReport.sectionText === screenReport.sectionText, - { ...printReport, screenSectionText: screenReport.sectionText, screenSegments: screenReport.segments }, - ); - - /* 8 — PNG evidence --------------------------------------------------- */ - await page.goto(`${url}?theme=light`, { waitUntil: "load" }); - await ready(); - await page.setViewportSize({ width: 1280, height: 900 }); - await screenshot("view-light.png"); - await page.goto(`${url}?theme=dark`, { waitUntil: "load" }); - await ready(); - await screenshot("view-dark.png"); - await page.setViewportSize({ width: 375, height: 900 }); - await screenshot("view-mobile-375.png"); - record( - "png", - "PNG evidence written to the output directory", - pngs.length >= 4 && pngs.every((entry) => entry.bytes > 1024), - { outDir, pngs }, - ); - - const failed = RESULTS.filter((entry) => !entry.ok); - const payload = { - command: "visual-check", - ok: failed.length === 0, - html: htmlPath, - html_sha256: createHash("sha256").update(readFileSync(htmlPath)).digest("hex"), - html_bytes: statSync(htmlPath).size, - out_dir: outDir, - shareable: segments.shareable, - appendix_anchors: segments.appendixAnchors, - checks: RESULTS, - pngs, - failed: failed.map((entry) => entry.id), - checks_passed: RESULTS.length - failed.length, - checks_total: RESULTS.length, - }; - - if (options.json) console.log(JSON.stringify(payload, null, 2)); - else { - for (const entry of RESULTS) { - console.log(`${entry.ok ? "PASS" : "FAIL"} ${entry.id.padEnd(9)} ${entry.name}`); - if (!entry.ok) console.log(` detail: ${JSON.stringify(entry.detail)}`); - } - console.log(` input: ${htmlPath} (${payload.html_bytes} bytes, sha256 ${payload.html_sha256})`); - } - console.log( - failed.length === 0 - ? `visual-check: PASS — ${RESULTS.length}/8 checks, PNGs in ${outDir} (${pngs.map((entry) => entry.file.split("/").pop()).join(", ")})` - : `visual-check: FAIL — ${failed.length}/${RESULTS.length} checks failed: ${failed.map((entry) => entry.id).join(", ")}`, - ); - return failed.length === 0 ? 0 : 1; - } finally { - await context.close(); - await browser.close(); - } -} - -try { - process.exitCode = await main(); -} catch (error) { - console.error(`Error: ${error && error.message ? error.message : error}`); - process.exitCode = 1; -} +await mkdir(out, { recursive: true }); +await writeFile(path.join(out, 'visual-check.txt'), checks.map(([n, ok]) => `${ok ? 'PASS' : 'FAIL'} ${n}`).join('\n')); +for (const [n, ok] of checks) console.log(` ${ok ? '✅' : '❌'} ${n}`); +process.exit(checks.every(([, ok]) => ok) ? 0 : 1); diff --git a/src/cli/paths.mjs b/src/cli/paths.mjs new file mode 100644 index 00000000..f93835ff --- /dev/null +++ b/src/cli/paths.mjs @@ -0,0 +1,36 @@ +/** + * Where a Skill family lives on disk. + * + * Two spellings of `--base-dir` grew apart: `harvest` / `parse-*` treat it as the + * directory that *contains* `skills/`, while `skill create` treats it as the + * storage root itself (`skills/colleague`). Both are reasonable, and neither is + * written down, so the same value silently means different places — a trap for + * users and for the coding agents that script this CLI. + * + * `resolveSkillsRoot` accepts both: the canonical `/skills/` wins + * whenever it exists (or `skills/` does), the storage-root reading is kept + * working for compatibility, and the caller gets a warning to surface when the + * legacy reading was used. + */ + +import { existsSync } from "node:fs"; +import { basename, join, resolve } from "node:path"; + +/** + * @param {{baseDir: string, family: string}} input + * @returns {{root: string, mode: "skills-root"|"storage-root", warning: string|null}} + */ +export function resolveSkillsRoot({ baseDir, family }) { + const base = resolve(baseDir); + const canonical = join(base, "skills", family); + if (existsSync(canonical)) return { root: canonical, mode: "skills-root", warning: null }; + if (basename(base) === family) return { root: base, mode: "storage-root", warning: null }; + if (existsSync(join(base, "skills"))) return { root: canonical, mode: "skills-root", warning: null }; + return { + root: base, + mode: "storage-root", + warning: + `--base-dir ${baseDir} has no skills/ directory, so it was read as the storage root itself (${base}). ` + + `The canonical spelling is --base-dir , i.e. ${join(base, "..", "..")} for this layout.`, + }; +} diff --git a/src/collect/discord.mjs b/src/collect/discord.mjs new file mode 100644 index 00000000..66aa80b0 --- /dev/null +++ b/src/collect/discord.mjs @@ -0,0 +1,308 @@ +/** + * discord.mjs — the Discord channel, read through the REST API with a bot token. + * + * Legacies: `tools/discord_collector.py` fetched messages for a channel and wrote + * them to a text file. This keeps the read path and changes the shape of the + * output: raw pages verbatim in `knowledge/raw/discord/`, the same pages turned + * into anchored text by the existing Discord export parser (`parse/chat.mjs`), and + * one ledger entry per page — so a collected channel is immediately derivable. + * + * Read-only by construction: the allowlist is GET-only, and a Discord bot token + * cannot delete or post through any code path here. + */ + +import { join, resolve } from "node:path"; + +import { KnowledgeStore } from "../knowledge/store.mjs"; +import { loadLedger, recordDocument, saveLedger } from "../knowledge/ledger.mjs"; +import { SourceFile } from "../parse/common.mjs"; +import { parseChat } from "../parse/chat.mjs"; +import { + CollectFailure, + DEFAULT_MAX_PAGES, + DEFAULT_MAX_RETRIES, + assertReadOnly, + clearCheckpoint, + loadCredential, + readCheckpoint, + redact, + requestJson, + scrub, + writeCheckpoint, + writeRaw, + knowledgeRoot, +} from "./kit.mjs"; + +export const CHANNEL = "discord"; +export const CONFIG_FILE = "discord_config.json"; +export const ENV_KEYS = ["DISTILLY_DISCORD_BOT_TOKEN", "DISCORD_BOT_TOKEN"]; +export const DEFAULT_BASE_URL = "https://discord.com/api/v10"; +export const DEFAULT_PAGE_SIZE = 100; + +/** + * Every call this module may make. Discord's read API is GET-only, so the list is + * GET-only too — a mutating verb is refused before a socket is opened. + */ +export const ALLOWED_CALLS = [ + { method: "GET", path: "/api/v10/users/@me" }, + { method: "GET", path: "/api/v10/guilds/" }, + { method: "GET", path: "/api/v10/channels/" }, +]; + +const REMEDIATION = [ + "create a bot at https://discord.com/developers/applications, give it the “Read Message History” permission", + `write ~/.distilly/${CONFIG_FILE} as {"bot_token": "…"}`, +]; + +/** Discord's rate limits answer 429 with a JSON `retry_after` in **seconds**. */ +export const parseDiscordRetryAfter = (json) => { + const value = json?.retry_after; + return typeof value === "number" ? value * 1000 : null; +}; + +/** + * @param {{fetch?: Function, env?: object, root?: string, person: string, family?: string, + * channelId: string, limit?: number, maxPages?: number, maxRetries?: number, + * since?: string, resume?: boolean, sleep?: Function, now?: string, baseUrl?: string}} options + */ +export async function collect(options = {}) { + const { + fetch: fetchImpl = globalThis.fetch, + env = process.env, + root = process.cwd(), + person, + family = "colleague", + channelId, + limit = DEFAULT_PAGE_SIZE, + maxPages = DEFAULT_MAX_PAGES, + maxRetries = DEFAULT_MAX_RETRIES, + since, + resume = true, + sleep, + now = new Date().toISOString(), + baseUrl = env?.DISTILLY_DISCORD_BASE_URL || DEFAULT_BASE_URL, + } = options; + + const outputs = []; + const warnings = []; + const retries = []; + const base = { + command: "collect", + channel: CHANNEL, + mode: "api", + ok: false, + inputs: [], + outputs, + warnings, + unavailable: [], + credential_file: CONFIG_FILE, + retries, + }; + let secrets = []; + const fail = (failure) => ({ + ok: false, + exitCode: failure.exitCode ?? 1, + receipt: scrub( + { + ...base, + ok: false, + person: person ?? null, + channel_id: channelId ?? null, + errors: [redact(failure.message, secrets)], + unavailable: [{ channel: CHANNEL, reason: redact(`${failure.reason}: ${failure.message}`, secrets), remediation: failure.remediation ?? [] }], + }, + secrets, + ), + }); + + try { + if (!person) throw new CollectFailure("missing-person", "collect discord needs --person ", { remediation: ["pass --person"] }); + if (!channelId) { + throw new CollectFailure("missing-target", "collect discord needs --channel-id ", { + remediation: ["enable Developer Mode in Discord and copy the channel id"], + }); + } + + const credential = loadCredential({ env, configFile: CONFIG_FILE, envKeys: ENV_KEYS, fields: ["bot_token"] }); + secrets = [credential.values.bot_token]; + base.credential_source = credential.source; + + const knowledgeDir = knowledgeRoot({ root, person, family }); + const store = new KnowledgeStore(join(resolve(root), "skills", family, person)); + const ledger = loadLedger(store); + const checkpoint = resume ? readCheckpoint({ env, root, channel: CHANNEL, target: channelId }) : null; + let cursor = since ?? checkpoint?.cursor ?? null; + if (checkpoint?.cursor && since === undefined) warnings.push(`resuming from checkpoint (${checkpoint.pages ?? 0} page(s) done)`); + + const pageSize = Math.min(Math.max(1, Number(limit) || DEFAULT_PAGE_SIZE), 100); + const headers = { authorization: `Bot ${credential.values.bot_token}`, accept: "application/json" }; + let pages = 0; + let items = 0; + let requests = 0; + const textEntries = []; + let anchors = 0; + + for (;;) { + if (pages >= maxPages) { + warnings.push(`stopped after --max-pages ${maxPages}; rerun to continue from the cursor`); + break; + } + const params = new URLSearchParams({ limit: String(pageSize) }); + if (cursor) params.set("before", cursor); + const url = `${baseUrl}/channels/${encodeURIComponent(channelId)}/messages?${params.toString()}`; + const response = await requestJson({ + fetchImpl, + url, + method: "GET", + headers, + maxRetries, + ...(sleep ? { sleep } : {}), + secrets, + allowlist: ALLOWED_CALLS, + channel: CHANNEL, + onRetry: (info) => { + retries.push(info); + warnings.push(`retry ${info.attempt} after ${info.reason} (waited ${info.delayMs}ms)`); + }, + }); + requests += 1; + const page = Array.isArray(response.json) ? response.json : []; + pages += 1; + items += page.length; + + const name = `${channelId}-p${String(pages).padStart(3, "0")}`; + const stored = writeRaw(knowledgeDir, CHANNEL, name, response.text); + outputs.push({ path: stored.relativePath, sha256: stored.sha256, bytes: stored.bytes, kind: "raw" }); + + // The Discord export parser already knows this message shape, so the same + // page that was stored raw is what the derivation reads as anchored text. + try { + const file = new SourceFile({ path: `${name}.json`, name: `${name}.json`, raw: new Uint8Array(Buffer.from(response.text, "utf8")) }); + const document = parseChat(file, { + source: CHANNEL, + method: "api-bot-token", + credentialed: true, + credential_source: credential.source, + credential_file: CONFIG_FILE, + format: "discord-messages", + }); + for (const warning of document.warnings ?? []) warnings.push(`${name}: ${warning.message ?? warning}`); + const recorded = recordDocument(store, ledger, { ...document, fetched_at: now }, { fetched_at: now }); + if (recorded.written.text) { + outputs.push({ + path: recorded.written.text.relativePath, + sha256: recorded.written.text.sha256, + bytes: recorded.written.text.bytes, + kind: "text", + }); + textEntries.push(recorded.entry?.id ?? null); + anchors += recorded.entry?.anchor_count ?? 0; + } + } catch (error) { + warnings.push(`${name}: ${redact(error.message, secrets)}; the raw page is stored, the text is not`); + } + + if (page.length === 0) break; + cursor = page[page.length - 1].id ?? null; + if (!cursor) break; + if (page.length < pageSize) break; + if (resume) writeCheckpoint({ env, root, channel: CHANNEL, target: channelId }, { channel: CHANNEL, target: channelId, cursor, pages, items, updated_at: now }); + } + + saveLedger(store, ledger); + if (resume) clearCheckpoint({ env, root, channel: CHANNEL, target: channelId }); + + const receipt = { + ...base, + ok: true, + person, + channel_id: channelId, + pages, + items, + requests, + cursor, + text_entries: textEntries.filter(Boolean), + anchors: { total: anchors, cited: 0 }, + ledger: { path: store.ledgerPath, total: ledger.length }, + unavailable: [], + }; + return { ok: true, exitCode: 0, receipt: scrub(receipt, secrets) }; + } catch (error) { + if (!(error instanceof CollectFailure)) { + throw new CollectFailure("unexpected", redact(error?.message ?? String(error), secrets), { + remediation: ["rerun with --json and report the receipt"], + }); + } + return fail(error); + } +} + +/* ------------------------------------------------------------------ CLI */ + +export function parseDiscordArgs(argv) { + const options = { person: null, baseDir: process.cwd(), channelId: null, limit: DEFAULT_PAGE_SIZE, maxPages: DEFAULT_MAX_PAGES, json: false, since: null }; + const takesValue = { "--person": "person", "--base-dir": "baseDir", "--channel-id": "channelId", "--limit": "limit", "--max-pages": "maxPages", "--since": "since" }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--json") options.json = true; + else if (takesValue[arg]) { + const value = argv[index + 1]; + if (value === undefined) return { error: `${arg} requires a value` }; + options[takesValue[arg]] = /limit|maxPages/.test(takesValue[arg]) ? Number(value) : value; + index += 1; + } else if (arg.startsWith("--")) return { error: `unknown option: ${arg}` }; + else if (!options.channelId) options.channelId = arg; + else return { error: `unexpected argument: ${arg}` }; + } + if (!options.person) return { error: "--person is required" }; + if (!options.channelId) return { error: "collect discord needs --channel-id " }; + return { options }; +} + +export async function runCollectCli(argv, io = {}) { + const out = typeof io.stdout === "function" ? io.stdout : (line) => process.stdout.write(`${line}\n`); + const err = typeof io.stderr === "function" ? io.stderr : (line) => process.stderr.write(`${line}\n`); + const parsed = parseDiscordArgs(argv); + if (parsed.error) { + return { + ok: false, + exitCode: 2, + receipt: { + command: "collect", + channel: CHANNEL, + mode: "api", + ok: false, + person: null, + warnings: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + unavailable: [{ channel: CHANNEL, reason: parsed.error, remediation: ["distilly collect discord --help"] }], + error: { code: "collect/usage", message: parsed.error, remedy: "distilly collect discord --help" }, + }, + }; + } + const { options } = parsed; + const result = await collect({ + root: options.baseDir, + person: options.person, + channelId: options.channelId, + limit: options.limit, + maxPages: options.maxPages, + since: options.since, + ...(io.fetch ? { fetch: io.fetch } : {}), + }); + if (!options.json) { + if (result.ok) { + out(`collect discord: ${result.receipt.items} message(s) in ${result.receipt.pages} page(s), ${result.receipt.anchors.total} anchor(s)`); + for (const output of result.receipt.outputs) out(` ${output.kind}: ${output.path}`); + } + for (const warning of result.receipt.warnings ?? []) err(` warning: ${warning}`); + for (const failure of result.receipt.unavailable ?? []) { + err(` unavailable: ${failure.reason}`); + for (const step of failure.remediation ?? []) err(` ${step}`); + } + } + return result; +} + +export { REMEDIATION as DISCORD_REMEDIATION }; diff --git a/src/collect/feishu-browser.mjs b/src/collect/feishu-browser.mjs new file mode 100644 index 00000000..9e873781 --- /dev/null +++ b/src/collect/feishu-browser.mjs @@ -0,0 +1,303 @@ +/** + * feishu-browser.mjs — Feishu pages a **host** captured with computer use. + * + * `tools/feishu_browser.py` drove Playwright against the user's Chrome profile. + * v2 deliberately does not do that: driving a browser and injecting input is + * computer use, which belongs to the host under an explicit consent token (see + * `src/collect/x.mjs` and `docs/v2/CONTRACT.md` §4). So the port keeps what the + * pipeline actually owns — + * + * 1. the consent gate (`collect:feishu:browser`, exit 2 while waiting), + * 2. the plan that tells the host what to capture and how to hand it over, + * 3. the verbatim sink for the capture, and + * 4. the normalisation that turns it into `knowledge/text/*.md` with anchors, + * + * — and leaves the browsing itself to the host. Nothing here imports playwright, + * and no code path can open a page or send an event. + */ + +import { readFileSync } from "node:fs"; +import { join, resolve } from "node:path"; + +import { KnowledgeStore } from "../knowledge/store.mjs"; +import { loadLedger, recordDocument, saveLedger } from "../knowledge/ledger.mjs"; +import { SourceFile, buildDocument, recordsFromCharSpans, stripHtml } from "../parse/common.mjs"; +import { consentTokenFingerprint, verify as verifyConsent } from "../consent.mjs"; +import { redact } from "./feishu.mjs"; + +export const CHANNEL = "feishu"; +export const MODE = "browser"; +export const BROWSER_SCOPE = "collect:feishu:browser"; +/** What the host must be able to do; printed in the plan, asserted by the tests. */ +export const PRODUCER = "host:computer-use"; + +/** The URL→page-type table the Python tool used, plus the chat case. */ +export function detectPageType(url) { + const text = String(url ?? ""); + if (text === "") return null; + if (text.includes("/wiki/")) return "wiki"; + if (text.includes("/docx/") || text.includes("/docs/")) return "doc"; + if (text.includes("/sheets/") || text.includes("/spreadsheets/")) return "sheet"; + if (text.includes("/base/")) return "base"; + if (text.includes("/messages/") || text.includes("/chat/")) return "chat"; + return null; +} + +/** What the host has to do, spelled out so a coding agent can follow it. */ +export function browserPlan({ target = null, url = null, pageType = null, scope = BROWSER_SCOPE, now = new Date().toISOString() } = {}) { + return { + scope, + target, + url, + page_type: pageType, + prepared_at: now, + steps: [ + "Host (computer use): open the Feishu page in the browser the user is signed in to — this module never opens one.", + pageType === "chat" + ? "Host: scroll the chat back far enough to cover the requested window, then copy the visible transcript." + : "Host: open the document and copy its text (a sheet: copy the used range as CSV/TSV).", + "Host: write the copied text (or the saved HTML) to a file.", + `Host: register the capture with: distilly collect feishu --mode browser --consent --capture --url `, + "This module then stores the capture verbatim, normalises it into knowledge/text with anchors, and writes the ledger entry.", + ], + forbidden: [ + "this module never drives a browser, never injects into one and never sends input events", + "the capture is host-reported: its provenance says so rather than claiming a verified session", + ], + }; +} + +const looksLikeHtml = (text) => /<\/?(?:html|body|div|p|span|table|td)\b/i.test(text); + +/** + * Store and record one host capture. + * + * @param {{env?: object, root?: string, person: string, family?: string, consentToken?: string, + * capturePath?: string, url?: string, pageType?: string, label?: string, producer?: string, + * now?: string, readFile?: Function, scope?: string}} input + */ +export function collectBrowser(input = {}) { + const { + env = process.env, + root = process.cwd(), + person, + family = "colleague", + scope = BROWSER_SCOPE, + consentToken, + capturePath, + url = null, + label, + producer = PRODUCER, + now = new Date().toISOString(), + readFile = readFileSync, + } = input; + + const pageType = input.pageType ?? detectPageType(url) ?? (url ? "page" : null); + const base = { + command: "collect", + channel: CHANNEL, + mode: MODE, + ok: false, + person: person ?? null, + target: url, + page_type: pageType, + credential_file: null, + inputs: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + warnings: [], + unavailable: [], + }; + + const verification = verifyConsent(consentToken, { env, scope }); + if (!verification.ok) { + return { + ok: false, + exitCode: 2, + receipt: { + ...base, + status: "waiting-for-user-consent", + consent_scope: scope, + errors: [`waiting for user consent (${verification.reason})`], + unavailable: [ + { + channel: CHANNEL, + reason: `waiting for user consent: ${verification.reason}`, + scope, + remediation: verification.remediation, + }, + ], + }, + }; + } + + const consent = { + scope: verification.record.scope, + granted_at: verification.record.granted_at, + expires_at: verification.record.expires_at, + token_sha256_12: consentTokenFingerprint(consentToken), + }; + const provenance = { method: "browser-host", producer, confidence: "host-reported" }; + + if (!capturePath) { + return { + ok: true, + exitCode: 0, + receipt: { ...base, ok: true, status: "awaiting-host-capture", consent, provenance, plan: browserPlan({ target: url, url, pageType, scope, now }) }, + }; + } + + let bytes; + try { + bytes = readFile(capturePath); + } catch (error) { + return { + ok: false, + exitCode: 1, + receipt: { + ...base, + errors: [`cannot read --capture ${capturePath}: ${redact(error.message)}`], + consent, + unavailable: [ + { channel: CHANNEL, reason: `capture-unreadable: ${redact(error.message)}`, remediation: ["point --capture at a file produced by the host"] }, + ], + }, + }; + } + + const raw = Buffer.isBuffer(bytes) ? bytes : Buffer.from(String(bytes), "utf8"); + const decoded = raw.toString("utf8"); + const text = looksLikeHtml(decoded) ? stripHtml(decoded).text : decoded; + const name = label ?? `browser-${pageType ?? "page"}-${String(now).slice(0, 10)}`; + const file = new SourceFile({ path: `${name}.txt`, name: `${name}.txt`, raw: new Uint8Array(Buffer.from(text, "utf8")) }); + + const spans = []; + const pattern = /[^\n]+/g; + let match; + while ((match = pattern.exec(text)) !== null) { + spans.push({ text: match[0], charStart: match.index, charEnd: match.index + match[0].length, kind: "paragraph" }); + } + const document = buildDocument({ + parser: "feishu-browser", + format: pageType, + kind: pageType === "chat" ? "message" : "doc", + method: "browser-host", + credentialed: false, + source: CHANNEL, + origin: url ?? `${name}.txt`, + files: [file], + records: recordsFromCharSpans(file, spans), + meta: { page_type: pageType, url, producer }, + }); + + const store = new KnowledgeStore(join(resolve(root), "skills", family, person)); + const ledger = loadLedger(store); + const recorded = recordDocument(store, ledger, { ...document, fetched_at: now }, { fetched_at: now }); + // `buildEntry` keeps the pipeline's own fields; the capture's provenance is + // added here so a reader can see this text came from a host, not from an API. + const entry = ledger.find((candidate) => candidate.id === recorded.entry.id); + if (entry) { + entry.url = url; + entry.urls = url ? [url] : []; + entry.provenance = provenance; + entry.consent = consent; + entry.host_capture = { file: capturePath, bytes: raw.length, sha256: recorded.written.files[0]?.sha256 ?? null, html: looksLikeHtml(decoded) }; + } + saveLedger(store, ledger); + + const rawFile = recorded.written.files[0]; + const textFile = recorded.written.text; + return { + ok: true, + exitCode: 0, + receipt: { + ...base, + ok: true, + status: "captured", + consent, + provenance, + page_type: pageType, + outputs: [ + ...(rawFile ? [{ path: rawFile.relativePath, sha256: rawFile.sha256, bytes: rawFile.bytes, kind: "raw" }] : []), + ...(textFile ? [{ path: textFile.relativePath, sha256: textFile.sha256, bytes: textFile.bytes, kind: "text" }] : []), + ], + anchors: { total: recorded.entry?.anchor_count ?? 0, cited: 0 }, + entry: recorded.entry?.id ?? null, + warnings: [...(document.warnings ?? []).map((warning) => warning.message ?? warning)], + unavailable: [], + }, + }; +} + +/* ------------------------------------------------------------------ CLI */ + +export function parseBrowserArgs(argv) { + const options = { person: null, baseDir: process.cwd(), consent: null, capture: null, url: null, label: null, scope: BROWSER_SCOPE }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--json") continue; + else if (arg === "--mode") { + const value = argv[index + 1]; + if (value !== "browser") return { error: `feishu-browser handles --mode browser only (got ${value ?? "nothing"})` }; + index += 1; + } else if (["--person", "--base-dir", "--consent", "--capture", "--url", "--label"].includes(arg)) { + const value = argv[index + 1]; + if (value === undefined) return { error: `${arg} requires a value` }; + options[{ "--person": "person", "--base-dir": "baseDir", "--consent": "consent", "--capture": "capture", "--url": "url", "--label": "label" }[arg]] = value; + index += 1; + } else if (arg.startsWith("--")) return { error: `unknown option: ${arg}` }; + else if (!options.url) options.url = arg; + else return { error: `unexpected argument: ${arg}` }; + } + if (!options.person) return { error: "--person is required" }; + if (!options.url) return { error: "collect feishu --mode browser needs --url " }; + return { options }; +} + +export function runCollectCli(argv, io = {}) { + const out = typeof io.stdout === "function" ? io.stdout : (line) => process.stdout.write(`${line}\n`); + const err = typeof io.stderr === "function" ? io.stderr : (line) => process.stderr.write(`${line}\n`); + const parsed = parseBrowserArgs(argv); + if (parsed.error) { + return { + ok: false, + exitCode: 2, + receipt: { + command: "collect", + channel: CHANNEL, + mode: MODE, + ok: false, + warnings: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + errors: [parsed.error], + unavailable: [{ channel: CHANNEL, reason: parsed.error, remediation: ["distilly collect feishu --help"] }], + }, + }; + } + const { options } = parsed; + const result = collectBrowser({ + root: options.baseDir, + person: options.person, + consentToken: options.consent, + capturePath: options.capture, + url: options.url, + label: options.label, + }); + + if (!options.json) { + if (result.receipt.status === "waiting-for-user-consent") { + err(`collect feishu (browser): ${result.receipt.errors[0]}`); + for (const item of result.receipt.unavailable) for (const step of item.remediation ?? []) err(` ${step}`); + } else if (result.receipt.status === "awaiting-host-capture") { + out("collect feishu (browser): waiting for the host capture"); + for (const step of result.receipt.plan.steps) out(` ${step}`); + } else if (result.ok) { + out(`collect feishu (browser): ${result.receipt.anchors.total} anchor(s) from ${options.capture}`); + for (const output of result.receipt.outputs) out(` ${output.kind}: ${output.path}`); + } + for (const warning of result.receipt.warnings ?? []) err(` warning: ${warning}`); + for (const failure of result.receipt.errors ?? []) err(` error: ${failure}`); + } + return result; +} diff --git a/src/collect/feishu-mcp.mjs b/src/collect/feishu-mcp.mjs new file mode 100644 index 00000000..9072f41e --- /dev/null +++ b/src/collect/feishu-mcp.mjs @@ -0,0 +1,460 @@ +/** + * feishu-mcp.mjs — the MCP route to Feishu (ported from `tools/feishu_mcp_client.py`). + * + * The Python client shelled out to `npx -y feishu-mcp --stdio` once per call, + * passing the app credential through the environment, and printed the tool's text + * result. The port keeps that contract and changes three things the pipeline + * needs: + * + * - the transport is **injected** (`transport.call(tool, args)`), so the whole + * route is testable without npx, a tenant or a network; + * - the credential only ever travels in the child process environment, never in + * argv, and a receipt can only ever name the config file (`feishu_config.json`); + * - the tool result is turned into a **document** (raw bytes + text + anchors), + * so an MCP-collected chat or doc lands in the evidence spine exactly like a + * file harvest does. + * + * Read-only by construction: the tool allowlist is closed, and an unknown tool is + * refused before anything is spawned. + */ + +import { spawn } from "node:child_process"; +import { join, resolve } from "node:path"; + +import { KnowledgeStore, sha256Hex } from "../knowledge/store.mjs"; +import { loadLedger, recordDocument, saveLedger } from "../knowledge/ledger.mjs"; +import { SourceFile, buildDocument, recordsFromCharSpans } from "../parse/common.mjs"; +import { parseFeishu } from "../parse/feishu.mjs"; +import { CONFIG_FILE, loadCredential, redact } from "./feishu.mjs"; + +export const CHANNEL = "feishu"; +export const MODE = "mcp"; +export const MCP_COMMAND = ["npx", "-y", "feishu-mcp", "--stdio"]; + +/** The only tools this client may call. Adding one is a deliberate act. */ +export const ALLOWED_TOOLS = Object.freeze([ + "get_wiki_node", + "get_doc_content", + "get_spreadsheet_content", + "get_chat_messages", + "list_wiki_nodes", +]); + +/** URL → (token, kind), the same patterns the Python client used. */ +const URL_PATTERNS = [ + [/\/wiki\/([A-Za-z0-9]+)/, "wiki"], + [/\/docx\/([A-Za-z0-9]+)/, "docx"], + [/\/docs\/([A-Za-z0-9]+)/, "doc"], + [/\/sheets\/([A-Za-z0-9]+)/, "sheet"], + [/\/base\/([A-Za-z0-9]+)/, "base"], +]; + +export function extractDocToken(url) { + for (const [pattern, kind] of URL_PATTERNS) { + const match = pattern.exec(String(url)); + if (match) return { token: match[1], kind }; + } + throw new Error(`cannot read a document token out of the URL: ${url}`); +} + +/** Which tool a URL maps to, and with which arguments. */ +export function toolForUrl(url) { + const { token, kind } = extractDocToken(url); + if (kind === "wiki") return { tool: "get_wiki_node", arguments: { token }, kind }; + if (kind === "docx" || kind === "doc") return { tool: "get_doc_content", arguments: { doc_token: token }, kind }; + if (kind === "sheet") return { tool: "get_spreadsheet_content", arguments: { spreadsheet_token: token }, kind }; + throw new Error(`no MCP tool reads a ${kind} document`); +} + +/** + * The default transport: one `npx -y feishu-mcp --stdio` process per call, JSON-RPC + * over stdin, credentials in the environment only. + */ +export function spawnMcpTransport({ command = MCP_COMMAND, timeoutMs = 30_000, spawnImpl = spawn } = {}) { + return { + async call(tool, args, { config, env = process.env } = {}) { + if (!ALLOWED_TOOLS.includes(tool)) throw new Error(`tool ${tool} is not on the allowlist`); + const payload = JSON.stringify({ jsonrpc: "2.0", method: "tools/call", params: { name: tool, arguments: args }, id: 1 }); + const childEnv = { + ...env, + FEISHU_APP_ID: config.app_id ?? "", + FEISHU_APP_SECRET: config.app_secret ?? "", + ...(config.mode === "user" && config.user_token ? { FEISHU_USER_ACCESS_TOKEN: config.user_token } : {}), + }; + return await new Promise((settle, fail) => { + const child = spawnImpl(command[0], command.slice(1), { env: childEnv, stdio: ["pipe", "pipe", "pipe"] }); + let stdout = ""; + let stderr = ""; + const timer = setTimeout(() => { + child.kill?.(); + fail(new Error(`feishu-mcp did not answer within ${timeoutMs}ms`)); + }, timeoutMs); + child.stdout.on("data", (chunk) => (stdout += chunk)); + child.stderr.on("data", (chunk) => (stderr += chunk)); + child.on("error", (error) => { + clearTimeout(timer); + fail(new Error(error.code === "ENOENT" ? "npx was not found; install Node.js, or npm install -g feishu-mcp" : error.message)); + }); + child.on("close", (code) => { + clearTimeout(timer); + if (code !== 0) { + fail(new Error(`feishu-mcp exited ${code}: ${stderr.slice(0, 200) || "no stderr"}`)); + return; + } + try { + settle(JSON.parse(stdout.slice(stdout.indexOf("{")))); + } catch (error) { + fail(new Error(`feishu-mcp returned a non-JSON answer: ${error.message}`)); + } + }); + child.stdin.end(payload); + }); + }, + }; +} + +/** + * MCP tool results wrap their payload: `{result: [{type:"text", text:"…"}]}` or a + * plain string, and an error comes back as `{error: …}`. Both shapes are read the + * same way the Python client read them. + */ +export function readToolResult(result) { + if (result && typeof result === "object" && "error" in result && result.error) { + return { error: typeof result.error === "string" ? result.error : JSON.stringify(result.error) }; + } + const payload = result && typeof result === "object" && "result" in result ? result.result : result; + if (typeof payload === "string") return { text: payload }; + if (Array.isArray(payload)) { + const parts = payload + .map((item) => (item && typeof item === "object" && item.type === "text" ? item.text : null)) + .filter((text) => typeof text === "string"); + if (parts.length > 0) return { text: parts.join("\n") }; + return { value: payload }; + } + if (payload && typeof payload === "object") return { value: payload }; + return { text: String(payload ?? "") }; +} + +/** Message arrays arrive as text (a JSON string) or already parsed. */ +export function asMessages({ text, value }) { + if (Array.isArray(value)) return value; + if (typeof text === "string") { + try { + const parsed = JSON.parse(text); + if (Array.isArray(parsed)) return parsed; + if (parsed && Array.isArray(parsed.items)) return parsed.items; + } catch { + return null; + } + } + return null; +} + +/** A document whose paragraphs are the lines of a fetched text (docs, sheets, wiki). */ +function documentFromText({ text, name, source, method, kind, credentialed, credentialSource, origin }) { + const file = new SourceFile({ path: name, name, raw: new Uint8Array(Buffer.from(text, "utf8")) }); + const spans = []; + const pattern = /[^\n]+/g; + let match; + while ((match = pattern.exec(text)) !== null) { + spans.push({ text: match[0], charStart: match.index, charEnd: match.index + match[0].length, kind: "paragraph" }); + } + return buildDocument({ + parser: "feishu-mcp", + format: kind, + kind: "doc", + method, + credentialed, + credential_source: credentialSource, + credential_file: CONFIG_FILE, + source, + origin: origin ?? name, + files: [file], + records: recordsFromCharSpans(file, spans), + }); +} + +/** + * Run one MCP read and record it. + * + * @param {{transport: object, config: object, tool: string, arguments: object, target: string, + * root: string, person: string, family?: string, source?: string, now?: string, + * env?: object, label?: string}} input + */ +export async function collectViaMcp(input) { + const { + transport, + config, + tool, + arguments: toolArgs, + target, + root, + person, + family = "colleague", + source = CHANNEL, + now = new Date().toISOString(), + env = process.env, + } = input; + + if (!ALLOWED_TOOLS.includes(tool)) { + return { ok: false, exitCode: 2, receipt: { command: "collect", channel: CHANNEL, mode: MODE, ok: false, warnings: [], errors: [`tool ${tool} is not on the allowlist`], unavailable: [{ channel: CHANNEL, reason: `unknown MCP tool ${tool}`, remediation: [`allowed: ${ALLOWED_TOOLS.join(", ")}`] }] } }; + } + + const warnings = []; + const secrets = [config.app_secret, config.user_token].filter(Boolean); + let result; + try { + result = await transport.call(tool, toolArgs, { config, env }); + } catch (error) { + return { + ok: false, + exitCode: 1, + receipt: { + command: "collect", + channel: CHANNEL, + mode: MODE, + ok: false, + person: person ?? null, + target: target ?? null, + credential_file: CONFIG_FILE, + warnings, + errors: [redact(error.message, secrets)], + unavailable: [ + { + channel: CHANNEL, + reason: redact(`MCP call failed: ${error.message}`, secrets), + remediation: ["npm install -g feishu-mcp", "check the app scopes: docs:doc:readonly, wiki:wiki:readonly, im:message:readonly"], + }, + ], + }, + }; + } + + const read = readToolResult(result); + if (read.error) { + return { + ok: false, + exitCode: 1, + receipt: { + command: "collect", + channel: CHANNEL, + mode: MODE, + ok: false, + person: person ?? null, + target: target ?? null, + credential_file: CONFIG_FILE, + warnings, + errors: [redact(read.error, secrets)], + unavailable: [{ channel: CHANNEL, reason: redact(`MCP returned an error: ${read.error}`, secrets), remediation: ["check the token scopes and the target id"] }], + }, + }; + } + + const messages = asMessages(read); + const label = input.label ?? `${tool}-${String(target ?? "result").replace(/[^A-Za-z0-9_-]+/g, "-").slice(0, 40)}`; + const rawText = typeof read.text === "string" ? read.text : JSON.stringify(read.value ?? messages ?? "", null, 2); + + const store = new KnowledgeStore(join(resolve(root), "skills", family, person)); + const ledger = loadLedger(store); + + let document; + if (messages) { + const file = new SourceFile({ path: `${label}.json`, name: `${label}.json`, raw: new Uint8Array(Buffer.from(JSON.stringify(messages), "utf8")) }); + document = parseFeishu(file, { + source, + method: `mcp-${tool}`, + credentialed: true, + credential_source: input.credentialSource ?? "feishu_config.json", + credential_file: CONFIG_FILE, + }); + warnings.push(...(document.warnings ?? []).map((warning) => warning.message ?? warning)); + } else { + document = documentFromText({ + text: rawText, + name: `${label}.txt`, + source, + method: `mcp-${tool}`, + kind: tool === "get_wiki_node" ? "wiki" : tool === "get_spreadsheet_content" ? "sheet" : "docx", + credentialed: true, + credentialSource: input.credentialSource ?? "feishu_config.json", + origin: target ?? label, + }); + } + + const recorded = recordDocument(store, ledger, { ...document, fetched_at: now }, { fetched_at: now }); + saveLedger(store, ledger); + const raw = recorded.written.files[0]; + const text = recorded.written.text; + const outputs = [ + ...(raw ? [{ path: raw.relativePath, sha256: raw.sha256, bytes: raw.bytes, kind: "raw" }] : []), + ...(text ? [{ path: text.relativePath, sha256: text.sha256, bytes: text.bytes, kind: "text" }] : []), + ]; + + const receipt = { + command: "collect", + channel: CHANNEL, + mode: MODE, + ok: true, + person, + target: target ?? null, + tool, + credential_file: CONFIG_FILE, + messages: messages ? messages.length : 0, + anchors: { total: recorded.entry?.anchor_count ?? 0, cited: 0 }, + outputs, + warnings, + unavailable: [], + }; + return { ok: true, exitCode: 0, receipt: secrets.reduce((value, secret) => redact(JSON.stringify(value), [secret]) && value, receipt) }; +} + +/** Messages from `get_chat_messages` when the tool returns raw API items. */ +export function normaliseMcpMessages(items) { + return (items ?? []).map((item) => ({ + message_id: item.message_id ?? item.id ?? null, + msg_type: item.msg_type ?? (item.content ? "text" : "unknown"), + create_time: item.create_time ?? item.timestamp ?? null, + sender: item.sender ?? { id: item.sender_id ?? null }, + body: item.body ?? { content: item.content ?? "" }, + })); +} + +/* ------------------------------------------------------------------ CLI */ + +/** + * `distilly collect feishu --mode mcp …` + * + * Argument parsing lives here rather than in the command layer so the two Feishu + * routes stay independently runnable (`collect feishu` = open API, this one = MCP). + */ +export function parseMcpArgs(argv) { + const options = { person: null, baseDir: process.cwd(), url: null, chatId: null, target: null, limit: 500, json: false, label: null }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--json") options.json = true; + else if (arg === "--mode") { + const value = argv[index + 1]; + if (value !== "mcp") return { error: `feishu-mcp handles --mode mcp only (got ${value ?? "nothing"})` }; + index += 1; + } else if (["--person", "--base-dir", "--url", "--chat-id", "--target", "--limit", "--label"].includes(arg)) { + const value = argv[index + 1]; + if (value === undefined) return { error: `${arg} requires a value` }; + const key = { "--person": "person", "--base-dir": "baseDir", "--url": "url", "--chat-id": "chatId", "--target": "target", "--limit": "limit", "--label": "label" }[arg]; + options[key] = key === "limit" ? Number(value) : value; + index += 1; + } else if (arg.startsWith("--")) return { error: `unknown option: ${arg}` }; + else if (!options.chatId) options.chatId = arg; + else return { error: `unexpected argument: ${arg}` }; + } + if (!options.url && !options.chatId) return { error: "collect feishu --mode mcp needs --url or --chat-id " }; + if (!options.person) return { error: "--person is required" }; + return { options }; +} + +export async function runCollectCli(argv, io = {}) { + const out = typeof io.stdout === "function" ? io.stdout : (line) => process.stdout.write(`${line}\n`); + const err = typeof io.stderr === "function" ? io.stderr : (line) => process.stderr.write(`${line}\n`); + const parsed = parseMcpArgs(argv); + if (parsed.error) { + return { + ok: false, + exitCode: 2, + receipt: { + command: "collect", + channel: CHANNEL, + mode: MODE, + ok: false, + person: null, + warnings: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + errors: [parsed.error], + unavailable: [{ channel: CHANNEL, reason: parsed.error, remediation: ["distilly collect feishu --help"] }], + }, + }; + } + const { options } = parsed; + + let config; + try { + const credential = loadCredential({ env: process.env }); + config = { ...credential.values, credential_source: credential.source }; + } catch (error) { + err(`collect feishu --mode mcp: ${error.message}`); + return { + ok: false, + exitCode: 1, + receipt: { + command: "collect", + channel: CHANNEL, + mode: MODE, + ok: false, + person: options.person, + credential_file: CONFIG_FILE, + warnings: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + errors: [error.message], + unavailable: [{ channel: CHANNEL, reason: error.message, remediation: ["write ~/.distilly/feishu_config.json with app_id and app_secret"] }], + }, + }; + } + + let tool; + let toolArgs; + let target; + try { + if (options.url) { + const mapped = toolForUrl(options.url); + tool = mapped.tool; + toolArgs = mapped.arguments; + target = options.url; + } else { + tool = "get_chat_messages"; + toolArgs = { chat_id: options.chatId, page_size: Math.min(Number(options.limit) || 50, 50) }; + target = options.chatId; + } + } catch (error) { + err(`collect feishu --mode mcp: ${error.message}`); + return { + ok: false, + exitCode: 2, + receipt: { + command: "collect", + channel: CHANNEL, + mode: MODE, + ok: false, + person: options.person, + warnings: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + errors: [error.message], + unavailable: [{ channel: CHANNEL, reason: error.message, remediation: ["pass a Feishu doc/wiki/sheet URL, or --chat-id"] }], + }, + }; + } + + const transport = io.transport ?? spawnMcpTransport(); + const result = await collectViaMcp({ + transport, + config, + tool, + arguments: toolArgs, + target, + label: options.label ?? undefined, + root: options.baseDir, + person: options.person, + credentialSource: config.credential_source, + }); + + if (!options.json) { + if (result.ok) { + out(`collect feishu (mcp): ${result.receipt.messages} message(s) via ${tool}, ${result.receipt.anchors.total} anchor(s)`); + for (const output of result.receipt.outputs) out(` ${output.kind}: ${output.path}`); + } + for (const warning of result.receipt.warnings ?? []) err(` warning: ${warning}`); + for (const failure of result.receipt.errors ?? []) err(` error: ${failure}`); + for (const item of result.receipt.unavailable ?? []) err(` unavailable: ${item.reason}`); + } + return result; +} diff --git a/src/collect/gmail.mjs b/src/collect/gmail.mjs new file mode 100644 index 00000000..035111e7 --- /dev/null +++ b/src/collect/gmail.mjs @@ -0,0 +1,390 @@ +/** + * gmail.mjs — Gmail messages, read with an OAuth refresh token. + * + * The read path is deliberately thin: list message ids, fetch each one as **raw + * MIME**, and hand the bytes to the email parser that already exists + * (`src/parse/email.mjs`). Headers, charsets, HTML fallbacks and attachment + * warnings are therefore identical to a locally harvested `.eml` — one + * implementation, two doors. + * + * Read-only by construction: the allowlist has one POST (the OAuth token + * exchange) and GETs under `/gmail/v1/users/`. Nothing here can send, label, + * trash or modify a message. + */ + +import { join, resolve } from "node:path"; + +import { KnowledgeStore } from "../knowledge/store.mjs"; +import { loadLedger, recordDocument, saveLedger } from "../knowledge/ledger.mjs"; +import { SourceFile } from "../parse/common.mjs"; +import { parseEmail } from "../parse/email.mjs"; +import { + CollectFailure, + DEFAULT_MAX_RETRIES, + clearCheckpoint, + loadCredential, + readCheckpoint, + redact, + requestJson, + scrub, + writeCheckpoint, +} from "./kit.mjs"; + +export const CHANNEL = "gmail"; +export const CONFIG_FILE = "gmail_config.json"; +export const ENV_KEYS = [ + "DISTILLY_GMAIL_CLIENT_ID", + "GMAIL_CLIENT_ID", + "DISTILLY_GMAIL_CLIENT_SECRET", + "GMAIL_CLIENT_SECRET", + "DISTILLY_GMAIL_REFRESH_TOKEN", + "GMAIL_REFRESH_TOKEN", +]; +export const DEFAULT_BASE_URL = "https://gmail.googleapis.com/gmail/v1"; +export const DEFAULT_TOKEN_URL = "https://oauth2.googleapis.com/token"; +export const DEFAULT_PAGE_SIZE = 100; +export const DEFAULT_MAX_MESSAGES = 200; + +export const ALLOWED_CALLS = [ + { method: "POST", path: "/token" }, + { method: "GET", path: "/gmail/v1/users/" }, +]; + +const REMEDIATION = [ + "create an OAuth client (desktop) in Google Cloud, enable the Gmail API, and add scope gmail.readonly", + "run the consent flow once and keep the refresh token", + `write ~/.distilly/${CONFIG_FILE} as {"client_id": "…", "client_secret": "…", "refresh_token": "…"}`, +]; + +/** Gmail returns base64url; `Buffer` needs the padding restored. */ +export function decodeRawMessage(raw) { + const text = String(raw ?? "").replace(/-/g, "+").replace(/_/g, "/"); + const padded = text + "=".repeat((4 - (text.length % 4)) % 4); + return Buffer.from(padded, "base64"); +} + +/** + * @param {{fetch?: Function, env?: object, root?: string, person: string, family?: string, + * query?: string, limit?: number, maxMessages?: number, maxRetries?: number, + * sleep?: Function, now?: string, baseUrl?: string, tokenUrl?: string}} options + */ +export async function collect(options = {}) { + const { + fetch: fetchImpl = globalThis.fetch, + env = process.env, + root = process.cwd(), + person, + family = "colleague", + query = "", + limit = DEFAULT_PAGE_SIZE, + maxMessages = DEFAULT_MAX_MESSAGES, + maxRetries = DEFAULT_MAX_RETRIES, + sleep, + now = new Date().toISOString(), + baseUrl = env?.DISTILLY_GMAIL_BASE_URL || DEFAULT_BASE_URL, + tokenUrl = env?.DISTILLY_GMAIL_TOKEN_URL || DEFAULT_TOKEN_URL, + } = options; + + const outputs = []; + const warnings = []; + const retries = []; + const base = { + command: "collect", + channel: CHANNEL, + mode: "api", + ok: false, + inputs: [], + outputs, + warnings, + unavailable: [], + credential_file: CONFIG_FILE, + retries, + }; + let secrets = []; + const fail = (failure) => ({ + ok: false, + exitCode: failure.exitCode ?? 1, + receipt: scrub( + { + ...base, + ok: false, + person: person ?? null, + query: query || null, + errors: [redact(failure.message, secrets)], + unavailable: [{ channel: CHANNEL, reason: redact(`${failure.reason}: ${failure.message}`, secrets), remediation: failure.remediation ?? [] }], + }, + secrets, + ), + }); + + try { + if (!person) throw new CollectFailure("missing-person", "collect gmail needs --person ", { remediation: ["pass --person"] }); + const credential = loadCredential({ + env, + configFile: CONFIG_FILE, + envKeys: ENV_KEYS, + fields: ["client_id", "client_secret", "refresh_token"], + }); + secrets = [credential.values.client_secret, credential.values.refresh_token]; + base.credential_source = credential.source; + + const onRetry = (info) => { + retries.push(info); + warnings.push(`retry ${info.attempt} after ${info.reason} (waited ${info.delayMs}ms)`); + }; + + const exchange = async () => { + const response = await requestJson({ + fetchImpl, + url: tokenUrl, + method: "POST", + headers: { "content-type": "application/x-www-form-urlencoded" }, + body: new URLSearchParams({ + grant_type: "refresh_token", + refresh_token: credential.values.refresh_token, + client_id: credential.values.client_id, + client_secret: credential.values.client_secret, + }).toString(), + maxRetries, + ...(sleep ? { sleep } : {}), + secrets, + allowlist: ALLOWED_CALLS, + channel: CHANNEL, + }); + const token = response.json?.access_token ?? null; + if (!token) throw new CollectFailure("auth-failed", "Google rejected the refresh token", { remediation: REMEDIATION }); + secrets = [credential.values.client_secret, credential.values.refresh_token, token]; + return token; + }; + + let token = await exchange(); + const headers = () => ({ authorization: `Bearer ${token}`, accept: "application/json" }); + + // A token can expire mid-run; one re-exchange keeps a long collection going. + const get = async (url) => { + try { + return await requestJson({ + fetchImpl, + url, + method: "GET", + headers: headers(), + maxRetries, + ...(sleep ? { sleep } : {}), + secrets, + allowlist: ALLOWED_CALLS, + channel: CHANNEL, + onRetry, + }); + } catch (error) { + if (!(error instanceof CollectFailure) || error.reason !== "unauthorized") throw error; + warnings.push("the access token expired mid-run; the refresh token was exchanged again"); + token = await exchange(); + return await requestJson({ + fetchImpl, + url, + method: "GET", + headers: headers(), + maxRetries, + ...(sleep ? { sleep } : {}), + secrets, + allowlist: ALLOWED_CALLS, + channel: CHANNEL, + onRetry, + }); + } + }; + + const store = new KnowledgeStore(join(resolve(root), "skills", family, person)); + const ledger = loadLedger(store); + const checkpoint = options.resume === false ? null : readCheckpoint({ env, root, channel: CHANNEL, target: query || "inbox" }); + const seen = new Set(checkpoint?.seen ?? []); + let pageToken = checkpoint?.page_token ?? null; + if (seen.size > 0) warnings.push(`resuming: ${seen.size} message(s) already fetched`); + + const pageSize = Math.min(Math.max(1, Number(limit) || DEFAULT_PAGE_SIZE), 500); + let pages = 0; + let fetched = 0; + let anchors = 0; + const textEntries = []; + + for (;;) { + if (fetched >= maxMessages) { + warnings.push(`stopped after --max-messages ${maxMessages}`); + break; + } + const params = new URLSearchParams({ maxResults: String(Math.min(pageSize, maxMessages - fetched)) }); + if (query) params.set("q", query); + if (pageToken) params.set("pageToken", pageToken); + const listing = await get(`${baseUrl}/users/me/messages?${params.toString()}`); + pages += 1; + const ids = (listing.json?.messages ?? []).map((entry) => entry.id).filter(Boolean); + pageToken = listing.json?.nextPageToken ?? null; + + for (const id of ids) { + if (seen.has(id)) continue; + const message = await get(`${baseUrl}/users/me/messages/${encodeURIComponent(id)}?format=raw`); + const bytes = decodeRawMessage(message.json?.raw); + if (bytes.length === 0) { + warnings.push(`message ${id} carried no raw MIME payload and was skipped`); + continue; + } + const file = new SourceFile({ path: `${id}.eml`, name: `${id}.eml`, raw: new Uint8Array(bytes) }); + let document; + try { + document = parseEmail(file, { + source: CHANNEL, + method: "api-oauth-refresh", + credentialed: true, + credential_source: credential.source, + credential_file: CONFIG_FILE, + }); + } catch (error) { + warnings.push(`message ${id}: ${redact(error.message, secrets)}; the raw bytes are stored, the text is not`); + document = null; + } + const recorded = recordDocument( + store, + ledger, + document + ? { ...document, fetched_at: now } + : { + parser: "gmail", + format: "eml", + kind: "email", + method: "api-oauth-refresh", + credentialed: true, + credential_source: credential.source, + credential_file: CONFIG_FILE, + source: CHANNEL, + origin: `gmail:${id}`, + files: [{ path: `${id}.eml`, name: `${id}.eml`, raw: new Uint8Array(bytes) }], + content: "", + segments: [], + entries: [], + warnings: ["the message could not be parsed; only the raw bytes are recorded"], + fetched_at: now, + }, + { fetched_at: now }, + ); + seen.add(id); + fetched += 1; + const raw = recorded.written.files[0]; + if (raw) outputs.push({ path: raw.relativePath, sha256: raw.sha256, bytes: raw.bytes, kind: "raw" }); + if (recorded.written.text) { + outputs.push({ + path: recorded.written.text.relativePath, + sha256: recorded.written.text.sha256, + bytes: recorded.written.text.bytes, + kind: "text", + }); + textEntries.push(recorded.entry?.id ?? null); + anchors += recorded.entry?.anchor_count ?? 0; + } + } + + writeCheckpoint( + { env, root, channel: CHANNEL, target: query || "inbox" }, + { channel: CHANNEL, target: query || "inbox", page_token: pageToken, pages, fetched, seen: [...seen].slice(-500), updated_at: now }, + ); + if (!pageToken || ids.length === 0) break; + } + + saveLedger(store, ledger); + clearCheckpoint({ env, root, channel: CHANNEL, target: query || "inbox" }); + + return { + ok: true, + exitCode: 0, + receipt: scrub( + { + ...base, + ok: true, + person, + query: query || null, + pages, + messages: fetched, + text_entries: textEntries.filter(Boolean), + anchors: { total: anchors, cited: 0 }, + ledger: { path: store.ledgerPath, total: ledger.length }, + unavailable: [], + }, + secrets, + ), + }; + } catch (error) { + if (!(error instanceof CollectFailure)) { + throw new CollectFailure("unexpected", redact(error?.message ?? String(error), secrets), { + remediation: ["rerun with --json and report the receipt"], + }); + } + return fail(error); + } +} + +/* ------------------------------------------------------------------ CLI */ + +export function parseGmailArgs(argv) { + const options = { person: null, baseDir: process.cwd(), query: "", limit: DEFAULT_PAGE_SIZE, maxMessages: DEFAULT_MAX_MESSAGES, json: false }; + const takesValue = { "--person": "person", "--base-dir": "baseDir", "--query": "query", "--limit": "limit", "--max-messages": "maxMessages" }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--json") options.json = true; + else if (takesValue[arg]) { + const value = argv[index + 1]; + if (value === undefined) return { error: `${arg} requires a value` }; + options[takesValue[arg]] = /limit|maxMessages/.test(takesValue[arg]) ? Number(value) : value; + index += 1; + } else if (arg.startsWith("--")) return { error: `unknown option: ${arg}` }; + else if (options.query === "") options.query = arg; + else return { error: `unexpected argument: ${arg}` }; + } + if (!options.person) return { error: "--person is required" }; + return { options }; +} + +export async function runCollectCli(argv, io = {}) { + const out = typeof io.stdout === "function" ? io.stdout : (line) => process.stdout.write(`${line}\n`); + const err = typeof io.stderr === "function" ? io.stderr : (line) => process.stderr.write(`${line}\n`); + const parsed = parseGmailArgs(argv); + if (parsed.error) { + return { + ok: false, + exitCode: 2, + receipt: { + command: "collect", + channel: CHANNEL, + mode: "api", + ok: false, + person: null, + warnings: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + unavailable: [{ channel: CHANNEL, reason: parsed.error, remediation: ["distilly collect gmail --help"] }], + error: { code: "collect/usage", message: parsed.error, remedy: "distilly collect gmail --help" }, + }, + }; + } + const { options } = parsed; + const result = await collect({ + root: options.baseDir, + person: options.person, + query: options.query, + limit: options.limit, + maxMessages: options.maxMessages, + ...(io.fetch ? { fetch: io.fetch } : {}), + }); + if (!options.json) { + if (result.ok) { + out(`collect gmail: ${result.receipt.messages} message(s) in ${result.receipt.pages} page(s), ${result.receipt.anchors.total} anchor(s)`); + for (const output of result.receipt.outputs) out(` ${output.kind}: ${output.path}`); + } + for (const warning of result.receipt.warnings ?? []) err(` warning: ${warning}`); + for (const failure of result.receipt.unavailable ?? []) { + err(` unavailable: ${failure.reason}`); + for (const step of failure.remediation ?? []) err(` ${step}`); + } + } + return result; +} + +export { REMEDIATION as GMAIL_REMEDIATION }; diff --git a/src/collect/kit.mjs b/src/collect/kit.mjs new file mode 100644 index 00000000..6d9e7b9e --- /dev/null +++ b/src/collect/kit.mjs @@ -0,0 +1,309 @@ +/** + * kit.mjs — the plumbing every credentialed collector needs, in one place. + * + * `src/collect/feishu.mjs` grew this by hand first (credential lookup, redaction, + * retry/backoff, a read-only allowlist, checkpointed cursors, verbatim raw storage, + * ledger registration). The channels added later share it from here instead of + * copying it, so one fix — a leak in a log line, a retry that ignores + * `Retry-After` — lands everywhere at once. + * + * Nothing here talks to a specific API: the caller supplies the base URL, the + * allowlist and the parser. What this module guarantees is the discipline in + * `docs/v2/CONTRACT.md`: credentials never reach stdout/stderr/a receipt (only the + * config *file name* does), every request is checked against a read-only + * allowlist before it is sent, and an interrupted run resumes from its cursor. + */ + +import { createHash } from "node:crypto"; +import { existsSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { dirname, join, resolve } from "node:path"; + +/** A failure a collector reports as a receipt, never as a stack trace. */ +export class CollectFailure extends Error { + constructor(reason, message, options = {}) { + super(message); + this.name = "CollectFailure"; + this.reason = reason; + this.exitCode = options.exitCode ?? 1; + this.remediation = options.remediation ?? []; + } +} + +export const DEFAULT_MAX_RETRIES = 4; +export const DEFAULT_MAX_BACKOFF_MS = 60_000; + +export const sha256Hex = (bytes) => createHash("sha256").update(Buffer.from(bytes)).digest("hex"); + +export const slug = (text, fallback = "target") => { + const cleaned = String(text ?? "") + .trim() + .replace(/[^\p{L}\p{N}._-]+/gu, "-") + .replace(/^-+|-+$/g, "") + .slice(0, 80); + return cleaned === "" ? fallback : cleaned; +}; + +/** Replace every occurrence of a secret with a fixed marker. */ +export function redact(text, secrets = []) { + let output = String(text ?? ""); + for (const secret of secrets) { + if (typeof secret !== "string" || secret.length < 4) continue; + output = output.split(secret).join("***"); + } + return output; +} + +/** Redact recursively, so no receipt field can carry a credential value. */ +export function scrub(value, secrets = []) { + if (typeof value === "string") return redact(value, secrets); + if (Array.isArray(value)) return value.map((item) => scrub(item, secrets)); + if (value && typeof value === "object") { + return Object.fromEntries(Object.entries(value).map(([key, item]) => [key, scrub(item, secrets)])); + } + return value; +} + +export function distillyHome(env = process.env) { + return env.DISTILLY_HOME && env.DISTILLY_HOME.trim() !== "" + ? env.DISTILLY_HOME + : join(homedir(), ".distilly"); +} + +/** `~/.distilly/`, with the pre-rename location as a fallback. */ +export function credentialPaths(configFile, env = process.env) { + return { + primary: join(distillyHome(env), configFile), + legacy: join(env.HOME ?? homedir(), ".colleague-skill", configFile), + }; +} + +/** + * Read a channel credential from the environment or the config file. + * + * @param {{configFile: string, envKeys: string[], fields: string[], env?: object, readFile?: Function}} input + * @returns {{values: object, source: string, configFile: string, path: string|null}} + */ +export function loadCredential(input) { + const { configFile, envKeys, fields, env = process.env, readFile = readFileSync } = input; + const pick = (name) => { + const value = env?.[name]; + return typeof value === "string" && value.trim() !== "" ? value.trim() : null; + }; + const fromEnv = {}; + let envComplete = true; + for (const [index, field] of fields.entries()) { + const value = pick(envKeys[index]); + if (value === null) envComplete = false; + else fromEnv[field] = value; + } + if (envComplete) return { values: fromEnv, source: "env", configFile, path: null }; + + const { primary, legacy } = credentialPaths(configFile, env); + for (const [path, source] of [ + [primary, "config"], + [legacy, "legacy-config"], + ]) { + if (!existsSync(path)) continue; + let parsed; + try { + parsed = JSON.parse(readFile(path, "utf8")); + } catch (error) { + throw new CollectFailure("bad-credential-file", `${configFile} is not valid JSON (${redact(error.message)}); rewrite it as {${fields.map((f) => `"${f}": "…"`).join(", ")}}`, { + remediation: [`write ~/.distilly/${configFile}`], + }); + } + const values = {}; + let complete = true; + for (const field of fields) { + const camel = field.replace(/_([a-z])/g, (_, letter) => letter.toUpperCase()); + const value = parsed[field] ?? parsed[camel] ?? null; + if (typeof value !== "string" || value.trim() === "") complete = false; + else values[field] = value.trim(); + } + if (!complete) { + throw new CollectFailure("incomplete-credential", `${configFile} is missing ${fields.join(" / ")}`, { + remediation: [`write ~/.distilly/${configFile} with ${fields.join(", ")}`], + }); + } + return { values, source, configFile, path }; + } + + throw new CollectFailure("no-credential", `no credential for this channel: set ${envKeys.join(" / ")} or write ~/.distilly/${configFile}`, { + remediation: [`write ~/.distilly/${configFile}`, `or export ${envKeys[0]}`], + }); +} + +/** + * Refuse anything outside the channel's read-only allowlist, before it is sent. + * + * @param {string} url + * @param {string} method + * @param {Array<{method: string, path: string|RegExp}>} allowlist + * @param {string} channel + */ +export function assertReadOnly(url, method, allowlist, channel) { + const verb = String(method ?? "GET").toUpperCase(); + const path = (() => { + try { + return new URL(url).pathname; + } catch { + return String(url); + } + })(); + const allowed = allowlist.some((entry) => { + if (entry.method !== verb) return false; + return entry.path instanceof RegExp ? entry.path.test(path) : path.startsWith(entry.path); + }); + if (!allowed) { + throw new CollectFailure("write-refused", `${channel} collection is read-only: ${verb} ${path} is not on the allowlist`, { + remediation: ["this build never posts, edits or deletes anything"], + }); + } +} + +/** Seconds from a `Retry-After` header (or a body's `retry_after`). */ +export function parseRetryAfter(headerValue, nowMs = Date.now()) { + if (headerValue === undefined || headerValue === null || headerValue === "") return null; + const text = String(headerValue).trim(); + if (/^\d+(\.\d+)?$/.test(text)) return Math.min(Number(text) * 1000, 15 * 60_000); + const at = Date.parse(text); + if (Number.isNaN(at)) return null; + return Math.max(0, Math.min(at - nowMs, 15 * 60_000)); +} + +export function backoffDelay(attempt, retryAfterMs = null, options = {}) { + const maxMs = options.maxMs ?? DEFAULT_MAX_BACKOFF_MS; + if (retryAfterMs !== null) return Math.min(retryAfterMs, maxMs); + const base = options.baseMs ?? 500; + return Math.min(base * 2 ** Math.max(0, attempt - 1), maxMs); +} + +export const defaultSleep = (ms) => new Promise((settle) => setTimeout(settle, ms)); + +/** + * One request with retry, backoff and redaction. + * + * @returns {Promise<{status: number, text: string, json: any, attempts: number}>} + */ +export async function requestJson(options) { + const { + fetchImpl, + url, + method = "GET", + headers = {}, + body, + maxRetries = DEFAULT_MAX_RETRIES, + sleep = defaultSleep, + secrets = [], + allowlist = null, + channel = "collect", + onRetry = () => {}, + acceptStatus = [], + parse = "json", + } = options; + + if (allowlist) assertReadOnly(url, method, allowlist, channel); + + let attempt = 0; + for (;;) { + attempt += 1; + const init = { method, headers, ...(body === undefined ? {} : { body: typeof body === "string" ? body : JSON.stringify(body) }) }; + const response = await fetchImpl(url, init); + const status = response.status; + if (status === 429 || (status >= 500 && status !== 501)) { + if (attempt > maxRetries) { + throw new CollectFailure("retry-exhausted", `${channel} still answering HTTP ${status} after ${maxRetries} retries`, { + remediation: ["retry later; the pages already fetched are on disk"], + }); + } + const retryAfterHeader = response.headers?.get?.("retry-after") ?? null; + const text = await response.text().catch(() => ""); + let bodyRetryAfter = null; + if (text) { + try { + const parsed = JSON.parse(text); + bodyRetryAfter = parsed?.retry_after ?? parsed?.error?.retry_after ?? null; + } catch { + bodyRetryAfter = null; + } + } + const retryAfterMs = + parseRetryAfter(retryAfterHeader) ?? + (typeof bodyRetryAfter === "number" ? Math.min(bodyRetryAfter * (bodyRetryAfter < 1000 ? 1000 : 1), 15 * 60_000) : null); + const delayMs = backoffDelay(attempt, retryAfterMs); + onRetry({ attempt, status, delayMs, reason: `HTTP ${status}` }); + await sleep(delayMs); + continue; + } + const text = await response.text(); + if (status >= 400 && !acceptStatus.includes(status)) { + throw new CollectFailure(status === 401 || status === 403 ? "unauthorized" : "http-error", `${channel} HTTP ${status}: ${redact(text.slice(0, 200), secrets)}`, { + remediation: ["check the credential and its scopes"], + }); + } + let json = null; + if (parse === "json" && text.trim() !== "") { + try { + json = JSON.parse(text); + } catch { + throw new CollectFailure("invalid-json", `${channel} returned a non-JSON body`, { + remediation: ["retry later; if it persists the endpoint may have changed"], + }); + } + } + return { status, text, json, attempts: attempt }; + } +} + +/* ------------------------------------------------------------------ storage */ + +/** `/skills///knowledge` — the Skill's evidence root. */ +export function knowledgeRoot({ root = process.cwd(), person, family = "colleague" } = {}) { + return person + ? join(resolve(root), "skills", slug(family), slug(person), "knowledge") + : join(resolve(root), "knowledge"); +} + +/** Store raw bytes verbatim, creating the bucket only now. */ +export function writeRaw(knowledgeDir, bucket, name, bytes) { + const dir = join(knowledgeDir, "raw", slug(bucket)); + mkdirSync(dir, { recursive: true }); + const path = join(dir, `${slug(name, "page")}.json`); + const staging = `${path}.${process.pid}.tmp`; + const buffer = Buffer.from(bytes); + writeFileSync(staging, buffer); + renameSync(staging, path); + const readBack = readFileSync(path); + if (!readBack.equals(buffer)) throw new CollectFailure("write-verify", `raw bytes changed on disk: ${path}`, {}); + return { path, relativePath: `raw/${slug(bucket)}/${slug(name, "page")}.json`, bytes: buffer.length, sha256: sha256Hex(buffer) }; +} + +/** A JSON cursor file under `$DISTILLY_HOME/state`, so an interrupted run resumes. */ +export function statePath({ env = process.env, root = process.cwd(), channel, target }) { + const key = sha256Hex(Buffer.from(`${resolve(root)}\n${target ?? ""}`, "utf8")).slice(0, 12); + return join(distillyHome(env), "state", `${slug(channel)}-${key}.json`); +} + +export function readCheckpoint({ env = process.env, root = process.cwd(), channel, target }) { + const path = statePath({ env, root, channel, target }); + if (!existsSync(path)) return null; + try { + return JSON.parse(readFileSync(path, "utf8")); + } catch { + return null; + } +} + +export function writeCheckpoint({ env = process.env, root = process.cwd(), channel, target }, value) { + const path = statePath({ env, root, channel, target }); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, `${JSON.stringify(value, null, 2)}\n`, "utf8"); + return path; +} + +export function clearCheckpoint({ env = process.env, root = process.cwd(), channel, target }) { + const path = statePath({ env, root, channel, target }); + if (existsSync(path)) rmSync(path, { force: true }); + return path; +} diff --git a/src/collect/notion.mjs b/src/collect/notion.mjs new file mode 100644 index 00000000..69a74cd4 --- /dev/null +++ b/src/collect/notion.mjs @@ -0,0 +1,418 @@ +/** + * notion.mjs — Notion pages, read with an internal integration token. + * + * Notion's read path is unusual: the API has almost no GETs — searching and + * listing blocks are `POST`s that change nothing. The allowlist therefore names + * those two POSTs explicitly and refuses everything else, which is the same + * guarantee the GET-only channels get: there is no code path here that can create, + * edit, delete or share anything. + * + * A fetched page becomes one ledger entry: the raw API responses verbatim under + * `knowledge/raw/notion/`, and the block text as anchored paragraphs — the same + * shape a harvested document has, so `retrospect` reads it without special cases. + */ + +import { join, resolve } from "node:path"; + +import { KnowledgeStore } from "../knowledge/store.mjs"; +import { loadLedger, recordDocument, saveLedger } from "../knowledge/ledger.mjs"; +import { SourceFile, buildDocument, recordsFromCharSpans } from "../parse/common.mjs"; +import { + CollectFailure, + DEFAULT_MAX_PAGES, + DEFAULT_MAX_RETRIES, + loadCredential, + readCheckpoint, + redact, + requestJson, + scrub, + writeCheckpoint, + clearCheckpoint, + writeRaw, +} from "./kit.mjs"; + +export const CHANNEL = "notion"; +export const CONFIG_FILE = "notion_config.json"; +export const ENV_KEYS = ["DISTILLY_NOTION_TOKEN", "NOTION_TOKEN"]; +export const DEFAULT_BASE_URL = "https://api.notion.com/v1"; +export const NOTION_VERSION = "2022-06-28"; + +/** + * The only calls this module makes. `POST /v1/search` and `POST + * /v1/databases/*\/query` are documented as read operations — they take a filter + * and return rows; nothing is written. + */ +export const ALLOWED_CALLS = [ + { method: "GET", path: "/v1/blocks/" }, + { method: "GET", path: "/v1/pages/" }, + { method: "GET", path: "/v1/users/" }, + { method: "POST", path: "/v1/search" }, + { method: "POST", path: /^\/v1\/databases\/[^/]+\/query$/ }, +]; + +const REMEDIATION = [ + "create an internal integration at https://www.notion.so/my-integrations and copy its token", + "share the page with the integration (⋯ → Connections)", + `write ~/.distilly/${CONFIG_FILE} as {"integration_token": "…"}`, +]; + +/** Block types whose text is dialogue or prose; everything else is named. */ +const TEXT_BLOCKS = new Map([ + ["paragraph", null], + ["heading_1", null], + ["heading_2", null], + ["heading_3", null], + ["bulleted_list_item", null], + ["numbered_list_item", null], + ["quote", null], + ["callout", null], + ["toggle", null], + ["to_do", null], + ["code", null], + ["template", null], +]); + +/** Flatten `rich_text[]` (and a code block's `caption`) into plain text. */ +export function blockText(block) { + const type = block?.type; + if (!type || !TEXT_BLOCKS.has(type)) return null; + const payload = block[type] ?? {}; + const rich = Array.isArray(payload.rich_text) ? payload.rich_text : []; + const text = rich.map((part) => part?.plain_text ?? part?.text?.content ?? "").join(""); + const caption = Array.isArray(payload.caption) ? payload.caption.map((part) => part?.plain_text ?? "").join("") : ""; + return `${text}${caption}`.trim(); +} + +/** How a page's title is rendered: the first rich-text of its title property. */ +export function pageTitle(page) { + const properties = page?.properties ?? {}; + for (const value of Object.values(properties)) { + if (value?.type === "title" && Array.isArray(value.title)) { + const text = value.title.map((part) => part?.plain_text ?? "").join("").trim(); + if (text !== "") return text; + } + } + return page?.id ?? "untitled"; +} + +/** + * Walk a page's children (breadth-first, depth-limited) and collect paragraphs. + * + * @returns {{paragraphs: string[], blocks: number, skipped: object, warnings: string[]}} + */ +export async function readBlocks(options) { + const { fetchImpl, baseUrl, headers, rootBlockId, maxDepth = 2, maxBlocks = 500, maxRetries, sleep, secrets, onRetry, depth = 0 } = options; + const paragraphs = []; + const warnings = []; + const skipped = {}; + let blocks = 0; + let cursor = null; + + do { + const params = new URLSearchParams({ page_size: "100" }); + if (cursor) params.set("start_cursor", cursor); + const url = `${baseUrl}/blocks/${encodeURIComponent(rootBlockId)}/children?${params.toString()}`; + const response = await requestJson({ + fetchImpl, + url, + method: "GET", + headers, + maxRetries, + ...(sleep ? { sleep } : {}), + secrets, + allowlist: ALLOWED_CALLS, + channel: CHANNEL, + onRetry, + }); + for (const block of response.json?.results ?? []) { + blocks += 1; + if (blocks > maxBlocks) break; + const text = blockText(block); + if (text !== null) { + if (text !== "") paragraphs.push(text); + continue; + } + if (block?.type === "child_page" && depth < maxDepth) { + const child = await readBlocks({ ...options, rootBlockId: block.id, depth: depth + 1, maxBlocks: maxBlocks - blocks }); + paragraphs.push(...child.paragraphs); + Object.assign(skipped, child.skipped); + warnings.push(...child.warnings); + blocks += child.blocks; + continue; + } + const type = block?.type ?? "unknown"; + skipped[type] = (skipped[type] ?? 0) + 1; + } + cursor = response.json?.has_more ? response.json?.next_cursor ?? null : null; + } while (cursor); + + if (Object.keys(skipped).length > 0) { + warnings.push( + `blocks without dialogue text were not anchored: ${Object.entries(skipped) + .map(([type, count]) => `${type}×${count}`) + .join(", ")}`, + ); + } + return { paragraphs, blocks, skipped, warnings }; +} + +/** + * @param {{fetch?: Function, env?: object, root?: string, person: string, family?: string, + * pageId: string, maxDepth?: number, maxRetries?: number, sleep?: Function, + * now?: string, baseUrl?: string}} options + */ +export async function collect(options = {}) { + const { + fetch: fetchImpl = globalThis.fetch, + env = process.env, + root = process.cwd(), + person, + family = "colleague", + pageId, + maxDepth = 2, + maxRetries = DEFAULT_MAX_RETRIES, + sleep, + now = new Date().toISOString(), + baseUrl = env?.DISTILLY_NOTION_BASE_URL || DEFAULT_BASE_URL, + } = options; + + const outputs = []; + const warnings = []; + const retries = []; + const base = { + command: "collect", + channel: CHANNEL, + mode: "api", + ok: false, + inputs: [], + outputs, + warnings, + unavailable: [], + credential_file: CONFIG_FILE, + retries, + }; + let secrets = []; + const fail = (failure) => ({ + ok: false, + exitCode: failure.exitCode ?? 1, + receipt: scrub( + { + ...base, + ok: false, + person: person ?? null, + page_id: pageId ?? null, + errors: [redact(failure.message, secrets)], + unavailable: [{ channel: CHANNEL, reason: redact(`${failure.reason}: ${failure.message}`, secrets), remediation: failure.remediation ?? [] }], + }, + secrets, + ), + }); + + try { + if (!person) throw new CollectFailure("missing-person", "collect notion needs --person ", { remediation: ["pass --person"] }); + if (!pageId) { + throw new CollectFailure("missing-target", "collect notion needs --page-id ", { + remediation: ["copy the page id from its URL (32 hex characters before the ?)"], + }); + } + const credential = loadCredential({ env, configFile: CONFIG_FILE, envKeys: ENV_KEYS, fields: ["integration_token"] }); + secrets = [credential.values.integration_token]; + base.credential_source = credential.source; + + const headers = { + authorization: `Bearer ${credential.values.integration_token}`, + "notion-version": NOTION_VERSION, + accept: "application/json", + }; + const id = normalisePageId(pageId); + const onRetry = (info) => { + retries.push(info); + warnings.push(`retry ${info.attempt} after ${info.reason} (waited ${info.delayMs}ms)`); + }; + + const pageResponse = await requestJson({ + fetchImpl, + url: `${baseUrl}/pages/${id}`, + method: "GET", + headers, + maxRetries, + ...(sleep ? { sleep } : {}), + secrets, + allowlist: ALLOWED_CALLS, + channel: CHANNEL, + onRetry, + }); + const title = pageTitle(pageResponse.json); + + const knowledgeDir = join(resolve(root), "skills", family, person, "knowledge"); + const stored = writeRaw(knowledgeDir, CHANNEL, `${id}-page`, pageResponse.text); + outputs.push({ path: stored.relativePath, sha256: stored.sha256, bytes: stored.bytes, kind: "raw" }); + + const read = await readBlocks({ + fetchImpl, + baseUrl, + headers, + rootBlockId: id, + maxDepth, + maxRetries, + sleep, + secrets, + onRetry, + }); + warnings.push(...read.warnings); + + const text = [`# ${title}`, "", ...read.paragraphs].join("\n"); + if (read.paragraphs.length === 0) { + throw new CollectFailure("no-text", `page ${id} has no paragraph blocks this client can read`, { + remediation: ["share the page with the integration, or check that it has text blocks"], + }); + } + + const file = new SourceFile({ path: `${id}.md`, name: `${id}.md`, raw: new Uint8Array(Buffer.from(text, "utf8")) }); + const spans = []; + const pattern = /[^\n]+/g; + let match; + while ((match = pattern.exec(text)) !== null) { + spans.push({ text: match[0], charStart: match.index, charEnd: match.index + match[0].length, kind: "paragraph" }); + } + const document = buildDocument({ + parser: "notion", + format: "blocks", + kind: "doc", + method: "api-integration-token", + credentialed: true, + credential_source: credential.source, + credential_file: CONFIG_FILE, + source: CHANNEL, + origin: `https://www.notion.so/${id}`, + files: [file], + records: recordsFromCharSpans(file, spans), + meta: { title, blocks: read.blocks, page_id: id }, + }); + + const store = new KnowledgeStore(join(resolve(root), "skills", family, person)); + const ledger = loadLedger(store); + const recorded = recordDocument(store, ledger, { ...document, fetched_at: now }, { fetched_at: now }); + saveLedger(store, ledger); + if (recorded.written.text) { + outputs.push({ + path: recorded.written.text.relativePath, + sha256: recorded.written.text.sha256, + bytes: recorded.written.text.bytes, + kind: "text", + }); + } + + return { + ok: true, + exitCode: 0, + receipt: scrub( + { + ...base, + ok: true, + person, + page_id: id, + title, + blocks: read.blocks, + paragraphs: read.paragraphs.length, + text_entries: [recorded.entry?.id ?? null].filter(Boolean), + anchors: { total: recorded.entry?.anchor_count ?? 0, cited: 0 }, + ledger: { path: store.ledgerPath, total: ledger.length }, + unavailable: [], + }, + secrets, + ), + }; + } catch (error) { + if (!(error instanceof CollectFailure)) { + throw new CollectFailure("unexpected", redact(error?.message ?? String(error), secrets), { + remediation: ["rerun with --json and report the receipt"], + }); + } + return fail(error); + } +} + +/** `https://www.notion.so/Title-<32 hex>?v=…` → `<32 hex>` (dashed ids accepted). */ +export function normalisePageId(input) { + const text = String(input ?? "").trim(); + const dashed = text.match(/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/i); + if (dashed) return dashed[0]; + const bare = text.match(/[0-9a-f]{32}/i); + if (bare) { + const value = bare[0]; + return `${value.slice(0, 8)}-${value.slice(8, 12)}-${value.slice(12, 16)}-${value.slice(16, 20)}-${value.slice(20)}`; + } + if (/^[0-9a-f-]{36}$/i.test(text)) return text; + throw new CollectFailure("bad-target", `cannot read a Notion page id out of "${text.slice(0, 60)}"`, { + remediation: ["pass the page URL or its 32-character id"], + }); +} + +/* ------------------------------------------------------------------ CLI */ + +export function parseNotionArgs(argv) { + const options = { person: null, baseDir: process.cwd(), pageId: null, maxDepth: 2, json: false }; + const takesValue = { "--person": "person", "--base-dir": "baseDir", "--page-id": "pageId", "--url": "pageId", "--max-depth": "maxDepth" }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--json") options.json = true; + else if (takesValue[arg]) { + const value = argv[index + 1]; + if (value === undefined) return { error: `${arg} requires a value` }; + options[takesValue[arg]] = takesValue[arg] === "maxDepth" ? Number(value) : value; + index += 1; + } else if (arg.startsWith("--")) return { error: `unknown option: ${arg}` }; + else if (!options.pageId) options.pageId = arg; + else return { error: `unexpected argument: ${arg}` }; + } + if (!options.person) return { error: "--person is required" }; + if (!options.pageId) return { error: "collect notion needs --page-id " }; + return { options }; +} + +export async function runCollectCli(argv, io = {}) { + const out = typeof io.stdout === "function" ? io.stdout : (line) => process.stdout.write(`${line}\n`); + const err = typeof io.stderr === "function" ? io.stderr : (line) => process.stderr.write(`${line}\n`); + const parsed = parseNotionArgs(argv); + if (parsed.error) { + return { + ok: false, + exitCode: 2, + receipt: { + command: "collect", + channel: CHANNEL, + mode: "api", + ok: false, + person: null, + warnings: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + unavailable: [{ channel: CHANNEL, reason: parsed.error, remediation: ["distilly collect notion --help"] }], + error: { code: "collect/usage", message: parsed.error, remedy: "distilly collect notion --help" }, + }, + }; + } + const { options } = parsed; + const result = await collect({ + root: options.baseDir, + person: options.person, + pageId: options.pageId, + maxDepth: options.maxDepth, + ...(io.fetch ? { fetch: io.fetch } : {}), + }); + if (!options.json) { + if (result.ok) { + out(`collect notion: "${result.receipt.title}" — ${result.receipt.paragraphs} paragraph(s), ${result.receipt.anchors.total} anchor(s)`); + for (const output of result.receipt.outputs) out(` ${output.kind}: ${output.path}`); + } + for (const warning of result.receipt.warnings ?? []) err(` warning: ${warning}`); + for (const failure of result.receipt.unavailable ?? []) { + err(` unavailable: ${failure.reason}`); + for (const step of failure.remediation ?? []) err(` ${step}`); + } + } + return result; +} + +export { REMEDIATION as NOTION_REMEDIATION }; diff --git a/src/collect/reddit.mjs b/src/collect/reddit.mjs new file mode 100644 index 00000000..7d37a240 --- /dev/null +++ b/src/collect/reddit.mjs @@ -0,0 +1,405 @@ +/** + * reddit.mjs — Reddit comments, read with an OAuth client credential. + * + * The only mutation this module performs is the OAuth token exchange — a POST that + * changes nothing a user can see, exactly like Feishu's tenant-token call. Every + * other request is a GET against a small allowlist, so there is no path here that + * can post, vote, edit or delete. + * + * A listing page is stored verbatim under `knowledge/raw/reddit/` and the comments + * it contains become anchored paragraphs (author + UTC time + body), which is the + * shape the derivation reads. `[deleted]` and `[removed]` bodies are skipped by + * name, never anchored as if someone had said them. + */ + +import { join, resolve } from "node:path"; + +import { KnowledgeStore } from "../knowledge/store.mjs"; +import { loadLedger, recordDocument, saveLedger } from "../knowledge/ledger.mjs"; +import { SourceFile, buildDocument, recordsFromCharSpans } from "../parse/common.mjs"; +import { + CollectFailure, + DEFAULT_MAX_PAGES, + DEFAULT_MAX_RETRIES, + clearCheckpoint, + loadCredential, + readCheckpoint, + redact, + requestJson, + scrub, + writeCheckpoint, + writeRaw, +} from "./kit.mjs"; + +export const CHANNEL = "reddit"; +export const CONFIG_FILE = "reddit_config.json"; +export const ENV_KEYS = ["DISTILLY_REDDIT_CLIENT_ID", "REDDIT_CLIENT_ID", "DISTILLY_REDDIT_CLIENT_SECRET", "REDDIT_CLIENT_SECRET"]; +export const DEFAULT_BASE_URL = "https://oauth.reddit.com"; +export const DEFAULT_TOKEN_URL = "https://www.reddit.com/api/v1/access_token"; +export const DEFAULT_PAGE_SIZE = 100; +export const USER_AGENT = "distilly/1.0 (read-only collector)"; + +/** The token exchange, plus GETs. Reddit's read API never needs another verb. */ +export const ALLOWED_CALLS = [ + { method: "POST", path: "/api/v1/access_token" }, + { method: "GET", path: "/api/v1/me" }, + { method: "GET", path: "/r/" }, + { method: "GET", path: "/user/" }, + { method: "GET", path: "/comments/" }, +]; + +const REMEDIATION = [ + "create a script app at https://www.reddit.com/prefs/apps (type: script) and copy the id and secret", + `write ~/.distilly/${CONFIG_FILE} as {"client_id": "…", "client_secret": "…"}`, +]; + +/** `[deleted]` / `[removed]` are placeholders, not something a person said. */ +export const isPlaceholder = (body) => { + const text = String(body ?? "").trim(); + return text === "" || text === "[deleted]" || text === "[removed]"; +}; + +/** A listing's comment children, flattened, with placeholders removed. */ +export function commentsFromListing(json) { + const children = json?.data?.children ?? []; + const comments = []; + const skipped = {}; + const walk = (nodes, depth = 0) => { + for (const node of nodes) { + if (node?.kind === "more") { + skipped.more = (skipped.more ?? 0) + 1; + continue; + } + const data = node?.data ?? node; + if (!data || typeof data !== "object") continue; + if (typeof data.body === "string") { + if (isPlaceholder(data.body)) skipped.placeholder = (skipped.placeholder ?? 0) + 1; + else { + comments.push({ + id: data.id ?? null, + author: data.author ?? "unknown", + body: data.body, + created_utc: typeof data.created_utc === "number" ? data.created_utc : null, + permalink: data.permalink ?? null, + depth, + score: typeof data.score === "number" ? data.score : null, + }); + } + } + const replies = data.replies?.data?.children; + if (Array.isArray(replies)) walk(replies, depth + 1); + } + }; + walk(children); + return { comments, skipped, after: json?.data?.after ?? null }; +} + +/** One anchor per comment: ` ：`. */ +export function redditDocument({ comments, file, options }) { + const text = comments.map((comment) => { + const at = comment.created_utc === null ? "" : new Date(comment.created_utc * 1000).toISOString(); + return `${at} ${comment.author}：${comment.body}`; + }).join("\n"); + const spans = []; + let offset = 0; + for (const [index, comment] of comments.entries()) { + const line = text.split("\n")[index]; + spans.push({ text: line, charStart: offset, charEnd: offset + line.length, kind: "comment", label: comment.id ?? undefined }); + offset += line.length + 1; + } + void file; + return { content: text, spans, options }; +} + +/** + * @param {{fetch?: Function, env?: object, root?: string, person: string, family?: string, + * target: string, kind?: "subreddit"|"user", limit?: number, maxPages?: number, + * maxRetries?: number, sleep?: Function, now?: string, baseUrl?: string, tokenUrl?: string}} options + */ +export async function collect(options = {}) { + const { + fetch: fetchImpl = globalThis.fetch, + env = process.env, + root = process.cwd(), + person, + family = "colleague", + target, + limit = DEFAULT_PAGE_SIZE, + maxPages = DEFAULT_MAX_PAGES, + maxRetries = DEFAULT_MAX_RETRIES, + sleep, + now = new Date().toISOString(), + baseUrl = env?.DISTILLY_REDDIT_BASE_URL || DEFAULT_BASE_URL, + tokenUrl = env?.DISTILLY_REDDIT_TOKEN_URL || DEFAULT_TOKEN_URL, + } = options; + + const outputs = []; + const warnings = []; + const retries = []; + const base = { + command: "collect", + channel: CHANNEL, + mode: "api", + ok: false, + inputs: [], + outputs, + warnings, + unavailable: [], + credential_file: CONFIG_FILE, + retries, + }; + let secrets = []; + const fail = (failure) => ({ + ok: false, + exitCode: failure.exitCode ?? 1, + receipt: scrub( + { + ...base, + ok: false, + person: person ?? null, + target: target ?? null, + errors: [redact(failure.message, secrets)], + unavailable: [{ channel: CHANNEL, reason: redact(`${failure.reason}: ${failure.message}`, secrets), remediation: failure.remediation ?? [] }], + }, + secrets, + ), + }); + + try { + if (!person) throw new CollectFailure("missing-person", "collect reddit needs --person ", { remediation: ["pass --person"] }); + if (!target) { + throw new CollectFailure("missing-target", "collect reddit needs --target ", { + remediation: ["pass --target programming, or --target someuser with --kind user"], + }); + } + const kind = options.kind ?? "subreddit"; + if (!["subreddit", "user"].includes(kind)) { + throw new CollectFailure("bad-kind", `--kind must be subreddit or user (got ${kind})`, { remediation: ["pass --kind subreddit|user"] }); + } + + const credential = loadCredential({ + env, + configFile: CONFIG_FILE, + envKeys: ENV_KEYS, + fields: ["client_id", "client_secret"], + }); + secrets = [credential.values.client_secret]; + base.credential_source = credential.source; + + const basic = Buffer.from(`${credential.values.client_id}:${credential.values.client_secret}`).toString("base64"); + const tokenResponse = await requestJson({ + fetchImpl, + url: tokenUrl, + method: "POST", + headers: { authorization: `Basic ${basic}`, "content-type": "application/x-www-form-urlencoded", "user-agent": USER_AGENT }, + body: "grant_type=client_credentials", + maxRetries, + ...(sleep ? { sleep } : {}), + secrets, + allowlist: ALLOWED_CALLS, + channel: CHANNEL, + }); + const token = tokenResponse.json?.access_token ?? null; + if (!token) { + throw new CollectFailure("auth-failed", "Reddit rejected the client credential", { remediation: REMEDIATION }); + } + secrets = [...secrets, token]; + + const knowledgeDir = join(resolve(root), "skills", family, person, "knowledge"); + const store = new KnowledgeStore(join(resolve(root), "skills", family, person)); + const ledger = loadLedger(store); + const checkpoint = options.resume === false ? null : readCheckpoint({ env, root, channel: CHANNEL, target: `${kind}:${target}` }); + let cursor = checkpoint?.cursor ?? null; + if (cursor) warnings.push(`resuming from checkpoint (${checkpoint.pages ?? 0} page(s) done)`); + + const pageSize = Math.min(Math.max(1, Number(limit) || DEFAULT_PAGE_SIZE), 100); + const headers = { authorization: `Bearer ${token}`, "user-agent": USER_AGENT, accept: "application/json" }; + const path = kind === "subreddit" ? `/r/${encodeURIComponent(target)}/comments` : `/user/${encodeURIComponent(target)}/comments`; + + let pages = 0; + let items = 0; + let comments = 0; + let anchors = 0; + const textEntries = []; + + for (;;) { + if (pages >= maxPages) { + warnings.push(`stopped after --max-pages ${maxPages}; rerun to continue from the cursor`); + break; + } + const params = new URLSearchParams({ limit: String(pageSize), raw_json: "1" }); + if (cursor) params.set("after", cursor); + const response = await requestJson({ + fetchImpl, + url: `${baseUrl}${path}?${params.toString()}`, + method: "GET", + headers, + maxRetries, + ...(sleep ? { sleep } : {}), + secrets, + allowlist: ALLOWED_CALLS, + channel: CHANNEL, + onRetry: (info) => { + retries.push(info); + warnings.push(`retry ${info.attempt} after ${info.reason} (waited ${info.delayMs}ms)`); + }, + }); + pages += 1; + const listing = commentsFromListing(response.json); + items += response.json?.data?.children?.length ?? 0; + comments += listing.comments.length; + for (const [type, count] of Object.entries(listing.skipped)) warnings.push(`${count} ${type} entr(ies) were skipped`); + + const name = `${kind}-${target}-p${String(pages).padStart(3, "0")}`; + const stored = writeRaw(knowledgeDir, CHANNEL, name, response.text); + outputs.push({ path: stored.relativePath, sha256: stored.sha256, bytes: stored.bytes, kind: "raw" }); + + if (listing.comments.length > 0) { + const text = listing.comments + .map((comment) => `${comment.created_utc === null ? "" : new Date(comment.created_utc * 1000).toISOString()} ${comment.author}：${comment.body}`) + .join("\n"); + const file = new SourceFile({ path: `${name}.txt`, name: `${name}.txt`, raw: new Uint8Array(Buffer.from(text, "utf8")) }); + const spans = []; + let offset = 0; + for (const line of text.split("\n")) { + spans.push({ text: line, charStart: offset, charEnd: offset + line.length, kind: "comment" }); + offset += line.length + 1; + } + const document = buildDocument({ + parser: "reddit", + format: "listing", + kind: "chat", + method: "api-oauth-client", + credentialed: true, + credential_source: credential.source, + credential_file: CONFIG_FILE, + source: CHANNEL, + origin: `https://www.reddit.com${path}`, + files: [file], + records: recordsFromCharSpans(file, spans), + meta: { target, kind, comments: listing.comments.length }, + }); + const recorded = recordDocument(store, ledger, { ...document, fetched_at: now }, { fetched_at: now }); + if (recorded.written.text) { + outputs.push({ + path: recorded.written.text.relativePath, + sha256: recorded.written.text.sha256, + bytes: recorded.written.text.bytes, + kind: "text", + }); + textEntries.push(recorded.entry?.id ?? null); + anchors += recorded.entry?.anchor_count ?? 0; + } + } + + cursor = listing.after; + if (!cursor) break; + if (options.resume !== false) { + writeCheckpoint({ env, root, channel: CHANNEL, target: `${kind}:${target}` }, { channel: CHANNEL, target, cursor, pages, items, updated_at: now }); + } + } + + saveLedger(store, ledger); + clearCheckpoint({ env, root, channel: CHANNEL, target: `${kind}:${target}` }); + + return { + ok: true, + exitCode: 0, + receipt: scrub( + { + ...base, + ok: true, + person, + target, + kind, + pages, + items, + comments, + cursor, + text_entries: textEntries.filter(Boolean), + anchors: { total: anchors, cited: 0 }, + ledger: { path: store.ledgerPath, total: ledger.length }, + unavailable: [], + }, + secrets, + ), + }; + } catch (error) { + if (!(error instanceof CollectFailure)) { + throw new CollectFailure("unexpected", redact(error?.message ?? String(error), secrets), { + remediation: ["rerun with --json and report the receipt"], + }); + } + return fail(error); + } +} + +/* ------------------------------------------------------------------ CLI */ + +export function parseRedditArgs(argv) { + const options = { person: null, baseDir: process.cwd(), target: null, kind: "subreddit", limit: DEFAULT_PAGE_SIZE, maxPages: DEFAULT_MAX_PAGES, json: false }; + const takesValue = { "--person": "person", "--base-dir": "baseDir", "--target": "target", "--kind": "kind", "--limit": "limit", "--max-pages": "maxPages" }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--json") options.json = true; + else if (takesValue[arg]) { + const value = argv[index + 1]; + if (value === undefined) return { error: `${arg} requires a value` }; + options[takesValue[arg]] = /limit|maxPages/.test(takesValue[arg]) ? Number(value) : value; + index += 1; + } else if (arg.startsWith("--")) return { error: `unknown option: ${arg}` }; + else if (!options.target) options.target = arg; + else return { error: `unexpected argument: ${arg}` }; + } + if (!options.person) return { error: "--person is required" }; + if (!options.target) return { error: "collect reddit needs --target " }; + return { options }; +} + +export async function runCollectCli(argv, io = {}) { + const out = typeof io.stdout === "function" ? io.stdout : (line) => process.stdout.write(`${line}\n`); + const err = typeof io.stderr === "function" ? io.stderr : (line) => process.stderr.write(`${line}\n`); + const parsed = parseRedditArgs(argv); + if (parsed.error) { + return { + ok: false, + exitCode: 2, + receipt: { + command: "collect", + channel: CHANNEL, + mode: "api", + ok: false, + person: null, + warnings: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + unavailable: [{ channel: CHANNEL, reason: parsed.error, remediation: ["distilly collect reddit --help"] }], + error: { code: "collect/usage", message: parsed.error, remedy: "distilly collect reddit --help" }, + }, + }; + } + const { options } = parsed; + const result = await collect({ + root: options.baseDir, + person: options.person, + target: options.target, + kind: options.kind, + limit: options.limit, + maxPages: options.maxPages, + ...(io.fetch ? { fetch: io.fetch } : {}), + }); + if (!options.json) { + if (result.ok) { + out(`collect reddit: ${result.receipt.comments} comment(s) in ${result.receipt.pages} page(s), ${result.receipt.anchors.total} anchor(s)`); + for (const output of result.receipt.outputs) out(` ${output.kind}: ${output.path}`); + } + for (const warning of result.receipt.warnings ?? []) err(` warning: ${warning}`); + for (const failure of result.receipt.unavailable ?? []) { + err(` unavailable: ${failure.reason}`); + for (const step of failure.remediation ?? []) err(` ${step}`); + } + } + return result; +} + +export { REMEDIATION as REDDIT_REMEDIATION }; diff --git a/src/commands/credentialed.mjs b/src/commands/credentialed.mjs new file mode 100644 index 00000000..d2e499f9 --- /dev/null +++ b/src/commands/credentialed.mjs @@ -0,0 +1,135 @@ +/** + * Credentialed commands — `collect`, `consent`, `transcribe` (from ds/07). + * + * Those modules own their flags, receipts and exit codes; this adapter only + * registers them with the command registry and guarantees CONTRACT §3: in + * `--json` mode stdout carries exactly one JSON object, whatever the module + * prints while it works (hence the stdout capture). + */ + +import { register } from "./index.mjs"; + +const CHANNELS = { + feishu: () => import("../collect/feishu.mjs"), + slack: () => import("../collect/slack.mjs"), + dingtalk: () => import("../collect/dingtalk.mjs"), + x: () => import("../collect/x.mjs"), +}; + +/** Run `fn` with stdout/stderr captured, so the dispatcher owns the output. */ +async function capture(fn) { + const originalOut = process.stdout.write.bind(process.stdout); + const originalErr = process.stderr.write.bind(process.stderr); + let out = ""; + let err = ""; + process.stdout.write = (chunk) => { + out += String(chunk); + return true; + }; + process.stderr.write = (chunk) => { + err += String(chunk); + return true; + }; + try { + const result = await fn(); + return { result, out, err }; + } finally { + process.stdout.write = originalOut; + process.stderr.write = originalErr; + } +} + +function parseReceiptFrom(text) { + const start = text.indexOf("{"); + if (start === -1) return null; + try { + return JSON.parse(text.slice(start)); + } catch { + return null; + } +} + +function forward(text, write) { + for (const line of text.split("\n")) if (line.trim()) write(line); +} + +async function runModule(load, argv, { json, reporter }) { + const module = await load(); + const entry = module.runCollectCli ?? module.runConsentCli ?? module.runTranscribeCli; + if (typeof entry !== "function") throw new Error("module exposes no CLI entry point"); + const { result, out, err } = await capture(() => entry(argv, {})); + if (!json) { + forward(out, (line) => reporter.line(line)); + forward(err, (line) => reporter.warn(line)); + } + const receipt = result?.receipt ?? parseReceiptFrom(out) ?? undefined; + const exitCode = result?.exitCode ?? (result?.ok === false ? 1 : 0); + return { receipt, exitCode }; +} + +const collectHelp = { + zh: [ + "用法 / Usage:", + " distilly collect [options] [--json]", + " distilly collect x --mode browser --consent # computer use，需显式同意", + "", + "需要凭据的渠道由脚本负责鉴权/分页/限流/续采；缺 key 或缺同意 → 响亮失败并给补救步骤，", + "回执与日志里只出现配置文件名，永不出现 key 值。", + ].join("\n"), + en: [ + "Distilly collect — credentialed channels (feishu, slack, dingtalk, x).", + "Browser collection requires `--consent `; without it the command exits 2.", + "Receipts and logs name the credential file, never its contents.", + ].join("\n"), +}; + +register("collect", { + summary: "需要凭据的渠道采集 / credentialed collection", + usage: "distilly collect [options] [--json]", + ...collectHelp, + async run({ argv, json, reporter }) { + const [channel, ...rest] = argv; + if (!channel || channel === "--help" || channel === "help") { + reporter.line(`Usage: distilly collect <${Object.keys(CHANNELS).join("|")}> [options] [--json]`); + return { receipt: undefined, exitCode: channel ? 0 : 2 }; + } + const load = CHANNELS[channel]; + if (!load) { + return { + receipt: { + command: "collect", + person: null, + ok: false, + inputs: [], + outputs: [], + anchors: { total: 0, cited: 0 }, + warnings: [], + unavailable: [], + error: { code: "collect/unknown-channel", message: `unsupported channel: ${channel}`, remedy: `known channels: ${Object.keys(CHANNELS).join(", ")}` }, + }, + exitCode: 2, + }; + } + return runModule(load, rest, { json, reporter }); + }, +}); + +register("consent", { + summary: "computer-use 同意管理 / consent tokens", + usage: "distilly consent [options] [--json]", + zh: ["用法 / Usage:", " distilly consent [--json]", "", "同意令牌存在 ~/.distilly/consent.json；没有令牌时 collect --mode browser 退出码 2。"].join("\n"), + en: ["Distilly consent — grant, list, verify, revoke or prune the computer-use consent tokens kept in ~/.distilly/consent.json."].join("\n"), + async run({ argv, json, reporter }) { + return runModule(() => import("../consent.mjs"), argv, { json, reporter }); + }, +}); + +register("transcribe", { + summary: "可选的转写后端 / optional transcription backend", + usage: "distilly transcribe [options] [--json]", + zh: ["用法 / Usage:", " distilly transcribe [--json]", "", "没有可用后端时明确报 unavailable，不静默降级；产物带 provenance{method,producer,confidence}。"].join("\n"), + en: ["Distilly transcribe — optional backend (OpenAI-compatible HTTP or a host capability). Reports `unavailable` instead of degrading silently, and records provenance with every transcript."].join("\n"), + async run({ argv, json, reporter }) { + return runModule(() => import("../optional/transcribe.mjs"), argv, { json, reporter }); + }, +}); diff --git a/src/commands/harvest.mjs b/src/commands/harvest.mjs new file mode 100644 index 00000000..076933e0 --- /dev/null +++ b/src/commands/harvest.mjs @@ -0,0 +1,202 @@ +/** + * `distilly harvest` — zero-credential intake: files in, `knowledge/` out. + * + * This is the glue ds/02 could not finish before its session ended: it walks the + * given paths, picks a parser by extension, and records every document through + * the shared ledger (raw bytes + anchored text + append-only index). + * + * Honest by construction: an unsupported or unreadable file is reported in + * `warnings` and in the receipt, never skipped silently, and a re-import of + * identical bytes appends nothing (the ledger dedupes by sha256). + */ + +import { readFileSync, readdirSync, statSync } from "node:fs"; +import { basename, extname, join, relative, resolve } from "node:path"; + +import { register } from "./index.mjs"; +import { KnowledgeStore, sha256Hex } from "../knowledge/store.mjs"; +import { loadLedger, recordDocument, saveLedger, ledgerStats } from "../knowledge/ledger.mjs"; +import { SourceFile, buildDocument, recordsFromCharSpans } from "../parse/common.mjs"; +import { parseSubtitle } from "../parse/subtitle.mjs"; +import { parseChat } from "../parse/chat.mjs"; + +const SUBTITLE_EXTENSIONS = new Set([".srt", ".vtt"]); +const TEXT_EXTENSIONS = new Set([".md", ".txt", ".text"]); +const CHAT_EXTENSIONS = new Set([".json"]); + +const help = { + zh: [ + "用法 / Usage:", + " distilly harvest --person [--base-dir ] [--source