PR-16 · Discord 与 Notion 两个采集渠道

.github/workflows/ci.yml

@@ -1,10 +1,13 @@
 name: CI
 
 on:
+  # The per-feature PRs are a stack whose bases are other `ds/*` branches; with the
+  # trigger list limited to the integration branches, none of them could ever run CI.
   push:
-    branches: [dot-skill-test, dot-skill, main]
+    branches: [dot-skill-test, dot-skill, main, 'ds/**']
   pull_request:
-    branches: [dot-skill-test, dot-skill, main]
+    branches: [dot-skill-test, dot-skill, main, 'ds/**']
+  workflow_dispatch:
 
 jobs:
   test:

docs/evidence/pr-16-discord-notion.md

@@ -0,0 +1,55 @@
+# PR-16 · Discord 与 Notion 两个采集渠道
+
+- 分支：`ds/16-discord-notion`（本地，已合并进 `dot-skill-test`）
+- 交付：`src/collect/kit.mjs`、`src/collect/discord.mjs`、`src/collect/notion.mjs`、
+  `src/parse/chat.mjs`（透传 method / 凭据来源 / 认 `author.username`）、
+  `src/commands/credentialed.mjs`、`tests/collect-discord-notion.test.mjs`
+
+## 1. 为什么
+
+`CONTRACT.md` §1 的采集命令行写着 8 个渠道，此前只实现 4 个（飞书/Slack/钉钉/X）。
+未实现的两个渠道此前回 `collect/planned-channel` 并写明所需凭据——诚实，但仍是缺口。
+这一轮把 Discord（bot token）与 Notion（内部集成 token）补上，`PENDING_CHANNELS`
+从 4 条缩到 2 条（reddit / gmail）。
+
+## 2. 三条纪律，与已有渠道一致
+
+| 纪律 | 实现 |
+| --- | --- |
+| 只读 | Discord 读 API 全是 GET，白名单就只列 GET；Notion 的读路径几乎全是 POST（search / 数据库查询），白名单**显式**列出这两个 POST 并拒绝其它一切。越权动词在建立连接之前被拒 |
+| 凭据不进回执 | 只允许出现配置文件名（`discord_config.json` / `notion_config.json`）；`redact`/`scrub` 覆盖回执的每个字段（含嵌套数组） |
+| 逐字入库 + 可派生 | 原始页 verbatim 落 `knowledge/raw/<channel>/`，同一次采集把它变成带锚点正文（Discord 交给**已有的导出解析器**，Notion 走段落化），一页/一页一个账本条目 |
+
+分页与限流：Discord 用 `before` 游标（一页不足 page size 即结束），429 读 body 里的
+`retry_after`（**秒**）退避——测试断言睡的是 1500ms；Notion 用 `next_cursor`。
+两处都支持断点续采（游标写在 `$DISTILLY_HOME/state/`）。
+
+## 3. 顺带修好的两处解析器透传
+
+- `parseChat` 现在尊重调用方的 `method` 与 `credentialed/credential_source/credential_file`：
+  导出来的是 `user-export`，API 采集来的是 `api-bot-token`——否则采集来的页面会在账本里
+  冒充用户导出。
+- 说话人取名认 REST 的 `author.username`（导出用 `name`，API 用 `username`）：
+  同一批消息无论是下载的还是采集的，人名一致（`林工` 而不是 `u1`）。
+- 顺带的一致性变化：chat 的正文文件名现在随 harvest 的 source 标签（默认取输入目录名），
+  与其它格式一致；`--source chat` 可恢复旧名。
+
+## 4. 怎么验
+
+```bash
+node --test tests/collect-discord-notion.test.mjs   # 10 条
+node --test tests/*.test.mjs                        # 364 pass / 0 fail
+node bin/distilly.mjs doctor                        # Planned channels: collect reddit, collect gmail
+```
+
+测试覆盖：两页分页（游标串接 + 逐字字节 + 锚点正文 + 一页一条目）、429 退避毫秒数、
+缺凭据零写入、写动词被拒、Notion 页面 → 段落（跳过块点名）、Notion 白名单（两个 POST 放行、
+`POST /v1/pages` 与 `PATCH` 拒绝）、页面 id 从 URL 解析与坏 id 响亮失败、空页面不算空文档、
+kit 的脱敏与凭据查找（环境变量 / 配置文件 / legacy 路径 / 非法 JSON）。
+
+## 5. 已知缺口
+
+- **reddit / gmail 仍未实现**（各自需要什么写在 `PENDING_CHANNELS`，`doctor` 会列出）。
+- Discord 的线程/回复关系、Notion 的数据库行查询（白名单已放行 `POST
+  /v1/databases/{id}/query`，但 CLI 尚未暴露 `--database-id`）留待后续。
+- Discord 附件、Notion 的图片与嵌入块不anchored，按类型计入 warnings。

package.json

@@ -20,7 +20,7 @@
     "CITATION.cff"
   ],
   "scripts": {
-    "test": "node --test \"tests/*.test.mjs\"",
+    "test": "node --test tests/*.test.mjs",
     "prepack": "node bin/distilly.mjs --check-package"
   },
   "keywords": [

scripts/acceptance.mjs

@@ -183,10 +183,17 @@ try {
   // 4. visual-check（八项）
   const vcScript = path.join(root, 'scripts/visual-check.mjs');
   const vc = spawnSync('node', [vcScript, htmlPath, '--out', evidenceDir], { cwd: workdir, encoding: 'utf8' });
+  // The hint that playwright is missing goes to **stderr**, so a row that quoted
+  // only stdout went red with an empty reason — a gate nobody can act on.
+  const vcDetail =
+    (vc.stdout ?? '').trim().split('\n').slice(-3).join(' / ') ||
+    (vc.stderr ?? '').trim().split('\n').slice(-2).join(' / ');
   if (/Cannot find module|ENOENT/.test(vc.stderr ?? '')) {
     record('visual-check 可用', false, 'scripts/visual-check.mjs 尚不存在（ds/03-render 的产出）');
+  } else if (/DISTILLY_PLAYWRIGHT_ROOT/.test(vc.stderr ?? '')) {
+    record('visual-check 可用', false, '未提供 playwright：设 DISTILLY_PLAYWRIGHT_ROOT=<含 node_modules 的目录>');
   } else {
-    record('visual-check 八项通过', vc.status === 0, (vc.stdout ?? '').trim().split('\n').slice(-3).join(' / '));
+    record('visual-check 八项通过', vc.status === 0, vcDetail);
   }
 } catch (error) {
   record('验收流程未中断', false, String(error.message).split('\n')[0]);

scripts/audit-objective.mjs

@@ -33,6 +33,22 @@ const record = (demand, ok, evidence, { gap = false } = {}) => {
 };
 
 const git = (...args) => execFileSync("git", args, { cwd: root, encoding: "utf8" }).trim();
+/**
+ * `git rev-parse --verify --quiet <ref>` → the ref's sha, or `null` when the ref
+ * does not exist.
+ *
+ * `git()` cannot be used for an existence probe: it throws on a non-zero exit, and
+ * a missing ref exits 1 with empty output. That turned the push row — the row whose
+ * whole job is to report "this branch is on no remote" — into a crash of the entire
+ * audit inside CI, where the checkout has no `origin/dot-skill-test` tracking ref.
+ */
+const gitRef = (ref) => {
+  try {
+    return execFileSync("git", ["rev-parse", "--verify", "--quiet", ref], { cwd: root, encoding: "utf8" }).trim();
+  } catch {
+    return null;
+  }
+};
 
 /** 1. Node single stack: no Python left anywhere in the tree. */
 {
@@ -219,10 +235,16 @@ const git = (...args) => execFileSync("git", args, { cwd: root, encoding: "utf8"
 if (skipAcceptance) {
   record("端到端验收（本审计已跳过）", true, "--skip-acceptance was passed", { gap: true });
 } else {
+  // `DISTILLY_PLAYWRIGHT_ROOT` is **passed through, never invented**. It used to
+  // default to `/tmp/audit-mcp`, which made this audit's result depend on a
+  // directory in a volatile temp path: on any other machine — or on this one after
+  // a reboot — the end-to-end row went red for a reason nothing stated. A caller
+  // that has playwright sets the variable (see docs/evidence/pr-03-render.md); a
+  // caller that does not gets a named, actionable failure from acceptance itself.
   const output = execFileSync(process.execPath, [join(root, "scripts", "acceptance.mjs")], {
     cwd: root,
     encoding: "utf8",
-    env: { ...process.env, DISTILLY_PLAYWRIGHT_ROOT: process.env.DISTILLY_PLAYWRIGHT_ROOT ?? "/tmp/audit-mcp" },
+    env: { ...process.env },
   });
   const summary = output.trim().split("\n").pop() ?? "";
   const match = /(\d+)\/(\d+)\s*通过/.exec(summary);
@@ -236,22 +258,31 @@ if (skipAcceptance) {
   // `upstream`. Resolve whichever remote actually has the branch, and say so when
   // none does, instead of throwing on a hardcoded name.
   const remotes = git("remote").split("\n").map((line) => line.trim()).filter(Boolean);
-  const tracking = remotes
-    .map((remote) => `${remote}/dot-skill-test`)
-    .find((ref) => git("rev-parse", "--verify", "--quiet", ref) !== "");
+  const tracking = remotes.map((remote) => `${remote}/dot-skill-test`).find((ref) => gitRef(ref) !== null);
   const ahead = tracking === undefined ? null : Number(git("rev-list", "--count", `${tracking}..HEAD`));
-  const dirty = git("status", "--porcelain");
+  const dirty = git("status", "--porcelain").split("\n").filter(Boolean);
 
   // "Pushed" is the thing the user asked for (an off-site copy). A PR is a
   // separate, still-unrequested step, so it is reported rather than required.
-  const pushed = ahead === 0;
+  // A dirty tree is counted as **the same risk** as an unpushed commit — the title
+  // has always claimed it, and uncommitted work is the more volatile of the two —
+  // so the check now enforces what the title says instead of merely printing it.
+  const pushed = ahead === 0 && dirty.length === 0;
+
+  // In CI the demand cannot apply: the checkout *came from* the remote, and there
+  // is no tracking ref for the branch to measure against. Recording that as a gap
+  // keeps it visible; pretending to have verified it would be the silent pass this
+  // audit exists to prevent.
+  const inCi = process.env.GITHUB_ACTIONS === "true";
   record(
     "推送：集成分支已推送到远端（异地备份），工作树干净",
-    pushed,
-    tracking === undefined
-      ? "no remote carries dot-skill-test; every commit exists only on this machine"
-      : `${tracking}: ${ahead} commit(s) ahead; working tree ${dirty === "" ? "clean" : "dirty"}; PR bodies staged in dst-evidence/PR-BODIES/`,
-    { gap: !pushed },
+    inCi ? true : pushed,
+    inCi
+      ? `CI checkout: ${remotes.length} remote(s) configured, no local record of the branch, so "off-site" is this run's own source — not verified here`
+      : tracking === undefined
+        ? "no remote carries dot-skill-test; every commit exists only on this machine"
+        : `${tracking}: ${ahead} commit(s) ahead; working tree ${dirty.length === 0 ? "clean" : `${dirty.length} path(s) dirty`}; PR bodies staged in dst-evidence/PR-BODIES/`,
+    { gap: inCi || !pushed },
   );
 }
 

src/collect/slack.mjs

@@ -88,9 +88,15 @@ export function distillyHome(env = process.env) {
 }
 
 export function credentialPaths(env = process.env) {
+  // `env.HOME ?? homedir()`, matching `kit.mjs`: the pre-rename
+  // `~/.colleague-skill/<file>` fallback has to move when a caller points `HOME`
+  // somewhere else. Reading the real home here ignored `env.HOME` — so an isolated
+  // run (a test, a sandboxed collect) could still pick up the credential sitting in
+  // the developer's own home, and the two channels disagreed about which file they
+  // had just read.
   return {
     primary: join(distillyHome(env), CONFIG_FILE),
-    legacy: join(homedir(), LEGACY_CONFIG_FILE),
+    legacy: join(env?.HOME ?? homedir(), LEGACY_CONFIG_FILE),
   };
 }