feat(babysitter): wake on real PR review feedback, not only verdicts

examples/babysitter/STANDALONE.md

@@ -31,8 +31,9 @@ drain from the bound lineage:
 ```
 
 The committed operator policy is `standalone-policy.json`: `botLogin` is
-`agent-relay-code[bot]` and the opt-in label is `babysit`. It contains no
-credential. With no `agentCli`, the agent runs the origin session's own CLI,
+`agent-relay-code[bot]`, the opt-in label is `babysit`, `reviewBots` lists the
+review bots whose feedback counts, and `ownAgents` lists our own agent
+accounts, whose comments never count. It contains no credential. With no `agentCli`, the agent runs the origin session's own CLI,
 so Claude and Codex sessions both work. The generated manifest therefore
 requires both harnesses explicitly; an operator must connect and declare both
 when deploying this source rather than relying on the static default branch
@@ -54,11 +55,31 @@ reported by `flows check`.
    `key=babysitter:<event>:<pr>@<head>`, which is the same for every delivery
    of the same event at the same live state.
 4. **Decline when nothing is actionable.** Closed, merged, draft or skip
-   label (`eligible`); the `babysit` label absent from *live* labels; this head
+   label (`eligible`); neither a Software Garden PR (a `relayflow/*` head in
+   the repository itself, in scope by default, drafts included, since Garden
+   drafts a PR whose checks fail) nor the `babysit` label on *live* labels
+   (removing it opts out; a label opt-in draft still declines); this head
    already reported by `botLogin` (its own comments' markers, all pages); or no
    failing check run or commit status, standing change request (all review
-   pages), or new authorised `@babysitter` directive (PR author, or
-   `OWNER`/`MEMBER`/`COLLABORATOR`, after Babysitter's last comment). The
+   pages), unanswered review feedback, or new authorised `@babysitter`
+   directive (PR author, or `OWNER`/`MEMBER`/`COLLABORATOR`, after
+   Babysitter's last comment). Review feedback is either an inline review
+   comment (a current line comment, or a file-level comment) with no later
+   reply in its thread from `botLogin` or a policy `ownAgents` account, or a
+   non-empty `COMMENTED` review body. Both must have become visible at or
+   after Babysitter's last comment: a pending review's comments date from
+   the review's submission. Like comments, feedback shares the journal
+   budget and is dropped oldest first past it. It counts only from the PR author,
+   `OWNER`/`MEMBER`/`COLLABORATOR`, or a policy `reviewBots` login; never from
+   `botLogin` or `ownAgents` (our own agents, whose "Fixed in …" replies must
+   not wake Babysitter, nor give it an `@babysitter` directive). Software Garden posts as the same app login as
+   Babysitter, so a `botLogin` comment is Babysitter's own (moves the window,
+   answers a thread) only when it carries a `<!-- babysitter:` marker. Inline comments, thread
+   replies included, wake it through `pull_request_review.submitted`: GitHub
+   files each one as its own submitted `COMMENTED` review, so the wake
+   contract and Cloud's trigger profile are unchanged. Evidence: every reply
+   sampled on flows#624/#628 has its own review with `submitted_at` equal to
+   its `created_at`. The
    signal read always fits the journal (50KB): long text is shortened and
    marked `[truncated]` (never below 200 characters), then the oldest
    comments are dropped. Failing checks and change requests are never

examples/babysitter/artifacts/babysitter-standalone.flow.ts

@@ -282,6 +282,7 @@ function agentTask(o, pr, head, changed) {
     "== What changed (live GitHub reread) ==",
     ...changed.failingChecks.map((c) => `- Failing check "${c.name}" (${c.conclusion}): ${c.summary || "no summary"}`),
     ...changed.changeRequests.map((r) => `- Changes requested by ${r.login}: ${r.body || "(no body)"}`),
+    ...changed.reviewFeedback.map((r) => r.kind === "inline" ? `- Review comment by ${r.login} on ${r.path}${r.line === void 0 ? "" : `:${r.line}`}: ${r.body}` : `- Review by ${r.login}: ${r.body}`),
     ...changed.directive ? [`- Directive from ${changed.directive.login}: ${changed.directive.body}`] : [],
     ...events ? ["", "== Origin session events (oldest first, bounded) ==", events] : [],
     "",
@@ -328,11 +329,12 @@ async function readSignals(c) {
     }
     throw new Error("Pagination exceeded; signals incomplete");
   }
-  const [runs, statuses, reviews, comments] = await Promise.all([
+  const [runs, statuses, reviews, comments, inline] = await Promise.all([
     pages(`/commits/${c.head}/check-runs?filter=latest`, "check_runs"),
     pages(`/commits/${c.head}/statuses`),
     pages(`/pulls/${c.number}/reviews`),
-    pages(`/issues/${c.number}/comments`)
+    pages(`/issues/${c.number}/comments`),
+    pages(`/pulls/${c.number}/comments`)
   ]);
   const bad = ["failure", "timed_out", "cancelled", "action_required", "startup_failure", "stale", "error"];
   const latestStatus = /* @__PURE__ */ new Map();
@@ -345,7 +347,7 @@ async function readSignals(c) {
   }
   const bot = c.botLogin.toLowerCase();
   const marker = `<!-- babysitter:report ${c.owner.toLowerCase()}/${c.repo.toLowerCase()}#${c.number}@${c.head} -->`;
-  const own = (m) => String(m.user?.login ?? "").toLowerCase() === bot;
+  const own = (m) => String(m.user?.login ?? "").toLowerCase() === bot && String(m.body ?? "").includes("<!-- babysitter:");
   let reported = false, lastOwn = -1;
   comments.forEach((m, i) => {
     if (!own(m)) return;
@@ -358,6 +360,39 @@ async function readSignals(c) {
   ].slice(0, 20);
   const requests = [...standing.values()].filter((r) => r.state === "CHANGES_REQUESTED").slice(0, 20);
   let recent = comments.slice(lastOwn + 1).slice(-50);
+  const author = c.author.toLowerCase(), bots = c.reviewBots.map((b) => b.toLowerCase());
+  const ours = [bot, ...c.ownAgents.map((a) => a.toLowerCase())];
+  const trusted = ["OWNER", "MEMBER", "COLLABORATOR"];
+  const from = (m) => {
+    const login = String(m.user?.login ?? "").toLowerCase();
+    if (!login || ours.includes(login)) return false;
+    if (login.endsWith("[bot]")) return bots.includes(login);
+    return login === author || trusted.includes(m.author_association);
+  };
+  const since = lastOwn < 0 ? "" : String(comments[lastOwn].created_at ?? "");
+  const answered = /* @__PURE__ */ new Map();
+  for (const m of inline) {
+    const login = String(m.user?.login ?? "").toLowerCase();
+    if (login === bot ? !own(m) : !ours.includes(login)) continue;
+    const thread = m.in_reply_to_id ?? m.id;
+    answered.set(thread, Math.max(answered.get(thread) ?? 0, m.id));
+  }
+  const submitted = new Map(reviews.map((r) => [r.id, String(r.submitted_at ?? "")]));
+  const visibleAt = (m) => [String(m.created_at ?? ""), submitted.get(m.pull_request_review_id) ?? ""].sort().at(-1);
+  const current = (m) => m.subject_type === "file" || typeof m.line === "number" && m.position !== null;
+  let feedback = [
+    ...inline.filter((m) => from(m) && current(m) && visibleAt(m) >= since && m.id > (answered.get(m.in_reply_to_id ?? m.id) ?? 0)).map((m) => ({
+      kind: "inline",
+      id: m.id,
+      login: m.user.login,
+      body: m.body,
+      path: m.path,
+      ...typeof m.line === "number" && m.subject_type !== "file" ? { line: m.line } : {},
+      thread: m.in_reply_to_id ?? m.id,
+      createdAt: visibleAt(m)
+    })),
+    ...reviews.filter((r) => r.state === "COMMENTED" && typeof r.body === "string" && r.body.trim() && from(r) && String(r.submitted_at ?? "") >= since).map((r) => ({ kind: "review", id: r.id, login: r.user.login, body: r.body, createdAt: r.submitted_at }))
+  ].sort((a, b) => String(a.createdAt).localeCompare(String(b.createdAt)) || a.id - b.id).slice(-100);
   const shape = (text3) => JSON.stringify({
     headSha: c.head,
     failingChecks: checks.map((r) => ({ name: cut(r.name, 200), conclusion: r.conclusion, summary: clip(r.summary, text3), url: cut(r.url, 500) })),
@@ -369,6 +404,12 @@ async function readSignals(c) {
       association: m.author_association ?? "NONE",
       body: clip(m.body, text3),
       createdAt: m.created_at
+    })),
+    reviewFeedback: feedback.map((r) => ({
+      ...r,
+      login: cut(r.login, 100),
+      body: clip(r.body, text3),
+      ...r.path ? { path: cut(r.path, 500) } : {}
     }))
   });
   let text2 = 4e3, out = shape(text2);
@@ -377,6 +418,10 @@ async function readSignals(c) {
     recent = recent.slice(1);
     out = shape(text2);
   }
+  while (Buffer.byteLength(out) > BUDGET && feedback.length > 1) {
+    feedback = feedback.slice(1);
+    out = shape(text2);
+  }
   const final = await get(`/pulls/${c.number}`);
   if (final.head?.sha !== c.head) throw new Error("Live PR head moved during signal capture");
   if (Buffer.byteLength(out) > BUDGET) throw new Error("PR signals exceed safe journal output size");
@@ -427,8 +472,8 @@ async function annotateReport(c) {
 ${(await current.json()).body}` }) });
   if (!res.ok) throw new Error(`GitHub comment PATCH: ${res.status}`);
 }
-async function readSignalsAt(f, pr, head, botLogin) {
-  const value = JSON.parse(await f.run(nodeCommand(readSignals, { owner: pr.owner, repo: pr.repo, number: pr.number, head, botLogin }), { timeout: "2m" }));
+async function readSignalsAt(f, pr, head, who) {
+  const value = JSON.parse(await f.run(nodeCommand(readSignals, { owner: pr.owner, repo: pr.repo, number: pr.number, head, ...who }), { timeout: "2m" }));
   if (value.headSha !== head) throw new Error("Signals were read for a different head");
   return value;
 }
@@ -486,6 +531,10 @@ function parsePolicy(value) {
   if (x.label !== void 0 && !text(x.label)) throw new Error("Invalid standalone Babysitter policy: label must be nonempty");
   if (x.agentCli !== void 0 && !text(x.agentCli)) throw new Error("Invalid standalone Babysitter policy: agentCli must be nonempty");
   if (x.agentModel !== void 0 && !text(x.agentModel)) throw new Error("Invalid standalone Babysitter policy: agentModel must be nonempty");
+  if (x.reviewBots !== void 0 && !(Array.isArray(x.reviewBots) && x.reviewBots.every((b) => text(b) && b.trim().toLowerCase().endsWith("[bot]"))))
+    throw new Error("Invalid standalone Babysitter policy: reviewBots must be a list of [bot] logins");
+  if (x.ownAgents !== void 0 && !(Array.isArray(x.ownAgents) && x.ownAgents.every(text)))
+    throw new Error("Invalid standalone Babysitter policy: ownAgents must be a list of logins");
   if (typeof x.agentCli === "string") {
     try {
       requiredReviewerModel(x.agentCli, x.agentModel);
@@ -496,6 +545,8 @@ function parsePolicy(value) {
   return {
     botLogin: x.botLogin.trim(),
     label: typeof x.label === "string" ? x.label.trim().toLowerCase() : "babysit",
+    reviewBots: [...new Set((x.reviewBots ?? []).map((b) => b.trim().toLowerCase()))],
+    ownAgents: [...new Set((x.ownAgents ?? []).map((a) => a.trim().toLowerCase()))],
     ...typeof x.agentCli === "string" ? { agentCli: x.agentCli.trim() } : {},
     ...typeof x.agentModel === "string" ? { agentModel: x.agentModel.trim() } : {}
   };
@@ -506,18 +557,23 @@ function boundPullRequest(value) {
   return { owner: pr.owner, repo: pr.repo, number: Number(pr.number), headSha: pr.headSha };
 }
 var reportMarker = (pr, head) => `<!-- babysitter:report ${pr.owner.toLowerCase()}/${pr.repo.toLowerCase()}#${pr.number}@${head} -->`;
+function gardenPullRequest(s, c) {
+  return typeof s.headRef === "string" && s.headRef.startsWith("relayflow/") && String(s.headRepo).toLowerCase() === `${c.owner}/${c.repo}`.toLowerCase();
+}
 function outOfScope(s, c, label) {
-  return eligible(s, c) ?? (Array.isArray(s.labels) && s.labels.some((l) => String(l).toLowerCase() === label) ? void 0 : `Live labels lack the "${label}" opt-in`);
+  const garden = gardenPullRequest(s, c);
+  return eligible(garden ? { ...s, draft: false } : s, c) ?? (garden || Array.isArray(s.labels) && s.labels.some((l) => String(l).toLowerCase() === label) ? void 0 : `Not a Software Garden PR and live labels lack the "${label}" opt-in`);
 }
-function whatChanged(s, author) {
+function whatChanged(s, author, ownAgents = []) {
   const directive = [...s.comments].reverse().find((m) => {
     const login = m.login.toLowerCase();
-    return DIRECTIVE.test(m.body) && !login.endsWith("[bot]") && (login === author.toLowerCase() || AUTHORISED.includes(m.association));
+    return DIRECTIVE.test(m.body) && !login.endsWith("[bot]") && !ownAgents.includes(login) && (login === author.toLowerCase() || AUTHORISED.includes(m.association));
   });
-  if (s.failingChecks.length === 0 && s.changeRequests.length === 0 && !directive) return void 0;
+  if (s.failingChecks.length === 0 && s.changeRequests.length === 0 && s.reviewFeedback.length === 0 && !directive) return void 0;
   return {
     failingChecks: s.failingChecks,
     changeRequests: s.changeRequests,
+    reviewFeedback: s.reviewFeedback,
     ...directive ? { directive: { login: directive.login, body: directive.body } } : {}
   };
 }
@@ -564,12 +620,17 @@ function createStandaloneBabysitter(policy, runtime = capabilities) {
       await report(`${wake.id}: ${skip}`);
       return f.done("declined");
     }
-    const signals = await readSignalsAt(f, pr, head, configured.botLogin);
+    const signals = await readSignalsAt(f, pr, head, {
+      botLogin: configured.botLogin,
+      author: String(live.author ?? ""),
+      reviewBots: configured.reviewBots,
+      ownAgents: configured.ownAgents
+    });
     if (signals.reported) {
       await report(`${wake.id}: head ${head} already reported`);
       return f.done("declined");
     }
-    const changed = whatChanged(signals, String(live.author));
+    const changed = whatChanged(signals, String(live.author), configured.ownAgents);
     if (!changed) {
       await report(`${wake.id}: nothing actionable at ${head}`);
       return f.done("declined");
@@ -621,7 +682,7 @@ function createStandaloneBabysitter(policy, runtime = capabilities) {
 }
 
 // standalone-entry.ts
-var standalone_entry_default = createStandaloneBabysitter({ "botLogin": "agent-relay-code[bot]", "label": "babysit" }, { enforcedAgentWriteScope: true });
+var standalone_entry_default = createStandaloneBabysitter({ "botLogin": "agent-relay-code[bot]", "label": "babysit", "reviewBots": ["chatgpt-codex-connector[bot]", "coderabbitai[bot]", "cubic-dev-ai[bot]", "cursor[bot]", "devin-ai-integration[bot]"], "ownAgents": ["AgentRelayBot"] }, { enforcedAgentWriteScope: true });
 export {
   standalone_entry_default as default
 };

examples/babysitter/artifacts/babysitter-standalone.manifest.json

@@ -1,10 +1,10 @@
 {
   "schemaVersion": 1,
   "artifact": "babysitter-standalone.flow.ts",
-  "bytes": 33927,
-  "sha256": "eed5e5de88d9477c3155c205310d4c4c060b8de2fcdb9ddbbe8c8c517cf22e2d",
+  "bytes": 37895,
+  "sha256": "70e27d2c653e17ccfe562e0b97f4ac4b843281bb7dcb8b6bfc6c00ddb24f15af",
   "policy": "../standalone-policy.json",
-  "policySha256": "3671bab03a1537c4cc6626009c17b1060b866e387fe57e79fdd8c580b89971b0",
+  "policySha256": "e2fd08aaef4ea922146ad1330d7b7d7161f078f9c19e1499aa272636ae6e39a2",
   "runtime": {
     "enforcedAgentWriteScope": true
   },

examples/babysitter/evidence/f1-feedback-filter/01-signals-red.txt

@@ -0,0 +1,54 @@
+$ node --experimental-strip-types --test tests/signals.test.ts
+✔ a failing commit status is a failing check; a newer success on the same context clears an older failure (2.913465ms)
+✔ reviews are paginated before standing verdicts are derived (0.962498ms)
+✔ a report older than any comment window is still found; only comments after the last report are returned (0.8608ms)
+✔ a marker in someone else's comment is not a report (0.413659ms)
+✔ a head that moves during the read refuses (1.100735ms)
+✔ failing check runs at the head are failing checks; running, passing and other-head runs are not (0.901415ms)
+✔ settleReport keeps the earliest report for a head and deletes a later duplicate of its own (0.869927ms)
+✔ a PR full of long reviews, comments and failing checks still fits the journal and keeps every actor (3.092858ms)
+✔ only a report for the current head counts, and report history never grows the payload (2.598259ms)
+✔ when shortened text still overflows, the oldest comments go first at the 200-char floor (12.446279ms)
+✖ inline review comments from the author, members and allowlisted review bots are feedback; outsiders, other bots and this bot are not (1.09783ms)
+✖ a thread Babysitter already answered, an outdated comment, and feedback before its last report are not feedback (0.755033ms)
+✖ a commented review body is feedback; an empty one, an outsider's, or a change request (already a verdict) is not (0.755585ms)
+✖ review feedback is shortened and, past the floor, dropped oldest first so the read still fits the journal (0.802462ms)
+ℹ tests 14
+ℹ suites 0
+ℹ pass 10
+ℹ fail 4
+ℹ cancelled 0
+ℹ skipped 0
+ℹ todo 0
+ℹ duration_ms 256.559028
+
+✖ failing tests:
+
+test at tests/signals.test.ts:173:1
+✖ inline review comments from the author, members and allowlisted review bots are feedback; outsiders, other bots and this bot are not (1.09783ms)
+  TypeError: Cannot read properties of undefined (reading 'map')
+      at TestContext.<anonymous> (file:///home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/babysitter-f1/examples/babysitter/tests/signals.test.ts:182:37)
+      at async Test.run (node:internal/test_runner/test:1409:7)
+      at async Test.processPendingSubtests (node:internal/test_runner/test:974:7)
+
+test at tests/signals.test.ts:187:1
+✖ a thread Babysitter already answered, an outdated comment, and feedback before its last report are not feedback (0.755033ms)
+  TypeError: Cannot read properties of undefined (reading 'map')
+      at TestContext.<anonymous> (file:///home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/babysitter-f1/examples/babysitter/tests/signals.test.ts:198:37)
+      at async Test.run (node:internal/test_runner/test:1409:7)
+      at async Test.processPendingSubtests (node:internal/test_runner/test:974:7)
+
+test at tests/signals.test.ts:201:1
+✖ a commented review body is feedback; an empty one, an outsider's, or a change request (already a verdict) is not (0.755585ms)
+  TypeError: Cannot read properties of undefined (reading 'map')
+      at TestContext.<anonymous> (file:///home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/babysitter-f1/examples/babysitter/tests/signals.test.ts:210:37)
+      at async Test.run (node:internal/test_runner/test:1409:7)
+      at async Test.processPendingSubtests (node:internal/test_runner/test:974:7)
+
+test at tests/signals.test.ts:214:1
+✖ review feedback is shortened and, past the floor, dropped oldest first so the read still fits the journal (0.802462ms)
+  TypeError: Cannot read properties of undefined (reading 'map')
+      at TestContext.<anonymous> (file:///home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/babysitter-f1/examples/babysitter/tests/signals.test.ts:219:32)
+      at async Test.run (node:internal/test_runner/test:1409:7)
+      at async Test.processPendingSubtests (node:internal/test_runner/test:974:7)
+exit=1