Repository navigation
feat(babysitter): wake on real PR review feedback, not only verdicts - #633
Conversation
The standalone Babysitter acted only on failing checks, CHANGES_REQUESTED reviews and @babysitter directives. Review feedback, meaning inline review comments and COMMENTED review bodies, never made a PR actionable. - signals: read /pulls/:n/comments. Feedback is an inline comment that is not outdated and not answered in its thread, or a non-empty COMMENTED review, newer than Babysitter's last comment. - Feedback counts only from the PR author, OWNER/MEMBER/COLLABORATOR, or a policy reviewBots login. It never counts from botLogin or a policy ownAgents account, so our own agents' "Fixed in ..." replies do not wake it. - policy: pin reviewBots (the five bots seen on recent AgentWorkforce PRs) and ownAgents (AgentRelayBot, kjgbot). - The artifact is rebuilt, so its sha256 is now af8d5174...; Cloud's pin must move with it (cloud C1). The wake contract is unchanged: GitHub sends pull_request_review.submitted for every review, including a single inline comment. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Session-Id: 497569fc-550a-44a0-a906-35c233c27ba6
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 3 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| const [runs, statuses, reviews, comments, inline] = await Promise.all([ | ||
| pages(`/commits/${c.head}/check-runs?filter=latest`, 'check_runs'), pages(`/commits/${c.head}/statuses`), | ||
| pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`), | ||
| pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`), pages(`/pulls/${c.number}/comments`), |
There was a problem hiding this comment.
🔴 Review thread replies never trigger diagnosis
A new thread reply reaches readSignals only if another subscribed event wakes the flow. The subscriptions omit pull_request_review_comment.created, so reply-only feedback goes undiagnosed.
Learn more
Babysitter only rereads GitHub when a registered webhook launches its body. GitHub delivers a reply added to an existing pull-request review thread as a review-comment creation event, without requiring a new review submission. The new review-comment read can find that reply, but the registered subscription set has no review-comment creation event, and admitDelivery rejects undeclared event types. Thus a reply without another wake remains unseen.
Example: A collaborator replies “the retry still races” to a review thread after the latest PR push. GitHub delivers pull_request_review_comment.created; the flow has no matching handler, so no diagnosis runs.
Recommended fix: Register pull_request_review_comment.created in subscriptions, update the declared trigger contract and Cloud webhook profile, and verify that a reply-only delivery reaches the body.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Not changed, with evidence. A thread reply already wakes Babysitter: GitHub files each reply as its own submitted COMMENTED review, so pull_request_review.submitted fires. I checked every sampled reply on flows#624/#628 (human UI replies and bot replies). Each has its own pull_request_review_id whose review is COMMENTED with submitted_at equal to the reply's created_at (e.g. review 5460128216 for comment 4221828885). This is review-object evidence, not a webhook delivery log. The first live run will confirm the delivery itself. Adding pull_request_review_comment would also need Cloud changes: flow-trigger-sources.ts rejects that family today, and so does the pinned standalone profile. STANDALONE.md now records this.
| ...inline.filter((m: any) => from(m) && typeof m.line === 'number' && String(m.created_at ?? '') > since | ||
| && m.id > (answered.get(m.in_reply_to_id ?? m.id) ?? 0)) | ||
| .map((m: any) => ({ kind: 'inline', id: m.id, login: m.user.login, body: m.body, path: m.path, line: m.line, | ||
| thread: m.in_reply_to_id ?? m.id, createdAt: m.created_at })), | ||
| ...reviews.filter((r: any) => r.state === 'COMMENTED' && typeof r.body === 'string' && r.body.trim() && from(r) |
There was a problem hiding this comment.
🟡 File-level review feedback is ignored
For a file-level review comment, m.line is null, so the inline filter discards it. If its COMMENTED review has no body, neither branch finds actionable feedback.
Learn more
GitHub allows a pull-request review comment on a file rather than a diff line. That comment has a file path and a null line. The filter excludes it, and a review with an empty body adds no separate feedback. whatChanged consequently returns no action when checks and directives are also empty.
Example: A reviewer comments “This module needs an integration test” on src/queue.ts as a file-level comment with line: null and submits a review with an empty body. Babysitter declines instead of diagnosing the comment.
Recommended fix: Accept file-level review comments with a valid file path and render their location without a line number in agentTask; continue excluding genuinely outdated line comments using the available original/current-line and subject metadata.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in 49b4b1b. subject_type === 'file' comments count and render as path with no line. Test: 'file-level review comments are feedback; a line comment GitHub marked outdated (no diff position) is not'.
| ...inline.filter((m: any) => from(m) && typeof m.line === 'number' && String(m.created_at ?? '') > since | ||
| && m.id > (answered.get(m.in_reply_to_id ?? m.id) ?? 0)) |
There was a problem hiding this comment.
🟡 Newly submitted pending comments are skipped
When a pending comment predates since, its later submission cannot pass the created_at filter. On a new head, the newly visible feedback is lost even without a report for that head.
Learn more
A pending GitHub review can contain comments created before the reviewer submits the review. This filter compares each comment's creation time with the bot's last issue-comment time, not the review submission time. A bot report on a previous head can land between those two times; after a new head arrives, createStandaloneBabysitter sees no report for that head but receives no feedback from the newly submitted review.
Example: A reviewer drafts a comment at 09:00; Babysitter reports on head A at 09:01; head B arrives at 09:02 without outdating that comment; the reviewer submits at 09:03. The 09:00 created_at fails the 09:01 cutoff and Babysitter declines head B.
Recommended fix: Associate inline comments with their review's submitted_at when deciding whether they became visible after the bot's last report; retain an explicit policy for immediately published replies and for drafts that remain pending.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in 49b4b1b. An inline comment now dates from max(created_at, its review's submitted_at). Test: 'a pending review comment counts from its review's submission…'.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b1a5809bd9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const [runs, statuses, reviews, comments, inline] = await Promise.all([ | ||
| pages(`/commits/${c.head}/check-runs?filter=latest`, 'check_runs'), pages(`/commits/${c.head}/statuses`), | ||
| pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`), | ||
| pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`), pages(`/pulls/${c.number}/comments`), |
There was a problem hiding this comment.
When feedback is posted as an individual diff comment or as a reply in an existing review thread, GitHub emits pull_request_review_comment.created; however, the inspected wake contract in examples/babysitter/subscriptions.ts registers only pull_request_review.submitted and .dismissed and does not accept the review-comment family. Consequently, this newly added API read will not run for those comments until an unrelated webhook arrives, so Babysitter can miss exactly the feedback and follow-ups this logic is intended to process. GitHub documents review-comment creation as a distinct webhook event: Webhook events and payloads.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not changed, with evidence. A thread reply already wakes Babysitter: GitHub files each reply as its own submitted COMMENTED review, so pull_request_review.submitted fires. I checked every sampled reply on flows#624/#628 (human UI replies and bot replies). Each has its own pull_request_review_id whose review is COMMENTED with submitted_at equal to the reply's created_at (e.g. review 5460128216 for comment 4221828885). This is review-object evidence, not a webhook delivery log. The first live run will confirm the delivery itself. Adding pull_request_review_comment would also need Cloud changes: flow-trigger-sources.ts rejects that family today, and so does the pinned standalone profile. STANDALONE.md now records this.
| ...inline.filter((m: any) => from(m) && typeof m.line === 'number' && String(m.created_at ?? '') > since | ||
| && m.id > (answered.get(m.in_reply_to_id ?? m.id) ?? 0)) |
There was a problem hiding this comment.
When a reviewer leaves feedback on an entire file rather than a diff line, GitHub returns a review comment with subject_type: "file", for which line is not required; the numeric-line predicate therefore drops the comment even though it is current and actionable. If the containing COMMENTED review has an empty summary, there is no review-body fallback and Babysitter treats the wake as having no feedback. Accept active file-level subjects and render their path without a line number; GitHub documents both line and file subject types and makes line optional for file comments: REST API endpoints for review comments.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 49b4b1b. subject_type === 'file' comments count and render as path with no line. Test: 'file-level review comments are feedback; a line comment GitHub marked outdated (no diff position) is not'.
… marker Two decisions from Khaliq and one finding: - Software Garden PRs are babysat by default. A Garden PR is one whose head is a same-repo relayflow/* branch. Any other PR still opts in with the `babysit` label, and removing the label opts it out. A fork cannot claim a relayflow/* name. - Software Garden posts as the same app login as Babysitter (agent-relay-code). A botLogin comment now counts as Babysitter's own only when it carries a `<!-- babysitter:` marker. Before this, the Garden's adversarial-review comment moved the "since last comment" window and marked review threads as answered. The artifact is rebuilt: sha256 9083e171... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Session-Id: 497569fc-550a-44a0-a906-35c233c27ba6
… still decline Garden drafts a PR whose checks fail, and getting it out of draft with the feedback fixed is the job (lead + Khaliq decision). Artifact sha256 9f04a000... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Session-Id: 497569fc-550a-44a0-a906-35c233c27ba6
|
Update (b9c6924, then head): decisions from Khaliq and the lead, plus one finding.
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
4 issues found and verified against the latest diff
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="examples/babysitter/signals.ts">
<violation number="1" location="examples/babysitter/signals.ts:70">
P2: Subscribe the body to `pull_request_review_comment.created`; replies to existing review threads do not submit a new review, so this inline-comment read otherwise runs only after an unrelated wake.</violation>
<violation number="2" location="examples/babysitter/signals.ts:112">
P2: Bot-authored PR feedback is ignored even when the bot is the PR author. Check the author/trusted association before applying the review-bot allowlist so the documented PR-author rule also applies to bot authors.</violation>
</file>
<file name="examples/babysitter/evidence/f1-feedback-filter/09-mutation.txt">
<violation number="1" location="examples/babysitter/evidence/f1-feedback-filter/09-mutation.txt:4">
P2: This mutation record can't be replayed against the code this PR ships. HEAD's tests/signals.test.ts has 16 tests, but the run records 15, and the one-expression guard it names (`if (login === bot ? !own(m) : !ours.includes(login)) continue;`) can't account for a 53-insertion/5-deletion diff-stat — pr-base..HEAD changes all of signals.ts by only 57/6. With the final marker-based `own` guard, 'compare to bot only' also fails the Garden test (unmarked bot post marks thread 1 answered), so the recorded single failure is not this mutation's effect on the committed code. Re-run the mutation against the committed revision, show a clean diff before/after, and update the counts.</violation>
</file>
<file name="examples/babysitter/tests/signals.test.ts">
<violation number="1" location="examples/babysitter/tests/signals.test.ts:210">
P3: Test 3 only positively asserts the allowlisted bot's COMMENTED review body is feedback. A member's non-empty COMMENTED body being included, and a COMMENTED review submitted before the last report being excluded by `since`, are never asserted, so both branches can regress without failing the suite. Add a member COMMENTED review with a real body to the expected feedback and one with an earlier `submitted_at` that must be excluded.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| if (login.endsWith('[bot]')) return bots.includes(login); | ||
| return login === author || trusted.includes(m.author_association); |
There was a problem hiding this comment.
P2: Bot-authored PR feedback is ignored even when the bot is the PR author. Check the author/trusted association before applying the review-bot allowlist so the documented PR-author rule also applies to bot authors.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/babysitter/signals.ts, line 112:
<comment>Bot-authored PR feedback is ignored even when the bot is the PR author. Check the author/trusted association before applying the review-bot allowlist so the documented PR-author rule also applies to bot authors.</comment>
<file context>
@@ -84,6 +99,36 @@ export async function readSignals(c: { owner: string; repo: string; number: numb
+ const from = (m: any) => {
+ const login = String(m.user?.login ?? '').toLowerCase();
+ if (!login || ours.includes(login)) return false;
+ if (login.endsWith('[bot]')) return bots.includes(login);
+ return login === author || trusted.includes(m.author_association);
+ };
</file context>
| if (login.endsWith('[bot]')) return bots.includes(login); | |
| return login === author || trusted.includes(m.author_association); | |
| if (login === author || trusted.includes(m.author_association)) return true; | |
| return login.endsWith('[bot]') && bots.includes(login); |
There was a problem hiding this comment.
Not changed. The bot authors we'd see are our own app (Software Garden posts as agent-relay-code, the same login as botLogin) or dependency bots like dependabot, whose own comments are not review feedback. Bot feedback is governed only by the explicit reviewBots allowlist. Letting an author bot through would reopen the self-wake loop the shared-login fix closed.
| const [runs, statuses, reviews, comments, inline] = await Promise.all([ | ||
| pages(`/commits/${c.head}/check-runs?filter=latest`, 'check_runs'), pages(`/commits/${c.head}/statuses`), | ||
| pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`), | ||
| pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`), pages(`/pulls/${c.number}/comments`), |
There was a problem hiding this comment.
P2: Subscribe the body to pull_request_review_comment.created; replies to existing review threads do not submit a new review, so this inline-comment read otherwise runs only after an unrelated wake.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/babysitter/signals.ts, line 70:
<comment>Subscribe the body to `pull_request_review_comment.created`; replies to existing review threads do not submit a new review, so this inline-comment read otherwise runs only after an unrelated wake.</comment>
<file context>
@@ -52,9 +65,9 @@ export async function readSignals(c: { owner: string; repo: string; number: numb
+ const [runs, statuses, reviews, comments, inline] = await Promise.all([
pages(`/commits/${c.head}/check-runs?filter=latest`, 'check_runs'), pages(`/commits/${c.head}/statuses`),
- pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`),
+ pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`), pages(`/pulls/${c.number}/comments`),
]);
const bad = ['failure', 'timed_out', 'cancelled', 'action_required', 'startup_failure', 'stale', 'error'];
</file context>
There was a problem hiding this comment.
Not changed, with evidence. A thread reply already wakes Babysitter: GitHub files each reply as its own submitted COMMENTED review, so pull_request_review.submitted fires. I checked every sampled reply on flows#624/#628 (human UI replies and bot replies). Each has its own pull_request_review_id whose review is COMMENTED with submitted_at equal to the reply's created_at (e.g. review 5460128216 for comment 4221828885). This is review-object evidence, not a webhook delivery log. The first live run will confirm the delivery itself. Adding pull_request_review_comment would also need Cloud changes: flow-trigger-sources.ts rejects that family today, and so does the pinned standalone profile. STANDALONE.md now records this.
| review(3, 'mallory', 'COMMENTED', 'ship it', 'NONE'), | ||
| review(4, 'carol', 'CHANGES_REQUESTED', 'Bound it', 'MEMBER'), | ||
| ] })); | ||
| assert.deepEqual(s.reviewFeedback.map((r: any) => [r.kind, r.id, r.login]), [['review', 1, 'coderabbitai[bot]']]); |
There was a problem hiding this comment.
P3: Test 3 only positively asserts the allowlisted bot's COMMENTED review body is feedback. A member's non-empty COMMENTED body being included, and a COMMENTED review submitted before the last report being excluded by since, are never asserted, so both branches can regress without failing the suite. Add a member COMMENTED review with a real body to the expected feedback and one with an earlier submitted_at that must be excluded.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/babysitter/tests/signals.test.ts, line 210:
<comment>Test 3 only positively asserts the allowlisted bot's COMMENTED review body is feedback. A member's non-empty COMMENTED body being included, and a COMMENTED review submitted before the last report being excluded by `since`, are never asserted, so both branches can regress without failing the suite. Add a member COMMENTED review with a real body to the expected feedback and one with an earlier `submitted_at` that must be excluded.</comment>
<file context>
@@ -162,3 +163,84 @@ test('when shortened text still overflows, the oldest comments go first at the 2
+ review(3, 'mallory', 'COMMENTED', 'ship it', 'NONE'),
+ review(4, 'carol', 'CHANGES_REQUESTED', 'Bound it', 'MEMBER'),
+ ] }));
+ assert.deepEqual(s.reviewFeedback.map((r: any) => [r.kind, r.id, r.login]), [['review', 1, 'coderabbitai[bot]']]);
+ assert.deepEqual(s.changeRequests.map((r: any) => r.id), [4]);
+});
</file context>
There was a problem hiding this comment.
Fixed in 49b4b1b. The new test asserts that a member's non-empty COMMENTED body counts (review 901) and that one submitted before the last report does not (review 902).
There was a problem hiding this comment.
2 issues found and verified against the latest diff
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="examples/babysitter/evidence/f1-feedback-filter/03-own-agents-red.txt">
<violation number="1" location="examples/babysitter/evidence/f1-feedback-filter/03-own-agents-red.txt:1">
P3: This red capture shows 3 failing tests at tests/signals.test.ts:224 and tests/standalone.test.ts:324/333, but that failing state exists in no committed revision: pr-base has none of these feedback tests, and at HEAD the same suites pass (04-all-babysitter-green.txt records 0 failures). Committing an exit=1 capture that cannot be reproduced from any commit, with no provenance, contradicts the committed green state and is not auditable. The repo's own convention annotates red captures (evidence/standalone/07-review-red.txt: "Code at 42f3828f, tests at 2b70cc26 (the red before the fix commit)"), and 09-mutation.txt already provides a reproducible red->green pair for the own-agents behavior. Annotate the revision the capture was taken against, or drop 01-03 and keep the mutation file's provenance pair.</violation>
</file>
<file name="examples/babysitter/signals.ts">
<violation number="1" location="examples/babysitter/signals.ts:70">
P2: The new feedback cutoff can replay already-handled comments because the issue-comments request does not enforce chronological order. Request `sort=created&direction=asc` here so `lastOwn`, `recent`, and `since` all refer to the latest bot report.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| const [runs, statuses, reviews, comments, inline] = await Promise.all([ | ||
| pages(`/commits/${c.head}/check-runs?filter=latest`, 'check_runs'), pages(`/commits/${c.head}/statuses`), | ||
| pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`), | ||
| pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`), pages(`/pulls/${c.number}/comments`), |
There was a problem hiding this comment.
P2: The new feedback cutoff can replay already-handled comments because the issue-comments request does not enforce chronological order. Request sort=created&direction=asc here so lastOwn, recent, and since all refer to the latest bot report.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/babysitter/signals.ts, line 70:
<comment>The new feedback cutoff can replay already-handled comments because the issue-comments request does not enforce chronological order. Request `sort=created&direction=asc` here so `lastOwn`, `recent`, and `since` all refer to the latest bot report.</comment>
<file context>
@@ -52,9 +65,9 @@ export async function readSignals(c: { owner: string; repo: string; number: numb
+ const [runs, statuses, reviews, comments, inline] = await Promise.all([
pages(`/commits/${c.head}/check-runs?filter=latest`, 'check_runs'), pages(`/commits/${c.head}/statuses`),
- pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`),
+ pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`), pages(`/pulls/${c.number}/comments`),
]);
const bad = ['failure', 'timed_out', 'cancelled', 'action_required', 'startup_failure', 'stale', 'error'];
</file context>
| pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments`), pages(`/pulls/${c.number}/comments`), | |
| pages(`/pulls/${c.number}/reviews`), pages(`/issues/${c.number}/comments?sort=created&direction=asc`), pages(`/pulls/${c.number}/comments`), |
There was a problem hiding this comment.
Not changed. GET /repos/{owner}/{repo}/issues/{issue_number}/comments has no sort/direction parameters (only since). Those exist on the repository-wide /issues/comments listing. The per-issue listing is returned in ascending order, and the existing 'only comments after the last report' test pins that use.
| @@ -0,0 +1,241 @@ | |||
| $ node --experimental-strip-types --test tests/signals.test.ts tests/standalone.test.ts | |||
There was a problem hiding this comment.
P3: This red capture shows 3 failing tests at tests/signals.test.ts:224 and tests/standalone.test.ts:324/333, but that failing state exists in no committed revision: pr-base has none of these feedback tests, and at HEAD the same suites pass (04-all-babysitter-green.txt records 0 failures). Committing an exit=1 capture that cannot be reproduced from any commit, with no provenance, contradicts the committed green state and is not auditable. The repo's own convention annotates red captures (evidence/standalone/07-review-red.txt: "Code at 42f3828, tests at 2b70cc2 (the red before the fix commit)"), and 09-mutation.txt already provides a reproducible red->green pair for the own-agents behavior. Annotate the revision the capture was taken against, or drop 01-03 and keep the mutation file's provenance pair.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/babysitter/evidence/f1-feedback-filter/03-own-agents-red.txt, line 1:
<comment>This red capture shows 3 failing tests at tests/signals.test.ts:224 and tests/standalone.test.ts:324/333, but that failing state exists in no committed revision: pr-base has none of these feedback tests, and at HEAD the same suites pass (04-all-babysitter-green.txt records 0 failures). Committing an exit=1 capture that cannot be reproduced from any commit, with no provenance, contradicts the committed green state and is not auditable. The repo's own convention annotates red captures (evidence/standalone/07-review-red.txt: "Code at 42f3828f, tests at 2b70cc26 (the red before the fix commit)"), and 09-mutation.txt already provides a reproducible red->green pair for the own-agents behavior. Annotate the revision the capture was taken against, or drop 01-03 and keep the mutation file's provenance pair.</comment>
<file context>
@@ -0,0 +1,241 @@
+$ node --experimental-strip-types --test tests/signals.test.ts tests/standalone.test.ts
+✔ a failing commit status is a failing check; a newer success on the same context clears an older failure (2.160205ms)
+✔ reviews are paginated before standing verdicts are derived (0.579688ms)
</file context>
There was a problem hiding this comment.
Addressed in 49b4b1b. evidence/f1-feedback-filter/README.md records the code state of every red capture. Red captures are intermediate working trees by design, because each commit holds tests and fix together. This one's extra failure is explained there: it came from the half-changed reader call site.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c176f9c. Configure here.
There was a problem hiding this comment.
1 existing issue remains and 1 new issue found across 22 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="examples/babysitter/evidence/f1-feedback-filter/11-garden-scope-red.txt">
<violation number="1" location="examples/babysitter/evidence/f1-feedback-filter/11-garden-scope-red.txt:25">
P2: This evidence file ends a chain of three consecutive failing captures (10, 11, 12) with no green counterpart for the final state. The last green full-suite run (04, pass 103/fail 0) predates the garden-scope and garden-draft tests now in HEAD, so the bundled evidence reads as the suite currently failing despite the "passing Babysitter tests" claim. Re-run the full suite at HEAD and capture a green run (or cite the CI run) so reviewers can tell an intentionally recorded intermediate failure from the current state.</violation>
</file>
Requires human review: Auto-approval blocked because this review re-detected 7 unresolved issues already reported by Cubic.
View guided diff | Turn on auto-fix | Re-trigger cubic
| ✔ unanswered review feedback alone is actionable and reaches the agent with its location (8.419447ms) | ||
| ✔ the reader is told the live PR author and the policy's review bots (1.984718ms) | ||
| ✔ review bots must be a list of [bot] logins (0.264722ms) | ||
| ✖ Software Garden PRs (same-repo relayflow/* heads) are in scope by default; any other PR needs the opt-in label (2.904317ms) |
There was a problem hiding this comment.
P2: This evidence file ends a chain of three consecutive failing captures (10, 11, 12) with no green counterpart for the final state. The last green full-suite run (04, pass 103/fail 0) predates the garden-scope and garden-draft tests now in HEAD, so the bundled evidence reads as the suite currently failing despite the "passing Babysitter tests" claim. Re-run the full suite at HEAD and capture a green run (or cite the CI run) so reviewers can tell an intentionally recorded intermediate failure from the current state.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/babysitter/evidence/f1-feedback-filter/11-garden-scope-red.txt, line 25:
<comment>This evidence file ends a chain of three consecutive failing captures (10, 11, 12) with no green counterpart for the final state. The last green full-suite run (04, pass 103/fail 0) predates the garden-scope and garden-draft tests now in HEAD, so the bundled evidence reads as the suite currently failing despite the "passing Babysitter tests" claim. Re-run the full suite at HEAD and capture a green run (or cite the CI run) so reviewers can tell an intentionally recorded intermediate failure from the current state.</comment>
<file context>
@@ -0,0 +1,53 @@
+✔ unanswered review feedback alone is actionable and reaches the agent with its location (8.419447ms)
+✔ the reader is told the live PR author and the policy's review bots (1.984718ms)
+✔ review bots must be a list of [bot] logins (0.264722ms)
+✖ Software Garden PRs (same-repo relayflow/* heads) are in scope by default; any other PR needs the opt-in label (2.904317ms)
+ℹ tests 24
+ℹ suites 0
</file context>
There was a problem hiding this comment.
Fixed in 49b4b1b. 04-all-babysitter-green.txt was re-captured at head: 114 tests, 107 pass, 0 fail, 7 todo, including the Garden scope/draft and round-1 tests.
…-second feedback; own-agent directives - File-level review comments (subject_type "file", no line) are feedback. A line comment GitHub marked outdated keeps its blob line but loses its diff position, so it is excluded. - A pending review's comments become visible on submission. They now date from max(created_at, the review's submitted_at). - The since-last-report cutoff is inclusive. Timestamps are per second, so feedback in the report's own second is no longer lost. - An @babysitter directive from an ownAgents account is not a directive. - kjgbot is removed from ownAgents: it may be a shared human account. - A file-level path renders without a line. - STANDALONE.md: precise visibility rules and the shared drop path. It also records why thread replies need no new subscription: each reply is its own submitted COMMENTED review. - Evidence: README with provenance for every red capture; a reproducible mutation record; the typecheck filter stated literally, with main's filtered count. Artifact sha256 is now 70e27d2c... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Session-Id: 497569fc-550a-44a0-a906-35c233c27ba6
There was a problem hiding this comment.
1 issue found across 17 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="examples/babysitter/evidence/f1-feedback-filter/README.md">
<violation number="1" location="examples/babysitter/evidence/f1-feedback-filter/README.md:3">
P3: The opening guarantee "Every file is the unedited output of the command on its first `$` line" is not true for this directory: `06-typecheck.txt` runs 4 commands (4 `$` lines plus hand-written header comments) and `09-mutation.txt` runs 7 commands with interleaved `#` annotations, and the table's own rows for them describe multi-step sequences ("see its header", "Literal snapshot, sed, diff, red run, cp restore, cmp, green run"). Reword so the claim matches the captures, e.g. that each file is the unedited output of the commands on its `$` lines, with red captures being pre-implementation trees.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| @@ -0,0 +1,23 @@ | |||
| # F1 evidence: what each capture is | |||
|
|
|||
| Every file is the unedited output of the command on its first `$` line. A red | |||
There was a problem hiding this comment.
P3: The opening guarantee "Every file is the unedited output of the command on its first $ line" is not true for this directory: 06-typecheck.txt runs 4 commands (4 $ lines plus hand-written header comments) and 09-mutation.txt runs 7 commands with interleaved # annotations, and the table's own rows for them describe multi-step sequences ("see its header", "Literal snapshot, sed, diff, red run, cp restore, cmp, green run"). Reword so the claim matches the captures, e.g. that each file is the unedited output of the commands on its $ lines, with red captures being pre-implementation trees.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/babysitter/evidence/f1-feedback-filter/README.md, line 3:
<comment>The opening guarantee "Every file is the unedited output of the command on its first `$` line" is not true for this directory: `06-typecheck.txt` runs 4 commands (4 `$` lines plus hand-written header comments) and `09-mutation.txt` runs 7 commands with interleaved `#` annotations, and the table's own rows for them describe multi-step sequences ("see its header", "Literal snapshot, sed, diff, red run, cp restore, cmp, green run"). Reword so the claim matches the captures, e.g. that each file is the unedited output of the commands on its `$` lines, with red captures being pre-implementation trees.</comment>
<file context>
@@ -0,0 +1,23 @@
+# F1 evidence: what each capture is
+
+Every file is the unedited output of the command on its first `$` line. A red
+capture is a snapshot of an intermediate working tree, taken before that change's
+implementation existed. It does not reproduce from any commit, because each commit
</file context>
…oposal paths - Remove the shipped-source waiver. The fixer now calls f.agent with a literal cli/model pair per origin CLI (claude/claude-sonnet-5, codex/gpt-5.6-sol), so the unmodified audit resolves both pairs. The fixer ignores the policy's agentCli. - Restore tests/standalone-artifact.test.ts from #633 unmodified. The build prints "Checked sha256:<sha> (<name>)", which that test accepts. The fixer artifact and the no-__require check are in the new tests/fixer-artifact.test.ts. - Read changed paths NUL-delimited, and diff with core.quotePath=false. A path git would still quote (quote, backslash, control char) refuses the proposal, so a newline cannot split a workflow path past the rule, and Cloud parses back exactly the checked paths. Fixer artifact sha256: 797ce37d... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Session-Id: 497569fc-550a-44a0-a906-35c233c27ba6
…oposal paths - Remove the shipped-source waiver. The fixer now calls f.agent with a literal cli/model pair per origin CLI (claude/claude-sonnet-5, codex/gpt-5.6-sol), so the unmodified audit resolves both pairs. The fixer ignores the policy's agentCli. - Restore tests/standalone-artifact.test.ts from #633 unmodified. The build prints "Checked sha256:<sha> (<name>)", which that test accepts. The fixer artifact and the no-__require check are in the new tests/fixer-artifact.test.ts. - Read changed paths NUL-delimited, and diff with core.quotePath=false. A path git would still quote (quote, backslash, control char) refuses the proposal, so a newline cannot split a workflow path past the rule, and Cloud parses back exactly the checked paths. Fixer artifact sha256: 797ce37d... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Session-Id: 497569fc-550a-44a0-a906-35c233c27ba6 Session-Id: 497569fc-550a-44a0-a906-35c233c27ba6
1 participant
F1 in the Babysitter owner plan. The plan's goal: a flow that wakes on PR feedback, loads the original session, and fixes the feedback in a sandbox.
What changes
Until now the standalone Babysitter acted only on failing checks,
CHANGES_REQUESTEDreviews, and@babysitterdirectives. A reviewer's inline comments, or a review bot's findings, could never make a PR actionable.signals.tsnow also reads/pulls/:n/comments, and returnsreviewFeedback:line != null), newer than Babysitter's last comment, with no later reply in their thread frombotLoginor anownAgentsaccount;COMMENTEDreview bodies, newer than Babysitter's last comment.OWNER/MEMBER/COLLABORATOR, or a policyreviewBotslogin. NeverbotLoginorownAgents.AgentRelayBot(MEMBER) posts "Fixed in …" thread replies on our PRs. WithoutownAgentsthose replies would count as human feedback and loop.Review comment by X on path:line: …orReview by X: ….pull_request_review.submittedfor every review, including a single inline comment. Cloud's trigger profile (checks,comments,events,repository,reviews) still matches.tests/standalone-artifact.test.tsasserts the exactstandalone-policy.json. I updated that assertion to the newreviewBots/ownAgentslists, because changing the policy is what this PR does. Please confirm both lists. Iskjgbotours?eed5e5de…→af8d51743982e018b1a0c0aceb66e5a4f038bf2a55ff7a75d543de8276e2322a(37,044 bytes).Not in this PR
Evidence (
examples/babysitter/evidence/f1-feedback-filter/, unedited output)01-signals-red.txt: the 4 new reader tests fail before the implementation.02-standalone-red.txt: the flow tests fail before the implementation.03-own-agents-red.txt: theownAgentstests fail before the implementation.04-all-babysitter-green.txt:tests 107, pass 100, fail 0, todo 7, exit 0. The 7 todos are the existing deliberate gates.05-artifact-check.txt:build-standalone.mjs --checkreproducesaf8d5174….06-typecheck.txt: in my local environment,main@fe60dd4chas 73 tsc errors, all environmental (missing@relayfile/relay-helpers/ai-histand a dual-packageCtx). This branch has 75: the same classes, plus 2 from the two newbody(f)test calls. 0 errors outside those classes. CI is the authority here.07-flows-check.txt:flows checkon the artifact passes.08-sdk-shipped-source.txt: the shipped-source model pins pass, 2/2.09-mutation.txt: revertingownAgentsthread-answering makesour own agent accounts are never feedback…fail. The file was restored byte-for-byte (cmpidentical) and the tests are green again.git diff --statline in that file is against HEAD (uncommitted work at that point), not the mutation alone.🤖 Generated with Claude Code
Note
Medium Risk
Changes when Babysitter runs on live PRs (new wake reasons, policy allowlists, and Garden auto-scope); misconfigured
reviewBots/ownAgentscould miss feedback or cause unwanted wakes, but behavior is bounded to diagnose-only with existing head-binding guards.Overview
Standalone Babysitter now treats unanswered PR review feedback as actionable, alongside failing checks, change requests, and
@babysitterdirectives.Signal read (
signals.ts) paginates/pulls/:n/commentsand returnsreviewFeedback: non-outdated inline threads and non-emptyCOMMENTEDreview bodies newer than Babysitter’s last marked comment, filtered to the PR author, collaborators, or policyreviewBots, and excludingbotLoginandownAgents(so our “Fixed in …” replies don’t re-wake the flow). Thread replies from Babysitter orownAgentsmark threads answered. Payload stays within the 50KB journal budget (truncate, then drop oldest feedback). The diagnose agent task lists each item with path/line when inline.Policy & scope:
standalone-policy.jsonpinsreviewBotsandownAgents. Same-reporelayflow/*Software Garden PRs are in scope by default (drafts included); other PRs still need thebabysitlabel. When Garden shares the app login, only comments with a<!-- babysitter:marker count as Babysitter’s own for the comment window and thread answers.The committed standalone artifact and manifest digest update accordingly; Cloud must move its pinned digest before deploy.
Reviewed by Cursor Bugbot for commit c176f9c. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
The Babysitter now wakes on real PR review feedback — inline review comments and non-empty
COMMENTEDreview bodies — not just failing checks,CHANGES_REQUESTEDreviews, and@babysitterdirectives./pulls/:n/commentsand lists each item in the agent task asReview comment by X on path:line: …orReview by X: ….OWNER/MEMBER/COLLABORATOR, or a policyreviewBotslogin; never frombotLoginorownAgents, so our "Fixed in …" thread replies can't loop the flow.submitted_at; the since-last-report cutoff is inclusive because timestamps are per-second.@babysitterdirective from anownAgentsaccount is not a directive.relayflow/*head branches) are babysat by default, including drafts — Garden opens draft PRs with failing checks, and getting them out of draft with feedback fixed is the job. Any other PR still opts in with thebabysitlabel, and label opt-in drafts still decline.botLogincomment counts as Babysitter's own only when it carries a<!-- babysitter:marker, so Garden's adversarial-review comment no longer moves the "since last comment" window or marks threads answered.standalone-policy.jsonnow pinsreviewBotsandownAgents(AgentRelayBot;kjgbotwas dropped as it may be a shared human account). The artifact digest moves to70e27d2c…, so Cloud must move its pinned digest before deploying this artifact.Written for commit 49b4b1b. Summary will update on new commits.
Agent Relay sessions
claudesession497569fc· opened viagh pr create· last active 2026-10-09