Repository navigation
chore(agents): review replay harness, and the 2026-10-21 process review - #794
Merged
Merged
Conversation
Pins a PR's review snapshot and its ledger on origin (refs/replay), replays the review or one lens on it with stage=replay, and scores the findings against the ledger. Replayed #752 round 1 and #775 rounds 1 and 2; results in docs/internal/PROCESS-REVIEW-2026-10-21.md, with the findings over the deliver and spec runs after #739 and the 22 changes they call for.
V3RON
added this pull request to stack #797
October 21, 2026 08:19
V3RON
marked this pull request as ready for review
October 21, 2026 08:54
V3RON
marked this pull request as draft
October 21, 2026 08:57
V3RON
marked this pull request as ready for review
October 21, 2026 09:04
V3RON
added a commit
that referenced
this pull request
Oct 21, 2026
## What this is The first layer of changes from `docs/internal/PROCESS-REVIEW-2026-10-21.md`: the ones that need no replay to judge, and the two measuring scripts the delivery and instruction changes depend on. Stacked on #794. - Item 3: the spec checker no longer pads five notes into every report. - Item 19: the budget guard counts awake time on the monotonic clock `clock.mjs` already used, so a laptop that sleeps does not spend its session budget (#771 lost its audit that way). `awake.mjs` is the shared helper. - Item 20: `pr-body-check.mjs` fails a PR whose Review section names a review that ran but has no `## Review ledger` comment (four of the last fifteen delivery PRs had none). - Item 21: `review-inputs.sh <pr> <previous> <previous base>` maps a previous commit across a rebase when every commit's patch-id matches, instead of refusing and leaving the coordinator to self-check; the reviewer and the deliver skill pass the base after a rebase or stack sync. - Item 22: a finding records the round that raised it, and the ledger comment starts with one `Round <r>: <commit>` line per check, so a round can be frozen and replayed from GitHub alone. `replay-freeze.mjs` checks against those lines. - Harness pieces 5 and 6: `instruction-reads.mjs` (how often each role opens each rule, brief and manual) and `delivery-counters.mjs` (CI lines sent to fix runs, stack syncs, empty builds, no-change fix runs, tests per Tests line, per merged PR). `transcripts.mjs` holds the transcript helpers the three measuring scripts share. ## Baseline the counters print today PRs merged since 2026-10-20: 26 CI lines, 12 stack syncs, 2 empty builds, 6 no-change fix runs, 200 tests over 128 Tests lines, test/src LOC 1.4. Build implementers opened their brief in 23 of 38 runs and `testing.md` in 5. *Written by an agent.*
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A review replay harness, and the process review it was built for.
docs/internal/PROCESS-REVIEW-2026-10-21.mdreads the deliver and spec runs after #739: where the time went, the obstacles (stale stacked bases, stack tooling, CI flakes, late ADR reviews, the round cap), review quality, spec-session holes, test growth and which instruction files agents actually open. It lists 22 changes, each with the number that judges it, and says the replay harness comes first.The harness:
review-inputs.shtakesREVIEW_HEAD,REVIEW_BASEandREVIEW_RULES, so a merged PR's snapshot can be reviewed again, with today's or another commit's rules.replay-freeze.mjspins a round's head, base and previous asrefs/replay/<PR>/<round>/*on origin and the PR's Review ledger as the blobrefs/replay/<PR>/truth, never in the checkout.review-replay.shbuilds the blind inputs for a snapshot and prints the reviewer's prompt, taggedstage=replay;--lensreplays one lens alone.review-score.mjsscores a replay's findings (JSON or a lens report) against the truth and appends a row to.agents/replay/results.jsonl.review-metrics.mjsleavesstage=replayruns out unless--replay; the reviewer has a replay clause.Replayed so far
Harness defects found by the examples and fixed in this PR: the coordinator reading the truth file, coordinators self-checking instead of spawning lenses, lens runs tagged as real reviews.
Written by an agent.