fix: one review rubric, and a session retry that knows why it failed (#166, #167) - #168
Merged
Merged
Conversation
Two items were rejected today for artefacts that had already been found, fixed and closed — in the other copy of the prompt. **#167 — the rubric was duplicated.** `executor.py` and `session_executor.py` each carried a REVIEW_PROMPT, and every correction made from measurement landed in the headless one: that a diff is the whole change, that the harness ran the checks and which ones, that the files the diff touched are supplied, that scope belongs to the task. The session reviewer had none of them. So R4 added a paragraph to a diagnostics panel and was rejected on "the diff does not demonstrate that it is specifically integrated into the existing diagnostics/support surface rather than merely adjacent" which is exactly the failure #156 documented and closed: a diff cannot show its surroundings, and the fix was to supply them. R3's rejection carried the same shape. Both also contained a real point — R3 widened a repository trait's return type, R4's wording claimed everything absent from the bundle was deliberate — so the reviews were not wrong. Half of each objection was simply a solved problem in the wrong file. The rubric is now imported rather than copied, and the two reviewer helpers are module-level functions both executors call, so a correction cannot land in one and not the other. Two tests guard it: the prompts must be the same object, because "roughly the same" is what lets drift back; and `_review` must actually pass the fields, because identical prompts with empty fields would keep the reviewers unequal while passing the first test. **#166 — a session retry was blind.** The agent's prompt carried the title, the brief and the checks. Not the reviewer's verdict, not the last failure. A session attempt is minutes of a real agent rather than one API call, so repeating one blind is the most expensive avoidable thing here — and R3's retry would have been sent the identical brief with no mention of the specific, actionable criticism it had just received. Not resumption: the item is re-read from the current brief, no prior diff is fed back, and a guided attempt consumes an attempt exactly as an unguided one does. A first attempt reads exactly as it did before. One thing worth recording because tests did not catch it: the first version of this borrowed `Executor._review_context` as an unbound method on a `SessionExecutor`, which has no `context_policy` and would have raised at runtime. The suite passed anyway, because the test that exercises that path lives on another branch. Extracting real shared functions removes the class of mistake rather than the instance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
I branched this from main, which does not carry #162, and re-applied the empty-diff fix here while rewriting _review. That left the same change in two open pull requests — my error in branching, not a second bug. Folding them: this branch already had the fix and was missing the test that proves it stays fixed, which is the half worth keeping. #162 closes as superseded rather than both being merged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two items were rejected today for artefacts already found, fixed and closed —
in the other copy of the prompt. Found running session mode against a second
repository (#56).
#167 — the rubric was duplicated
executor.pyandsession_executor.pyeach carried aREVIEW_PROMPT. Everycorrection made from measurement landed in the headless one:
So R4 added a paragraph to a diagnostics panel and was rejected on:
That is precisely what #156 documented and closed — a diff cannot show its
surroundings, and the fix was to supply them. R3's rejection carried the same
shape.
Both rejections also contained a real point — R3 widened a repository
trait's return type, R4's wording claimed everything absent from the bundle was
deliberate. The reviews were not wrong. Half of each objection was a solved
problem in the wrong file.
The rubric is now imported rather than copied, and
review_context/review_checks_promptare module-level functions both executors call.Two tests guard it: the prompts must be the same object, because "roughly
the same" is what lets drift back; and
_reviewmust actually pass the fields,because identical prompts with empty fields would keep the reviewers unequal
while passing the first test.
#166 — a session retry was blind
The agent's prompt carried the title, the brief and the checks. Not the
reviewer's verdict, not the last failure. R3's retry would have been sent the
identical brief with no mention of the specific criticism it had just received.
A session attempt is minutes of a real agent rather than one API call, so
repeating one blind is the most expensive avoidable thing the harness does.
Not resumption: the item is re-read from the current brief, no prior diff
is fed back, and a guided attempt consumes an attempt exactly as an unguided
one does. A first attempt reads exactly as before — asserted.
One thing the tests did not catch
The first version of this borrowed
Executor._review_contextas an unboundmethod on a
SessionExecutor, which has nocontext_policyand would haveraised at runtime. The suite passed anyway, because the test exercising that
path lives on another branch. Extracting real shared functions removes the
class of mistake rather than the instance.
All four gates green.
🤖 Generated with Claude Code