Skip to content

A finding is a hypothesis at the evaluator-fed fix stages #167

Description

@aaddrick

Depends on #162.

Problem

The two contrarian revision stages tell the model to verify a challenger's claim before acting on it:

  • :4262-4264 "A challenger finding is a HYPOTHESIS, not a directive: verify each against the actual tree/state first. Adopt it if confirmed; REBUT it with concrete evidence if wrong. Do not capitulate to an unverified claim, gates oscillate precisely when a wrong Major is adopted without verification."
  • :4403-4405 the same, plus a requirement to cite new evidence when overturning an adjudicated decision.

None of the fix stages carry it. They open with "Address code review feedback" and paste the reviewer's text.

There is also no way for a fixer to disagree. FIX_SCHEMA at :641 has status, commit, files_changed, fixes_applied, summary, error, notes_for_downstream. A fixer that believes a finding is wrong has only two moves: comply, or return status: 'error' and fail the issue.

This matters because reversals are majority-beneficial. In one decomposition of 58.19% "sycophantic" behaviour, 43.52% was movement toward the correct answer and only 14.66% away from it. A blanket never-comply rule would discard the majority that were corrections. The correct shape is evidence-gated, which is what settledBlock at :1861-1862 already asks for at the gates.

Scope: three stages, not five

Apply to the evaluator-fed fix stages, where the input is a judge's opinion:

  • quality-fix :2957, fed rev.comments from a code-reviewer agent
  • test-quality-fix :3479, fed v.comments from a test-validator agent
  • pr-fix :4675, fed spec.comments and code.comments

Do not apply to the oracle-fed stages:

  • test-fix :3430, fed t.failures from a real suite run
  • browser-fix :3146, fed bw.failures from a real browser run

Both already carry an explicit anti-rebuttal guard ("Fix the real defect, do NOT delete or weaken assertions just to make the failure disappear"). Telling those fixers a failing test is a hypothesis inverts a correct guard. Grounding a gate in something checkable removes oscillation rather than reducing it, and a real failure is an absorbing boundary with nothing to negotiate.

Change

  1. Lift the verify-then-adopt-or-rebut framing into a shared constant beside HANDOFF_ASK (:1867) and COMMIT_SHA_ASK (:1872), and use it at the three evaluator-fed stages only.
  2. Add rebutted: [{ finding_id, evidence }] to FIX_SCHEMA, referencing the finding ids the dependency issue introduces.
  3. Ship a deterministic consumer in the same issue. A prompt rule guarding a prompt rule is exactly the pattern docs/architecture/metrics.md records code review rejecting once already, and a real guardrail has to be deterministic. The rule: a fix returning rebutted.length > 0 && (fixes_applied || []).length === 0 records its own disposition, and on the final iteration of its gate forces carried-unresolved plus a Verification Gaps line rather than allowing an approval.

Prompt text, schema field, and JS rule ship together so the change reverts as a unit.

Explicitly not doing

The original draft also proposed stripping reviewer attribution ('Spec review:' / 'Code review:') from the text handed to fixers. That is dropped. The relevant finding endorses external attribution as one of its two acceptable options ("attribute externally, or strip attribution entirely"), what :4680-4681 does is external attribution, and no fix prompt contains authority framing today. Removing it would cost the fixer the ability to tell two reviewers apart for no measured benefit.

Acceptance criteria

  • The three evaluator-fed stages carry the framing. The two oracle-fed stages do not, and their anti-rebuttal guards are untouched.
  • rebutted entries require cited evidence; an entry with an empty evidence is treated as no rebuttal.
  • The JS rule is unit-tested above the split marker.
  • A fix stage that rebuts every finding and applies none terminates its gate as carried-unresolved with a Verification Gaps line, never as approved.
  • A fix stage that rebuts some findings and fixes the rest proceeds normally.
  • A fix stage that returns no rebutted field behaves exactly as it does today.
  • node --test stays green. Extend tests/quality-loop.test.js, tests/pr-review-gate.test.js, tests/test-loop.test.js.

Documentation target

Document in CHANGELOG.md and in docs/architecture/gate-hygiene.md, adding its row to docs/architecture/index.md's file map if not already present.

Do NOT edit docs/architecture/pipeline.md, metrics.md, or failure-semantics.md. Their moved prose is hash-frozen by tests/architecture-provenance.test.js, and since the tech-docs stage runs at :4707, after the last test-loop execution, a frozen-prose edit surfaces only as a red CI check on the batch PR. New pages are authored text and are explicitly allowed; see docs/architecture/CLAUDE.md.

Metadata

Metadata

Assignees

Labels

gate-hygieneGate hygiene: findings, caps, and per-problem budgets (sequenced)

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions