Skip to content

fix: preserve confidence spread after verifier phase merges - #104

Open
YMuskrat wants to merge 4 commits into
AI-Safety-Institute:mainfrom
YMuskrat:fix/verifier-confidence-spread
Open

YMuskrat wants to merge 4 commits into
AI-Safety-Institute:mainfrom
YMuskrat:fix/verifier-confidence-spread

Conversation

@YMuskrat

@YMuskrat YMuskrat commented Oct 27, 2026 •

Copy link
Copy Markdown

A verifier correction can merge neighbouring phases whose confidence values still differ. In that case, the exported spread was always zero. I reproduced this with final confidences [0.95, 0.60, 0.95]: the expected spread is 0.229, but the phases frame contained 0.

In _apply_verdicts() in src/transect/scanners/phases_verify.py, stitch_phases() already rebuilds the merged phases and recomputes their statistics after a relabel. The final if p.verifier.overturned block then overwrote confidence_spread with 0.0. decision_phases() stored that value, and phases_df() exported it unchanged in frames["phases"].confidence_spread.

Removing that assignment preserves the calculation from the final turn confidences. Uniform confidence values still naturally produce zero spread.

Five regression tests cover merged and separate phases, including the public scanner and exported frame. Three fail before the fix. The unit suite passes (560 tests), along with Ruff, mypy, Pyright and package builds. Seven browser tests were skipped because Chromium could not run here. Local tests disabled only Scout’s progress memory monitor because /proc is unavailable.

Related to #101.

@alexandraabbas alexandraabbas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review posted on behalf of @alexandraabbas (written with Claude Code).

Thanks, the diagnosis is right: stitch_phases() already recomputes the spread from the final per-turn confidences after a relabel, and the confidence_spread = 0.0 override in _apply_verdicts() discards that for a merged phase. Removing the line is the correct fix. A few things before this can go in.

Must fix

  1. Rebase and re-bump. The branch bumps 0.4.4 to 0.4.5, but main is at 0.7.0. Rebase onto main and bump to 0.7.1 (patch) in pyproject.toml and uv.lock.
  2. Keep the docs true. The column contract in src/transect/frames/phases.py (the confidence_spread entry) was not touched. That docstring is the frame contract per AGENTS.md, so state the new semantics there, not only in the scanner docstring.
  3. Say what the spread means for a mixed-source phase. After this change a phase with confidence_source = "verifier" can carry a spread computed over two judge confidences and one verifier confidence. The per-turn rows for overturned turns are stamped confidence_pm = 0.0, so the two surfaces now describe the overturned turn differently. Either document plainly that the phase spread is over the final per-turn confidences whatever their source (and that confidence_source names the overturning verdict, not the whole sample), or restrict the spread to rows matching confidence_source. We prefer the first (document it) since it is the smaller change, but the contract has to say it.
  4. The StitchedPhase field description overstates. "after any verifier updates and display merging" is not what happens: a weak relabel or a non-overturning review never re-stitches (applied_any stays False). Word it as "recomputed from the final per-turn confidences when an applied overturn re-stitches the phases".

Trim the tests

The suite target is a small, heavily parametrized set (see AGENTS.md). Five new tests for a one-line removal is more than it needs:

  • Keep the parametrized _apply_verdicts test; it pins the behaviour well.
  • Drop test_merged_confidence_spread_survives_scanning_and_frame_projection, or reduce it to a single assertion that phases_df passes the recomputed spread through. It re-runs the "merge-with-unreviewed" case end to end and re-asserts review-population facts (original_confidence, verifier_confidence, n_relabelled) that reviewed_frame() tests already cover in the same file.

Once those are in we are happy to merge.

@YMuskrat
YMuskrat force-pushed the fix/verifier-confidence-spread branch from 53d2540 to 2921737 Compare November 18, 2026 00:06
@YMuskrat

Copy link
Copy Markdown
Author

Thank you for the review and sorry for the late reply. I rebased on main and bumped to 0.7.1. I clarified that when an overturn restitches a phase, the spread uses all final turn confidences, including both judge and verifier values. The docs now explain that confidence_source="verifier" identifies the representative overturning verdict. I kept the parametrized regression test and removed the duplicate end-to-end test. make check passed locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants