Skip to content

fix: preserve native references through scan and reload - #124

Open
YalcinkayaE wants to merge 2 commits into
AI-Safety-Institute:mainfrom
YalcinkayaE:codex/preserve-message-references
Open

YalcinkayaE wants to merge 2 commits into
AI-Safety-Institute:mainfrom
YalcinkayaE:codex/preserve-message-references

Conversation

@YalcinkayaE

@YalcinkayaE YalcinkayaE commented Nov 5, 2026 •

Copy link
Copy Markdown

Unbatched cohort_llm_scanner results now retain Scout's native message and event citations through scan, storage and reload. Previously, the solo, cohort and k-roll wrappers created new results without their references. turns_frame now exposes the retained citations as message_references and event_references.

This is the narrow reference-preservation stopgap for #77. It does not implement finding-level matching, citation prompts or viewer links.

Current implementation

The current head, 54607ca, includes the revision addressing the maintainer's review:

  • Preserve and deduplicate first-round citations across unbatched voting regimes, including dissenting members. Add verifier citations only when the review succeeds; invalid labels and verifier exceptions add none.
  • Return verifier references separately from the stored VerifierReview; no private reference attribute is used.
  • Document that message references resolve to transcript messages with messages="all" or a custom loader that preserves source message IDs. Generated loader-item IDs are not claimed to identify original transcript messages.
  • Read reference columns directly and decode Scout's persisted JSON representation. Empty citations remain empty lists; a missing column fails explicitly.
  • Exclude the two reference columns from default report-table projection. They remain available on layer frames and can be selected explicitly. Batched results do not retain per-unit citations.

Validation and hosted status

Local validation recorded for 54607ca includes the full make check suite with UI dependencies installed, locked-environment checks and package build. Regression coverage exercises voting regimes, successful and failed verifier reviews, and scan/store/reload. A Chromium check covers the default report table, including null and empty metadata.

These are local results. The current-head CI run still concludes action_required and has no executed jobs; maintainer workflow approval and hosted verification remain pending.

Tests use scripted judges; no paid judge-model calls were made.

Version and merge order

This branch bumps Transect from 0.7.0 to 0.7.1; the lockfile change is limited to Transect's package-version line. As requested in review, #117 is first in the merge order. Once it merges, rebase this PR onto the updated main and apply the next patch version. #105 needs its own corresponding rebase and version update.

@alexandraabbas alexandraabbas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review posted on behalf of @alexandraabbas (written with Claude Code).

Thanks, this is a sensible stopgap for #77 and the scope statement is clear: unbatched results only, no viewer links, no citation prompts. CI is approved and green on your branches. Points before merge:

Must fix

  1. Loader-item citations point at ids that exist nowhere. digest_turns (and the per-call loader in the add-a-layer skill) build each item's ChatMessageUser without an id, so Scout falls back to an auto-generated one. A judge answer of "see [M1]" on a digest turn is then stored as a message reference whose id matches no transcript message and changes on every scan. turns_frame exposes it with no marker, which is the opposite of the location #77 asks for. Either drop references for items whose messages carry no source id, or state in the cohort_llm_scanner docstring, the skill and the README that references resolve to transcript messages only for messages="all" scans or a loader that preserves source ids. The skill paragraph currently steers authors toward the per-turn loader case, which is exactly where the ids are meaningless.
  2. A failed verifier review still contributes citations. In _verify_judgement() the references are captured before the v_label is None check, so a review that ends with status="error" (label outside the vocabulary) still feeds its references into _merge_references(). The failed-verifier test only covers the exception path. Move the capture next to the ok return.
  3. Return the verifier's references instead of a PrivateAttr. _verify_judgement() writes result._references from outside the model and both callers read it back. Return tuple[VerifierReview, list[Reference]] and keep VerifierReview as the stored record only. Fixing this also fixes point 2 naturally.
  4. Browser check. A Table() block with no columns takes every frame column, so every unbatched judged layer table now carries two list-of-struct columns into inspect-viz. Python conversion passing says nothing about how DuckDB-WASM renders them (AGENTS.md, "Verify report changes in a real browser"). Either exclude the reference columns from the Table default or check them in the $0 demo-log loop and say so in the PR.

Smaller points

  • _item_row yields None when the results table has no message_references column, which is indistinguishable from "the judge cited nothing" ([]). Read the column directly so an older store fails loudly, per "Honest absence".
  • The json.loads(...) if isinstance(references, str) branch exists because Scout's results table serialises references as JSON text while an in-process Result carries a list. Say so in a one-line comment.
  • In the batch path _vote_and_verify() now collects per-unit verifier references that the batch Result never writes. Keep references out of _UnitDecision for that path, or state the limit at the batch Result construction.

Keep it smaller

  • Seven tests in 175 lines for a pass-through is more than it needs. test_reference_identity_includes_type unit-tests a ten-line helper the k-roll and cohort cases already exercise, and test_absent_or_unknown_citations_do_not_create_references tests Scout's own reference resolution. The regime-parametrized test, the verifier test (with the failed-review and error-status cases as parameters) and the store-and-reload test cover the change.
  • The README paragraph, the skill paragraph and the cohort_llm_scanner docstring each state the same four facts. Keep the full statement in the scanner docstring and reduce the other two to a sentence that points there, plus the source-id caveat from point 1.

Version

0.7.1 collides with #105 and #117 as your description notes. We will take #117 first; the others re-bump after rebasing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants