Repository navigation
fix: preserve native references through scan and reload - #124
Open
YalcinkayaE wants to merge 2 commits into
Open
YalcinkayaE wants to merge 2 commits into
YalcinkayaE wants to merge 2 commits into
Conversation
alexandraabbas
left a comment
Collaborator
There was a problem hiding this comment.
Review posted on behalf of @alexandraabbas (written with Claude Code).
Thanks, this is a sensible stopgap for #77 and the scope statement is clear: unbatched results only, no viewer links, no citation prompts. CI is approved and green on your branches. Points before merge:
Must fix
- Loader-item citations point at ids that exist nowhere.
digest_turns(and the per-call loader in the add-a-layer skill) build each item'sChatMessageUserwithout an id, so Scout falls back to an auto-generated one. A judge answer of "see [M1]" on a digest turn is then stored as amessagereference whose id matches no transcript message and changes on every scan.turns_frameexposes it with no marker, which is the opposite of the location #77 asks for. Either drop references for items whose messages carry no source id, or state in thecohort_llm_scannerdocstring, the skill and the README that references resolve to transcript messages only formessages="all"scans or a loader that preserves source ids. The skill paragraph currently steers authors toward the per-turn loader case, which is exactly where the ids are meaningless. - A failed verifier review still contributes citations. In
_verify_judgement()the references are captured before thev_label is Nonecheck, so a review that ends withstatus="error"(label outside the vocabulary) still feeds its references into_merge_references(). The failed-verifier test only covers the exception path. Move the capture next to theokreturn. - Return the verifier's references instead of a
PrivateAttr._verify_judgement()writesresult._referencesfrom outside the model and both callers read it back. Returntuple[VerifierReview, list[Reference]]and keepVerifierReviewas the stored record only. Fixing this also fixes point 2 naturally. - Browser check. A
Table()block with nocolumnstakes every frame column, so every unbatched judged layer table now carries two list-of-struct columns into inspect-viz. Python conversion passing says nothing about how DuckDB-WASM renders them (AGENTS.md, "Verify report changes in a real browser"). Either exclude the reference columns from the Table default or check them in the $0 demo-log loop and say so in the PR.
Smaller points
_item_rowyieldsNonewhen the results table has nomessage_referencescolumn, which is indistinguishable from "the judge cited nothing" ([]). Read the column directly so an older store fails loudly, per "Honest absence".- The
json.loads(...) if isinstance(references, str)branch exists because Scout's results table serialises references as JSON text while an in-processResultcarries a list. Say so in a one-line comment. - In the batch path
_vote_and_verify()now collects per-unit verifier references that the batchResultnever writes. Keep references out of_UnitDecisionfor that path, or state the limit at the batchResultconstruction.
Keep it smaller
- Seven tests in 175 lines for a pass-through is more than it needs.
test_reference_identity_includes_typeunit-tests a ten-line helper the k-roll and cohort cases already exercise, andtest_absent_or_unknown_citations_do_not_create_referencestests Scout's own reference resolution. The regime-parametrized test, the verifier test (with the failed-review and error-status cases as parameters) and the store-and-reload test cover the change. - The README paragraph, the skill paragraph and the
cohort_llm_scannerdocstring each state the same four facts. Keep the full statement in the scanner docstring and reduce the other two to a sentence that points there, plus the source-id caveat from point 1.
Version
0.7.1 collides with #105 and #117 as your description notes. We will take #117 first; the others re-bump after rebasing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unbatched
cohort_llm_scannerresults now retain Scout's native message and event citations through scan, storage and reload. Previously, the solo, cohort and k-roll wrappers created new results without their references.turns_framenow exposes the retained citations asmessage_referencesandevent_references.This is the narrow reference-preservation stopgap for #77. It does not implement finding-level matching, citation prompts or viewer links.
Current implementation
The current head,
54607ca, includes the revision addressing the maintainer's review:VerifierReview; no private reference attribute is used.messages="all"or a custom loader that preserves source message IDs. Generated loader-item IDs are not claimed to identify original transcript messages.Validation and hosted status
Local validation recorded for
54607caincludes the fullmake checksuite with UI dependencies installed, locked-environment checks and package build. Regression coverage exercises voting regimes, successful and failed verifier reviews, and scan/store/reload. A Chromium check covers the default report table, including null and empty metadata.These are local results. The current-head CI run still concludes
action_requiredand has no executed jobs; maintainer workflow approval and hosted verification remain pending.Tests use scripted judges; no paid judge-model calls were made.
Version and merge order
This branch bumps Transect from 0.7.0 to 0.7.1; the lockfile change is limited to Transect's package-version line. As requested in review, #117 is first in the merge order. Once it merges, rebase this PR onto the updated main and apply the next patch version. #105 needs its own corresponding rebase and version update.