You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: add report index and collection reliability - #100
When an evaluation contains multiple epochs, it can be useful to have an overview index page that collates and summarises them, including summaries of the collection-wide scanner reliabilities. This PR does the following:
Renders multiple selected transcripts as individual run/epoch reports with a separate index and back links. Open the index by default; keep single-transcript output unchanged.
Shows report links and scan status, then collection reliability grouped by scanner, judge setup and vocabulary, followed by execution coverage. Include pooled ballot agreement, equal-sample agreement and original verifier outcomes, with agreement colours and folded column explanations.
Implements agent advice on how to run cross-epoch narrative analysis so that this is more robust and reliable if users want to understand qualitatively how different epochs play out.
Superseded by #121 (merged as 2655747, version 0.7.0), which carries this PR's run index on the rebased main: one report per epoch plus report_index.html first in report_paths, the epoch reports with their recorded execution status, the collection reliability pooled per judge setup and vocabulary (from the frames' own judge_agreement and the original ballots with transcript-scoped unit keys, verifier relabels and spot-check as two columns), and the run-wide scan status. Not carried over, by design: equal_sample_agreement (always equal to the plain mean under one-sample scans), the eligibility re-count (the frames already stamp it), the position-based file names and anchors, and the cross-run narratives recipe. Thank you for the groundwork, Kozzy; the design notes and renders are on #121.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When an evaluation contains multiple epochs, it can be useful to have an overview index page that collates and summarises them, including summaries of the collection-wide scanner reliabilities. This PR does the following: