Skip to content

feat: add report index and collection reliability - #100

Closed
kvoudouris-aisi wants to merge 3 commits into
mainfrom
kvoudouris/index
Closed

kvoudouris-aisi wants to merge 3 commits into
mainfrom
kvoudouris/index

Conversation

@kvoudouris-aisi

Copy link
Copy Markdown
Collaborator

When an evaluation contains multiple epochs, it can be useful to have an overview index page that collates and summarises them, including summaries of the collection-wide scanner reliabilities. This PR does the following:

  • Renders multiple selected transcripts as individual run/epoch reports with a separate index and back links. Open the index by default; keep single-transcript output unchanged.
  • Shows report links and scan status, then collection reliability grouped by scanner, judge setup and vocabulary, followed by execution coverage. Include pooled ballot agreement, equal-sample agreement and original verifier outcomes, with agreement colours and folded column explanations.
  • Implements agent advice on how to run cross-epoch narrative analysis so that this is more robust and reliable if users want to understand qualitatively how different epochs play out.

@kvoudouris-aisi

Copy link
Copy Markdown
Collaborator Author

CC @domdathan-aisi

@alexandraabbas

Copy link
Copy Markdown
Collaborator

Superseded by #121 (merged as 2655747, version 0.7.0), which carries this PR's run index on the rebased main: one report per epoch plus report_index.html first in report_paths, the epoch reports with their recorded execution status, the collection reliability pooled per judge setup and vocabulary (from the frames' own judge_agreement and the original ballots with transcript-scoped unit keys, verifier relabels and spot-check as two columns), and the run-wide scan status. Not carried over, by design: equal_sample_agreement (always equal to the plain mean under one-sample scans), the eligibility re-count (the frames already stamp it), the position-based file names and anchors, and the cross-run narratives recipe. Thank you for the groundwork, Kozzy; the design notes and renders are on #121.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants