Skip to content

feat: run index for a sample scanned over several epochs - #121

Merged
alexandraabbas merged 7 commits into
mainfrom
report/index
Nov 5, 2026
Merged

alexandraabbas merged 7 commits into
mainfrom
report/index

Conversation

@alexandraabbas

Copy link
Copy Markdown
Collaborator

Takeover of #100 (Kozzy's run index), rebased onto main and reduced to what the package can stand behind. Closes #100 as superseded once merged.

What changes

  • One report per epoch plus a run index. When a sample was scanned over several epochs, render() writes report_epoch1.html, report_epoch2.html, ... plus report_index.html, first in report_paths and opened by default; a single epoch renders one report as before. Each epoch report links back to the index. The index is rendered by report/render.py (render_index), reads no transcripts and loads no chart runtime.
  • Reports table: each epoch's report with its recorded execution status, Errors recorded when any scanner recorded an error for that transcript (from scan_status.errors), else Ran.
  • Collection reliability (report/collection.py): each judged surface the scan ran (Phases, Sub-agents, custom layers declaring an audit), pooled over the epochs with one row per recorded judge setup and label vocabulary: valid ballots against the setup's expected count, the frames' own per-unit judge_agreement averaged, pairwise agreement, α and AC1 from the original ballots with transcript-scoped unit keys, verifier relabels and spot-check overturns. A single judge gets a row saying there is no agreement to measure; a surface whose frames break their contract gets a row naming the breach; epochs without a recorded setup pool into one "no judged units" row.
  • Run-wide scan status repeated on the index.
  • Version 0.7.0; the committed fixture stores regenerated at that version.

What is deliberately not carried over from #100

  • reliability.equal_sample_agreement: transect() scans one sample per invocation, so the statistic always equalled the plain mean.
  • The eligibility re-count that nulled judge_agreement where it counted fewer than two ballots: the frames already stamp that decision, and the re-count made the index disagree with the per-epoch audits.
  • The run001_epoch2 file names and #transcript-N anchors: every file holds one epoch.
  • The cross-run narratives recipe and its pointers in README, AGENTS.md and the skills: a workflow nothing in the package implements.

Validation

Rendered on the three-epoch fixture with a two-roll gpt-5-mini judge and verifier (table with numbers), the two-epoch Petri demo with a solo judge (single-judge row), and single-transcript scans (no index); links and back links checked in Chromium. make check passes (674 tests).

Review

Two no-context reviews of the original PR and one of the takeover; confirmed findings applied (README sections lost in a trim restored, no-regime epochs pooled, Sub-agents units limited to classified lanes, vocabulary keyed as a set, verifier counts from relabel_rate, NaN epochs refused, docstrings and legend wording).

🤖 Generated with Claude Code

https://claude.ai/code/session_01TJKrL8rQH1dN8B49eALQZn

kvoudouris-aisi and others added 7 commits November 4, 2026 16:43
Takeover of the contributor's index: one report per epoch plus
report_index.html first in report_paths, with each epoch's recorded
execution status, the judged surfaces' reliability pooled per recorded
judge setup and vocabulary from the frames' own agreement and the
original ballots, and the run-wide scan status; a single judge says it
has no agreement to measure; a surface that breaks its frame contract
gets a row naming the breach. Rendered from the report orchestrator,
files named by epoch, links by file. The balanced statistic and the
cross-run narrative recipe are not carried. Version 0.7.0, fixture
stores regenerated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TJKrL8rQH1dN8B49eALQZn
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TJKrL8rQH1dN8B49eALQZn
Epochs with judged units but no recorded judge setup get a row saying
so; a non-voting row carries no agreement numbers; the units the judge
never saw share one constant with the per-epoch audit; column guards
name the frames' contract; the narrow-column rule targets alpha and
AC1; copy says "no data" as everywhere else; README and the report
guide name both verifier columns. Tests merged: denominators over
surfaces and setups in one parametrized test, the errored epoch as one
case, the quoted file links pinned through a stem with a space, the
single-judge pipeline run and the markup pins dropped.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TJKrL8rQH1dN8B49eALQZn
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TJKrL8rQH1dN8B49eALQZn
@alexandraabbas
alexandraabbas marked this pull request as ready for review November 5, 2026 09:39
@alexandraabbas
alexandraabbas merged commit 2655747 into main Nov 5, 2026
3 checks passed
@alexandraabbas
alexandraabbas deleted the report/index branch November 5, 2026 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants