Skip to content

feat: Phase C C7 family-stratified precision@k report -- detects family inflation in benchmark top-k - #889

Open
cschanhniem wants to merge 1 commit into
mainfrom
feat/c7-family-stratified-report
Open

cschanhniem wants to merge 1 commit into
mainfrom
feat/c7-family-stratified-report

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Implements C7 from NEXT_100_PR_MAP: family-stratified precision@k.

What

  • family_stratified_report.py: compute_family_stratified_report() + format_family_stratified_report() + FamilyStratifiedReport
  • Detects when a single AMP family dominates the top-k ranked sequences (>60% dominance threshold)
  • Per-family precision@k computed for each family in the top-k window
  • family_inflation_likely=True when dominant family fraction >= 0.60 AND >= 0.50 overall threshold
  • 63 tests; BASELINE 7150 to 7213

Why

A model that learns to rank one family highly (e.g. all defensins) will appear to have good precision@k while actually just fitting within-family patterns. This report makes that family-inflation artifact visible.

Disconfirming pass

  • Report is evidence only, does not block ranking.
  • References BMC-0004 (family-stratified split) for publication-grade testing.
  • No safety/release policy touched.

cschanhniem added a commit that referenced this pull request Jul 10, 2026
…ype by requiring all declared enemies to be beaten (#889) (#890)

Co-authored-by: OpenCode <opencode@example.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant