Skip to content

feat: Phase C C4 similarity-neighbor distribution report -- novelty shortcut visibility for benchmark audits - #884

Merged
cschanhniem merged 1 commit into
mainfrom
feat/c4-similarity-neighbor-report
Jul 10, 2026
Merged

cschanhniem merged 1 commit into
mainfrom
feat/c4-similarity-neighbor-report

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Implements C4 from NEXT_100_PR_MAP: similarity-neighbor distribution report for novelty honesty.

What

  • src/openamp_foundry/evidence/similarity_neighbor_report.py: compute_similarity_report() + format_similarity_report() + SimilarityNeighborReport dataclass
  • SIMILARITY_SHORTCUT_THRESHOLD = 0.80 — fraction above which a sequence is considered high-similarity to references
  • SHORTCUT_FRACTION_THRESHOLD = 0.50 — if >50% of positives are high-similarity to known AMPs, novelty_shortcut_likely=True
  • Character-overlap similarity (positional match / max length); notice included that alignment tools are needed for publication-grade analysis
  • 63 tests in tests/evidence/test_similarity_neighbor_report.py
  • BASELINE: 6898 → 6961

Why

Benchmark improvements can be driven by the model learning that benchmark positives look like known AMPs in sequence space, not by discovering genuinely novel candidates. This report makes that shortcut visible and auditable.

Disconfirming pass

  • Similarity metric is positional only; noted in NOTICE. Does not replace alignment.
  • Report flags the shortcut; does not block anything — it is evidence, not policy.
  • No safety/release policy touched.

@cschanhniem
cschanhniem merged commit 6d900dc into main Jul 10, 2026
0 of 2 checks passed
cschanhniem added a commit that referenced this pull request Jul 10, 2026
…s for governed benchmark scripts (#884) (#885)

Co-authored-by: OpenCode <opencode@example.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant