Skip to content

feat: Phase F F8 failure mode similarity report -- FMS- schema benchmarks whether rejected candidates resemble known failure modes, pattern-repeat flag, calibration action trigger, 63 tests - #927

Merged
cschanhniem merged 1 commit into
mainfrom
feat/f8-failure-mode-similarity-report
Jul 10, 2026
Merged

cschanhniem merged 1 commit into
mainfrom
feat/f8-failure-mode-similarity-report

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Implements F8 from NEXT_100_PR_MAP: benchmark for rejected candidates resembling known failure modes.

What

  • src/openamp_foundry/evidence/failure_mode_similarity_report.py: FMS- prefix schema
  • FailureModeSimilarityReport dataclass: 14 fields
  • VALID_FAILURE_MODE_CATEGORIES (11 values): hemolysis_risk, low_selectivity, toxicity_flag, dual_use_concern, known_resistance_motif, charge_shortcut, length_outside_range, low_novelty, sequence_degeneracy, poor_charge_distribution, other
  • VALID_SIMILARITY_ASSESSMENT_STATUSES (5 values): low_similarity, moderate_similarity, high_similarity, pattern_repeated, insufficient_data
  • HIGH_SIMILARITY_THRESHOLD = 0.60, PATTERN_REPEATED_THRESHOLD = 0.80
  • 11 validation rules: prefix, failure mode category, similarity status, benchmark status, fraction range, fraction consistency check, min candidates gate, dry_lab_only, pattern_repeated_flag consistency, non-blank fields, n_with_failure_mode <= n_rejected
  • validate_failure_mode_similarity_report(), format_failure_mode_similarity_report()
  • tests/evidence/test_failure_mode_similarity_report.py: 63 tests, 5 classes
  • BASELINE: 8284 -> 8347

Why

A pipeline that repeatedly nominates candidates with the same failure-mode signature (hemolysis risk, dual-use concern, low novelty) is not learning from history. FMS- records surface this pattern machine-readably so calibration can correct it. The pattern_repeated_flag and calibration_action_recommended fields give calibration reviewers a machine-checkable gate.

Disconfirming pass

  • dry_lab_only enforced on all reports
  • fraction consistency check prevents manual override of statistics
  • pattern_repeated_flag consistency with PATTERN_REPEATED_THRESHOLD = 0.80 prevents under-reporting
  • min candidates gate prevents over-claiming from micro-batches

…arks whether rejected candidates resemble known failure modes, pattern-repeat flag, calibration action trigger, 63 tests (#926)
@cschanhniem
cschanhniem merged commit 19a7c2e into main Jul 10, 2026
1 check failed
cschanhniem added a commit that referenced this pull request Jul 10, 2026
…actual shows rank changes IF gate opened without applying them, results_applied_to_ranking enforcement, 63 tests (#927) (#929)

Co-authored-by: OpenCode <opencode@example.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant