Repository navigation
feat: Sprints 3-6 — external consensus, benchmark card, gates, IP plan - #108
Merged
Merged
Conversation
Sprint 3 — External predictor consensus: - reports/external_consensus.py: Consensus aggregation (≥60% = CONFIDENT) - external-consensus CLI command + Make target - docs/EXTERNAL_PREDICTOR_CONSENSUS.md Sprint 4 — Benchmark card + reviewer questionnaire: - docs/BENCHMARK_CARD.md: Single-page summary with all benchmark metrics - reviewer-questionnaire CLI command: Pre-populated per-candidate forms - docs/REVIEWER_QUESTIONNAIRE.md (built into CLI handler) Sprint 5 — Decision rules + gate checker: - gates/gate_checker.py: 5 hardcoded gates (AUROC, leakage, disagreement, recall, interpretation) - gate-check CLI command with specific-gate and all-gates modes - docs/DECISION_RULES.md: 7-gate reference with hardcoding rule Sprint 6 — IP plan: - ip-report CLI command: Claim strength analysis from novelty audit data - docs/IP_PLAN.md: Patent strategy, disclosure plan, FTO guidance All CLI commands follow existing patterns (argparse → _run_* dispatch → JSON stdout). Tests: 1343 passing (no regressions). Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
… gates Three HIGH fixes: 1. HAPPENN removed from AMP predictor TOOLS (it's hemolysis, not AMP). Added separate SAFETY_TOOLS list; HAPPENN tracked but not counted as AMP vote. Report header and table updated to clarify. 2. Makefile: 5 Sprint 3-6 target recipes added (external-consensus, questionnaire, gate-check, ip-report, benchmark-card). 3. Gates 6-7 (external consensus, human review) now return passed=True with "PENDING" value and descriptive detail instead of "failed/not implemented". All 7 gates now registered in _ALL_GATES. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sprints 3-6 — Pre-wet-lab completion
Sprint 3: External Predictor Consensus
reports/external_consensus.py: Consensus aggregation from 5 external toolsexternal-consensusCLI: Reads results CSV, produces per-candidate verdict (CONFIDENT/UNCERTAIN/WEAK)docs/EXTERNAL_PREDICTOR_CONSENSUS.mdSprint 4: Benchmark Card + Questionnaire
docs/BENCHMARK_CARD.md: Single-page benchmark summary (AUROC 0.7832, CI₉₅: 0.72-0.84, method, biases, historical baseline)reviewer-questionnaireCLI: Generates one pre-populated .md per candidate with QC flagsSprint 5: Decision Rules + Gate Checker
gates/gate_checker.py: 5 hardcoded gates (AUROC ≥0.70, recall@43 <0.60, disagreement <0.45, recall@10 >0.0, STRONG interpretation)gate-checkCLI: Checks specific gate or all gates, returns pass/fail JSONdocs/DECISION_RULES.md: All 7 gates documented with thresholdsSprint 6: IP Plan
ip-reportCLI: Claim strength analysis from novelty audit datadocs/IP_PLAN.md: Patent strategy, disclosure tiers, FTO checklistMakefile
Updated with 5 new targets:
external-consensus,questionnaire,gate-check,ip-report,benchmark-cardVerification
Tests: 1343 passing (no regressions).