Repository navigation
feat(calibration): add lab-result intake module + synthetic example data (v0.5.19) - #133
Merged
Merged
Conversation
…ata (v0.5.19) Lore: add calibration intake module to join pilot panel predictions with lab result actuals. Honest minimum-cohort-size gate prevents small-sample theater. Synthetic example data clearly labeled SYNTHETIC in every file. Co-authored-by: OmX <omx@openamp.local>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The wet-lab feedback loop had lab-result ingestion (
data/lab_results.py,lab-result-reportCLI) but no way to JOIN pilot panel predictions with lab-result actuals. Without that join, any future recalibration has no intake valve and reviewers cannot see prediction-vs-actual alignment.This PR adds the missing join layer.
Changes
New module:
openamp_foundry.calibration.intakebuild_calibration_intake_report(panel_csv, results_dir)joins a pilot panel CSV (computational predictions) with a directory of validated lab result JSON files (experimental actuals) oncandidate_id.write_calibration_intake_json/write_calibration_intake_markdownproduce machine-readable and human-readable review artifacts.MIN_COHORT_SIZE=5): aggregate cohort metrics are NOT reported whenn < 5. Below the gate the metric is markedinsufficient_data: Trueand no point estimate is produced. This prevents small-sample theater.activity_vs_active_mic(MIC ≤ 32 µg/mL active) andrich_selectivity_vs_high_hemolysis(hemolysis ≥ 10% high-risk).New CLI:
openamp-foundry calibration-intakeMakefile targets
make lab-result-intake PANEL=<panel.csv> RESULTS_DIR=<dir>(real data)make lab-result-intake-example(synthetic demo, no args needed)Synthetic example data (clearly labeled SYNTHETIC)
examples/lab_results/RES-SYN-001..005.json— 5 synthetic JSONs covering active hit, inactive candidate, low hemolysis, high hemolysis, control failureexamples/lab_results_panel.csv— 8-candidate synthetic pilot panelexamples/lab_results/README.md— prominent SYNTHETIC disclaimerEvery JSON file's
disclaimerfield explicitly states it is a synthetic example. The README repeats the warning prominently.Tests
29 new tests in
tests/test_calibration_intake.py. Total: 1614 passing (was 1585). Coverage of:MIN_COHORT_SIZE)rich_selectivityabove threshold means LOWER predicted hemolysis)Documentation updates
docs/ROADMAP.md— new v0.5.19 entrydocs/METRICS_CURRENT.md— header updated, change log entry addedAGENTS.md— Priority 7 marked as having v0.5.19 progressCHANGELOG.md— v0.5.19 section addedMakefile— test count updated from 1287 to 1614Honest Limitations (documented in module docstring, CLI help, README)
MIN_COHORT_SIZE=5. Below the gate, no point estimate is reported. This prevents small-sample theater where 2-3 MIC readouts could swing an "AUROC" by 0.30.docs/DECISION_RULES.md,docs/WAVE2_PLAN.md).Verification
$ make lab-result-intake-example { "status": "ok", "n_panel_candidates": 8, "n_lab_results": 5, "n_matched_candidates": 4, "n_orphan_lab_results": 0, "cohort_metrics": { "activity_vs_active_mic": {"n": 3, "insufficient_data": true}, "rich_selectivity_vs_high_hemolysis": {"n": 2, "insufficient_data": true} } } $ python -m pytest tests/ -q 1614 passed, 1 skipped in 28.55s $ ruff check src/openamp_foundry/calibration/ tests/test_calibration_intake.py All checks passed!The cohort metrics correctly report
insufficient_data: truebecause the synthetic example only has 3 MIC results and 2 hemolysis results (both below the 5-sample gate). This is exactly the honest behavior we want — a single MIC readout cannot swing an aggregate metric.Future loop benefit
examples/lab_results/directory with real JSON is the ONLY change needed to get a real prediction-vs-actual report.