Skip to content

feat: Phase O O2 prediction drift monitor schema -- earliest warning signal for pipeline score distribution shifts - #838

Merged
cschanhniem merged 2 commits into
mainfrom
feat/o2-prediction-drift-monitor
Jul 9, 2026
Merged

cschanhniem merged 2 commits into
mainfrom
feat/o2-prediction-drift-monitor

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Summary

  • PredictionDriftEntry: 15 fields tracking mean/std score shifts between reference and evaluation batches
  • 8 validation rules: monitor_id prefix DRM-, batch uniqueness, score bounds [0,1], std non-negative, shift magnitude consistency ±0.001, population ≥1, notes length ≤400, notes required when drift_flag=True
  • 4 warning conditions: unreported drift (shift ≥0.1 without flag), small populations (<10), variance explosion (eval_std > 2× ref_std)
  • fix: enforce uppercase sequence in CandidateSummaryCard validation (pre-existing test failure)
  • Phase O O2 complete

Test plan

  • make agent-check passes
  • All tests pass (1 pre-existing failure fixed: test_lowercase_sequence_fails)

@cschanhniem
cschanhniem merged commit 23e45f2 into main Jul 9, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant