Skip to content

feat: Phase H H6 simulation uncertainty calibration report -- SUC- schema tracks whether module confidence intervals are empirically calibrated, overconfidence flag, 63 tests - #923

Merged
cschanhniem merged 1 commit into
mainfrom
feat/h6-simulation-uncertainty-calibration-report
Jul 10, 2026
Merged

cschanhniem merged 1 commit into
mainfrom
feat/h6-simulation-uncertainty-calibration-report

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Implements H6 from NEXT_100_PR_MAP: simulation uncertainty calibration report.

What

  • src/openamp_foundry/simulation/uncertainty_calibration_report.py: SUC- prefix schema
  • SimulationUncertaintyCalibrationReport dataclass: 14 fields
  • VALID_SIMULATION_MODULES (7 values): membrane_proxy, structure_proxy, ensemble_checker, external_adapter, dummy_module, baseline_registry, other
  • VALID_CALIBRATION_METHODS (6 values): isotonic_regression, platt_scaling, temperature_scaling, histogram_binning, empirical_coverage_only, none_applied
  • VALID_UNCERTAINTY_ASSESSMENT_STATUSES (5 values): well_calibrated, overconfident, underconfident, insufficient_data, not_assessed
  • OVERCONFIDENCE_THRESHOLD = 0.15, MIN_SAMPLES_FOR_CALIBRATION = 10
  • 10 validation rules: prefix, module, method, status, coverage range, calibration_error consistency, insufficient_data gate, dry_lab_only, overconfidence_flag consistency, non-blank fields
  • validate_uncertainty_calibration_report(), format_uncertainty_calibration_report()
  • tests/simulation/test_uncertainty_calibration_report.py: 63 tests, 5 classes
  • BASELINE: 8221 -> 8284

Why

A simulation module that claims 80% confidence intervals should actually contain the true value 80% of the time. Without SUC- records, over-confident modules silently inflate trust in rankings. The overconfidence_flag and cheap_baseline_beats_module fields give calibration reviewers a machine-checkable gate before a module gets ranking authority.

Disconfirming pass

  • dry_lab_only enforced on all reports (cannot be set False)
  • insufficient_data status required when n_samples < 10 (prevents over-claiming calibration from small samples)
  • calibration_error must equal abs(expected - empirical) within 0.01 tolerance (prevents manual override of error magnitude)

…hema tracks whether module confidence intervals are empirically calibrated, overconfidence flag, 63 tests (#923)
@cschanhniem
cschanhniem merged commit 2cb9cec into main Jul 10, 2026
1 check failed
cschanhniem added a commit that referenced this pull request Jul 10, 2026
…hema tracks whether module confidence intervals are empirically calibrated, overconfidence flag, 63 tests (#923) (#924)

Co-authored-by: OpenCode <opencode@example.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant