Skip to content

fix(calibration): report calibration and evaluation period metrics separately#5

Merged
DarriEy merged 1 commit into
mainfrom
fix/final-evaluation-period-metrics
Jul 19, 2026
Merged

fix(calibration): report calibration and evaluation period metrics separately#5
DarriEy merged 1 commit into
mainfrom
fix/final-evaluation-period-metrics

Conversation

@DarriEy

@DarriEy DarriEy commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

Problem

run_final_evaluation returned only final_metrics, so the results saver's final_result.get('calibration_metrics', {}) wrote empty dicts. Every HEC-HMS run persisted:

"calibration_metrics": {},
"evaluation_metrics": {},

…even though the log showed a computed KGE. The scores were only recoverable from best_score in the separate best-params file.

The metrics were also computed over the whole post-warmup series as a single block rather than per period, so the evaluation period was never scored at all. Every run logged EVALUATION PERIOD: No evaluation period configured despite the config setting one — silently removing HEC-HMS from any calibration-vs-evaluation generalisation comparison.

Found while reproducing the SYMFLUENCE Paper 3 experiments, where all 17 HEC-HMS calibration cells had empty metrics.

Fix

Split the periods the way the sibling HBV plugin already does, reusing its warmup semantics:

  • drop warmup from the full simulation first, then filter to the period — matching how each calibration iteration was scored (run 2002–2009 → drop 365 warmup days → filter to 2004–2007)
  • the evaluation period already starts after warmup, so it takes no further skip
  • return calibration_metrics and evaluation_metrics alongside final_metrics, keeping unprefixed aliases in final_metrics for existing readers

Verification

Re-ran a HEC-HMS calibration on Bow at Banff with the patched package:

before after
calibration_metrics {} 17 metrics, Calib_KGE 0.851
evaluation_metrics {} 17 metrics, Eval_KGE 0.658

Calib_KGE 0.851 matches the optimiser's own best_score for that run — an independent check that the calibration-period split is correct. Eval_KGE is information that previously did not exist.

Existing test suite passes (3 passed); ruff clean.

🤖 Generated with Claude Code

…parately

`run_final_evaluation` returned only `final_metrics`, so the results saver's
`final_result.get('calibration_metrics', {})` wrote empty dicts and every
HEC-HMS run persisted:

    "calibration_metrics": {},
    "evaluation_metrics": {},

even though the log showed a computed KGE. The scores were only recoverable
from `best_score` in the separate best-params file.

The metrics were also computed over the whole post-warmup series as a single
block rather than per period, so the evaluation period was never scored at
all — every run logged "EVALUATION PERIOD: No evaluation period configured"
despite the config setting one. That silently removed HEC-HMS from any
calibration-vs-evaluation generalisation comparison.

Split the periods the way the sibling HBV plugin already does, reusing its
warmup semantics: drop warmup from the FULL simulation first, then filter to
the period (matching how each calibration iteration was scored); the
evaluation period starts after warmup so it takes no further skip. Return
`calibration_metrics` and `evaluation_metrics` alongside `final_metrics`, and
keep unprefixed aliases in `final_metrics` for existing readers.

Verified on Bow at Banff: the persisted JSON now carries Calib_KGE 0.851 —
matching the optimiser's own best_score for that run — and Eval_KGE 0.658,
17 metrics per period, where both dicts were previously empty.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@DarriEy
DarriEy merged commit 5493f68 into main Jul 19, 2026
2 checks passed
@DarriEy
DarriEy deleted the fix/final-evaluation-period-metrics branch July 19, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant