fix(calibration): report calibration and evaluation period metrics separately#5
Merged
Merged
Conversation
…parately
`run_final_evaluation` returned only `final_metrics`, so the results saver's
`final_result.get('calibration_metrics', {})` wrote empty dicts and every
HEC-HMS run persisted:
"calibration_metrics": {},
"evaluation_metrics": {},
even though the log showed a computed KGE. The scores were only recoverable
from `best_score` in the separate best-params file.
The metrics were also computed over the whole post-warmup series as a single
block rather than per period, so the evaluation period was never scored at
all — every run logged "EVALUATION PERIOD: No evaluation period configured"
despite the config setting one. That silently removed HEC-HMS from any
calibration-vs-evaluation generalisation comparison.
Split the periods the way the sibling HBV plugin already does, reusing its
warmup semantics: drop warmup from the FULL simulation first, then filter to
the period (matching how each calibration iteration was scored); the
evaluation period starts after warmup so it takes no further skip. Return
`calibration_metrics` and `evaluation_metrics` alongside `final_metrics`, and
keep unprefixed aliases in `final_metrics` for existing readers.
Verified on Bow at Banff: the persisted JSON now carries Calib_KGE 0.851 —
matching the optimiser's own best_score for that run — and Eval_KGE 0.658,
17 metrics per period, where both dicts were previously empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
run_final_evaluationreturned onlyfinal_metrics, so the results saver'sfinal_result.get('calibration_metrics', {})wrote empty dicts. Every HEC-HMS run persisted:…even though the log showed a computed KGE. The scores were only recoverable from
best_scorein the separate best-params file.The metrics were also computed over the whole post-warmup series as a single block rather than per period, so the evaluation period was never scored at all. Every run logged
EVALUATION PERIOD: No evaluation period configureddespite the config setting one — silently removing HEC-HMS from any calibration-vs-evaluation generalisation comparison.Found while reproducing the SYMFLUENCE Paper 3 experiments, where all 17 HEC-HMS calibration cells had empty metrics.
Fix
Split the periods the way the sibling HBV plugin already does, reusing its warmup semantics:
calibration_metricsandevaluation_metricsalongsidefinal_metrics, keeping unprefixed aliases infinal_metricsfor existing readersVerification
Re-ran a HEC-HMS calibration on Bow at Banff with the patched package:
calibration_metrics{}evaluation_metrics{}Calib_KGE 0.851matches the optimiser's ownbest_scorefor that run — an independent check that the calibration-period split is correct.Eval_KGEis information that previously did not exist.Existing test suite passes (3 passed); ruff clean.
🤖 Generated with Claude Code