Refuse to report an ETA while every lane is still warming up - #9
Merged
EHxuban11 merged 1 commit intoAug 4, 2026
Merged
Conversation
A run's opening epochs carry the image-cache fill, allocator warmup and cuDNN autotuning, so wall/epochs_done there is not a per-epoch cost, it is a startup cost being annualised across 100 epochs. observations() already excluded those points from the cost-model fit, but the per-lane remaining-time loop called _epoch_seconds(record) directly with no such guard, so the same contamination arrived by another route. Measured 2026-08: a cache-filling ec-s campaign with zero completed epochs reported "est. remaining 64.9h"; a fresh rfdetr-n campaign reported 90h; an earlier yolox-tiny campaign reported 16.3h. All three were the same artifact. A number that wrong is worse than no number, because it gets used for money decisions. Now: a lane only contributes its own per-epoch rate once past MIN_EPOCHS_OBSERVED, and when lanes are running but none has cleared warmup the estimate returns available=False with a reason instead of a figure. The dashboard renders "ETA not available yet (warmup/cache fill)". A campaign that has not started has no misleading evidence, only no evidence, and keeps the long-standing "no observations yet" answer. Claude-Session: https://claude.ai/code/session_01D8uB5n3e1zYjix18gT2Y4w
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third in the series after #7 (wrong numbers) and #8 (incomplete artifacts). This one is wrong estimates.
The bug
Three campaigns, three badly wrong headline ETAs, all the same artifact:
A run's opening epochs carry the image-cache fill, allocator warmup and cuDNN autotuning.
wall / epochs_donethere is not a per-epoch cost, it is a startup cost being annualised across 100 epochs.observations()already knew this — it excludes running datasets belowMIN_EPOCHS_OBSERVEDfrom the cost-model fit. But the per-lane remaining-time loop called_epoch_seconds(record)directly with no guard, so the contamination came back in through the other door and dominated the makespan simulation.What changes
MIN_EPOCHS_OBSERVED; before that it falls back to the fitted model, which was built from vetted points.available: Falseplus areason, instead of a number.ETA not available yet (warmup/cache fill)with the reason as a tooltip.The distinction that matters
A campaign that hasn't started has no misleading evidence, just no evidence — that keeps the long-standing
p50 = 0.0/"no observations yet"behaviour. Only the case where lanes are actively running but all still warming returns unavailable.I initially conflated those two and broke
test_eta_reports_its_own_method_when_it_knows_nothing. Narrowing the guard was the right fix rather than weakening that assertion.Check
tests/test_eta_refuses_to_guess.py(new, 5 tests): the real ec-s situation returns unavailable; zero-epochs returns unavailable; it becomes available once one lane clears warmup; available as soon as any dataset finishes; and a warming lane with a huge wall does not drive the estimate.Full harness suite: 158 passed.
Not verified
MIN_EPOCHS_OBSERVED = 5is inherited, not tuned here. For a ~250 s epoch that is ~20 minutes of "no ETA" at campaign start, which I think is the right trade against a 10x-wrong number, but it is a judgement call.predicted_total_secondsare unchanged; only the availability gate and the per-lane rate are touched.