Skip to content

Refuse to report an ETA while every lane is still warming up - #9

Merged
EHxuban11 merged 1 commit into
rf100vl-harnessfrom
fix/eta-refuses-to-guess-during-cache-fill
Aug 4, 2026
Merged

EHxuban11 merged 1 commit into
rf100vl-harnessfrom
fix/eta-refuses-to-guess-during-cache-fill

Conversation

@EHxuban11

Copy link
Copy Markdown
Contributor

Third in the series after #7 (wrong numbers) and #8 (incomplete artifacts). This one is wrong estimates.

The bug

Three campaigns, three badly wrong headline ETAs, all the same artifact:

campaign reported reality
ec-s, cache filling, 0 epochs done est. remaining 64.9 h unknown, but not that
rfdetr-n, fresh dataset, 0 epochs done 90 h unknown
yolox-tiny (earlier) 16.3 h actual campaign 4.4 h

A run's opening epochs carry the image-cache fill, allocator warmup and cuDNN autotuning. wall / epochs_done there is not a per-epoch cost, it is a startup cost being annualised across 100 epochs.

observations() already knew this — it excludes running datasets below MIN_EPOCHS_OBSERVED from the cost-model fit. But the per-lane remaining-time loop called _epoch_seconds(record) directly with no guard, so the contamination came back in through the other door and dominated the makespan simulation.

What changes

  • A lane contributes its own per-epoch rate only once past MIN_EPOCHS_OBSERVED; before that it falls back to the fitted model, which was built from vetted points.
  • When lanes are running but none has cleared warmup and nothing has finished, return available: False plus a reason, instead of a number.
  • Dashboard renders ETA not available yet (warmup/cache fill) with the reason as a tooltip.

The distinction that matters

A campaign that hasn't started has no misleading evidence, just no evidence — that keeps the long-standing p50 = 0.0 / "no observations yet" behaviour. Only the case where lanes are actively running but all still warming returns unavailable.

I initially conflated those two and broke test_eta_reports_its_own_method_when_it_knows_nothing. Narrowing the guard was the right fix rather than weakening that assertion.

Check

tests/test_eta_refuses_to_guess.py (new, 5 tests): the real ec-s situation returns unavailable; zero-epochs returns unavailable; it becomes available once one lane clears warmup; available as soon as any dataset finishes; and a warming lane with a huge wall does not drive the estimate.

Full harness suite: 158 passed.

Not verified

  • MIN_EPOCHS_OBSERVED = 5 is inherited, not tuned here. For a ~250 s epoch that is ~20 minutes of "no ETA" at campaign start, which I think is the right trade against a 10x-wrong number, but it is a judgement call.
  • The p90 band and predicted_total_seconds are unchanged; only the availability gate and the per-lane rate are touched.

A run's opening epochs carry the image-cache fill, allocator warmup and
cuDNN autotuning, so wall/epochs_done there is not a per-epoch cost, it is
a startup cost being annualised across 100 epochs.

observations() already excluded those points from the cost-model fit, but
the per-lane remaining-time loop called _epoch_seconds(record) directly
with no such guard, so the same contamination arrived by another route.

Measured 2026-08: a cache-filling ec-s campaign with zero completed epochs
reported "est. remaining 64.9h"; a fresh rfdetr-n campaign reported 90h;
an earlier yolox-tiny campaign reported 16.3h. All three were the same
artifact. A number that wrong is worse than no number, because it gets
used for money decisions.

Now: a lane only contributes its own per-epoch rate once past
MIN_EPOCHS_OBSERVED, and when lanes are running but none has cleared
warmup the estimate returns available=False with a reason instead of a
figure. The dashboard renders "ETA not available yet (warmup/cache fill)".

A campaign that has not started has no misleading evidence, only no
evidence, and keeps the long-standing "no observations yet" answer.

Claude-Session: https://claude.ai/code/session_01D8uB5n3e1zYjix18gT2Y4w
@EHxuban11
EHxuban11 merged commit 0de9e57 into rf100vl-harness Aug 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant