Bound the VAX image test's wait for sshd on the clock - #13
Merged
Conversation
The wait counted attempts -- 180 of them, spaced by sleep 20 -- and the comment claimed that came to about an hour. It only does when every attempt fails instantly, which is what happens while the guest's ssh port is still closed: slirp answers with an RST. An attempt against a guest that accepts the connection and then stalls costs the whole ConnectTimeout instead, which is 60s here, making the same loop nearly four hours. That is not hypothetical: a guest that stalled mid-boot sat in this step for over two hours before anyone noticed, and would have kept the runner for four. Take the deadline off the clock so the bound holds regardless of what an individual attempt costs. The overshoot is now just the attempt in flight when the hour passes, rather than a multiple of the whole budget. Also drop the "~15 minutes" figure from the comment while rewriting it. A healthy guest reaches sshd in about two minutes; the host keys stopped being generated on first boot a while ago, which is what that number described. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M9LHkNMNNfDf2Fa1ieGT8g
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The wait counted attempts — 180 of them, spaced by
sleep 20— and the comment claimed that came to about an hour:It only comes to an hour when every attempt fails instantly, which is what happens while the guest's ssh port is still closed: slirp answers with an RST. An attempt against a guest that accepts the connection and then stalls costs the whole
ConnectTimeoutinstead — 60s here — making the same loop nearly four hours.That is not hypothetical. Two VAX jobs on the same commit failed the same nominal way and took wildly different amounts of time:
Test Image (vax)10.1, vax11.0, vaxSo the bound depended on how the guest was broken, which is the one thing a timeout should not depend on.
The fix
Take the deadline off the clock, so it holds regardless of what an individual attempt costs:
The overshoot is now just the attempt in flight when the hour passes, instead of a multiple of the whole budget. Verified against a stubbed
ssh_vaxat 1/1000th scale — exits immediately on success, and honours the budget whether attempts fail instantly or burn time (overshooting by one attempt, ~80s at real scale).The comment also loses its "~15 minutes" figure. A healthy guest reaches sshd in about two minutes; that number described first-boot host key generation, which the image stopped doing when the keys started being baked in.
Scope
Branched from
master, and touches only the VAX test step, so it is independent of both #11 and the ntpdate fix in #12. This is purely about the harness bounding itself — #12 addressed one cause of a stalled guest, whereas this makes any future cause cost an hour instead of four.Generated by Claude Code