fix(jobs): wait for a fresh pane's prompt before typing a command - #39
Merged
Conversation
Launching into a pane created moments earlier raced the shell's first
prompt. The pty buffers the typed command, so the kernel's canonical
echo lands in the output stream BEFORE the prompt; when the shell
finally prints it, the prompt glues itself to the job's first output
line ("sh-5.3$ 1"). That polluted reads and per-job log files, and
the prompt landed inside the job's output slice because baselineLines
was captured before it printed.
This was the root cause of the flaky job-log integration test
(sidemux-pbg): under CPU load the fixture's fresh pane lost the race
2 of 3 full-suite runs, and readJobLog's first output element became
'sh-5.3$ 1' instead of '1'. Reproduced deterministically with a
30-iteration stress run under load; zero failures after this fix.
launch() now polls pane state until the pane stops being completely
blank (cursor at origin, no history) before typing, capped at 2s for
shells that print no prompt at all. cursor_x is added to
PANE_STATE_FORMAT so 'prompt printed' is detectable on row 0.
Closes sidemux-pbg
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the flaky job-log integration test (sidemux-pbg) at its real cause — a launch race that also affects production.
Root cause
Launching a job into a pane created moments earlier races the shell's first prompt. The pty buffers the typed command and the kernel's canonical echo lands in the raw output stream before the prompt; when the shell finally prints it, the prompt glues itself to the job's first output line:
This pollutes reads and per-job logs, and the prompt lands inside the job's output slice because
baselineLinesis captured before it prints. Under CPU load the integration fixture lost this race in 2 of 3 full-suite runs (expected [ …(203) ] to include 1).Fix
JobManager.launch()now waits for the pane to stop being completely blank — cursor at origin, zero history — before typing, polling every 25ms with a 2s cap (a shell that prints no prompt still proceeds).cursor_xis added toPANE_STATE_FORMATso a one-line prompt on row 0 is detectable.Verification
Closes sidemux-pbg