Repository navigation
Description
🤖 Generated by the Agentic Engineer
Description
KSail's post-merge Go validation ended without a vulnerability-scan result when its hosted runner shut down. Attribute the interruption and restore a completed, fail-closed scan; a shutdown must never be interpreted as a clean scan or an ordinary timeout.
Steps to Reproduce
- Inspect KSail main validation attempt 1 at
4463e5eed299b78d839af4975e3cf65e48b7f0d2. - Read the vulnerability job. The scan starts at 17:43:47 UTC and receives a runner shutdown at 17:53:48 UTC on 2026-10-20.
- Compare the invoked shared workflow at
61693f6ef2740f113ecd2a4ae01840557abe44e8: its scan-step deadline is 24 minutes and job deadline is 60 minutes. The timeout classifier and second scan never execute.
This is observed failure evidence, not a deterministic reproduction. Resource exhaustion is a hypothesis; the available log does not establish the shutdown's cause.
Expected Behavior
A complete scan verdict, or a bounded operational failure with enough evidence to diagnose the runner interruption.
Actual Behavior
The scanner step is cancelled, the job and required validation fail, and there is no completed scan verdict. No finding was cleared and no scanner retry was requested.
Environment
Hosted Linux runner; Go 1.27.1; govulncheck 1.8.0; unchanged reviewed scan configuration.
Acceptance Criteria
- Attribute or reproduce the premature shutdown using bounded resource/runner evidence.
- Record a completed scan on the affected source, or a current main descendant that includes it, with the exact source and native job identity.
- If a workflow repair is needed, retain the finite deadlines, finding rejection and timeout-only retry controls, with positive and negative regression checks.
Additional Context
Priority rationale: this interrupts default-branch verification. The diagnostic source is published; KSail adoption is tracked in its native blocking issue. Interruption attribution remains unverified. The separate release's Go proxy transport failure is a distinct failure class.
Delivered on: 2026-11-15
Default-branch acceptance is now verified: KSail source cc0650bf593658a3ac8e6360831fc442ca211f01 is a descendant of the originally affected source (347 commits ahead, none behind), and run 43731873440 / scanner job 132866121627 completed with native diagnostics and an explicit no-blocking-vulnerabilities evaluator verdict. KSail #8406 and its waiting record #8409 are completed. The stale blocked label is removed; the original interruption still needs bounded causal evidence before this investigation can close. A successful scan is not proof of what caused an earlier runner loss.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- StatusShow more project fields🫴 Ready
Activity
devantler commentedon Oct 25, 2026
Refinement (kept open, future check 2026-11-03).
What I checked just now:
c6505ba3, a descendant of4463e5ee, finished its Vulnerability Scan job withsuccess(00:49:16 to 00:59:25 UTC on 2026-10-25, about ten minutes, well inside the 24-minute step deadline).ubuntu-latestlabel began moving to Ubuntu 26 on 19 October, one day before the incident. That is context, not evidence of cause.Proposed exit: if no scanner runner-shutdown recurs on a consuming repository's default branch by 2026-11-03, close this as not reproducible, with the completed scan above as the verdict record. A recurrence before then gets attributed from its own fresh logs. The separate proxy-download failure class is handled in #672 (PR #746).
devantler commentedon Nov 1, 2026
A new native failure belongs to this investigation: KSail required Go run 41283844401, job 124781088005, source
7f1088b73986cfbca329668df6621ae0d0bc42fd, ended with GitHub's “hosted runner lost communication with the server” annotation. It has no completed vulnerability verdict; the log download currently returns BlobNotFound. The browser's increasing elapsed counter while the API said in progress does not prove the scanner continued executing past its step deadline. CPU starvation, memory pressure and network loss remain unverified causes.A real healthy base run at
dd03dae58c7e673b75fca8fc6181c60a030f0847completed the scan in about 12 minutes and the job in 13m33s, using Go 1.27.3, govulncheck 1.8.0 built with Go 1.27.2 and GOMEMLIMIT=8GiB. Product Go sources, dependency manifests, allowlist and scanner helper bytes are unchanged by the failed PR; only the workflow adoption and CI contract fixture differ.One failed-job recovery was started and read back as attempt 2 at the unchanged source. All security checks remain required. This issue stays open: a recovery pass would not establish the shutdown's root cause, and another failure requires further diagnosis rather than repeated blind retries.
devantler commentedon Nov 4, 2026
Re-check, 4 November 2026 (the 3 November exit date has passed). The "close as not reproducible" exit doesn't apply: the shutdown came back on KSail's default branch before the deadline. In main CI run 41212866444 (31 October, attempt 1), the Vulnerability Scan job ended with GitHub's "hosted runner lost communication with the server" annotation while its scan step was still running. Neither the timeout classifier nor the retry ran. The 1 November PR-run case above (41283844401) completed its scan on attempt 2. I also found one main scan step cancelled mid-run (40737176674, 28 October). Its cause isn't recorded, so I'm noting it, not counting it.
This stays open. Next step: attribute the 31 October recurrence from its own logs and runner evidence (first acceptance criterion), not retry blindly.
🤖 Generated with Claude Code
devantler commentedon Nov 7, 2026
Two more default-branch recurrences on 7 November 2026. Both ended with GitHub's "hosted runner lost communication with the server" annotation while the scan step was still running. Neither the timeout classifier nor the retry ran, and neither has a scan verdict. Both log downloads return
BlobNotFound.16ee5f6a(main push)New observation: both jobs ran far past the scan step's 24-minute deadline before the runner was declared lost.
timeout-minutesis enforced by the runner process itself, so a deadline that never fired suggests the runner stopped responding, rather than the scan finishing late. That fits CPU or memory starvation better than network loss. It is still a hypothesis: no resource evidence exists for these jobs.That makes four runner losses on main in eight days (31 Oct, 1 Nov on a PR, and two today). I re-ran the failed jobs of the 18:30 run. Attempt 2 completed the Vulnerability Scan with
success, so16ee5f6anow has a completed scan verdict and main is green.chore: agent-claim #671 nonce=d34efe91b6b85ee2763b0be122d17034
devantler commentedon Nov 8, 2026
The scan needs about as much memory as the runner has
This covers the first acceptance criterion: what makes the runner stop. Two measurements point the same way.
1. Peak memory of the exact scan. I ran the shared action's command (
govulncheck@v1.8.0 -format=json -scan=symbol ./...) on KSailmainatc97b8966with Go 1.27.3 and the workflow'sGOMEMLIMIT=8GiB, measured with/usr/bin/time -l:GOMAXPROCS=4, like the hosted runnerA public repository's
ubuntu-latestrunner has 16 GB. The scan alone reaches that, before counting the runner agent, harden-runner and thegochild processes.GOMEMLIMITcannot stop this. It is a soft limit, and the Go runtime goes past it when the live data is bigger than the limit, which it must be here (the peak is about twice the limit). The workflow's own comment on the job already describes this failure: the host stops the runner mid-scan.Limits of this measurement: it ran on macOS arm64, not Linux amd64, so the exact figure on the runner will differ. The gap to the 16 GB ceiling is small either way, which also explains why the same commit sometimes passes on a re-run.
2. Which jobs lose their runner. I checked every failed job in the 150 most recent failed KSail runs (3 November 17:20 UTC to now) for GitHub's "hosted runner lost communication" annotation. There were 8:
main, 3 on pull requests)Analyze (go)Both affected jobs build an analysis of the whole program in memory. No light job lost a runner in that window. This count leaves out runs that a re-run later turned green, so the 7 November 18:30 case is not in it.
This also explains the earlier puzzle: the 24-minute step deadline never fired. The runner process enforces that deadline, and it was starved or stopped before it could.
One detail does not fit cleanly. In job 128402435699 the API marks the scan step
successat 12:35:00, and the runner is reported lost at 13:19:36. So that loss came after the scan step ended. Memory left behind by the scan is one possible reason. I have not proved it.Possible repairs (not decided)
Each repair must keep the finite deadlines, the finding rejection and the timeout-only retry:
-scan=symbolover separate groups of packages so that no single call graph holds the whole module. This keeps the same reachability-based verdict. It needs a check that the combined findings match one full scan.-scan=package) to save memory. That changes which findings block a merge.CodeQL
Analyze (go)is GitHub-managed, so it needs a separate decision. It should not be changed as part of this issue.Not changed: no workflow, no required check, no retry. The board is above its in-progress limit, so this is recorded as refinement, not started as a PR.
devantler commentedon Nov 12, 2026
One more recurrence, on a pull request on 12 November 2026: run 43301993183 / job 131476468245 (devantler-tech/ksail#8349, a CI-only change) started at 20:41:15Z and ended at 21:29:16Z with the same "hosted runner lost communication with the server" annotation while the scan step was still running. The scan step and both retry steps have no conclusion, and the log download returns
BlobNotFound. Only the failed job was re-run (attempt 2).A second recurrence the same evening: run 43318693885 / job 131535804820 (devantler-tech/ksail#8350, a CI/test-only change) started at 22:39:53Z and ended at 23:34:56Z the same way: lost runner during the scan step, no step conclusions. Only the failed job was re-run.
devantler commentedon Nov 14, 2026
Three more recurrences on 14 November 2026, all on KSail pull requests, all at about 48–55 minutes with the scan step still running and "The hosted runner lost communication with the server":
siderolabsbumpSuccessful scans in the same window took about 15 minutes (for example run 43609665253, 15:21–15:36). So the runner is lost at a consistent point well past a normal scan, which fits the scan stalling or exhausting the runner rather than a random infrastructure drop. Log downloads return
BlobNotFound, as before. The two pull-request jobs were re-run (attempt 2). That is 3 of the last 8 KSail validation runs, so this now blocks merges regularly.devantler commentedon Nov 14, 2026
A fourth recurrence today, with a usable log this time. The re-run of devantler-tech/ksail#8391's scan (job 132556375362, 19:18–19:30 UTC) reached
Scan and enforce reachable findings(govulncheckwithGOMEMLIMIT=8GiB) and, about 10 minutes into it, logged "The runner has received a shutdown signal" and was cancelled. The timeout retry steps were skipped, so the job failed without any scan verdict. This is the same 10-minute shutdown as the original 20 October case, so these failures look like the scan itself making the hosted runner unavailable, not like a slow scan.devantler commentedon Nov 14, 2026
A fifth recurrence, in a different shape. The second re-run of devantler-tech/ksail#8391's scan (job 132562329243, 19:40–20:36 UTC) ran for 56 minutes and ended with "The hosted runner lost communication with the server", the same annotation as the three earlier 48–55 minute cases. So today shows both modes on one pull request: a shutdown about 10 minutes into
govulncheck, and a lost runner near the job deadline.One correlation worth testing, not yet a finding: the job's own annotation says
ubuntu-latestbegan moving to Ubuntu 26 on 19 October 2026, and the first recorded case in this issue is 20 October. Comparing the runner image version of a failed job against a passing one would confirm or rule it out.devantler-tech/ksail#8391 is parked on this issue rather than re-run again.
chore: agent-claim #671 nonce=26175ca5af552fe255c27e2e16712750
47 remaining items
devantler commentedon Nov 16, 2026
Post-merge publication follow-up for #950 is complete at
d70d50665465117f01104313e53057374a96a1a3: main CI, manifest publication, Code Quality, security analysis, and repository drift verification all completed successfully.This closes the unfinished workflow collection in the previous handoff. It does not establish live manifest reconciliation or affected-consumer scanner acceptance: the catalogue's production-interface vulnerability job was skipped, so its green CI is not a consumer scan. No new KSail workflow run was returned for the interval beginning at this merge.
Keep #671 open. The next proof remains a native affected-consumer scan bound to the new shared workflow revision, with the owned swap setup and cleanup and the enforced finding verdict. Historical runner-loss attribution remains separate and unverified.
chore: agent-claim #671 nonce=996b614ec8df2a50cfbcbd89c9740b39
devantler commentedon Nov 16, 2026
An affected consumer has now exercised the published repair. Platform required Go run 43811790996, attempt 1 / scanner job 133115087955 completed successfully for source
2304047d95a5e392cac7708bc4ffb696c5eb4986after #950 merged.The native log binds both the shared scanner checkout and action execution to
d70d50665465117f01104313e53057374a96a1a3. It records 4 GiB swap creation and activation, scanner exit 0, the evaluator's no-blocking-vulnerabilities verdict, and the owned swap retirement step executingswapoffand removal successfully. The timeout-only retry was skipped. Scan diagnostics recorded 5.47 seconds and peak RSS 2,169,240 KiB.This proves the new lifecycle ran on Platform's ordinary required-workflow path. Its smaller scan does not establish that swap prevents KSail's runner losses, nor attribute historical interruptions. Keep #671 open for affected KSail acceptance and bounded attribution evidence; the original acceptance criteria are unchanged.
devantler commentedon Nov 16, 2026
A KSail consumer still loses its hosted runner after #950. I re-ran the failed vulnerability job of ksail#8391 at
7fc93dd3(a change that does not touch the scanned code). Run 43735166748, attempt 3, job 133168311777 ran on the merged workflow: "Prepare bounded swap headroom for the scanner" succeeded. The scan step then ran from 08:06 to 08:54 UTC and ended with "The hosted runner lost communication with the server", with no scan verdict, no timeout retry and no swap retirement.So the 4 GiB swap fixed the smaller Platform scan (05:45 note above), but not the KSail one. KSail is the larger module graph (about 14.6 GiB peak measured earlier against a 16 GB runner). The next options are more headroom, a larger runner for this one consumer, or narrowing what the scan loads.
devantler commentedon Nov 16, 2026
Two cheaper routes for KSail's scan were measured on KSail
mainatcc0650bf5with the reviewed scanner (govulncheck v1.8.0, Go 1.27.4). Neither fixes the runner losses.1. Run a package-level scan first and skip the expensive scan when it is clean. A symbol-level finding always has a package-level finding too, so a clean package-level result would prove the full scan clean at almost no cost. The package-level scan took 3.7 s and peaked at about 200 MiB. But KSail imports 15 packages that have advisories (for example GO-2025-3521 and GO-2026-6905), so the full symbol-level scan would still run on every KSail change. This route does not help KSail.
2. Split
./...into smaller scans. The scanner's memory grows with the packages it loads.go list -depsshows thatpkg/clialone loads 5,936 of the module's 5,985 packages, so every split that includes the CLI loads almost the whole graph again. This route does not help either.What is left — each needs a decision this issue has not made yet:
ksailexecutable. It uses much less memory, but it covers only the code linked into the shipped binary, not every package. That changes what this required check covers, so it needs its own review against the gate's purpose.Parked PRs that wait on this issue, such as devantler-tech/ksail#8391, keep their current re-entry condition.
chore: agent-claim #671 nonce=6c1a9838cb7719fab3aedbb4c1f56ddf
devantler commentedon Nov 16, 2026
New default-branch interruption: KSail run 43898923987, attempt 1 / vulnerability job 133410590872, source
fc4e3d24c8d814fe6a058e87b55841788cb5267e, failed after the hosted runner lost communication. The native scan step ran from 15:41:59 to 15:53:26 UTC on 16 November, then was cancelled after 11m27s. The timeout classifier and retry were skipped. This is no completed scan verdict and does not establish an OOM or a configured timeout.The attempt's native
referenced_workflowsbinds this invocation tovalidate-go-project.yaml@ac1f346773da6f47444c283927fb1ce31187fe21. That immutable workflow lacks the scanner swap preparation/retirement added by #950; current shared main includes both. This failure therefore does not test the #950 mitigation, and source publication alone does not establish adoption by every caller. The failed job log returned HTTP 404, and the run's complete two-artifact listing contained no scanner diagnostics, so causal attribution remains unavailable.One manual failed-jobs retry was accepted as attempt 2 at the same KSail source; its vulnerability job
133455425099was observed running. No further retry is armed. Collect that native result once, then advance the existing consumer rollout and bounded attribution work. Keep the original acceptance criteria open; neither a retry nor earlier successful descendant scans clear this new incomplete scan.chore: agent-claim #671 nonce=0cb4247b6073f50c36aecae30a714860
devantler commentedon Nov 16, 2026
The single failed-jobs retry of KSail run 43898923987, attempt 2 has completed with failure at the same source,
fc4e3d24c8d814fe6a058e87b55841788cb5267e. Its vulnerability job 133455425099 reports another hosted-runner communication loss. The native scan step was cancelled from 16:50:20 to 17:01:16 UTC on 16 November: 10m56s, before its configured 24-minute deadline. This is an incomplete scan, with no vulnerability verdict and no proven OOM or timeout cause.The run's immutable workflow reference is still
validate-go-project.yaml@ac1f346773da6f47444c283927fb1ce31187fe21, which lacks the scanner swap lifecycle. The complete artifact listing still contains only the patch and Calico binary artifacts, with no scanner diagnostics. Neither attempt tests the mitigation released in shared workflow v21.7.2.No further manual retry is armed. The next route is the existing KSail adoption PR #8420, currently draft at
ab559283c580aed2c14a902306639fa0877ef03bwith system tests pending. Finish its current-head readiness, then verify a native main invocation actually uses the released workflow and completes the scan and swap cleanup. KSail #8419 retains that acceptance; this investigation retains its unresolved attribution criterion.chore: agent-claim #671 nonce=ca3ee3586be3cb6697fcf67595e2b69b
chore: agent-claim #671 nonce=5bb9d786a9a8083f7c9124748c7b5f10
devantler commentedon Nov 17, 2026
The adoption handoff above is superseded: KSail #8420 is merged and KSail #8419 has completed its native acceptance. The historical runner-loss investigation remains open here.
Independent readback of KSail main CI 43976765109 at
1f11639aa011dfce2b2971edaae1adf9ebc0daf7confirms the actual invocation referencesvalidate-go-project.yaml@d70d50665465117f01104313e53057374a96a1a3, which contains the bounded swap mitigation. Its native vulnerability job 133780587573 completed successfully:swapoff.This establishes adoption and a completed scan with the mitigation. It does not attribute the earlier runner losses, prove an OOM, or prove the interruption cannot recur. Preserve the unfinished causal-evidence criterion. The next investigation step is bounded runner/resource evidence from an actual interruption under the adopted workflow, rather than another adoption PR or an unbounded retry. No further retry was requested by this verification.