Skip to content

Diagnose Go scanner shutdowns before their configured deadline #671

Open
1 of 1 issue completed
Open

Description

@devantler

🤖 Generated by the Agentic Engineer

Description

KSail's post-merge Go validation ended without a vulnerability-scan result when its hosted runner shut down. Attribute the interruption and restore a completed, fail-closed scan; a shutdown must never be interpreted as a clean scan or an ordinary timeout.

Steps to Reproduce

  1. Inspect KSail main validation attempt 1 at 4463e5eed299b78d839af4975e3cf65e48b7f0d2.
  2. Read the vulnerability job. The scan starts at 17:43:47 UTC and receives a runner shutdown at 17:53:48 UTC on 2026-10-20.
  3. Compare the invoked shared workflow at 61693f6ef2740f113ecd2a4ae01840557abe44e8: its scan-step deadline is 24 minutes and job deadline is 60 minutes. The timeout classifier and second scan never execute.

This is observed failure evidence, not a deterministic reproduction. Resource exhaustion is a hypothesis; the available log does not establish the shutdown's cause.

Expected Behavior

A complete scan verdict, or a bounded operational failure with enough evidence to diagnose the runner interruption.

Actual Behavior

The scanner step is cancelled, the job and required validation fail, and there is no completed scan verdict. No finding was cleared and no scanner retry was requested.

Environment

Hosted Linux runner; Go 1.27.1; govulncheck 1.8.0; unchanged reviewed scan configuration.

Acceptance Criteria

  • Attribute or reproduce the premature shutdown using bounded resource/runner evidence.
  • Record a completed scan on the affected source, or a current main descendant that includes it, with the exact source and native job identity.
  • If a workflow repair is needed, retain the finite deadlines, finding rejection and timeout-only retry controls, with positive and negative regression checks.

Additional Context

Priority rationale: this interrupts default-branch verification. The diagnostic source is published; KSail adoption is tracked in its native blocking issue. Interruption attribution remains unverified. The separate release's Go proxy transport failure is a distinct failure class.

Delivered on: 2026-11-15

Default-branch acceptance is now verified: KSail source cc0650bf593658a3ac8e6360831fc442ca211f01 is a descendant of the originally affected source (347 commits ahead, none behind), and run 43731873440 / scanner job 132866121627 completed with native diagnostics and an explicit no-blocking-vulnerabilities evaluator verdict. KSail #8406 and its waiting record #8409 are completed. The stale blocked label is removed; the original interruption still needs bounded causal evidence before this investigation can close. A successful scan is not proof of what caused an earlier runner loss.

Activity

  1. added theissue type on Oct 20, 2026
  2. moved this to 📥 Backlog in 🌊 Project Boardon Oct 20, 2026
  3. devantler commented on Oct 25, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer (interactive Claude Code session)

    Refinement (kept open, future check 2026-11-03).

    What I checked just now:

    • Completed scan on a descendant (second acceptance criterion): KSail main run 40148089495 at c6505ba3, a descendant of 4463e5ee, finished its Vulnerability Scan job with success (00:49:16 to 00:59:25 UTC on 2026-10-25, about ten minutes, well inside the 24-minute step deadline).
    • No recurrence: KSail main has 46 CI runs from 2026-10-20 17:00 UTC onward, including the incident run; 36 passed. None of the six later failures was a scanner runner shutdown. Two were Vulnerability Scan failures with real reachable-advisory verdicts (runs 39529498180 and 39531840057). One (39473856395) failed on module-proxy stream errors in other jobs, which is the Retry Go module downloads before the vulnerability scan so a proxy blip does not turn main red #672 class. The rest failed in unrelated jobs.
    • Attribution (first criterion) is still open. The original job log only shows the shutdown signal, so the cause cannot be attributed after the fact. The same runs carried GitHub's notice that the ubuntu-latest label began moving to Ubuntu 26 on 19 October, one day before the incident. That is context, not evidence of cause.

    Proposed exit: if no scanner runner-shutdown recurs on a consuming repository's default branch by 2026-11-03, close this as not reproducible, with the completed scan above as the verdict record. A recurrence before then gets attributed from its own fresh logs. The separate proxy-download failure class is handled in #672 (PR #746).

  4. devantler commented on Nov 1, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    A new native failure belongs to this investigation: KSail required Go run 41283844401, job 124781088005, source 7f1088b73986cfbca329668df6621ae0d0bc42fd, ended with GitHub's “hosted runner lost communication with the server” annotation. It has no completed vulnerability verdict; the log download currently returns BlobNotFound. The browser's increasing elapsed counter while the API said in progress does not prove the scanner continued executing past its step deadline. CPU starvation, memory pressure and network loss remain unverified causes.

    A real healthy base run at dd03dae58c7e673b75fca8fc6181c60a030f0847 completed the scan in about 12 minutes and the job in 13m33s, using Go 1.27.3, govulncheck 1.8.0 built with Go 1.27.2 and GOMEMLIMIT=8GiB. Product Go sources, dependency manifests, allowlist and scanner helper bytes are unchanged by the failed PR; only the workflow adoption and CI contract fixture differ.

    One failed-job recovery was started and read back as attempt 2 at the unchanged source. All security checks remain required. This issue stays open: a recovery pass would not establish the shutdown's root cause, and another failure requires further diagnosis rather than repeated blind retries.

  5. devantler commented on Nov 4, 2026

    @devantler
    ContributorAuthor

    Re-check, 4 November 2026 (the 3 November exit date has passed). The "close as not reproducible" exit doesn't apply: the shutdown came back on KSail's default branch before the deadline. In main CI run 41212866444 (31 October, attempt 1), the Vulnerability Scan job ended with GitHub's "hosted runner lost communication with the server" annotation while its scan step was still running. Neither the timeout classifier nor the retry ran. The 1 November PR-run case above (41283844401) completed its scan on attempt 2. I also found one main scan step cancelled mid-run (40737176674, 28 October). Its cause isn't recorded, so I'm noting it, not counting it.

    This stays open. Next step: attribute the 31 October recurrence from its own logs and runner evidence (first acceptance criterion), not retry blindly.

    🤖 Generated with Claude Code

  6. devantler commented on Nov 7, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Two more default-branch recurrences on 7 November 2026. Both ended with GitHub's "hosted runner lost communication with the server" annotation while the scan step was still running. Neither the timeout classifier nor the retry ran, and neither has a scan verdict. Both log downloads return BlobNotFound.

    Run / job Source Job started → lost (UTC)
    42372147601 / 128402435699 main push 12:22:33 → 13:19:36 (~57 min)
    42401586135 / 128490521168 16ee5f6a (main push) 18:30:46 → 19:24:48 (~54 min)

    New observation: both jobs ran far past the scan step's 24-minute deadline before the runner was declared lost. timeout-minutes is enforced by the runner process itself, so a deadline that never fired suggests the runner stopped responding, rather than the scan finishing late. That fits CPU or memory starvation better than network loss. It is still a hypothesis: no resource evidence exists for these jobs.

    That makes four runner losses on main in eight days (31 Oct, 1 Nov on a PR, and two today). I re-ran the failed jobs of the 18:30 run. Attempt 2 completed the Vulnerability Scan with success, so 16ee5f6a now has a completed scan verdict and main is green.

  7. added a commit that references this issue on Nov 8, 2026
  8. devantler commented on Nov 8, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The scan needs about as much memory as the runner has

    This covers the first acceptance criterion: what makes the runner stop. Two measurements point the same way.

    1. Peak memory of the exact scan. I ran the shared action's command (govulncheck@v1.8.0 -format=json -scan=symbol ./...) on KSail main at c97b8966 with Go 1.27.3 and the workflow's GOMEMLIMIT=8GiB, measured with /usr/bin/time -l:

    Run Wall time Peak memory
    All 10 CPUs 5m28s 15.7 GB
    GOMAXPROCS=4, like the hosted runner 3m36s 16.6 GB

    A public repository's ubuntu-latest runner has 16 GB. The scan alone reaches that, before counting the runner agent, harden-runner and the go child processes. GOMEMLIMIT cannot stop this. It is a soft limit, and the Go runtime goes past it when the live data is bigger than the limit, which it must be here (the peak is about twice the limit). The workflow's own comment on the job already describes this failure: the host stops the runner mid-scan.

    Limits of this measurement: it ran on macOS arm64, not Linux amd64, so the exact figure on the runner will differ. The gap to the 16 GB ceiling is small either way, which also explains why the same commit sometimes passes on a re-run.

    2. Which jobs lose their runner. I checked every failed job in the 150 most recent failed KSail runs (3 November 17:20 UTC to now) for GitHub's "hosted runner lost communication" annotation. There were 8:

    Job Losses
    🛡️ Vulnerability Scan 5 (2 on main, 3 on pull requests)
    CodeQL Analyze (go) 3
    Every other job (tests, lint, system tests, builds) 0

    Both affected jobs build an analysis of the whole program in memory. No light job lost a runner in that window. This count leaves out runs that a re-run later turned green, so the 7 November 18:30 case is not in it.

    This also explains the earlier puzzle: the 24-minute step deadline never fired. The runner process enforces that deadline, and it was starved or stopped before it could.

    One detail does not fit cleanly. In job 128402435699 the API marks the scan step success at 12:35:00, and the runner is reported lost at 13:19:36. So that loss came after the scan step ended. Memory left behind by the scan is one possible reason. I have not proved it.

    Possible repairs (not decided)

    Each repair must keep the finite deadlines, the finding rejection and the timeout-only retry:

    • Split the scan. Run -scan=symbol over separate groups of packages so that no single call graph holds the whole module. This keeps the same reachability-based verdict. It needs a check that the combined findings match one full scan.
    • Use a larger runner for this job only. This is the simplest change, but it costs money, so it needs the maintainer's decision.
    • Do not lower the scan level (for example -scan=package) to save memory. That changes which findings block a merge.

    CodeQL Analyze (go) is GitHub-managed, so it needs a separate decision. It should not be changed as part of this issue.

    Not changed: no workflow, no required check, no retry. The board is above its in-progress limit, so this is recorded as refinement, not started as a PR.

  9. devantler commented on Nov 12, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    One more recurrence, on a pull request on 12 November 2026: run 43301993183 / job 131476468245 (devantler-tech/ksail#8349, a CI-only change) started at 20:41:15Z and ended at 21:29:16Z with the same "hosted runner lost communication with the server" annotation while the scan step was still running. The scan step and both retry steps have no conclusion, and the log download returns BlobNotFound. Only the failed job was re-run (attempt 2).

    A second recurrence the same evening: run 43318693885 / job 131535804820 (devantler-tech/ksail#8350, a CI/test-only change) started at 22:39:53Z and ended at 23:34:56Z the same way: lost runner during the scan step, no step conclusions. Only the failed job was re-run.

  10. devantler commented on Nov 14, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Three more recurrences on 14 November 2026, all on KSail pull requests, all at about 48–55 minutes with the scan step still running and "The hosted runner lost communication with the server":

    Run / job Branch Job window (UTC)
    43601499297 / 132467212519 Dependabot siderolabs bump 13:48–14:36
    43614692072 / 132531598167 devantler-tech/ksail#7483 17:43–18:38
    43625845906 / 132541582269 devantler-tech/ksail#8391 18:19–19:07

    Successful scans in the same window took about 15 minutes (for example run 43609665253, 15:21–15:36). So the runner is lost at a consistent point well past a normal scan, which fits the scan stalling or exhausting the runner rather than a random infrastructure drop. Log downloads return BlobNotFound, as before. The two pull-request jobs were re-run (attempt 2). That is 3 of the last 8 KSail validation runs, so this now blocks merges regularly.

  11. devantler commented on Nov 14, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    A fourth recurrence today, with a usable log this time. The re-run of devantler-tech/ksail#8391's scan (job 132556375362, 19:18–19:30 UTC) reached Scan and enforce reachable findings (govulncheck with GOMEMLIMIT=8GiB) and, about 10 minutes into it, logged "The runner has received a shutdown signal" and was cancelled. The timeout retry steps were skipped, so the job failed without any scan verdict. This is the same 10-minute shutdown as the original 20 October case, so these failures look like the scan itself making the hosted runner unavailable, not like a slow scan.

  12. devantler commented on Nov 14, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    A fifth recurrence, in a different shape. The second re-run of devantler-tech/ksail#8391's scan (job 132562329243, 19:40–20:36 UTC) ran for 56 minutes and ended with "The hosted runner lost communication with the server", the same annotation as the three earlier 48–55 minute cases. So today shows both modes on one pull request: a shutdown about 10 minutes into govulncheck, and a lost runner near the job deadline.

    One correlation worth testing, not yet a finding: the job's own annotation says ubuntu-latest began moving to Ubuntu 26 on 19 October 2026, and the first recorded case in this issue is 20 October. Comparing the runner image version of a failed job against a passing one would confirm or rule it out.

    devantler-tech/ksail#8391 is parked on this issue rather than re-run again.

  13. added a commit that references this issue on Nov 15, 2026
  14. 47 remaining items

  15. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Post-merge publication follow-up for #950 is complete at d70d50665465117f01104313e53057374a96a1a3: main CI, manifest publication, Code Quality, security analysis, and repository drift verification all completed successfully.

    This closes the unfinished workflow collection in the previous handoff. It does not establish live manifest reconciliation or affected-consumer scanner acceptance: the catalogue's production-interface vulnerability job was skipped, so its green CI is not a consumer scan. No new KSail workflow run was returned for the interval beginning at this merge.

    Keep #671 open. The next proof remains a native affected-consumer scan bound to the new shared workflow revision, with the owned swap setup and cleanup and the enforced finding verdict. Historical runner-loss attribution remains separate and unverified.

  16. added a commit that references this issue on Nov 16, 2026
  17. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    An affected consumer has now exercised the published repair. Platform required Go run 43811790996, attempt 1 / scanner job 133115087955 completed successfully for source 2304047d95a5e392cac7708bc4ffb696c5eb4986 after #950 merged.

    The native log binds both the shared scanner checkout and action execution to d70d50665465117f01104313e53057374a96a1a3. It records 4 GiB swap creation and activation, scanner exit 0, the evaluator's no-blocking-vulnerabilities verdict, and the owned swap retirement step executing swapoff and removal successfully. The timeout-only retry was skipped. Scan diagnostics recorded 5.47 seconds and peak RSS 2,169,240 KiB.

    This proves the new lifecycle ran on Platform's ordinary required-workflow path. Its smaller scan does not establish that swap prevents KSail's runner losses, nor attribute historical interruptions. Keep #671 open for affected KSail acceptance and bounded attribution evidence; the original acceptance criteria are unchanged.

  18. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    A KSail consumer still loses its hosted runner after #950. I re-ran the failed vulnerability job of ksail#8391 at 7fc93dd3 (a change that does not touch the scanned code). Run 43735166748, attempt 3, job 133168311777 ran on the merged workflow: "Prepare bounded swap headroom for the scanner" succeeded. The scan step then ran from 08:06 to 08:54 UTC and ended with "The hosted runner lost communication with the server", with no scan verdict, no timeout retry and no swap retirement.

    So the 4 GiB swap fixed the smaller Platform scan (05:45 note above), but not the KSail one. KSail is the larger module graph (about 14.6 GiB peak measured earlier against a 16 GB runner). The next options are more headroom, a larger runner for this one consumer, or narrowing what the scan loads.

  19. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Two cheaper routes for KSail's scan were measured on KSail main at cc0650bf5 with the reviewed scanner (govulncheck v1.8.0, Go 1.27.4). Neither fixes the runner losses.

    1. Run a package-level scan first and skip the expensive scan when it is clean. A symbol-level finding always has a package-level finding too, so a clean package-level result would prove the full scan clean at almost no cost. The package-level scan took 3.7 s and peaked at about 200 MiB. But KSail imports 15 packages that have advisories (for example GO-2025-3521 and GO-2026-6905), so the full symbol-level scan would still run on every KSail change. This route does not help KSail.

    2. Split ./... into smaller scans. The scanner's memory grows with the packages it loads. go list -deps shows that pkg/cli alone loads 5,936 of the module's 5,985 packages, so every split that includes the CLI loads almost the whole graph again. This route does not help either.

    What is left — each needs a decision this issue has not made yet:

    • A runner with more memory for this one consumer. It could be a self-hosted organisation runner (see feat(ci): activate bounded organization runners for KSail preflight platform#4544, currently parked) or a paid larger hosted runner. The paid runner is a spending decision.
    • A binary-mode scan of the built ksail executable. It uses much less memory, but it covers only the code linked into the shipped binary, not every package. That changes what this required check covers, so it needs its own review against the gate's purpose.
    • A smaller dependency graph in KSail itself. This is product work in ksail and cannot be done quickly.

    Parked PRs that wait on this issue, such as devantler-tech/ksail#8391, keep their current re-entry condition.

  20. added a commit that references this issue on Nov 16, 2026
  21. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    New default-branch interruption: KSail run 43898923987, attempt 1 / vulnerability job 133410590872, source fc4e3d24c8d814fe6a058e87b55841788cb5267e, failed after the hosted runner lost communication. The native scan step ran from 15:41:59 to 15:53:26 UTC on 16 November, then was cancelled after 11m27s. The timeout classifier and retry were skipped. This is no completed scan verdict and does not establish an OOM or a configured timeout.

    The attempt's native referenced_workflows binds this invocation to validate-go-project.yaml@ac1f346773da6f47444c283927fb1ce31187fe21. That immutable workflow lacks the scanner swap preparation/retirement added by #950; current shared main includes both. This failure therefore does not test the #950 mitigation, and source publication alone does not establish adoption by every caller. The failed job log returned HTTP 404, and the run's complete two-artifact listing contained no scanner diagnostics, so causal attribution remains unavailable.

    One manual failed-jobs retry was accepted as attempt 2 at the same KSail source; its vulnerability job 133455425099 was observed running. No further retry is armed. Collect that native result once, then advance the existing consumer rollout and bounded attribution work. Keep the original acceptance criteria open; neither a retry nor earlier successful descendant scans clear this new incomplete scan.

  22. added a commit that references this issue on Nov 16, 2026
  23. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The single failed-jobs retry of KSail run 43898923987, attempt 2 has completed with failure at the same source, fc4e3d24c8d814fe6a058e87b55841788cb5267e. Its vulnerability job 133455425099 reports another hosted-runner communication loss. The native scan step was cancelled from 16:50:20 to 17:01:16 UTC on 16 November: 10m56s, before its configured 24-minute deadline. This is an incomplete scan, with no vulnerability verdict and no proven OOM or timeout cause.

    The run's immutable workflow reference is still validate-go-project.yaml@ac1f346773da6f47444c283927fb1ce31187fe21, which lacks the scanner swap lifecycle. The complete artifact listing still contains only the patch and Calico binary artifacts, with no scanner diagnostics. Neither attempt tests the mitigation released in shared workflow v21.7.2.

    No further manual retry is armed. The next route is the existing KSail adoption PR #8420, currently draft at ab559283c580aed2c14a902306639fa0877ef03b with system tests pending. Finish its current-head readiness, then verify a native main invocation actually uses the released workflow and completes the scan and swap cleanup. KSail #8419 retains that acceptance; this investigation retains its unresolved attribution criterion.

  24. added 2 commits that reference this issue on Nov 17, 2026
  25. devantler commented on Nov 17, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The adoption handoff above is superseded: KSail #8420 is merged and KSail #8419 has completed its native acceptance. The historical runner-loss investigation remains open here.

    Independent readback of KSail main CI 43976765109 at 1f11639aa011dfce2b2971edaae1adf9ebc0daf7 confirms the actual invocation references validate-go-project.yaml@d70d50665465117f01104313e53057374a96a1a3, which contains the bounded swap mitigation. Its native vulnerability job 133780587573 completed successfully:

    • The first scan ran from 23:10:35 to 23:23:03 UTC on 16 November and reported no blocking vulnerabilities. Both timeout-retry steps were skipped.
    • Preparation created and verified this job's 4 GiB swap file as active. The always-run retirement step completed successfully for the same file, with native execution evidence for swapoff.
    • The overall main run is now successful after its single failed-jobs retry; the source and reviewed workflow binding are unchanged.

    This establishes adoption and a completed scan with the mitigation. It does not attribute the earlier runner losses, prove an OOM, or prove the interruption cannot recur. Preserve the unfinished causal-evidence criterion. The next investigation step is bounded runner/resource evidence from an actual interruption under the adopted workflow, rather than another adoption PR or an unbounded retry. No further retry was requested by this verification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions