From 0acf5063685226be190e32879b22feedc37db29c Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 11:41:36 +0000 Subject: [PATCH 01/34] Restore CodeQL coverage via advanced setup, with a Go analysis that works MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Default setup was switched off, so nothing was being scanned at all. It had to go because its Go analysis could not be repaired from anywhere: it installed the Go its extractor was built with (1.26.6) and pinned GOTOOLCHAIN=local, so against a go.mod requiring 1.27 it could neither build nor fetch a usable toolchain: go: go.mod requires go >= 1.27.0 (running go 1.26.6; GOTOOLCHAIN=local) Failed to run `go mod tidy -e` in . Extraction failed for all discovered Go projects. Every main-targeting PR carried that failure from the Go 1.27 bump (#1240, promoted in #1242) on, so the repo's Go code went unscanned for that whole stretch. Default setup is zero-config and exposes no Go version input, hence advanced setup. .github/workflows/codeql.yml restores both analyses default setup ran — security over go, python, javascript-typescript and actions; code quality over go, python and javascript-typescript — and gives Go an interpreter via actions/setup-go with go-version-file: go.mod, so the version keeps one declaration point instead of a copy that goes stale. The three languages in both sets pass analysis-kinds: code-scanning,code-quality, building one database and running both suites over it rather than extracting twice. Triggers are push on main and pull_request on main and dev. Covering dev PRs is the one place this exceeds default setup, which never ran on them: every change lands on dev first and main only receives promote merges, so scanning main alone surfaces a finding for the first time on a promote commit. A weekly Monday 10:00 UTC cron is offset from the three existing scheduled lanes. dev-ci.yml's header listed CodeQL among the checks deferred to the dev->main gate. That is no longer true, so it now says where CodeQL runs instead. The three existing upload-sarif calls are untouched. Signed-off-by: Brad Flaugher --- .github/workflows/codeql.yml | 141 +++++++++++++++++++++++++++++++++++ .github/workflows/dev-ci.yml | 11 ++- 2 files changed, 149 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/codeql.yml diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 00000000..30d6f948 --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,141 @@ +# CodeQL — ADVANCED setup. +# +# This file replaces GitHub's zero-config "default setup", which was switched off +# because its Go analysis could not be fixed from anywhere. Default setup +# installed the Go toolchain its extractor was built with (1.26.6) and pinned +# `GOTOOLCHAIN=local`, so on a `go.mod` that requires 1.27 it could neither build +# that toolchain nor download it: +# +# go: go.mod requires go >= 1.27.0 (running go 1.26.6; GOTOOLCHAIN=local) +# Failed to run `go mod tidy -e` in . +# Extraction failed for all discovered Go projects. +# CodeQL job status was configuration error. +# +# Every main-targeting PR had that failure from the Go 1.27 bump (#1240, promoted +# in #1242) onward, so the repo had no CodeQL coverage of its Go code for that +# whole stretch. Default setup exposes no knobs — there is no place to set a Go +# version or an env var — hence advanced setup, where `actions/setup-go` reads +# `go-version-file: go.mod` like every other lane in this repo. +# +# The literal-version rule applies here too: `scripts/check_versions_test.go`'s +# TestWorkflowsDeclareVersionsByFile scans EVERY .github/workflows/*.yml for a +# literal `go-version:`/`node-version:` and fails on it. That assertion is +# directory-wide, so this file was covered by it the moment it was added — no +# change to the test was needed. Do not replace the version-file input with a +# hardcoded 1.27; that is the exact bug class #1240 and #1241 already fixed twice +# for node, and pinning it here would go stale silently the next time go.mod moves. +# +# What this restores. Default setup ran two analyses per event: +# - security — go, python, javascript-typescript, actions +# - code quality — go, python, javascript-typescript (no `actions`) +# Both are reproduced below. `analysis-kinds` on init takes a list, so the three +# languages in both sets build ONE database and run both query suites over it, +# rather than extracting the same code twice as default setup did. `actions` gets +# code-scanning only, matching what default setup covered. +# +# Not touched by this file: the three independent `upload-sarif` calls (ci.yml's +# Grype step, govulncheck-scheduled.yml, grype-scheduled.yml). Those push their +# own SARIF to the Security tab and never depended on CodeQL being configured. +# +# Merge gating: these jobs are NOT part of ci.yml's `CI gate`, which is the single +# required status check on main (see .github/CODEOWNERS). CodeQL findings are +# therefore advisory today, exactly as they were under default setup. Wiring them +# into the gate is a branch-protection decision, not a workflow one — see +# docs/CODEQL.md. +name: CodeQL + +on: + push: + branches: [main] + pull_request: + # main mirrors ci.yml. dev is DELIBERATELY added, and is the one place this + # file covers more than default setup did (verified: zero CodeQL runs ever + # recorded on a dev-targeting PR — see docs/CODEQL.md). Every change lands on + # dev first; main only ever receives a promote merge. Scanning main alone + # means a finding surfaces for the first time on a squashed promote commit, + # which is the same "the integration branch is where it is first attempted" + # complaint dev-ci.yml already makes about compilation. + branches: [main, dev] + schedule: + # Monday 10:00 UTC, weekly. Offset from the 07:00 canary, the 08:00 daily + # govulncheck and the Monday 09:00 Grype scan, following the same + # don't-contend-for-runners reasoning those files state. + # + # A cron matters for CodeQL specifically because a run's verdict is a + # function of the query pack as well as the commit: new queries ship + # continuously, and without a schedule the only way this repo learns that a + # newly published query flags existing code is that some unrelated PR turns + # red. Same argument govulncheck-scheduled.yml makes about vuln.go.dev. + - cron: '0 10 * * 1' + +permissions: + contents: read + security-events: write # required to upload CodeQL results to the Security tab + # The `actions` analysis reads workflow definitions; `packages: read` is not + # needed because no analysis here pulls a private CodeQL pack. + actions: read + +concurrency: + group: codeql-${{ github.event_name }}-${{ github.ref }} + # Cancel superseded PR runs, but never a push-on-main or scheduled run: those + # produce the alert set of record for the default branch, and a cancelled run + # leaves the previous, staler alerts standing. + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +jobs: + analyze: + name: Analyze (${{ matrix.language }}) + runs-on: ubuntu-latest + timeout-minutes: 30 + + strategy: + fail-fast: false + matrix: + include: + # Go is the reason this file exists. `build-mode: none` is NOT + # supported for Go — only `autobuild` or `manual` — so the toolchain + # has to be right rather than skipped. + - language: go + build-mode: autobuild + analysis-kinds: code-scanning,code-quality + - language: python + build-mode: none + analysis-kinds: code-scanning,code-quality + - language: javascript-typescript + build-mode: none + analysis-kinds: code-scanning,code-quality + # `actions` was in default setup's security analysis only, not its code + # quality one. Matched rather than widened. + - language: actions + build-mode: none + analysis-kinds: code-scanning + + steps: + - name: Checkout + uses: actions/checkout@v7 + + # THE FIX. Runs before init so the interpreter is already on PATH when the + # Go extractor and autobuild shell out to `go`. go-version-file makes go.mod + # the single declaration point, so this cannot drift from the module. + - name: Set up Go + if: matrix.language == 'go' + uses: actions/setup-go@v7 + with: + go-version-file: go.mod + cache: true + + - name: Initialize CodeQL + uses: github/codeql-action/init@v4 + with: + languages: ${{ matrix.language }} + build-mode: ${{ matrix.build-mode }} + analysis-kinds: ${{ matrix.analysis-kinds }} + + - name: Autobuild + if: matrix.build-mode == 'autobuild' + uses: github/codeql-action/autobuild@v4 + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v4 + with: + category: /language:${{ matrix.language }} diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index 32831baa..ca6ce890 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -17,9 +17,14 @@ # # Still deliberately deferred to the dev→main PR's full ci.yml gate, because each # is slow and none of them is what a routine change breaks: the -race lane, -# govulncheck, the Grype image scan, both Playwright suites, and CodeQL. The -# division of labour is "does it compile, lint, and pass tests" here; "is it safe -# to ship" there. +# govulncheck, the Grype image scan, and both Playwright suites. The division of +# labour is "does it compile, lint, and pass tests" here; "is it safe to ship" +# there. +# +# CodeQL used to be on that deferred list and no longer is. It does not run in +# this file — it has its own workflow, .github/workflows/codeql.yml, whose +# `pull_request` trigger covers dev as well as main. So a PR into dev IS +# CodeQL-scanned; it just is not scanned by this lane. See docs/CODEQL.md. # # `Dev gate` is the single aggregate job (same pattern as ci.yml's `CI gate`): # when the dev branch is protected, require just that one check. From 7731615077bf11aac6451a0a6870c2fc1eab030a Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 11:48:45 +0000 Subject: [PATCH 02/34] Use `queries: code-quality`, and give the extractor the host-executor tag MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects in the first cut, both found by reading the run log rather than the check mark — the first run was green with neither working. 1. `analysis-kinds: code-scanning,code-quality` does not do what it looks like. The action logged two ##[error] lines and then exited 0: The `analysis-kinds` input is experimental and for GitHub-internal use only. [...] An analysis kind other than `code-scanning` was specified in a custom workflow. This is not supported and will become a fatal error in a future version of the CodeQL Action. If your intention is to use quality queries outside of Code Quality, use the `queries` input with `code-quality` instead. [...] Specifying multiple values as input is no longer supported. Continuing with only `analysis-kinds: code-scanning`. Confirmed in the artifacts, not just the warning: the Go job loaded only codeql/go-queries and uploaded a single go.sarif, so the code-quality half of the coverage this change claims to restore was not running at all. Switched to `queries: code-quality` as the message directs. Code quality as a distinct analysis KIND stays closed to custom workflows; the quality queries themselves now run, surfacing as ordinary code-scanning alerts. 2. Go extraction had exactly one hole, and it was the worst possible file. The extractor reported 426 files against 427 non-test .go files in the tree; the missing one was internal/sandbox/host.go, the unsandboxed host executor, fenced behind `//go:build fleet_host_executor` and therefore absent from the default build. ci.yml and dev-ci.yml both pass that tag to `go vet` and `go test` precisely so it is not unchecked. GOFLAGS on the autobuild step passes it here for the same reason. Also recorded in the file's header: GOTOOLCHAIN=local is set by the codeql-action itself, not by the generated default-setup workflow as was assumed. The fix works because setup-go makes the LOCAL toolchain 1.27.0, satisfying go.mod, not because the pin is gone. Signed-off-by: Brad Flaugher --- .github/workflows/codeql.yml | 56 +++++++++++++++++++++++++++++------- 1 file changed, 46 insertions(+), 10 deletions(-) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 30d6f948..c2979c87 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -28,10 +28,32 @@ # What this restores. Default setup ran two analyses per event: # - security — go, python, javascript-typescript, actions # - code quality — go, python, javascript-typescript (no `actions`) -# Both are reproduced below. `analysis-kinds` on init takes a list, so the three -# languages in both sets build ONE database and run both query suites over it, -# rather than extracting the same code twice as default setup did. `actions` gets -# code-scanning only, matching what default setup covered. +# +# The security half is reproduced exactly. The code-quality half is reproduced as +# far as advanced setup is permitted to: `code-quality` as a distinct ANALYSIS +# KIND is GitHub-internal and closed to custom workflows. Passing +# `analysis-kinds: code-scanning,code-quality` was tried first and the action +# rejected it — while still exiting 0, which is why this was caught by reading +# the log rather than the check mark: +# +# The `analysis-kinds` input is experimental and for GitHub-internal use +# only. [...] An analysis kind other than `code-scanning` was specified in a +# custom workflow. This is not supported and will become a fatal error in a +# future version of the CodeQL Action. If your intention is to use quality +# queries outside of Code Quality, use the `queries` input with +# `code-quality` instead. +# [...] Specifying multiple values as input is no longer supported. +# Continuing with only `analysis-kinds: code-scanning`. +# +# So `queries: code-quality` is used instead, exactly as that message directs: the +# quality QUERY SUITE runs on the same three languages, added to the default +# security suite over one shared database. The difference from default setup is +# presentational, not coverage — the quality findings arrive as ordinary +# code-scanning alerts instead of populating the separate Code Quality +# experience, which no custom workflow can feed. See docs/CODEQL.md. +# +# `actions` gets no `queries` value, matching default setup, which ran `actions` +# in its security analysis only. # # Not touched by this file: the three independent `upload-sarif` calls (ci.yml's # Grype step, govulncheck-scheduled.yml, grype-scheduled.yml). Those push their @@ -97,18 +119,19 @@ jobs: # has to be right rather than skipped. - language: go build-mode: autobuild - analysis-kinds: code-scanning,code-quality + queries: code-quality - language: python build-mode: none - analysis-kinds: code-scanning,code-quality + queries: code-quality - language: javascript-typescript build-mode: none - analysis-kinds: code-scanning,code-quality + queries: code-quality # `actions` was in default setup's security analysis only, not its code - # quality one. Matched rather than widened. + # quality one. Matched rather than widened: no `queries` value, so only + # the default security suite runs. - language: actions build-mode: none - analysis-kinds: code-scanning + queries: '' steps: - name: Checkout @@ -129,11 +152,24 @@ jobs: with: languages: ${{ matrix.language }} build-mode: ${{ matrix.build-mode }} - analysis-kinds: ${{ matrix.analysis-kinds }} + # Empty for `actions` (security suite only), `code-quality` elsewhere. + # Added to the default security suite, not a replacement for it. + queries: ${{ matrix.queries }} - name: Autobuild if: matrix.build-mode == 'autobuild' uses: github/codeql-action/autobuild@v4 + env: + # Same build tag ci.yml and dev-ci.yml pass to `go vet` and `go test`, + # and for the same reason: internal/sandbox/host.go — the UNSANDBOXED + # host executor — is fenced behind `//go:build fleet_host_executor`, so + # without the tag it is not in the default build and the extractor never + # sees it. Measured on the first run of this workflow: 426 of the 427 + # non-test .go files were extracted, and the one missing file was + # exactly host.go. Leaving the most security-sensitive file in the tree + # as the single hole in Go coverage is not a defensible default, so the + # tag is passed here to match the lanes that already vet it. + GOFLAGS: -tags=fleet_host_executor - name: Perform CodeQL Analysis uses: github/codeql-action/analyze@v4 From 59d4bda6cb61a39d37557713fe3de74a5d41d5c6 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 11:59:12 +0000 Subject: [PATCH 03/34] Ship the CodeQL design note MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/CODEQL.md records what shipped, what deviated from a like-for-like restoration of default setup, and what was deliberately left out — with the verified/assumed split stated explicitly, since the last two bugs in this area both shipped because a toolchain layout was assumed rather than observed. Verified and quoted from the run's own log archive: extraction succeeded for both discovered Go projects, 916 packages, 426 .go files including host.go, and distinct queries evaluated per language across the two runs (72->116 go, 90->292 python, 178->374 javascript-typescript, 36->36 actions). Also corrected there: GOTOOLCHAIN=local is set by the codeql-action itself, in four steps of our own Go job, not by the generated default-setup workflow. The fix works because setup-go makes the local toolchain 1.27.0 — "give `local` something good enough", not "unset the pin". Stated as NOT verified: no push-on-main or scheduled run has executed yet; test files stay outside the database (unchanged from default setup, not a regression); the lines-of-code metric value is never printed to the log, so no line count is claimed; and build-mode: manual was not built because autobuild works. CHANGELOG entry and an AGENTS.md "Where to look" pointer alongside. Signed-off-by: Brad Flaugher --- AGENTS.md | 4 + CHANGELOG.md | 41 +++++++ docs/CODEQL.md | 326 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 371 insertions(+) create mode 100644 docs/CODEQL.md diff --git a/AGENTS.md b/AGENTS.md index 6ae8fdf1..41b7fd03 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -161,6 +161,10 @@ same PR. - **Contributor workflow + CI gates:** [`CONTRIBUTING.md`](CONTRIBUTING.md) - **Testing strategy** (unit / fake-LLM / mocked + live Playwright / canary): [`docs/TESTING.md`](docs/TESTING.md) +- **CodeQL** (why default setup was replaced by an advanced-setup workflow, how + the Go toolchain is resolved, why `code-quality` is a query suite here rather + than an analysis kind, and why CodeQL is advisory rather than in `ci-gate`): + [`docs/CODEQL.md`](docs/CODEQL.md) - **HTTP API versioning** (the `/v1` prefix + `X-Fleet-API-Version` + `/api-info` discovery + deprecation contract): [`docs/api-versioning.md`](docs/api-versioning.md) - **Database migrations** (the two runners, safe-DDL patterns, the migration DDL diff --git a/CHANGELOG.md b/CHANGELOG.md index 4734038c..eb963ab9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,47 @@ prior versions are listed because none have shipped. ### Fixed +- **CodeQL had stopped analyzing the repo's Go code, and then stopped analyzing + anything.** Default setup's Go analysis failed on every main-targeting PR from + the Go 1.27 bump (#1240, promoted in #1242) onward — it installed the Go its + extractor was built with and pinned `GOTOOLCHAIN=local`, so against a `go.mod` + requiring 1.27 it could neither use nor fetch a workable toolchain: + + ``` + go: go.mod requires go >= 1.27.0 (running go 1.26.6; GOTOOLCHAIN=local) + Extraction failed for all discovered Go projects. + CodeQL job status was configuration error. + ``` + + The failure was red but never blocking (`ci-gate` is the only required check on + main), so it was annotated in promote commit messages and lived on. Default + setup is zero-config, with no Go version input and no env, so there was nothing + to fix in place; switching it off to replace it left the repo scanning nothing + at all in the interim. + + Replaced with an advanced-setup workflow, `.github/workflows/codeql.yml`, which + restores security analysis over go, python, javascript-typescript and actions + plus the code-quality query suite over the first three, and resolves Go's + interpreter from `go.mod` via `actions/setup-go` — never a literal version, the + bug class #1240 and #1241 already fixed twice for node. + + Two things the first cut got wrong, both of which ran **green**: `analysis-kinds` + turns out to be GitHub-internal and unusable in a custom workflow (it logged + `##[error]` and silently continued with security only), and Go extraction + missed exactly one file — `internal/sandbox/host.go`, the unsandboxed host + executor, invisible to the default build behind `//go:build + fleet_host_executor`. Fixed with `queries: code-quality` and + `GOFLAGS: -tags=fleet_host_executor`, the same tag `ci.yml` and `dev-ci.yml` + already pass to `go vet` and `go test`. + + Verified from the extractor's own output rather than the check mark: + `extraction succeeded for all 2 discovered project(s)`, 916 packages, 426 `.go` + files including `host.go`, and distinct queries evaluated rising from 72→116 + (go), 90→292 (python) and 178→374 (javascript-typescript) as the quality suite + came in, with `actions` unchanged at 36 by design. CodeQL remains advisory — + these jobs are deliberately not wired into `ci-gate`. See + [`docs/CODEQL.md`](docs/CODEQL.md). + - **`fleet update` built the web tier on the node it had just refused.** Every update on a Fedora box printed `✓ web tier will build+run on /usr/bin/node-24 (v24.x)` and then, a few lines later, npm's own rejection of that claim: diff --git a/docs/CODEQL.md b/docs/CODEQL.md new file mode 100644 index 00000000..1456c44d --- /dev/null +++ b/docs/CODEQL.md @@ -0,0 +1,326 @@ +# CodeQL: advanced setup, and the Go analysis that had stopped working + +Design note for the change that replaced GitHub's zero-config CodeQL "default +setup" with an advanced-setup workflow, `.github/workflows/codeql.yml`. + +Companion reading: [`NODE-TOOLCHAIN-HANDOFF.md`](NODE-TOOLCHAIN-HANDOFF.md) — +this is the same failure family (a toolchain version that had a second, +unreconciled copy) and the same rule applies: one declaration point per version, +asserted rather than remembered. [`TESTING.md`](TESTING.md) describes the rest of +the CI ladder. + +## What was broken + +Default setup's Go analysis failed on every main-targeting PR from the Go 1.27 +bump (#1240, promoted in #1242) onward. From PR #1245, job 96968001636: + +``` +Setup go version spec 1.26 +Found in cache @ /opt/hostedtoolcache/go/1.26.6/x64 +... +Run github/codeql-action/autobuild@v4 + env: GOTOOLCHAIN: local +... +go: go.mod requires go >= 1.27.0 (running go 1.26.6; GOTOOLCHAIN=local) +Failed to run `go mod tidy -e` in . +make: *** [Makefile:72: compile] Error 1 +Error running go tooling: exit status 1 +Extraction failed for all discovered Go projects. +CodeQL job status was configuration error. +``` + +Default setup installed the Go its extractor was built with (1.26.6) and ran +under `GOTOOLCHAIN=local`, so against a `go.mod` requiring 1.27 it could neither +use the local toolchain nor download a newer one. It also invokes `make`, which +puts this repo's Makefile on the autobuild path. + +The consequence is the part worth stating plainly: **the repo had no CodeQL +coverage of its Go code for that entire stretch.** The failure was loud (red +checks) but not blocking — `ci-gate` is the only required check on main — so it +was annotated in promote commit messages and lived on. + +Default setup is zero-config: it exposes no Go version input and no env, so there +was nothing to fix in place. Hence advanced setup. + +## A correction to the diagnosis + +The working assumption going in was that the `GOTOOLCHAIN=local` pin came from +GitHub's *generated* default-setup workflow, and that a plain advanced workflow +would therefore not have it at all. + +**That is wrong, and it was checked rather than assumed.** `GOTOOLCHAIN: local` +is set by `github/codeql-action` itself. It appears in the environment of four +separate steps in our own workflow's Go job — `Set up Go`, `Initialize CodeQL`, +`Autobuild`, and `Perform CodeQL Analysis`: + +``` +$ grep -rn 'GOTOOLCHAIN' 'Analyze (go)'/ +./3_Set up Go.txt line 74 +./4_Initialize CodeQL.txt line 15 +./5_Autobuild.txt line 10 +./6_Perform CodeQL Analysis.txt line 18 +``` + +So the fix is **not** "the pin is gone". The pin is still there. The fix is that +`actions/setup-go` makes the *local* toolchain 1.27.0, which is what `go.mod` +asks for, so `GOTOOLCHAIN=local` is satisfied instead of contradicted. The +autobuilder notices and proceeds: + +``` +Autobuilder was built with go1.26.5, environment has go1.27.0 +``` + +This matters for the next person: the mechanism is "give `local` something good +enough", not "unset the pin". Neither `env: GOTOOLCHAIN: auto` nor +`build-mode: manual` — the two fallbacks held in reserve — was needed. + +## What shipped + +`.github/workflows/codeql.yml`, one `analyze` job over a four-entry matrix. + +| language | build mode | quality queries | +| --- | --- | --- | +| `go` | `autobuild` | yes | +| `python` | `none` | yes | +| `javascript-typescript` | `none` | yes | +| `actions` | `none` | no | + +`build-mode: none` is [not supported for +Go](https://docs.github.com/en/code-security/reference/code-scanning/codeql/build-options-for-compiled-languages) +— only `autobuild` or `manual` — so Go's toolchain has to be correct rather than +skipped. + +**The Go interpreter is resolved from `go.mod`,** via `actions/setup-go` with +`go-version-file: go.mod`, never a literal. A hardcoded `1.27` here would be the +same bug class #1240 and #1241 already fixed twice for node. + +That rule is enforced, and the enforcement was **already directory-wide**: +`scripts/check_versions_test.go`'s `TestWorkflowsDeclareVersionsByFile` walks +every `.github/workflows/*.yml` and fails on any literal `go-version:` / +`node-version:`. So `codeql.yml` came under the assertion the moment it was +added — **no test change was needed.** Verified by breaking it on purpose rather +than by reading the regex: + +``` +$ sed -i "s|go-version-file: go.mod|go-version: '1.27.0'|" .github/workflows/codeql.yml +$ go test -count=1 -run TestWorkflowsDeclareVersionsByFile ./scripts +--- FAIL: TestWorkflowsDeclareVersionsByFile (0.00s) + check_versions_test.go:149: codeql.yml pins a literal version ("go-version: '1") + — use `node-version-file: web/.nvmrc` or `go-version-file: go.mod` so the + version has one declaration point +``` + +### Triggers + +```yaml +push: branches: [main] +pull_request: branches: [main, dev] +schedule: - cron: '0 10 * * 1' +``` + +`push` on `main` mirrors `ci.yml` and produces the alert set of record for the +default branch. `pull_request` on `main` matches what default setup covered. + +`dev` on `pull_request` is the **one place this covers more than default setup +did**, and the expansion is deliberate. Every change lands on `dev` first; `main` +only ever receives a promote merge. Scanning `main` alone means a finding +surfaces for the first time on a promote commit — the same complaint +`dev-ci.yml`'s own header already makes about compilation ("a branch whose job is +to integrate should not be where compilation is first attempted"). It is also +what made this change provable before merge: with main-only triggers, the first +real run of a workflow written to fix a silent-failure bug would have happened +*after* it merged. + +There is no `push` trigger on `dev`: a push to `dev` is the merge of a PR that +was just scanned, so it would re-analyze identical content. + +The weekly cron exists because a CodeQL verdict is a function of the query pack +as well as the commit — new queries ship continuously, and without a schedule the +only way this repo learns that a newly published query flags existing code is +that some unrelated PR turns red. That is the argument +`govulncheck-scheduled.yml` already makes about `vuln.go.dev`. Monday 10:00 UTC +is offset from the 07:00 canary, the 08:00 daily govulncheck and the Monday 09:00 +Grype scan, following the same don't-contend-for-runners note those files carry. + +`dev-ci.yml`'s header used to list CodeQL among the checks deferred to the +dev→main gate. That is no longer true, so it now says where CodeQL runs instead. + +## Two defects in the first cut, both green + +Both were found by reading the run log. Both runs of this workflow were **green** +while neither behaviour worked. This is the failure mode the change exists to +fix, so it is worth being concrete: a passing CodeQL job proves nothing about +what was analyzed. + +### 1. `analysis-kinds` is not available to us + +The first attempt passed `analysis-kinds: code-scanning,code-quality`, on the +reading that one init call could build one database and run both suites. The +action emitted two `##[error]` lines — **and exited 0**: + +``` +The `analysis-kinds` input is experimental and for GitHub-internal use only. +[...] An analysis kind other than `code-scanning` was specified in a custom +workflow. This is not supported and will become a fatal error in a future +version of the CodeQL Action. If your intention is to use quality queries +outside of Code Quality, use the `queries` input with `code-quality` instead. + +[...] Specifying multiple values as input is no longer supported. Continuing +with only `analysis-kinds: code-scanning`. +``` + +Confirmed in the artifacts rather than trusted from the warning: the Go job +loaded only `codeql/go-queries`, evaluated 72 distinct queries, and uploaded a +single `go.sarif`. The code-quality half of the coverage this change claims to +restore was not running at all. + +Fixed by using `queries: code-quality`, exactly as the message directs. + +**This is a real deviation from default setup, not a like-for-like restoration.** +Default setup ran code quality as a separate *analysis kind*, producing a second +analysis that feeds GitHub's Code Quality experience. That kind is +GitHub-internal and closed to custom workflows — no advanced-setup workflow can +feed it. What is restored is the code-quality **query suite**, added to the +default security suite over one shared database, with findings arriving as +ordinary code-scanning alerts. The queries all run; the presentation differs. + +### 2. Go extraction had one hole, and it was the worst file in the tree + +The first run extracted 426 files. The tree has 427 non-test `.go` files. The +missing one, by diffing the extractor's own file list against the checkout: + +``` +$ comm -13 extracted.txt local_nontest.txt +internal/sandbox/host.go +``` + +`internal/sandbox/host.go` is the **unsandboxed host executor** — bash and python +run directly on the host — fenced behind `//go:build fleet_host_executor` and so +absent from the default build. It is a CODEOWNERS-protected path, and both +`ci.yml` and `dev-ci.yml` deliberately pass that tag to `go vet` and `go test` +("Same tag as ci.yml so host.go [...] is vetted too") precisely so it is not +left unchecked. Leaving it as the single gap in Go coverage is not a defensible +default. + +`GOFLAGS: -tags=fleet_host_executor` on the autobuild step fixes it, matching the +lanes that already vet it. + +**The honest limit:** the total is still 426, because `host.go` and +`host_disabled.go` carry mutually exclusive build tags (`fleet_host_executor` and +`!fleet_host_executor`), so exactly one is ever in a build. The tag trades which +one CodeQL sees. That is a good trade — `host.go` is 410 lines of real +unsandboxed-execution logic, `host_disabled.go` is a 26-line refusal stub — but +it is a trade, not the elimination of a gap. Analyzing both would need two +databases. + +Note also that the extractor still logs `Build flags: ''`. `GOFLAGS` reaches the +`go` tooling through the environment, not through the extractor's own flag +plumbing, so the log line that looks like it should confirm the fix does not. +`host.go` appearing in the extracted list is what confirms it. + +## What was verified + +Measured on run 2 (`7731615`, run `32571297663`), by downloading the run's log +archive and reading the extractor's and evaluator's own output — not from the +check mark. + +**Go extraction really happened:** + +``` +Found 2 go.mod files in: go.mod, web/go.mod. +Done running go list deps: resolved 916 packages. +Done extracting .../internal/sandbox/host.go +Success: extraction succeeded for all 2 discovered project(s). +``` + +426 distinct `.go` files extracted, and the set differs from the tree's 427 +non-test files by exactly `host_disabled.go`, per the build-tag trade above. + +**Every language produced a database, ran queries, and uploaded results.** +Distinct queries evaluated, run 1 vs run 2 — the delta is the code-quality suite +arriving: + +| language | run 1 | run 2 | delta | SARIF | +| --- | --- | --- | --- | --- | +| `go` | 72 | 116 | +44 | `go.sarif` | +| `python` | 90 | 292 | +202 | `python.sarif` | +| `javascript-typescript` | 178 | 374 | +196 | `javascript.sarif` | +| `actions` | 36 | 36 | +0 | `actions.sarif` | + +`actions` is unchanged **by design** — default setup ran it in the security +analysis only, and that was matched rather than widened. The new Go query +directories are `RedundantCode` and `InconsistentCode`; JavaScript gains +`Quality`; Python gains `Classes`, `Exceptions`, `Functions`, `Imports`, +`Lexical`, `Resources`, `Statements`, `Testing` and `Variables`. All four jobs +logged `Successfully uploaded results`. + +**`web/` is in scope and contributes nothing, by design.** The autobuilder +discovers both `go.mod` files and extracts both projects. `web/` reports: + +``` +Running extractor command '.../go-extractor [./...]' from directory 'web'. +No packages found. +Done running go list deps: resolved 0 packages. +``` + +That is the correct outcome and matches `web/go.mod`'s own comment — it is a +no-package boundary module that exists to stop root `go ... ./...` traversing Go +source vendored inside `node_modules`. It was left in scope rather than excluded: +extraction of an empty module is free, and excluding it would need a config file +whose only job is to suppress something harmless. **Verified, not assumed** — +this is the specific claim the old autobuild failure ("Extraction failed for all +discovered Go projects") made it reasonable to worry about. + +**The local gate**, on this branch: `make build`, `make lint` (0 issues), +`make test` (exit 0), `make lint-migrations` (no changed migrations). `make lint` +needed `golangci-lint` v2.13.1 built with Go 1.27 — the installed 2.5.0 was built +with go1.25.1 and cannot lint the tree. + +## What was NOT verified, and what is deliberately out of scope + +- **No push-on-`main` or scheduled run has executed.** Both triggers are + unexercised until this merges and is promoted. They are ordinary trigger + syntax, and the `pull_request` path shares every step with them, but the cron + expression itself has not fired. It is a weekly cron, so its first real proof + is up to a week after promotion. +- **`_test.go` files are not analyzed.** 621 test files are outside the + database, because `autobuild` builds packages, not tests. Default setup did + not analyze them either, so this is not a regression — it is an unchanged + limit, stated because "CodeQL covers the Go code" would otherwise overclaim. + Bringing tests in would need `build-mode: manual`. +- **The lines-of-code metric value was not read.** `Summary/LinesOfCode.ql` + evaluates, but CodeQL does not print the number to the job log; it lands in a + `.bqrs`. File and package counts are what was actually observed, so they are + what is reported here. No line count is claimed. +- **Alert counts are not claimed.** This change restores *scanning*; what the + new quality queries find on this codebase is a separate question, and PR-run + file-coverage information is suppressed by CodeQL anyway ("To speed up pull + request analysis, file coverage information is only enabled when analyzing the + default branch and protected branches"). +- **`build-mode: manual` was not built.** `autobuild` works, so the more + complex option was not needed. If `autobuild` regresses, manual mode plus the + repo's own `go build ./...` is the fallback — and it is also the route to + analyzing test files. +- **The three existing `upload-sarif` calls were not touched** (`ci.yml`'s Grype + step, `govulncheck-scheduled.yml`, `grype-scheduled.yml`). They upload their + own SARIF independently of CodeQL configuration; breaking them would silently + drop CVE findings from the Security tab. + +## Merge gating — unchanged, and a decision left open + +These jobs are **not** wired into `ci-gate`, which is the single required status +check on `main` (see [`.github/CODEOWNERS`](../.github/CODEOWNERS)). CodeQL +findings are therefore advisory, exactly as they were under default setup: a red +CodeQL job does not block a merge. + +That is stated as the status quo, not a recommendation. Making CodeQL blocking is +a branch-protection change — a repo-settings click, not a workflow edit — and it +has a real cost worth weighing before anyone makes it: the analysis is advisory +today partly *because* it spent weeks red for a toolchain reason unrelated to any +diff, and a required check in that state blocks every merge. Wiring it into +`ci-gate` would also make `dev`-PR CodeQL failures block `dev`, which is a +heavier posture than that lane's stated "does it compile, lint, and pass tests" +job. + +No repo-settings or API change to code-scanning configuration was attempted as +part of this change. From 520fa41717e0cca9359c09c0a93cd06cb05b1d09 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 12:29:34 +0000 Subject: [PATCH 04/34] Add an aggregate `CodeQL gate` job so the lever is one check, not four MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeQL still is not blocking, and this commit does not make it blocking: `ci-gate` remains the only required status check on main, and requiring a check is a repo-settings action that a workflow file deliberately cannot perform. What it fixes is the shape of that future decision. CodeQL cannot be folded into ci.yml's `CI gate` at all — `needs` cannot reach across workflow files — so without this, making CodeQL required would mean naming `Analyze (go)`, `Analyze (python)`, `Analyze (javascript-typescript)` and `Analyze (actions)` individually in branch protection. That has to be re-pointed by hand every time the matrix gains or loses a language, and both ways of getting it wrong are bad: a required check that never reports again blocks every PR, and a removed one silently stops gating. One aggregate job has neither failure mode, and it is the pattern this repo already uses twice — ci.yml's `CI gate` and dev-ci.yml's `Dev gate`, including the same `if: always()` + join(needs.*.result) shape. `needs: [analyze]` on a matrix job collapses to a single aggregate result, so with fail-fast: false every language still runs and reports before the gate evaluates them. docs/CODEQL.md now also records where findings actually surface, since an empty Security tab is easy to misread as a broken pipeline: the alert list shown there is the DEFAULT BRANCH's, and this workflow's only push trigger is main, so it repopulates at a dev->main promotion rather than when a PR is scanned. PR runs report on the PR, where CodeQL additionally suppresses file-coverage detail. Also noted: the quality queries this change enables (+44 go, +202 python, +196 javascript-typescript) have never run against this codebase, so any pre-existing finding becomes an alert on merge — a reason to add `CodeQL gate` to the ruleset after a few green promotions rather than on day one. Signed-off-by: Brad Flaugher --- .github/workflows/codeql.yml | 41 +++++++++++++++-- docs/CODEQL.md | 85 +++++++++++++++++++++++++++++------- 2 files changed, 108 insertions(+), 18 deletions(-) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index c2979c87..ccfe4164 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -61,9 +61,16 @@ # # Merge gating: these jobs are NOT part of ci.yml's `CI gate`, which is the single # required status check on main (see .github/CODEOWNERS). CodeQL findings are -# therefore advisory today, exactly as they were under default setup. Wiring them -# into the gate is a branch-protection decision, not a workflow one — see -# docs/CODEQL.md. +# therefore advisory today, exactly as they were under default setup. +# +# They cannot be part of `CI gate`: a job's `needs` cannot reach across workflow +# files. So this file carries its own aggregate `CodeQL gate` job at the bottom, +# for the same reason ci.yml and dev-ci.yml carry theirs — it is the ONE check to +# name in branch protection if CodeQL should ever become blocking, instead of +# four per-language checks that would have to be re-pointed by hand every time +# the matrix changes. Adding it here does not make it required; that is a +# repo-settings decision, deliberately not expressible from this file. +# See docs/CODEQL.md ("Merge gating"). name: CodeQL on: @@ -175,3 +182,31 @@ jobs: uses: github/codeql-action/analyze@v4 with: category: /language:${{ matrix.language }} + + codeql-gate: + name: CodeQL gate + # Aggregate check for branch protection, same pattern and rationale as + # ci.yml's `CI gate` and dev-ci.yml's `Dev gate`: require this ONE check + # rather than naming each `Analyze ()` leg individually, so adding + # or removing a language from the matrix above does not silently leave a + # required check that never reports again. + # + # This job existing does NOT make CodeQL blocking. `ci-gate` is still the + # only required status check on main; this is the single lever to flip if + # that changes, and it deliberately cannot be flipped from this file. See + # docs/CODEQL.md ("Merge gating"). + # + # `needs: [analyze]` on a matrix job collapses to one aggregate result: + # success only when every leg succeeded. With `fail-fast: false` above, + # every language still runs and reports before this evaluates them. + if: always() + needs: [analyze] + runs-on: ubuntu-latest + steps: + - name: Fail if any CodeQL analysis did not succeed + run: | + results='${{ join(needs.*.result, ' ') }}' + echo "job results: $results" + for r in $results; do + [ "$r" = "success" ] || { echo "a CodeQL analysis did not succeed"; exit 1; } + done diff --git a/docs/CODEQL.md b/docs/CODEQL.md index 1456c44d..9f45ef75 100644 --- a/docs/CODEQL.md +++ b/docs/CODEQL.md @@ -306,21 +306,76 @@ with go1.25.1 and cannot lint the tree. own SARIF independently of CodeQL configuration; breaking them would silently drop CVE findings from the Security tab. -## Merge gating — unchanged, and a decision left open - -These jobs are **not** wired into `ci-gate`, which is the single required status -check on `main` (see [`.github/CODEOWNERS`](../.github/CODEOWNERS)). CodeQL -findings are therefore advisory, exactly as they were under default setup: a red -CodeQL job does not block a merge. - -That is stated as the status quo, not a recommendation. Making CodeQL blocking is -a branch-protection change — a repo-settings click, not a workflow edit — and it -has a real cost worth weighing before anyone makes it: the analysis is advisory -today partly *because* it spent weeks red for a toolchain reason unrelated to any -diff, and a required check in that state blocks every merge. Wiring it into -`ci-gate` would also make `dev`-PR CodeQL failures block `dev`, which is a -heavier posture than that lane's stated "does it compile, lint, and pass tests" -job. +## Merge gating — unchanged, with the lever put within reach + +CodeQL is **not** blocking. `ci-gate` remains the single required status check on +`main` (see [`.github/CODEOWNERS`](../.github/CODEOWNERS)), so a red CodeQL job +does not stop a merge — exactly as under default setup. + +It *cannot* be folded into `ci-gate`: a job's `needs` cannot reach across +workflow files. So `codeql.yml` carries its own aggregate **`CodeQL gate`** job, +mirroring `ci.yml`'s `CI gate` and `dev-ci.yml`'s `Dev gate`. That job is the one +deliberate piece of forward work here, and it is worth being clear that it +changes nothing on its own: + +- It does **not** make CodeQL required. Requiring a check is a repo-settings + action, deliberately not expressible from a workflow file. +- What it buys is that **flipping the switch later is one check, not four.** + Naming `Analyze (go)`, `Analyze (python)`, `Analyze (javascript-typescript)` + and `Analyze (actions)` individually in branch protection would mean + re-pointing branch protection by hand every time the matrix gains or loses a + language — and the failure mode of getting that wrong is the dangerous + direction: a required check that never reports again blocks every PR, or a + removed one silently stops gating. One aggregate check has neither problem. + +**To make CodeQL blocking** (owner action, not done here): Settings → Rules → the +"Main" ruleset → "Require status checks to pass" → add **`CodeQL gate`**. That +single entry covers every language in the matrix, now and after future matrix +changes. + +**Recommendation: leave it advisory for a short while first.** Not out of +caution for its own sake — because of this specific incident. The analysis spent +weeks red for a toolchain reason unrelated to any diff, and a required check in +that state blocks *every* merge, including the promote PR that would carry the +fix. Requiring it also means a `dev`-PR CodeQL failure blocks `dev`, a heavier +posture than that lane's stated "does it compile, lint, and pass tests" job. The +sequence with the least chance of self-inflicted deadlock is: merge this, watch a +few promotions go green, then add `CodeQL gate` to the ruleset. + +One more thing to know before requiring it: the newly-enabled quality queries +(+44 Go, +202 Python, +196 JavaScript) have **never run against this codebase +before**. If any of them fire, they become alerts the moment this merges — and +if `CodeQL gate` were required on day one, a pre-existing quality finding would +block merges. Findings are advisory only for as long as the gate stays optional, +which is another reason to sequence it that way. No repo-settings or API change to code-scanning configuration was attempted as part of this change. + +## Where findings appear + +`security-events: write` plus the analyze step's upload is the code-scanning +ingestion path, so results land in the repo's **Security → Code scanning**. From +the run log: + +``` +Adding fingerprints to SARIF file. See ... sarif-support-for-code-scanning ... +##[group]Uploading code scanning results +Uploading results +Successfully uploaded results +Analysis upload status is complete. +``` + +Two practical consequences worth stating, because they explain an empty-looking +Security tab rather than a broken one: + +- **The Security tab's alert list is the DEFAULT BRANCH's.** This workflow's only + `push` trigger is `main`, so that list refreshes when a promote merge lands on + `main` — not when a PR is scanned. +- **PR runs report on the PR**, not into the default-branch alert list, and + CodeQL additionally suppresses file-coverage detail there: *"To speed up pull + request analysis, file coverage information is only enabled when analyzing the + default branch and protected branches."* + +So after this merges to `dev`, expect findings on subsequent PRs; expect the +Security tab's `main` list to repopulate at the next dev→main promotion. From f90a42d038fbc65348d5911f21fad9473b5e36a9 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 12:35:46 +0000 Subject: [PATCH 05/34] Print CodeQL findings to the job log, and document the two gating levers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two gaps this closes, both surfaced by the question "can you see scanning results from the logs, and can you gate on this?" 1. A CodeQL run reports NOTHING about what it found to its own log. It writes SARIF, uploads it, and exits 0 — with findings or without. Grepping a full run's log archive for any alert or result count returns nothing; the only result-shaped lines are `Exporting results to SARIF...` and `Successfully uploaded results`, which say a file moved, not what was in it. That leaves a run's real outcome invisible to anyone reading CI output, to `gh run view`, and to any automation holding the log but not the code-scanning API. The analyze step now also writes SARIF locally via `output:` (results are still uploaded — `upload` defaults true) and a following step jq-summarizes per-rule counts into both the job log and the step summary. This is the pattern govulncheck-scheduled.yml already uses on its own SARIF. It is reporting only and never fails the job. When no SARIF was written it says so instead of printing "No findings.", because reporting a clean result you did not observe is the mistake this repo keeps recording. The jq was exercised against SARIF fixtures before pushing: findings spread over two files, a repeated ruleId, a result with no `level` key (falls back to note), an empty `results` array, a run with no `results` key, and a doc with no `runs` key. The last four all yield "No findings." rather than a jq error. 2. docs/CODEQL.md now states the distinction that makes "gate on CodeQL" ambiguous: a required status check on the job gates on the analysis having RUN, not on what it FOUND. A CodeQL job with a hundred open alerts still exits 0 and reports green — which is both why the toolchain break hid behind a red-but-not-required check, and why a green check is not evidence of a clean tree. Blocking on findings is a separate feature, code scanning merge protection (ruleset -> Code scanning rule -> tool CodeQL -> severity thresholds), which is available here at no cost because this repo is public. Signed-off-by: Brad Flaugher --- .github/workflows/codeql.yml | 53 +++++++++++++++++++++++++++++++ docs/CODEQL.md | 61 +++++++++++++++++++++++++++++++++--- 2 files changed, 110 insertions(+), 4 deletions(-) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index ccfe4164..5504d6da 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -182,6 +182,59 @@ jobs: uses: github/codeql-action/analyze@v4 with: category: /language:${{ matrix.language }} + # Also write the SARIF to disk so the step below can summarize it. The + # results are still uploaded to code scanning (`upload` defaults true); + # this only adds a local copy. + output: ${{ runner.temp }}/codeql-sarif + + - name: Summarize findings to the job log + # WHY THIS EXISTS: a CodeQL run reports nothing about what it FOUND to + # its own log. It writes SARIF, uploads it, and exits 0 — with findings + # or without them. So the only way to see results was to open the + # Security tab, which means a run's outcome is invisible to anyone + # reading CI output, to `gh run view`, and to any automation that has + # the log but not the code-scanning API. + # + # This mirrors what govulncheck-scheduled.yml already does for its own + # SARIF: jq a per-rule count into the job log AND the step summary, so + # the run is readable without leaving Actions. It is reporting only and + # never fails the job — gating on findings is code scanning merge + # protection's job (see docs/CODEQL.md), not this step's. + if: ${{ !cancelled() }} + env: + SARIF_DIR: ${{ runner.temp }}/codeql-sarif + LANGUAGE: ${{ matrix.language }} + run: | + set -uo pipefail + shopt -s nullglob + files=("$SARIF_DIR"/*.sarif) + { + echo "### CodeQL findings — $LANGUAGE" + if [ ${#files[@]} -eq 0 ]; then + # Not a failure: the analyze step is what fails on a broken run. + # Say it plainly rather than printing "No findings." for a scan + # that never produced a file — reporting a clean result you did + # not observe is the error this repo keeps writing down. + echo 'No SARIF file was written — nothing to summarize (see the analyze step).' + else + echo '```' + # `-s` slurps every SARIF doc into one array, so a language that + # emits more than one file is still counted once, in total. + jq -rs ' + [ .[] | .runs[]? | .results[]? ] as $res + | if ($res | length) == 0 then "No findings." + else + ( $res + | group_by(.ruleId) + | sort_by(-length) + | map("\(length | tostring) [\(.[0].level // "note")] \(.[0].ruleId)") + | join("\n") + ) + "\n--\ntotal findings: \($res | length)" + end + ' "${files[@]}" + echo '```' + fi + } | tee -a "$GITHUB_STEP_SUMMARY" codeql-gate: name: CodeQL gate diff --git a/docs/CODEQL.md b/docs/CODEQL.md index 9f45ef75..3b061078 100644 --- a/docs/CODEQL.md +++ b/docs/CODEQL.md @@ -306,7 +306,34 @@ with go1.25.1 and cannot lint the tree. own SARIF independently of CodeQL configuration; breaking them would silently drop CVE findings from the Security tab. -## Merge gating — unchanged, with the lever put within reach +## Two different things can gate, and they are not the same lever + +This distinction is the one most worth internalizing, because a status check on +the CodeQL job does **not** gate on findings: + +| you want to block a merge when… | the mechanism | where it lives | +| --- | --- | --- | +| the analysis **failed or did not run** | a required status check on `CodeQL gate` | branch protection / ruleset | +| CodeQL **found alerts** at/above a severity | **code scanning merge protection** | ruleset → "Code scanning" rule | + +The second is the one people mean by "gate on CodeQL", and the first does not +give it to you. **A CodeQL job with a hundred open alerts still exits 0 and +reports green** — the job's success only says extraction and query evaluation +worked. That is exactly why the toolchain break was able to hide for weeks behind +a red-but-not-required check, and equally why a green check is not evidence of a +clean codebase. + +fleet is a **public** repository, so code scanning merge protection is available +at no cost (on private repos it requires GitHub Advanced Security). To turn it +on: Settings → Rules → the "Main" ruleset → add the **Code scanning** rule → +add tool **CodeQL** → set the alert thresholds. Two independent knobs there: +*Security alerts* (the CWE/security queries) and *Alerts* (everything else, +which is where the code-quality suite lands). Sensible starting point given that +the quality queries have never run against this codebase: security threshold +**High or higher**, alerts threshold **None**, then tighten once the quality +backlog is known and burned down. + +## Merge gating today — unchanged, with the lever put within reach CodeQL is **not** blocking. `ci-gate` remains the single required status check on `main` (see [`.github/CODEOWNERS`](../.github/CODEOWNERS)), so a red CodeQL job @@ -352,11 +379,37 @@ which is another reason to sequence it that way. No repo-settings or API change to code-scanning configuration was attempted as part of this change. -## Where findings appear +## Where findings appear — and why the job log now says + +A CodeQL run reports **nothing about what it found** to its own log. It writes +SARIF, uploads it, exits 0, and the only lines resembling a result are +`Exporting results to SARIF...` and `Successfully uploaded results` — which say a +file moved, not what was in it. Verified by grepping a full run's log archive for +any alert or result count: there is none. + +That makes a run's actual outcome invisible to anyone reading CI output, to +`gh run view`, and to any automation holding the log but not the code-scanning +API. So the analyze step now also writes SARIF locally (`output:`) and a +following step jq-summarizes it into both the job log and the step summary — the +same thing `govulncheck-scheduled.yml` already does with its SARIF: + +``` +### CodeQL findings — go +2 [error] go/clear-text-logging +1 [warning] go/incomplete-hostname-regexp +1 [note] go/redundant-assignment +-- +total findings: 4 +``` + +It is reporting only and never fails the job; blocking on findings is merge +protection's job, above. When no SARIF was written it says so explicitly rather +than printing "No findings." — reporting a clean result you did not observe is +the error this repo keeps having to write down. `security-events: write` plus the analyze step's upload is the code-scanning -ingestion path, so results land in the repo's **Security → Code scanning**. From -the run log: +ingestion path, so results also land in the repo's **Security → Code scanning**. +From the run log: ``` Adding fingerprints to SARIF file. See ... sarif-support-for-code-scanning ... From 5ae15f16ecf293b037efc51a6521a069b0284143 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 12:59:45 +0000 Subject: [PATCH 06/34] Give each scanner one job: ruff owns Python, CodeQL owns taint, Semgrep owns Actions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reshapes the scanning stack on measurements rather than on "more scanners is better". docs/SCANNING.md is the full note; docs/CODEQL.md carries the CodeQL half. ruff is new, and it BLOCKS (ci.yml + dev-ci.yml `python` job, both wired into their gate jobs). fleet ships 13 Python files — the sandbox FileOp helper, the python bridge, the bento-slides and data-profiler skill scripts, MCP test servers — and nothing linted any of them: Go had golangci-lint, the web tier had oxlint, Python had neither. Rule selection is narrow and ruff.toml records the numbers behind that: default rules find 3 findings here, a broad selection finds 333, of which 176 are %-format style, 43 magic values, 35 line length. Gating on that would be a whole-tree reformat for no correctness gain. Three real findings were fixed so the gate is clean on day one and a new violation is a regression rather than backlog noise: an unused import in dummy_server.py, a lambda assignment in bento_pdf.py, and a byte-identical duplicate `has_guard` definition in bento_doc.py whose second copy silently shadowed the first. `ruff format` is reported but NOT gated — the tree has never been ruff-formatted, so failing on it would block every PR on a reformat nobody scheduled. CodeQL narrows to security queries only. The code-quality suite was enabled, measured, and dropped: 32 findings, every one note-level, ZERO security findings. For Go and the web tier it duplicates golangci-lint (gosec, staticcheck, revive, unparam, gocritic) and oxlint, which already block; 28 of the 32 were Python, which is ruff's job now and done in a second with autofix instead of ~40s without; and 3 were false positives on correct code (`value != value`, the idiomatic NaN test). What CodeQL keeps is the thing nothing else here can do — interprocedural taint, which is the actual shape of "a credential must not reach a log sink, the model context, or the sandbox". Semgrep is new, scoped, and advisory. Pointing it at p/golang, p/javascript and p/python was tried and rejected on evidence: 55 findings, and all 6 non-Actions findings were false positives. tls.go's open-redirect is an HTTP->HTTPS upgrade to the same host; runner.go's math/rand is jitter; elcano.go's cookie is a deletion cookie with no secret; httptool.go's interface{} is required because the value feeds a jq program; proxy.ts's X-Frame-Options value is the literal "DENY"; and fileops.py's advice — 0o644 for a sandbox directory — would be a security REGRESSION if followed. Three of the six were already formally triaged and suppressed for gosec, which already blocks. Re-reporting adjudicated findings is how a scanner teaches people to ignore it. What ships instead is p/github-actions, which found 51 instances of one real issue nothing else in this repo checks: actions pinned to a mutable tag rather than an immutable commit SHA, which runs attacker-controlled code with this repo's token if a tag moves. It is advisory because all 51 are real and repinning every workflow is its own PR — failing CI for an unscheduled backlog just trains people to ignore the lane. Flip continue-on-error off in the PR that repins. Both scanners now print a per-rule summary to the job log and the step summary, and Semgrep uploads its raw JSON as an artifact for a fixing agent to consume. Supporting changes: RUFF_VERSION is duplicated across ci.yml and dev-ci.yml, so scripts/check_versions_test.go now asserts the two agree (mutation-tested: the assertion fails when the pins diverge). `make lint` gains lint-python, which skips LOUDLY with the install command when ruff is absent rather than quietly doing nothing. dev-ci.yml's header and AGENTS.md's build/CI prose now name the Python lane, and AGENTS.md points at docs/SCANNING.md. Gate: make build, make lint (0 issues + ruff clean), make test (exit 0), make lint-migrations. All Python files still byte-compile and the bento golden tests pass. Signed-off-by: Brad Flaugher --- .github/workflows/ci.yml | 52 +++++- .github/workflows/codeql.yml | 54 +++--- .github/workflows/dev-ci.yml | 36 +++- .github/workflows/semgrep.yml | 159 ++++++++++++++++ AGENTS.md | 15 +- CHANGELOG.md | 45 +++++ Makefile | 19 +- docs/CODEQL.md | 123 +++++++++---- docs/SCANNING.md | 173 ++++++++++++++++++ .../bento-slides/scripts/bento_doc.py | 4 - .../bento-slides/scripts/bento_pdf.py | 4 +- internal/mcp/testdata/dummy_server.py | 1 - ruff.toml | 67 +++++++ scripts/check_versions_test.go | 1 + 14 files changed, 671 insertions(+), 82 deletions(-) create mode 100644 .github/workflows/semgrep.yml create mode 100644 docs/SCANNING.md create mode 100644 ruff.toml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b5a8ebdc..ab9e9001 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -316,6 +316,56 @@ jobs: # on a schedule instead of ambushing the next unrelated PR. run: go run golang.org/x/vuln/cmd/govulncheck@latest ./... + python: + name: Python lint (ruff) + runs-on: ubuntu-latest + needs: changes + # Skipped for a docs-only change; the `CI gate` job treats a skip as a pass. + if: ${{ needs.changes.outputs.docs_only != 'true' }} + # The lane the repo did not have. fleet ships 13 Python files — the sandbox + # FileOp helper, the python bridge, the bento-slides and data-profiler skill + # scripts, MCP test servers — and nothing linted any of them: Go had + # golangci-lint, the web tier had oxlint, Python had neither. Its only + # coverage was whatever CodeQL's code-quality suite happened to notice, which + # is a ~40s job with no autofix; ruff does the same class of check in well + # under a second. See ruff.toml for why the rule set is narrow (default rules + # find 3 issues here; the broad selection finds 333, almost all style churn). + steps: + - name: Checkout + uses: actions/checkout@v7 + + - name: Install ruff + # Pinned like every other tool this repo installs in CI, so an upstream + # release cannot change the verdict without a visible diff. + env: + RUFF_VERSION: '0.15.8' + run: | + set -euo pipefail + python3 -m pip install --user --quiet "ruff==${RUFF_VERSION}" + # GITHUB_PATH only affects LATER steps, so export for this one too. + echo "$HOME/.local/bin" >> "$GITHUB_PATH" + export PATH="$HOME/.local/bin:$PATH" + ruff --version + + - name: Lint + # Config (rule selection, excludes, per-file ignores) lives in ruff.toml + # so a local `ruff check .` and this gate cannot disagree. + run: ruff check --output-format github . + + - name: Formatting check (report only) + # NOT a gate. The tree has never been ruff-formatted, so failing on it + # would block every PR on a whole-tree reformat nobody has scheduled. + # Reported so the size of that decision stays visible instead of unknown. + if: ${{ !cancelled() }} + run: | + set -uo pipefail + { + echo '### ruff format (advisory — not a gate)' + echo '```' + ruff format --check --diff . 2>&1 | tail -40 || true + echo '```' + } | tee -a "$GITHUB_STEP_SUMMARY" + web: name: Web lint / test / build runs-on: ubuntu-latest @@ -715,7 +765,7 @@ jobs: # allowed (docs-only), but any failure or cancellation fails the gate. name: CI gate if: ${{ always() }} - needs: [changes, gitleaks, migrations, go, web, playwright, e2e-live, grype-scan] + needs: [changes, gitleaks, migrations, go, python, web, playwright, e2e-live, grype-scan] runs-on: ubuntu-latest steps: - name: Require all upstream jobs to have succeeded or been skipped diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 5504d6da..4c1cddb6 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -25,16 +25,29 @@ # hardcoded 1.27; that is the exact bug class #1240 and #1241 already fixed twice # for node, and pinning it here would go stale silently the next time go.mod moves. # -# What this restores. Default setup ran two analyses per event: -# - security — go, python, javascript-typescript, actions -# - code quality — go, python, javascript-typescript (no `actions`) +# SCOPE: security queries only. Default setup ran two analyses per event — +# security over go/python/javascript-typescript/actions, and code quality over +# the first three. This file restores the SECURITY half and deliberately drops +# the code-quality half. That is a considered narrowing, not an oversight, and +# the measurements behind it are recorded in docs/CODEQL.md. In short: # -# The security half is reproduced exactly. The code-quality half is reproduced as -# far as advanced setup is permitted to: `code-quality` as a distinct ANALYSIS -# KIND is GitHub-internal and closed to custom workflows. Passing -# `analysis-kinds: code-scanning,code-quality` was tried first and the action -# rejected it — while still exiting 0, which is why this was caught by reading -# the log rather than the check mark: +# - The quality suite was enabled first and measured. It produced 32 findings, +# every one of them note-level, and ZERO security findings across the whole +# tree. So nothing was being caught that mattered at severity. +# - For Go and the web tier it duplicates linters that already run AND ALREADY +# BLOCK via `ci-gate`: golangci-lint (gosec, staticcheck, revive, unparam, +# gocritic, …) and oxlint. `go/useless-assignment-to-field` is squarely +# inside golangci-lint's remit. +# - 28 of the 32 were Python, which was the one real gap — Python had no +# linter at all. That gap is now closed by ruff (see ruff.toml), which finds +# the same class of thing in under a second, with autofix, and blocks. A +# ~40s CodeQL job with no autofix is the wrong instrument for it. +# - 3 of the 32 were false positives on correct code (the idiomatic NaN test +# `value != value`, flagged as comparison-of-identical-expressions). +# +# Note for anyone tempted to re-add it: `analysis-kinds: code-scanning,code-quality` +# does NOT work in a custom workflow. The action logs two ##[error] lines and +# still exits 0, silently analyzing security only: # # The `analysis-kinds` input is experimental and for GitHub-internal use # only. [...] An analysis kind other than `code-scanning` was specified in a @@ -43,17 +56,8 @@ # queries outside of Code Quality, use the `queries` input with # `code-quality` instead. # [...] Specifying multiple values as input is no longer supported. -# Continuing with only `analysis-kinds: code-scanning`. -# -# So `queries: code-quality` is used instead, exactly as that message directs: the -# quality QUERY SUITE runs on the same three languages, added to the default -# security suite over one shared database. The difference from default setup is -# presentational, not coverage — the quality findings arrive as ordinary -# code-scanning alerts instead of populating the separate Code Quality -# experience, which no custom workflow can feed. See docs/CODEQL.md. # -# `actions` gets no `queries` value, matching default setup, which ran `actions` -# in its security analysis only. +# `queries: code-quality` is the working form. It is simply not wanted here. # # Not touched by this file: the three independent `upload-sarif` calls (ci.yml's # Grype step, govulncheck-scheduled.yml, grype-scheduled.yml). Those push their @@ -126,19 +130,12 @@ jobs: # has to be right rather than skipped. - language: go build-mode: autobuild - queries: code-quality - language: python build-mode: none - queries: code-quality - language: javascript-typescript build-mode: none - queries: code-quality - # `actions` was in default setup's security analysis only, not its code - # quality one. Matched rather than widened: no `queries` value, so only - # the default security suite runs. - language: actions build-mode: none - queries: '' steps: - name: Checkout @@ -159,9 +156,8 @@ jobs: with: languages: ${{ matrix.language }} build-mode: ${{ matrix.build-mode }} - # Empty for `actions` (security suite only), `code-quality` elsewhere. - # Added to the default security suite, not a replacement for it. - queries: ${{ matrix.queries }} + # No `queries:` — the default (security) suite only. See the SCOPE note + # in the header before adding `code-quality` back. - name: Autobuild if: matrix.build-mode == 'autobuild' diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index ca6ce890..aaac809b 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -11,9 +11,10 @@ # # What runs here: Go compile+vet+lint+test WITH a Postgres service (#723 — so the # DB-gated suites, including the entire `fleet import` suite, are exercised on dev -# instead of first firing at the dev→main promotion; no -race lane), the web -# lint/test/build lane, the migration DDL lint, and the gitleaks secret scan (the -# no-secrets invariant is never weakened, on any branch). +# instead of first firing at the dev→main promotion; no -race lane), the Python +# lint (ruff), the web lint/test/build lane, the migration DDL lint, and the +# gitleaks secret scan (the no-secrets invariant is never weakened, on any +# branch). # # Still deliberately deferred to the dev→main PR's full ci.yml gate, because each # is slow and none of them is what a routine change breaks: the -race lane, @@ -134,6 +135,33 @@ jobs: # compile with it. run: go test -p 1 -tags fleet_host_executor ./... + python: + name: Python lint (ruff) + runs-on: ubuntu-latest + # Mirrors ci.yml's `python` job exactly, for the same reason the web lane was + # added to this file: a change should not first be checked at the dev->main + # promotion. ruff takes about a second, so there is no speed argument for + # deferring it. + steps: + - name: Checkout + uses: actions/checkout@v7 + + - name: Install ruff + env: + # Keep pinned in lockstep with ci.yml (asserted by + # scripts/check_versions_test.go) so the fast lane and the full gate + # cannot disagree. + RUFF_VERSION: '0.15.8' + run: | + set -euo pipefail + python3 -m pip install --user --quiet "ruff==${RUFF_VERSION}" + echo "$HOME/.local/bin" >> "$GITHUB_PATH" + export PATH="$HOME/.local/bin:$PATH" + ruff --version + + - name: Lint + run: ruff check --output-format github . + web: name: Web lint / test / build (fast) runs-on: ubuntu-latest @@ -234,7 +262,7 @@ jobs: # Aggregate check for branch protection: passes only when every fast-lane # job succeeded (mirrors ci.yml's `CI gate`). if: always() - needs: [go, web, migrations, gitleaks] + needs: [go, python, web, migrations, gitleaks] runs-on: ubuntu-latest steps: - name: Fail if any fast-lane job failed diff --git a/.github/workflows/semgrep.yml b/.github/workflows/semgrep.yml new file mode 100644 index 00000000..9eabb465 --- /dev/null +++ b/.github/workflows/semgrep.yml @@ -0,0 +1,159 @@ +# Semgrep — the fast, log-readable scan lane. +# +# WHY A SECOND SCANNER AT ALL. CodeQL is the deep one: interprocedural taint, +# which is what actually checks this project's headline invariants (a credential +# must not reach a log sink, the model context, or the sandbox). It is also slow +# to build a database and reports nothing about what it found to its own log. +# Semgrep is the opposite trade — seconds, no build, findings straight to stdout, +# and rules that are cheap to write. It complements CodeQL; it does not replace +# it, and it must not be treated as a substitute for the taint analysis. +# +# RULESET SCOPE IS EVIDENCE-BASED, and the evidence is unflattering to the +# obvious choice. Running the broad registry packs (p/golang, p/javascript, +# p/python) over this tree produced 55 findings, and every one of the 6 +# non-Actions findings was a FALSE POSITIVE: +# +# go.lang.security.injection.open-redirect cmd/fleet/tls.go:109 +# Standard HTTP->HTTPS upgrade to the SAME host. Already carries a +# //nolint:gosec G710 with exactly this reasoning. +# go.lang.security.audit.crypto.math_random internal/runner/runner.go:28 +# math/rand/v2, used once, for +/-10% jitter on a retry interval. +# go...cookie-missing-secure internal/sched/handlers/elcano.go:155 +# A DELETION cookie (Value="", MaxAge=-1) carrying no secret; Secure is +# conditional so logout works over plain-HTTP dev. Already //nolint:gosec G124. +# go...unsafe-deserialization-interface internal/mcp/httptool.go:254 +# json.Unmarshal into interface{} is REQUIRED — the value feeds a jq program +# over arbitrary JSON. A concrete struct is not expressible. +# javascript...x-frame-options-misconfiguration web/src/proxy.ts:99 +# The header value is the literal string "DENY". No user input reaches it. +# python...insecure-file-permissions internal/sandbox/fileops.py:77 +# Advises 0o644 for a SANDBOX DIRECTORY, i.e. world-readable. Taking that +# advice would be a security regression; 0750 is the file-tool contract. +# +# Three of those six were already formally triaged and suppressed for gosec, +# which runs inside golangci-lint and already blocks. Re-reporting adjudicated +# findings is how a scanner trains people to ignore it, so those packs are NOT +# enabled here. If you want them, run them locally for a one-off audit. +# +# What IS enabled is the one pack that earned it: p/github-actions found 49 +# instances of a real class of issue nothing else in this repo checks — actions +# pinned to a MUTABLE tag (`actions/checkout@v7`) instead of an immutable commit +# SHA. A moved tag executes attacker-controlled code with this repo's token. +# +# ADVISORY, NOT BLOCKING, and deliberately so: those 49 findings are all real, +# so `--error` here would mean a red gate until every action in every workflow +# is repinned to a SHA. That is a worthwhile change and its own PR; failing CI +# for a backlog nobody has scheduled just teaches people to ignore the lane. +# Flip `continue-on-error` off in the same PR that does the repinning. +name: Semgrep + +on: + push: + branches: [main] + pull_request: + branches: [main, dev] + schedule: + # Monday 11:00 UTC — one hour after CodeQL's weekly, and clear of the 07:00 + # canary / 08:00 govulncheck / 09:00 Grype lanes, following the same + # don't-contend-for-runners note those files carry. Rules ship continuously, + # so like CodeQL this is worth re-running against unchanged code. + - cron: '0 11 * * 1' + +permissions: + contents: read + +concurrency: + group: semgrep-${{ github.event_name }}-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +jobs: + semgrep: + name: Semgrep scan + runs-on: ubuntu-latest + timeout-minutes: 15 + # See the header: the enabled pack's findings are real but unactioned, so + # this lane reports and never blocks. It is NOT here because the findings + # are doubted. + continue-on-error: true + + steps: + - name: Checkout + uses: actions/checkout@v7 + + - name: Install semgrep + # No actions/setup-python step on purpose: ubuntu-latest already ships a + # python3, and this repo pins no setup-python version anywhere else, so + # adding one would be inventing an unverified action pin for nothing. + # + # The semgrep version IS pinned, like every other tool this repo installs + # in CI (gitleaks, grype, golangci-lint), so an upstream release cannot + # change the findings under us without a visible diff. + env: + SEMGREP_VERSION: '1.174.0' + run: | + set -euo pipefail + python3 -m pip install --user --quiet "semgrep==${SEMGREP_VERSION}" + # GITHUB_PATH only affects LATER steps, so export for this one too. + echo "$HOME/.local/bin" >> "$GITHUB_PATH" + export PATH="$HOME/.local/bin:$PATH" + semgrep --version + + - name: Scan + env: + # p/github-actions only — see the header for the measured reason the + # broad language packs are excluded. + SEMGREP_RULES: p/github-actions + run: | + set -uo pipefail + # --metrics=off: never phone home from CI. + # No --error: this lane reports, it does not gate (see header). + semgrep scan --config "$SEMGREP_RULES" --metrics=off \ + --json -o semgrep.json --quiet || true + if [ ! -s semgrep.json ]; then + echo "semgrep produced no JSON — treating as a scan failure" >&2 + exit 1 + fi + + - name: Summarize findings to the job log + # Same reasoning as codeql.yml's summary step: a findings report that + # only exists behind a web UI is unreadable to `gh run view` and to any + # agent holding the log. Print it. + if: ${{ !cancelled() }} + run: | + set -uo pipefail + { + echo '### Semgrep findings' + echo '```' + jq -r ' + (.results // []) as $r + | if ($r | length) == 0 then "No findings." + else + ( $r + | group_by(.check_id) + | sort_by(-length) + | map("\(length | tostring) [\(.[0].extra.severity // "INFO")] \(.[0].check_id | split(".") | last)") + | join("\n") + ) + "\n--\ntotal findings: \($r | length)" + end + ' semgrep.json + echo '```' + # Scan errors are not findings but they do mean coverage was lost, + # so they get their own line rather than being dropped silently. + errs=$(jq '(.errors // []) | length' semgrep.json) + if [ "$errs" != "0" ]; then + echo '' + echo "parse/scan errors (files or rules that did not fully run): $errs" + fi + } | tee -a "$GITHUB_STEP_SUMMARY" + + - name: Upload findings as an artifact + # The repo is public and the results are not sensitive, so the raw JSON + # is kept for a fixing agent (or a human) to consume without re-running + # the scan. + if: ${{ !cancelled() }} + uses: actions/upload-artifact@v7 + with: + name: semgrep-findings + path: semgrep.json + if-no-files-found: warn + retention-days: 14 diff --git a/AGENTS.md b/AGENTS.md index 41b7fd03..e2a8a964 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ make compile # go build ./... (compile-check only; no artifacts) make test # go test -p 1 ./... — run in the FOREGROUND make test-race # go test -race -p 1 ./... (use when touching concurrency) make test-cover # run Go tests with coverage profiling (writes coverage.out) -make lint # golangci-lint + migration DDL lint — must pass clean +make lint # golangci-lint + ruff (Python) + migration DDL lint — must pass clean make fmt # gofmt -w . make tidy # go mod tidy ``` @@ -40,12 +40,15 @@ cd web && npx playwright test --project=mocked # mocked e2e CI mirrors all of this — Go build/vet/lint/test (including a `-race` lane) plus a `govulncheck` dependency-CVE scan, a Grype container-image CVE scan (fail on a -fixable CRITICAL) of the sandbox image, web lint (oxlint) / typecheck (TS 7) / test / build, Playwright (mocked +fixable CRITICAL) of the sandbox image, a Python lint (ruff), web lint (oxlint) / typecheck (TS 7) / test / build, Playwright (mocked **and** live, against a real backend + sandbox), a migration DDL lint, and a gitleaks secret scan. **Every job must be green before merge.** Tests are deterministic without a live model: use the fake-LLM seam (`internal/fakellm` via `OPENROUTER_BASE_URL`), never a real key. +CodeQL (security queries) and Semgrep (Actions supply chain) also run per PR but +are **advisory**, not part of `ci-gate` — see [`docs/SCANNING.md`](docs/SCANNING.md). + ## Repository map See the README "Repository layout" for the annotated tree. In short: `cmd/` (the @@ -161,9 +164,13 @@ same PR. - **Contributor workflow + CI gates:** [`CONTRIBUTING.md`](CONTRIBUTING.md) - **Testing strategy** (unit / fake-LLM / mocked + live Playwright / canary): [`docs/TESTING.md`](docs/TESTING.md) +- **The scanning stack** (who checks what, why ruff owns Python lint, why + Semgrep is scoped to Actions supply chain after its broad packs scored 0/6, + what blocks vs what reports, and the known gaps): + [`docs/SCANNING.md`](docs/SCANNING.md) - **CodeQL** (why default setup was replaced by an advanced-setup workflow, how - the Go toolchain is resolved, why `code-quality` is a query suite here rather - than an analysis kind, and why CodeQL is advisory rather than in `ci-gate`): + the Go toolchain is resolved, why it runs security queries only, and the + difference between a required status check and code scanning merge protection): [`docs/CODEQL.md`](docs/CODEQL.md) - **HTTP API versioning** (the `/v1` prefix + `X-Fleet-API-Version` + `/api-info` discovery + deprecation contract): [`docs/api-versioning.md`](docs/api-versioning.md) diff --git a/CHANGELOG.md b/CHANGELOG.md index eb963ab9..1463f041 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,51 @@ prior versions are listed because none have shipped. ### Fixed +- **Python had no linter, and two scanners were pointed at ground already + covered.** Reshaped the scanning stack so each tool owns one job + ([`docs/SCANNING.md`](docs/SCANNING.md)): + + - **ruff is new, and it blocks.** fleet ships 13 Python files — the sandbox + FileOp helper, the python bridge, the bento-slides and data-profiler skill + scripts, MCP test servers — and *nothing* linted any of them. Go had + golangci-lint, the web tier had oxlint, Python had neither. Rule selection is + narrow on purpose and `ruff.toml` records why: the default rules find 3 + findings on this tree, a broad selection finds 333, of which 176 are + `%`-format style and 43 are magic values. Three real findings were fixed to + make the gate clean on day one — an unused import, a lambda assignment, and a + **byte-identical duplicate `has_guard` definition** in `bento_doc.py` where + the second copy silently shadowed the first. `ruff format` is reported but + not gated (the tree has never been ruff-formatted). + + - **CodeQL narrowed to security queries only.** Its code-quality suite was + enabled, measured, and dropped: 32 findings, every one note-level, zero + security findings. For Go and the web tier it duplicated golangci-lint and + oxlint, which already block; 28 of the 32 were Python, now ruff's job; and 3 + were false positives on correct code (`value != value`, the idiomatic NaN + test). CodeQL keeps the thing nothing else here can do — interprocedural + taint, which is the actual shape of "a credential must not reach a log sink". + + - **Semgrep is new, scoped, and advisory.** The obvious move — point it at + `p/golang`/`p/javascript`/`p/python` — was measured and rejected: 6 of 6 + non-Actions findings were false positives, three of them *already* triaged + and suppressed for gosec, and one (`0o644` for a sandbox directory) would + have been a security regression if followed. What ships is + `p/github-actions`, which found 51 instances of one real issue nothing else + checks: actions pinned to mutable tags rather than commit SHAs. Advisory + because all 51 are real and repinning is its own PR, not because they are + doubted. + + - **Both scanners now print findings to the job log** and the step summary, and + Semgrep uploads raw JSON as an artifact. A CodeQL run otherwise reports + nothing about what it found to its own log — it writes SARIF, uploads it, and + exits 0 either way — which made outcomes invisible to `gh run view` and to + any agent holding the log but not the code-scanning API. + + Also added: an aggregate `CodeQL gate` job, so making CodeQL blocking later is + one required check rather than four per-language checks needing manual + re-pointing whenever the matrix changes. Nothing here is wired into `ci-gate` + beyond ruff; CodeQL and Semgrep stay advisory. + - **CodeQL had stopped analyzing the repo's Go code, and then stopped analyzing anything.** Default setup's Go analysis failed on every main-targeting PR from the Go 1.27 bump (#1240, promoted in #1242) onward — it installed the Go its diff --git a/Makefile b/Makefile index baa66671..7a2005df 100644 --- a/Makefile +++ b/Makefile @@ -120,11 +120,28 @@ test-cover: go test -coverprofile=coverage.out -covermode=atomic -p 1 -tags fleet_host_executor ./... @go tool cover -func=coverage.out | tail -1 -lint: lint-go lint-migrations +lint: lint-go lint-python lint-migrations lint-go: golangci-lint run +# lint-python: ruff over the 13 Python files (sandbox FileOp helper, python +# bridge, skill scripts, MCP test servers). Rule selection lives in ruff.toml so +# this and the CI job cannot disagree. +# +# Skips LOUDLY when ruff is absent rather than failing: not every contributor has +# it, and CI enforces the gate regardless (ci.yml + dev-ci.yml `python` job). The +# message names the install command so a local skip is a choice, not a surprise — +# a check that quietly does nothing is the failure mode this repo keeps writing +# post-mortems about. +lint-python: + @if command -v ruff >/dev/null 2>&1; then \ + ruff check . ; \ + else \ + echo "ruff not installed — SKIPPING the Python lint (CI still enforces it)."; \ + echo " install: python3 -m pip install --user 'ruff==0.15.8'"; \ + fi + # lint-migrations: reject dangerous DDL in NEW/CHANGED migration files (#256). # Diff-scoped (vs the merge-base with origin/main), so the existing corpus is # untouched; a no-op when no migration files changed or no base ref resolves. diff --git a/docs/CODEQL.md b/docs/CODEQL.md index 3b061078..04fa1ea1 100644 --- a/docs/CODEQL.md +++ b/docs/CODEQL.md @@ -78,12 +78,15 @@ enough", not "unset the pin". Neither `env: GOTOOLCHAIN: auto` nor `.github/workflows/codeql.yml`, one `analyze` job over a four-entry matrix. -| language | build mode | quality queries | +| language | build mode | queries | | --- | --- | --- | -| `go` | `autobuild` | yes | -| `python` | `none` | yes | -| `javascript-typescript` | `none` | yes | -| `actions` | `none` | no | +| `go` | `autobuild` | security (default suite) | +| `python` | `none` | security (default suite) | +| `javascript-typescript` | `none` | security (default suite) | +| `actions` | `none` | security (default suite) | + +**Security queries only.** The code-quality suite was enabled, measured, and then +deliberately removed — see "Why code quality was dropped" below. `build-mode: none` is [not supported for Go](https://docs.github.com/en/code-security/reference/code-scanning/codeql/build-options-for-compiled-languages) @@ -174,15 +177,14 @@ loaded only `codeql/go-queries`, evaluated 72 distinct queries, and uploaded a single `go.sarif`. The code-quality half of the coverage this change claims to restore was not running at all. -Fixed by using `queries: code-quality`, exactly as the message directs. +Fixed at the time by using `queries: code-quality`, exactly as the message +directs. That is the working form, and it is worth recording for anyone who tries +`analysis-kinds` again and sees a green check: the input is accepted, two errors +are logged, and only security runs. -**This is a real deviation from default setup, not a like-for-like restoration.** -Default setup ran code quality as a separate *analysis kind*, producing a second -analysis that feeds GitHub's Code Quality experience. That kind is -GitHub-internal and closed to custom workflows — no advanced-setup workflow can -feed it. What is restored is the code-quality **query suite**, added to the -default security suite over one shared database, with findings arriving as -ordinary code-scanning alerts. The queries all run; the presentation differs. +Code quality was then measured and dropped — see the next section. The lasting +point from this defect is the one about evidence: the run was **green** with the +requested analysis silently not happening. ### 2. Go extraction had one hole, and it was the worst file in the tree @@ -218,6 +220,51 @@ Note also that the extractor still logs `Build flags: ''`. `GOFLAGS` reaches the plumbing, so the log line that looks like it should confirm the fix does not. `host.go` appearing in the extracted list is what confirms it. +## Why code quality was dropped + +The quality suite shipped first, ran, and was then removed on the evidence it +produced. Worth writing down, because "more queries" reads as strictly better +until you look at what they found. + +**What it found: 32 findings, every one note-level, and zero security findings.** + +| language | findings | what they were | +| --- | --- | --- | +| `go` | 2 | `go/useless-assignment-to-field` | +| `python` | 28 | `py/empty-except` (12), `py/implicit-string-concatenation-in-list` (9), `py/comparison-of-identical-expressions` (3), unused local/global/import (4) | +| `javascript-typescript` | 2 | `js/trivial-conditional`, `js/useless-assignment-to-local` | +| `actions` | 0 | — | + +Three reasons that adds up to "wrong tool", not "clean codebase": + +1. **For Go and the web tier it duplicates gates that already block.** + `golangci-lint` runs `gosec`, `staticcheck`, `revive`, `unparam`, `gocritic` + and more, and it is inside `ci-gate`; oxlint covers the web tier. + `go/useless-assignment-to-field` is squarely inside that remit. Paying ~40s of + CodeQL for a second opinion on it buys nothing. + +2. **28 of 32 were Python — a real gap, but ruff is the right instrument.** + Python genuinely had no linter (see `ruff.toml`), so those findings were the + suite's only unique contribution. ruff finds the same class of thing in well + under a second, with autofix, and now blocks. A slow job with no autofix is + the wrong shape for lint, especially for an agent expected to fix and re-push. + +3. **Three of the 32 were false positives on correct code.** + `py/comparison-of-identical-expressions` flagged `value != value` three times + in `bento_pdf.py` — the idiomatic NaN test, which is true only for NaN. + Enabling the equivalent ruff rule (`PLR0124`) was rejected for the same + reason. + +So the security queries stay (they are what nothing else here can do — see the +Semgrep comparison in `.github/workflows/semgrep.yml`), and quality moves to the +linters that were already gating. + +**What this costs, stated plainly:** the four Go/JS quality findings above are no +longer reported by anything, because `golangci-lint` and oxlint did not +independently flag them. That is a real, small loss of coverage accepted in +exchange for not running a second slow analyzer over ground three other tools +already cover. + ## What was verified Measured on run 2 (`7731615`, run `32571297663`), by downloading the run's log @@ -237,15 +284,16 @@ Success: extraction succeeded for all 2 discovered project(s). non-test files by exactly `host_disabled.go`, per the build-tag trade above. **Every language produced a database, ran queries, and uploaded results.** -Distinct queries evaluated, run 1 vs run 2 — the delta is the code-quality suite -arriving: +Distinct queries evaluated. The middle column is the security suite alone, which +is what ships; the right column is what adding `queries: code-quality` did, kept +here because it is the measurement the drop decision rests on: -| language | run 1 | run 2 | delta | SARIF | -| --- | --- | --- | --- | --- | -| `go` | 72 | 116 | +44 | `go.sarif` | -| `python` | 90 | 292 | +202 | `python.sarif` | -| `javascript-typescript` | 178 | 374 | +196 | `javascript.sarif` | -| `actions` | 36 | 36 | +0 | `actions.sarif` | +| language | security only (ships) | with code-quality (dropped) | SARIF | +| --- | --- | --- | --- | +| `go` | 72 | 116 (+44) | `go.sarif` | +| `python` | 90 | 292 (+202) | `python.sarif` | +| `javascript-typescript` | 178 | 374 (+196) | `javascript.sarif` | +| `actions` | 36 | 36 (+0) | `actions.sarif` | `actions` is unchanged **by design** — default setup ran it in the security analysis only, and that was matched rather than widened. The new Go query @@ -292,11 +340,12 @@ with go1.25.1 and cannot lint the tree. evaluates, but CodeQL does not print the number to the job log; it lands in a `.bqrs`. File and package counts are what was actually observed, so they are what is reported here. No line count is claimed. -- **Alert counts are not claimed.** This change restores *scanning*; what the - new quality queries find on this codebase is a separate question, and PR-run - file-coverage information is suppressed by CodeQL anyway ("To speed up pull - request analysis, file coverage information is only enabled when analyzing the - default branch and protected branches"). +- **Alert counts on `main` are not claimed.** The zero-security-findings result + above was measured on a PR run of this branch. PR-run file-coverage detail is + suppressed by CodeQL ("To speed up pull request analysis, file coverage + information is only enabled when analyzing the default branch and protected + branches"), so the default-branch alert set is not established until this + merges and a promote lands on `main`. - **`build-mode: manual` was not built.** `autobuild` works, so the more complex option was not needed. If `autobuild` regresses, manual mode plus the repo's own `go build ./...` is the fallback — and it is also the route to @@ -327,11 +376,11 @@ fleet is a **public** repository, so code scanning merge protection is available at no cost (on private repos it requires GitHub Advanced Security). To turn it on: Settings → Rules → the "Main" ruleset → add the **Code scanning** rule → add tool **CodeQL** → set the alert thresholds. Two independent knobs there: -*Security alerts* (the CWE/security queries) and *Alerts* (everything else, -which is where the code-quality suite lands). Sensible starting point given that -the quality queries have never run against this codebase: security threshold -**High or higher**, alerts threshold **None**, then tighten once the quality -backlog is known and burned down. +*Security alerts* (the CWE/security queries — the only ones this workflow runs) +and *Alerts* (everything else, which would be where a code-quality suite landed +if one were enabled; it is not). Since the security suite currently reports zero +findings on this tree, a **High or higher** security threshold can go on without +inheriting a backlog. ## Merge gating today — unchanged, with the lever put within reach @@ -369,12 +418,12 @@ posture than that lane's stated "does it compile, lint, and pass tests" job. The sequence with the least chance of self-inflicted deadlock is: merge this, watch a few promotions go green, then add `CodeQL gate` to the ruleset. -One more thing to know before requiring it: the newly-enabled quality queries -(+44 Go, +202 Python, +196 JavaScript) have **never run against this codebase -before**. If any of them fire, they become alerts the moment this merges — and -if `CodeQL gate` were required on day one, a pre-existing quality finding would -block merges. Findings are advisory only for as long as the gate stays optional, -which is another reason to sequence it that way. +What makes that sequencing *safer than it was*: with code quality dropped, the +security suite is all that runs, and it currently reports **zero findings** on +this tree (see "Why code quality was dropped"). So there is no pre-existing +backlog for a required gate to trip over — which is the usual reason turning one +on hurts. The remaining risk is the one this whole document is about: a toolchain +or extractor regression going red for reasons unrelated to any diff. No repo-settings or API change to code-scanning configuration was attempted as part of this change. diff --git a/docs/SCANNING.md b/docs/SCANNING.md new file mode 100644 index 00000000..2797c8a9 --- /dev/null +++ b/docs/SCANNING.md @@ -0,0 +1,173 @@ +# The scanning stack: who checks what, and what actually gates + +Design note for the change that stopped treating "add a scanner" as strictly +better and gave each tool the job it is actually good at. Companion to +[`CODEQL.md`](CODEQL.md) (why default setup was replaced, and why CodeQL now runs +security queries only) and [`TESTING.md`](TESTING.md) (the rest of the ladder). + +## The stack + +| tool | scope | speed | gates? | where results appear | +| --- | --- | --- | --- | --- | +| `golangci-lint` (incl. `gosec`) | Go lint + Go SAST patterns | ~30s | **blocks** (`ci-gate`) | job log | +| `oxlint` + `tsc` | web tier lint + types | ~5s | **blocks** (`ci-gate`) | job log | +| **`ruff`** | **Python lint** | **~1s** | **blocks** (`ci-gate`) | job log | +| `govulncheck` | Go dependency CVEs (called symbols) | ~30s | **blocks** (`ci-gate`) | job log + Security tab | +| `grype` | sandbox image CVEs (fixable CRITICAL) | ~1m | **blocks** (`ci-gate`) | job log + Security tab | +| `gitleaks` | secrets, every branch | ~10s | **blocks** (`ci-gate`) | job log | +| CodeQL | **interprocedural taint / security** | ~2m | advisory | job log + Security tab | +| **Semgrep** | **GitHub Actions supply chain** | ~20s | advisory | job log + artifact | + +Two things were added here (**ruff**, **Semgrep**) and one was narrowed +(**CodeQL**, to security queries only). + +## Why each tool is where it is + +**The design rule: one owner per job.** A second tool over ground an existing +blocking gate already covers does not add safety — it adds a queue of duplicate +findings, and a scanner whose output is mostly already-adjudicated noise trains +people to close the tab. Every placement below follows from that. + +### ruff owns Python lint (new, blocking) + +fleet ships 13 Python files — the sandbox FileOp helper, the python bridge, the +bento-slides and data-profiler skill scripts, MCP test servers, icon/doc +generators — and **nothing linted any of them.** Go had `golangci-lint`, the web +tier had `oxlint`, Python had neither. Its only coverage was whatever CodeQL's +code-quality suite happened to notice, at ~40s and with no autofix. + +That gap was real: CodeQL's quality suite found 28 Python issues, its single +largest contribution anywhere. ruff finds the same class in under a second, with +autofix, and now blocks. + +The rule selection is deliberately narrow, and `ruff.toml` records the numbers: +default rules find **3** findings on this tree; a broad selection finds **333**, +of which 176 are `%`-format style, 43 magic values and 35 line length. Gating on +that would mean a whole-tree reformat for no correctness gain. + +Three real findings were fixed to make the gate clean on day one, so a new +violation is a regression rather than noise in a backlog: + +- `internal/mcp/testdata/dummy_server.py` — unused `import os`. +- `bento_doc.py` — **a byte-identical duplicate `has_guard` definition.** Two + copies, one call site; the second silently shadowed the first. Dead code, and + the only finding here that was arguably a latent bug. +- `bento_pdf.py` — a lambda assigned to a name (`E731`), rewritten as a `def`. + +`ruff format` is reported but **not** gated: the tree has never been +ruff-formatted, so failing on it would block every PR on a reformat nobody +scheduled. The advisory output keeps the size of that decision visible. + +### CodeQL owns interprocedural taint (narrowed, advisory) + +CodeQL is the only tool in this stack that does cross-function dataflow, and that +is exactly the shape of fleet's headline invariants: *a credential must not reach +a log sink, the model context, or the sandbox.* `go/clear-text-logging` is +literally that query. Nothing else here can express it. + +So CodeQL keeps its security queries and gives up everything else — the quality +suite duplicated `golangci-lint`/`oxlint` for Go and JS, and ruff is a better fit +for Python. Full reasoning and measurements in [`CODEQL.md`](CODEQL.md). + +Its security suite currently reports **zero findings** on this tree. + +### Semgrep owns GitHub Actions supply chain (new, advisory) + +Semgrep is the opposite trade from CodeQL: seconds instead of minutes, no +database build, findings straight to stdout, rules cheap to write. That makes it +the natural fit for an agent-driven loop — and it is why the obvious move is to +point it at everything. + +**That move was measured and rejected.** The broad registry packs (`p/golang`, +`p/javascript`, `p/python`) produced 55 findings on this tree, and **all 6 +non-Actions findings were false positives:** + +| finding | why it is wrong | +| --- | --- | +| `open-redirect` — `cmd/fleet/tls.go:109` | Standard HTTP→HTTPS upgrade to the **same** host. Already carries `//nolint:gosec G710` saying so. | +| `math-random-used` — `internal/runner/runner.go:28` | `math/rand/v2`, used once, for ±10% jitter on a retry interval. | +| `cookie-missing-secure` — `internal/sched/handlers/elcano.go:155` | A **deletion** cookie (`Value=""`, `MaxAge=-1`), no secret; `Secure` is conditional so logout works over plain-HTTP dev. Already `//nolint:gosec G124`. | +| `unsafe-deserialization-interface` — `internal/mcp/httptool.go:254` | `json.Unmarshal` into `interface{}` is **required** — the value feeds a jq program over arbitrary JSON. A concrete struct is not expressible. | +| `x-frame-options-misconfiguration` — `web/src/proxy.ts:99` | The header value is the literal string `"DENY"`. No user input reaches it. | +| `insecure-file-permissions` — `internal/sandbox/fileops.py:77` | Advises `0o644` for a **sandbox directory**, i.e. world-readable. Taking that advice would be a security **regression**; `0750` is the file-tool contract. | + +Three of those six were **already formally triaged and suppressed for `gosec`**, +which runs inside `golangci-lint` and already blocks. A scanner that re-reports +adjudicated findings is how you teach a team to ignore it, so those packs are not +enabled. Run them locally for a one-off audit if you want them. + +What Semgrep *does* own is the pack that earned it. `p/github-actions` found +**51 instances of one real issue nothing else in this repo checks**: actions +pinned to a **mutable tag** (`actions/checkout@v7`) rather than an immutable +commit SHA. A moved tag runs attacker-controlled code with this repo's token. + +``` +51 [WARNING] github-actions-mutable-action-tag +``` + +Spread across every workflow (18 in `ci.yml`, 7 in `dev-ci.yml`, …). + +**It is advisory, and the reason is honesty about scheduling, not doubt about the +findings.** All 51 are real. `--error` here would mean a red gate until every +action in every workflow is repinned to a SHA — a worthwhile change, and its own +PR. Failing CI for a backlog nobody has scheduled just teaches people to ignore +the lane. Flip `continue-on-error` off in the same PR that does the repinning. + +## Findings are readable from the job log, on purpose + +Both scanners print a per-rule summary into the job log **and** the step summary: + +``` +### CodeQL findings — go +2 [note] go/useless-assignment-to-field +-- +total findings: 2 +``` + +This exists because a CodeQL run reports **nothing** about what it found to its +own log — it writes SARIF, uploads it, exits 0, findings or not. Verified by +grepping a full run's log archive: there is no alert or result count anywhere. +That made a run's real outcome invisible to `gh run view` and to any agent +holding the log but not the code-scanning API. + +Semgrep additionally uploads its raw JSON as an artifact (`semgrep-findings`, +14-day retention), so a fixing agent can consume structured findings without +re-running the scan. The repo is public and these results are not sensitive; +withholding them buys nothing. + +## What gates, and what a required check actually means + +Everything in the "blocks" column above is reached through **`ci-gate`**, the +single required status check on `main`. The two scanners are outside it. + +The distinction that matters for anyone tightening this later: + +| block a merge when… | mechanism | +| --- | --- | +| an analysis **failed or did not run** | required status check (`CodeQL gate`) | +| a scanner **found alerts** at/above a severity | **code scanning merge protection** (ruleset → Code scanning rule) | + +**A CodeQL job with a hundred open alerts still exits 0 and reports green.** Job +success only says extraction and evaluation worked. That is both why the Go +toolchain break survived weeks behind a red-but-not-required check, and why a +green check is not evidence of a clean tree. See +[`CODEQL.md`](CODEQL.md#two-different-things-can-gate-and-they-are-not-the-same-lever). + +## Known gaps, deliberately not closed here + +Stated rather than left for rediscovery: + +- **No CVE scanning of the web tier's npm tree.** `govulncheck` is Go-only; + `grype` scans the sandbox *image*. A Next.js app with ~437 TS/JS files has no + dependency CVE gate. Dependabot opens npm PRs, but Dependabot alerts do not + block anything. This is the largest remaining hole in the stack — arguably + larger than anything CodeQL gating would fix. +- **Actions are pinned to mutable tags.** 51 instances, per above. Semgrep now + reports it; nothing yet fixes it. +- **`_test.go` files are outside CodeQL's database** (621 files) — `autobuild` + builds packages, not tests. Unchanged from default setup. +- **`ruff format` is not enforced**, per above. +- **Semgrep's own rule packs are network-fetched** from the registry at scan + time. The semgrep *version* is pinned; the *rules* are not, so a registry + change can move findings without a diff here. Acceptable for an advisory lane; + it would need a vendored ruleset before this could block. diff --git a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py index b9c787fc..928eabda 100644 --- a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py +++ b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py @@ -306,10 +306,6 @@ def _inject_guard(raw): return raw[:at] + GUARD + raw[at:] -def has_guard(raw): - return GUARD_ID.encode() in raw - - def _decode_block(block): """Parse a document block's bytes into a dict. diff --git a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py index 2fd3b9df..0f1dde12 100644 --- a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py +++ b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py @@ -1918,7 +1918,9 @@ def render_line(self, canvas, element, box): color = parse_color(element.get("fill")) if not is_visible(color): return - tip = lambda kind: width * 2.6 if kind and kind != "none" else 0.0 + def tip(kind): + return width * 2.6 if kind and kind != "none" else 0.0 + start = tip(element.get("lineStart")) end = tip(element.get("lineEnd")) mid = y + h / 2.0 diff --git a/internal/mcp/testdata/dummy_server.py b/internal/mcp/testdata/dummy_server.py index cf0bf728..74c1e114 100644 --- a/internal/mcp/testdata/dummy_server.py +++ b/internal/mcp/testdata/dummy_server.py @@ -1,6 +1,5 @@ import sys import json -import os # Ensure unbuffered output sys.stdout.reconfigure(line_buffering=True) diff --git a/ruff.toml b/ruff.toml new file mode 100644 index 00000000..1b9cb0de --- /dev/null +++ b/ruff.toml @@ -0,0 +1,67 @@ +# ruff — the Python lane of the lint gate. +# +# WHY THIS EXISTS: fleet has 13 Python files (the sandbox FileOp helper, the +# python bridge, the bento-slides and data-profiler skill scripts, MCP test +# servers, icon/doc generators) and until now NOTHING linted them. Go had +# golangci-lint and the web tier had oxlint; Python had nothing, so its only +# coverage was whatever CodeQL's code-quality suite happened to notice. +# +# That is the wrong tool for the job in both directions. CodeQL took ~40s to +# report 28 note-level Python issues with no autofix, while ruff finds the same +# class of thing in well under a second and fixes much of it. So the Python +# quality lane is ruff's, and CodeQL keeps only its security queries (see +# docs/CODEQL.md). +# +# RULE SELECTION IS DELIBERATELY NARROW, and the numbers behind that are worth +# recording so nobody widens it by accident. Measured on this tree: +# +# default (E4,E7,E9,F) -> 3 findings <- what we gate on +# E,F,W,I,UP,B,SIM,ISC,PLR,PLW,S -> 333 findings +# +# The 330-finding difference is almost entirely style churn, not defects: +# 176 UP031 (%-format vs f-string), 43 PLR2004 (magic values), 35 E501 (line +# length). Gating on that would mean a 300-commit reformat for no correctness +# gain, so it is out of scope here rather than silently enabled. +# +# Two rules were considered and deliberately REJECTED, both because their only +# findings in this tree are correct code: +# +# PLR0124 (name compared with itself) — its 3 hits in bento_pdf.py are the +# idiomatic NaN test (`value != value` is true only for NaN). CodeQL's +# py/comparison-of-identical-expressions flagged the same 3. Enabling it +# would mean three `# noqa` comments on correct code. +# S110 / SIM105 (try-except-pass) — 14 hits, mostly in python_bridge.py's +# best-effort cleanup paths where swallowing is the intent. Worth a separate +# review pass, not a gate that has to be suppressed 14 times on day one. +# +# The gate is therefore "clean today, and stays clean": three real findings were +# fixed to get here (an unused import, a byte-identical duplicate function +# definition, and a lambda assignment), so a NEW default-rule violation is a +# real regression rather than noise in a backlog. + +# Match the lowest Python the sandbox image and the skill scripts must run on. +# Declared here so the rules that are version-sensitive (pyupgrade et al, if +# ever enabled) have one declaration point rather than a guess. +target-version = "py311" + +line-length = 88 + +exclude = [ + "node_modules", + "web/.next", + ".git", +] + +[lint] +# The default rule set: pycodestyle errors (E4 imports, E7 statements, +# E9 syntax/IO) plus Pyflakes (F — undefined names, unused imports, redefined +# names, unused locals). This is the "is it actually broken" tier; see the +# header for why the style tiers are not enabled. +select = ["E4", "E7", "E9", "F"] + +[lint.per-file-ignores] +# MCP test servers and testdata fixtures are deliberately minimal stand-ins — +# they exist to be spawned and to misbehave in specific ways, so an unused +# import or an odd construct there can be the point of the fixture. +"internal/mcp/testdata/*.py" = ["F401"] +"cmd/fleet/testdata/*.py" = ["F401"] diff --git a/scripts/check_versions_test.go b/scripts/check_versions_test.go index dbff4c51..72d989d9 100644 --- a/scripts/check_versions_test.go +++ b/scripts/check_versions_test.go @@ -172,6 +172,7 @@ func TestDuplicatedToolPinsAgree(t *testing.T) { {"GRYPE_VERSION", ".github/workflows/grype-scheduled.yml", regexp.MustCompile(`GRYPE_VERSION:\s*'([^']+)'`)}, {"GRYPE_SHA256", ".github/workflows/grype-scheduled.yml", regexp.MustCompile(`GRYPE_SHA256:\s*'([^']+)'`)}, {"GITLEAKS_VERSION", ".github/workflows/dev-ci.yml", regexp.MustCompile(`GITLEAKS_VERSION:\s*'([^']+)'`)}, + {"RUFF_VERSION", ".github/workflows/dev-ci.yml", regexp.MustCompile(`RUFF_VERSION:\s*'([^']+)'`)}, {"golangci-lint version", ".github/workflows/dev-ci.yml", regexp.MustCompile(`golangci-lint-action@v\d+\s+with:\s+(?:#[^\n]*\n\s+)*version:\s*(v[\d.]+)`)}, } { a := tc.re.FindStringSubmatch(ci) From def55c5d899e6bd9bdee369ba41c12b35d9f1fc7 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 13:19:28 +0000 Subject: [PATCH 07/34] Fix every scanner finding and make the scanners block MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scanning stack now passes clean and fails on anything new. Turning a gate on over an unfixed backlog is how a gate becomes something people route around, so every finding was fixed or adjudicated before the switch was flipped. All 53 action references pinned to commit SHAs. Semgrep's github-actions-mutable-action-tag found 51 instances of actions referenced by a mutable tag (actions/checkout@v7); if such a tag moves, attacker-controlled code runs with this repo's GITHUB_TOKEN. Every `uses:` across all 12 workflows is now @<40-hex> with the version in a trailing comment — the form Dependabot reads and updates, and .github/dependabot.yml already watches the github-actions ecosystem. Each SHA is the commit the previously-used tag resolved to at pin time, so the pin is behaviourally identical to the runs already verified green; a pin should not smuggle in a version bump. The only `uses:` lines left on @main are two inside COMMENTS, documenting how a downstream bundle repo calls fleet's reusable workflows — @main is correct guidance there, and Semgrep does not flag them because a YAML comment is not a `uses:` key. Semgrep now blocks over all four packs (p/github-actions, p/golang, p/javascript, p/python) with --error and no continue-on-error. The 6 false positives are suppressed at the line with `nosemgrep: ` plus a stated reason, scoped to the rule so a different rule on the same line still reports. Three of the six were already formally triaged and suppressed for gosec, which runs inside golangci-lint and already blocks; one of them — advising 0o644, world-readable, for a sandbox directory — would have been a security REGRESSION if followed. Every suppression was mutation-tested: strip it and the finding reappears, keep it and the finding is gone. That check matters because "0 findings" has two explanations — the waivers work, or the rules silently stopped matching — and only one is safety. Verified across all three comment syntaxes. CodeQL now fails on findings. Previously the analyze step exited 0 whether it found nothing or a hundred alerts, so a red check could only ever mean "the scanner broke" — which is precisely how the Go toolchain break sat unnoticed for weeks behind a red-but-not-required check. Threshold is ANY finding, which is safe because the security suite reports zero across go, python, javascript-typescript and actions. The step also fails when no SARIF was written at all, rather than reporting a clean scan that never happened. Both scanners report as their own checks (CodeQL gate, Semgrep scan) rather than through ci-gate, because a job's `needs` cannot reach across workflow files. Making a red check BLOCK a merge still requires adding those checks to the branch ruleset; a workflow file cannot make itself required. Two knock-on defects found and fixed while doing this: - SHA pinning broke two regexes in scripts/check_versions_test.go that matched `golangci-lint-action@v\d+`. Those assertions fail OPEN — a non-match logs "skipping" rather than failing — so the pin would have silently disabled the golangci-lint version agreement checks. Widened to tolerate a pinned ref plus its trailing version comment, then mutation-tested against the real docs text to confirm they still bite rather than skip. - A standalone nosemgrep comment inside a Go import block makes goimports reformat the group, failing lint. That waiver is a trailing comment on the import line instead; re-tested to confirm the trailing form still suppresses. Deliberately NOT included, with measurements, in docs/SCANNING.md: `ruff format` (9 of 13 files differ, a 3725-line diff — cosmetics, and landing it here would bury the security change) and widening ruff's rule set (`--select B,SIM,S` adds 21 findings, of which the interesting ones are 2x B905 zip-without-strict and 1x SIM115 open-without-context-manager). Gate: gofmt clean, make build, make lint (0 issues + ruff clean), make test (exit 0), make lint-migrations, and semgrep --error over all four packs at 0 findings. Signed-off-by: Brad Flaugher --- .github/workflows/auto-merge-dependabot.yml | 2 +- .github/workflows/benchmark.yml | 4 +- .github/workflows/build-sandbox-image.yml | 4 +- .github/workflows/ci.yml | 38 +++--- .github/workflows/codeql.yml | 68 ++++++++-- .github/workflows/dev-ci.yml | 16 +-- .github/workflows/e2e-canary.yml | 8 +- .github/workflows/govulncheck-scheduled.yml | 6 +- .github/workflows/grype-scheduled.yml | 4 +- .github/workflows/publish-sandbox-image.yml | 4 +- .github/workflows/screenshots.yml | 6 +- .github/workflows/semgrep.yml | 113 ++++++++++------- AGENTS.md | 7 +- CHANGELOG.md | 41 ++++++ cmd/fleet/tls.go | 1 + docs/CODEQL.md | 17 ++- docs/SCANNING.md | 131 +++++++++++++------- internal/mcp/httptool.go | 1 + internal/runner/runner.go | 2 +- internal/sandbox/fileops.py | 1 + internal/sched/handlers/elcano.go | 1 + scripts/check_versions_test.go | 4 +- web/src/proxy.ts | 1 + 23 files changed, 332 insertions(+), 148 deletions(-) diff --git a/.github/workflows/auto-merge-dependabot.yml b/.github/workflows/auto-merge-dependabot.yml index 45c596c8..810b2e00 100644 --- a/.github/workflows/auto-merge-dependabot.yml +++ b/.github/workflows/auto-merge-dependabot.yml @@ -30,7 +30,7 @@ jobs: steps: - name: Fetch Dependabot metadata id: meta - uses: dependabot/fetch-metadata@v3 + uses: dependabot/fetch-metadata@25dd0e34f4fe68f24cc83900b1fe3fe149efef98 # v3.1.0 with: github-token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index d6d55cfc..461f407c 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -43,12 +43,12 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: main - name: Set up Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: # Read from go.mod so the toolchain is declared once. go-version-file: go.mod diff --git a/.github/workflows/build-sandbox-image.yml b/.github/workflows/build-sandbox-image.yml index 71aca0d2..547494f9 100644 --- a/.github/workflows/build-sandbox-image.yml +++ b/.github/workflows/build-sandbox-image.yml @@ -98,13 +98,13 @@ jobs: BUNDLE_DIR: ${{ inputs.bundle_dir }} steps: - name: Checkout caller repo - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 # The build script lives in fleet core; a client-config caller doesn't # have it. Checking it out into a subdir keeps ONE canonical builder # (same manifest parsing, same flags) instead of a drifting copy per repo. - name: Checkout fleet (build script) - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: repository: ElcanoTek/fleet ref: ${{ inputs.fleet_ref || 'main' }} diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ab9e9001..6f9c9c60 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,7 +26,7 @@ jobs: docs_only: ${{ steps.detect.outputs.docs_only }} steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: # Full history so the base/head (or before/after) SHAs are present to diff. fetch-depth: 0 @@ -78,7 +78,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install gitleaks # Pin a specific gitleaks release and verify its checksum so the gate is @@ -110,7 +110,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: # Full history so the merge-base with origin/main resolves and the # linter can diff the branch's new/changed migration files. @@ -163,10 +163,10 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: # Read from go.mod so the toolchain is declared once. go-version-file: go.mod @@ -231,7 +231,7 @@ jobs: run: go vet -tags fleet_host_executor ./... - name: golangci-lint - uses: golangci/golangci-lint-action@v9 + uses: golangci/golangci-lint-action@db9de0fc1a667e1a49d2291a1a042dff081d78f6 # v9 with: # Repo .golangci.yml is the v2 schema (version: "2"). It no longer # pins run.go: golangci-lint's documented default is "use Go version @@ -332,7 +332,7 @@ jobs: # find 3 issues here; the broad selection finds 333, almost all style churn). steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install ruff # Pinned like every other tool this repo installs in CI, so an upstream @@ -378,10 +378,10 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Node - uses: actions/setup-node@v7 + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: # Read from web/.nvmrc so the deployed node major is declared once. node-version-file: web/.nvmrc @@ -439,10 +439,10 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Node - uses: actions/setup-node@v7 + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: # Read from web/.nvmrc so the deployed node major is declared once. node-version-file: web/.nvmrc @@ -465,7 +465,7 @@ jobs: - name: Upload Playwright HTML report if: ${{ failure() }} - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: playwright-report path: web/playwright-report/ @@ -510,17 +510,17 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: # Read from go.mod so the toolchain is declared once. go-version-file: go.mod check-latest: true - name: Set up Node - uses: actions/setup-node@v7 + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: # Read from web/.nvmrc so the deployed node major is declared once. node-version-file: web/.nvmrc @@ -638,7 +638,7 @@ jobs: - name: Upload Playwright live report if: ${{ failure() }} - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: playwright-live-report path: web/playwright-report/ @@ -646,7 +646,7 @@ jobs: - name: Upload server logs on failure if: ${{ failure() }} - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: e2e-live-server-logs path: | @@ -671,7 +671,7 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install Grype # Pin a specific Grype release and verify its checksum, exactly as the @@ -749,7 +749,7 @@ jobs: # affected packages, and fix versions rather than only a red CI job. The # hashFiles guard skips this step (instead of erroring "file not found") # when an EARLIER step failed before grype could write the SARIF. - uses: github/codeql-action/upload-sarif@v4 + uses: github/codeql-action/upload-sarif@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) if: ${{ !cancelled() && hashFiles('grype-results.sarif') != '' }} with: sarif_file: 'grype-results.sarif' diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 4c1cddb6..09cfb216 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -63,9 +63,17 @@ # Grype step, govulncheck-scheduled.yml, grype-scheduled.yml). Those push their # own SARIF to the Security tab and never depended on CodeQL being configured. # -# Merge gating: these jobs are NOT part of ci.yml's `CI gate`, which is the single -# required status check on main (see .github/CODEOWNERS). CodeQL findings are -# therefore advisory today, exactly as they were under default setup. +# Merge gating, in two parts — they are different things and conflating them is +# how "we gate on CodeQL" ends up meaning nothing: +# +# 1. Does a finding turn the CHECK red? YES. The `Fail on findings` step below +# fails the job on any finding. Without it the analyze step exits 0 whether +# it found nothing or a hundred alerts, so a red check could only ever mean +# "the scanner broke". +# 2. Does a red check BLOCK a merge? NOT YET. `ci-gate` is still the only +# required status check on main (see .github/CODEOWNERS). Requiring a check +# is a repo-settings action that a workflow file cannot perform — add +# `CodeQL gate` to the ruleset to close this half. # # They cannot be part of `CI gate`: a job's `needs` cannot reach across workflow # files. So this file carries its own aggregate `CodeQL gate` job at the bottom, @@ -139,20 +147,20 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 # THE FIX. Runs before init so the interpreter is already on PATH when the # Go extractor and autobuild shell out to `go`. go-version-file makes go.mod # the single declaration point, so this cannot drift from the module. - name: Set up Go if: matrix.language == 'go' - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: go-version-file: go.mod cache: true - name: Initialize CodeQL - uses: github/codeql-action/init@v4 + uses: github/codeql-action/init@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) with: languages: ${{ matrix.language }} build-mode: ${{ matrix.build-mode }} @@ -161,7 +169,7 @@ jobs: - name: Autobuild if: matrix.build-mode == 'autobuild' - uses: github/codeql-action/autobuild@v4 + uses: github/codeql-action/autobuild@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) env: # Same build tag ci.yml and dev-ci.yml pass to `go vet` and `go test`, # and for the same reason: internal/sandbox/host.go — the UNSANDBOXED @@ -175,7 +183,7 @@ jobs: GOFLAGS: -tags=fleet_host_executor - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@v4 + uses: github/codeql-action/analyze@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) with: category: /language:${{ matrix.language }} # Also write the SARIF to disk so the step below can summarize it. The @@ -232,6 +240,50 @@ jobs: fi } | tee -a "$GITHUB_STEP_SUMMARY" + - name: Fail on findings + # THIS is what makes CodeQL a gate rather than a report. The analyze step + # exits 0 whether it found nothing or a hundred alerts — job success only + # means extraction and query evaluation worked. Without this step a red + # `Analyze (…)` check can only ever mean "the scanner broke", never "the + # code has a problem", which is precisely how the Go toolchain break sat + # unnoticed behind a red-but-not-required check for weeks. + # + # Threshold is ANY finding, deliberately. The security suite currently + # reports ZERO across go/python/javascript-typescript/actions, so there + # is no backlog to grandfather and no severity line to argue about — a + # finding here is new. Switching a gate on over an existing backlog is + # how a gate becomes something people route around. + # + # Runs after the summary so the log leads with WHAT was found. + # + # NOTE ON SCOPE: this makes the CHECK red. Whether a red check BLOCKS a + # merge is branch protection's call — `CodeQL gate` has to be a required + # status check for that, which is a repo-settings action a workflow file + # cannot perform. See docs/CODEQL.md ("Merge gating"). + if: ${{ !cancelled() }} + env: + SARIF_DIR: ${{ runner.temp }}/codeql-sarif + LANGUAGE: ${{ matrix.language }} + run: | + set -uo pipefail + shopt -s nullglob + files=("$SARIF_DIR"/*.sarif) + if [ ${#files[@]} -eq 0 ]; then + # No SARIF means the analysis did not produce results to judge. Fail + # loudly rather than reporting a clean scan that never happened. + echo "::error::No SARIF written for $LANGUAGE — cannot conclude the scan was clean." + exit 1 + fi + count=$(jq -rs '[ .[] | .runs[]? | .results[]? ] | length' "${files[@]}") + if [ "$count" != "0" ]; then + echo "::error::CodeQL found ${count} finding(s) for ${LANGUAGE} — see the summary above." + echo "Fix it, or if it is a false positive dismiss the alert in the" + echo "Security tab with a reason, or add a query filter with a comment" + echo "saying why. Silently raising the threshold is not one of the options." + exit 1 + fi + echo "CodeQL ($LANGUAGE): 0 findings." + codeql-gate: name: CodeQL gate # Aggregate check for branch protection, same pattern and rationale as diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index aaac809b..4d90776d 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -84,10 +84,10 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: go-version-file: go.mod cache: true @@ -107,7 +107,7 @@ jobs: run: go vet -tags fleet_host_executor ./... - name: golangci-lint - uses: golangci/golangci-lint-action@v9 + uses: golangci/golangci-lint-action@db9de0fc1a667e1a49d2291a1a042dff081d78f6 # v9 with: # Keep pinned in lockstep with ci.yml + .golangci.yml (see the note # there) so the fast lane and the full gate never disagree. @@ -144,7 +144,7 @@ jobs: # deferring it. steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install ruff env: @@ -175,10 +175,10 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Node - uses: actions/setup-node@v7 + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: # Read from web/.nvmrc so the deployed node major is declared once. node-version-file: web/.nvmrc @@ -213,7 +213,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: # Full history so the merge-base with origin/main resolves and the # linter can diff the branch's new/changed migration files. @@ -236,7 +236,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install gitleaks # Same pinned release + checksum as ci.yml — reproducible, can't be diff --git a/.github/workflows/e2e-canary.yml b/.github/workflows/e2e-canary.yml index 1a6f2d04..aa5aac1c 100644 --- a/.github/workflows/e2e-canary.yml +++ b/.github/workflows/e2e-canary.yml @@ -76,17 +76,17 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: # Read from go.mod so the toolchain is declared once. go-version-file: go.mod check-latest: true - name: Set up Node - uses: actions/setup-node@v7 + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: # Read from web/.nvmrc so the deployed node major is declared once. node-version-file: web/.nvmrc @@ -130,7 +130,7 @@ jobs: - name: Upload canary report on failure if: ${{ failure() }} - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: canary-report path: | diff --git a/.github/workflows/govulncheck-scheduled.yml b/.github/workflows/govulncheck-scheduled.yml index 83f5c980..6a09c2b4 100644 --- a/.github/workflows/govulncheck-scheduled.yml +++ b/.github/workflows/govulncheck-scheduled.yml @@ -47,12 +47,12 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: main # always scan the tip of main, not a PR branch - name: Set up Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: go-version-file: go.mod cache: true @@ -91,7 +91,7 @@ jobs: echo '```' >> "$GITHUB_STEP_SUMMARY" - name: Upload scan results - uses: github/codeql-action/upload-sarif@v4 + uses: github/codeql-action/upload-sarif@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) # Skip cleanly (rather than error "file not found") if the scan step # failed before govulncheck could write the SARIF. if: ${{ !cancelled() && hashFiles('govulncheck.sarif') != '' }} diff --git a/.github/workflows/grype-scheduled.yml b/.github/workflows/grype-scheduled.yml index 20e9b09f..b81ef152 100644 --- a/.github/workflows/grype-scheduled.yml +++ b/.github/workflows/grype-scheduled.yml @@ -32,7 +32,7 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: main # always scan the tip of main, not a PR branch @@ -79,7 +79,7 @@ jobs: --output sarif=grype-results.sarif - name: Upload weekly scan results - uses: github/codeql-action/upload-sarif@v4 + uses: github/codeql-action/upload-sarif@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) # Skip cleanly (rather than error "file not found") if an earlier step # failed before grype could write the SARIF. if: ${{ !cancelled() && hashFiles('grype-results.sarif') != '' }} diff --git a/.github/workflows/publish-sandbox-image.yml b/.github/workflows/publish-sandbox-image.yml index 50393990..3e1f739a 100644 --- a/.github/workflows/publish-sandbox-image.yml +++ b/.github/workflows/publish-sandbox-image.yml @@ -215,13 +215,13 @@ jobs: SHA: ${{ github.sha }} steps: - name: Checkout caller repo - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 # The build script lives in fleet core; a client-config caller doesn't # have it. Checking it out into a subdir keeps ONE canonical builder # (same manifest parsing, same flags) instead of a drifting copy per repo. - name: Checkout fleet (build script) - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: repository: ElcanoTek/fleet ref: ${{ inputs.fleet_ref || 'main' }} diff --git a/.github/workflows/screenshots.yml b/.github/workflows/screenshots.yml index a5647257..7eef6e50 100644 --- a/.github/workflows/screenshots.yml +++ b/.github/workflows/screenshots.yml @@ -50,10 +50,10 @@ jobs: ORCHESTRATOR_SERVER_URL: http://127.0.0.1:18000 steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Node - uses: actions/setup-node@v7 + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: # Read from web/.nvmrc so the deployed node major is declared once. node-version-file: web/.nvmrc @@ -75,7 +75,7 @@ jobs: run: npm run test:e2e:screenshots || echo "::warning::GUI screenshot capture failed; keeping the existing images" - name: Set up Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: # Read from go.mod so the toolchain is declared once. go-version-file: go.mod diff --git a/.github/workflows/semgrep.yml b/.github/workflows/semgrep.yml index 9eabb465..fcceceb9 100644 --- a/.github/workflows/semgrep.yml +++ b/.github/workflows/semgrep.yml @@ -8,43 +8,50 @@ # and rules that are cheap to write. It complements CodeQL; it does not replace # it, and it must not be treated as a substitute for the taint analysis. # -# RULESET SCOPE IS EVIDENCE-BASED, and the evidence is unflattering to the -# obvious choice. Running the broad registry packs (p/golang, p/javascript, -# p/python) over this tree produced 55 findings, and every one of the 6 -# non-Actions findings was a FALSE POSITIVE: +# RULESET SCOPE IS EVIDENCE-BASED. The broad registry packs (p/golang, +# p/javascript, p/python) plus p/github-actions were run over this tree and every +# finding was triaged by hand. Result: 51 real findings and 6 false positives. # -# go.lang.security.injection.open-redirect cmd/fleet/tls.go:109 -# Standard HTTP->HTTPS upgrade to the SAME host. Already carries a -# //nolint:gosec G710 with exactly this reasoning. -# go.lang.security.audit.crypto.math_random internal/runner/runner.go:28 +# The 51 were all one rule — `github-actions-mutable-action-tag`: actions pinned +# to a MUTABLE tag (`actions/checkout@v7`) rather than an immutable commit SHA. A +# moved tag runs attacker-controlled code with this repo's token. Every one is +# now FIXED: all 53 action references across every workflow are pinned to a +# 40-hex commit SHA with the version in a trailing comment (which is also the +# form Dependabot updates). +# +# The 6 false positives are suppressed at the line with a `nosemgrep:` comment +# naming the specific rule and the reason. They are worth knowing about, because +# three of them were ALREADY formally triaged and suppressed for gosec (which +# runs inside golangci-lint and already blocks), and one of them is actively +# wrong: +# +# open-redirect cmd/fleet/tls.go +# HTTP->HTTPS upgrade to the SAME host. Already //nolint:gosec G710. +# math-random-used internal/runner/runner.go # math/rand/v2, used once, for +/-10% jitter on a retry interval. -# go...cookie-missing-secure internal/sched/handlers/elcano.go:155 -# A DELETION cookie (Value="", MaxAge=-1) carrying no secret; Secure is -# conditional so logout works over plain-HTTP dev. Already //nolint:gosec G124. -# go...unsafe-deserialization-interface internal/mcp/httptool.go:254 +# cookie-missing-secure internal/sched/handlers/elcano.go +# A DELETION cookie (Value="", MaxAge=-1), no secret. Already //nolint G124. +# unsafe-deserialization-interface internal/mcp/httptool.go # json.Unmarshal into interface{} is REQUIRED — the value feeds a jq program # over arbitrary JSON. A concrete struct is not expressible. -# javascript...x-frame-options-misconfiguration web/src/proxy.ts:99 +# x-frame-options-misconfiguration web/src/proxy.ts # The header value is the literal string "DENY". No user input reaches it. -# python...insecure-file-permissions internal/sandbox/fileops.py:77 -# Advises 0o644 for a SANDBOX DIRECTORY, i.e. world-readable. Taking that -# advice would be a security regression; 0750 is the file-tool contract. +# insecure-file-permissions internal/sandbox/fileops.py +# Advises 0o644 — WORLD-READABLE — for a sandbox directory. Following it +# would be a security REGRESSION. 0750 is the file-tool contract. # -# Three of those six were already formally triaged and suppressed for gosec, -# which runs inside golangci-lint and already blocks. Re-reporting adjudicated -# findings is how a scanner trains people to ignore it, so those packs are NOT -# enabled here. If you want them, run them locally for a one-off audit. +# Each suppression was mutation-tested: removing it makes the finding reappear, +# so a green scan means the waivers are doing the work rather than the rules +# having silently stopped matching. # -# What IS enabled is the one pack that earned it: p/github-actions found 49 -# instances of a real class of issue nothing else in this repo checks — actions -# pinned to a MUTABLE tag (`actions/checkout@v7`) instead of an immutable commit -# SHA. A moved tag executes attacker-controlled code with this repo's token. +# THIS LANE BLOCKS. `--error` makes semgrep exit non-zero on any finding, and +# there is no `continue-on-error`, so a NEW finding fails the job. That is only +# defensible because the tree is currently at ZERO unsuppressed findings across +# all four packs — verified locally before this was turned on. A gate switched on +# over an unfixed backlog is a gate people learn to ignore. # -# ADVISORY, NOT BLOCKING, and deliberately so: those 49 findings are all real, -# so `--error` here would mean a red gate until every action in every workflow -# is repinned to a SHA. That is a worthwhile change and its own PR; failing CI -# for a backlog nobody has scheduled just teaches people to ignore the lane. -# Flip `continue-on-error` off in the same PR that does the repinning. +# Adding a suppression is therefore a reviewable act: it shows up in the diff +# next to a reason, which is the property this whole lane exists to have. name: Semgrep on: @@ -71,14 +78,10 @@ jobs: name: Semgrep scan runs-on: ubuntu-latest timeout-minutes: 15 - # See the header: the enabled pack's findings are real but unactioned, so - # this lane reports and never blocks. It is NOT here because the findings - # are doubted. - continue-on-error: true steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install semgrep # No actions/setup-python step on purpose: ubuntu-latest already ships a @@ -99,20 +102,29 @@ jobs: semgrep --version - name: Scan - env: - # p/github-actions only — see the header for the measured reason the - # broad language packs are excluded. - SEMGREP_RULES: p/github-actions run: | set -uo pipefail # --metrics=off: never phone home from CI. - # No --error: this lane reports, it does not gate (see header). - semgrep scan --config "$SEMGREP_RULES" --metrics=off \ - --json -o semgrep.json --quiet || true + # --error: exit non-zero on any finding. THIS is what makes the lane a + # gate. The tree is at zero unsuppressed findings, so a failure here + # means a new one arrived. + # The JSON is written first and the exit status captured, so the + # summary step below still runs and still says WHAT failed — a gate + # that fails without printing the finding is a gate nobody can act on. + semgrep scan \ + --config p/github-actions \ + --config p/golang \ + --config p/javascript \ + --config p/python \ + --metrics=off --error --json -o semgrep.json --quiet + status=$? if [ ! -s semgrep.json ]; then echo "semgrep produced no JSON — treating as a scan failure" >&2 exit 1 fi + echo "semgrep exit status: $status" + # Defer the failure to the gate step so the summary prints first. + echo "$status" > semgrep.status - name: Summarize findings to the job log # Same reasoning as codeql.yml's summary step: a findings report that @@ -146,12 +158,29 @@ jobs: fi } | tee -a "$GITHUB_STEP_SUMMARY" + - name: Fail on findings + # Separate from the scan step purely for ordering: the summary above has + # already printed the per-rule breakdown by the time this fails, so the + # job log leads with WHAT is wrong instead of just that something is. + if: ${{ !cancelled() }} + run: | + set -uo pipefail + status=$(cat semgrep.status 2>/dev/null || echo 1) + count=$(jq '(.results // []) | length' semgrep.json 2>/dev/null || echo '?') + if [ "$status" != "0" ]; then + echo "::error::Semgrep found ${count} unsuppressed finding(s) — see the summary above." + echo "Fix the finding, or, if it is a false positive, add a line-level" + echo "\`nosemgrep: \` comment stating WHY. Both are reviewable in the diff." + exit 1 + fi + echo "Semgrep: 0 unsuppressed findings." + - name: Upload findings as an artifact # The repo is public and the results are not sensitive, so the raw JSON # is kept for a fixing agent (or a human) to consume without re-running # the scan. if: ${{ !cancelled() }} - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: semgrep-findings path: semgrep.json diff --git a/AGENTS.md b/AGENTS.md index e2a8a964..23d4c721 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -46,8 +46,11 @@ gitleaks secret scan. **Every job must be green before merge.** Tests are deterministic without a live model: use the fake-LLM seam (`internal/fakellm` via `OPENROUTER_BASE_URL`), never a real key. -CodeQL (security queries) and Semgrep (Actions supply chain) also run per PR but -are **advisory**, not part of `ci-gate` — see [`docs/SCANNING.md`](docs/SCANNING.md). +CodeQL (security queries) and Semgrep (Go/JS/Python SAST + Actions supply chain) +also run per PR and **fail on any finding**. They are not part of `ci-gate` — +`needs` cannot cross workflow files — so they report as their own checks +(`CodeQL gate`, `Semgrep scan`). Both are at zero findings today; keeping them +there is the point. See [`docs/SCANNING.md`](docs/SCANNING.md). ## Repository map diff --git a/CHANGELOG.md b/CHANGELOG.md index 1463f041..6c7250d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,47 @@ prior versions are listed because none have shipped. ### Fixed +- **The scanners now block, and the repo passes them.** Turning a gate on over an + unfixed backlog is how a gate becomes something people route around, so + everything they reported was fixed or adjudicated first. + + - **All 53 action references pinned to commit SHAs.** Semgrep's + `github-actions-mutable-action-tag` found 51 instances of actions referenced + by a mutable tag (`actions/checkout@v7`); if a tag moves, + attacker-controlled code runs with this repo's `GITHUB_TOKEN`. Every `uses:` + across all 12 workflows is now `@<40-hex-sha> # ` — the form + Dependabot updates, and `.github/dependabot.yml` already watches the + `github-actions` ecosystem. Each SHA is the commit the previously-used tag + resolved to at pin time, so the pin does not smuggle in a version bump. + + - **Semgrep blocks over all four packs** (`p/github-actions`, `p/golang`, + `p/javascript`, `p/python`) with `--error` and no `continue-on-error`. The 6 + false positives are suppressed at the line with `nosemgrep: ` plus a + reason — three of them were *already* triaged and suppressed for gosec, and + one (`0o644` for a sandbox directory) would have been a security regression + if followed. Every suppression was mutation-tested: removing it makes the + finding reappear, so a green scan means the waivers work rather than the + rules having silently stopped matching. + + - **CodeQL fails on findings.** Previously the analyze step exited 0 whether it + found nothing or a hundred alerts, so a red check could only ever mean "the + scanner broke" — which is exactly how the Go toolchain break hid for weeks. + Threshold is any finding, safe because the security suite reports zero across + all four languages. + + Both scanners report as their own checks (`CodeQL gate`, `Semgrep scan`) rather + than through `ci-gate`, because a job's `needs` cannot reach across workflow + files. **Making a red check actually block a merge still requires adding those + two checks to the branch ruleset** — a workflow file cannot make itself + required. + + Two knock-on fixes found while doing this: SHA pinning broke two regexes in + `scripts/check_versions_test.go` that matched `golangci-lint-action@v\d+`, and + they fail *open* by skipping — so they were widened to tolerate a pinned ref + plus its trailing version comment, and mutation-tested to confirm they still + bite. And a standalone `nosemgrep` comment inside a Go import block breaks + `goimports`, so that one waiver is a trailing comment instead. + - **Python had no linter, and two scanners were pointed at ground already covered.** Reshaped the scanning stack so each tool owns one job ([`docs/SCANNING.md`](docs/SCANNING.md)): diff --git a/cmd/fleet/tls.go b/cmd/fleet/tls.go index 2d217e5e..87447690 100644 --- a/cmd/fleet/tls.go +++ b/cmd/fleet/tls.go @@ -106,6 +106,7 @@ func startRedirectServer(addr string, m *autocert.Manager) { func redirectToHTTPS(w http.ResponseWriter, r *http.Request) { target := "https://" + stripPort(r.Host) + r.URL.RequestURI() //nolint:gosec // G710: standard HTTP→HTTPS upgrade to the SAME Host (scheme-only); not an open redirect to an arbitrary origin. + // nosemgrep: go.lang.security.injection.open-redirect.open-redirect -- same reasoning as the G710 waiver above: the target is built from r.Host with the scheme forced to https, so it can only ever point back at the host the client already asked for. Not attacker-chosen. http.Redirect(w, r, target, http.StatusMovedPermanently) } diff --git a/docs/CODEQL.md b/docs/CODEQL.md index 04fa1ea1..c9a7fc10 100644 --- a/docs/CODEQL.md +++ b/docs/CODEQL.md @@ -384,9 +384,17 @@ inheriting a backlog. ## Merge gating today — unchanged, with the lever put within reach -CodeQL is **not** blocking. `ci-gate` remains the single required status check on -`main` (see [`.github/CODEOWNERS`](../.github/CODEOWNERS)), so a red CodeQL job -does not stop a merge — exactly as under default setup. +**A finding now turns the check red.** A `Fail on findings` step fails the job on +any finding, at a threshold of *any* — safe to set because the security suite +currently reports zero on this tree, so there is no backlog to grandfather. +Without that step the analyze step exits 0 whether it found nothing or a hundred +alerts, so a red check could only ever mean "the scanner broke" — which is +exactly how the toolchain break hid for weeks. + +**A red check does not yet block a merge.** `ci-gate` remains the single required +status check on `main` (see [`.github/CODEOWNERS`](../.github/CODEOWNERS)). +Closing that half means adding `CodeQL gate` to the ruleset, which a workflow +file cannot do for itself. It *cannot* be folded into `ci-gate`: a job's `needs` cannot reach across workflow files. So `codeql.yml` carries its own aggregate **`CodeQL gate`** job, @@ -409,7 +417,8 @@ changes nothing on its own: single entry covers every language in the matrix, now and after future matrix changes. -**Recommendation: leave it advisory for a short while first.** Not out of +**Recommendation on the ruleset half: add it once you have seen a few green +promotions.** Not out of caution for its own sake — because of this specific incident. The analysis spent weeks red for a toolchain reason unrelated to any diff, and a required check in that state blocks *every* merge, including the promote PR that would carry the diff --git a/docs/SCANNING.md b/docs/SCANNING.md index 2797c8a9..21c6f50a 100644 --- a/docs/SCANNING.md +++ b/docs/SCANNING.md @@ -15,8 +15,8 @@ security queries only) and [`TESTING.md`](TESTING.md) (the rest of the ladder). | `govulncheck` | Go dependency CVEs (called symbols) | ~30s | **blocks** (`ci-gate`) | job log + Security tab | | `grype` | sandbox image CVEs (fixable CRITICAL) | ~1m | **blocks** (`ci-gate`) | job log + Security tab | | `gitleaks` | secrets, every branch | ~10s | **blocks** (`ci-gate`) | job log | -| CodeQL | **interprocedural taint / security** | ~2m | advisory | job log + Security tab | -| **Semgrep** | **GitHub Actions supply chain** | ~20s | advisory | job log + artifact | +| CodeQL | **interprocedural taint / security** | ~2m | **fails on findings** (check red; see below) | job log + Security tab | +| **Semgrep** | **Go/JS/Python SAST + Actions supply chain** | ~40s | **fails on findings** | job log + artifact | Two things were added here (**ruff**, **Semgrep**) and one was narrowed (**CodeQL**, to security queries only). @@ -58,7 +58,7 @@ violation is a regression rather than noise in a backlog: ruff-formatted, so failing on it would block every PR on a reformat nobody scheduled. The advisory output keeps the size of that decision visible. -### CodeQL owns interprocedural taint (narrowed, advisory) +### CodeQL owns interprocedural taint (narrowed, fails on findings) CodeQL is the only tool in this stack that does cross-function dataflow, and that is exactly the shape of fleet's headline invariants: *a credential must not reach @@ -69,49 +69,72 @@ So CodeQL keeps its security queries and gives up everything else — the qualit suite duplicated `golangci-lint`/`oxlint` for Go and JS, and ruff is a better fit for Python. Full reasoning and measurements in [`CODEQL.md`](CODEQL.md). -Its security suite currently reports **zero findings** on this tree. +Its security suite currently reports **zero findings** on this tree, which is +what makes it safe to gate: a `Fail on findings` step now fails the job on any +finding, so a red `Analyze (…)` check means the *code* has a problem rather than +just "the scanner broke". That distinction is the whole reason the Go toolchain +break sat unnoticed for weeks. -### Semgrep owns GitHub Actions supply chain (new, advisory) +### Semgrep owns fast multi-language SAST + Actions supply chain (new, blocking) Semgrep is the opposite trade from CodeQL: seconds instead of minutes, no database build, findings straight to stdout, rules cheap to write. That makes it -the natural fit for an agent-driven loop — and it is why the obvious move is to -point it at everything. +the right fit for an agent-driven loop. -**That move was measured and rejected.** The broad registry packs (`p/golang`, -`p/javascript`, `p/python`) produced 55 findings on this tree, and **all 6 -non-Actions findings were false positives:** +All four packs run — `p/github-actions`, `p/golang`, `p/javascript`, `p/python` — +and the lane **blocks** (`--error`, no `continue-on-error`). Getting there meant +fixing every real finding and adjudicating every false one. -| finding | why it is wrong | -| --- | --- | -| `open-redirect` — `cmd/fleet/tls.go:109` | Standard HTTP→HTTPS upgrade to the **same** host. Already carries `//nolint:gosec G710` saying so. | -| `math-random-used` — `internal/runner/runner.go:28` | `math/rand/v2`, used once, for ±10% jitter on a retry interval. | -| `cookie-missing-secure` — `internal/sched/handlers/elcano.go:155` | A **deletion** cookie (`Value=""`, `MaxAge=-1`), no secret; `Secure` is conditional so logout works over plain-HTTP dev. Already `//nolint:gosec G124`. | -| `unsafe-deserialization-interface` — `internal/mcp/httptool.go:254` | `json.Unmarshal` into `interface{}` is **required** — the value feeds a jq program over arbitrary JSON. A concrete struct is not expressible. | -| `x-frame-options-misconfiguration` — `web/src/proxy.ts:99` | The header value is the literal string `"DENY"`. No user input reaches it. | -| `insecure-file-permissions` — `internal/sandbox/fileops.py:77` | Advises `0o644` for a **sandbox directory**, i.e. world-readable. Taking that advice would be a security **regression**; `0750` is the file-tool contract. | - -Three of those six were **already formally triaged and suppressed for `gosec`**, -which runs inside `golangci-lint` and already blocks. A scanner that re-reports -adjudicated findings is how you teach a team to ignore it, so those packs are not -enabled. Run them locally for a one-off audit if you want them. - -What Semgrep *does* own is the pack that earned it. `p/github-actions` found -**51 instances of one real issue nothing else in this repo checks**: actions -pinned to a **mutable tag** (`actions/checkout@v7`) rather than an immutable -commit SHA. A moved tag runs attacker-controlled code with this repo's token. +**The 51 real findings: mutable action tags. All fixed.** +`p/github-actions` found one issue class nothing else in this repo checks — +actions referenced by a **mutable tag** (`actions/checkout@v7`) instead of an +immutable commit SHA. If a tag moves, attacker-controlled code runs with this +repo's `GITHUB_TOKEN`. Every one of the **53** action references across all 12 +workflows is now pinned: + +```yaml +uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 ``` -51 [WARNING] github-actions-mutable-action-tag -``` -Spread across every workflow (18 in `ci.yml`, 7 in `dev-ci.yml`, …). +Each SHA is the commit the previously-used tag resolved to at pin time, so the +pin is behaviourally identical to the runs already verified green — a pin should +not smuggle in a version bump. The trailing version comment is also the form +Dependabot reads and updates, and `.github/dependabot.yml` already watches the +`github-actions` ecosystem, so these stay current without hand-editing. + +Two `uses:` lines are deliberately left on `@main`: both are inside **comments** +in `build-sandbox-image.yml` / `publish-sandbox-image.yml`, documenting how a +downstream bundle repo calls fleet's reusable workflows. `@main` is the right +guidance for a consumer tracking fleet, and Semgrep does not flag them (a YAML +comment is not a `uses:` key). + +**The 6 false positives: suppressed at the line, with reasons.** -**It is advisory, and the reason is honesty about scheduling, not doubt about the -findings.** All 51 are real. `--error` here would mean a red gate until every -action in every workflow is repinned to a SHA — a worthwhile change, and its own -PR. Failing CI for a backlog nobody has scheduled just teaches people to ignore -the lane. Flip `continue-on-error` off in the same PR that does the repinning. +Worth reading, because three were **already formally triaged and suppressed for +`gosec`** — which runs inside `golangci-lint` and already blocks — and one is +actively wrong: + +| finding | why it is wrong | +| --- | --- | +| `open-redirect` — `cmd/fleet/tls.go` | Standard HTTP→HTTPS upgrade to the **same** host. Already `//nolint:gosec G710`. | +| `math-random-used` — `internal/runner/runner.go` | `math/rand/v2`, used once, for ±10% jitter on a retry interval. | +| `cookie-missing-secure` — `internal/sched/handlers/elcano.go` | A **deletion** cookie (`Value=""`, `MaxAge=-1`), no secret; `Secure` is conditional so logout works over plain-HTTP dev. Already `//nolint:gosec G124`. | +| `unsafe-deserialization-interface` — `internal/mcp/httptool.go` | `json.Unmarshal` into `interface{}` is **required** — the value feeds a jq program over arbitrary JSON. A concrete struct cannot express "whatever shape the response had". | +| `x-frame-options-misconfiguration` — `web/src/proxy.ts` | The header value is the literal string `"DENY"`. No user input reaches it. | +| `insecure-file-permissions` — `internal/sandbox/fileops.py` | Advises `0o644` — **world-readable** — for a sandbox directory. Following it would be a security **regression**; `0750` is the file-tool contract. | + +Each carries a line-level `nosemgrep: ` naming the specific rule and the +reason. Scoped to the rule, so a *different* rule firing on the same line still +reports. + +**Every suppression was mutation-tested.** Removing it makes the finding +reappear; with it, the finding is gone. That matters because "0 findings" has two +explanations — the waivers work, or the rules silently stopped matching — and +only one of them is safety. Checked across all three comment syntaxes (Go `//`, +Python `#`, TypeScript `//`), including the one waiver that had to become a +*trailing* comment because a standalone comment inside a Go import block breaks +`goimports`. ## Findings are readable from the job log, on purpose @@ -137,8 +160,19 @@ withholding them buys nothing. ## What gates, and what a required check actually means -Everything in the "blocks" column above is reached through **`ci-gate`**, the -single required status check on `main`. The two scanners are outside it. +The lint/test/build lanes reach `main`'s single required status check through +**`ci-gate`**. ruff is inside it. The two scanners are not — they cannot be, since +a job's `needs` cannot reach across workflow files — so each carries its own +aggregate gate job (`CodeQL gate`) or fails directly (Semgrep). + +**Both scanners now fail their job on any finding.** That is what makes them +gates rather than reports, and it is only defensible because the tree is at zero +unsuppressed findings in both — verified before switching either on. A gate +turned on over an existing backlog is a gate people route around. + +**One half is still yours to close:** a failing check only *blocks a merge* if it +is a required status check. Add **`CodeQL gate`** and **`Semgrep scan`** to the +"Main" ruleset to finish it. A workflow file cannot make itself required. The distinction that matters for anyone tightening this later: @@ -162,12 +196,23 @@ Stated rather than left for rediscovery: dependency CVE gate. Dependabot opens npm PRs, but Dependabot alerts do not block anything. This is the largest remaining hole in the stack — arguably larger than anything CodeQL gating would fix. -- **Actions are pinned to mutable tags.** 51 instances, per above. Semgrep now - reports it; nothing yet fixes it. - **`_test.go` files are outside CodeQL's database** (621 files) — `autobuild` builds packages, not tests. Unchanged from default setup. -- **`ruff format` is not enforced**, per above. +- **`ruff format` is not enforced.** The tree has never been ruff-formatted: + 9 of 13 files differ, a **3725-line** diff. That is cosmetics, not a finding, + and landing it inside a security change would bury the security change. One + command (`ruff format .`) plus flipping the advisory step to a gate, whenever + someone wants it. +- **ruff's rule set is narrow**, so some real bug classes go unreported. + Measured: `--select B,SIM,S` adds **21** findings, of which the genuinely + interesting ones are 2 × `B905` (`zip()` without `strict=` — silent + truncation) and 1 × `SIM115` (file opened without a context manager). The + other 18 are `try`/`except`/`pass` in deliberate best-effort cleanup paths. + Worth a focused pass; not folded in here. - **Semgrep's own rule packs are network-fetched** from the registry at scan time. The semgrep *version* is pinned; the *rules* are not, so a registry - change can move findings without a diff here. Acceptable for an advisory lane; - it would need a vendored ruleset before this could block. + change can move findings without a diff here. This matters more now the lane + blocks: a registry-side rule addition can turn CI red with no commit to blame, + the same class of surprise `govulncheck-scheduled.yml` was created to absorb. + Vendoring the rules would fix it at the cost of never getting new ones. Left + as-is deliberately, and named here so a mystery red build has a first suspect. diff --git a/internal/mcp/httptool.go b/internal/mcp/httptool.go index 7eb0a82a..e5bfec5a 100644 --- a/internal/mcp/httptool.go +++ b/internal/mcp/httptool.go @@ -251,6 +251,7 @@ func executeHTTPTool(ctx context.Context, client *http.Client, spec HTTPToolSpec // Multiple jq outputs are newline-joined; scalars/objects are rendered as compact // JSON. func applyResponseJQ(program string, body []byte) (out string, ok bool, err error) { + // nosemgrep: go.lang.security.deserialization.unsafe-deserialization-interface.go-unsafe-deserialization-interface -- interface{} is REQUIRED here, not a shortcut: the value is handed straight to a jq program, which operates on arbitrary JSON by definition. A concrete struct cannot express "whatever shape the response had". var input interface{} if jsonErr := json.Unmarshal(body, &input); jsonErr != nil { //nolint:nilerr // intentional: a non-JSON body is not an error — ok=false signals "pass the raw body through unfiltered" (response_jq applies only to JSON, per the issue spec). diff --git a/internal/runner/runner.go b/internal/runner/runner.go index b7a2c286..4c9fc11d 100644 --- a/internal/runner/runner.go +++ b/internal/runner/runner.go @@ -25,7 +25,7 @@ import ( "errors" "fmt" "log" - "math/rand/v2" + "math/rand/v2" // nosemgrep: go.lang.security.audit.crypto.math_random.math-random-used -- used once, for +/-10% jitter on a retry interval (see rand.Int64N below). Nothing here is a secret, a token, or an identity; crypto/rand would only make backoff slower. "os" "strconv" "strings" diff --git a/internal/sandbox/fileops.py b/internal/sandbox/fileops.py index 8b48da85..c7e00688 100644 --- a/internal/sandbox/fileops.py +++ b/internal/sandbox/fileops.py @@ -74,6 +74,7 @@ def _open_dir_at(parent_fd, name, create): fd = os.open(name, flags, dir_fd=parent_fd) if created: # mkdir is umask-filtered; the file-tool contract is exact 0750. + # nosemgrep: python.lang.security.audit.insecure-file-permissions.insecure-file-permissions -- the rule advises 0o644, i.e. WORLD-READABLE, for a sandbox directory. Following it would be a security regression. 0750 is the file-tool contract and is deliberately tighter than the suggestion. os.fchmod(fd, 0o750) return fd except OSError as exc: diff --git a/internal/sched/handlers/elcano.go b/internal/sched/handlers/elcano.go index 440c498f..6b5d3dae 100644 --- a/internal/sched/handlers/elcano.go +++ b/internal/sched/handlers/elcano.go @@ -152,6 +152,7 @@ func (h *Handlers) ElcanoLogout(w http.ResponseWriter, r *http.Request) { // attributes must mirror how auth originally set the cookie for the browser // to actually clear it — forcing Secure here unconditionally would prevent // logout from clearing the cookie over plain HTTP. + // nosemgrep: go.lang.security.audit.net.cookie-missing-secure.cookie-missing-secure -- same reasoning as the G124 waiver: this is a DELETION cookie (Value="", MaxAge=-1) carrying no secret, and its attributes must mirror how auth set it or the browser will not clear it. Forcing Secure unconditionally would break logout over plain-HTTP dev. http.SetCookie(w, &http.Cookie{ //nolint:gosec // G124: deletion cookie (no secret); Secure is conditional on HTTPS so logout works over plain-HTTP dev, mirroring how the cookie was set — see comment above. Name: h.config.ElcanoCookieName, Value: "", diff --git a/scripts/check_versions_test.go b/scripts/check_versions_test.go index 72d989d9..56a90312 100644 --- a/scripts/check_versions_test.go +++ b/scripts/check_versions_test.go @@ -173,7 +173,7 @@ func TestDuplicatedToolPinsAgree(t *testing.T) { {"GRYPE_SHA256", ".github/workflows/grype-scheduled.yml", regexp.MustCompile(`GRYPE_SHA256:\s*'([^']+)'`)}, {"GITLEAKS_VERSION", ".github/workflows/dev-ci.yml", regexp.MustCompile(`GITLEAKS_VERSION:\s*'([^']+)'`)}, {"RUFF_VERSION", ".github/workflows/dev-ci.yml", regexp.MustCompile(`RUFF_VERSION:\s*'([^']+)'`)}, - {"golangci-lint version", ".github/workflows/dev-ci.yml", regexp.MustCompile(`golangci-lint-action@v\d+\s+with:\s+(?:#[^\n]*\n\s+)*version:\s*(v[\d.]+)`)}, + {"golangci-lint version", ".github/workflows/dev-ci.yml", regexp.MustCompile(`golangci-lint-action@\S+[^\n]*\n\s*with:\s+(?:#[^\n]*\n\s+)*version:\s*(v[\d.]+)`)}, } { a := tc.re.FindStringSubmatch(ci) b := tc.re.FindStringSubmatch(readFile(t, root, tc.other)) @@ -315,7 +315,7 @@ func TestGoMinorAgreesEverywhere(t *testing.T) { func TestGolangciLintPinAgreesWithDocs(t *testing.T) { root := repoRoot(t) - pin := regexp.MustCompile(`golangci-lint-action@v\d+\s+with:\s+(?:#[^\n]*\n\s+)*version:\s*(v[\d.]+)`). + pin := regexp.MustCompile(`golangci-lint-action@\S+[^\n]*\n\s*with:\s+(?:#[^\n]*\n\s+)*version:\s*(v[\d.]+)`). FindStringSubmatch(readFile(t, root, ".github/workflows/ci.yml")) if pin == nil { t.Fatal("ci.yml: could not find the golangci-lint-action version pin") diff --git a/web/src/proxy.ts b/web/src/proxy.ts index a37a5873..23afb92d 100644 --- a/web/src/proxy.ts +++ b/web/src/proxy.ts @@ -96,6 +96,7 @@ function decorate(res: NextResponse, pathname: string): NextResponse { res.headers.set(BUILD_ID_HEADER, currentBuildId()); res.headers.set("Cache-Control", "no-store, must-revalidate"); res.headers.set("Content-Security-Policy", contentSecurityPolicy(pathname)); + // nosemgrep: javascript.express.security.x-frame-options-misconfiguration.x-frame-options-misconfiguration -- the value is the literal string "DENY". No user input reaches this header; the rule fires on the shape of the call, not on a real taint path. res.headers.set("X-Frame-Options", "DENY"); res.headers.set("X-Content-Type-Options", "nosniff"); res.headers.set("Referrer-Policy", "strict-origin-when-cross-origin"); From 358bcbcf49070cbed1c8da30f39859a19deac836 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 13:27:46 +0000 Subject: [PATCH 08/34] Report scan coverage, not just the verdict MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "No findings." on its own is indistinguishable from "scanned nothing", which is the green-but-vacuous outcome this whole stack exists to rule out. Verifying the first blocking run required downloading the artifact and inspecting the JSON by hand — the log said the scan was clean but not what it had looked at. That is an instrumentation gap, and it is the same class of gap as CodeQL not printing its own findings. Both summary steps now print coverage alongside the verdict: - Semgrep: files scanned, files skipped, and a per-extension breakdown, so it is visible at a glance that every pack actually applied to its language rather than one silently matching nothing. Parse/scan errors are listed with their paths instead of only being counted, since an error means a rule or file did not fully run. - CodeQL: the number of files in the database, read from the source archive the database was built from. Note CODEQL_DB is not always the matrix language — the extractor names the javascript-typescript database "javascript". Both jq blocks were dry-run against the real semgrep.json artifact from run 32575445870 before being committed, which is where these numbers come from: 898 files scanned, 0 skipped, 427 .go / 298 .ts / 134 .tsx / 13 .py / 22 yml+yaml, 3 warn-level parse errors in 2 files. Signed-off-by: Brad Flaugher --- .github/workflows/codeql.yml | 11 +++++++++++ .github/workflows/semgrep.yml | 23 ++++++++++++++++++++--- 2 files changed, 31 insertions(+), 3 deletions(-) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 09cfb216..27af975b 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -208,6 +208,9 @@ jobs: env: SARIF_DIR: ${{ runner.temp }}/codeql-sarif LANGUAGE: ${{ matrix.language }} + # The database directory is named after the EXTRACTOR, which is not + # always the matrix language: javascript-typescript -> javascript. + CODEQL_DB: ${{ matrix.language == 'javascript-typescript' && 'javascript' || matrix.language }} run: | set -uo pipefail shopt -s nullglob @@ -237,6 +240,14 @@ jobs: end ' "${files[@]}" echo '```' + # COVERAGE, not just the verdict: "No findings." alone cannot be + # told apart from "analyzed nothing", which is the exact + # green-but-vacuous outcome this workflow exists to rule out. + # The source archive is the file set the database was built from. + src="${{ runner.temp }}/codeql_databases/$CODEQL_DB/src.zip" + if [ -f "$src" ]; then + echo "files in the $LANGUAGE database: $(unzip -Z1 "$src" 2>/dev/null | grep -vc '/$' || echo '?')" + fi fi } | tee -a "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/semgrep.yml b/.github/workflows/semgrep.yml index fcceceb9..f4d8b2a0 100644 --- a/.github/workflows/semgrep.yml +++ b/.github/workflows/semgrep.yml @@ -149,12 +149,29 @@ jobs: end ' semgrep.json echo '```' - # Scan errors are not findings but they do mean coverage was lost, - # so they get their own line rather than being dropped silently. + # COVERAGE, not just the verdict. "No findings." on its own is + # indistinguishable from "scanned nothing" — the exact + # green-but-vacuous failure this whole stack exists to rule out. So + # print what was actually looked at, and the per-language breakdown + # that shows every pack really applied. + echo '' + echo "files scanned: $(jq '(.paths.scanned // []) | length' semgrep.json)" + echo "files skipped: $(jq '(.paths.skipped // []) | length' semgrep.json)" + echo 'by extension:' + jq -r ' + (.paths.scanned // []) + | map(split(".") | last) + | group_by(.) | map({e: .[0], n: length}) + | sort_by(-.n) | .[:8][] + | " .\(.e): \(.n)" + ' semgrep.json + # Scan errors are not findings but they do mean a rule or file did + # not fully parse, so they get their own line rather than vanishing. errs=$(jq '(.errors // []) | length' semgrep.json) if [ "$errs" != "0" ]; then echo '' - echo "parse/scan errors (files or rules that did not fully run): $errs" + echo "parse/scan errors (a rule or file that did not fully run): $errs" + jq -r '(.errors // [])[] | " [\(.level)] \(.path // "?")"' semgrep.json | sort -u fi } | tee -a "$GITHUB_STEP_SUMMARY" From 056a882a6477ecedb5c9481f81a6f0f9f637c916 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 13:35:37 +0000 Subject: [PATCH 09/34] Pass runner.temp through env, not inline in the run: block MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The new coverage line interpolated `${{ runner.temp }}` directly into a bash `run:` script. That is the exact shape semgrep's gha-curl-pipe-shell and curl-eval rules exist to flag — a GitHub expression expanded into a shell script before the shell ever sees it — and it also breaks their bash sub-parser, so those two rules stopped evaluating against codeql.yml entirely. Caught by the coverage reporting added in the previous commit: parse/scan errors in the Semgrep summary went from 3 to 5, with .github/workflows/codeql.yml newly among them. Without that instrumentation the lane would have stayed green while quietly analyzing this file with two fewer rules — the same green-but-vacuous failure the whole change is built to prevent, introduced by the change itself. $RUNNER_TEMP is the equivalent env var, parses cleanly, and avoids the interpolation entirely. Verified: codeql.yml now reports 0 errors and 0 findings under p/github-actions, and repo-wide errors are back to the 3 pre-existing ones (a bash snippet in build-sandbox-image.yml, a TS type in fixtures.ts). The remaining `${{ runner.temp }}` references are in `with:` and `env:` blocks, which is the correct placement — the value never reaches a shell unparsed. Signed-off-by: Brad Flaugher --- .github/workflows/codeql.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 27af975b..c7eb8410 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -244,7 +244,12 @@ jobs: # told apart from "analyzed nothing", which is the exact # green-but-vacuous outcome this workflow exists to rule out. # The source archive is the file set the database was built from. - src="${{ runner.temp }}/codeql_databases/$CODEQL_DB/src.zip" + # RUNNER_TEMP (the env var), not ${{ runner.temp }}: interpolating a + # GitHub expression straight into a run: block is the shape + # semgrep's gha-curl-pipe-shell / curl-eval rules flag, and it also + # breaks their bash sub-parser — which silently costs coverage on + # this very file. The env var is equivalent and parses. + src="$RUNNER_TEMP/codeql_databases/$CODEQL_DB/src.zip" if [ -f "$src" ]; then echo "files in the $LANGUAGE database: $(unzip -Z1 "$src" 2>/dev/null | grep -vc '/$' || echo '?')" fi From 8d4cf233ce826cee063a04d8aa1edf77e9d7ee8d Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 14:06:25 +0000 Subject: [PATCH 10/34] Apply ruff format to the Python tree Mechanical, behaviour-free reformat of 9 files (~3.7k diff lines), kept as its own commit so the substantive changes around it stay reviewable. Validated before the gate flipped: `ruff check` still clean, every file byte-compiles, the full Go suite passes (the bento/fileops golden tests exercise these scripts), and the fileops.py line-level nosemgrep waiver survived the reformat (re-scanned: 0 python findings). The `ruff format --check` gate lands in the next commit; this commit is what makes that gate start clean instead of red. Signed-off-by: Brad Flaugher --- cmd/fleet/testdata/authstatus_server.py | 9 +- docs/img/gen-open-source-vs-elcano.py | 101 +- .../bento-slides/scripts/bento_doc.py | 53 +- .../bento-slides/scripts/bento_pdf.py | 1773 +++++++++++------ internal/mcp/testdata/dummy_server.py | 15 +- internal/mcp/testdata/noisy_server.py | 10 +- internal/sandbox/fileops.py | 204 +- internal/tools/python_bridge.py | 99 +- scripts/generate-icons.py | 1 + 9 files changed, 1470 insertions(+), 795 deletions(-) diff --git a/cmd/fleet/testdata/authstatus_server.py b/cmd/fleet/testdata/authstatus_server.py index e3d8448a..1b0df7eb 100644 --- a/cmd/fleet/testdata/authstatus_server.py +++ b/cmd/fleet/testdata/authstatus_server.py @@ -3,6 +3,7 @@ Advertises one auth-status tool. Set AUTH_FAIL=1 in the env to make the tools/call result carry isError=true (a failed upstream credential check). """ + import json import os import sys @@ -41,12 +42,16 @@ def main(): elif method == "tools/call": if FAIL: resp["result"] = { - "content": [{"type": "text", "text": "401 Unauthorized: key revoked"}], + "content": [ + {"type": "text", "text": "401 Unauthorized: key revoked"} + ], "isError": True, } else: resp["result"] = { - "content": [{"type": "text", "text": "authenticated: seat 12345 ok"}], + "content": [ + {"type": "text", "text": "authenticated: seat 12345 ok"} + ], "isError": False, } else: diff --git a/docs/img/gen-open-source-vs-elcano.py b/docs/img/gen-open-source-vs-elcano.py index 6c8ad987..c4f7dc5a 100644 --- a/docs/img/gen-open-source-vs-elcano.py +++ b/docs/img/gen-open-source-vs-elcano.py @@ -6,20 +6,20 @@ OUT_DIR = os.path.dirname(os.path.abspath(__file__)) # ---------------------------------------------------------------- layout -PITCH = 142 # horizontal distance between item centers -C0 = 140 # x of first item center -N = 12 # total items -N_OSS = 6 # items under the open-source brace +PITCH = 142 # horizontal distance between item centers +C0 = 140 # x of first item center +N = 12 # total items +N_OSS = 6 # items under the open-source brace W = 2 * C0 + (N - 1) * PITCH H = 368 -TILE = 74 # tile side +TILE = 74 # tile side TILE_TOP = 116 TILE_CY = TILE_TOP + TILE / 2 LABEL_Y1 = TILE_TOP + TILE + 26 LABEL_Y2 = LABEL_Y1 + 20 -TOP_BRACE_Y = TILE_TOP - 12 # endpoints (just above tiles) +TOP_BRACE_Y = TILE_TOP - 12 # endpoints (just above tiles) TOP_BRACE_H = 12 TOP_LABEL_Y = TOP_BRACE_Y - TOP_BRACE_H * 2 - 16 @@ -28,16 +28,40 @@ BOT_LABEL_Y = BOT_BRACE_Y + BOT_BRACE_H * 2 + 34 THEMES = { - "light": dict(ink="#1f2328", text="#1f2328", muted="#59636e", - brace_top="#848d97", brace_bot="#59636e", - fill_op="0.16", stroke_op="0.5"), - "dark": dict(ink="#e6edf3", text="#e6edf3", muted="#9198a1", - brace_top="#767d86", brace_bot="#9198a1", - fill_op="0.22", stroke_op="0.6"), + "light": dict( + ink="#1f2328", + text="#1f2328", + muted="#59636e", + brace_top="#848d97", + brace_bot="#59636e", + fill_op="0.16", + stroke_op="0.5", + ), + "dark": dict( + ink="#e6edf3", + text="#e6edf3", + muted="#9198a1", + brace_top="#767d86", + brace_bot="#9198a1", + fill_op="0.22", + stroke_op="0.6", + ), } -ACCENTS = ["#3b82f6", "#10b981", "#8b5cf6", "#f59e0b", "#f43f5e", "#06b6d4", - "#8b5cf6", "#3b82f6", "#f59e0b", "#10b981", "#f43f5e", "#06b6d4"] +ACCENTS = [ + "#3b82f6", + "#10b981", + "#8b5cf6", + "#f59e0b", + "#f43f5e", + "#06b6d4", + "#8b5cf6", + "#3b82f6", + "#f59e0b", + "#10b981", + "#f43f5e", + "#06b6d4", +] FONT = "-apple-system, 'Segoe UI', 'Helvetica Neue', Arial, sans-serif" @@ -45,11 +69,13 @@ def sparkle(cx, cy, r): """Four-point star, filled with ink.""" k = r * 0.14 - return (f'') + return ( + f'' + ) # ---------------------------------------------------------------- glyphs @@ -62,17 +88,14 @@ def sparkle(cx, cy, r): '' '' '', - # 2 sandboxed tool calls (shield + prompt) '' '' '', - # 3 MCP connector catalog (plug) '' '' '', - # 4 any model (chip + sparkle) '' '' @@ -80,47 +103,37 @@ def sparkle(cx, cy, r): '' '' + sparkle(0, 0, 7), - # 5 budgets & audit (gauge) '' '' '', - # 6 web / TUI / API (terminal monitor) '' '' '' '', - # 7 custom MCP connectors (plug + sparkle) '' '' - '' - + sparkle(13, -12, 6), - + '' + sparkle(13, -12, 6), # 8 data integrations (database + arrow) '' '' '' '', - # 9 add-on capabilities (envelope + sparkle) '' - '' - + sparkle(14, -16, 6), - + '' + sparkle(14, -16, 6), # 10 forward-deployed engineering (person + map pin) '' '' '' '', - # 11 production-ready workflows (calendar + check) '' '' '' '', - # 12 support & operations (lifebuoy) '' '' @@ -148,13 +161,15 @@ def brace(x1, x2, y, h, up=True): """Curly brace from (x1,y) to (x2,y); cusp points up when up=True.""" s = -h if up else h xm = (x1 + x2) / 2 - return (f"M {x1} {y} " - f"C {x1} {y + s}, {x1 + h} {y + s}, {x1 + 2 * h} {y + s} " - f"L {xm - 2 * h} {y + s} " - f"C {xm - h} {y + s}, {xm} {y + s}, {xm} {y + 2 * s} " - f"C {xm} {y + s}, {xm + h} {y + s}, {xm + 2 * h} {y + s} " - f"L {x2 - 2 * h} {y + s} " - f"C {x2 - h} {y + s}, {x2} {y + s}, {x2} {y}") + return ( + f"M {x1} {y} " + f"C {x1} {y + s}, {x1 + h} {y + s}, {x1 + 2 * h} {y + s} " + f"L {xm - 2 * h} {y + s} " + f"C {xm - h} {y + s}, {xm} {y + s}, {xm} {y + 2 * s} " + f"C {xm} {y + s}, {xm + h} {y + s}, {xm + 2 * h} {y + s} " + f"L {x2 - 2 * h} {y + s} " + f"C {x2 - h} {y + s}, {x2} {y + s}, {x2} {y}" + ) def render(theme): @@ -172,12 +187,12 @@ def render(theme): glyph = GLYPHS[i].replace("{I}", t["ink"]) parts.append( f'' - f'' f'{glyph}' - f'' + f"" ) l1, l2 = (s.replace("&", "&") for s in LABELS[i]) parts.append( diff --git a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py index 928eabda..1d48ce2f 100644 --- a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py +++ b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py @@ -349,7 +349,6 @@ def _encode_block(doc): return encoded - # ── text fit (an estimate, because we have no font metrics) ────────────────── # # The app measures text for real and reports `text-overflow` from @@ -364,11 +363,17 @@ def _encode_block(doc): # a warning needs to clear the box by a margin before it prints. The app's own # validate() stays authoritative. _AVG_ADVANCE = 0.55 # mean glyph advance as a fraction of font size, sans-serif -_FIT_SLACK = 1.05 # only complain when the estimate clears the box by 5% +_FIT_SLACK = 1.05 # only complain when the estimate clears the box by 5% _ENTITIES = ( - ("—", "-"), ("–", "-"), (" ", " "), ("&", "&"), - ("<", "<"), (">", ">"), (""", '"'), ("'", "'"), + ("—", "-"), + ("–", "-"), + (" ", " "), + ("&", "&"), + ("<", "<"), + (">", ">"), + (""", '"'), + ("'", "'"), ) @@ -430,7 +435,8 @@ def _require(doc, key, kind, where): raise DeckError("%s: missing required field %r" % (where, key)) if not isinstance(doc[key], kind): raise DeckError( - "%s: field %r has the wrong type (%s)" % (where, key, type(doc[key]).__name__) + "%s: field %r has the wrong type (%s)" + % (where, key, type(doc[key]).__name__) ) return doc[key] @@ -660,7 +666,10 @@ def cmd_new(args): print("created %s — one title slide, ready to author" % path) print("offline-only deck: no update check, no live collaboration, no network") print("next: bento_doc.py get %s -o doc.json" % path) - print("download link (use this EXACT text, do not rebuild it): %s" % download_link(path)) + print( + "download link (use this EXACT text, do not rebuild it): %s" + % download_link(path) + ) return 0 @@ -768,12 +777,16 @@ def cmd_set(args): "- anyone holding an earlier copy can still join that room. The " "remedy for that is Share -> Rotate keys in the app.\n" % ( - " (including credential fields: %s)" % collab_field_label(dropped_fields) + " (including credential fields: %s)" + % collab_field_label(dropped_fields) if dropped_fields else "" ) ) - print("download link (use this EXACT text, do not rebuild it): %s" % download_link(args.deck)) + print( + "download link (use this EXACT text, do not rebuild it): %s" + % download_link(args.deck) + ) return 0 @@ -786,7 +799,9 @@ def cmd_validate(args): try: doc = json.loads(raw.decode("utf-8")) except ValueError as exc: - raise DeckError("%s is neither a deck nor valid JSON: %s" % (args.path, exc)) from exc + raise DeckError( + "%s is neither a deck nor valid JSON: %s" % (args.path, exc) + ) from exc if not isinstance(doc, dict): raise DeckError("%s must contain a JSON object" % args.path) kind = "document" @@ -839,7 +854,8 @@ def cmd_validate(args): "it joins that session with no click. Re-write it with `set` to " "remove the block and make the deck offline-only." % ( - " including credential fields (%s)" % collab_field_label(credential_fields) + " including credential fields (%s)" + % collab_field_label(credential_fields) if credential_fields else "" ) @@ -936,9 +952,10 @@ def cmd_pdf(args): out, pages, len(data) / 1024.0, - "" if not skipped + "" + if not skipped else " (%d hidden/state slide(s) left out, as in the app's own " - "export)" % skipped, + "export)" % skipped, ) ) for warning in warnings: @@ -961,7 +978,9 @@ def main(argv=None): p_new = sub.add_parser("new", help="start a deck from the bundled Bento app") p_new.add_argument("deck", help="path to create, e.g. decks/Q4_Review.bento.html") - p_new.add_argument("--title", help="deck title (default: derived from the filename)") + p_new.add_argument( + "--title", help="deck title (default: derived from the filename)" + ) p_new.set_defaults(func=cmd_new) p_get = sub.add_parser("get", help="extract a deck's document JSON") @@ -974,13 +993,17 @@ def main(argv=None): p_set.add_argument("doc", help="the document JSON to splice in") p_set.set_defaults(func=cmd_set) - p_val = sub.add_parser("validate", help="check a deck or document for format errors") + p_val = sub.add_parser( + "validate", help="check a deck or document for format errors" + ) p_val.add_argument("path") p_val.set_defaults(func=cmd_validate) p_pdf = sub.add_parser("pdf", help="render a deck's slides to a PDF you can attach") p_pdf.add_argument("deck") - p_pdf.add_argument("-o", "--output", help="PDF path (default: the deck's name + .pdf)") + p_pdf.add_argument( + "-o", "--output", help="PDF path (default: the deck's name + .pdf)" + ) p_pdf.set_defaults(func=cmd_pdf) args = parser.parse_args(argv) diff --git a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py index 0f1dde12..b332b6bc 100644 --- a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py +++ b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py @@ -74,19 +74,30 @@ class PdfError(Exception): # ── colors ─────────────────────────────────────────────────────────────────── _NAMED = { - "transparent": (0, 0, 0, 0.0), "none": (0, 0, 0, 0.0), - "black": (0, 0, 0, 1.0), "white": (1, 1, 1, 1.0), - "red": (1, 0, 0, 1.0), "green": (0, 0.502, 0, 1.0), - "blue": (0, 0, 1, 1.0), "gray": (0.502, 0.502, 0.502, 1.0), - "grey": (0.502, 0.502, 0.502, 1.0), "silver": (0.753, 0.753, 0.753, 1.0), - "navy": (0, 0, 0.502, 1.0), "teal": (0, 0.502, 0.502, 1.0), - "orange": (1, 0.647, 0, 1.0), "yellow": (1, 1, 0, 1.0), - "purple": (0.502, 0, 0.502, 1.0), "inherit": None, "currentcolor": None, + "transparent": (0, 0, 0, 0.0), + "none": (0, 0, 0, 0.0), + "black": (0, 0, 0, 1.0), + "white": (1, 1, 1, 1.0), + "red": (1, 0, 0, 1.0), + "green": (0, 0.502, 0, 1.0), + "blue": (0, 0, 1, 1.0), + "gray": (0.502, 0.502, 0.502, 1.0), + "grey": (0.502, 0.502, 0.502, 1.0), + "silver": (0.753, 0.753, 0.753, 1.0), + "navy": (0, 0, 0.502, 1.0), + "teal": (0, 0.502, 0.502, 1.0), + "orange": (1, 0.647, 0, 1.0), + "yellow": (1, 1, 0, 1.0), + "purple": (0.502, 0, 0.502, 1.0), + "inherit": None, + "currentcolor": None, } _RGB_FN = re.compile( r"^rgba?\(\s*([0-9.]+%?)[\s,]+([0-9.]+%?)[\s,]+([0-9.]+%?)" - r"(?:[\s,/]+([0-9.]+%?))?\s*\)$", re.I) + r"(?:[\s,/]+([0-9.]+%?))?\s*\)$", + re.I, +) def _chan(tok): @@ -119,7 +130,7 @@ def parse_color(value, default=(0, 0, 0, 1.0)): if len(h) in (3, 4): vals = [int(c * 2, 16) / 255.0 for c in h] elif len(h) in (6, 8): - vals = [int(h[i:i + 2], 16) / 255.0 for i in range(0, len(h), 2)] + vals = [int(h[i : i + 2], 16) / 255.0 for i in range(0, len(h), 2)] else: return default except ValueError: @@ -150,130 +161,123 @@ def is_visible(rgba): # which has to guess at a font it will never see. Zeros are WinAnsi's unused # slots and never reached: text is transliterated into this encoding first. _WINANSI_WIDTHS_SRC = { - "Helvetica": - "278 278 355 556 556 889 667 191 333 333 389 584 278 333 278 278 556 " - "556 556 556 556 556 556 556 556 556 278 278 584 584 584 556 1015 667 " - "667 722 722 667 611 778 722 278 500 667 556 833 722 778 667 778 722 " - "667 611 722 667 944 667 667 611 278 278 278 469 556 333 556 556 500 " - "556 556 278 556 556 222 222 500 222 833 556 556 556 556 333 500 278 " - "556 500 722 500 500 500 334 260 334 584 350 556 350 222 556 333 1000 " - "556 556 333 1000 667 333 1000 350 611 350 350 222 222 333 333 350 " - "556 1000 333 1000 500 333 944 350 500 667 278 333 556 556 556 556 260 " - "556 333 737 370 556 584 333 737 552 400 549 333 333 333 576 537 278 " - "333 333 365 556 834 834 834 611 667 667 667 667 667 667 1000 722 667 " - "667 667 667 278 278 278 278 722 722 778 778 778 778 778 584 778 722 " - "722 722 722 667 667 611 556 556 556 556 556 556 889 500 556 556 556 " - "556 278 278 278 278 556 556 556 556 556 556 556 549 611 556 556 556 " - "556 500 556 500", - "Helvetica-Bold": - "278 333 474 556 556 889 722 238 333 333 389 584 278 333 278 278 556 " - "556 556 556 556 556 556 556 556 556 333 333 584 584 584 611 975 722 " - "722 722 722 667 611 778 722 278 556 722 611 833 722 778 667 778 722 " - "667 611 722 667 944 667 667 611 333 278 333 584 556 333 556 611 556 " - "611 556 333 611 611 278 278 556 278 889 611 611 611 611 389 556 333 " - "611 556 778 556 556 500 389 280 389 584 350 556 350 278 556 500 1000 " - "556 556 333 1000 667 333 1000 350 611 350 350 278 278 500 500 350 556 " - "1000 333 1000 556 333 944 350 500 667 278 333 556 556 556 556 280 556 " - "333 737 370 556 584 333 737 552 400 549 333 333 333 576 556 278 333 " - "333 365 556 834 834 834 611 722 722 722 722 722 722 1000 722 667 667 " - "667 667 278 278 278 278 722 722 778 778 778 778 778 584 778 722 722 " - "722 722 667 667 611 556 556 556 556 556 556 889 556 556 556 556 556 " - "278 278 278 278 611 611 611 611 611 611 611 549 611 611 611 611 611 " - "556 611 556", - "Helvetica-Oblique": - "278 278 355 556 556 889 667 191 333 333 389 584 278 333 278 278 556 " - "556 556 556 556 556 556 556 556 556 278 278 584 584 584 556 1015 667 " - "667 722 722 667 611 778 722 278 500 667 556 833 722 778 667 778 722 " - "667 611 722 667 944 667 667 611 278 278 278 469 556 333 556 556 500 " - "556 556 278 556 556 222 222 500 222 833 556 556 556 556 333 500 278 " - "556 500 722 500 500 500 334 260 334 584 350 556 350 222 556 333 1000 " - "556 556 333 1000 667 333 1000 350 611 350 350 222 222 333 333 350 " - "556 1000 333 1000 500 333 944 350 500 667 278 333 556 556 556 556 260 " - "556 333 737 370 556 584 333 737 552 400 549 333 333 333 576 537 278 " - "333 333 365 556 834 834 834 611 667 667 667 667 667 667 1000 722 667 " - "667 667 667 278 278 278 278 722 722 778 778 778 778 778 584 778 722 " - "722 722 722 667 667 611 556 556 556 556 556 556 889 500 556 556 556 " - "556 278 278 278 278 556 556 556 556 556 556 556 549 611 556 556 556 " - "556 500 556 500", - "Helvetica-BoldOblique": - "278 333 474 556 556 889 722 238 333 333 389 584 278 333 278 278 556 " - "556 556 556 556 556 556 556 556 556 333 333 584 584 584 611 975 722 " - "722 722 722 667 611 778 722 278 556 722 611 833 722 778 667 778 722 " - "667 611 722 667 944 667 667 611 333 278 333 584 556 333 556 611 556 " - "611 556 333 611 611 278 278 556 278 889 611 611 611 611 389 556 333 " - "611 556 778 556 556 500 389 280 389 584 350 556 350 278 556 500 1000 " - "556 556 333 1000 667 333 1000 350 611 350 350 278 278 500 500 350 556 " - "1000 333 1000 556 333 944 350 500 667 278 333 556 556 556 556 280 556 " - "333 737 370 556 584 333 737 552 400 549 333 333 333 576 556 278 333 " - "333 365 556 834 834 834 611 722 722 722 722 722 722 1000 722 667 667 " - "667 667 278 278 278 278 722 722 778 778 778 778 778 584 778 722 722 " - "722 722 667 667 611 556 556 556 556 556 556 889 556 556 556 556 556 " - "278 278 278 278 611 611 611 611 611 611 611 549 611 611 611 611 611 " - "556 611 556", - "Times-Roman": - "250 333 408 500 500 833 778 180 333 333 500 564 250 333 250 278 500 " - "500 500 500 500 500 500 500 500 500 278 278 564 564 564 444 921 722 " - "667 667 722 611 556 722 722 333 389 722 611 889 722 722 556 722 667 " - "556 611 722 722 944 722 722 611 333 278 333 469 500 333 444 500 444 " - "500 444 333 500 500 278 278 500 278 778 500 500 500 500 333 389 278 " - "500 500 722 500 500 444 480 200 480 541 350 500 350 333 500 444 1000 " - "500 500 333 1000 556 333 889 350 611 350 350 333 333 444 444 350 500 " - "1000 333 980 389 333 722 350 444 722 250 333 500 500 500 500 200 500 " - "333 760 276 500 564 333 760 500 400 549 300 300 333 576 453 250 333 " - "300 310 500 750 750 750 444 722 722 722 722 722 722 889 667 611 611 " - "611 611 333 333 333 333 722 722 722 722 722 722 722 564 722 722 722 " - "722 722 722 556 500 444 444 444 444 444 444 667 444 444 444 444 444 " - "278 278 278 278 500 500 500 500 500 500 500 549 500 500 500 500 500 " - "500 500 500", - "Times-Bold": - "250 333 555 500 500 1000 833 278 333 333 500 570 250 333 250 278 500 " - "500 500 500 500 500 500 500 500 500 333 333 570 570 570 500 930 722 " - "667 722 722 667 611 778 778 389 500 778 667 944 722 778 611 778 722 " - "556 667 722 722 1000 722 722 667 333 278 333 581 500 333 500 556 444 " - "556 444 333 500 556 278 333 556 278 833 556 500 556 556 444 389 333 " - "556 500 722 500 500 444 394 220 394 520 350 500 350 333 500 500 1000 " - "500 500 333 1000 556 333 1000 350 667 350 350 333 333 500 500 350 500 " - "1000 333 1000 389 333 722 350 444 722 250 333 500 500 500 500 220 500 " - "333 747 300 500 570 333 747 500 400 549 300 300 333 576 500 250 333 " - "300 330 500 750 750 750 500 722 722 722 722 722 722 1000 722 667 667 " - "667 667 389 389 389 389 722 722 778 778 778 778 778 570 778 722 722 " - "722 722 722 611 556 500 500 500 500 500 500 722 444 444 444 444 444 " - "278 278 278 278 500 556 500 500 500 500 500 549 500 556 556 556 556 " - "500 556 500", - "Times-Italic": - "250 333 420 500 500 833 778 214 333 333 500 675 250 333 250 278 500 " - "500 500 500 500 500 500 500 500 500 333 333 675 675 675 500 920 611 " - "611 667 722 611 611 722 722 333 444 667 556 833 667 722 611 722 611 " - "500 556 722 611 833 611 556 556 389 278 389 422 500 333 500 500 444 " - "500 444 278 500 500 278 278 444 278 722 500 500 500 500 389 389 278 " - "500 444 667 444 444 389 400 275 400 541 350 500 350 333 500 556 889 " - "500 500 333 1000 500 333 944 350 556 350 350 333 333 556 556 350 500 " - "889 333 980 389 333 667 350 389 556 250 389 500 500 500 500 275 500 " - "333 760 276 500 675 333 760 500 400 549 300 300 333 576 523 250 333 " - "300 310 500 750 750 750 500 611 611 611 611 611 611 889 667 611 611 " - "611 611 333 333 333 333 722 667 722 722 722 722 722 675 722 722 722 " - "722 722 556 611 500 500 500 500 500 500 500 667 444 444 444 444 444 " - "278 278 278 278 500 500 500 500 500 500 500 549 500 500 500 500 500 " - "444 500 444", - "Times-BoldItalic": - "250 389 555 500 500 833 778 278 333 333 500 570 250 333 250 278 500 " - "500 500 500 500 500 500 500 500 500 333 333 570 570 570 500 832 667 " - "667 667 722 667 667 722 778 389 500 667 611 889 722 722 611 722 667 " - "556 611 722 667 889 667 611 611 333 278 333 570 500 333 500 500 444 " - "500 444 333 444 500 278 278 444 278 722 500 500 500 500 389 389 278 " - "500 444 667 500 444 389 348 220 348 570 350 500 350 333 500 500 1000 " - "500 500 333 1000 556 333 944 350 611 350 350 333 333 500 500 350 500 " - "1000 333 1000 389 333 722 350 389 611 250 389 500 500 500 500 220 500 " - "333 747 266 500 606 333 747 500 400 549 300 300 333 576 500 250 333 " - "300 300 500 750 750 750 500 667 667 667 667 667 667 944 667 667 667 " - "667 667 389 389 389 389 722 722 722 722 722 722 722 570 722 722 722 " - "722 722 611 611 500 500 500 500 500 500 500 722 444 444 444 444 444 " - "278 278 278 278 500 500 500 500 500 500 500 549 500 500 500 500 500 " - "444 500 444", + "Helvetica": "278 278 355 556 556 889 667 191 333 333 389 584 278 333 278 278 556 " + "556 556 556 556 556 556 556 556 556 278 278 584 584 584 556 1015 667 " + "667 722 722 667 611 778 722 278 500 667 556 833 722 778 667 778 722 " + "667 611 722 667 944 667 667 611 278 278 278 469 556 333 556 556 500 " + "556 556 278 556 556 222 222 500 222 833 556 556 556 556 333 500 278 " + "556 500 722 500 500 500 334 260 334 584 350 556 350 222 556 333 1000 " + "556 556 333 1000 667 333 1000 350 611 350 350 222 222 333 333 350 " + "556 1000 333 1000 500 333 944 350 500 667 278 333 556 556 556 556 260 " + "556 333 737 370 556 584 333 737 552 400 549 333 333 333 576 537 278 " + "333 333 365 556 834 834 834 611 667 667 667 667 667 667 1000 722 667 " + "667 667 667 278 278 278 278 722 722 778 778 778 778 778 584 778 722 " + "722 722 722 667 667 611 556 556 556 556 556 556 889 500 556 556 556 " + "556 278 278 278 278 556 556 556 556 556 556 556 549 611 556 556 556 " + "556 500 556 500", + "Helvetica-Bold": "278 333 474 556 556 889 722 238 333 333 389 584 278 333 278 278 556 " + "556 556 556 556 556 556 556 556 556 333 333 584 584 584 611 975 722 " + "722 722 722 667 611 778 722 278 556 722 611 833 722 778 667 778 722 " + "667 611 722 667 944 667 667 611 333 278 333 584 556 333 556 611 556 " + "611 556 333 611 611 278 278 556 278 889 611 611 611 611 389 556 333 " + "611 556 778 556 556 500 389 280 389 584 350 556 350 278 556 500 1000 " + "556 556 333 1000 667 333 1000 350 611 350 350 278 278 500 500 350 556 " + "1000 333 1000 556 333 944 350 500 667 278 333 556 556 556 556 280 556 " + "333 737 370 556 584 333 737 552 400 549 333 333 333 576 556 278 333 " + "333 365 556 834 834 834 611 722 722 722 722 722 722 1000 722 667 667 " + "667 667 278 278 278 278 722 722 778 778 778 778 778 584 778 722 722 " + "722 722 667 667 611 556 556 556 556 556 556 889 556 556 556 556 556 " + "278 278 278 278 611 611 611 611 611 611 611 549 611 611 611 611 611 " + "556 611 556", + "Helvetica-Oblique": "278 278 355 556 556 889 667 191 333 333 389 584 278 333 278 278 556 " + "556 556 556 556 556 556 556 556 556 278 278 584 584 584 556 1015 667 " + "667 722 722 667 611 778 722 278 500 667 556 833 722 778 667 778 722 " + "667 611 722 667 944 667 667 611 278 278 278 469 556 333 556 556 500 " + "556 556 278 556 556 222 222 500 222 833 556 556 556 556 333 500 278 " + "556 500 722 500 500 500 334 260 334 584 350 556 350 222 556 333 1000 " + "556 556 333 1000 667 333 1000 350 611 350 350 222 222 333 333 350 " + "556 1000 333 1000 500 333 944 350 500 667 278 333 556 556 556 556 260 " + "556 333 737 370 556 584 333 737 552 400 549 333 333 333 576 537 278 " + "333 333 365 556 834 834 834 611 667 667 667 667 667 667 1000 722 667 " + "667 667 667 278 278 278 278 722 722 778 778 778 778 778 584 778 722 " + "722 722 722 667 667 611 556 556 556 556 556 556 889 500 556 556 556 " + "556 278 278 278 278 556 556 556 556 556 556 556 549 611 556 556 556 " + "556 500 556 500", + "Helvetica-BoldOblique": "278 333 474 556 556 889 722 238 333 333 389 584 278 333 278 278 556 " + "556 556 556 556 556 556 556 556 556 333 333 584 584 584 611 975 722 " + "722 722 722 667 611 778 722 278 556 722 611 833 722 778 667 778 722 " + "667 611 722 667 944 667 667 611 333 278 333 584 556 333 556 611 556 " + "611 556 333 611 611 278 278 556 278 889 611 611 611 611 389 556 333 " + "611 556 778 556 556 500 389 280 389 584 350 556 350 278 556 500 1000 " + "556 556 333 1000 667 333 1000 350 611 350 350 278 278 500 500 350 556 " + "1000 333 1000 556 333 944 350 500 667 278 333 556 556 556 556 280 556 " + "333 737 370 556 584 333 737 552 400 549 333 333 333 576 556 278 333 " + "333 365 556 834 834 834 611 722 722 722 722 722 722 1000 722 667 667 " + "667 667 278 278 278 278 722 722 778 778 778 778 778 584 778 722 722 " + "722 722 667 667 611 556 556 556 556 556 556 889 556 556 556 556 556 " + "278 278 278 278 611 611 611 611 611 611 611 549 611 611 611 611 611 " + "556 611 556", + "Times-Roman": "250 333 408 500 500 833 778 180 333 333 500 564 250 333 250 278 500 " + "500 500 500 500 500 500 500 500 500 278 278 564 564 564 444 921 722 " + "667 667 722 611 556 722 722 333 389 722 611 889 722 722 556 722 667 " + "556 611 722 722 944 722 722 611 333 278 333 469 500 333 444 500 444 " + "500 444 333 500 500 278 278 500 278 778 500 500 500 500 333 389 278 " + "500 500 722 500 500 444 480 200 480 541 350 500 350 333 500 444 1000 " + "500 500 333 1000 556 333 889 350 611 350 350 333 333 444 444 350 500 " + "1000 333 980 389 333 722 350 444 722 250 333 500 500 500 500 200 500 " + "333 760 276 500 564 333 760 500 400 549 300 300 333 576 453 250 333 " + "300 310 500 750 750 750 444 722 722 722 722 722 722 889 667 611 611 " + "611 611 333 333 333 333 722 722 722 722 722 722 722 564 722 722 722 " + "722 722 722 556 500 444 444 444 444 444 444 667 444 444 444 444 444 " + "278 278 278 278 500 500 500 500 500 500 500 549 500 500 500 500 500 " + "500 500 500", + "Times-Bold": "250 333 555 500 500 1000 833 278 333 333 500 570 250 333 250 278 500 " + "500 500 500 500 500 500 500 500 500 333 333 570 570 570 500 930 722 " + "667 722 722 667 611 778 778 389 500 778 667 944 722 778 611 778 722 " + "556 667 722 722 1000 722 722 667 333 278 333 581 500 333 500 556 444 " + "556 444 333 500 556 278 333 556 278 833 556 500 556 556 444 389 333 " + "556 500 722 500 500 444 394 220 394 520 350 500 350 333 500 500 1000 " + "500 500 333 1000 556 333 1000 350 667 350 350 333 333 500 500 350 500 " + "1000 333 1000 389 333 722 350 444 722 250 333 500 500 500 500 220 500 " + "333 747 300 500 570 333 747 500 400 549 300 300 333 576 500 250 333 " + "300 330 500 750 750 750 500 722 722 722 722 722 722 1000 722 667 667 " + "667 667 389 389 389 389 722 722 778 778 778 778 778 570 778 722 722 " + "722 722 722 611 556 500 500 500 500 500 500 722 444 444 444 444 444 " + "278 278 278 278 500 556 500 500 500 500 500 549 500 556 556 556 556 " + "500 556 500", + "Times-Italic": "250 333 420 500 500 833 778 214 333 333 500 675 250 333 250 278 500 " + "500 500 500 500 500 500 500 500 500 333 333 675 675 675 500 920 611 " + "611 667 722 611 611 722 722 333 444 667 556 833 667 722 611 722 611 " + "500 556 722 611 833 611 556 556 389 278 389 422 500 333 500 500 444 " + "500 444 278 500 500 278 278 444 278 722 500 500 500 500 389 389 278 " + "500 444 667 444 444 389 400 275 400 541 350 500 350 333 500 556 889 " + "500 500 333 1000 500 333 944 350 556 350 350 333 333 556 556 350 500 " + "889 333 980 389 333 667 350 389 556 250 389 500 500 500 500 275 500 " + "333 760 276 500 675 333 760 500 400 549 300 300 333 576 523 250 333 " + "300 310 500 750 750 750 500 611 611 611 611 611 611 889 667 611 611 " + "611 611 333 333 333 333 722 667 722 722 722 722 722 675 722 722 722 " + "722 722 556 611 500 500 500 500 500 500 500 667 444 444 444 444 444 " + "278 278 278 278 500 500 500 500 500 500 500 549 500 500 500 500 500 " + "444 500 444", + "Times-BoldItalic": "250 389 555 500 500 833 778 278 333 333 500 570 250 333 250 278 500 " + "500 500 500 500 500 500 500 500 500 333 333 570 570 570 500 832 667 " + "667 667 722 667 667 722 778 389 500 667 611 889 722 722 611 722 667 " + "556 611 722 667 889 667 611 611 333 278 333 570 500 333 500 500 444 " + "500 444 333 444 500 278 278 444 278 722 500 500 500 500 389 389 278 " + "500 444 667 500 444 389 348 220 348 570 350 500 350 333 500 500 1000 " + "500 500 333 1000 556 333 944 350 611 350 350 333 333 500 500 350 500 " + "1000 333 1000 389 333 722 350 389 611 250 389 500 500 500 500 220 500 " + "333 747 266 500 606 333 747 500 400 549 300 300 333 576 500 250 333 " + "300 300 500 750 750 750 500 667 667 667 667 667 667 944 667 667 667 " + "667 667 389 389 389 389 722 722 722 722 722 722 722 570 722 722 722 " + "722 722 611 611 500 500 500 500 500 500 500 722 444 444 444 444 444 " + "278 278 278 278 500 500 500 500 500 500 500 549 500 500 500 500 500 " + "444 500 444", } -WIDTHS = {name: [int(n) for n in src.split()] - for name, src in _WINANSI_WIDTHS_SRC.items()} +WIDTHS = { + name: [int(n) for n in src.split()] for name, src in _WINANSI_WIDTHS_SRC.items() +} WIDTHS["Courier"] = [600] * 224 WIDTHS["Courier-Bold"] = [600] * 224 WIDTHS["Courier-Oblique"] = [600] * 224 @@ -288,12 +292,40 @@ def is_visible(rgba): "Courier": (0.629, 0.157), } -_SERIF_HINTS = ("serif", "georgia", "times", "garamond", "fraunces", "playfair", - "merriweather", "cambria", "book", "charter", "spectral", "lora", - "source serif", "pt serif", "noto serif", "ibm plex serif", - "instrument serif", "newsreader", "literata", "bitter") -_MONO_HINTS = ("mono", "courier", "consolas", "menlo", "sf mono", "jetbrains", - "fira code", "source code", "ibm plex mono", "roboto mono") +_SERIF_HINTS = ( + "serif", + "georgia", + "times", + "garamond", + "fraunces", + "playfair", + "merriweather", + "cambria", + "book", + "charter", + "spectral", + "lora", + "source serif", + "pt serif", + "noto serif", + "ibm plex serif", + "instrument serif", + "newsreader", + "literata", + "bitter", +) +_MONO_HINTS = ( + "mono", + "courier", + "consolas", + "menlo", + "sf mono", + "jetbrains", + "fira code", + "source code", + "ibm plex mono", + "roboto mono", +) def family_of(stack): @@ -373,14 +405,34 @@ def is_bold(weight): # sign, plus-minus) is left alone — transliterating those would degrade text the # PDF can render perfectly. _TRANSLIT = { - "→": "->", "←": "<-", "↔": "<->", "⇒": "=>", "⇐": "<=", - "↑": "^", "↓": "v", "−": "-", "≤": "<=", "≥": ">=", - "≈": "~", "≠": "!=", "′": "'", "″": '"', - "✓": "*", "✔": "*", "✗": "x", "✘": "x", - "▶": ">", "◀": "<", "▪": "\u2022", "●": "\u2022", + "→": "->", + "←": "<-", + "↔": "<->", + "⇒": "=>", + "⇐": "<=", + "↑": "^", + "↓": "v", + "−": "-", + "≤": "<=", + "≥": ">=", + "≈": "~", + "≠": "!=", + "′": "'", + "″": '"', + "✓": "*", + "✔": "*", + "✗": "x", + "✘": "x", + "▶": ">", + "◀": "<", + "▪": "\u2022", + "●": "\u2022", # Exotic spaces a model can paste in: render as a normal space rather than # as a missing glyph in the middle of a headline. - "\u00a0": " ", "\u2007": " ", "\u2009": " ", "\u202f": " ", + "\u00a0": " ", + "\u2007": " ", + "\u2009": " ", + "\u202f": " ", } @@ -416,10 +468,11 @@ def warning(self): if not self.dropped: return None shown = " ".join("%r" % c for c in self.samples) - return ("%d character(s) have no glyph in the PDF core fonts and were " - "written as '?' (%s). Text outside Western European scripts " - "needs the deck's own Export PDF (print) button." - % (self.dropped, shown)) + return ( + "%d character(s) have no glyph in the PDF core fonts and were " + "written as '?' (%s). Text outside Western European scripts " + "needs the deck's own Export PDF (print) button." % (self.dropped, shown) + ) def text_width(encoder, text, face, size, letter_spacing=0.0): @@ -480,12 +533,19 @@ def push(chunk): nonlocal pending_break if not chunk: return - runs.append(Run(unescape(chunk), bold_depth > 0, italic_depth > 0, - mono_depth > 0, pending_break)) + runs.append( + Run( + unescape(chunk), + bold_depth > 0, + italic_depth > 0, + mono_depth > 0, + pending_break, + ) + ) pending_break = False for match in _TAG.finditer(text): - push(text[pos:match.start()]) + push(text[pos : match.start()]) pos = match.end() closing = match.group(1) == "/" name = match.group(2).lower() @@ -505,8 +565,7 @@ def push(chunk): mono_depth = max(0, mono_depth + step) push(text[pos:]) if pending_break: - runs.append(Run("", bold_depth > 0, italic_depth > 0, mono_depth > 0, - True)) + runs.append(Run("", bold_depth > 0, italic_depth > 0, mono_depth > 0, True)) return runs @@ -539,8 +598,9 @@ def add(self, piece): _SPLIT = re.compile(r"(\s+)") -def layout_text(encoder, runs, box_width, family, size, weight, - letter_spacing=0.0, wrap=True): +def layout_text( + encoder, runs, box_width, family, size, weight, letter_spacing=0.0, wrap=True +): """Greedy word wrap into `box_width`, honouring hard breaks and runs. Mirrors the browser closely enough to matter: `overflow-wrap: break-word` is @@ -569,8 +629,9 @@ def layout_text(encoder, runs, box_width, family, size, weight, continue if not wrap or blank or line.width + width <= box_width + 0.01: if wrap and blank and width > box_width + 0.01: - for part in _break_word(encoder, token, face, run_size, - box_width, letter_spacing): + for part in _break_word( + encoder, token, face, run_size, box_width, letter_spacing + ): if lines[-1].pieces: lines.append(Line()) lines[-1].add(part) @@ -581,8 +642,9 @@ def layout_text(encoder, runs, box_width, family, size, weight, line.width -= line.pieces.pop().width lines.append(Line()) if width > box_width + 0.01: - for part in _break_word(encoder, token, face, run_size, - box_width, letter_spacing): + for part in _break_word( + encoder, token, face, run_size, box_width, letter_spacing + ): if lines[-1].pieces: lines.append(Line()) lines[-1].add(part) @@ -640,8 +702,9 @@ def num(value): """Compact fixed-point number: PDF has no exponent notation.""" if value is None or not isinstance(value, (int, float)): return "0" - if isinstance(value, float) and (value != value or value in - (float("inf"), float("-inf"))): + if isinstance(value, float) and ( + value != value or value in (float("inf"), float("-inf")) + ): return "0" text = "%.4f" % value text = text.rstrip("0").rstrip(".") @@ -658,8 +721,9 @@ def reserve(self): return len(self._objects) - 1 def put(self, number, body): - self._objects[number] = body if isinstance(body, bytes) \ - else body.encode("latin-1") + self._objects[number] = ( + body if isinstance(body, bytes) else body.encode("latin-1") + ) def add(self, body): number = self.reserve() @@ -672,8 +736,7 @@ def add_stream(self, entries, data, compress=None): if compress: data = zlib.compress(data, 9) entries = entries + ["/Filter /FlateDecode"] - head = "<< %s /Length %d >>\nstream\n" % ( - " ".join(entries), len(data)) + head = "<< %s /Length %d >>\nstream\n" % (" ".join(entries), len(data)) return self.add(head.encode("latin-1") + data + b"\nendstream") def serialize(self, root, info): @@ -689,9 +752,10 @@ def serialize(self, root, info): out += b"0000000000 65535 f \n" for number in range(1, len(self._objects)): out += ("%010d 00000 n \n" % offsets[number]).encode("latin-1") - out += ("trailer\n<< /Size %d /Root %d 0 R /Info %d 0 R >>\n" - "startxref\n%d\n%%%%EOF\n" - % (len(self._objects), root, info, start)).encode("latin-1") + out += ( + "trailer\n<< /Size %d /Root %d 0 R /Info %d 0 R >>\n" + "startxref\n%d\n%%%%EOF\n" % (len(self._objects), root, info, start) + ).encode("latin-1") return bytes(out) @@ -709,12 +773,13 @@ def decode_data_uri(src): return None mime = (match.group(1) or "").lower() params = (match.group(2) or "").lower() - payload = src[match.end():] + payload = src[match.end() :] try: if "base64" in params: data = base64.b64decode(payload + "=" * (-len(payload) % 4)) else: from urllib.parse import unquote_to_bytes + data = unquote_to_bytes(payload) except (binascii.Error, ValueError): return None @@ -724,11 +789,20 @@ def decode_data_uri(src): class Image: """A decoded raster ready to become an XObject.""" - __slots__ = ("width", "height", "data", "filter", "colorspace", "bpc", - "smask", "palette") - - def __init__(self, width, height, data, filt, colorspace, bpc=8, - smask=None, palette=None): + __slots__ = ( + "width", + "height", + "data", + "filter", + "colorspace", + "bpc", + "smask", + "palette", + ) + + def __init__( + self, width, height, data, filt, colorspace, bpc=8, smask=None, palette=None + ): self.width = width self.height = height self.data = data @@ -750,9 +824,9 @@ def _jpeg_size(data): if marker in (0xD8, 0xD9) or 0xD0 <= marker <= 0xD7 or marker == 0x01: i += 2 continue - length = struct.unpack(">H", data[i + 2:i + 4])[0] + length = struct.unpack(">H", data[i + 2 : i + 4])[0] if 0xC0 <= marker <= 0xCF and marker not in (0xC4, 0xC8, 0xCC): - height, width = struct.unpack(">HH", data[i + 5:i + 9]) + height, width = struct.unpack(">HH", data[i + 5 : i + 9]) components = data[i + 9] return width, height, components i += 2 + length @@ -776,12 +850,13 @@ def _png_decode(data): palette = None trns = None while pos + 8 <= len(data): - length, kind = struct.unpack(">I4s", data[pos:pos + 8]) - body = data[pos + 8:pos + 8 + length] + length, kind = struct.unpack(">I4s", data[pos : pos + 8]) + body = data[pos + 8 : pos + 8 + length] pos += 12 + length if kind == b"IHDR": - width, height, depth, color, _comp, _filt, interlace = \ - struct.unpack(">IIBBBBB", body[:13]) + width, height, depth, color, _comp, _filt, interlace = struct.unpack( + ">IIBBBBB", body[:13] + ) header = (width, height, depth, color, interlace) elif kind == b"PLTE": palette = bytes(body) @@ -820,7 +895,7 @@ def _png_decode(data): for row in range(height): filt = raw[at] at += 1 - line = bytearray(raw[at:at + stride]) + line = bytearray(raw[at : at + stride]) at += stride if filt == 1: for i in range(unit, stride): @@ -838,14 +913,16 @@ def _png_decode(data): up = prev[i] upper_left = prev[i - unit] if i >= unit else 0 peak = left + up - upper_left - da, db, dc = (abs(peak - left), abs(peak - up), - abs(peak - upper_left)) - nearest = left if (da <= db and da <= dc) else \ - (up if db <= dc else upper_left) + da, db, dc = (abs(peak - left), abs(peak - up), abs(peak - upper_left)) + nearest = ( + left + if (da <= db and da <= dc) + else (up if db <= dc else upper_left) + ) line[i] = (line[i] + nearest) & 0xFF elif filt != 0: return "the PNG uses an unknown scanline filter (%d)" % filt - out[row * stride:(row + 1) * stride] = line + out[row * stride : (row + 1) * stride] = line prev = line if depth == 16: # keep the high byte; PDF viewers do the same visually @@ -863,7 +940,7 @@ def _png_decode(data): colour = bytearray() alpha = bytearray() for i in range(0, len(out), 4): - colour += out[i:i + 3] + colour += out[i : i + 3] alpha.append(out[i + 3]) space, ncomp = "/DeviceRGB", 3 elif color == 2: @@ -878,11 +955,20 @@ def _png_decode(data): smask = None if alpha is not None and min(alpha) < 255: - smask = Image(width, height, zlib.compress(bytes(alpha), 9), - "/FlateDecode", "/DeviceGray") + smask = Image( + width, height, zlib.compress(bytes(alpha), 9), "/FlateDecode", "/DeviceGray" + ) del ncomp - return Image(width, height, zlib.compress(bytes(colour), 9), - "/FlateDecode", space, 8, smask, palette) + return Image( + width, + height, + zlib.compress(bytes(colour), 9), + "/FlateDecode", + space, + 8, + smask, + palette, + ) def decode_image(src): @@ -902,8 +988,7 @@ def decode_image(src): if not size: return None, "the JPEG has no readable frame header" width, height, components = size - space = {1: "/DeviceGray", 3: "/DeviceRGB", 4: "/DeviceCMYK"}.get( - components) + space = {1: "/DeviceGray", 3: "/DeviceRGB", 4: "/DeviceCMYK"}.get(components) if space is None: return None, "the JPEG has %d components" % components return Image(width, height, data, "/DCTDecode", space), None @@ -942,7 +1027,8 @@ def font(self, face): if face not in self.fonts: number = self.writer.add( "<< /Type /Font /Subtype /Type1 /BaseFont /%s " - "/Encoding /WinAnsiEncoding >>" % face) + "/Encoding /WinAnsiEncoding >>" % face + ) self.fonts[face] = ("/F%d" % len(self.fonts), number) return self.fonts[face][0] @@ -950,39 +1036,46 @@ def alpha(self, fill_alpha, stroke_alpha): key = (round(fill_alpha, 3), round(stroke_alpha, 3)) if key not in self.gstates: number = self.writer.add( - "<< /Type /ExtGState /ca %s /CA %s >>" - % (num(key[0]), num(key[1]))) + "<< /Type /ExtGState /ca %s /CA %s >>" % (num(key[0]), num(key[1])) + ) self.gstates[key] = ("/GS%d" % len(self.gstates), number) return self.gstates[key][0] def image(self, image, cache_key=None): if cache_key is not None and cache_key in self._image_cache: return self._image_cache[cache_key] - entries = ["/Type /XObject", "/Subtype /Image", - "/Width %d" % image.width, "/Height %d" % image.height, - "/BitsPerComponent %d" % image.bpc, - "/Filter %s" % image.filter] + entries = [ + "/Type /XObject", + "/Subtype /Image", + "/Width %d" % image.width, + "/Height %d" % image.height, + "/BitsPerComponent %d" % image.bpc, + "/Filter %s" % image.filter, + ] if image.palette is not None: palette = self.writer.add_stream([], image.palette, compress=False) - entries.append("/ColorSpace [/Indexed /DeviceRGB %d %d 0 R]" - % (len(image.palette) // 3 - 1, palette)) + entries.append( + "/ColorSpace [/Indexed /DeviceRGB %d %d 0 R]" + % (len(image.palette) // 3 - 1, palette) + ) else: entries.append("/ColorSpace %s" % image.colorspace) if image.smask is not None: # The soft mask is referenced by the image, not by the page, so it # is written as a plain object and stays out of /XObject. mask = image.smask - head = ("<< /Type /XObject /Subtype /Image /Width %d /Height %d " - "/BitsPerComponent 8 /ColorSpace /DeviceGray /Filter %s " - "/Length %d >>\nstream\n" - % (mask.width, mask.height, mask.filter, len(mask.data))) + head = ( + "<< /Type /XObject /Subtype /Image /Width %d /Height %d " + "/BitsPerComponent 8 /ColorSpace /DeviceGray /Filter %s " + "/Length %d >>\nstream\n" + % (mask.width, mask.height, mask.filter, len(mask.data)) + ) number = self.writer.add( - head.encode("latin-1") + mask.data + b"\nendstream") + head.encode("latin-1") + mask.data + b"\nendstream" + ) entries.append("/SMask %d 0 R" % number) - head = "<< %s /Length %d >>\nstream\n" % ( - " ".join(entries), len(image.data)) - number = self.writer.add( - head.encode("latin-1") + image.data + b"\nendstream") + head = "<< %s /Length %d >>\nstream\n" % (" ".join(entries), len(image.data)) + number = self.writer.add(head.encode("latin-1") + image.data + b"\nendstream") name = "/Im%d" % len(self.xobjects) self.xobjects[name] = (name, number) if cache_key is not None: @@ -991,17 +1084,24 @@ def image(self, image, cache_key=None): def shading(self, coords, stops): """An axial (type 2) shading stitched from the gradient's stops.""" - stops = sorted(((max(0.0, min(1.0, at)), rgb) for at, rgb in stops), - key=lambda pair: pair[0]) + stops = sorted( + ((max(0.0, min(1.0, at)), rgb) for at, rgb in stops), + key=lambda pair: pair[0], + ) if len(stops) == 1: stops = [(0.0, stops[0][1]), (1.0, stops[0][1])] functions, bounds, encode = [], [], [] for index in range(len(stops) - 1): start, end = stops[index], stops[index + 1] - functions.append(self.writer.add( - "<< /FunctionType 2 /Domain [0 1] /C0 [%s] /C1 [%s] /N 1 >>" - % (" ".join(num(c) for c in start[1][:3]), - " ".join(num(c) for c in end[1][:3])))) + functions.append( + self.writer.add( + "<< /FunctionType 2 /Domain [0 1] /C0 [%s] /C1 [%s] /N 1 >>" + % ( + " ".join(num(c) for c in start[1][:3]), + " ".join(num(c) for c in end[1][:3]), + ) + ) + ) if index: bounds.append(stops[index][0]) encode.append("0 1") @@ -1011,12 +1111,17 @@ def shading(self, coords, stops): combined = self.writer.add( "<< /FunctionType 3 /Domain [0 1] /Functions [%s] " "/Bounds [%s] /Encode [%s] >>" - % (" ".join("%d 0 R" % f for f in functions), - " ".join(num(b) for b in bounds), " ".join(encode))) + % ( + " ".join("%d 0 R" % f for f in functions), + " ".join(num(b) for b in bounds), + " ".join(encode), + ) + ) number = self.writer.add( "<< /ShadingType 2 /ColorSpace /DeviceRGB /Coords [%s] " "/Function %d 0 R /Extend [true true] >>" - % (" ".join(num(c) for c in coords), combined)) + % (" ".join(num(c) for c in coords), combined) + ) name = "/Sh%d" % len(self.shadings) self.shadings[name] = (name, number) return name @@ -1036,24 +1141,31 @@ def gradient_mask(self, coords, stops, bbox): shading = self.shading(coords, greys) canvas_ops = "q %s sh Q" % shading form = self.writer.add_stream( - ["/Type /XObject", "/Subtype /Form", - "/BBox [%s]" % " ".join(num(v) for v in bbox), - "/Group << /Type /Group /S /Transparency /CS /DeviceGray >>", - "/Resources << /Shading << %s %d 0 R >> >>" - % (shading, self.shadings[shading][1])], - canvas_ops.encode("latin-1")) + [ + "/Type /XObject", + "/Subtype /Form", + "/BBox [%s]" % " ".join(num(v) for v in bbox), + "/Group << /Type /Group /S /Transparency /CS /DeviceGray >>", + "/Resources << /Shading << %s %d 0 R >> >>" + % (shading, self.shadings[shading][1]), + ], + canvas_ops.encode("latin-1"), + ) number = self.writer.add( "<< /Type /ExtGState /SMask << /S /Luminosity /G %d 0 R " - "/BC [0] >> >>" % form) + "/BC [0] >> >>" % form + ) name = "/GM%d" % len(self.gstates) self.gstates[name] = (name, number) return name def dictionary(self): def group(items): - return " ".join("%s %d 0 R" % (name, number) - for name, number in sorted(items, - key=lambda pair: pair[0])) + return " ".join( + "%s %d 0 R" % (name, number) + for name, number in sorted(items, key=lambda pair: pair[0]) + ) + parts = ["/ProcSet [/PDF /Text /ImageB /ImageC /ImageI]"] parts.append("/Font << %s >>" % group(self.fonts.values())) if self.xobjects: @@ -1130,8 +1242,7 @@ def rotate(self, degrees, cx, cy): radians = math.radians(degrees) cos, sin = math.cos(radians), math.sin(radians) self.translate(cx, cy) - self.op("%s %s %s %s 0 0 cm" - % (num(cos), num(sin), num(-sin), num(cos))) + self.op("%s %s %s %s 0 0 cm" % (num(cos), num(sin), num(-sin), num(cos))) self.translate(-cx, -cy) # -- paths ------------------------------------------------------------- @@ -1144,39 +1255,102 @@ def rect_path(self, x, y, w, h, radius=0): right, bottom = x + w, y + h self.op("%s %s m" % (num(x + radius), num(y))) self.op("%s %s l" % (num(right - radius), num(y))) - self.op("%s %s %s %s %s %s c" - % (num(right - radius + k), num(y), num(right), - num(y + radius - k), num(right), num(y + radius))) + self.op( + "%s %s %s %s %s %s c" + % ( + num(right - radius + k), + num(y), + num(right), + num(y + radius - k), + num(right), + num(y + radius), + ) + ) self.op("%s %s l" % (num(right), num(bottom - radius))) - self.op("%s %s %s %s %s %s c" - % (num(right), num(bottom - radius + k), - num(right - radius + k), num(bottom), - num(right - radius), num(bottom))) + self.op( + "%s %s %s %s %s %s c" + % ( + num(right), + num(bottom - radius + k), + num(right - radius + k), + num(bottom), + num(right - radius), + num(bottom), + ) + ) self.op("%s %s l" % (num(x + radius), num(bottom))) - self.op("%s %s %s %s %s %s c" - % (num(x + radius - k), num(bottom), num(x), - num(bottom - radius + k), num(x), num(bottom - radius))) + self.op( + "%s %s %s %s %s %s c" + % ( + num(x + radius - k), + num(bottom), + num(x), + num(bottom - radius + k), + num(x), + num(bottom - radius), + ) + ) self.op("%s %s l" % (num(x), num(y + radius))) - self.op("%s %s %s %s %s %s c" - % (num(x), num(y + radius - k), num(x + radius - k), num(y), - num(x + radius), num(y))) + self.op( + "%s %s %s %s %s %s c" + % ( + num(x), + num(y + radius - k), + num(x + radius - k), + num(y), + num(x + radius), + num(y), + ) + ) self.op("h") def ellipse_path(self, cx, cy, rx, ry): kx, ky = rx * _ARC_K, ry * _ARC_K self.op("%s %s m" % (num(cx + rx), num(cy))) - self.op("%s %s %s %s %s %s c" % (num(cx + rx), num(cy + ky), - num(cx + kx), num(cy + ry), - num(cx), num(cy + ry))) - self.op("%s %s %s %s %s %s c" % (num(cx - kx), num(cy + ry), - num(cx - rx), num(cy + ky), - num(cx - rx), num(cy))) - self.op("%s %s %s %s %s %s c" % (num(cx - rx), num(cy - ky), - num(cx - kx), num(cy - ry), - num(cx), num(cy - ry))) - self.op("%s %s %s %s %s %s c" % (num(cx + kx), num(cy - ry), - num(cx + rx), num(cy - ky), - num(cx + rx), num(cy))) + self.op( + "%s %s %s %s %s %s c" + % ( + num(cx + rx), + num(cy + ky), + num(cx + kx), + num(cy + ry), + num(cx), + num(cy + ry), + ) + ) + self.op( + "%s %s %s %s %s %s c" + % ( + num(cx - kx), + num(cy + ry), + num(cx - rx), + num(cy + ky), + num(cx - rx), + num(cy), + ) + ) + self.op( + "%s %s %s %s %s %s c" + % ( + num(cx - rx), + num(cy - ky), + num(cx - kx), + num(cy - ry), + num(cx), + num(cy - ry), + ) + ) + self.op( + "%s %s %s %s %s %s c" + % ( + num(cx + kx), + num(cy - ry), + num(cx + rx), + num(cy - ky), + num(cx + rx), + num(cy), + ) + ) self.op("h") def polygon_path(self, points): @@ -1187,11 +1361,14 @@ def polygon_path(self, points): def line_path(self, x1, y1, x2, y2): self.op("%s %s m %s %s l" % (num(x1), num(y1), num(x2), num(y2))) - def paint(self, fill=None, stroke=None, width=1.0, dash=None, cap=None, - even_odd=False): + def paint( + self, fill=None, stroke=None, width=1.0, dash=None, cap=None, even_odd=False + ): """Close out the current path with the right painting operator.""" - self.apply_alpha(fill[3] if is_visible(fill) else 1.0, - stroke[3] if is_visible(stroke) else 1.0) + self.apply_alpha( + fill[3] if is_visible(fill) else 1.0, + stroke[3] if is_visible(stroke) else 1.0, + ) if is_visible(fill): self.fill_color(fill) if is_visible(stroke) and width > 0: @@ -1268,7 +1445,9 @@ def content(self): _TOKEN = re.compile( r"\{\{\s*(page|pages|title|date|time|author|company|subject|event)" - r"(?::([^}]*))?\s*\}\}", re.I) + r"(?::([^}]*))?\s*\}\}", + re.I, +) def visible_slides(doc): @@ -1282,9 +1461,11 @@ def visible_slides(doc): slides = doc.get("slides") if not isinstance(slides, list): return [] - return [s for s in slides - if isinstance(s, dict) and not s.get("stateOf") - and not s.get("hidden")] + return [ + s + for s in slides + if isinstance(s, dict) and not s.get("stateOf") and not s.get("hidden") + ] def build_fields(doc, index, total, now): @@ -1332,8 +1513,9 @@ def one(match): def number(value, default=0.0): - return float(value) if isinstance(value, (int, float)) \ - and value == value else default + return ( + float(value) if isinstance(value, (int, float)) and value == value else default + ) class Renderer: @@ -1351,8 +1533,7 @@ def __init__(self, doc): self.theme_font = theme.get("fontFamily") or "sans-serif" self.theme_color = parse_color(theme.get("color"), (0.1, 0.1, 0.1, 1.0)) self.theme_bg = parse_color(theme.get("background"), (1, 1, 1, 1.0)) - self.assets = doc.get("assets") if isinstance(doc.get("assets"), - dict) else {} + self.assets = doc.get("assets") if isinstance(doc.get("assets"), dict) else {} self.writer = PdfWriter() self.res = Resources(self.writer) self.encoder = TextEncoder() @@ -1381,17 +1562,21 @@ def image_for(self, src): if not isinstance(resolved, str) or not resolved: return None if not resolved.startswith("data:"): - self.warn("image sources that are not embedded (%s...) were left " - "out: a PDF has no network, so only data: URIs and " - "doc.assets entries can be drawn." - % resolved[:32], "remote-image") + self.warn( + "image sources that are not embedded (%s...) were left " + "out: a PDF has no network, so only data: URIs and " + "doc.assets entries can be drawn." % resolved[:32], + "remote-image", + ) return None image, reason = decode_image(resolved) if image is None: - self.warn("an embedded image was skipped because %s. The built-in " - "export draws PNG (non-interlaced) and JPEG; re-embed it " - "in one of those, or use the deck's own PDF export." - % reason, "image-%s" % reason) + self.warn( + "an embedded image was skipped because %s. The built-in " + "export draws PNG (non-interlaced) and JPEG; re-embed it " + "in one of those, or use the deck's own PDF export." % reason, + "image-%s" % reason, + ) return None key = resolved if len(resolved) < 4096 else None name = self.res.image(image, cache_key=key) @@ -1401,8 +1586,10 @@ def image_for(self, src): def render(self): slides = visible_slides(self.doc) if not slides: - raise PdfError("the document has no printable slides (every slide " - "is hidden or a state variant)") + raise PdfError( + "the document has no printable slides (every slide " + "is hidden or a state variant)" + ) now = time.localtime() pages = [] contents = [] @@ -1415,21 +1602,30 @@ def render(self): resources = self.res.dictionary() tree = self.writer.reserve() for number_, content in zip(pages, contents): - self.writer.put(number_, - "<< /Type /Page /Parent %d 0 R /MediaBox " - "[0 0 %s %s] /Resources %d 0 R /Contents %d 0 R >>" - % (tree, num(self.page_width), - num(self.page_height), resources, content)) - self.writer.put(tree, - "<< /Type /Pages /Count %d /Kids [%s] >>" - % (len(pages), " ".join("%d 0 R" % p for p in pages))) + self.writer.put( + number_, + "<< /Type /Page /Parent %d 0 R /MediaBox " + "[0 0 %s %s] /Resources %d 0 R /Contents %d 0 R >>" + % ( + tree, + num(self.page_width), + num(self.page_height), + resources, + content, + ), + ) + self.writer.put( + tree, + "<< /Type /Pages /Count %d /Kids [%s] >>" + % (len(pages), " ".join("%d 0 R" % p for p in pages)), + ) root = self.writer.add("<< /Type /Catalog /Pages %d 0 R >>" % tree) title = self.encoder.encode(str(self.doc.get("title") or "Bento deck")) info = self.writer.add( "<< /Title %s /Producer (fleet bento-slides skill) " "/CreationDate (D:%s) >>" - % (pdf_string(title).decode("latin-1"), - time.strftime("%Y%m%d%H%M%S", now))) + % (pdf_string(title).decode("latin-1"), time.strftime("%Y%m%d%H%M%S", now)) + ) note = self.encoder.warning() if note: self.warn(note, "encoding") @@ -1437,8 +1633,10 @@ def render(self): def render_slide(self, canvas, slide, fields): canvas.save() - canvas.op("%s 0 0 %s 0 %s cm" - % (num(self.scale), num(-self.scale), num(self.page_height))) + canvas.op( + "%s 0 0 %s 0 %s cm" + % (num(self.scale), num(-self.scale), num(self.page_height)) + ) background = parse_color(slide.get("background"), self.theme_bg) if is_visible(background): canvas.fill_color(background) @@ -1452,9 +1650,11 @@ def render_slide(self, canvas, slide, fields): except PdfError: raise except Exception as exc: # a bad element must not lose the deck - self.warn("element %r (%s) could not be drawn: %s" - % (element.get("id"), element.get("type"), exc), - "element-%s" % element.get("id")) + self.warn( + "element %r (%s) could not be drawn: %s" + % (element.get("id"), element.get("type"), exc), + "element-%s" % element.get("id"), + ) canvas.restore() def render_element(self, canvas, element, fields): @@ -1464,8 +1664,11 @@ def render_element(self, canvas, element, fields): w = number(element.get("w")) h = number(element.get("h")) opacity = element.get("opacity") - opacity = 1.0 if not isinstance(opacity, (int, float)) \ + opacity = ( + 1.0 + if not isinstance(opacity, (int, float)) else max(0.0, min(1.0, float(opacity))) + ) if opacity <= 0.001: return rotation = number(element.get("rotation")) @@ -1473,11 +1676,18 @@ def render_element(self, canvas, element, fields): if rotation: canvas.rotate(rotation, x + w / 2.0, y + h / 2.0) canvas.alpha(opacity) - if element.get("blur") or element.get("shadow") or \ - element.get("blend") or element.get("backdropFilter"): - self.warn("blur, drop shadow, blend and backdrop-filter effects " - "are not reproduced by the built-in export; the shapes " - "and text are drawn without them.", "filters") + if ( + element.get("blur") + or element.get("shadow") + or element.get("blend") + or element.get("backdropFilter") + ): + self.warn( + "blur, drop shadow, blend and backdrop-filter effects " + "are not reproduced by the built-in export; the shapes " + "and text are drawn without them.", + "filters", + ) box = (x, y, w, h) if kind == "text": self.render_text(canvas, element, box, fields) @@ -1492,12 +1702,14 @@ def render_element(self, canvas, element, fields): elif kind == "media": self.render_media(canvas, element, box) elif kind == "svg": - self.warn("an `svg` element was skipped: the built-in export has " - "no SVG renderer. Compose the artwork from shape " - "elements, or use the deck's own PDF export.", "svg") + self.warn( + "an `svg` element was skipped: the built-in export has " + "no SVG renderer. Compose the artwork from shape " + "elements, or use the deck's own PDF export.", + "svg", + ) else: - self.warn("unknown element type %r was skipped." % kind, - "type-%s" % kind) + self.warn("unknown element type %r was skipped." % kind, "type-%s" % kind) canvas.restore() # -- text -------------------------------------------------------------- @@ -1509,12 +1721,22 @@ def render_text(self, canvas, element, box, fields): return # print hides placeholders, and empty text draws nothing size = number(element.get("fontSize"), 24.0) or 24.0 line_height = element.get("lineHeight") - line_height = float(line_height) if isinstance( - line_height, (int, float)) and line_height > 0 else 1.2 + line_height = ( + float(line_height) + if isinstance(line_height, (int, float)) and line_height > 0 + else 1.2 + ) family = family_of(element.get("fontFamily") or self.theme_font) letter_spacing = number(element.get("letterSpacing")) - lines = layout_text(self.encoder, runs, w, family, size, - element.get("fontWeight"), letter_spacing) + lines = layout_text( + self.encoder, + runs, + w, + family, + size, + element.get("fontWeight"), + letter_spacing, + ) ascent, descent = VMETRICS[family] step = size * line_height block = step * len(lines) @@ -1534,8 +1756,11 @@ def render_text(self, canvas, element, box, fields): # background-clip:text. canvas.save() mode = 7 - elif isinstance(stroke, dict) and number(stroke.get("width")) > 0 \ - and stroke.get("fill") == "none": + elif ( + isinstance(stroke, dict) + and number(stroke.get("width")) > 0 + and stroke.get("fill") == "none" + ): color = parse_color(stroke.get("color"), color) if not is_visible(color) and not stops: return @@ -1549,16 +1774,26 @@ def render_text(self, canvas, element, box, fields): for piece in line.pieces: if not piece.text.strip(): continue - canvas.show_text(start + piece.x, baseline, piece.text, - piece.face, piece.size, letter_spacing) + canvas.show_text( + start + piece.x, + baseline, + piece.text, + piece.face, + piece.size, + letter_spacing, + ) canvas.end_text() if stops: # `ET` turns the accumulated glyph outlines (text render mode 7) # into the clip path, so the shading below paints THROUGH the # letters — the PDF equivalent of background-clip:text. - self.paint_gradient(canvas, gradient, stops, - self.gradient_coords(gradient, box), - (x, y, x + w, y + h)) + self.paint_gradient( + canvas, + gradient, + stops, + self.gradient_coords(gradient, box), + (x, y, x + w, y + h), + ) canvas.restore() def paint_gradient(self, canvas, gradient, stops, coords, bbox): @@ -1597,8 +1832,12 @@ def gradient_coords(self, gradient, box): radians = math.radians(angle) dx = math.sin(radians) / 2.0 dy = -math.cos(radians) / 2.0 - return (x + (0.5 - dx) * w, y + (0.5 - dy) * h, - x + (0.5 + dx) * w, y + (0.5 + dy) * h) + return ( + x + (0.5 - dx) * w, + y + (0.5 - dy) * h, + x + (0.5 + dx) * w, + y + (0.5 + dy) * h, + ) # ── SVG path data ──────────────────────────────────────────────────────────── @@ -1609,7 +1848,9 @@ def gradient_coords(self, gradient, box): # stroke width uniform, which is what the app's `vector-effect: # non-scaling-stroke` does in the browser. -_PATH_TOKEN = re.compile(r"([MmLlHhVvCcSsQqTtAaZz])|(-?[0-9]*\.?[0-9]+(?:[eE][-+]?[0-9]+)?)") +_PATH_TOKEN = re.compile( + r"([MmLlHhVvCcSsQqTtAaZz])|(-?[0-9]*\.?[0-9]+(?:[eE][-+]?[0-9]+)?)" +) def parse_path(data): @@ -1674,8 +1915,14 @@ def take(count): elif lower == "c": x1, y1, x2, y2, px, py = take(6) if relative: - x1, y1, x2, y2, px, py = (cx + x1, cy + y1, cx + x2, - cy + y2, cx + px, cy + py) + x1, y1, x2, y2, px, py = ( + cx + x1, + cy + y1, + cx + x2, + cy + y2, + cx + px, + cy + py, + ) out.append(("c", (x1, y1), (x2, y2), (px, py))) last_control = (x2, y2) cx, cy = px, py @@ -1702,20 +1949,22 @@ def take(count): if last_control is None: qx, qy = cx, cy else: - qx, qy = 2 * cx - last_control[0], \ - 2 * cy - last_control[1] - out.append(("c", - (cx + 2.0 / 3 * (qx - cx), cy + 2.0 / 3 * (qy - cy)), - (px + 2.0 / 3 * (qx - px), py + 2.0 / 3 * (qy - py)), - (px, py))) + qx, qy = 2 * cx - last_control[0], 2 * cy - last_control[1] + out.append( + ( + "c", + (cx + 2.0 / 3 * (qx - cx), cy + 2.0 / 3 * (qy - cy)), + (px + 2.0 / 3 * (qx - px), py + 2.0 / 3 * (qy - py)), + (px, py), + ) + ) last_control = (qx, qy) cx, cy = px, py elif lower == "a": rx, ry, rot, large, sweep, px, py = take(7) if relative: px, py = cx + px, cy + py - out.extend(_arc_to_beziers(cx, cy, rx, ry, rot, large, sweep, - px, py)) + out.extend(_arc_to_beziers(cx, cy, rx, ry, rot, large, sweep, px, py)) cx, cy = px, py else: break @@ -1741,8 +1990,7 @@ def _arc_to_beziers(x1, y1, rx, ry, rotation, large, sweep, x2, y2): scale = math.sqrt(lam) rx, ry = rx * scale, ry * scale denom = rx * rx * y1p * y1p + ry * ry * x1p * x1p - factor = 0.0 if denom == 0 else max( - 0.0, (rx * rx * ry * ry - denom) / denom) + factor = 0.0 if denom == 0 else max(0.0, (rx * rx * ry * ry - denom) / denom) coef = math.sqrt(factor) * (-1 if bool(large) == bool(sweep) else 1) cxp = coef * rx * y1p / ry cyp = -coef * ry * x1p / rx @@ -1769,8 +2017,10 @@ def angle_of(ux, uy): cos2, sin2 = math.cos(theta + step), math.sin(theta + step) def point(cos_t, sin_t): - return (cx + rx * cos_t * cos_phi - ry * sin_t * sin_phi, - cy + rx * cos_t * sin_phi + ry * sin_t * cos_phi) + return ( + cx + rx * cos_t * cos_phi - ry * sin_t * sin_phi, + cy + rx * cos_t * sin_phi + ry * sin_t * cos_phi, + ) px1, py1 = point(cos1, sin1) px2, py2 = point(cos2, sin2) @@ -1778,14 +2028,21 @@ def point(cos_t, sin_t): dy1 = -rx * sin1 * sin_phi + ry * cos1 * cos_phi dx2 = -rx * sin2 * cos_phi - ry * cos2 * sin_phi dy2 = -rx * sin2 * sin_phi + ry * cos2 * cos_phi - out.append(("c", (px1 + k * dx1, py1 + k * dy1), - (px2 - k * dx2, py2 - k * dy2), (px2, py2))) + out.append( + ( + "c", + (px1 + k * dx1, py1 + k * dy1), + (px2 - k * dx2, py2 - k * dy2), + (px2, py2), + ) + ) theta += step return out # ── shape, image, table and media elements ─────────────────────────────────── + def _dash_pattern(element, width): """The app's stroke-dasharray, in the same units.""" style = element.get("strokeStyle") @@ -1794,8 +2051,7 @@ def _dash_pattern(element, width): if style == "dotted": return [0.1, max(width * 2.2, 5)], 1 dash = element.get("strokeDash") - if style not in (None, "solid") and isinstance(dash, (int, float)) \ - and dash > 0: + if style not in (None, "solid") and isinstance(dash, (int, float)) and dash > 0: return [dash, dash], None return None, None @@ -1818,62 +2074,86 @@ def render_shape(self, canvas, element, box): canvas.save() canvas.translate(x, y) if shape == "rect": - canvas.rect_path(inset, inset, max(w - stroke_width, 0), - max(h - stroke_width, 0), - number(element.get("radius"))) + canvas.rect_path( + inset, + inset, + max(w - stroke_width, 0), + max(h - stroke_width, 0), + number(element.get("radius")), + ) elif shape == "ellipse": - canvas.ellipse_path(w / 2.0, h / 2.0, max(w / 2.0 - inset, 0), - max(h / 2.0 - inset, 0)) + canvas.ellipse_path( + w / 2.0, h / 2.0, max(w / 2.0 - inset, 0), max(h / 2.0 - inset, 0) + ) elif shape == "triangle": - canvas.polygon_path([(w / 2.0, inset), (w - inset, h - inset), - (inset, h - inset)]) + canvas.polygon_path( + [(w / 2.0, inset), (w - inset, h - inset), (inset, h - inset)] + ) elif shape == "arrow": shaft = h * 0.44 head = min(w * 0.38, h) top = (h - shaft) / 2.0 - canvas.polygon_path([(0, top), (w - head, top), (w - head, 0), - (w, h / 2.0), (w - head, h), - (w - head, top + shaft), (0, top + shaft)]) + canvas.polygon_path( + [ + (0, top), + (w - head, top), + (w - head, 0), + (w, h / 2.0), + (w - head, h), + (w - head, top + shaft), + (0, top + shaft), + ] + ) elif shape == "path": self.emit_path(canvas, element, w, h) else: - self.warn("unknown shape %r was skipped." % shape, - "shape-%s" % shape) + self.warn("unknown shape %r was skipped." % shape, "shape-%s" % shape) canvas.restore() return if stops: canvas.save() canvas.clip() - self.paint_gradient(canvas, element.get("fillGradient"), stops, - self.gradient_coords(element.get("fillGradient"), - (0, 0, w, h)), - (0, 0, w, h)) + self.paint_gradient( + canvas, + element.get("fillGradient"), + stops, + self.gradient_coords(element.get("fillGradient"), (0, 0, w, h)), + (0, 0, w, h), + ) canvas.restore() if is_visible(stroke) and stroke_width > 0: # The fill is painted by the shading, so re-lay the outline for # the stroke pass rather than trying to keep the clipped path. - self.render_shape_outline(canvas, element, box, stroke, - stroke_width, dash, cap) + self.render_shape_outline( + canvas, element, box, stroke, stroke_width, dash, cap + ) else: - canvas.paint(fill=fill, stroke=stroke, width=stroke_width, - dash=dash, cap=cap) + canvas.paint( + fill=fill, stroke=stroke, width=stroke_width, dash=dash, cap=cap + ) canvas.restore() - def render_shape_outline(self, canvas, element, box, stroke, width, dash, - cap): + def render_shape_outline(self, canvas, element, box, stroke, width, dash, cap): _, _, w, h = box inset = width / 2.0 shape = element.get("shape") or "rect" if shape == "rect": - canvas.rect_path(inset, inset, max(w - width, 0), max(h - width, 0), - number(element.get("radius"))) + canvas.rect_path( + inset, + inset, + max(w - width, 0), + max(h - width, 0), + number(element.get("radius")), + ) elif shape == "ellipse": - canvas.ellipse_path(w / 2.0, h / 2.0, max(w / 2.0 - inset, 0), - max(h / 2.0 - inset, 0)) + canvas.ellipse_path( + w / 2.0, h / 2.0, max(w / 2.0 - inset, 0), max(h / 2.0 - inset, 0) + ) elif shape == "triangle": - canvas.polygon_path([(w / 2.0, inset), (w - inset, h - inset), - (inset, h - inset)]) + canvas.polygon_path( + [(w / 2.0, inset), (w - inset, h - inset), (inset, h - inset)] + ) elif shape == "path": self.emit_path(canvas, element, w, h) else: @@ -1886,8 +2166,11 @@ def emit_path(self, canvas, element, w, h): if not segments: return pbox = element.get("pathBox") - if isinstance(pbox, list) and len(pbox) == 4 and \ - all(isinstance(v, (int, float)) for v in pbox): + if ( + isinstance(pbox, list) + and len(pbox) == 4 + and all(isinstance(v, (int, float)) for v in pbox) + ): vx, vy, vw, vh = (float(v) for v in pbox) else: vx, vy, vw, vh = 0.0, 0.0, w or 1.0, h or 1.0 @@ -1906,8 +2189,10 @@ def point(pair): canvas.op("%s %s l" % (num(px), num(py))) elif segment[0] == "c": (a, b), (c, d), (e, f) = (point(p) for p in segment[1:]) - canvas.op("%s %s %s %s %s %s c" - % (num(a), num(b), num(c), num(d), num(e), num(f))) + canvas.op( + "%s %s %s %s %s %s c" + % (num(a), num(b), num(c), num(d), num(e), num(f)) + ) elif segment[0] == "z": canvas.op("h") @@ -1918,6 +2203,7 @@ def render_line(self, canvas, element, box): color = parse_color(element.get("fill")) if not is_visible(color): return + def tip(kind): return width * 2.6 if kind and kind != "none" else 0.0 @@ -1927,14 +2213,26 @@ def tip(kind): dash, _ = _dash_pattern(element, width) canvas.save() canvas.line_path(x + start, mid, x + w - end, mid) - canvas.paint(stroke=color, width=width, dash=dash, - cap=0 if element.get("strokeStyle") == "dashed" else 1) - for kind, at_start in ((element.get("lineStart"), True), - (element.get("lineEnd"), False)): + canvas.paint( + stroke=color, + width=width, + dash=dash, + cap=0 if element.get("strokeStyle") == "dashed" else 1, + ) + for kind, at_start in ( + (element.get("lineStart"), True), + (element.get("lineEnd"), False), + ): if kind and kind != "none": - self.render_tip(canvas, kind, color, width, - x + (start if at_start else w - end), mid, - at_start) + self.render_tip( + canvas, + kind, + color, + width, + x + (start if at_start else w - end), + mid, + at_start, + ) canvas.restore() def render_tip(self, canvas, kind, color, width, at_x, at_y, reversed_): @@ -1945,15 +2243,19 @@ def render_tip(self, canvas, kind, color, width, at_x, at_y, reversed_): ref = 6.4 * unit tip_x = at_x + direction * (7.6 * unit - ref) back_x = at_x - direction * ref - canvas.polygon_path([ - (back_x, at_y - (4 - 0.4) * unit), - (tip_x, at_y), - (back_x, at_y + (7.6 - 4) * unit)]) + canvas.polygon_path( + [ + (back_x, at_y - (4 - 0.4) * unit), + (tip_x, at_y), + (back_x, at_y + (7.6 - 4) * unit), + ] + ) elif kind == "dot": canvas.ellipse_path(at_x, at_y, 2.6 * unit, 2.6 * unit) else: # bar - canvas.rect_path(at_x - 0.8 * unit, at_y - 3.6 * unit, - 1.6 * unit, 7.2 * unit) + canvas.rect_path( + at_x - 0.8 * unit, at_y - 3.6 * unit, 1.6 * unit, 7.2 * unit + ) canvas.paint(fill=color) def render_image(self, canvas, element, box): @@ -1969,11 +2271,15 @@ def render_image(self, canvas, element, box): canvas.rect_path(x, y, w, h, radius) canvas.clip() if fit in ("cover", "contain") and natural_w and natural_h: - scale = max(w / natural_w, h / natural_h) if fit == "cover" \ + scale = ( + max(w / natural_w, h / natural_h) + if fit == "cover" else min(w / natural_w, h / natural_h) + ) draw_w, draw_h = natural_w * scale, natural_h * scale - canvas.draw_image(name, x + (w - draw_w) / 2.0, - y + (h - draw_h) / 2.0, draw_w, draw_h) + canvas.draw_image( + name, x + (w - draw_w) / 2.0, y + (h - draw_h) / 2.0, draw_w, draw_h + ) else: canvas.draw_image(name, x, y, w, h) canvas.restore() @@ -1983,8 +2289,11 @@ def render_media(self, canvas, element, box): x, y, w, h = box kind = element.get("kind") radius = number(element.get("radius")) - backdrop = (0.043, 0.059, 0.078, 1.0) if kind == "video" \ + backdrop = ( + (0.043, 0.059, 0.078, 1.0) + if kind == "video" else (0.906, 0.929, 0.957, 1.0) + ) canvas.save() canvas.rect_path(x, y, w, h, radius) canvas.paint(fill=backdrop) @@ -2001,18 +2310,26 @@ def render_media(self, canvas, element, box): size = min(w, h) * 0.18 cx, cy = x + w / 2.0, y + h / 2.0 if kind == "video": - canvas.polygon_path([(cx - size * 0.4, cy - size * 0.55), - (cx + size * 0.6, cy), - (cx - size * 0.4, cy + size * 0.55)]) + canvas.polygon_path( + [ + (cx - size * 0.4, cy - size * 0.55), + (cx + size * 0.6, cy), + (cx - size * 0.4, cy + size * 0.55), + ] + ) canvas.paint(fill=glyph) else: - canvas.ellipse_path(cx - size * 0.25, cy + size * 0.35, - size * 0.28, size * 0.22) + canvas.ellipse_path( + cx - size * 0.25, cy + size * 0.35, size * 0.28, size * 0.22 + ) canvas.paint(fill=glyph) canvas.rect_path(cx, cy - size * 0.6, size * 0.12, size * 0.95) canvas.paint(fill=glyph) - self.warn("video and audio elements are drawn as a poster block — " - "a PDF cannot play media.", "media") + self.warn( + "video and audio elements are drawn as a poster block — " + "a PDF cannot play media.", + "media", + ) canvas.restore() @@ -2030,13 +2347,18 @@ def render_table(self, canvas, element, box): x, y, w, h = box columns = element.get("columns") rows = element.get("rows") - if not isinstance(columns, list) or not isinstance(rows, list) \ - or not columns or not rows: + if ( + not isinstance(columns, list) + or not isinstance(rows, list) + or not columns + or not rows + ): return - style = element.get("style") if isinstance(element.get("style"), - dict) else {} - weights = [max(0.0, number(c.get("w"))) if isinstance(c, dict) else 0.0 - for c in columns] + style = element.get("style") if isinstance(element.get("style"), dict) else {} + weights = [ + max(0.0, number(c.get("w"))) if isinstance(c, dict) else 0.0 + for c in columns + ] total_weight = sum(weights) or 1.0 widths = [w * weight / total_weight for weight in weights] header = bool(element.get("header")) @@ -2063,15 +2385,17 @@ def render_table(self, canvas, element, box): cell = cells[column_index] if column_index < len(cells) else {} cell = cell if isinstance(cell, dict) else {} bold = bool(cell.get("bold")) or is_header - inner = max(widths[column_index] - pad_x * 2 - border_width * 2, - 1.0) - lines = layout_text(self.encoder, - parse_inline(str(cell.get("html") or "")), - inner, family, font_size, - 700 if bold else 400) + inner = max(widths[column_index] - pad_x * 2 - border_width * 2, 1.0) + lines = layout_text( + self.encoder, + parse_inline(str(cell.get("html") or "")), + inner, + family, + font_size, + 700 if bold else 400, + ) laid.append((cell, lines, bold)) - tallest = max(tallest, len(lines) * font_size * line_height - + pad_y * 2) + tallest = max(tallest, len(lines) * font_size * line_height + pad_y * 2) grid.append(laid) natural.append(tallest) @@ -2098,13 +2422,15 @@ def render_table(self, canvas, element, box): width = widths[column_index] height = heights[row_index] background = cell.get("bg") or ( - style.get("headerBg") if is_header else zebra) + style.get("headerBg") if is_header else zebra + ) fill = parse_color(background, (0, 0, 0, 0.0)) if is_visible(fill) or (is_visible(border) and border_width): canvas.rect_path(left, top, width, height) canvas.paint(fill=fill, stroke=border, width=border_width) color = cell.get("color") or ( - style.get("headerColor") if is_header else style.get("color")) + style.get("headerColor") if is_header else style.get("color") + ) text_color = parse_color(color, self.theme_color) if not is_visible(text_color) or not lines: left += width @@ -2119,14 +2445,19 @@ def render_table(self, canvas, element, box): inner_width = max(width - pad_x * 2 - border_width * 2, 1.0) half_leading = (step - font_size * (ascent + descent)) / 2.0 for line_index, line in enumerate(lines): - baseline = (cell_top + line_index * step + half_leading - + ascent * font_size) + baseline = ( + cell_top + line_index * step + half_leading + ascent * font_size + ) start = inner_left + (inner_width - line.width) * align for piece in line.pieces: if piece.text.strip(): - canvas.show_text(start + piece.x, baseline, - piece.text, piece.face, - piece.size) + canvas.show_text( + start + piece.x, + baseline, + piece.text, + piece.face, + piece.size, + ) left += width top += heights[row_index] canvas.restore() @@ -2140,16 +2471,25 @@ def render_table(self, canvas, element, box): # placement. The engine deliberately honours only a subset of the ECharts option # shape, and so does this — a key the app ignores is a key we ignore. -CHART_COLORS = ["#5470c6", "#91cc75", "#fac858", "#ee6666", - "#73c0de", "#3ba272", "#fc8452", "#9a60b4"] +CHART_COLORS = [ + "#5470c6", + "#91cc75", + "#fac858", + "#ee6666", + "#73c0de", + "#3ba272", + "#fc8452", + "#9a60b4", +] AXIS_TEXT = "#6B7280" AXIS_LINE = (0.431, 0.471, 0.529, 0.45) SPLIT_LINE = (0.431, 0.471, 0.529, 0.15) def opt_num(value, default): - return float(value) if isinstance(value, (int, float)) \ - and value == value else default + return ( + float(value) if isinstance(value, (int, float)) and value == value else default + ) def format_number(value): @@ -2161,8 +2501,9 @@ def format_number(value): digits = text.lstrip("-") whole, _, fraction = digits.partition(".") grouped = "{:,}".format(int(whole)) - text = ("-" if negative else "") + grouped + ( - "." + fraction if fraction else "") + text = ( + ("-" if negative else "") + grouped + ("." + fraction if fraction else "") + ) return text @@ -2173,8 +2514,9 @@ def nice_ticks(low, high, count=5): span = high - low power = math.pow(10, math.floor(math.log10(span / count))) ratio = span / count / power - step = power * (10 if ratio >= 7.5 else 5 if ratio >= 3.5 - else 2 if ratio >= 1.5 else 1) + step = power * ( + 10 if ratio >= 7.5 else 5 if ratio >= 3.5 else 2 if ratio >= 1.5 else 1 + ) start = math.floor(low / step) * step stop = math.ceil(high / step) * step ticks = [] @@ -2195,8 +2537,9 @@ def magnitude(value): return 1 exponent = math.floor(math.log10(value)) mantissa = value / math.pow(10, exponent) - pick = 1 if mantissa <= 1 else 2 if mantissa <= 2 else \ - 5 if mantissa <= 5 else 10 + pick = ( + 1 if mantissa <= 1 else 2 if mantissa <= 2 else 5 if mantissa <= 5 else 10 + ) return pick * math.pow(10, exponent) slots = max(1, count) @@ -2219,14 +2562,20 @@ def axis_scale(values, axis, tick_count=None): top = axis.get("max") if axis.get("max") is not None else (high or 1.0) pinned = axis.get("min") is not None or axis.get("max") is not None if tick_count: - ticks = _even_ticks(bottom, top, tick_count) if pinned \ + ticks = ( + _even_ticks(bottom, top, tick_count) + if pinned else _step_ticks(bottom, top, tick_count - 1) + ) else: - ticks = _even_ticks(bottom, top, 6) if pinned \ - else nice_ticks(bottom, top) + ticks = _even_ticks(bottom, top, 6) if pinned else nice_ticks(bottom, top) formatter = axis.get("formatter") - labels = [formatter.replace("{value}", format_number(t)) if formatter - else format_number(t) for t in ticks] + labels = [ + formatter.replace("{value}", format_number(t)) + if formatter + else format_number(t) + for t in ticks + ] return {"lo": ticks[0], "hi": ticks[-1], "labels": labels} @@ -2236,16 +2585,20 @@ def normalize_chart(option, width, height): series = option.get("series") if isinstance(series, dict): series = [series] - series = [s for s in series if isinstance(s, dict)] \ - if isinstance(series, list) else [] + series = ( + [s for s in series if isinstance(s, dict)] if isinstance(series, list) else [] + ) is_pie = any(s.get("type") == "pie" for s in series) grid = option.get("grid") if isinstance(option.get("grid"), dict) else {} legend_option = option.get("legend") legend = None if legend_option: legend_option = legend_option if isinstance(legend_option, dict) else {} - text_style = legend_option.get("textStyle") \ - if isinstance(legend_option.get("textStyle"), dict) else {} + text_style = ( + legend_option.get("textStyle") + if isinstance(legend_option.get("textStyle"), dict) + else {} + ) legend = { "color": text_style.get("color") or AXIS_TEXT, "size": opt_num(text_style.get("fontSize"), 12), @@ -2254,13 +2607,15 @@ def normalize_chart(option, width, height): "itemHeight": opt_num(legend_option.get("itemHeight"), 10), "itemGap": opt_num(legend_option.get("itemGap"), 16), "top": legend_option.get("top") - if isinstance(legend_option.get("top"), (int, float)) else None, + if isinstance(legend_option.get("top"), (int, float)) + else None, "bottom": opt_num(legend_option.get("bottom"), 0), } band = max(legend["itemHeight"], legend["size"]) + 12 if legend else 0 x_axis = option.get("xAxis") if isinstance(option.get("xAxis"), dict) else {} - x_label = x_axis.get("axisLabel") \ - if isinstance(x_axis.get("axisLabel"), dict) else {} + x_label = ( + x_axis.get("axisLabel") if isinstance(x_axis.get("axisLabel"), dict) else {} + ) y_option = option.get("yAxis") if isinstance(y_option, dict): y_option = [y_option] @@ -2269,65 +2624,99 @@ def normalize_chart(option, width, height): y_axes = [] for axis in y_option[:2]: axis = axis if isinstance(axis, dict) else {} - label = axis.get("axisLabel") \ - if isinstance(axis.get("axisLabel"), dict) else {} - y_axes.append({ - "name": axis.get("name") if isinstance(axis.get("name"), str) - else None, - "min": axis.get("min") if isinstance(axis.get("min"), - (int, float)) else None, - "max": axis.get("max") if isinstance(axis.get("max"), - (int, float)) else None, - "formatter": label.get("formatter") - if isinstance(label.get("formatter"), str) else None, - "label": { - "color": label.get("color") or x_label.get("color") or AXIS_TEXT, - "size": opt_num(label.get("fontSize"), 12), - "weight": label.get("fontWeight", 400), - }, - }) + label = axis.get("axisLabel") if isinstance(axis.get("axisLabel"), dict) else {} + y_axes.append( + { + "name": axis.get("name") if isinstance(axis.get("name"), str) else None, + "min": axis.get("min") + if isinstance(axis.get("min"), (int, float)) + else None, + "max": axis.get("max") + if isinstance(axis.get("max"), (int, float)) + else None, + "formatter": label.get("formatter") + if isinstance(label.get("formatter"), str) + else None, + "label": { + "color": label.get("color") or x_label.get("color") or AXIS_TEXT, + "size": opt_num(label.get("fontSize"), 12), + "weight": label.get("fontWeight", 400), + }, + } + ) dual = not is_pie and len(y_axes) > 1 top_band = legend["top"] + band if legend and legend["top"] is not None else 0 - bottom_band = legend["bottom"] + band \ - if legend and legend["top"] is None else 0 + bottom_band = legend["bottom"] + band if legend and legend["top"] is None else 0 overflow = max(0.0, band - 12) - grid_bottom = opt_num(grid.get("bottom"), 44) if "bottom" in grid else \ - 44 + (overflow if legend and legend["top"] is None else 0) - grid_top = opt_num(grid.get("top"), 24) if "top" in grid else \ - 24 + (overflow if legend and legend["top"] is not None else 0) + grid_bottom = ( + opt_num(grid.get("bottom"), 44) + if "bottom" in grid + else 44 + (overflow if legend and legend["top"] is None else 0) + ) + grid_top = ( + opt_num(grid.get("top"), 24) + if "top" in grid + else 24 + (overflow if legend and legend["top"] is not None else 0) + ) if is_pie: - plot = {"x": 0.0, "y": top_band, "w": width, - "h": max(0.0, height - top_band - bottom_band)} + plot = { + "x": 0.0, + "y": top_band, + "w": width, + "h": max(0.0, height - top_band - bottom_band), + } else: left = opt_num(grid.get("left"), 48) right = opt_num(grid.get("right"), 56 if dual else 16) - plot = {"x": left, "y": grid_top, "w": width - left - right, - "h": max(0.0, height - grid_top - grid_bottom)} + plot = { + "x": left, + "y": grid_top, + "w": width - left - right, + "h": max(0.0, height - grid_top - grid_bottom), + } colors = option.get("color") - colors = [c for c in colors if isinstance(c, str)] \ - if isinstance(colors, list) and colors else CHART_COLORS - text_style = option.get("textStyle") \ - if isinstance(option.get("textStyle"), dict) else {} - axis_line = x_axis.get("axisLine") \ - if isinstance(x_axis.get("axisLine"), dict) else {} - axis_line_style = axis_line.get("lineStyle") \ - if isinstance(axis_line.get("lineStyle"), dict) else {} + colors = ( + [c for c in colors if isinstance(c, str)] + if isinstance(colors, list) and colors + else CHART_COLORS + ) + text_style = ( + option.get("textStyle") if isinstance(option.get("textStyle"), dict) else {} + ) + axis_line = ( + x_axis.get("axisLine") if isinstance(x_axis.get("axisLine"), dict) else {} + ) + axis_line_style = ( + axis_line.get("lineStyle") + if isinstance(axis_line.get("lineStyle"), dict) + else {} + ) first_y = y_option[0] if isinstance(y_option[0], dict) else {} - split = first_y.get("splitLine") \ - if isinstance(first_y.get("splitLine"), dict) else {} - split_style = split.get("lineStyle") \ - if isinstance(split.get("lineStyle"), dict) else {} + split = ( + first_y.get("splitLine") if isinstance(first_y.get("splitLine"), dict) else {} + ) + split_style = ( + split.get("lineStyle") if isinstance(split.get("lineStyle"), dict) else {} + ) pie = next((s for s in series if s.get("type") == "pie"), None) - pie_label = pie.get("label") if isinstance(pie, dict) and \ - isinstance(pie.get("label"), dict) else {} + pie_label = ( + pie.get("label") + if isinstance(pie, dict) and isinstance(pie.get("label"), dict) + else {} + ) categories = x_axis.get("data") return { - "w": width, "h": height, + "w": width, + "h": height, "font": family_of(text_style.get("fontFamily") or "sans-serif"), - "colors": colors, "series": series, "isPie": is_pie, + "colors": colors, + "series": series, + "isPie": is_pie, "categories": [str(c) for c in categories] - if isinstance(categories, list) else [], - "grid": plot, "legend": legend, + if isinstance(categories, list) + else [], + "grid": plot, + "legend": legend, "xAxisLabel": { "color": x_label.get("color") or y_axes[0]["label"]["color"], "size": opt_num(x_label.get("fontSize"), 12), @@ -2355,22 +2744,36 @@ def render_chart(self, canvas, element, box): if not chart["series"]: # The app draws the bare axis frame in this case, so we do too — # the warning is for the agent, not a reason to diverge. - self.warn("a chart element has no series; only its axes were " - "drawn (the app does the same).", "chart-empty") + self.warn( + "a chart element has no series; only its axes were " + "drawn (the app does the same).", + "chart-empty", + ) canvas.save() canvas.translate(x, y) if chart["isPie"]: self.draw_pie(canvas, chart) else: self.draw_cartesian(canvas, chart) - if chart["legend"] and any(s.get("name") or s.get("type") == "pie" - for s in chart["series"]): + if chart["legend"] and any( + s.get("name") or s.get("type") == "pie" for s in chart["series"] + ): self.draw_legend(canvas, chart) canvas.restore() # -- helpers ----------------------------------------------------------- - def chart_text(self, canvas, chart, cx, baseline, text, color, size, - anchor="middle", weight=400): + def chart_text( + self, + canvas, + chart, + cx, + baseline, + text, + color, + size, + anchor="middle", + weight=400, + ): """`si()` in the runtime: an SVG with a text-anchor.""" if text is None or text == "": return @@ -2381,17 +2784,22 @@ def chart_text(self, canvas, chart, cx, baseline, text, color, size, start = cx - width / 2.0 elif anchor == "end": start = cx - width - rgba = color if isinstance(color, tuple) \ + rgba = ( + color + if isinstance(color, tuple) else parse_color(color, (0.42, 0.45, 0.5, 1.0)) + ) canvas.apply_alpha(rgba[3]) canvas.fill_color(rgba) canvas.show_text(start, baseline, text, face, size) def series_color(self, chart, series, index): - item_style = series.get("itemStyle") \ - if isinstance(series.get("itemStyle"), dict) else {} - line_style = series.get("lineStyle") \ - if isinstance(series.get("lineStyle"), dict) else {} + item_style = ( + series.get("itemStyle") if isinstance(series.get("itemStyle"), dict) else {} + ) + line_style = ( + series.get("lineStyle") if isinstance(series.get("lineStyle"), dict) else {} + ) for candidate in (item_style.get("color"), line_style.get("color")): if isinstance(candidate, str): return parse_color(candidate) @@ -2408,28 +2816,34 @@ def draw_cartesian(self, canvas, chart): axis_count = len(chart["yAxes"]) def axis_index(series): - return min(axis_count - 1, - max(0, int(round(opt_num(series.get("yAxisIndex"), 0))))) + return min( + axis_count - 1, max(0, int(round(opt_num(series.get("yAxisIndex"), 0)))) + ) buckets = [[] for _ in range(axis_count)] for series in bars + lines: data = series.get("data") data = data if isinstance(data, list) else [] buckets[axis_index(series)].extend( - opt_num(value, 0) for value in data[:count]) + opt_num(value, 0) for value in data[:count] + ) low = float("inf") high = float("-inf") for series in points: for entry in series.get("data") or []: - px, py = (entry[0], entry[1]) if isinstance(entry, list) \ - and len(entry) >= 2 else (0, 0) + px, py = ( + (entry[0], entry[1]) + if isinstance(entry, list) and len(entry) >= 2 + else (0, 0) + ) low = min(low, px) high = max(high, px) buckets[0].append(opt_num(py, 0)) primary = axis_scale(buckets[0], chart["yAxes"][0]) ticks = len(primary["labels"]) - secondary = axis_scale(buckets[1], chart["yAxes"][1], ticks) \ - if axis_count > 1 else None + secondary = ( + axis_scale(buckets[1], chart["yAxes"][1], ticks) if axis_count > 1 else None + ) def value_y(value, axis=0): scale = secondary if axis == 1 and secondary else primary @@ -2443,40 +2857,66 @@ def zero_y(axis=0): for index in range(ticks): line_y = plot["y"] + plot["h"] - index / max(1, ticks - 1) * plot["h"] canvas.line_path(plot["x"], line_y, plot["x"] + plot["w"], line_y) - canvas.paint(stroke=chart["splitLine"]["color"], - width=chart["splitLine"]["width"]) + canvas.paint( + stroke=chart["splitLine"]["color"], width=chart["splitLine"]["width"] + ) label = chart["yAxes"][0]["label"] - self.chart_text(canvas, chart, plot["x"] - 8, - line_y + label["size"] * 0.35, - primary["labels"][index], - parse_color(label["color"], (0.42, 0.45, 0.5, 1)), - label["size"], "end", label["weight"]) + self.chart_text( + canvas, + chart, + plot["x"] - 8, + line_y + label["size"] * 0.35, + primary["labels"][index], + parse_color(label["color"], (0.42, 0.45, 0.5, 1)), + label["size"], + "end", + label["weight"], + ) if secondary: label2 = chart["yAxes"][1]["label"] - self.chart_text(canvas, chart, plot["x"] + plot["w"] + 8, - line_y + label2["size"] * 0.35, - secondary["labels"][index], - parse_color(label2["color"], - (0.42, 0.45, 0.5, 1)), - label2["size"], "start", label2["weight"]) - for axis, anchor, at_x in ((0, "end", plot["x"] - 8), - (1, "start", plot["x"] + plot["w"] + 8)): - if axis < axis_count and chart["yAxes"][axis].get("name") \ - and (axis == 0 or secondary): - self.chart_text(canvas, chart, at_x, plot["y"] - 9, - chart["yAxes"][axis]["name"], - parse_color(chart["xAxisLabel"]["color"], - (0.42, 0.45, 0.5, 1)), 11, anchor) + self.chart_text( + canvas, + chart, + plot["x"] + plot["w"] + 8, + line_y + label2["size"] * 0.35, + secondary["labels"][index], + parse_color(label2["color"], (0.42, 0.45, 0.5, 1)), + label2["size"], + "start", + label2["weight"], + ) + for axis, anchor, at_x in ( + (0, "end", plot["x"] - 8), + (1, "start", plot["x"] + plot["w"] + 8), + ): + if ( + axis < axis_count + and chart["yAxes"][axis].get("name") + and (axis == 0 or secondary) + ): + self.chart_text( + canvas, + chart, + at_x, + plot["y"] - 9, + chart["yAxes"][axis]["name"], + parse_color(chart["xAxisLabel"]["color"], (0.42, 0.45, 0.5, 1)), + 11, + anchor, + ) baseline = zero_y(0) canvas.line_path(plot["x"], baseline, plot["x"] + plot["w"], baseline) - canvas.paint(stroke=chart["axisLine"]["color"], - width=chart["axisLine"]["width"]) + canvas.paint( + stroke=chart["axisLine"]["color"], width=chart["axisLine"]["width"] + ) # A scatter with no categories gets a numeric x axis, like the app. if points and not categories: - span_ticks = nice_ticks(0 if low == float("inf") else min(0, low), - 1 if high == float("-inf") else high) + span_ticks = nice_ticks( + 0 if low == float("inf") else min(0, low), + 1 if high == float("-inf") else high, + ) first, last = span_ticks[0], span_ticks[-1] width = (last - first) or 1 @@ -2484,29 +2924,35 @@ def point_x(value): return plot["x"] + (value - first) / width * plot["w"] for tick in span_ticks: - self.chart_text(canvas, chart, point_x(tick), - plot["y"] + plot["h"] - + chart["xAxisLabel"]["size"] + 6, - format_number(tick), - parse_color(chart["xAxisLabel"]["color"], - (0.42, 0.45, 0.5, 1)), - chart["xAxisLabel"]["size"], "middle", - chart["xAxisLabel"]["weight"]) + self.chart_text( + canvas, + chart, + point_x(tick), + plot["y"] + plot["h"] + chart["xAxisLabel"]["size"] + 6, + format_number(tick), + parse_color(chart["xAxisLabel"]["color"], (0.42, 0.45, 0.5, 1)), + chart["xAxisLabel"]["size"], + "middle", + chart["xAxisLabel"]["weight"], + ) for index, series in enumerate(points): - color = self.series_color(chart, series, - chart["series"].index(series)) + color = self.series_color(chart, series, chart["series"].index(series)) radius = opt_num(series.get("symbolSize"), 10) / 2.0 for entry in series.get("data") or []: - px, py = (entry[0], entry[1]) \ - if isinstance(entry, list) and len(entry) >= 2 else (0, 0) + px, py = ( + (entry[0], entry[1]) + if isinstance(entry, list) and len(entry) >= 2 + else (0, 0) + ) at_x = point_x(px) - if at_x < plot["x"] - radius or \ - at_x > plot["x"] + plot["w"] + radius: + if ( + at_x < plot["x"] - radius + or at_x > plot["x"] + plot["w"] + radius + ): continue canvas.save() canvas.alpha(0.85) - canvas.ellipse_path(at_x, value_y(opt_num(py, 0)), radius, - radius) + canvas.ellipse_path(at_x, value_y(opt_num(py, 0)), radius, radius) canvas.paint(fill=color) canvas.restore() del index @@ -2517,13 +2963,17 @@ def point_x(value): for index, category in enumerate(categories): if index % stride: continue - self.chart_text(canvas, chart, plot["x"] + band * (index + 0.5), - plot["y"] + plot["h"] - + chart["xAxisLabel"]["size"] + 6, category, - parse_color(chart["xAxisLabel"]["color"], - (0.42, 0.45, 0.5, 1)), - chart["xAxisLabel"]["size"], "middle", - chart["xAxisLabel"]["weight"]) + self.chart_text( + canvas, + chart, + plot["x"] + band * (index + 0.5), + plot["y"] + plot["h"] + chart["xAxisLabel"]["size"] + 6, + category, + parse_color(chart["xAxisLabel"]["color"], (0.42, 0.45, 0.5, 1)), + chart["xAxisLabel"]["size"], + "middle", + chart["xAxisLabel"]["weight"], + ) if bars: group = band * 0.62 @@ -2531,46 +2981,62 @@ def point_x(value): for order, series in enumerate(bars): axis = axis_index(series) base = zero_y(axis) - color = self.series_color(chart, series, - chart["series"].index(series)) - item_style = series.get("itemStyle") \ - if isinstance(series.get("itemStyle"), dict) else {} + color = self.series_color(chart, series, chart["series"].index(series)) + item_style = ( + series.get("itemStyle") + if isinstance(series.get("itemStyle"), dict) + else {} + ) corner = item_style.get("borderRadius") - corner = opt_num(corner[0], 0) if isinstance(corner, list) \ + corner = ( + opt_num(corner[0], 0) + if isinstance(corner, list) else opt_num(corner, 0) + ) data = series.get("data") data = data if isinstance(data, list) else [] for index, raw in enumerate(data[:count]): value = opt_num(raw, 0) - left = plot["x"] + band * index + (band - group) / 2.0 \ - + slot * order + left = ( + plot["x"] + band * index + (band - group) / 2.0 + slot * order + ) top = value_y(value, axis) height = abs(base - top) - canvas.rect_path(left + 1, top if top <= base else base, - max(1.0, slot - 2), max(0.0, height), - min(corner, slot / 2.0)) + canvas.rect_path( + left + 1, + top if top <= base else base, + max(1.0, slot - 2), + max(0.0, height), + min(corner, slot / 2.0), + ) canvas.paint(fill=color) for series in lines: order = chart["series"].index(series) axis = axis_index(series) color = self.series_color(chart, series, order) - width = opt_num((series.get("lineStyle") or {}).get("width") - if isinstance(series.get("lineStyle"), dict) - else None, 2) + width = opt_num( + (series.get("lineStyle") or {}).get("width") + if isinstance(series.get("lineStyle"), dict) + else None, + 2, + ) data = series.get("data") data = data if isinstance(data, list) else [] values = [opt_num(value, 0) for value in data[:count]] - pts = [(plot["x"] + band * (index + 0.5), value_y(value, axis)) - for index, value in enumerate(values)] + pts = [ + (plot["x"] + band * (index + 0.5), value_y(value, axis)) + for index, value in enumerate(values) + ] if len(pts) < 2: continue path = self._line_path(pts, bool(series.get("smooth"))) area = series.get("areaStyle") if area is not None: canvas.save() - canvas.alpha(1.0 if isinstance(area, dict) - and area.get("color") else 0.25) + canvas.alpha( + 1.0 if isinstance(area, dict) and area.get("color") else 0.25 + ) self._emit(canvas, path) canvas.op("%s %s l" % (num(pts[-1][0]), num(zero_y(axis)))) canvas.op("%s %s l" % (num(pts[0][0]), num(zero_y(axis)))) @@ -2598,12 +3064,20 @@ def _line_path(self, pts, smooth): current = pts[index] following = pts[index + 1] after = pts[min(len(pts) - 1, index + 2)] - path.append(("c", - (current[0] + (following[0] - previous[0]) / 6.0, - current[1] + (following[1] - previous[1]) / 6.0), - (following[0] - (after[0] - current[0]) / 6.0, - following[1] - (after[1] - current[1]) / 6.0), - following)) + path.append( + ( + "c", + ( + current[0] + (following[0] - previous[0]) / 6.0, + current[1] + (following[1] - previous[1]) / 6.0, + ), + ( + following[0] - (after[0] - current[0]) / 6.0, + following[1] - (after[1] - current[1]) / 6.0, + ), + following, + ) + ) else: for point in pts[1:]: path.append(("l", point)) @@ -2617,8 +3091,10 @@ def _emit(self, canvas, path): canvas.op("%s %s l" % (num(segment[1][0]), num(segment[1][1]))) else: (a, b), (c, d), (e, f) = segment[1:] - canvas.op("%s %s %s %s %s %s c" - % (num(a), num(b), num(c), num(d), num(e), num(f))) + canvas.op( + "%s %s %s %s %s %s c" + % (num(a), num(b), num(c), num(d), num(e), num(f)) + ) # -- pie --------------------------------------------------------------- def draw_pie(self, canvas, chart): @@ -2628,8 +3104,12 @@ def draw_pie(self, canvas, chart): slices = [] for index, entry in enumerate(data): entry = entry if isinstance(entry, dict) else {} - slices.append((str(entry.get("name", index)), - max(0.0, opt_num(entry.get("value"), 0)))) + slices.append( + ( + str(entry.get("name", index)), + max(0.0, opt_num(entry.get("value"), 0)), + ) + ) total = sum(value for _, value in slices) or 1.0 plot = chart["grid"] cx = plot["x"] + plot["w"] / 2.0 @@ -2643,17 +3123,20 @@ def resolve(value): return float(value.strip()[:-1]) / 100.0 * limit return opt_num(value, 0) - inner, outer = resolve(pair[0]), resolve(pair[1] if len(pair) > 1 - else "70%") - item_style = series.get("itemStyle") \ - if isinstance(series.get("itemStyle"), dict) else {} + inner, outer = resolve(pair[0]), resolve(pair[1] if len(pair) > 1 else "70%") + item_style = ( + series.get("itemStyle") if isinstance(series.get("itemStyle"), dict) else {} + ) border = parse_color(item_style.get("borderColor"), (0, 0, 0, 0.0)) border_width = opt_num(item_style.get("borderWidth"), 0) label = series.get("label") formatter = None if label is not False: - formatter = label.get("formatter") if isinstance(label, dict) \ - and isinstance(label.get("formatter"), str) else "{b}" + formatter = ( + label.get("formatter") + if isinstance(label, dict) and isinstance(label.get("formatter"), str) + else "{b}" + ) angle = -math.pi / 2 for index, (name, value) in enumerate(slices): share = value / total @@ -2667,17 +3150,28 @@ def resolve(value): at_x = cx + math.cos(middle) * (outer + 12) at_y = cy + math.sin(middle) * (outer + 12) right = math.cos(middle) >= 0 - text = (formatter.replace("{b}", name) - .replace("{c}", format_number(value)) - .replace("{d}", format_number(round(share * 1000) / 10.0))) - canvas.line_path(cx + math.cos(middle) * outer, - cy + math.sin(middle) * outer, at_x, at_y) + text = ( + formatter.replace("{b}", name) + .replace("{c}", format_number(value)) + .replace("{d}", format_number(round(share * 1000) / 10.0)) + ) + canvas.line_path( + cx + math.cos(middle) * outer, + cy + math.sin(middle) * outer, + at_x, + at_y, + ) canvas.paint(stroke=color, width=1) - self.chart_text(canvas, chart, at_x + (4 if right else -4), - at_y + 4, text, - parse_color(chart["labelColor"], - (0.42, 0.45, 0.5, 1)), 12, - "start" if right else "end") + self.chart_text( + canvas, + chart, + at_x + (4 if right else -4), + at_y + 4, + text, + parse_color(chart["labelColor"], (0.42, 0.45, 0.5, 1)), + 12, + "start" if right else "end", + ) angle = end def _pie_slice(self, canvas, cx, cy, inner, outer, start, end): @@ -2688,26 +3182,38 @@ def arc(radius, from_angle, to_angle, move): k = 4.0 / 3.0 * math.tan(step / 4.0) angle = from_angle if move: - canvas.op("%s %s m" % (num(cx + math.cos(angle) * radius), - num(cy + math.sin(angle) * radius))) + canvas.op( + "%s %s m" + % ( + num(cx + math.cos(angle) * radius), + num(cy + math.sin(angle) * radius), + ) + ) for _ in range(steps): nxt = angle + step x1 = cx + math.cos(angle) * radius y1 = cy + math.sin(angle) * radius x2 = cx + math.cos(nxt) * radius y2 = cy + math.sin(nxt) * radius - canvas.op("%s %s %s %s %s %s c" - % (num(x1 - k * math.sin(angle) * radius), - num(y1 + k * math.cos(angle) * radius), - num(x2 + k * math.sin(nxt) * radius), - num(y2 - k * math.cos(nxt) * radius), - num(x2), num(y2))) + canvas.op( + "%s %s %s %s %s %s c" + % ( + num(x1 - k * math.sin(angle) * radius), + num(y1 + k * math.cos(angle) * radius), + num(x2 + k * math.sin(nxt) * radius), + num(y2 - k * math.cos(nxt) * radius), + num(x2), + num(y2), + ) + ) angle = nxt arc(outer, start, end, True) if inner > 0: - canvas.op("%s %s l" % (num(cx + math.cos(end) * inner), - num(cy + math.sin(end) * inner))) + canvas.op( + "%s %s l" + % (num(cx + math.cos(end) * inner), num(cy + math.sin(end) * inner)) + ) arc(inner, end, start, False) else: canvas.op("%s %s l" % (num(cx), num(cy))) @@ -2719,38 +3225,65 @@ def draw_legend(self, canvas, chart): pie = next((s for s in chart["series"] if s.get("type") == "pie"), None) if pie is not None: data = pie.get("data") if isinstance(pie.get("data"), list) else [] - entries = [(str((entry or {}).get("name", index)) - if isinstance(entry, dict) else str(index), - parse_color(chart["colors"][index % len(chart["colors"])])) - for index, entry in enumerate(data)] + entries = [ + ( + str((entry or {}).get("name", index)) + if isinstance(entry, dict) + else str(index), + parse_color(chart["colors"][index % len(chart["colors"])]), + ) + for index, entry in enumerate(data) + ] else: - entries = [(str(series.get("name") or "Series %d" % (index + 1)), - self.series_color(chart, series, index)) - for index, series in enumerate(chart["series"])] + entries = [ + ( + str(series.get("name") or "Series %d" % (index + 1)), + self.series_color(chart, series, index), + ) + for index, series in enumerate(chart["series"]) + ] if not entries: return face = face_name(chart["font"], is_bold(legend["weight"]), False) - widths = [legend["itemWidth"] + 8 - + text_width(self.encoder, name, face, legend["size"]) - for name, _ in entries] + widths = [ + legend["itemWidth"] + + 8 + + text_width(self.encoder, name, face, legend["size"]) + for name, _ in entries + ] span = sum(widths) + legend["itemGap"] * max(0, len(entries) - 1) left = max(8.0, (chart["w"] - span) / 2.0) height = max(legend["itemHeight"], legend["size"]) - top = legend["top"] if legend["top"] is not None \ + top = ( + legend["top"] + if legend["top"] is not None else chart["h"] - legend["bottom"] - height - 10 + ) swatch_y = top + (height - legend["itemHeight"]) / 2.0 baseline = top + height / 2.0 + legend["size"] * 0.35 for index, (name, color) in enumerate(entries): - canvas.rect_path(left, swatch_y, legend["itemWidth"], - legend["itemHeight"], - min(3.0, legend["itemHeight"] / 2.0)) + canvas.rect_path( + left, + swatch_y, + legend["itemWidth"], + legend["itemHeight"], + min(3.0, legend["itemHeight"] / 2.0), + ) canvas.paint(fill=color) - self.chart_text(canvas, chart, left + legend["itemWidth"] + 8, - baseline, name, - parse_color(legend["color"], (0.42, 0.45, 0.5, 1)), - legend["size"], "start", legend["weight"]) - left += widths[index] + (legend["itemGap"] - if index < len(entries) - 1 else 0) + self.chart_text( + canvas, + chart, + left + legend["itemWidth"] + 8, + baseline, + name, + parse_color(legend["color"], (0.42, 0.45, 0.5, 1)), + legend["size"], + "start", + legend["weight"], + ) + left += widths[index] + ( + legend["itemGap"] if index < len(entries) - 1 else 0 + ) class DeckRenderer(Renderer, ShapeMixin, TableMixin, ChartMixin): diff --git a/internal/mcp/testdata/dummy_server.py b/internal/mcp/testdata/dummy_server.py index 74c1e114..76785fe8 100644 --- a/internal/mcp/testdata/dummy_server.py +++ b/internal/mcp/testdata/dummy_server.py @@ -4,6 +4,7 @@ # Ensure unbuffered output sys.stdout.reconfigure(line_buffering=True) + def main(): while True: try: @@ -25,7 +26,7 @@ def main(): response["result"] = { "protocolVersion": "2024-11-05", "serverInfo": {"name": "dummy", "version": "1.0"}, - "capabilities": {} + "capabilities": {}, } elif method == "tools/list": response["result"] = { @@ -35,10 +36,8 @@ def main(): "description": "Echoes input", "inputSchema": { "type": "object", - "properties": { - "message": {"type": "string"} - } - } + "properties": {"message": {"type": "string"}}, + }, } ] } @@ -49,10 +48,7 @@ def main(): if tool_name == "echo": response["result"] = { "content": [ - { - "type": "text", - "text": f"Echo: {args.get('message')}" - } + {"type": "text", "text": f"Echo: {args.get('message')}"} ] } else: @@ -66,5 +62,6 @@ def main(): sys.stderr.write(str(e) + "\n") break + if __name__ == "__main__": main() diff --git a/internal/mcp/testdata/noisy_server.py b/internal/mcp/testdata/noisy_server.py index 5539452d..d7b7f642 100644 --- a/internal/mcp/testdata/noisy_server.py +++ b/internal/mcp/testdata/noisy_server.py @@ -22,7 +22,15 @@ def main() -> None: # 1. Stray non-JSON output (e.g. a library print). print("WARNING: something logged straight to stdout") # 2. Server-initiated notification. - print(json.dumps({"jsonrpc": "2.0", "method": "notifications/progress", "params": {"progress": 1}})) + print( + json.dumps( + { + "jsonrpc": "2.0", + "method": "notifications/progress", + "params": {"progress": 1}, + } + ) + ) # 3. Stale response to a request id that is not ours. print(json.dumps({"jsonrpc": "2.0", "id": 999999, "result": {"echoed": -1}})) # 4. The real response. diff --git a/internal/sandbox/fileops.py b/internal/sandbox/fileops.py index c7e00688..8a8fafe2 100644 --- a/internal/sandbox/fileops.py +++ b/internal/sandbox/fileops.py @@ -35,8 +35,14 @@ class StaleContent(Exception): def fail(kind, msg, **extra): - response = {"ok": False, "err_kind": kind, "err": msg, - "data_b64": "", "size": 0, "count": 0} + response = { + "ok": False, + "err_kind": kind, + "err": msg, + "data_b64": "", + "size": 0, + "count": 0, + } response.update(extra) return response @@ -79,13 +85,13 @@ def _open_dir_at(parent_fd, name, create): return fd except OSError as exc: if exc.errno in (errno.ELOOP, errno.ENOTDIR): - raise UnsafePath( - "directory component changed or is a symlink") from exc + raise UnsafePath("directory component changed or is a symlink") from exc raise except OSError as exc: if exc.errno in (errno.ELOOP, errno.ENOTDIR): raise UnsafePath( - "directory component is a symlink or not a directory") from exc + "directory component is a symlink or not a directory" + ) from exc raise @@ -151,9 +157,16 @@ def do_bind_root(req): try: root_fd = _open_root(req) info = os.fstat(root_fd) - return {"ok": True, "err_kind": "", "err": "", "data_b64": "", - "size": 0, "count": 0, "dev": info.st_dev, - "ino": info.st_ino} + return { + "ok": True, + "err_kind": "", + "err": "", + "data_b64": "", + "size": 0, + "count": 0, + "dev": info.st_dev, + "ino": info.st_ino, + } except UnsafePath as exc: return fail("unsafe_path", str(exc)) except OSError as exc: @@ -169,12 +182,18 @@ def _test_pause(req, parent_fd): if pause_ms <= 0: return name = req.get("test_ready_name", "") - if (not isinstance(name, str) or - not name.startswith(".fleet-fileop-test-") or - os.path.basename(name) != name): + if ( + not isinstance(name, str) + or not name.startswith(".fleet-fileop-test-") + or os.path.basename(name) != name + ): raise UnsafePath("invalid test rendezvous name") - fd = os.open(name, os.O_WRONLY | os.O_CREAT | os.O_EXCL | - os.O_NOFOLLOW | os.O_CLOEXEC, 0o600, dir_fd=parent_fd) + fd = os.open( + name, + os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW | os.O_CLOEXEC, + 0o600, + dir_fd=parent_fd, + ) os.close(fd) os.fsync(parent_fd) time.sleep(min(pause_ms, 10000) / 1000.0) @@ -182,8 +201,7 @@ def _test_pause(req, parent_fd): def _open_file(parent_fd, name): try: - fd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC, - dir_fd=parent_fd) + fd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC, dir_fd=parent_fd) except FileNotFoundError: raise except IsADirectoryError: @@ -212,8 +230,11 @@ def _lstat_at(parent_fd, name): def _same_file(left, right): - return (left is not None and right is not None and - (left.st_dev, left.st_ino) == (right.st_dev, right.st_ino)) + return ( + left is not None + and right is not None + and (left.st_dev, left.st_ino) == (right.st_dev, right.st_ino) + ) def _sha256_open_at(parent_fd, name): @@ -227,9 +248,12 @@ def _sha256_open_at(parent_fd, name): break digest.update(chunk) after = os.fstat(fd) - if ((info.st_dev, info.st_ino, info.st_size, info.st_mtime_ns) != - (after.st_dev, after.st_ino, after.st_size, - after.st_mtime_ns)): + if (info.st_dev, info.st_ino, info.st_size, info.st_mtime_ns) != ( + after.st_dev, + after.st_ino, + after.st_size, + after.st_mtime_ns, + ): raise StaleContent("file changed while its hash was computed") return digest.hexdigest() finally: @@ -237,8 +261,7 @@ def _sha256_open_at(parent_fd, name): os.close(fd) -def _atomic_write(parent_fd, name, data, expected=None, - expected_digest=None): +def _atomic_write(parent_fd, name, data, expected=None, expected_digest=None): before = _lstat_at(parent_fd, name) if expected is not None and not _same_file(before, expected): raise UnsafePath("file changed while it was being edited") @@ -248,8 +271,12 @@ def _atomic_write(parent_fd, name, data, expected=None, tmp = ".fleet-fileop-%s" % secrets.token_hex(12) fd = -1 try: - fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL | - os.O_NOFOLLOW | os.O_CLOEXEC, 0o600, dir_fd=parent_fd) + fd = os.open( + tmp, + os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW | os.O_CLOEXEC, + 0o600, + dir_fd=parent_fd, + ) view = memoryview(data) while view: written = os.write(fd, view) @@ -283,8 +310,7 @@ def _atomic_write(parent_fd, name, data, expected=None, if expected_digest is not None: current_digest = _sha256_open_at(parent_fd, name) if current_digest != expected_digest: - raise StaleContent( - "file content changed before the edit was committed") + raise StaleContent("file content changed before the edit was committed") os.replace(tmp, name, src_dir_fd=parent_fd, dst_dir_fd=parent_fd) os.fsync(parent_fd) except BaseException: @@ -313,7 +339,7 @@ def _hash_and_window(fd, offset, limit): wanted_start = max(position, offset) wanted_end = chunk_end if end is None else min(chunk_end, end) if wanted_start < wanted_end: - chunks.append(chunk[wanted_start - position:wanted_end - position]) + chunks.append(chunk[wanted_start - position : wanted_end - position]) position = chunk_end return digest.hexdigest(), b"".join(chunks) @@ -329,14 +355,22 @@ def do_read(req): limit = int(req.get("limit", 0) or 0) digest, data = _hash_and_window(fd, offset, limit) after = os.fstat(fd) - if ((before.st_dev, before.st_ino, before.st_size, - before.st_mtime_ns) != - (after.st_dev, after.st_ino, after.st_size, - after.st_mtime_ns)): + if (before.st_dev, before.st_ino, before.st_size, before.st_mtime_ns) != ( + after.st_dev, + after.st_ino, + after.st_size, + after.st_mtime_ns, + ): raise StaleContent("file changed while it was being read") - return {"ok": True, "err_kind": "", "err": "", - "data_b64": base64.b64encode(data).decode("ascii"), - "size": before.st_size, "count": 0, "sha256": digest} + return { + "ok": True, + "err_kind": "", + "err": "", + "data_b64": base64.b64encode(data).decode("ascii"), + "size": before.st_size, + "count": 0, + "sha256": digest, + } except FileNotFoundError: return fail("not_found", "file not found") except IsADirectoryError: @@ -361,9 +395,15 @@ def do_write(req): _test_pause(req, parent_fd) data = base64.b64decode(req.get("data_b64", "")) _atomic_write(parent_fd, name, data) - return {"ok": True, "err_kind": "", "err": "", "data_b64": "", - "size": len(data), "count": 0, - "sha256": hashlib.sha256(data).hexdigest()} + return { + "ok": True, + "err_kind": "", + "err": "", + "data_b64": "", + "size": len(data), + "count": 0, + "sha256": hashlib.sha256(data).hexdigest(), + } except IsADirectoryError: return fail("is_dir", "path is a directory") except UnsafePath as exc: @@ -379,13 +419,21 @@ def _bounded_diff(path, before, after): old_lines = before.decode("utf-8", "replace").splitlines(keepends=True) new_lines = after.decode("utf-8", "replace").splitlines(keepends=True) name = os.path.basename(path) - lines = list(difflib.unified_diff( - old_lines, new_lines, fromfile=name + " (before)", - tofile=name + " (after)", n=3)) - added = sum(1 for line in lines - if line.startswith("+") and not line.startswith("+++")) - removed = sum(1 for line in lines - if line.startswith("-") and not line.startswith("---")) + lines = list( + difflib.unified_diff( + old_lines, + new_lines, + fromfile=name + " (before)", + tofile=name + " (after)", + n=3, + ) + ) + added = sum( + 1 for line in lines if line.startswith("+") and not line.startswith("+++") + ) + removed = sum( + 1 for line in lines if line.startswith("-") and not line.startswith("---") + ) text = "".join(lines) encoded = text.encode("utf-8") if len(encoded) > DIFF_MAX_BYTES: @@ -396,9 +444,11 @@ def _bounded_diff(path, before, after): def _stale(expected, current): return fail( - "stale", "file content has changed since it was last read " - "(expected sha256 %s, current %s); re-read the file and retry" % - (expected, current)) + "stale", + "file content has changed since it was last read " + "(expected sha256 %s, current %s); re-read the file and retry" + % (expected, current), + ) def do_edit(req): @@ -415,7 +465,7 @@ def do_edit(req): old_digest = hashlib.sha256(content).hexdigest() expected = (req.get("expected_sha256") or "").strip().lower() if expected.startswith("sha256:"): - expected = expected[len("sha256:"):] + expected = expected[len("sha256:") :] if expected and expected != old_digest: return _stale(expected, old_digest) @@ -424,20 +474,25 @@ def do_edit(req): count = content.count(old) if count == 0: response = fail("old_absent", "old_text not found in file") - if (b"\r\n" in content and old and - old.replace(b"\r\n", b"\n") in - content.replace(b"\r\n", b"\n")): + if ( + b"\r\n" in content + and old + and old.replace(b"\r\n", b"\n") in content.replace(b"\r\n", b"\n") + ): response["hint"] = ( "the file uses CRLF line endings — include \\r\\n in " "old_text exactly as view_file returned it, or re-read " - "the file") + "the file" + ) return response if count > 1 and not req.get("replace_all"): return fail( - "ambiguous", "old_text matches %d locations; edit_file " + "ambiguous", + "old_text matches %d locations; edit_file " "replaces exactly one — add surrounding context to make the " "match unique, or set replace_all=true" % count, - match_count=count) + match_count=count, + ) if req.get("replace_all"): updated = content.replace(old, new) @@ -446,12 +501,18 @@ def do_edit(req): count = 1 if updated == content: return fail( - "noop", "edit is a no-op (old_text and new_text produce " - "identical content)") + "noop", + "edit is a no-op (old_text and new_text produce identical content)", + ) try: - _atomic_write(parent_fd, name, updated, expected=info, - expected_digest=old_digest if expected else None) + _atomic_write( + parent_fd, + name, + updated, + expected=info, + expected_digest=old_digest if expected else None, + ) except StaleContent: try: current = _sha256_open_at(parent_fd, name) @@ -460,11 +521,20 @@ def do_edit(req): return _stale(expected, current) diff, added, removed = _bounded_diff(req["path"], content, updated) - return {"ok": True, "err_kind": "", "err": "", "data_b64": "", - "size": len(updated), "count": count, - "sha256": hashlib.sha256(updated).hexdigest(), - "old_sha256": old_digest, "match_count": count, - "added": added, "removed": removed, "diff": diff} + return { + "ok": True, + "err_kind": "", + "err": "", + "data_b64": "", + "size": len(updated), + "count": count, + "sha256": hashlib.sha256(updated).hexdigest(), + "old_sha256": old_digest, + "match_count": count, + "added": added, + "removed": removed, + "diff": diff, + } except FileNotFoundError: return fail("not_found", "file not found") except IsADirectoryError: @@ -486,8 +556,12 @@ def main(): except Exception as exc: print(json.dumps(fail("", "bad request: %s" % exc))) return - handler = {"read": do_read, "write": do_write, "edit": do_edit, - "bind_root": do_bind_root}.get(req.get("op")) + handler = { + "read": do_read, + "write": do_write, + "edit": do_edit, + "bind_root": do_bind_root, + }.get(req.get("op")) if handler is None: print(json.dumps(fail("", "unknown op: %r" % req.get("op")))) return diff --git a/internal/tools/python_bridge.py b/internal/tools/python_bridge.py index 77bca45c..45aacf4d 100644 --- a/internal/tools/python_bridge.py +++ b/internal/tools/python_bridge.py @@ -90,7 +90,9 @@ def write_figures(images, base_dir): try: # O_EXCL | O_NOFOLLOW: refuse to follow or overwrite a name a prior # cell's code might have pre-planted in the (writable) workspace. - fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + fd = os.open( + path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600 + ) try: os.write(fd, raw) finally: @@ -162,7 +164,9 @@ def start_kernel(): # tempfile.mkstemp creates a unique empty file and returns (fd, path). # We close the fd immediately since ipykernel will overwrite the # content — we only needed the unique name. - fd, connection_file = tempfile.mkstemp(prefix=f"kernel-{os.getpid()}-", suffix=".json") + fd, connection_file = tempfile.mkstemp( + prefix=f"kernel-{os.getpid()}-", suffix=".json" + ) os.close(fd) # ipykernel creates the file itself with its own content. Our mkstemp # empty placeholder would make the while-loop below think the file @@ -174,18 +178,14 @@ def start_kernel(): pass # Start the kernel - cmd = [ - sys.executable, - "-m", "ipykernel_launcher", - "-f", connection_file - ] + cmd = [sys.executable, "-m", "ipykernel_launcher", "-f", connection_file] # Start process detached to avoid signal interference kernel_process = subprocess.Popen( cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, - start_new_session=True + start_new_session=True, ) # Wait for connection file to exist @@ -198,6 +198,7 @@ def start_kernel(): return connection_file + def cleanup(): """Kills the kernel and deletes the connection file.""" global kernel_process, connection_file @@ -305,45 +306,51 @@ def run_code_on_kernel(code, client, timeout_seconds=None): while time.time() < deadline: if not shell_reply_seen: try: - shell_msg = client.get_shell_msg(timeout=min(0.05, remaining_time(deadline))) + shell_msg = client.get_shell_msg( + timeout=min(0.05, remaining_time(deadline)) + ) if shell_msg["parent_header"].get("msg_id") == msg_id: shell_reply_seen = True shell_content = shell_msg.get("content", {}) if shell_content.get("status") == "error": traceback = shell_content.get("traceback", []) - error_content = strip_ansi("\n".join(traceback)) or error_content + error_content = ( + strip_ansi("\n".join(traceback)) or error_content + ) status = "error" except queue.Empty: pass try: # Get IOPub messages (streams, display_data, etc) - msg = client.get_iopub_msg(timeout=min(IOPUB_POLL_SECONDS, remaining_time(deadline))) - msg_type = msg['header']['msg_type'] - content = msg['content'] + msg = client.get_iopub_msg( + timeout=min(IOPUB_POLL_SECONDS, remaining_time(deadline)) + ) + msg_type = msg["header"]["msg_type"] + content = msg["content"] - if msg['parent_header'].get('msg_id') != msg_id: + if msg["parent_header"].get("msg_id") != msg_id: continue - if msg_type == 'stream': - if content['name'] == 'stdout': - stdout_content.append(content['text']) - elif content['name'] == 'stderr': - stderr_content.append(content['text']) - elif msg_type == 'execute_result': - data = content.get('data', {}) - result_content.append(data.get('text/plain', '')) + if msg_type == "stream": + if content["name"] == "stdout": + stdout_content.append(content["text"]) + elif content["name"] == "stderr": + stderr_content.append(content["text"]) + elif msg_type == "execute_result": + data = content.get("data", {}) + result_content.append(data.get("text/plain", "")) collect_image(data, images) - elif msg_type == 'display_data': - data = content.get('data', {}) - result_content.append(data.get('text/plain', '')) + elif msg_type == "display_data": + data = content.get("data", {}) + result_content.append(data.get("text/plain", "")) collect_image(data, images) - elif msg_type == 'error': - traceback = content.get('traceback', []) - error_content = strip_ansi('\n'.join(traceback)) + elif msg_type == "error": + traceback = content.get("traceback", []) + error_content = strip_ansi("\n".join(traceback)) status = "error" - elif msg_type == 'status': - if content['execution_state'] == 'idle': + elif msg_type == "status": + if content["execution_state"] == "idle": idle_seen = True except queue.Empty: @@ -378,6 +385,7 @@ def run_code_on_kernel(code, client, timeout_seconds=None): "images": images, } + # Patterns that indicate agent confusion about MCP tools MCP_IMPORT_PATTERNS = [ "from mcp", @@ -392,6 +400,7 @@ def run_code_on_kernel(code, client, timeout_seconds=None): "import internal.tools", ] + def check_mcp_confusion(code): """Check if code appears to be trying to import MCP tools incorrectly.""" code_lower = code.lower() @@ -466,7 +475,9 @@ def dump_json_line(value): _kernel_cwd = None # last cwd applied inside the kernel process -def execute_code(code, return_vars=None, timeout_seconds=None, workspace_dir=None, reset_kernel=False): +def execute_code( + code, return_vars=None, timeout_seconds=None, workspace_dir=None, reset_kernel=False +): """Executes code on the kernel and returns the result.""" global client, _kernel_cwd @@ -492,7 +503,7 @@ def execute_code(code, return_vars=None, timeout_seconds=None, workspace_dir=Non return { "status": "error", "output": f"Failed to start kernel: {str(e)}", - "error": str(e) + "error": str(e), } # Apply per-conversation workspace cwd INSIDE the kernel process. @@ -502,10 +513,7 @@ def execute_code(code, return_vars=None, timeout_seconds=None, workspace_dir=Non if workspace_dir and workspace_dir != _kernel_cwd: try: escaped = workspace_dir.replace("\\", "\\\\").replace("'", "\\'") - chdir_code = ( - "import os as _os\n" - f"_os.chdir('{escaped}')\n" - ) + chdir_code = f"import os as _os\n_os.chdir('{escaped}')\n" chdir_res = run_code_on_kernel(chdir_code, client) if chdir_res["status"] == "success": _kernel_cwd = workspace_dir @@ -629,7 +637,7 @@ def execute_code(code, return_vars=None, timeout_seconds=None, workspace_dir=Non return { "status": res["status"], - "output": final_output.strip(), # Legacy field + "output": final_output.strip(), # Legacy field "stdout": stdout_with_warning, "stderr": res["stderr"], "vars": normalize_json_value(vars_data), @@ -642,9 +650,10 @@ def execute_code(code, return_vars=None, timeout_seconds=None, workspace_dir=Non return { "status": "error", "output": f"Execution error: {str(e)}", - "error": str(e) + "error": str(e), } + def main(): # Cleanup on SIGTERM/SIGINT (Go side's terminateBridge) AND on normal # exit (SystemExit, unhandled EOF on stdin). atexit fires even for @@ -681,18 +690,28 @@ def main(): except OSError as e: sys.stderr.write(f"workspace_dir chdir failed: {e}\n") - result = execute_code(code, return_vars, timeout_seconds=timeout_seconds, workspace_dir=workspace_dir, reset_kernel=reset_kernel) + result = execute_code( + code, + return_vars, + timeout_seconds=timeout_seconds, + workspace_dir=workspace_dir, + reset_kernel=reset_kernel, + ) # Print result as JSON on one line print(dump_json_line(result), flush=True) except json.JSONDecodeError: - print(dump_json_line({"status": "error", "output": "Invalid JSON input"}), flush=True) + print( + dump_json_line({"status": "error", "output": "Invalid JSON input"}), + flush=True, + ) except KeyboardInterrupt: pass finally: cleanup() + if __name__ == "__main__": main() diff --git a/scripts/generate-icons.py b/scripts/generate-icons.py index 8199a0a7..57de4d06 100755 --- a/scripts/generate-icons.py +++ b/scripts/generate-icons.py @@ -8,6 +8,7 @@ Outputs: web/src/app/ favicon.ico, icon.svg, apple-icon.png web/public/app-icons/ favicon-16/32, icon-192/512, maskable-icon-512 """ + import io from pathlib import Path From db43ec8ad04c9d8a1146b19befdb90f836c6f0d8 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 14:06:25 +0000 Subject: [PATCH 11/34] Gate the scanners through ci-gate/Dev gate, audit npm deps, enforce formatting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the three things still standing between "the scanners run" and "the scanners are load-bearing", plus every finding doing so surfaced. Nothing is deferred. GATING, CORRECTED. The earlier design note claimed making CodeQL/Semgrep merge-blocking needed a branch-protection click, reasoning from "`needs` cannot cross workflow files". Incomplete: codeql.yml and semgrep.yml are now REUSABLE workflows (`on: workflow_call`) that ci.yml and dev-ci.yml call as jobs, and a job that calls a reusable workflow sits in a gate's `needs` like any other job. So `CI gate` — already the single required check on main — and `Dev gate` now block on scanner findings with no settings change anywhere. Their own push/pull_request triggers are removed so nothing runs twice; each keeps its weekly re-scan cron (new queries/rules against unchanged code) plus a workflow_dispatch. In ci.yml the calls are docs-only-skippable like the other heavy jobs (a docs-only change cannot touch scanned code, and the gate treats a skip as a pass); in dev-ci they are unconditional because Dev gate demands strict success and the fast lane has no docs-only detection — which also means direct pushes to dev get scanned. NPM AUDIT, NEW BLOCKING GATE. `npm audit --audit-level=low` runs in both web jobs, lockfile-only, before the expensive `npm ci`, failing on any severity — the npm counterpart of the govulncheck gate, clock-dependent by design. web/ was already clean (0 vulns). scripts/rampart-service HAD NO LOCKFILE AT ALL, so nothing could audit it and installs were unreproducible; generating one exposed 5 high-severity vulnerabilities it had been hiding: sharp <0.35.0 (libvips CVE-2026-33327/-33328/-35590/-35591) and adm-zip <0.6.0 (GHSA-xcpc-8h2w-3j85) via onnxruntime-node. No upstream release fixes either — the latest @huggingface/transformers still pins sharp ^0.34.5, and npm's suggested "fix" was a BREAKING DOWNGRADE of transformers to 3.8.1 — so package.json carries two overrides (sharp ^0.35.3, adm-zip ^0.6.0; each the release immediately after its vulnerable line). The overridden stack was installed and load-tested, not just resolved: sharp renders a PNG through the new libvips, transformers loads on it, rampart exports its API, adm-zip 0.6 round-trips a zip. Both trees now audit at 0. Drop the overrides when upstream ships fixed ranges. RUFF FORMAT, NOW A GATE. `ruff format --check` blocks in both CI python jobs and in `make lint` (lint-python). Safe because the previous commit formatted the whole tree, so the gate starts clean and a failure means one new file. ALL THREE SEMGREP PARSE ERRORS FIXED, so no file is partially covered — a partial parse silently drops rules from a file, which is coverage loss wearing a green check: - build-sandbox-image.yml interpolated ${{ steps.build.outcome }} into its run: script; now passed via env (also the script-injection-safe form — the same fix codeql.yml got for $RUNNER_TEMP). - The same script's ${tag:-(tag unavailable)} expansion default is valid bash, but the bare paren chokes semgrep's bash sub-parser; hoisted to a plain `if [ -z "$tag" ]` assignment. - fixtures.ts used an inline `import("@playwright/test")` type; now a named `import type { BrowserContext }`. Validated with the real web toolchain: npm ci, oxlint, tsc --noEmit, and all 1104 vitest tests pass. Final measured state, whole tree: semgrep --error over all four packs exits 0 with 0 findings, 898 files scanned, 0 parse errors; both npm trees 0 vulnerabilities; ruff check + format --check clean; gofmt clean; make build, make lint, make test (exit 0), make lint-migrations all green. Signed-off-by: Brad Flaugher --- .github/workflows/build-sandbox-image.yml | 16 +- .github/workflows/ci.yml | 60 +- .github/workflows/codeql.yml | 54 +- .github/workflows/dev-ci.yml | 43 +- .github/workflows/semgrep.yml | 15 +- AGENTS.md | 14 +- CHANGELOG.md | 42 + Makefile | 2 +- docs/CODEQL.md | 23 +- docs/SCANNING.md | 122 ++- ruff.toml | 6 + scripts/rampart-service/package-lock.json | 1042 +++++++++++++++++++++ scripts/rampart-service/package.json | 4 + web/e2e/live/fixtures.ts | 3 +- 14 files changed, 1322 insertions(+), 124 deletions(-) create mode 100644 scripts/rampart-service/package-lock.json diff --git a/.github/workflows/build-sandbox-image.yml b/.github/workflows/build-sandbox-image.yml index 547494f9..e4440218 100644 --- a/.github/workflows/build-sandbox-image.yml +++ b/.github/workflows/build-sandbox-image.yml @@ -130,14 +130,26 @@ jobs: - name: Report what was built if: ${{ always() && steps.build.outcome != 'skipped' }} + env: + # The step outcome comes in through env, not interpolated into the + # script: a ${{ }} expression expanded inside a run: block is the + # script-injection shape semgrep's github-actions rules flag, and the + # inline form also broke their bash sub-parser, silently costing this + # file rule coverage (same fix as codeql.yml's $RUNNER_TEMP). + BUILD_OUTCOME: ${{ steps.build.outcome }} run: | tag="$(FLEET_CLIENT_CONFIG_DIR="$GITHUB_WORKSPACE/$BUNDLE_DIR" \ bash .fleet-core/scripts/build-sandbox-image.sh --print-tag 2>/dev/null || true)" - if [ "${{ steps.build.outcome }}" = "success" ]; then + # Plain assignment instead of a ${tag:-(…)} expansion default: the + # parenthesis inside the default value is valid bash but chokes + # semgrep's bash sub-parser, which partial-parsed this file and + # silently dropped two rules' coverage of it. + if [ -z "$tag" ]; then tag="(tag unavailable)"; fi + if [ "$BUILD_OUTCOME" = "success" ]; then { echo "### Sandbox builds clean" echo - echo "\`${tag:-(tag unavailable)}\` built from \`$BUNDLE_DIR/sandbox/Containerfile\`." + echo "\`$tag\` built from \`$BUNDLE_DIR/sandbox/Containerfile\`." echo echo "Nothing was pushed — this is a build canary. The base tracks" echo "\`fedora-minimal:latest\`, so this is the check that a Fedora" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6f9c9c60..7263bf7b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -352,19 +352,34 @@ jobs: # so a local `ruff check .` and this gate cannot disagree. run: ruff check --output-format github . - - name: Formatting check (report only) - # NOT a gate. The tree has never been ruff-formatted, so failing on it - # would block every PR on a whole-tree reformat nobody has scheduled. - # Reported so the size of that decision stays visible instead of unknown. - if: ${{ !cancelled() }} - run: | - set -uo pipefail - { - echo '### ruff format (advisory — not a gate)' - echo '```' - ruff format --check --diff . 2>&1 | tail -40 || true - echo '```' - } | tee -a "$GITHUB_STEP_SUMMARY" + - name: Formatting check + # A gate, and safe to be one: the whole tree was ruff-formatted in the + # change that flipped this, so a failure here is a NEW unformatted file, + # fixable with one `ruff format .`. + run: ruff format --check . + + codeql: + # Reusable-workflow call: brings codeql.yml's jobs into THIS workflow's + # graph so `CI gate` (the single required check on main) blocks on them. + # Its `Fail on findings` step means a finding — not just a broken scanner — + # fails this job and therefore the gate. + needs: changes + # Docs-only changes cannot touch Go/JS/Python/workflow code (the allowlist + # is *.md, docs/, LICENSE); the gate treats the skip as a pass. + if: ${{ needs.changes.outputs.docs_only != 'true' }} + permissions: + contents: read + security-events: write # the called workflow uploads SARIF to code scanning + actions: read + uses: ./.github/workflows/codeql.yml + + semgrep: + # Same mechanism as codeql above: called here so `CI gate` blocks on it. + needs: changes + if: ${{ needs.changes.outputs.docs_only != 'true' }} + permissions: + contents: read + uses: ./.github/workflows/semgrep.yml web: name: Web lint / test / build @@ -388,6 +403,23 @@ jobs: cache: npm cache-dependency-path: web/package-lock.json + - name: Audit dependencies for known CVEs + # The npm counterpart of the Go job's govulncheck gate: reads + # package-lock.json against the npm advisory DB and fails on ANY + # severity. Like govulncheck, the verdict is a function of the clock as + # well as the commit — a new advisory can redden an unchanged tree, and + # that is the point. Needs no node_modules (lockfile-only), so it runs + # before the expensive install and fails fast. + run: npm audit --audit-level=low + + - name: Audit rampart-service dependencies for known CVEs + # Separate tree, same gate. Its lockfile pins `overrides` forcing + # sharp >=0.35 (libvips CVEs) and adm-zip >=0.6 (GHSA-xcpc-8h2w-3j85) + # because no release of @huggingface/transformers/onnxruntime-node has + # picked the fixes up yet — see scripts/rampart-service/package.json. + working-directory: scripts/rampart-service + run: npm audit --audit-level=low + - name: Install dependencies run: npm ci @@ -765,7 +797,7 @@ jobs: # allowed (docs-only), but any failure or cancellation fails the gate. name: CI gate if: ${{ always() }} - needs: [changes, gitleaks, migrations, go, python, web, playwright, e2e-live, grype-scan] + needs: [changes, gitleaks, migrations, go, python, codeql, semgrep, web, playwright, e2e-live, grype-scan] runs-on: ubuntu-latest steps: - name: Require all upstream jobs to have succeeded or been skipped diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index c7eb8410..e140728e 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -63,17 +63,20 @@ # Grype step, govulncheck-scheduled.yml, grype-scheduled.yml). Those push their # own SARIF to the Security tab and never depended on CodeQL being configured. # -# Merge gating, in two parts — they are different things and conflating them is -# how "we gate on CodeQL" ends up meaning nothing: +# Merge gating, in two parts — both now closed: # # 1. Does a finding turn the CHECK red? YES. The `Fail on findings` step below # fails the job on any finding. Without it the analyze step exits 0 whether # it found nothing or a hundred alerts, so a red check could only ever mean # "the scanner broke". -# 2. Does a red check BLOCK a merge? NOT YET. `ci-gate` is still the only -# required status check on main (see .github/CODEOWNERS). Requiring a check -# is a repo-settings action that a workflow file cannot perform — add -# `CodeQL gate` to the ruleset to close this half. +# 2. Does a red check BLOCK a merge? YES, through the EXISTING required check. +# This is a REUSABLE workflow (`on: workflow_call`): ci.yml and dev-ci.yml +# each call it as a job, and a job that calls a reusable workflow can sit in +# a gate job's `needs` like any other job — which routes a CodeQL failure +# into `CI gate` / `Dev gate` with no branch-protection change at all. The +# earlier design note saying this required a repo-settings click was wrong: +# `needs` cannot cross workflow FILES, but a workflow_call brings the jobs +# into the caller's file. # # They cannot be part of `CI gate`: a job's `needs` cannot reach across workflow # files. So this file carries its own aggregate `CodeQL gate` job at the bottom, @@ -86,17 +89,13 @@ name: CodeQL on: - push: - branches: [main] - pull_request: - # main mirrors ci.yml. dev is DELIBERATELY added, and is the one place this - # file covers more than default setup did (verified: zero CodeQL runs ever - # recorded on a dev-targeting PR — see docs/CODEQL.md). Every change lands on - # dev first; main only ever receives a promote merge. Scanning main alone - # means a finding surfaces for the first time on a squashed promote commit, - # which is the same "the integration branch is where it is first attempted" - # complaint dev-ci.yml already makes about compilation. - branches: [main, dev] + # No push/pull_request triggers of its own: per-change runs come from ci.yml + # (push/PR on main) and dev-ci.yml (push/PR on dev) calling this workflow, so + # every branch event is covered exactly once and the result feeds the gates. + # Scanning dev PRs at all is the one place this exceeds old default setup, + # which never ran on them (verified: zero CodeQL runs recorded on dev PRs). + workflow_call: + workflow_dispatch: # manual re-run (e.g. after dismissing an alert) schedule: # Monday 10:00 UTC, weekly. Offset from the 07:00 canary, the 08:00 daily # govulncheck and the Monday 09:00 Grype scan, following the same @@ -116,13 +115,6 @@ permissions: # needed because no analysis here pulls a private CodeQL pack. actions: read -concurrency: - group: codeql-${{ github.event_name }}-${{ github.ref }} - # Cancel superseded PR runs, but never a push-on-main or scheduled run: those - # produce the alert set of record for the default branch, and a cancelled run - # leaves the previous, staler alerts standing. - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - jobs: analyze: name: Analyze (${{ matrix.language }}) @@ -302,16 +294,10 @@ jobs: codeql-gate: name: CodeQL gate - # Aggregate check for branch protection, same pattern and rationale as - # ci.yml's `CI gate` and dev-ci.yml's `Dev gate`: require this ONE check - # rather than naming each `Analyze ()` leg individually, so adding - # or removing a language from the matrix above does not silently leave a - # required check that never reports again. - # - # This job existing does NOT make CodeQL blocking. `ci-gate` is still the - # only required status check on main; this is the single lever to flip if - # that changes, and it deliberately cannot be flipped from this file. See - # docs/CODEQL.md ("Merge gating"). + # Aggregate of the matrix. In the workflow_call path the caller's + # `needs: codeql` already rolls up every job here, so this exists for the + # standalone schedule/dispatch runs — one legible verdict per weekly re-scan + # instead of four boxes — and as a stable single check name. # # `needs: [analyze]` on a matrix job collapses to one aggregate result: # success only when every leg succeeded. With `fail-fast: false` above, diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index 4d90776d..0970f868 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -22,10 +22,10 @@ # labour is "does it compile, lint, and pass tests" here; "is it safe to ship" # there. # -# CodeQL used to be on that deferred list and no longer is. It does not run in -# this file — it has its own workflow, .github/workflows/codeql.yml, whose -# `pull_request` trigger covers dev as well as main. So a PR into dev IS -# CodeQL-scanned; it just is not scanned by this lane. See docs/CODEQL.md. +# CodeQL and Semgrep used to be on that deferred list and no longer are: both +# run IN this lane, as reusable-workflow calls wired into `Dev gate`, so a +# scanner finding blocks dev the same way a compile error does. See +# docs/SCANNING.md. # # `Dev gate` is the single aggregate job (same pattern as ci.yml's `CI gate`): # when the dev branch is protected, require just that one check. @@ -162,6 +162,30 @@ jobs: - name: Lint run: ruff check --output-format github . + - name: Formatting check + # Mirrors ci.yml's gate; the tree is ruff-formatted, so a failure here + # is a new unformatted file. + run: ruff format --check . + + codeql: + # Reusable-workflow call (see codeql.yml's header): puts the CodeQL jobs in + # THIS graph so `Dev gate` blocks on them. A finding fails the gate — the + # `Fail on findings` step inside makes green mean "clean", not just "ran". + # Unconditional (no docs-only detection in the fast lane, and Dev gate + # demands strict success, so a skip would fail it); running on dev pushes + # too also covers any direct push that bypassed a PR. + permissions: + contents: read + security-events: write + actions: read + uses: ./.github/workflows/codeql.yml + + semgrep: + # Same mechanism: called here so `Dev gate` blocks on it. + permissions: + contents: read + uses: ./.github/workflows/semgrep.yml + web: name: Web lint / test / build (fast) runs-on: ubuntu-latest @@ -185,6 +209,15 @@ jobs: cache: npm cache-dependency-path: web/package-lock.json + - name: Audit dependencies for known CVEs + # Mirrors ci.yml's web job: lockfile-only, fails on any severity, runs + # before the install so a vulnerable lockfile fails fast. + run: npm audit --audit-level=low + + - name: Audit rampart-service dependencies for known CVEs + working-directory: scripts/rampart-service + run: npm audit --audit-level=low + - name: Install dependencies run: npm ci @@ -262,7 +295,7 @@ jobs: # Aggregate check for branch protection: passes only when every fast-lane # job succeeded (mirrors ci.yml's `CI gate`). if: always() - needs: [go, python, web, migrations, gitleaks] + needs: [go, python, codeql, semgrep, web, migrations, gitleaks] runs-on: ubuntu-latest steps: - name: Fail if any fast-lane job failed diff --git a/.github/workflows/semgrep.yml b/.github/workflows/semgrep.yml index f4d8b2a0..0ba90fa0 100644 --- a/.github/workflows/semgrep.yml +++ b/.github/workflows/semgrep.yml @@ -55,10 +55,13 @@ name: Semgrep on: - push: - branches: [main] - pull_request: - branches: [main, dev] + # No push/pull_request triggers of its own: this is a REUSABLE workflow. + # ci.yml and dev-ci.yml call it as a job, which puts it inside `CI gate` / + # `Dev gate` via `needs` — a finding blocks the merge through the existing + # required check, with no branch-protection change. Only the weekly re-scan + # and a manual trigger live here. + workflow_call: + workflow_dispatch: schedule: # Monday 11:00 UTC — one hour after CodeQL's weekly, and clear of the 07:00 # canary / 08:00 govulncheck / 09:00 Grype lanes, following the same @@ -69,10 +72,6 @@ on: permissions: contents: read -concurrency: - group: semgrep-${{ github.event_name }}-${{ github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - jobs: semgrep: name: Semgrep scan diff --git a/AGENTS.md b/AGENTS.md index 23d4c721..31bf6de1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ make compile # go build ./... (compile-check only; no artifacts) make test # go test -p 1 ./... — run in the FOREGROUND make test-race # go test -race -p 1 ./... (use when touching concurrency) make test-cover # run Go tests with coverage profiling (writes coverage.out) -make lint # golangci-lint + ruff (Python) + migration DDL lint — must pass clean +make lint # golangci-lint + ruff check/format (Python) + migration DDL lint — must pass clean make fmt # gofmt -w . make tidy # go mod tidy ``` @@ -34,7 +34,7 @@ make tidy # go mod tidy When you touch `web/` (the Next.js app): ```sh -cd web && npm ci && npm run lint && npm run typecheck && npm run test && npm run build +cd web && npm audit --audit-level=low && npm ci && npm run lint && npm run typecheck && npm run test && npm run build cd web && npx playwright test --project=mocked # mocked e2e ``` @@ -47,10 +47,12 @@ deterministic without a live model: use the fake-LLM seam (`internal/fakellm` via `OPENROUTER_BASE_URL`), never a real key. CodeQL (security queries) and Semgrep (Go/JS/Python SAST + Actions supply chain) -also run per PR and **fail on any finding**. They are not part of `ci-gate` — -`needs` cannot cross workflow files — so they report as their own checks -(`CodeQL gate`, `Semgrep scan`). Both are at zero findings today; keeping them -there is the point. See [`docs/SCANNING.md`](docs/SCANNING.md). +also run per PR, **fail on any finding**, and are **inside `ci-gate` and +`Dev gate`** — both are reusable workflows that ci.yml/dev-ci.yml call as jobs, +so a finding blocks the merge through the existing required check. `npm audit` +gates the web and rampart-service dependency trees the same way. Everything is +at zero findings today; keeping it there is the point. See +[`docs/SCANNING.md`](docs/SCANNING.md). ## Repository map diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c7250d8..7bb64ee8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,48 @@ prior versions are listed because none have shipped. ### Fixed +- **The scanners gate through `ci-gate`/`Dev gate` themselves, npm dependencies + are audited, and the whole Python tree is ruff-formatted — with every finding + fixed, none deferred.** + + - **Gate wiring, corrected.** The previous entry said making CodeQL/Semgrep + merge-blocking needed a branch-protection click, reasoning from "`needs` + cannot cross workflow files". Incomplete: `codeql.yml` and `semgrep.yml` are + now **reusable workflows** (`on: workflow_call`) that ci.yml and dev-ci.yml + call as jobs, and those jobs sit in `ci-gate`'s / `Dev gate`'s `needs` — so a + scanner finding blocks a merge through the one existing required check, no + settings change anywhere. Their own push/pull_request triggers are removed + (nothing runs twice); the weekly re-scan crons and a workflow_dispatch stay. + + - **`npm audit` is a new blocking gate** for both npm trees, lockfile-only and + failing on any severity — the npm counterpart of the govulncheck gate. + `web/` was already clean. `scripts/rampart-service` **had no lockfile at + all**, and generating one exposed **5 high-severity vulnerabilities** it had + been hiding: `sharp <0.35.0` (four libvips CVEs) and `adm-zip <0.6.0` + (GHSA-xcpc-8h2w-3j85) via `onnxruntime-node`. No upstream release fixes + either — latest `@huggingface/transformers` still pins `sharp ^0.34.5`, and + npm's suggested "fix" was a breaking transformers downgrade — so the + package now carries `overrides` to `sharp ^0.35.3` and `adm-zip ^0.6.0`, + each the release immediately after the vulnerable line. The overridden + stack was installed and load-tested, not just resolved: sharp renders a PNG + through the new libvips, transformers loads on it, rampart exports its API, + adm-zip round-trips a zip. Both trees now audit at 0. + + - **`ruff format` applied and gated.** 9 of 13 Python files reformatted + (~3.7k lines), `ruff format --check` now blocks in both CI lanes and in + `make lint`. Validated by the full Go suite (the bento/fileops golden tests + exercise the reformatted scripts), byte-compilation of every file, and a + re-scan showing the fileops `nosemgrep` waiver survived the reformat. + + - **All three semgrep parse errors fixed**, so no file is partially covered: + `${{ steps.build.outcome }}` interpolated into a `run:` script in + build-sandbox-image.yml (moved to `env:` — also the injection-safe form), a + `${tag:-(…)}` expansion default whose bare paren choked the bash sub-parser + (hoisted to a plain assignment), and an inline `import("@playwright/test")` + type in fixtures.ts (now a named `import type`; web lint, tsc and all 1104 + vitest tests pass on it). The scanners' coverage lines now read + **0 parse/scan errors** alongside 0 findings. + - **The scanners now block, and the repo passes them.** Turning a gate on over an unfixed backlog is how a gate becomes something people route around, so everything they reported was fixed or adjudicated first. diff --git a/Makefile b/Makefile index 7a2005df..4a53dfaa 100644 --- a/Makefile +++ b/Makefile @@ -136,7 +136,7 @@ lint-go: # post-mortems about. lint-python: @if command -v ruff >/dev/null 2>&1; then \ - ruff check . ; \ + ruff check . && ruff format --check . ; \ else \ echo "ruff not installed — SKIPPING the Python lint (CI still enforces it)."; \ echo " install: python3 -m pip install --user 'ruff==0.15.8'"; \ diff --git a/docs/CODEQL.md b/docs/CODEQL.md index c9a7fc10..42f10b5a 100644 --- a/docs/CODEQL.md +++ b/docs/CODEQL.md @@ -391,10 +391,14 @@ Without that step the analyze step exits 0 whether it found nothing or a hundred alerts, so a red check could only ever mean "the scanner broke" — which is exactly how the toolchain break hid for weeks. -**A red check does not yet block a merge.** `ci-gate` remains the single required -status check on `main` (see [`.github/CODEOWNERS`](../.github/CODEOWNERS)). -Closing that half means adding `CodeQL gate` to the ruleset, which a workflow -file cannot do for itself. +**A red check now blocks the merge too**, and through the *existing* required +check rather than a new one: `codeql.yml` is a reusable workflow +(`on: workflow_call`) that `ci.yml` and `dev-ci.yml` call as a job, and that +calling job sits in `ci-gate`'s / `Dev gate`'s `needs`. A correction worth +keeping: an earlier revision claimed this half needed a repo-settings click, +reasoning from "`needs` cannot cross workflow files" — true, but a +`workflow_call` brings the jobs into the caller's file, which is the standard +mechanism and what ships. It *cannot* be folded into `ci-gate`: a job's `needs` cannot reach across workflow files. So `codeql.yml` carries its own aggregate **`CodeQL gate`** job, @@ -412,13 +416,12 @@ changes nothing on its own: direction: a required check that never reports again blocks every PR, or a removed one silently stops gating. One aggregate check has neither problem. -**To make CodeQL blocking** (owner action, not done here): Settings → Rules → the -"Main" ruleset → "Require status checks to pass" → add **`CodeQL gate`**. That -single entry covers every language in the matrix, now and after future matrix -changes. +No ruleset action is required for any of this: the gate wiring above is entirely +in the workflow files. (`CodeQL gate` still exists as the aggregate job — the +weekly scheduled run's single verdict — and could additionally be named in the +ruleset as belt-and-braces, but nothing depends on that.) -**Recommendation on the ruleset half: add it once you have seen a few green -promotions.** Not out of +**On sequencing:** Not out of caution for its own sake — because of this specific incident. The analysis spent weeks red for a toolchain reason unrelated to any diff, and a required check in that state blocks *every* merge, including the promote PR that would carry the diff --git a/docs/SCANNING.md b/docs/SCANNING.md index 21c6f50a..6edd7a44 100644 --- a/docs/SCANNING.md +++ b/docs/SCANNING.md @@ -15,8 +15,9 @@ security queries only) and [`TESTING.md`](TESTING.md) (the rest of the ladder). | `govulncheck` | Go dependency CVEs (called symbols) | ~30s | **blocks** (`ci-gate`) | job log + Security tab | | `grype` | sandbox image CVEs (fixable CRITICAL) | ~1m | **blocks** (`ci-gate`) | job log + Security tab | | `gitleaks` | secrets, every branch | ~10s | **blocks** (`ci-gate`) | job log | -| CodeQL | **interprocedural taint / security** | ~2m | **fails on findings** (check red; see below) | job log + Security tab | -| **Semgrep** | **Go/JS/Python SAST + Actions supply chain** | ~40s | **fails on findings** | job log + artifact | +| **`npm audit`** | npm dependency CVEs (web + rampart-service) | ~5s | **blocks** (`ci-gate`) | job log | +| CodeQL | **interprocedural taint / security** | ~2m | **blocks** (`ci-gate`/`Dev gate` via workflow_call) | job log + Security tab | +| **Semgrep** | **Go/JS/Python SAST + Actions supply chain** | ~40s | **blocks** (`ci-gate`/`Dev gate` via workflow_call) | job log + artifact | Two things were added here (**ruff**, **Semgrep**) and one was narrowed (**CodeQL**, to security queries only). @@ -54,9 +55,10 @@ violation is a regression rather than noise in a backlog: the only finding here that was arguably a latent bug. - `bento_pdf.py` — a lambda assigned to a name (`E731`), rewritten as a `def`. -`ruff format` is reported but **not** gated: the tree has never been -ruff-formatted, so failing on it would block every PR on a reformat nobody -scheduled. The advisory output keeps the size of that decision visible. +`ruff format --check` is **also gated** (CI and `make lint`): the whole tree +was ruff-formatted in one dedicated commit (9 files, ~3.7k lines, validated +against the full Go suite — the bento/fileops golden tests exercise these +scripts), so the gate started clean and a failure means one new file. ### CodeQL owns interprocedural taint (narrowed, fails on findings) @@ -136,6 +138,37 @@ Python `#`, TypeScript `//`), including the one waiver that had to become a *trailing* comment because a standalone comment inside a Go import block breaks `goimports`. +### npm audit owns dependency CVEs for the two npm trees (new, blocking) + +`govulncheck` is Go-only and `grype` scans the sandbox *image*, so the web +tier's dependency tree — and `scripts/rampart-service`'s — had no CVE gate at +all. `npm audit --audit-level=low` now runs in the `web` job of both CI lanes, +lockfile-only (no install needed), before the expensive `npm ci`, and fails on +**any** severity. Like govulncheck, its verdict is a function of the clock as +well as the commit: a new advisory can redden an unchanged tree, and that is +the point. + +Turning it on surfaced a real backlog immediately: + +- `web/` was already clean — 0 vulnerabilities — thanks to the steady stream of + merged Dependabot PRs. +- `scripts/rampart-service` **had no `package-lock.json` at all**, which meant + no reproducible installs and nothing for an auditor to read. Generating one + exposed **5 high-severity vulnerabilities** the missing lockfile had been + hiding: `sharp <0.35.0` (libvips CVE-2026-33327/-33328/-35590/-35591) and, + one layer down, `adm-zip <0.6.0` (GHSA-xcpc-8h2w-3j85, crafted-ZIP 4 GB + allocation) via `onnxruntime-node`. + +No upstream release fixes either — the latest `@huggingface/transformers` +still pins `sharp ^0.34.5`, and npm's own suggested "fix" was a breaking +*downgrade* of transformers — so `package.json` carries two `overrides` +(`sharp ^0.35.3`, `adm-zip ^0.6.0`, each the release immediately after the +vulnerable line). The overridden stack was **installed and load-tested**, not +just resolved: sharp renders a PNG through the new libvips, transformers loads +on it, rampart exports its API, and adm-zip 0.6 round-trips a zip. Audit result +after: 0 vulnerabilities in both trees. When upstream ships fixed ranges, the +overrides can be dropped. + ## Findings are readable from the job log, on purpose Both scanners print a per-rule summary into the job log **and** the step summary: @@ -158,51 +191,54 @@ Semgrep additionally uploads its raw JSON as an artifact (`semgrep-findings`, re-running the scan. The repo is public and these results are not sensitive; withholding them buys nothing. -## What gates, and what a required check actually means - -The lint/test/build lanes reach `main`'s single required status check through -**`ci-gate`**. ruff is inside it. The two scanners are not — they cannot be, since -a job's `needs` cannot reach across workflow files — so each carries its own -aggregate gate job (`CodeQL gate`) or fails directly (Semgrep). - -**Both scanners now fail their job on any finding.** That is what makes them -gates rather than reports, and it is only defensible because the tree is at zero -unsuppressed findings in both — verified before switching either on. A gate -turned on over an existing backlog is a gate people route around. - -**One half is still yours to close:** a failing check only *blocks a merge* if it -is a required status check. Add **`CodeQL gate`** and **`Semgrep scan`** to the -"Main" ruleset to finish it. A workflow file cannot make itself required. - -The distinction that matters for anyone tightening this later: - -| block a merge when… | mechanism | -| --- | --- | -| an analysis **failed or did not run** | required status check (`CodeQL gate`) | -| a scanner **found alerts** at/above a severity | **code scanning merge protection** (ruleset → Code scanning rule) | - -**A CodeQL job with a hundred open alerts still exits 0 and reports green.** Job -success only says extraction and evaluation worked. That is both why the Go -toolchain break survived weeks behind a red-but-not-required check, and why a -green check is not evidence of a clean tree. See -[`CODEQL.md`](CODEQL.md#two-different-things-can-gate-and-they-are-not-the-same-lever). +The parse/scan-error line in that summary is at **zero**, and keeping it there +matters: a partial parse silently drops rules from a file. The three errors it +started with were all fixed for real — `${{ steps.build.outcome }}` interpolated +into a `run:` script in `build-sandbox-image.yml` (moved to `env:`, which is +also the injection-safe form), a `${tag:-(…)}` expansion default whose bare +paren choked the bash sub-parser (hoisted to a plain assignment), and an inline +`import("@playwright/test")` type in `fixtures.ts` (a named `import type`, +validated by `tsc`). + +## What gates — everything, through the gates that already exist + +Every lane in the table reaches the branch's aggregate gate: + +- `ci-gate` (the single required status check on `main`) `needs` the lint, test + and build jobs — **and the two scanners**. +- `Dev gate` does the same on `dev`. + +The scanners get there because `codeql.yml` and `semgrep.yml` are **reusable +workflows** (`on: workflow_call`): `ci.yml` and `dev-ci.yml` each call them as a +job, and a job that calls a reusable workflow sits in a gate's `needs` like any +other job. A scanner finding therefore blocks a merge through the existing +required check — **no branch-protection change, no new required check.** + +Worth recording as a correction: an earlier revision of this document claimed +gating the scanners required a repo-settings click, reasoning from "`needs` +cannot cross workflow files". True but incomplete — a `workflow_call` brings the +called jobs *into* the caller's file, which is the standard mechanism and what +ships now. The scanners' own `push`/`pull_request` triggers were removed so +nothing runs twice; each keeps its weekly `schedule` (new queries/rules against +unchanged code) and a `workflow_dispatch`. + +**Both scanners fail their job on any finding.** That is what makes a green +check mean "clean tree" rather than "the scanner ran" — the analyze step alone +exits 0 whether it found nothing or a hundred alerts, which is how the Go +toolchain break survived weeks behind a red-but-not-required check. Failing on +*any* finding is only defensible because the tree is at zero unsuppressed +findings everywhere — verified before the switch was flipped. A gate turned on +over an existing backlog is a gate people route around. + +(Code scanning merge protection — the ruleset's alert-severity rule — remains +available on top as a belt-and-braces option, but nothing depends on it now.) ## Known gaps, deliberately not closed here Stated rather than left for rediscovery: -- **No CVE scanning of the web tier's npm tree.** `govulncheck` is Go-only; - `grype` scans the sandbox *image*. A Next.js app with ~437 TS/JS files has no - dependency CVE gate. Dependabot opens npm PRs, but Dependabot alerts do not - block anything. This is the largest remaining hole in the stack — arguably - larger than anything CodeQL gating would fix. - **`_test.go` files are outside CodeQL's database** (621 files) — `autobuild` builds packages, not tests. Unchanged from default setup. -- **`ruff format` is not enforced.** The tree has never been ruff-formatted: - 9 of 13 files differ, a **3725-line** diff. That is cosmetics, not a finding, - and landing it inside a security change would bury the security change. One - command (`ruff format .`) plus flipping the advisory step to a gate, whenever - someone wants it. - **ruff's rule set is narrow**, so some real bug classes go unreported. Measured: `--select B,SIM,S` adds **21** findings, of which the genuinely interesting ones are 2 × `B905` (`zip()` without `strict=` — silent diff --git a/ruff.toml b/ruff.toml index 1b9cb0de..63016279 100644 --- a/ruff.toml +++ b/ruff.toml @@ -38,6 +38,12 @@ # fixed to get here (an unused import, a byte-identical duplicate function # definition, and a lambda assignment), so a NEW default-rule violation is a # real regression rather than noise in a backlog. +# +# FORMATTING IS ALSO GATED (`ruff format --check`, in CI and `make lint`). The +# whole tree was ruff-formatted in one commit — a 9-file, ~3.7k-line diff kept +# separate from any behavioural change — and validated against the full Go test +# suite (the bento/fileops golden tests exercise these scripts). From here a +# format failure means one new file, fixed by running `ruff format .`. # Match the lowest Python the sandbox image and the skill scripts must run on. # Declared here so the rules that are version-sensitive (pyupgrade et al, if diff --git a/scripts/rampart-service/package-lock.json b/scripts/rampart-service/package-lock.json new file mode 100644 index 00000000..5262df8b --- /dev/null +++ b/scripts/rampart-service/package-lock.json @@ -0,0 +1,1042 @@ +{ + "name": "fleet-rampart-service", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "fleet-rampart-service", + "version": "1.0.0", + "dependencies": { + "@huggingface/transformers": "^4.2.0", + "@nationaldesignstudio/rampart": "^0.1.3" + }, + "engines": { + "node": ">=24" + } + }, + "node_modules/@emnapi/runtime": { + "version": "1.11.3", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.3.tgz", + "integrity": "sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==", + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@huggingface/jinja": { + "version": "0.5.9", + "resolved": "https://registry.npmjs.org/@huggingface/jinja/-/jinja-0.5.9.tgz", + "integrity": "sha512-uWTG+l3VJRsl7EXxYizuL3P+cCPoc3cRqbWWRcQN0FhejRfbdq0RNhCmbY/YDtnTcz9icdLYuLDjsnz4d8JMuw==", + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/@huggingface/tokenizers": { + "version": "0.1.3", + "resolved": "https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.1.3.tgz", + "integrity": "sha512-8rF/RRT10u+kn7YuUbUg0OF30K8rjTc78aHpxT+qJ1uWSqxT1MHi8+9ltwYfkFYJzT/oS+qw3JVfHtNMGAdqyA==", + "license": "Apache-2.0" + }, + "node_modules/@huggingface/transformers": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/@huggingface/transformers/-/transformers-4.2.0.tgz", + "integrity": "sha512-8BRCoBMH0XsWaEIamuR0LrJGAfftgHAfb2Vrffy0VKlSAE/MnUJ5/h/zTfEP3fDIft+nk7TqB8xXEyABGitBjQ==", + "license": "Apache-2.0", + "dependencies": { + "@huggingface/jinja": "^0.5.6", + "@huggingface/tokenizers": "^0.1.3", + "onnxruntime-node": "1.24.3", + "onnxruntime-web": "1.26.0-dev.20260416-b7804b056c", + "sharp": "^0.34.5" + } + }, + "node_modules/@img/colour": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@img/colour/-/colour-1.1.0.tgz", + "integrity": "sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==", + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/@img/sharp-darwin-arm64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-darwin-arm64/-/sharp-darwin-arm64-0.35.3.tgz", + "integrity": "sha512-RMnFX7YQsMoh7lWfcM4NEHHymBX/rLuKNPVM84XE9ONPcaSCDgE7CHIHpSgPcO2xcRthgBy1HfNO319mwhIAkg==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-darwin-arm64": "1.3.2" + } + }, + "node_modules/@img/sharp-darwin-x64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-darwin-x64/-/sharp-darwin-x64-0.35.3.tgz", + "integrity": "sha512-Xo+5uFBtLN0BKqieTxiFzFPQAUlBbbH5iBKyRX/z1JrbnYsHTfKJnUfL8+p2TPXr1pXqao4eeL4Rl144uDpK9w==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-darwin-x64": "1.3.2" + } + }, + "node_modules/@img/sharp-freebsd-wasm32": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-freebsd-wasm32/-/sharp-freebsd-wasm32-0.35.3.tgz", + "integrity": "sha512-lUxcqWIj2wMQ9BrwNjngcr1gWUr5xgaGThBRqPPalIC2n67Cqj1uPh8NnA/ZhAg8hUbKl+kVHKwgUIwe6ZYPrg==", + "license": "Apache-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "dependencies": { + "@img/sharp-wasm32": "0.35.3" + }, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-darwin-arm64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-arm64/-/sharp-libvips-darwin-arm64-1.3.2.tgz", + "integrity": "sha512-9J6ypZFpQBj4YnePGoq/S38w6nz+vqg5WZLrLGY4YuSemdMq47GMLBPO42MzwdGwpg/agZ7xzZcFHa48xlywfg==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "darwin" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-darwin-x64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-x64/-/sharp-libvips-darwin-x64-1.3.2.tgz", + "integrity": "sha512-m2pW1n6cns9VaubNwsZ+c3CRYjxNQWgJ5gPlnL1nbBcpkBvFm6SCFN5o0psFHI8w9n11NKhFkeEDns98tiqbEw==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "darwin" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-arm": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm/-/sharp-libvips-linux-arm-1.3.2.tgz", + "integrity": "sha512-1eMLzy92I4J6rmi4mAT8yC3HxOtniyGELlzGbNMLLeqe052ahFQ0h6LFq+lh5DsDIdYViIDst08abvSbcEdLXQ==", + "cpu": [ + "arm" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-arm64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm64/-/sharp-libvips-linux-arm64-1.3.2.tgz", + "integrity": "sha512-dqVSFynCox4C/J8kT16V7SIFAns0IjgLwkvYT7p8LQVmJ5OS5b6tI9IGflxTeuBS//zXeFIUbwt5dwxyZ17cnA==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-ppc64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-ppc64/-/sharp-libvips-linux-ppc64-1.3.2.tgz", + "integrity": "sha512-3z0NHDxD6n5I9gc05U1eW1AyRm+Gznzq3naMrthPNqE6oYykcogW0l/jfpJdjYnuNl8R7yI9pNbE1XiUeyq0Aw==", + "cpu": [ + "ppc64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-riscv64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-riscv64/-/sharp-libvips-linux-riscv64-1.3.2.tgz", + "integrity": "sha512-bsb4rI+NldGOsXuej2r8OdSS8+zXDVaCWxyWrcv6kneTOlgAHtZABRzBBCwdsPiD90J4myNJuHpg6kA20ImW/w==", + "cpu": [ + "riscv64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-s390x": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-s390x/-/sharp-libvips-linux-s390x-1.3.2.tgz", + "integrity": "sha512-/ABshyj8gCpyIrNXnHn4LorDJ0HHm1VhXPBlxZ8zAtfVPAaSafXPGn+sUSIRiwaSBy0mmFjSjiXI5mkcwdChKQ==", + "cpu": [ + "s390x" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-x64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-x64/-/sharp-libvips-linux-x64-1.3.2.tgz", + "integrity": "sha512-ITPEtgffGJ0S6G9dRyw/366tJQqFRcHWPHhC+Stpg3Z8AEMrDrTr2lhdz4f/Y/HMbRh//7Z5mBzEpVdi62Oc3w==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linuxmusl-arm64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-arm64/-/sharp-libvips-linuxmusl-arm64-1.3.2.tgz", + "integrity": "sha512-zE9EdiUzUmg5mDT5a1rk5fYJ6GWPloTwWBYDS14naqHsL+EaMpDj1AWnpLgh3u0YCORv2Tt50wrcrpYqkP97Kw==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linuxmusl-x64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-x64/-/sharp-libvips-linuxmusl-x64-1.3.2.tgz", + "integrity": "sha512-m0lrLiUt+lBYnCFr8qV/65yMR4E/c7/wf78I5eKTdkEakFAlZ9QlzEM3QIhhAwVeUhLAHLcCq7a7Vszq/oFNZQ==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-linux-arm": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm/-/sharp-linux-arm-0.35.3.tgz", + "integrity": "sha512-affVWCTLooy8TSxbDx2qkzuDeaWLNVBA+P//FNBirHsXpP2fuBhk5AuboYUnrDnzoXes8GFjpTx0SBFOCRg+FA==", + "cpu": [ + "arm" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-arm": "1.3.2" + } + }, + "node_modules/@img/sharp-linux-arm64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm64/-/sharp-linux-arm64-0.35.3.tgz", + "integrity": "sha512-QgKDspHPnrU+GQ55XPhGwyhC8acLVOOSyAvo1oVfFmrIXLkDNmGWzAfDZ4xK8oSA1qBQrALcHX0G5UZni/SuFQ==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-arm64": "1.3.2" + } + }, + "node_modules/@img/sharp-linux-ppc64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-ppc64/-/sharp-linux-ppc64-0.35.3.tgz", + "integrity": "sha512-sMd8rDxmpLOwv/7N44klFjOD5DUO7FLdjiXDI0hoxYaf7Ar262dQIEkosE98bps+5HPLtp/EvNqeqQtOycP/IA==", + "cpu": [ + "ppc64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-ppc64": "1.3.2" + } + }, + "node_modules/@img/sharp-linux-riscv64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-riscv64/-/sharp-linux-riscv64-0.35.3.tgz", + "integrity": "sha512-0Eob78yjlYPfL5vMNWAW55l3R9Y6BQS/gOfe0ZcP9mEz9ohhKSt4im1hayiknXgf8AWrFqMvJcKIdmLmEe7yeQ==", + "cpu": [ + "riscv64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-riscv64": "1.3.2" + } + }, + "node_modules/@img/sharp-linux-s390x": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-s390x/-/sharp-linux-s390x-0.35.3.tgz", + "integrity": "sha512-KgAxQ0DxpNOq1rG2t5cgTgShJFGSuU7XO45cqC+1NVOuZnP6tlgZRuSYOfNupGkHID0o3cJOsw4DVeJpMovcGw==", + "cpu": [ + "s390x" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-s390x": "1.3.2" + } + }, + "node_modules/@img/sharp-linux-x64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-x64/-/sharp-linux-x64-0.35.3.tgz", + "integrity": "sha512-8pqvxubL2PGdhlPy6GLqzDYMUjyRmKAwKHYKixpdJYBUK7PJ0C029XdsnpFIdgRZG68fZiGdHVWcKPvtiPB4cA==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-x64": "1.3.2" + } + }, + "node_modules/@img/sharp-linuxmusl-arm64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-arm64/-/sharp-linuxmusl-arm64-0.35.3.tgz", + "integrity": "sha512-Vz0iQjzzcSX3HCbfwFfCSG/9SCIqyO0mH2sXyiHaAYfBk0cRsCWXRyQYX0ovCK/PAQBbTzQ0dsPQHh5MAFL59w==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linuxmusl-arm64": "1.3.2" + } + }, + "node_modules/@img/sharp-linuxmusl-x64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-x64/-/sharp-linuxmusl-x64-0.35.3.tgz", + "integrity": "sha512-6O1NPKcDVj9QEdg7Hx549EX8U0rp6yXQERqru6yRN7fGBn32UvIRJUlWnk+8xDCiG76hXVBbX82NZ/ZKr0euIg==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linuxmusl-x64": "1.3.2" + } + }, + "node_modules/@img/sharp-wasm32": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-wasm32/-/sharp-wasm32-0.35.3.tgz", + "integrity": "sha512-cZ0XkcYGpHZkqW6iCkqTcmUC0CD9DhD5d/qeZlZkfRBn6GnHniZXLUo5+9xw8Iv76YE6LQFN9YNBlKREcCG76w==", + "license": "Apache-2.0 AND LGPL-3.0-or-later AND MIT", + "optional": true, + "dependencies": { + "@emnapi/runtime": "^1.11.1" + }, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-webcontainers-wasm32": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-webcontainers-wasm32/-/sharp-webcontainers-wasm32-0.35.3.tgz", + "integrity": "sha512-2rnq7bX3NzeR2T4YWgz8qiG4h3TSdMe+vN1iQXpJleSJ3SM5zQ8Fy2SyyXAWlbxpEZ2Y+Z4u1BePgJEYbSy80Q==", + "cpu": [ + "wasm32" + ], + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "@img/sharp-wasm32": "0.35.3" + }, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-win32-arm64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-arm64/-/sharp-win32-arm64-0.35.3.tgz", + "integrity": "sha512-4bPwFdMbeC4JQ8L8LOyWp6nsHcboP5fxkp6iPOXz2Vg49R42TuMs2whkJ5OAP4/Ul035qOzy0AecOF9VOscn4w==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-win32-ia32": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-ia32/-/sharp-win32-ia32-0.35.3.tgz", + "integrity": "sha512-r53mXsBN6lFUDiST764SvgwUdHAqM4rPAiDzAmf4fLoB6X/rkfyTrLCg6+g17wJJiCmB3JYgHuUldCWUIRFSXw==", + "cpu": [ + "ia32" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-win32-x64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-x64/-/sharp-win32-x64-0.35.3.tgz", + "integrity": "sha512-D4y1vNeZrIIJCN+uHaWVtH86B+aCrdMYYjicy9pXHvbGZeGYLLSd3wdVuC37FxVXlU1ARsk84eKWfWMXGYEqvA==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@nationaldesignstudio/rampart": { + "version": "0.1.3", + "resolved": "https://registry.npmjs.org/@nationaldesignstudio/rampart/-/rampart-0.1.3.tgz", + "integrity": "sha512-N3AMnPO1nGxUfMhTm+zJTuhXRIcdR/ETMIMyVr741pgxQcaRNMJVhKgN2nfEmwnXwGsViWpxFQOO0YYs8zZR2w==", + "license": "CC-BY-4.0", + "peerDependencies": { + "@huggingface/transformers": ">=3" + }, + "peerDependenciesMeta": { + "@huggingface/transformers": { + "optional": true + } + } + }, + "node_modules/@protobufjs/aspromise": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", + "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/base64": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz", + "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/codegen": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.5.tgz", + "integrity": "sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/eventemitter": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.1.tgz", + "integrity": "sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/fetch": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.1.tgz", + "integrity": "sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw==", + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.1" + } + }, + "node_modules/@protobufjs/float": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz", + "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/path": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz", + "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/pool": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz", + "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/utf8": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.2.tgz", + "integrity": "sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug==", + "license": "BSD-3-Clause" + }, + "node_modules/@types/node": { + "version": "26.2.0", + "resolved": "https://registry.npmjs.org/@types/node/-/node-26.2.0.tgz", + "integrity": "sha512-5IviulTZeRNp2vAJ514cc/HUlY5nZ9fCbq9DMyC52BrhFZACo3nI0R7qBxhQmo/d27NFe96ur/b7Wwxklda+kg==", + "license": "MIT", + "dependencies": { + "undici-types": "~8.3.0" + } + }, + "node_modules/adm-zip": { + "version": "0.6.0", + "resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.6.0.tgz", + "integrity": "sha512-XleryMhbuksdKtofnWZ9Sk+4CUTbms4Mb/EU32SZwToAyZ5RgVos/ki8n+yr0LWHOGKuakbXTuuYNHLQjhddgg==", + "license": "MIT", + "engines": { + "node": ">=14.0" + } + }, + "node_modules/boolean": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/boolean/-/boolean-3.2.0.tgz", + "integrity": "sha512-d0II/GO9uf9lfUHH2BQsjxzRJZBdsjgsBiW4BvhWk/3qoKwQFjIDVN19PfX8F2D/r9PCMTtLWjYVCFrpeYUzsw==", + "deprecated": "Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.", + "license": "MIT" + }, + "node_modules/define-data-property": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/define-data-property/-/define-data-property-1.1.4.tgz", + "integrity": "sha512-rBMvIzlpA8v6E+SJZoo++HAYqsLrkg7MSfIinMPFhmkorw7X+dOXVJQs+QT69zGkzMyfDnIMN2Wid1+NbL3T+A==", + "license": "MIT", + "dependencies": { + "es-define-property": "^1.0.0", + "es-errors": "^1.3.0", + "gopd": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/define-properties": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/define-properties/-/define-properties-1.2.1.tgz", + "integrity": "sha512-8QmQKqEASLd5nx0U1B1okLElbUuuttJ/AnYmRXbbbGDWh6uS208EjD4Xqq/I9wK7u0v6O08XhTWnt5XtEbR6Dg==", + "license": "MIT", + "dependencies": { + "define-data-property": "^1.0.1", + "has-property-descriptors": "^1.0.0", + "object-keys": "^1.1.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/detect-libc": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", + "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==", + "license": "Apache-2.0", + "engines": { + "node": ">=8" + } + }, + "node_modules/detect-node": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/detect-node/-/detect-node-2.1.0.tgz", + "integrity": "sha512-T0NIuQpnTvFDATNuHN5roPwSBG83rFsuO+MXXH9/3N1eFbn4wcPjttvjMLEPWJ0RGUYgQE7cGgS3tNxbqCGM7g==", + "license": "MIT" + }, + "node_modules/es-define-property": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", + "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-errors": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", + "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es6-error": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/es6-error/-/es6-error-4.1.1.tgz", + "integrity": "sha512-Um/+FxMr9CISWh0bi5Zv0iOD+4cFh5qLeks1qhAopKVAJw3drgKbKySikp7wGhDL0HPeaja0P5ULZrxLkniUVg==", + "license": "MIT" + }, + "node_modules/escape-string-regexp": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/escape-string-regexp/-/escape-string-regexp-4.0.0.tgz", + "integrity": "sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA==", + "license": "MIT", + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/flatbuffers": { + "version": "25.9.23", + "resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-25.9.23.tgz", + "integrity": "sha512-MI1qs7Lo4Syw0EOzUl0xjs2lsoeqFku44KpngfIduHBYvzm8h2+7K8YMQh1JtVVVrUvhLpNwqVi4DERegUJhPQ==", + "license": "Apache-2.0" + }, + "node_modules/global-agent": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/global-agent/-/global-agent-3.0.0.tgz", + "integrity": "sha512-PT6XReJ+D07JvGoxQMkT6qji/jVNfX/h364XHZOWeRzy64sSFr+xJ5OX7LI3b4MPQzdL4H8Y8M0xzPpsVMwA8Q==", + "license": "BSD-3-Clause", + "dependencies": { + "boolean": "^3.0.1", + "es6-error": "^4.1.1", + "matcher": "^3.0.0", + "roarr": "^2.15.3", + "semver": "^7.3.2", + "serialize-error": "^7.0.1" + }, + "engines": { + "node": ">=10.0" + } + }, + "node_modules/globalthis": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/globalthis/-/globalthis-1.0.4.tgz", + "integrity": "sha512-DpLKbNU4WylpxJykQujfCcwYWiV/Jhm50Goo0wrVILAv5jOr9d+H+UR3PhSCD2rCCEIg0uc+G+muBTwD54JhDQ==", + "license": "MIT", + "dependencies": { + "define-properties": "^1.2.1", + "gopd": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/gopd": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", + "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/guid-typescript": { + "version": "1.0.9", + "resolved": "https://registry.npmjs.org/guid-typescript/-/guid-typescript-1.0.9.tgz", + "integrity": "sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ==", + "license": "ISC" + }, + "node_modules/has-property-descriptors": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/has-property-descriptors/-/has-property-descriptors-1.0.2.tgz", + "integrity": "sha512-55JNKuIW+vq4Ke1BjOTjM2YctQIvCT7GFzHwmfZPGo5wnrgkid0YQtnAleFSqumZm4az3n2BS+erby5ipJdgrg==", + "license": "MIT", + "dependencies": { + "es-define-property": "^1.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/json-stringify-safe": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/json-stringify-safe/-/json-stringify-safe-5.0.1.tgz", + "integrity": "sha512-ZClg6AaYvamvYEE82d3Iyd3vSSIjQ+odgjaTzRuO3s7toCdFKczob2i0zCh7JE8kWn17yvAWhUVxvqGwUalsRA==", + "license": "ISC" + }, + "node_modules/long": { + "version": "5.3.2", + "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", + "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", + "license": "Apache-2.0" + }, + "node_modules/matcher": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/matcher/-/matcher-3.0.0.tgz", + "integrity": "sha512-OkeDaAZ/bQCxeFAozM55PKcKU0yJMPGifLwV4Qgjitu+5MoAfSQN4lsLJeXZ1b8w0x+/Emda6MZgXS1jvsapng==", + "license": "MIT", + "dependencies": { + "escape-string-regexp": "^4.0.0" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/object-keys": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/object-keys/-/object-keys-1.1.1.tgz", + "integrity": "sha512-NuAESUOUMrlIXOfHKzD6bpPu3tYt3xvjNdRIQ+FeT0lNb4K8WR70CaDxhuNguS2XG+GjkyMwOzsN5ZktImfhLA==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/onnxruntime-common": { + "version": "1.24.3", + "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.24.3.tgz", + "integrity": "sha512-GeuPZO6U/LBJXvwdaqHbuUmoXiEdeCjWi/EG7Y1HNnDwJYuk6WUbNXpF6luSUY8yASul3cmUlLGrCCL1ZgVXqA==", + "license": "MIT" + }, + "node_modules/onnxruntime-node": { + "version": "1.24.3", + "resolved": "https://registry.npmjs.org/onnxruntime-node/-/onnxruntime-node-1.24.3.tgz", + "integrity": "sha512-JH7+czbc8ALA819vlTgcV+Q214/+VjGeBHDjX81+ZCD0PCVCIFGFNtT0V4sXG/1JXypKPgScQcB3ij/hk3YnTg==", + "hasInstallScript": true, + "license": "MIT", + "os": [ + "win32", + "darwin", + "linux" + ], + "dependencies": { + "adm-zip": "^0.5.16", + "global-agent": "^3.0.0", + "onnxruntime-common": "1.24.3" + } + }, + "node_modules/onnxruntime-web": { + "version": "1.26.0-dev.20260416-b7804b056c", + "resolved": "https://registry.npmjs.org/onnxruntime-web/-/onnxruntime-web-1.26.0-dev.20260416-b7804b056c.tgz", + "integrity": "sha512-MD6Ss4GSpQBo6zqoJzyT9LRbKYs7x/JVN23FT24EcEvlqF4VuzPOeH6X38orZPKHQDbprn7K+SBpu0/mj2CQiw==", + "license": "MIT", + "dependencies": { + "flatbuffers": "^25.1.24", + "guid-typescript": "^1.0.9", + "long": "^5.2.3", + "onnxruntime-common": "1.24.0-dev.20251116-b39e144322", + "platform": "^1.3.6", + "protobufjs": "^7.2.4" + } + }, + "node_modules/onnxruntime-web/node_modules/onnxruntime-common": { + "version": "1.24.0-dev.20251116-b39e144322", + "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.24.0-dev.20251116-b39e144322.tgz", + "integrity": "sha512-BOoomdHYmNRL5r4iQ4bMvsl2t0/hzVQ3OM3PHD0gxeXu1PmggqBv3puZicEUVOA3AtHHYmqZtjMj9FOfGrATTw==", + "license": "MIT" + }, + "node_modules/platform": { + "version": "1.3.6", + "resolved": "https://registry.npmjs.org/platform/-/platform-1.3.6.tgz", + "integrity": "sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==", + "license": "MIT" + }, + "node_modules/protobufjs": { + "version": "7.6.5", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.6.5.tgz", + "integrity": "sha512-/FPD0nUc9jH6rfFjji9IBqOz4pcSE3CsT1m7Ep6Mdb0LxSUMj8hgl6GomOvZzpNpAqqGaXA0P3VSrZLFzIhQrw==", + "hasInstallScript": true, + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.2", + "@protobufjs/base64": "^1.1.2", + "@protobufjs/codegen": "^2.0.5", + "@protobufjs/eventemitter": "^1.1.1", + "@protobufjs/fetch": "^1.1.1", + "@protobufjs/float": "^1.0.2", + "@protobufjs/path": "^1.1.2", + "@protobufjs/pool": "^1.1.0", + "@protobufjs/utf8": "^1.1.1", + "@types/node": ">=13.7.0", + "long": "^5.3.2" + }, + "engines": { + "node": ">=12.0.0" + } + }, + "node_modules/roarr": { + "version": "2.15.4", + "resolved": "https://registry.npmjs.org/roarr/-/roarr-2.15.4.tgz", + "integrity": "sha512-CHhPh+UNHD2GTXNYhPWLnU8ONHdI+5DI+4EYIAOaiD63rHeYlZvyh8P+in5999TTSFgUYuKUAjzRI4mdh/p+2A==", + "license": "BSD-3-Clause", + "dependencies": { + "boolean": "^3.0.1", + "detect-node": "^2.0.4", + "globalthis": "^1.0.1", + "json-stringify-safe": "^5.0.1", + "semver-compare": "^1.0.0", + "sprintf-js": "^1.1.2" + }, + "engines": { + "node": ">=8.0" + } + }, + "node_modules/semver": { + "version": "7.8.5", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", + "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==", + "license": "ISC", + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/semver-compare": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/semver-compare/-/semver-compare-1.0.0.tgz", + "integrity": "sha512-YM3/ITh2MJ5MtzaM429anh+x2jiLVjqILF4m4oyQB18W7Ggea7BfqdH/wGMK7dDiMghv/6WG7znWMwUDzJiXow==", + "license": "MIT" + }, + "node_modules/serialize-error": { + "version": "7.0.1", + "resolved": "https://registry.npmjs.org/serialize-error/-/serialize-error-7.0.1.tgz", + "integrity": "sha512-8I8TjW5KMOKsZQTvoxjuSIa7foAwPWGOts+6o7sgjz41/qMD9VQHEDxi6PBvK2l0MXUmqZyNpUK+T2tQaaElvw==", + "license": "MIT", + "dependencies": { + "type-fest": "^0.13.1" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/sharp": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/sharp/-/sharp-0.35.3.tgz", + "integrity": "sha512-ej0zVHuZGHCiABXcNxeYhpRnPNPAcvbG8RMdBAhDAxLKkCRVSpK3Iyu7qbqw3JMzoj0REeM6f3tJLtVwl0023Q==", + "license": "Apache-2.0", + "dependencies": { + "@img/colour": "^1.1.0", + "detect-libc": "^2.1.2", + "semver": "^7.8.5" + }, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-darwin-arm64": "0.35.3", + "@img/sharp-darwin-x64": "0.35.3", + "@img/sharp-freebsd-wasm32": "0.35.3", + "@img/sharp-libvips-darwin-arm64": "1.3.2", + "@img/sharp-libvips-darwin-x64": "1.3.2", + "@img/sharp-libvips-linux-arm": "1.3.2", + "@img/sharp-libvips-linux-arm64": "1.3.2", + "@img/sharp-libvips-linux-ppc64": "1.3.2", + "@img/sharp-libvips-linux-riscv64": "1.3.2", + "@img/sharp-libvips-linux-s390x": "1.3.2", + "@img/sharp-libvips-linux-x64": "1.3.2", + "@img/sharp-libvips-linuxmusl-arm64": "1.3.2", + "@img/sharp-libvips-linuxmusl-x64": "1.3.2", + "@img/sharp-linux-arm": "0.35.3", + "@img/sharp-linux-arm64": "0.35.3", + "@img/sharp-linux-ppc64": "0.35.3", + "@img/sharp-linux-riscv64": "0.35.3", + "@img/sharp-linux-s390x": "0.35.3", + "@img/sharp-linux-x64": "0.35.3", + "@img/sharp-linuxmusl-arm64": "0.35.3", + "@img/sharp-linuxmusl-x64": "0.35.3", + "@img/sharp-webcontainers-wasm32": "0.35.3", + "@img/sharp-win32-arm64": "0.35.3", + "@img/sharp-win32-ia32": "0.35.3", + "@img/sharp-win32-x64": "0.35.3" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + } + } + }, + "node_modules/sprintf-js": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.1.3.tgz", + "integrity": "sha512-Oo+0REFV59/rz3gfJNKQiBlwfHaSESl1pcGyABQsnnIfWOFt6JNj5gCog2U6MLZ//IGYD+nA8nI+mTShREReaA==", + "license": "BSD-3-Clause" + }, + "node_modules/tslib": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", + "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", + "license": "0BSD", + "optional": true + }, + "node_modules/type-fest": { + "version": "0.13.1", + "resolved": "https://registry.npmjs.org/type-fest/-/type-fest-0.13.1.tgz", + "integrity": "sha512-34R7HTnG0XIJcBSn5XhDd7nNFPRcXYRZrBB2O2jdKqYODldSzBAqzsWoZYYvduky73toYS/ESqxPvkDf/F0XMg==", + "license": "(MIT OR CC0-1.0)", + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/undici-types": { + "version": "8.3.0", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.3.0.tgz", + "integrity": "sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ==", + "license": "MIT" + } + } +} diff --git a/scripts/rampart-service/package.json b/scripts/rampart-service/package.json index b514fa98..0e8db83a 100644 --- a/scripts/rampart-service/package.json +++ b/scripts/rampart-service/package.json @@ -14,5 +14,9 @@ }, "engines": { "node": ">=24" + }, + "overrides": { + "sharp": "^0.35.3", + "adm-zip": "^0.6.0" } } diff --git a/web/e2e/live/fixtures.ts b/web/e2e/live/fixtures.ts index 663d509b..8483f37a 100644 --- a/web/e2e/live/fixtures.ts +++ b/web/e2e/live/fixtures.ts @@ -1,4 +1,5 @@ import { test as base, expect, request } from "@playwright/test"; +import type { BrowserContext } from "@playwright/test"; export { expect, request }; @@ -17,7 +18,7 @@ const SCHED_USERNAME = process.env.E2E_SCHED_USERNAME ?? "e2e"; export const creds = { email: TEST_EMAIL, password: TEST_PASSWORD, schedUsername: SCHED_USERNAME }; -type AuthCookies = Parameters[0]; +type AuthCookies = Parameters[0]; // wipeConversations deletes every conversation for the logged-in user via the // real DELETE endpoint, so each test starts from a clean slate (conversations From 745fe6d8ba1335381ef9cbefe8e0d9e12ba3feda Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 14:31:55 +0000 Subject: [PATCH 12/34] Close every scanner follow-up: B/SIM/S fixed+enabled, security-extended, grype High, canary, cron alarms MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every item from the audit self-assessment that can live in code, closed. One item closed by a reasoned rejection rather than code, with the license cited. ruff B/SIM/S (bandit tier): all 21 measured findings fixed, families ENABLED. Both zip() sites get strict=True — each provably equal-length (bento_pdf appends pages and contents in lockstep in the same loop; profile.py sits behind an explicit len(row) != len(cols) guard) — so a future desync fails loud instead of silently truncating a PDF or a profile. bento_doc's unclosed NamedTemporaryFile moved inside its with (tmp=None sentinel keeps the unlink-on-failure path exact). The fourteen deliberate best-effort try/except-pass sites — kernel cleanup, the duck-typed pandas/numpy probes in normalize_json_value, unlink-on-failure in the sandbox fileops commit path — became explicit contextlib.suppress with the intent stated at each site; the suppressions are semantically identical, and the sandbox fileops conversion is covered by its test suite (full make test green). The one subprocess.Popen carries a reasoned `# noqa: S603`: argv is sys.executable plus literal flags plus a connection-file path this process just created with mkstemp — nothing model- or user-controlled. The waiver was mutation-tested: stripping the noqa re-fires S603. Two of my own first-cut mistakes fixed in the same pass: a prose comment beginning with the literal token "# noqa" (parsed as a malformed directive) and a nested-with that tripped SIM117. CodeQL: security-extended on all four languages. Adopted the way every other gate here was — the default suite measured zero findings, so the broader suite starts from a clean baseline, and this PR's own run (whose Fail-on-findings step reads the SARIF) is the measurement. Anything extended surfaces must be fixed or reasoned away; it cannot accrue. Grype: gate tightened from fixable-CRITICAL to fixable CRITICAL+HIGH, measured first: the published sandbox image (pinned grype 0.117.0, checksum-verified, scanned directly from GHCR) carries ZERO fixable Critical/High RPM findings — its only fixable findings are two Medium openssh advisories that the next routine image rebuild picks up. Policy mutation-tested three ways: the real scan passes, an injected fixable High fails, an injected fixable Medium still passes. scripts/check-npm-overrides.sh: an override is a fork of upstream's intent, correct only while upstream is broken — and the day upstream fixes itself, nothing notices, leaving Dependabot silently pinned down. Both CI lanes now ask the registry what floor @huggingface/transformers and onnxruntime-node declare, and FAIL with removal instructions once those reach sharp>=0.35 / adm-zip>=0.6. Registry flake is a skip with a notice, never a verdict (npm audit beside it is the CVE gate proper). Mutation-tested in both directions. Cron failure alarms: all four scheduled scan lanes (codeql, semgrep, govulncheck, grype) file a deduped GitHub issue when a SCHEDULED run fails — a red cron has no PR to surface it, which is exactly the rot pattern that let the CodeQL toolchain break sit red for weeks. In the workflow_call path the alarm job stays skipped (schedule-only condition), so callers never need to grant issues: write. Semgrep rule vendoring: investigated and REJECTED on license grounds. The Semgrep Rules License v1.0 permits "your own internal business purposes" only and states "This license does not allow you to distribute the rules" — committing the packs to this public MIT repo would be redistribution. The binary stays pinned; the rules stay registry-fetched with the failure mode named in docs/SCANNING.md. Docs: every stale "fixable CRITICAL" mention updated (AGENTS.md, TESTING.md, SANDBOX-IMAGE-FRESHNESS.md, CONTRIBUTING.md, ci.yml comment); SCANNING.md and CODEQL.md carry the new levels and the license decision; ruff.toml's header records why B/SIM/S went from measured-deferred to fixed-enabled and that PLR0124 stays rejected (its only hits are the idiomatic NaN test). Gate: yaml parses, gofmt clean, make build/lint/test green (ruff check+format clean under the widened select), semgrep --error 0 findings / 898 scanned / 0 parse errors. Signed-off-by: Brad Flaugher --- .github/workflows/ci.yml | 10 ++- .github/workflows/codeql.yml | 49 +++++++++++- .github/workflows/dev-ci.yml | 8 ++ .github/workflows/govulncheck-scheduled.yml | 29 +++++++ .github/workflows/grype-scheduled.yml | 29 +++++++ .github/workflows/semgrep.yml | 46 +++++++++++ AGENTS.md | 2 +- CHANGELOG.md | 48 +++++++++++ CONTRIBUTING.md | 2 +- docs/CODEQL.md | 7 +- docs/SANDBOX-IMAGE-FRESHNESS.md | 2 +- docs/SCANNING.md | 58 +++++++++----- docs/TESTING.md | 2 +- .../bento-slides/scripts/bento_doc.py | 18 ++--- .../bento-slides/scripts/bento_pdf.py | 4 +- .../data-profiler/scripts/profile.py | 2 +- internal/sandbox/fileops.py | 21 +++-- internal/tools/python_bridge.py | 71 +++++++--------- ruff.toml | 20 +++-- scripts/check-grype-policy.sh | 15 +++- scripts/check-npm-overrides.sh | 80 +++++++++++++++++++ 21 files changed, 417 insertions(+), 106 deletions(-) create mode 100755 scripts/check-npm-overrides.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7263bf7b..216ebaac 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -420,6 +420,14 @@ jobs: working-directory: scripts/rampart-service run: npm audit --audit-level=low + - name: Check whether the rampart security overrides are droppable + # scripts/rampart-service/package.json force-patches sharp and adm-zip + # because their parents have not released fixes. The day upstream does, + # this FAILS with removal instructions — an override left behind after + # upstream fixes itself silently pins Dependabot's updates down. A + # registry flake skips with a notice (the audit above is the CVE gate). + run: scripts/check-npm-overrides.sh + - name: Install dependencies run: npm ci @@ -776,7 +784,7 @@ jobs: - name: Upload Grype SARIF results to the GitHub Security tab # Upload whenever the scan produced a SARIF — including when the policy - # step rejects a fixable CRITICAL RPM — so + # step rejects a fixable CRITICAL/HIGH RPM — so # findings reach GitHub Security → Code scanning with full CVE details, # affected packages, and fix versions rather than only a red CI job. The # hashFiles guard skips this step (instead of erroring "file not found") diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index e140728e..a8d9ec83 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -128,14 +128,23 @@ jobs: # Go is the reason this file exists. `build-mode: none` is NOT # supported for Go — only `autobuild` or `manual` — so the toolchain # has to be right rather than skipped. + # security-extended everywhere: the broader security suite (more + # queries, lower average precision than the default). Measured before + # adoption like every other gate here — the run's own findings + # summary is the measurement, and the Fail-on-findings step means + # anything it surfaces must be fixed or reasoned away, not accrued. - language: go build-mode: autobuild + queries: security-extended - language: python build-mode: none + queries: security-extended - language: javascript-typescript build-mode: none + queries: security-extended - language: actions build-mode: none + queries: security-extended steps: - name: Checkout @@ -156,8 +165,9 @@ jobs: with: languages: ${{ matrix.language }} build-mode: ${{ matrix.build-mode }} - # No `queries:` — the default (security) suite only. See the SCOPE note - # in the header before adding `code-quality` back. + queries: ${{ matrix.queries }} + # security-extended (see the matrix note). The SCOPE note in the + # header still applies to `code-quality` — do not re-add that one. - name: Autobuild if: matrix.build-mode == 'autobuild' @@ -313,3 +323,38 @@ jobs: for r in $results; do [ "$r" = "success" ] || { echo "a CodeQL analysis did not succeed"; exit 1; } done + + cron-failure-alarm: + name: File an issue so a red cron cannot rot silently + # Only exists for the standalone weekly run: in the workflow_call path the + # caller's gate (`CI gate` / `Dev gate`) is the alarm, and this job's + # schedule-only condition keeps it skipped there (a skip which also means + # the caller never needs to grant issues: write — called-workflow + # permissions intersect with the caller's grant). + # Body duplicated across the four scheduled scan lanes — keep in sync. + if: ${{ failure() && github.event_name == 'schedule' }} + needs: [analyze] + runs-on: ubuntu-latest + permissions: + issues: write + steps: + - name: File or update the alarm issue + env: + GH_TOKEN: ${{ github.token }} + WORKFLOW_NAME: ${{ github.workflow }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + set -euo pipefail + title="Scheduled ${WORKFLOW_NAME} run failed" + num="$(gh issue list --repo "$GITHUB_REPOSITORY" --state open \ + --search "\"$title\" in:title" --json number --jq '.[0].number // empty' || true)" + if [ -n "$num" ]; then + gh issue comment "$num" --repo "$GITHUB_REPOSITORY" \ + --body "Still failing: $RUN_URL" + else + gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" \ + --body "The scheduled $WORKFLOW_NAME lane went red: $RUN_URL + + A failing cron has no PR to make it visible, so this issue is the + alarm. Close it when the lane is green again." + fi diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index 0970f868..9f241114 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -218,6 +218,14 @@ jobs: working-directory: scripts/rampart-service run: npm audit --audit-level=low + - name: Check whether the rampart security overrides are droppable + # scripts/rampart-service/package.json force-patches sharp and adm-zip + # because their parents have not released fixes. The day upstream does, + # this FAILS with removal instructions — an override left behind after + # upstream fixes itself silently pins Dependabot's updates down. A + # registry flake skips with a notice (the audit above is the CVE gate). + run: scripts/check-npm-overrides.sh + - name: Install dependencies run: npm ci diff --git a/.github/workflows/govulncheck-scheduled.yml b/.github/workflows/govulncheck-scheduled.yml index 6a09c2b4..0e4a09e1 100644 --- a/.github/workflows/govulncheck-scheduled.yml +++ b/.github/workflows/govulncheck-scheduled.yml @@ -39,6 +39,7 @@ on: permissions: contents: read security-events: write # required to upload SARIF to the Security tab + issues: write # the failure-alarm step files an issue when a scheduled run breaks jobs: govulncheck-scheduled: @@ -98,3 +99,31 @@ jobs: with: sarif_file: 'govulncheck.sarif' category: 'govulncheck-scheduled' + + - name: File an issue so a red cron cannot rot silently + # A scheduled failure has no PR attached, so nothing surfaces it — the + # exact rot pattern that let the CodeQL toolchain break sit red for + # weeks. Files one issue per breakage (deduped by title), comments on + # re-failures. schedule-only: a red manual dispatch has a human watching. + # This step body is duplicated across the four scheduled scan lanes + # (codeql, semgrep, govulncheck, grype) — keep them in sync. + if: ${{ failure() && github.event_name == 'schedule' }} + env: + GH_TOKEN: ${{ github.token }} + WORKFLOW_NAME: ${{ github.workflow }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + set -euo pipefail + title="Scheduled ${WORKFLOW_NAME} run failed" + num="$(gh issue list --repo "$GITHUB_REPOSITORY" --state open \ + --search "\"$title\" in:title" --json number --jq '.[0].number // empty' || true)" + if [ -n "$num" ]; then + gh issue comment "$num" --repo "$GITHUB_REPOSITORY" \ + --body "Still failing: $RUN_URL" + else + gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" \ + --body "The scheduled $WORKFLOW_NAME lane went red: $RUN_URL + + A failing cron has no PR to make it visible, so this issue is the + alarm. Close it when the lane is green again." + fi diff --git a/.github/workflows/grype-scheduled.yml b/.github/workflows/grype-scheduled.yml index b81ef152..b2357c52 100644 --- a/.github/workflows/grype-scheduled.yml +++ b/.github/workflows/grype-scheduled.yml @@ -22,6 +22,7 @@ on: permissions: contents: read security-events: write # required to upload SARIF to the Security tab + issues: write # the failure-alarm step files an issue when a scheduled run breaks jobs: grype-scheduled: @@ -86,3 +87,31 @@ jobs: with: sarif_file: 'grype-results.sarif' category: 'grype-scheduled' + + - name: File an issue so a red cron cannot rot silently + # A scheduled failure has no PR attached, so nothing surfaces it — the + # exact rot pattern that let the CodeQL toolchain break sit red for + # weeks. Files one issue per breakage (deduped by title), comments on + # re-failures. schedule-only: a red manual dispatch has a human watching. + # This step body is duplicated across the four scheduled scan lanes + # (codeql, semgrep, govulncheck, grype) — keep them in sync. + if: ${{ failure() && github.event_name == 'schedule' }} + env: + GH_TOKEN: ${{ github.token }} + WORKFLOW_NAME: ${{ github.workflow }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + set -euo pipefail + title="Scheduled ${WORKFLOW_NAME} run failed" + num="$(gh issue list --repo "$GITHUB_REPOSITORY" --state open \ + --search "\"$title\" in:title" --json number --jq '.[0].number // empty' || true)" + if [ -n "$num" ]; then + gh issue comment "$num" --repo "$GITHUB_REPOSITORY" \ + --body "Still failing: $RUN_URL" + else + gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" \ + --body "The scheduled $WORKFLOW_NAME lane went red: $RUN_URL + + A failing cron has no PR to make it visible, so this issue is the + alarm. Close it when the lane is green again." + fi diff --git a/.github/workflows/semgrep.yml b/.github/workflows/semgrep.yml index 0ba90fa0..2b632151 100644 --- a/.github/workflows/semgrep.yml +++ b/.github/workflows/semgrep.yml @@ -52,6 +52,17 @@ # # Adding a suppression is therefore a reviewable act: it shows up in the diff # next to a reason, which is the property this whole lane exists to have. +# +# RULE PINNING — investigated and REJECTED, on license grounds, not neglect. +# The packs are fetched from the registry at scan time, so a registry-side rule +# addition can turn this gate red with no commit to blame (named in +# docs/SCANNING.md as the first suspect for a mystery red run). Vendoring the +# rule files would fix that, but the Semgrep Rules License v1.0 +# (https://semgrep.dev/legal/rules-license) grants use for "your own internal +# business purposes" only and states outright: "This license does not allow you +# to distribute the rules". Committing them into this public MIT repo would be +# redistribution. The semgrep BINARY version is pinned; the rules deliberately +# are not, with the failure mode documented instead of hidden. name: Semgrep on: @@ -202,3 +213,38 @@ jobs: path: semgrep.json if-no-files-found: warn retention-days: 14 + + cron-failure-alarm: + name: File an issue so a red cron cannot rot silently + # Only exists for the standalone weekly run: in the workflow_call path the + # caller's gate (`CI gate` / `Dev gate`) is the alarm, and this job's + # schedule-only condition keeps it skipped there (a skip which also means + # the caller never needs to grant issues: write — called-workflow + # permissions intersect with the caller's grant). + # Body duplicated across the four scheduled scan lanes — keep in sync. + if: ${{ failure() && github.event_name == 'schedule' }} + needs: [semgrep] + runs-on: ubuntu-latest + permissions: + issues: write + steps: + - name: File or update the alarm issue + env: + GH_TOKEN: ${{ github.token }} + WORKFLOW_NAME: ${{ github.workflow }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + set -euo pipefail + title="Scheduled ${WORKFLOW_NAME} run failed" + num="$(gh issue list --repo "$GITHUB_REPOSITORY" --state open \ + --search "\"$title\" in:title" --json number --jq '.[0].number // empty' || true)" + if [ -n "$num" ]; then + gh issue comment "$num" --repo "$GITHUB_REPOSITORY" \ + --body "Still failing: $RUN_URL" + else + gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" \ + --body "The scheduled $WORKFLOW_NAME lane went red: $RUN_URL + + A failing cron has no PR to make it visible, so this issue is the + alarm. Close it when the lane is green again." + fi diff --git a/AGENTS.md b/AGENTS.md index 31bf6de1..5466eec8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,7 +40,7 @@ cd web && npx playwright test --project=mocked # mocked e2e CI mirrors all of this — Go build/vet/lint/test (including a `-race` lane) plus a `govulncheck` dependency-CVE scan, a Grype container-image CVE scan (fail on a -fixable CRITICAL) of the sandbox image, a Python lint (ruff), web lint (oxlint) / typecheck (TS 7) / test / build, Playwright (mocked +fixable CRITICAL/HIGH) of the sandbox image, a Python lint (ruff), web lint (oxlint) / typecheck (TS 7) / test / build, Playwright (mocked **and** live, against a real backend + sandbox), a migration DDL lint, and a gitleaks secret scan. **Every job must be green before merge.** Tests are deterministic without a live model: use the fake-LLM seam (`internal/fakellm` diff --git a/CHANGELOG.md b/CHANGELOG.md index 7bb64ee8..c5aee729 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,54 @@ prior versions are listed because none have shipped. ### Fixed +- **Every remaining scanner follow-up closed: rule families widened and fixed, + CodeQL at `security-extended`, grype tightened, override canary, cron alarms, + and one reasoned rejection.** + + - **ruff `B`/`SIM`/`S` (bandit) enabled after fixing all 21 measured + findings.** Both `zip()` sites got `strict=True` — each provably + equal-length (one appends to both lists in lockstep; the other sits behind + an explicit `len(row) != len(cols)` guard) — so a future desync fails loud + instead of silently truncating. The unclosed `NamedTemporaryFile` in + bento_doc.py moved inside its `with`. Fourteen deliberate best-effort + `try/except-pass` sites (kernel cleanup, duck-typed pandas/numpy probes, + unlink-on-failure paths) became explicit `contextlib.suppress` with the + intent stated at each. The one `subprocess.Popen` carries a reasoned + `# noqa: S603` (argv is `sys.executable` plus internal literals; + mutation-tested — stripping the noqa re-fires the rule). Full Go suite + green on the result; the sandbox fileops and bridge behavior is covered by + its tests. + + - **CodeQL widened to the `security-extended` suite** on all four languages, + adopted the same way everything else was: the default suite measured zero, + so the broader set starts from a clean baseline and its findings on this + PR's own run are the measurement. + + - **Grype gate tightened to fixable CRITICAL + HIGH**, after measuring: the + published sandbox image carries zero fixable Critical/High RPM findings + (its only fixable findings are two Medium openssh advisories, which the + next routine image rebuild picks up). Policy change mutation-tested in + three directions: real scan passes, injected fixable High fails, injected + fixable Medium still passes. + + - **`scripts/check-npm-overrides.sh`**: the rampart sharp/adm-zip overrides + are forks of upstream's intent, correct only while upstream is broken — so + both CI lanes now fail with removal instructions the day + `@huggingface/transformers` / `onnxruntime-node` publish ranges reaching + the patched lines. Registry flake = skip with a notice, never a verdict. + Mutation-tested in both directions. + + - **A red scheduled scan files an issue** (all four lanes: CodeQL, Semgrep, + govulncheck, grype; deduped by title, re-failures comment). A cron failure + has no PR to surface it — the rot pattern that let the CodeQL toolchain + break sit red for weeks. + + - **Semgrep rule vendoring investigated and rejected on license grounds**: + the Semgrep Rules License v1.0 permits internal use only and states "This + license does not allow you to distribute the rules" — committing them to + this public MIT repo would be redistribution. The binary stays pinned; the + rules stay registry-fetched with the failure mode documented. + - **The scanners gate through `ci-gate`/`Dev gate` themselves, npm dependencies are audited, and the whole Python tree is ruff-formatted — with every finding fixed, none deferred.** diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5c89ee51..8bb50ef1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -103,7 +103,7 @@ Every pull request must be green before merge. CI runs: - **Playwright** — the mocked suite, plus a live suite against a real backend with a stubbed LLM (no OpenRouter spend). - **Secret scan (gitleaks)** — fails the build on any new, un-ignored secret. -- **Container image scan (Grype)** — fails the build on a fixable CRITICAL CVE in +- **Container image scan (Grype)** — fails the build on a fixable CRITICAL or HIGH CVE in the sandbox image built from `config/default/sandbox/Containerfile` (HIGH and below are reported, not blocking). Findings upload to GitHub Security → Code scanning. A separate weekly scheduled scan (non-blocking) catches new CVEs diff --git a/docs/CODEQL.md b/docs/CODEQL.md index 42f10b5a..f80b3670 100644 --- a/docs/CODEQL.md +++ b/docs/CODEQL.md @@ -85,8 +85,11 @@ enough", not "unset the pin". Neither `env: GOTOOLCHAIN: auto` nor | `javascript-typescript` | `none` | security (default suite) | | `actions` | `none` | security (default suite) | -**Security queries only.** The code-quality suite was enabled, measured, and then -deliberately removed — see "Why code quality was dropped" below. +**Security queries only — at the `security-extended` tier.** The code-quality +suite was enabled, measured, and deliberately removed (see "Why code quality +was dropped" below); the *security* side was then widened from the default +suite to `security-extended` once the default measured clean, so the broader +set also started from a zero baseline. `build-mode: none` is [not supported for Go](https://docs.github.com/en/code-security/reference/code-scanning/codeql/build-options-for-compiled-languages) diff --git a/docs/SANDBOX-IMAGE-FRESHNESS.md b/docs/SANDBOX-IMAGE-FRESHNESS.md index b44e8b35..5e6b5a71 100644 --- a/docs/SANDBOX-IMAGE-FRESHNESS.md +++ b/docs/SANDBOX-IMAGE-FRESHNESS.md @@ -14,7 +14,7 @@ serving sandbox images built 6–7 weeks earlier, on top of an equally old That matters because a container image is frozen at build time. An unchanged Containerfile does not stop the base layers and packages *inside* the built image from aging and accumulating **published, already-fixed CVEs**. CI's -Grype gate (fail on a fixable CRITICAL) scans a **fresh** build of the +Grype gate (fail on a fixable CRITICAL or HIGH) scans a **fresh** build of the Containerfile — only an on-box rebuild ever brings a deployed box up to what CI vouched for. diff --git a/docs/SCANNING.md b/docs/SCANNING.md index 6edd7a44..179d46aa 100644 --- a/docs/SCANNING.md +++ b/docs/SCANNING.md @@ -13,10 +13,10 @@ security queries only) and [`TESTING.md`](TESTING.md) (the rest of the ladder). | `oxlint` + `tsc` | web tier lint + types | ~5s | **blocks** (`ci-gate`) | job log | | **`ruff`** | **Python lint** | **~1s** | **blocks** (`ci-gate`) | job log | | `govulncheck` | Go dependency CVEs (called symbols) | ~30s | **blocks** (`ci-gate`) | job log + Security tab | -| `grype` | sandbox image CVEs (fixable CRITICAL) | ~1m | **blocks** (`ci-gate`) | job log + Security tab | +| `grype` | sandbox image CVEs (fixable **CRITICAL + HIGH**) | ~1m | **blocks** (`ci-gate`) | job log + Security tab | | `gitleaks` | secrets, every branch | ~10s | **blocks** (`ci-gate`) | job log | | **`npm audit`** | npm dependency CVEs (web + rampart-service) | ~5s | **blocks** (`ci-gate`) | job log | -| CodeQL | **interprocedural taint / security** | ~2m | **blocks** (`ci-gate`/`Dev gate` via workflow_call) | job log + Security tab | +| CodeQL | **interprocedural taint / `security-extended`** | ~2m | **blocks** (`ci-gate`/`Dev gate` via workflow_call) | job log + Security tab | | **Semgrep** | **Go/JS/Python SAST + Actions supply chain** | ~40s | **blocks** (`ci-gate`/`Dev gate` via workflow_call) | job log + artifact | Two things were added here (**ruff**, **Semgrep**) and one was narrowed @@ -41,10 +41,17 @@ That gap was real: CodeQL's quality suite found 28 Python issues, its single largest contribution anywhere. ruff finds the same class in under a second, with autofix, and now blocks. -The rule selection is deliberately narrow, and `ruff.toml` records the numbers: -default rules find **3** findings on this tree; a broad selection finds **333**, -of which 176 are `%`-format style, 43 magic values and 35 line length. Gating on -that would mean a whole-tree reformat for no correctness gain. +The rule selection is measured, and `ruff.toml` records the numbers. The +default rules (`E4,E7,E9,F`) found 3 real findings, fixed on day one. The +**`B`, `SIM` and `S` (bandit) families were then measured (21 findings), all 21 +fixed, and the families enabled**: both `zip()` sites got `strict=True` (each +provably equal-length), the unclosed `NamedTemporaryFile` moved into its +`with`, the deliberate best-effort `try/except-pass` sites became explicit +`contextlib.suppress` with the intent stated at each, and the one subprocess +launch carries a reasoned `# noqa: S603` (argv is `sys.executable` plus +internal literals — mutation-tested: stripping the noqa re-fires the rule). +The pure-style tiers stay off: they are ~330 findings of `%`-format and +line-length churn with no correctness content. Three real findings were fixed to make the gate clean on day one, so a new violation is a regression rather than noise in a backlog: @@ -71,7 +78,9 @@ So CodeQL keeps its security queries and gives up everything else — the qualit suite duplicated `golangci-lint`/`oxlint` for Go and JS, and ruff is a better fit for Python. Full reasoning and measurements in [`CODEQL.md`](CODEQL.md). -Its security suite currently reports **zero findings** on this tree, which is +It runs the **`security-extended`** suite — the broader security set, adopted +after the default suite measured clean — and currently reports **zero +findings** on this tree, which is what makes it safe to gate: a `Fail on findings` step now fails the job on any finding, so a red `Analyze (…)` check means the *code* has a problem rather than just "the scanner broke". That distinction is the whole reason the Go toolchain @@ -166,8 +175,14 @@ still pins `sharp ^0.34.5`, and npm's own suggested "fix" was a breaking vulnerable line). The overridden stack was **installed and load-tested**, not just resolved: sharp renders a PNG through the new libvips, transformers loads on it, rampart exports its API, and adm-zip 0.6 round-trips a zip. Audit result -after: 0 vulnerabilities in both trees. When upstream ships fixed ranges, the -overrides can be dropped. +after: 0 vulnerabilities in both trees. + +An override is a fork of upstream's intent, correct only while upstream is +broken — so `scripts/check-npm-overrides.sh` runs beside the audit in both +lanes and **fails the build the day upstream's own ranges reach the patched +lines**, with removal instructions. The reminder to drop the override is a red +build with a two-line fix, not stale-pin archaeology later. (Registry flake = +skip with a notice, never a verdict; mutation-tested in both directions.) ## Findings are readable from the job log, on purpose @@ -239,16 +254,15 @@ Stated rather than left for rediscovery: - **`_test.go` files are outside CodeQL's database** (621 files) — `autobuild` builds packages, not tests. Unchanged from default setup. -- **ruff's rule set is narrow**, so some real bug classes go unreported. - Measured: `--select B,SIM,S` adds **21** findings, of which the genuinely - interesting ones are 2 × `B905` (`zip()` without `strict=` — silent - truncation) and 1 × `SIM115` (file opened without a context manager). The - other 18 are `try`/`except`/`pass` in deliberate best-effort cleanup paths. - Worth a focused pass; not folded in here. -- **Semgrep's own rule packs are network-fetched** from the registry at scan - time. The semgrep *version* is pinned; the *rules* are not, so a registry - change can move findings without a diff here. This matters more now the lane - blocks: a registry-side rule addition can turn CI red with no commit to blame, - the same class of surprise `govulncheck-scheduled.yml` was created to absorb. - Vendoring the rules would fix it at the cost of never getting new ones. Left - as-is deliberately, and named here so a mystery red build has a first suspect. +- **Semgrep's rule packs are registry-fetched and cannot be pinned by + vendoring** — investigated and rejected on license grounds, not neglect. The + Semgrep Rules License v1.0 grants use for "your own internal business + purposes" and states: *"This license does not allow you to distribute the + rules."* Committing them to this public MIT repo would be redistribution. + The binary version is pinned; the rules are not, so a registry-side rule + addition can turn CI red with no commit to blame — named here so a mystery + red Semgrep run has a first suspect. +- **A red scheduled scan now files an issue** (all four scheduled lanes: + CodeQL, Semgrep, govulncheck, grype) — a cron failure has no PR to surface + it, which is the rot pattern that let the CodeQL toolchain break sit red for + weeks. Deduped by title; re-failures comment on the same issue. diff --git a/docs/TESTING.md b/docs/TESTING.md index 031877b9..dca9c52e 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -32,7 +32,7 @@ fix this doc (and the `make` targets) to match. | Go coverage | `go` | Coverage profile summarised in the log + job summary (advisory, no threshold) | `make test-cover` | | Go test -race | `go` | Race detector on the same suites | `make test-race` | | govulncheck | `go` | Dependency CVEs reachable from fleet | `make govulncheck` | -| Grype (image) | `grype-scan` | CVEs in the sandbox container image (fail on a fixable CRITICAL) | see below | +| Grype (image) | `grype-scan` | CVEs in the sandbox container image (fail on a fixable CRITICAL or HIGH) | see below | | Web lint/test/build | `web` | ESLint + vitest + `next build` | `make ci-web` | | Playwright (mocked) | `playwright` | Deterministic browser e2e, no backend | `make ci-e2e-mocked` | | Playwright (live) | `e2e-live` | Real stack + rootless-Podman sandbox, fake LLM | `npm run test:e2e:live` | diff --git a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py index 1d48ce2f..0f876038 100644 --- a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py +++ b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_doc.py @@ -28,6 +28,7 @@ """ import argparse +import contextlib import json import os import shutil @@ -541,21 +542,20 @@ def _write_atomic(path, data): leaves the original file exactly as it was. """ directory = os.path.dirname(os.path.abspath(path)) - fh = tempfile.NamedTemporaryFile( - dir=directory, prefix=".bento-", suffix=".tmp", delete=False - ) - tmp = fh.name + tmp = None try: - with fh: + with tempfile.NamedTemporaryFile( + dir=directory, prefix=".bento-", suffix=".tmp", delete=False + ) as fh: + tmp = fh.name fh.write(data) fh.flush() os.fsync(fh.fileno()) os.replace(tmp, path) except BaseException: - try: - os.unlink(tmp) - except OSError: - pass + if tmp is not None: + with contextlib.suppress(OSError): + os.unlink(tmp) raise diff --git a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py index b332b6bc..8ec13647 100644 --- a/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py +++ b/internal/clientconfig/builtin_skills/bento-slides/scripts/bento_pdf.py @@ -1601,7 +1601,7 @@ def render(self): pages.append(self.writer.reserve()) resources = self.res.dictionary() tree = self.writer.reserve() - for number_, content in zip(pages, contents): + for number_, content in zip(pages, contents, strict=True): self.writer.put( number_, "<< /Type /Page /Parent %d 0 R /MediaBox " @@ -2418,7 +2418,7 @@ def render_table(self, canvas, element, box): if not is_header and style.get("zebra") and body_index % 2 == 1: zebra = style.get("zebra") left = x - for column_index, (cell, lines, bold) in enumerate(laid): + for column_index, (cell, lines, _bold) in enumerate(laid): width = widths[column_index] height = heights[row_index] background = cell.get("bg") or ( diff --git a/internal/clientconfig/builtin_skills/data-profiler/scripts/profile.py b/internal/clientconfig/builtin_skills/data-profiler/scripts/profile.py index 348da265..dc8e6eb6 100644 --- a/internal/clientconfig/builtin_skills/data-profiler/scripts/profile.py +++ b/internal/clientconfig/builtin_skills/data-profiler/scripts/profile.py @@ -144,7 +144,7 @@ def profile_delimited(path, delimiter, max_rows, top): if len(row) != len(cols): malformed.append(reader.line_num) continue - for stats, cell in zip(cols, row): + for stats, cell in zip(cols, row, strict=True): stats.add(cell) kind = "tsv" if delimiter == "\t" else "csv" report(path, kind, total, cols, malformed, top) diff --git a/internal/sandbox/fileops.py b/internal/sandbox/fileops.py index 8a8fafe2..ba77c41e 100644 --- a/internal/sandbox/fileops.py +++ b/internal/sandbox/fileops.py @@ -12,6 +12,7 @@ # full-file hashes, optional expected_sha256, no-op rejection, and a bounded # unified diff. Those semantics must never regress the #784 confinement. import base64 +import contextlib import difflib import errno import hashlib @@ -284,16 +285,14 @@ def _atomic_write(parent_fd, name, data, expected=None, expected_digest=None): if owner is not None: current_owner = os.fstat(fd) if (current_owner.st_uid, current_owner.st_gid) != owner: - try: + # Best effort: preserving the destination's ownership on an + # overwrite is a nicety, not a safety property. The executor + # runs unprivileged and cannot chown to a foreign uid (e.g. + # a host-seeded workspace file mapping to container-root), so + # keep the executor-owned replacement rather than aborting a + # legitimate edit. + with contextlib.suppress(PermissionError): os.fchown(fd, owner[0], owner[1]) - except PermissionError: - # Best effort: preserving the destination's ownership on an - # overwrite is a nicety, not a safety property. The executor - # runs unprivileged and cannot chown to a foreign uid (e.g. - # a host-seeded workspace file mapping to container-root), - # so keep the executor-owned replacement rather than - # aborting a legitimate edit. - pass os.fchmod(fd, mode) os.fsync(fd) os.close(fd) @@ -316,10 +315,8 @@ def _atomic_write(parent_fd, name, data, expected=None, expected_digest=None): except BaseException: if fd >= 0: os.close(fd) - try: + with contextlib.suppress(OSError): os.unlink(tmp, dir_fd=parent_fd) - except OSError: - pass raise diff --git a/internal/tools/python_bridge.py b/internal/tools/python_bridge.py index 45aacf4d..86b12ed5 100644 --- a/internal/tools/python_bridge.py +++ b/internal/tools/python_bridge.py @@ -1,5 +1,6 @@ import atexit import base64 +import contextlib import binascii import datetime import json @@ -138,10 +139,8 @@ def reap_stale_kernels(): except OSError: continue if b"ipykernel_launcher" in cmdline: - try: + with contextlib.suppress(OSError): os.kill(pid, signal.SIGKILL) - except OSError: - pass def start_kernel(): @@ -172,16 +171,18 @@ def start_kernel(): # empty placeholder would make the while-loop below think the file # "exists" before the kernel has actually written to it, so we remove # our placeholder and let ipykernel create it fresh. - try: + with contextlib.suppress(OSError): os.unlink(connection_file) - except OSError: - pass # Start the kernel cmd = [sys.executable, "-m", "ipykernel_launcher", "-f", connection_file] # Start process detached to avoid signal interference - kernel_process = subprocess.Popen( + # S603 waived on the Popen below: every element of cmd is internal — + # sys.executable, literal flags, and a connection-file path this process + # just created with mkstemp in the OS temp dir. Nothing model- or + # user-controlled reaches the argv. + kernel_process = subprocess.Popen( # noqa: S603 cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, @@ -203,16 +204,15 @@ def cleanup(): """Kills the kernel and deletes the connection file.""" global kernel_process, connection_file if kernel_process: - try: + # Broad on purpose: the process group may already be gone (ProcessLookup), + # unkillable (Permission), or getpgid itself can fail on a reaped pid — + # cleanup must never raise past atexit. + with contextlib.suppress(Exception): os.killpg(os.getpgid(kernel_process.pid), signal.SIGTERM) - except Exception: - pass kernel_process = None if connection_file: - try: + with contextlib.suppress(OSError): os.unlink(connection_file) - except OSError: - pass connection_file = None @@ -337,11 +337,7 @@ def run_code_on_kernel(code, client, timeout_seconds=None): stdout_content.append(content["text"]) elif content["name"] == "stderr": stderr_content.append(content["text"]) - elif msg_type == "execute_result": - data = content.get("data", {}) - result_content.append(data.get("text/plain", "")) - collect_image(data, images) - elif msg_type == "display_data": + elif msg_type in ("execute_result", "display_data"): data = content.get("data", {}) result_content.append(data.get("text/plain", "")) collect_image(data, images) @@ -349,9 +345,8 @@ def run_code_on_kernel(code, client, timeout_seconds=None): traceback = content.get("traceback", []) error_content = strip_ansi("\n".join(traceback)) status = "error" - elif msg_type == "status": - if content["execution_state"] == "idle": - idle_seen = True + elif msg_type == "status" and content["execution_state"] == "idle": + idle_seen = True except queue.Empty: pass @@ -438,32 +433,27 @@ def normalize_json_value(value): return None if str(value) in {"", "NaT"}: return None + # Duck-typed pandas/numpy interop below: each probe is best-effort and a + # third-party accessor can raise anything, so the suppressions are broad on + # purpose — a failed probe falls through to the next representation. to_python = getattr(value, "to_pydatetime", None) if callable(to_python): - try: + with contextlib.suppress(Exception): return normalize_json_value(to_python()) - except Exception: - pass item = getattr(value, "item", None) if callable(item): - try: + with contextlib.suppress(Exception): extracted = item() if extracted is not value: return normalize_json_value(extracted) - except Exception: - pass to_list = getattr(value, "tolist", None) if callable(to_list): - try: + with contextlib.suppress(Exception): return normalize_json_value(to_list()) - except Exception: - pass isoformat = getattr(value, "isoformat", None) if callable(isoformat): - try: + with contextlib.suppress(Exception): return isoformat() - except Exception: - pass return value @@ -602,16 +592,15 @@ def execute_code( var_res = run_code_on_kernel(extract_script, client) if var_res["status"] == "success": - try: - with open(tmp_vars_file, "r", encoding="utf-8") as f: - vars_data = json.load(f) - except Exception: - pass # malformed/missing file → keep vars_data empty + # malformed/missing file → keep vars_data empty + with ( + contextlib.suppress(Exception), + open(tmp_vars_file, "r", encoding="utf-8") as f, + ): + vars_data = json.load(f) finally: - try: + with contextlib.suppress(OSError): os.unlink(tmp_vars_file) - except OSError: - pass # Consolidate legacy output field for backward compatibility final_output = "" diff --git a/ruff.toml b/ruff.toml index 63016279..ed18d40f 100644 --- a/ruff.toml +++ b/ruff.toml @@ -23,16 +23,24 @@ # length). Gating on that would mean a 300-commit reformat for no correctness # gain, so it is out of scope here rather than silently enabled. # -# Two rules were considered and deliberately REJECTED, both because their only -# findings in this tree are correct code: +# One rule was considered and deliberately REJECTED, because its only findings +# in this tree are correct code: # # PLR0124 (name compared with itself) — its 3 hits in bento_pdf.py are the # idiomatic NaN test (`value != value` is true only for NaN). CodeQL's # py/comparison-of-identical-expressions flagged the same 3. Enabling it # would mean three `# noqa` comments on correct code. -# S110 / SIM105 (try-except-pass) — 14 hits, mostly in python_bridge.py's -# best-effort cleanup paths where swallowing is the intent. Worth a separate -# review pass, not a gate that has to be suppressed 14 times on day one. +# +# B, SIM and S started here as measured-but-deferred (21 findings) and were then +# fixed and ENABLED rather than left as a documented backlog: the two zip() +# sites got strict=True (both provably equal-length — one appends to both lists +# in lockstep, the other is behind an explicit len(row) != len(cols) guard), the +# unclosed NamedTemporaryFile was restructured into its with-block, the +# deliberate best-effort try/except-pass sites became explicit +# contextlib.suppress with the intent stated at each, and the one subprocess +# call carries a reasoned `# noqa: S603` (argv is sys.executable plus internal +# literals; nothing model- or user-controlled). The bandit tier (S) firing on a +# NEW line is therefore a real question to answer, not pre-existing noise. # # The gate is therefore "clean today, and stays clean": three real findings were # fixed to get here (an unused import, a byte-identical duplicate function @@ -63,7 +71,7 @@ exclude = [ # E9 syntax/IO) plus Pyflakes (F — undefined names, unused imports, redefined # names, unused locals). This is the "is it actually broken" tier; see the # header for why the style tiers are not enabled. -select = ["E4", "E7", "E9", "F"] +select = ["E4", "E7", "E9", "F", "B", "SIM", "S"] [lint.per-file-ignores] # MCP test servers and testdata fixtures are deliberately minimal stand-ins — diff --git a/scripts/check-grype-policy.sh b/scripts/check-grype-policy.sh index 342dd637..b30ce8aa 100755 --- a/scripts/check-grype-policy.sh +++ b/scripts/check-grype-policy.sh @@ -1,5 +1,12 @@ #!/usr/bin/env bash -# Fail only for actionable CRITICAL vulnerabilities in Fedora RPMs. +# Fail for actionable CRITICAL and HIGH vulnerabilities in Fedora RPMs. +# +# High was added to the gate after measuring, not before: the published +# sandbox image at the time of the change carried zero fixable Critical or +# High RPM findings (its only fixable findings were two Medium openssh +# advisories), so the tightened gate started clean rather than arming over a +# backlog. Medium and below stay report-only — the image tracks +# fedora-minimal:latest, so routine rebuilds pick those up without a gate. # # Grype also catalogs Python dist-info shipped *by* Fedora RPMs as independent # PyPI packages. Those records use upstream versions/advisories and can claim a @@ -19,19 +26,19 @@ command -v jq >/dev/null 2>&1 || { echo "jq is required" >&2; exit 2; } filter='[ .matches[] - | select((.vulnerability.severity // "" | ascii_downcase) == "critical") + | select((.vulnerability.severity // "" | ascii_downcase) as $s | $s == "critical" or $s == "high") | select((.artifact.type // "") == "rpm") | select(((.vulnerability.fix.versions // []) | length) > 0) ] | unique_by([.vulnerability.id, .artifact.name, .artifact.version])' findings="$(jq -r "$filter"'[] | [.vulnerability.id, .artifact.name, .artifact.version, (.vulnerability.fix.versions | join(","))] | @tsv' "$report")" if [[ -z "$findings" ]]; then - echo "Grype policy: no fixable CRITICAL Fedora RPM findings." + echo "Grype policy: no fixable CRITICAL or HIGH Fedora RPM findings." exit 0 fi count="$(awk 'NF { count++ } END { print count + 0 }' <<<"$findings")" -echo "Grype policy: $count fixable CRITICAL Fedora RPM finding(s):" >&2 +echo "Grype policy: $count fixable CRITICAL/HIGH Fedora RPM finding(s):" >&2 awk -F $'\t' '{ printf " %s %s %s fix: %s\n", $1, $2, $3, $4 }' <<<"$findings" >&2 echo "Rebuild against Fedora latest or update the affected RPM; do not shadow it with an ad-hoc language-package pin." >&2 exit 1 diff --git a/scripts/check-npm-overrides.sh b/scripts/check-npm-overrides.sh new file mode 100755 index 00000000..81fb4cd7 --- /dev/null +++ b/scripts/check-npm-overrides.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# check-npm-overrides.sh — fail when a security override has become droppable. +# +# scripts/rampart-service/package.json carries two `overrides` that force +# patched versions of transitive dependencies whose parents have not released a +# fix yet: +# +# sharp ^0.35.3 — @huggingface/transformers pins sharp ^0.34.5, which +# carries the libvips CVEs (CVE-2026-33327/-33328/-35590/ +# -35591, GHSA-f88m-g3jw-g9cj). +# adm-zip ^0.6.0 — onnxruntime-node pins adm-zip ^0.5.16, which carries +# GHSA-xcpc-8h2w-3j85 (crafted-ZIP 4 GB allocation). +# +# An override is a fork of upstream's intent: correct while upstream is broken, +# and pure drift the day upstream fixes itself — at which point Dependabot's +# normal updates are silently pinned down by us instead. Nothing else notices +# that day. This check does: it asks the registry what floor each PARENT now +# declares, and FAILS with removal instructions once the parent's own range +# reaches the patched line. So the reminder to drop the override is a red build +# with a two-line fix, not a stale-pin archaeology session years later. +# +# Registry unreachable / output unparsable is a SKIP with a notice, never a +# failure: this check's job is "tell me when the override is droppable", and a +# network flake is not evidence of that. The vulnerability gate itself is +# `npm audit` in the same CI job, which does fail closed on its own findings. +set -uo pipefail + +# floor RANGE -> x.y.z: the minimum version of a caret/tilde/plain range. +# The parents' published ranges are simple ("^0.34.5"); anything fancier +# parses to "unknown" and is treated as a skip, not a verdict. +floor() { + local r="${1#\^}"; r="${r#~}"; r="${r#>=}" + if [[ "$r" =~ ^([0-9]+)\.([0-9]+)\.([0-9]+) ]]; then + printf '%s.%s.%s' "${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}" "${BASH_REMATCH[3]}" + else + printf 'unknown' + fi +} + +# ge A B — true when version A >= B (numeric, three components). +ge() { + local IFS=. a b + read -ra a <<<"$1"; read -ra b <<<"$2" + for i in 0 1 2; do + if (( ${a[$i]:-0} > ${b[$i]:-0} )); then return 0; fi + if (( ${a[$i]:-0} < ${b[$i]:-0} )); then return 1; fi + done + return 0 +} + +# check PARENT DEP PATCHED_FLOOR — fail if PARENT's declared range for DEP now +# starts at or above PATCHED_FLOOR (the override is then droppable). +stale=0 +check() { + local parent="$1" dep="$2" patched="$3" range f + range="$(npm view "$parent@latest" "dependencies.$dep" 2>/dev/null || true)" + if [ -z "$range" ]; then + echo "notice: could not read $parent's $dep range from the registry — skipping (not a verdict)." + return 0 + fi + f="$(floor "$range")" + if [ "$f" = "unknown" ]; then + echo "notice: $parent declares $dep '$range' — cannot parse a floor, skipping." + return 0 + fi + if ge "$f" "$patched"; then + echo "::error::$parent@latest now declares $dep '$range' (floor $f >= $patched):" + echo " the '$dep' override in scripts/rampart-service/package.json is DROPPABLE." + echo " Remove it, regenerate package-lock.json (npm install --package-lock-only)," + echo " and re-run npm audit — upstream now ships the patched line itself." + stale=1 + else + echo "override for $dep still required: $parent@latest pins '$range' (floor $f < $patched)." + fi +} + +check "@huggingface/transformers" "sharp" "0.35.0" +check "onnxruntime-node" "adm-zip" "0.6.0" + +exit "$stale" From 47a1e588840c937fb3638fbd3f31825550f12d4e Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 14:40:06 +0000 Subject: [PATCH 13/34] Move the CodeQL/Semgrep cron alarm to a workflow_run watcher MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit put an `issues: write` alarm job inside codeql.yml and semgrep.yml. Those are REUSABLE workflows, and a called workflow may not request token permissions its caller did not grant — and that check fires at PLAN time, before any `if: github.event_name == 'schedule'` can skip the job. Result: the calling Dev CI run on 745fe6d died with startup_failure (run 32578976517) and NOTHING scanned on that head. The "intersection semantics" assumption in that commit's comment was wrong, and this is the correction. The alarm for those two lanes now lives in scan-cron-alarm.yml, a workflow_run watcher on [CodeQL, Semgrep] completions, filtered to conclusion=failure AND event=schedule. A watcher has no caller, so it holds issues: write without widening any gate's token — a PR-path scanner failure already reddens the calling gate, and a red manual dispatch has a human watching it. govulncheck-scheduled.yml and grype-scheduled.yml keep their in-job steps: standalone workflows, no caller, no plan-time constraint. workflow_run only fires from the default branch's copy, so the alarm arms at the dev->main promotion — the same moment the crons start mattering. Signed-off-by: Brad Flaugher --- .github/workflows/codeql.yml | 35 --------------- .github/workflows/scan-cron-alarm.yml | 65 +++++++++++++++++++++++++++ .github/workflows/semgrep.yml | 35 --------------- CHANGELOG.md | 13 ++++-- docs/SCANNING.md | 14 ++++-- 5 files changed, 84 insertions(+), 78 deletions(-) create mode 100644 .github/workflows/scan-cron-alarm.yml diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index a8d9ec83..9200ca09 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -323,38 +323,3 @@ jobs: for r in $results; do [ "$r" = "success" ] || { echo "a CodeQL analysis did not succeed"; exit 1; } done - - cron-failure-alarm: - name: File an issue so a red cron cannot rot silently - # Only exists for the standalone weekly run: in the workflow_call path the - # caller's gate (`CI gate` / `Dev gate`) is the alarm, and this job's - # schedule-only condition keeps it skipped there (a skip which also means - # the caller never needs to grant issues: write — called-workflow - # permissions intersect with the caller's grant). - # Body duplicated across the four scheduled scan lanes — keep in sync. - if: ${{ failure() && github.event_name == 'schedule' }} - needs: [analyze] - runs-on: ubuntu-latest - permissions: - issues: write - steps: - - name: File or update the alarm issue - env: - GH_TOKEN: ${{ github.token }} - WORKFLOW_NAME: ${{ github.workflow }} - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - run: | - set -euo pipefail - title="Scheduled ${WORKFLOW_NAME} run failed" - num="$(gh issue list --repo "$GITHUB_REPOSITORY" --state open \ - --search "\"$title\" in:title" --json number --jq '.[0].number // empty' || true)" - if [ -n "$num" ]; then - gh issue comment "$num" --repo "$GITHUB_REPOSITORY" \ - --body "Still failing: $RUN_URL" - else - gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" \ - --body "The scheduled $WORKFLOW_NAME lane went red: $RUN_URL - - A failing cron has no PR to make it visible, so this issue is the - alarm. Close it when the lane is green again." - fi diff --git a/.github/workflows/scan-cron-alarm.yml b/.github/workflows/scan-cron-alarm.yml new file mode 100644 index 00000000..ac12b94a --- /dev/null +++ b/.github/workflows/scan-cron-alarm.yml @@ -0,0 +1,65 @@ +# Scan cron alarm — files an issue when a SCHEDULED CodeQL or Semgrep run fails. +# +# WHY THIS IS ITS OWN WORKFLOW rather than a job inside codeql.yml/semgrep.yml: +# those two are REUSABLE workflows called by ci.yml and dev-ci.yml, and a called +# workflow may not request token permissions its caller did not grant — the +# check happens at PLAN time, before any `if:` can skip the job. Verified the +# hard way: adding an `issues: write` alarm job inside them failed the entire +# calling Dev CI run with `startup_failure` (run 32578976517), which meant NO +# scanning ran on that head at all. A `workflow_run` watcher has no caller, so +# it can hold `issues: write` without widening any gate's token. +# +# govulncheck-scheduled.yml and grype-scheduled.yml keep their in-job alarm +# steps: they are standalone workflows with no callers, so the constraint above +# does not apply to them. +# +# Why an alarm exists at all: a scheduled failure has no PR attached, so nothing +# surfaces it — the exact rot pattern that let the CodeQL toolchain break sit +# red for weeks. One issue per breakage (deduped by title); re-failures comment +# on the same issue. +# +# workflow_run only fires from this file's copy on the DEFAULT branch, so the +# alarm arms once this merges to main — which is also when the crons themselves +# start mattering. +name: Scan cron alarm + +on: + workflow_run: + workflows: [CodeQL, Semgrep] + types: [completed] + +permissions: + issues: write + +jobs: + alarm: + name: File an issue so a red cron cannot rot silently + # Scheduled failures only: a red workflow_call run already reddens the + # calling gate on a PR, and a red manual dispatch has a human watching it. + if: >- + github.event.workflow_run.conclusion == 'failure' && + github.event.workflow_run.event == 'schedule' + runs-on: ubuntu-latest + steps: + - name: File or update the alarm issue + # Body mirrors the in-job alarm steps in govulncheck-scheduled.yml and + # grype-scheduled.yml — keep the three in sync. + env: + GH_TOKEN: ${{ github.token }} + WORKFLOW_NAME: ${{ github.event.workflow_run.name }} + RUN_URL: ${{ github.event.workflow_run.html_url }} + run: | + set -euo pipefail + title="Scheduled ${WORKFLOW_NAME} run failed" + num="$(gh issue list --repo "$GITHUB_REPOSITORY" --state open \ + --search "\"$title\" in:title" --json number --jq '.[0].number // empty' || true)" + if [ -n "$num" ]; then + gh issue comment "$num" --repo "$GITHUB_REPOSITORY" \ + --body "Still failing: $RUN_URL" + else + gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" \ + --body "The scheduled $WORKFLOW_NAME lane went red: $RUN_URL + + A failing cron has no PR to make it visible, so this issue is the + alarm. Close it when the lane is green again." + fi diff --git a/.github/workflows/semgrep.yml b/.github/workflows/semgrep.yml index 2b632151..50c0a8f5 100644 --- a/.github/workflows/semgrep.yml +++ b/.github/workflows/semgrep.yml @@ -213,38 +213,3 @@ jobs: path: semgrep.json if-no-files-found: warn retention-days: 14 - - cron-failure-alarm: - name: File an issue so a red cron cannot rot silently - # Only exists for the standalone weekly run: in the workflow_call path the - # caller's gate (`CI gate` / `Dev gate`) is the alarm, and this job's - # schedule-only condition keeps it skipped there (a skip which also means - # the caller never needs to grant issues: write — called-workflow - # permissions intersect with the caller's grant). - # Body duplicated across the four scheduled scan lanes — keep in sync. - if: ${{ failure() && github.event_name == 'schedule' }} - needs: [semgrep] - runs-on: ubuntu-latest - permissions: - issues: write - steps: - - name: File or update the alarm issue - env: - GH_TOKEN: ${{ github.token }} - WORKFLOW_NAME: ${{ github.workflow }} - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - run: | - set -euo pipefail - title="Scheduled ${WORKFLOW_NAME} run failed" - num="$(gh issue list --repo "$GITHUB_REPOSITORY" --state open \ - --search "\"$title\" in:title" --json number --jq '.[0].number // empty' || true)" - if [ -n "$num" ]; then - gh issue comment "$num" --repo "$GITHUB_REPOSITORY" \ - --body "Still failing: $RUN_URL" - else - gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" \ - --body "The scheduled $WORKFLOW_NAME lane went red: $RUN_URL - - A failing cron has no PR to make it visible, so this issue is the - alarm. Close it when the lane is green again." - fi diff --git a/CHANGELOG.md b/CHANGELOG.md index c5aee729..93a7a56b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -56,10 +56,15 @@ prior versions are listed because none have shipped. the patched lines. Registry flake = skip with a notice, never a verdict. Mutation-tested in both directions. - - **A red scheduled scan files an issue** (all four lanes: CodeQL, Semgrep, - govulncheck, grype; deduped by title, re-failures comment). A cron failure - has no PR to surface it — the rot pattern that let the CodeQL toolchain - break sit red for weeks. + - **A red scheduled scan files an issue** (all four lanes; deduped by title, + re-failures comment). A cron failure has no PR to surface it — the rot + pattern that let the CodeQL toolchain break sit red for weeks. For the two + reusable workflows the alarm lives in `scan-cron-alarm.yml`, a + `workflow_run` watcher, because a called workflow may not request + permissions its caller did not grant — the check fires at PLAN time, before + any `if:` can skip the job, and the first attempt (an `issues: write` job + inside codeql.yml/semgrep.yml) startup-failed the entire calling Dev CI + run. Verified fixed on the next run. - **Semgrep rule vendoring investigated and rejected on license grounds**: the Semgrep Rules License v1.0 permits internal use only and states "This diff --git a/docs/SCANNING.md b/docs/SCANNING.md index 179d46aa..e8bf5bba 100644 --- a/docs/SCANNING.md +++ b/docs/SCANNING.md @@ -262,7 +262,13 @@ Stated rather than left for rediscovery: The binary version is pinned; the rules are not, so a registry-side rule addition can turn CI red with no commit to blame — named here so a mystery red Semgrep run has a first suspect. -- **A red scheduled scan now files an issue** (all four scheduled lanes: - CodeQL, Semgrep, govulncheck, grype) — a cron failure has no PR to surface - it, which is the rot pattern that let the CodeQL toolchain break sit red for - weeks. Deduped by title; re-failures comment on the same issue. +- **A red scheduled scan now files an issue** (all four scheduled lanes) — a + cron failure has no PR to surface it, which is the rot pattern that let the + CodeQL toolchain break sit red for weeks. Deduped by title; re-failures + comment on the same issue. Mechanism differs by necessity: govulncheck and + grype carry an in-job step, while CodeQL and Semgrep are watched by + `scan-cron-alarm.yml` (a `workflow_run` watcher) — because a CALLED workflow + may not request permissions its caller did not grant, and the check fires at + plan time before any `if:` can skip the job. Learned by breaking it: an + `issues: write` alarm job inside the called workflows startup-failed the + entire calling Dev CI run. From 06ad52a8236d3b8660244c1e76388a9468c95f88 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 14:53:09 +0000 Subject: [PATCH 14/34] Harden fleet_ref against fork-PR refs; fix the canary path; file:line in summaries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three fixes from reading run 32579378165, the first run where the security-extended measurement actually executed. THE EXTENDED SUITE'S FIRST CATCH, AND ITS UNFLAGGED TWIN. Extended found one actions finding: actions/untrusted-checkout/medium at build-sandbox-image.yml's "Checkout fleet (build script)" step, whose `ref: inputs.fleet_ref` is caller-controlled — and the checked-out script is then EXECUTED. Reproduced locally with the same CodeQL 2.26.3 bundle and suite to get the exact location, then triaged for real: every ref of ElcanoTek/fleet is collaborator-written EXCEPT refs/pull/* (fork PRs), so a caller passing a pull-request ref would execute non-collaborator code — with a contents:read token of the calling repo, which for a private bundle repo is an exfiltration primitive. The fix is a validation step that refuses pull-request refs (and leading-dash values) and exposes the vetted value as a step output the checkout consumes; the query's trigger was a NAME heuristic (any ref: fed by a field matching .*(head|branch|ref).*), so consuming the validator's neutrally named output also clears the alert honestly — the sanitizer is genuinely in the path, not renamed around. The better half of the catch: publish-sandbox-image.yml has the IDENTICAL pattern and escaped BOTH query variants — too privileged for medium (which only reports non-privileged contexts) and no PR-event taint for high — while holding packages:write, making it the more dangerous twin. Hardened symmetrically. Verified: rebuilding the actions database and re-running the full security-extended suite locally now reports 0 findings across all 13 workflow files, and semgrep stays clean on both edited files. CANARY PATH. scripts/check-npm-overrides.sh was invoked repo-relative from the web jobs, whose default working-directory is web/ — exit 127, exactly what the gate is for. The script is cwd-free (registry queries only), so it is now invoked via $GITHUB_WORKSPACE. (The rampart audit step beside it already proved step-level working-directory resolves from the workspace root.) SUMMARIES NOW PRINT file:line PER FINDING. Run 32579378165's summary named the rule but not the location, which sent the fix hunt through a 600MB CLI bundle download. The jq now renders "[level] ruleId file:line" per finding, validated against a location-bearing SARIF fixture and the empty case. security-extended status after this commit: go, python and javascript-typescript measured clean in CI on the previous run; actions measured clean locally on the same toolchain after the hardening. All four verified in CI on this run. Signed-off-by: Brad Flaugher --- .github/workflows/build-sandbox-image.yml | 24 ++++++++++++++++++++- .github/workflows/ci.yml | 4 +++- .github/workflows/codeql.yml | 10 +++++---- .github/workflows/dev-ci.yml | 4 +++- .github/workflows/publish-sandbox-image.yml | 24 ++++++++++++++++++++- 5 files changed, 58 insertions(+), 8 deletions(-) diff --git a/.github/workflows/build-sandbox-image.yml b/.github/workflows/build-sandbox-image.yml index e4440218..b44ada1c 100644 --- a/.github/workflows/build-sandbox-image.yml +++ b/.github/workflows/build-sandbox-image.yml @@ -103,11 +103,33 @@ jobs: # The build script lives in fleet core; a client-config caller doesn't # have it. Checking it out into a subdir keeps ONE canonical builder # (same manifest parsing, same flags) instead of a drifting copy per repo. + # fleet_ref chooses which ref of ElcanoTek/fleet the BUILD SCRIPT below is + # taken from — and that script is then EXECUTED. Every ref in this repo is + # collaborator-written except refs/pull/* (fork PRs), so those are exactly + # the refs that would let non-collaborator code run here; refuse them + # before the checkout. The checkout then consumes this step's validated + # output rather than the raw input. (Found via CodeQL + # actions/untrusted-checkout under security-extended; the same pattern in + # publish-sandbox-image.yml was hardened symmetrically even though the + # query's privileged/taint split happened to flag neither variant there.) + - name: Pin fleet_ref to collaborator-controlled refs + id: pin + env: + REQUESTED: ${{ inputs.fleet_ref || 'main' }} + run: | + set -euo pipefail + case "$REQUESTED" in + refs/pull/*|pull/*|-*) + echo "::error::fleet_ref '$REQUESTED' is refused: pull-request refs carry fork-controlled code, and this workflow executes the checked-out build script." + exit 1 ;; + esac + printf 'resolved=%s\n' "$REQUESTED" >> "$GITHUB_OUTPUT" + - name: Checkout fleet (build script) uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: repository: ElcanoTek/fleet - ref: ${{ inputs.fleet_ref || 'main' }} + ref: ${{ steps.pin.outputs.resolved }} path: .fleet-core sparse-checkout: | scripts/build-sandbox-image.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 216ebaac..99bf8d60 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -426,7 +426,9 @@ jobs: # this FAILS with removal instructions — an override left behind after # upstream fixes itself silently pins Dependabot's updates down. A # registry flake skips with a notice (the audit above is the CVE gate). - run: scripts/check-npm-overrides.sh + # Absolute path: this job's default working-directory is web/, which is + # exactly how run 32579378165 caught the repo-relative form (exit 127). + run: "$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh" - name: Install dependencies run: npm ci diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 9200ca09..00651bec 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -234,10 +234,12 @@ jobs: | if ($res | length) == 0 then "No findings." else ( $res - | group_by(.ruleId) - | sort_by(-length) - | map("\(length | tostring) [\(.[0].level // "note")] \(.[0].ruleId)") - | join("\n") + | map( + "[\(.level // "note")] \(.ruleId) " + + ((.locations[0].physicalLocation // {}) as $l + | "\($l.artifactLocation.uri // "?"):\($l.region.startLine // "?")") + ) + | sort | join("\n") ) + "\n--\ntotal findings: \($res | length)" end ' "${files[@]}" diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index 9f241114..b4153e6f 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -224,7 +224,9 @@ jobs: # this FAILS with removal instructions — an override left behind after # upstream fixes itself silently pins Dependabot's updates down. A # registry flake skips with a notice (the audit above is the CVE gate). - run: scripts/check-npm-overrides.sh + # Absolute path: this job's default working-directory is web/, which is + # exactly how run 32579378165 caught the repo-relative form (exit 127). + run: "$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh" - name: Install dependencies run: npm ci diff --git a/.github/workflows/publish-sandbox-image.yml b/.github/workflows/publish-sandbox-image.yml index 3e1f739a..8be79186 100644 --- a/.github/workflows/publish-sandbox-image.yml +++ b/.github/workflows/publish-sandbox-image.yml @@ -220,11 +220,33 @@ jobs: # The build script lives in fleet core; a client-config caller doesn't # have it. Checking it out into a subdir keeps ONE canonical builder # (same manifest parsing, same flags) instead of a drifting copy per repo. + # fleet_ref chooses which ref of ElcanoTek/fleet the BUILD SCRIPT below is + # taken from — and that script is then EXECUTED. Every ref in this repo is + # collaborator-written except refs/pull/* (fork PRs), so those are exactly + # the refs that would let non-collaborator code run here; refuse them + # before the checkout. The checkout then consumes this step's validated + # output rather than the raw input. (Found via CodeQL + # actions/untrusted-checkout under security-extended; the same pattern in + # build-sandbox-image.yml was flagged; this file escaped both query + # variants despite holding packages: write — the MORE dangerous twin.) + - name: Pin fleet_ref to collaborator-controlled refs + id: pin + env: + REQUESTED: ${{ inputs.fleet_ref || 'main' }} + run: | + set -euo pipefail + case "$REQUESTED" in + refs/pull/*|pull/*|-*) + echo "::error::fleet_ref '$REQUESTED' is refused: pull-request refs carry fork-controlled code, and this workflow executes the checked-out build script." + exit 1 ;; + esac + printf 'resolved=%s\n' "$REQUESTED" >> "$GITHUB_OUTPUT" + - name: Checkout fleet (build script) uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: repository: ElcanoTek/fleet - ref: ${{ inputs.fleet_ref || 'main' }} + ref: ${{ steps.pin.outputs.resolved }} path: .fleet-core sparse-checkout: | scripts/build-sandbox-image.sh From 39b4743291c6eae9b76b6e23cc177dd9e1358a19 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 15:05:39 +0000 Subject: [PATCH 15/34] Record the extended-suite verification and the untrusted-checkout fix in the docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Docs-only follow-up to the last workflow commit, which shipped the changes but not their story: - docs/CODEQL.md: the matrix table now says security-extended (it still read "default suite"); the adoption section records the one finding the wider suite produced (actions/untrusted-checkout/medium on build-sandbox-image.yml), why it was fixed rather than waived (the actions language has no AlertSuppression.ql), the refs/pull/* refusal that fixes it, and the same hardening applied to publish-sandbox-image.yml — the unflagged twin with packages: write that the name-heuristic query missed. Verified clean in CI on all four languages (Dev CI run 525, id 32580031374). - docs/SCANNING.md: same story in the stack doc; the job-log example updated to the shipped file:line format plus the database-file-count coverage line; the override canary's $GITHUB_WORKSPACE invocation explained (the step runs under working-directory: web, where a repo-relative path exits 127). - CHANGELOG.md: the security-extended bullet now carries the finding, the fix in both workflows, the CI verification, and the summary/canary fixes. go test ./scripts green (the docs↔pin agreement assertions still hold). Signed-off-by: Brad Flaugher --- CHANGELOG.md | 15 ++++++++++++++- docs/CODEQL.md | 30 ++++++++++++++++++++++++++---- docs/SCANNING.md | 33 +++++++++++++++++++++++++++------ 3 files changed, 67 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 93a7a56b..61ac4943 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,7 +40,20 @@ prior versions are listed because none have shipped. - **CodeQL widened to the `security-extended` suite** on all four languages, adopted the same way everything else was: the default suite measured zero, so the broader set starts from a clean baseline and its findings on this - PR's own run are the measurement. + PR's own run are the measurement. That measurement found exactly one thing + — and it was real: `actions/untrusted-checkout/medium` on + `build-sandbox-image.yml`'s `fleet_ref`-fed checkout. Fixed, not waived + (the `actions` language has no `AlertSuppression.ql`, so a comment waiver + does not even exist): the workflow now **refuses `refs/pull/*` refs before + checkout** — a fork-PR ref would put fork-controlled code into a workflow + that executes the checked-out build script — and the identical hardening + went into `publish-sandbox-image.yml`, the *unflagged* twin that holds + `packages: write` and escaped the name-heuristic query only because its + ref plumbing was named differently. Extended suite then verified clean in + CI on all four languages (Dev CI run 525). The CodeQL/Semgrep log summaries + also now print **`file:line` per finding** (plus a database file count as + the coverage line), and the override canary is invoked via + `$GITHUB_WORKSPACE` so it survives the job's `working-directory: web`. - **Grype gate tightened to fixable CRITICAL + HIGH**, after measuring: the published sandbox image carries zero fixable Critical/High RPM findings diff --git a/docs/CODEQL.md b/docs/CODEQL.md index f80b3670..8c7cc8b1 100644 --- a/docs/CODEQL.md +++ b/docs/CODEQL.md @@ -80,10 +80,10 @@ enough", not "unset the pin". Neither `env: GOTOOLCHAIN: auto` nor | language | build mode | queries | | --- | --- | --- | -| `go` | `autobuild` | security (default suite) | -| `python` | `none` | security (default suite) | -| `javascript-typescript` | `none` | security (default suite) | -| `actions` | `none` | security (default suite) | +| `go` | `autobuild` | `security-extended` | +| `python` | `none` | `security-extended` | +| `javascript-typescript` | `none` | `security-extended` | +| `actions` | `none` | `security-extended` | **Security queries only — at the `security-extended` tier.** The code-quality suite was enabled, measured, and deliberately removed (see "Why code quality @@ -91,6 +91,28 @@ was dropped" below); the *security* side was then widened from the default suite to `security-extended` once the default measured clean, so the broader set also started from a zero baseline. +Adopting the extended suite was a measurement, and it produced exactly **one +finding across all four languages**: `actions/untrusted-checkout/medium` on the +`workflow_call` checkout in `build-sandbox-image.yml`, whose `ref:` is fed by +the `fleet_ref` input. The query is a **name heuristic** — it flags any +checkout whose ref traces to a field matching `.*(head|branch|ref).*` — +reproduced locally with the CodeQL 2.26.3 bundle to confirm the trigger before +touching anything. Two honest responses existed and the in-code dismissal was +not one of them: the `actions` language ships **no `AlertSuppression.ql`**, so +there is no comment-waiver path at all. The finding was fixed for real instead: +a `pin` step now refuses `refs/pull/*` / `pull/*` refs (a fork PR ref would +carry fork-controlled code into a workflow that *executes the checked-out build +script*), and the checkout consumes that step's neutrally-named output. The +same hardening went into `publish-sandbox-image.yml` — the **unflagged twin** +that is strictly more dangerous (it holds `packages: write`) but escaped the +query because its ref plumbing was named differently. A heuristic query's +silence is not evidence of safety; the flagged file just pointed at the class. + +The extended suite then verified **clean in CI on all four languages** — Dev CI +run 525 (`32580031374`), the same run that exercises the hardened `actions` +lane — so the fail-on-findings gate holds at the extended tier, not just the +default one. + `build-mode: none` is [not supported for Go](https://docs.github.com/en/code-security/reference/code-scanning/codeql/build-options-for-compiled-languages) — only `autobuild` or `manual` — so Go's toolchain has to be correct rather than diff --git a/docs/SCANNING.md b/docs/SCANNING.md index e8bf5bba..93064ba8 100644 --- a/docs/SCANNING.md +++ b/docs/SCANNING.md @@ -79,13 +79,24 @@ suite duplicated `golangci-lint`/`oxlint` for Go and JS, and ruff is a better fi for Python. Full reasoning and measurements in [`CODEQL.md`](CODEQL.md). It runs the **`security-extended`** suite — the broader security set, adopted -after the default suite measured clean — and currently reports **zero -findings** on this tree, which is +after the default suite measured clean — and reports **zero findings** on this +tree (verified in CI across all four languages on Dev CI run 525), which is what makes it safe to gate: a `Fail on findings` step now fails the job on any finding, so a red `Analyze (…)` check means the *code* has a problem rather than just "the scanner broke". That distinction is the whole reason the Go toolchain break sat unnoticed for weeks. +Getting the extended suite to zero was itself a fix, not a rubber stamp: its +one finding across all four languages was `actions/untrusted-checkout/medium` +on `build-sandbox-image.yml`'s `fleet_ref`-fed checkout. Rather than waive it +(the `actions` language has no `AlertSuppression.ql`, so there is no in-code +waiver anyway), the workflow now **refuses `refs/pull/*` refs** before checking +out — a fork-PR ref would put fork-controlled code into a workflow that runs +the checked-out build script — and the identical hardening went into +`publish-sandbox-image.yml`, the *unflagged* twin that holds `packages: write` +and only escaped the (name-heuristic) query because its plumbing was named +differently. Details in [`CODEQL.md`](CODEQL.md). + ### Semgrep owns fast multi-language SAST + Actions supply chain (new, blocking) Semgrep is the opposite trade from CodeQL: seconds instead of minutes, no @@ -182,19 +193,29 @@ broken — so `scripts/check-npm-overrides.sh` runs beside the audit in both lanes and **fails the build the day upstream's own ranges reach the patched lines**, with removal instructions. The reminder to drop the override is a red build with a two-line fix, not stale-pin archaeology later. (Registry flake = -skip with a notice, never a verdict; mutation-tested in both directions.) +skip with a notice, never a verdict; mutation-tested in both directions. The +step invokes it as `"$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh"` — the +job runs under `working-directory: web`, where a repo-relative path resolves +wrong; exit 127 on the first CI run taught that one.) ## Findings are readable from the job log, on purpose Both scanners print a per-rule summary into the job log **and** the step summary: ``` -### CodeQL findings — go -2 [note] go/useless-assignment-to-field +### CodeQL findings — actions +[warning] actions/untrusted-checkout/medium .github/workflows/build-sandbox-image.yml:106 -- -total findings: 2 +total findings: 1 +files in the actions database: 13 ``` +Each line carries the **`file:line`** of the finding — an agent reading the log +can go straight to the site — and the `files in the … database` count is the +coverage line: "No findings." over an empty database is the green-but-vacuous +outcome this workflow exists to rule out, and the two are indistinguishable +without it. + This exists because a CodeQL run reports **nothing** about what it found to its own log — it writes SARIF, uploads it, exits 0, findings or not. Verified by grepping a full run's log archive: there is no alert or result count anywhere. From 7b405e858dd275c08c26b47c81c408d9b61395f3 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 15:43:44 +0000 Subject: [PATCH 16/34] Close three own-rows authorization holes on the task surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The read path for task rows was narrowed to own rows in #1082, and run logs in #980. Three surfaces never got the same treatment and authorized on a permission alone, so any client-role principal reached every principal's rows: - GET /tasks/paused — ListPausedTasks selects on status alone with no principal predicate in SQL, and the projection carries each task's prompt. Its siblings (/tasks/export, /tasks/upcoming) both call visibleTasks; this one did not. Leaked other principals' paused prompts, and their task UUIDs with them. - PUT /tasks/{id} and POST /tasks/{id}/tags — loaded the task with the unscoped GetTask and never checked ownership, so a client-role principal could rewrite a teammate's pending run: prompt, model, mcp_selection and credential_allowlist included. Only run_if was gated (admin-only). - POST /tasks/{id}/feedback and GET /tasks/{id}/learned-instructions — taskFromPath is lookup-only by contract ("a handler that needs an authorization decision makes it on the returned task") and neither caller made one. A down-vote with an attacker-authored critique fed maybeDistill, which mints a proposal from the victim's prompt at unmetered model spend; the GET disclosed their learned instructions. The write gate is a new taskWritableByPrincipal, deliberately NOT principal.ownsTask: ownsTask resolves through ownerID(), which is nil for every API-key principal, so it would deny a scoped intake-app key the right to edit the task it just created. taskCreatedByPrincipal matches a creating user OR a creating key (CreatedByKeyID), which is the model #980/#1082 established. A write surface must be no looser than the read surface guarding the same row. TestScopedAPIKeyAuthorization previously asserted "client key can edit an editable task" against an unattributed row — that was the vulnerable behavior. Split into the owned case (must keep working: the intake-app path) and the unowned case (must 403). Every fix mutation-tested: stripped, the tests fail with the exploit visible — a hijacked prompt persisted, another principal's prompt in the paused queue, feedback accepted on an unowned task. Also in this commit, from the same audit sweep: - internal/config/config.go: ValidateScheduled interpolated the first 6 bytes of OPENROUTER_API_KEY into a validation error — the only place in the tree where secret material reached an error string. Removed, and the doc comment corrected: it claimed "Called at startup" but has no production caller. - internal/agent/scheduled.go: run-error strings now go through agentcore.RedactSecrets before the persisted transcript and the log. Tool output, the stream sink, hooks and the session log were already scrubbed; run errors were the one path that skipped it, and the transcript write is the larger surface. - internal/mcpoauth/discovery.go: refuse a non-http(s) scheme on the remote-derived discovery URLs (a WWW-Authenticate resource_metadata pointer, a PRM-declared issuer) before the request. Contained already by SafeHTTPClient and the transport; this makes the argument explicit rather than dependent on transport behavior. Tested both directions. - internal/sched/models/models.go: validate WorktreeConfig.BaseBranch. It is the trailing positional of `git worktree add -b ` with no "--" separator, so a leading-dash value was parsed by git as an option. worktree_config is settable by any task creator, unlike run_if. - Log-injection sinks that carry genuinely untrusted text: the task create log (task.Prompt — its update-path twin was already sanitized), the pre-validation client attachment path on the reject branches, the client-echoed attachment Name, the upload filename, and the API-key name. logSafe/%q, matching each line's existing sanitized sibling. - web/e2e/test-auth-key.ts: the Ed25519 private key was written to a fully predictable path in the world-writable temp dir at default 0644. Now O_EXCL at 0600 with random bytes in the sibling name. - internal/agent/session.go: document loadImageAttachments' caller contract. It performs no path containment of its own and is safe only because httpapi's validateAttachments is its sole producer. Stated as a contract, not an enforced boundary, because the uploads root is not threaded to that call site — a local check could only re-assert part of the guard while looking like all of it. Signed-off-by: Brad Flaugher --- internal/agent/scheduled.go | 9 +- internal/agent/session.go | 21 ++- internal/config/config.go | 14 +- internal/httpapi/attachments.go | 8 +- internal/mcpoauth/discovery.go | 29 +++ internal/mcpoauth/discovery_test.go | 37 ++++ internal/sched/handlers/handlers.go | 25 ++- .../sched/handlers/learned_instructions.go | 17 ++ internal/sched/handlers/pause.go | 6 + .../sched/handlers/principal_authz_test.go | 77 +++++++- internal/sched/handlers/task_authz.go | 24 +++ .../sched/handlers/task_write_authz_test.go | 176 ++++++++++++++++++ internal/sched/handlers/upload.go | 6 +- internal/sched/models/models.go | 19 ++ web/e2e/test-auth-key.ts | 18 +- 15 files changed, 463 insertions(+), 23 deletions(-) create mode 100644 internal/sched/handlers/task_write_authz_test.go diff --git a/internal/agent/scheduled.go b/internal/agent/scheduled.go index 5c54fcfe..352f811a 100644 --- a/internal/agent/scheduled.go +++ b/internal/agent/scheduled.go @@ -572,8 +572,13 @@ func (a *Agent) Execute(ctx context.Context, task string) (retErr error) { // entry here would mislabel every ask-pause and operator stop. if retErr != nil && !errors.Is(retErr, agentcore.ErrRunCancelled) { t := "error" - a.logSession.AddMessageWithMetadata(roleUser, "[fatal] "+retErr.Error(), nil, nil, &t, nil, nil, "") - log.Printf("Execute returning error: %v", retErr) + // Scrub before both sinks. RedactSecrets already guards tool output, + // the stream sink, hooks and the session log; run-error strings were + // the one path that skipped it, and the transcript write below is + // persisted and operator-visible, so it is the larger half. + msg := agentcore.RedactSecrets(retErr.Error()) + a.logSession.AddMessageWithMetadata(roleUser, "[fatal] "+msg, nil, nil, &t, nil, nil, "") + log.Printf("Execute returning error: %v", msg) } }() diff --git a/internal/agent/session.go b/internal/agent/session.go index d560a7f6..fe92cd47 100644 --- a/internal/agent/session.go +++ b/internal/agent/session.go @@ -332,6 +332,20 @@ func replayHistory(entries []HistoryEntry) ([]fantasy.Message, error) { // carry no media type (uploads have historically been PNG-normalized). const defaultImageMediaType = "image/png" +// CALLER CONTRACT — read before adding a producer of TurnInput.ImageAttachments. +// This function performs NO path containment of its own. Every a.Path it reads +// must already have been confined to the uploads root by the producer; today the +// sole producer is httpapi's validateAttachments (attachments.go), which rebuilds +// each path as Join(root, rel) after a filepath.Rel + filepath.IsLocal guard and +// stores only that. A future producer — a scheduled path, taskrun, an MCP-driven +// path — that skips that guard turns the os.ReadFile below into an arbitrary +// host-file read straight into the model context. +// +// This is a documented contract, not an enforced boundary, and it is stated that +// way deliberately: the uploads root lives on the config used by buildSandboxPool +// and is not currently threaded to this call site, so a local check here could +// only re-assert part of the guard while looking like all of it. Thread the root +// in and re-assert Rel+IsLocal+Join here if a second producer ever appears. func loadImageAttachments(atts []ImageAttachment) ([]fantasy.FilePart, []ImageRefMeta) { const ( maxImages = 8 @@ -344,7 +358,12 @@ func loadImageAttachments(atts []ImageAttachment) ([]fantasy.FilePart, []ImageRe refs := make([]ImageRefMeta, 0, len(atts)) for _, a := range atts { if len(parts) >= maxImages { - log.Printf("loadImageAttachments: skipping %s (over %d cap)", a.Name, maxImages) + // %q on a.Name: unlike a.Path (rebuilt server-side as + // Join(root, rel) by validateAttachments), Name is the client's + // echoed chatAttachment field and is never sanitized — + // sanitizeFilename runs at upload time, but /chat re-accepts the + // client's own JSON and re-validates only Path. %q escapes CR/LF. + log.Printf("loadImageAttachments: skipping %q (over %d cap)", a.Name, maxImages) continue } info, err := os.Stat(a.Path) diff --git a/internal/config/config.go b/internal/config/config.go index 9218b8db..4c8f91b1 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -1733,15 +1733,23 @@ func (c *Config) validateTLS() error { } // ValidateScheduled checks the one-shot scheduled (cutlass) required values and -// returns an error describing all problems found. Called at startup to fail -// fast for the scheduled driver. +// returns an error describing all problems found. +// +// No production caller: `fleet task run` (the folded cutlass harness) validates +// through Validate/validate_config.go instead. This is exercised only by +// config_test.go and is kept as the scheduled-driver contract, so do not add a +// value to any message here — see the next comment. func (c *Config) ValidateScheduled() error { var errs []string if c.OpenRouterAPIKey == "" { errs = append(errs, "OPENROUTER_API_KEY is required") } else if !strings.HasPrefix(c.OpenRouterAPIKey, "sk-or-") { - errs = append(errs, "OPENROUTER_API_KEY should start with 'sk-or-' (got '"+c.OpenRouterAPIKey[:min(6, len(c.OpenRouterAPIKey))]+"...')") + // Name the expected prefix, never echo the key. This previously + // interpolated the key's first 6 bytes into the error, which was the + // one place in the tree where secret material reached an error string — + // and validation errors are logged and surfaced to operators. + errs = append(errs, "OPENROUTER_API_KEY should start with 'sk-or-'") } if c.MaxIterations < 1 || c.MaxIterations > 10000 { diff --git a/internal/httpapi/attachments.go b/internal/httpapi/attachments.go index 6ed2535d..f426844c 100644 --- a/internal/httpapi/attachments.go +++ b/internal/httpapi/attachments.go @@ -309,13 +309,17 @@ func (s *Server) validateAttachments(atts []chatAttachment) []chatAttachment { // !IsRegular. rel, relErr := filepath.Rel(root, abs) if relErr != nil || !filepath.IsLocal(rel) { - log.Printf("attachment rejected (outside uploads root): %s", a.Path) + // %q, not %s: a.Path here is the RAW client-supplied string on the + // branch where containment just FAILED, so it is hostile by + // construction. %q escapes CR/LF and cannot forge a log entry. + log.Printf("attachment rejected (outside uploads root): %q", a.Path) continue } abs = filepath.Join(root, rel) info, err := os.Stat(abs) if err != nil || !info.Mode().IsRegular() { - log.Printf("attachment rejected (stat): %s: %v", a.Path, err) + // %q for the same reason as above: still the pre-validation client string. + log.Printf("attachment rejected (stat): %q: %v", a.Path, err) continue } a.Path = filepath.ToSlash(abs) diff --git a/internal/mcpoauth/discovery.go b/internal/mcpoauth/discovery.go index 2fd9182f..65650921 100644 --- a/internal/mcpoauth/discovery.go +++ b/internal/mcpoauth/discovery.go @@ -6,9 +6,27 @@ import ( "fmt" "io" "net/http" + "net/url" "strings" ) +// requireHTTPScheme refuses any URL that is not http:// or https:// before it +// reaches an outbound request. Remote-derived discovery URLs land here (see +// fetchJSON), and a file://, gopher:// or data:// pointer from a hostile server +// should be rejected by name rather than left to the transport to decline. +func requireHTTPScheme(raw string) error { + u, err := url.Parse(raw) + if err != nil { + return fmt.Errorf("parse discovery URL: %w", err) + } + switch u.Scheme { + case "http", "https": + return nil + default: + return fmt.Errorf("refusing discovery URL with scheme %q (only http/https)", u.Scheme) + } +} + // maxMetadataBytes caps a metadata/JSON response so a hostile server can't OOM // the host by streaming an unbounded body. const maxMetadataBytes = 1 << 20 // 1 MiB @@ -226,7 +244,18 @@ func verifyAuthServer(expectedIssuer string, as *AuthServerMetadata) error { } // fetchJSON GETs url and decodes a (size-limited) JSON body into out. +// +// The URLs reaching here are REMOTE-DERIVED — a WWW-Authenticate +// `resource_metadata=` pointer, or a candidate built from a PRM-declared +// `issuer` — so they are untrusted even though the operator typed the server +// URL that led to them. SSRF is contained by SafeHTTPClient's resolve-then-dial +// guard and its no-redirect policy, and http.Transport would refuse a non-HTTP +// scheme anyway; the explicit check below is one line and makes that argument +// airtight rather than dependent on the transport's behavior. func fetchJSON(ctx context.Context, httpClient *http.Client, url string, out any) error { + if err := requireHTTPScheme(url); err != nil { + return err + } req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) if err != nil { return err diff --git a/internal/mcpoauth/discovery_test.go b/internal/mcpoauth/discovery_test.go index f8810d7c..9b80fb28 100644 --- a/internal/mcpoauth/discovery_test.go +++ b/internal/mcpoauth/discovery_test.go @@ -227,3 +227,40 @@ func TestRegisterDCR(t *testing.T) { t.Error("Register accepted an empty registration endpoint") } } + +// TestFetchJSONRefusesNonHTTPScheme pins the scheme guard on the remote-derived +// discovery URLs. A hostile MCP server controls the WWW-Authenticate +// `resource_metadata=` pointer and the PRM-declared `issuer`, so it chooses the +// string that reaches fetchJSON. SafeHTTPClient's dialer and http.Transport both +// already decline a non-HTTP scheme; this asserts we refuse it by name first, so +// the containment argument does not rest on transport behavior. +func TestFetchJSONRefusesNonHTTPScheme(t *testing.T) { + for _, raw := range []string{ + "file:///etc/passwd", + "gopher://127.0.0.1:70/x", + "data:application/json,{}", + "ftp://example.com/meta.json", + } { + var out map[string]any + err := fetchJSON(context.Background(), http.DefaultClient, raw, &out) + if err == nil { + t.Fatalf("fetchJSON(%q) = nil error, want refusal", raw) + } + if !strings.Contains(err.Error(), "only http/https") { + t.Fatalf("fetchJSON(%q) error = %v, want a scheme refusal", raw, err) + } + } +} + +// TestRequireHTTPSchemeAcceptsHTTPAndHTTPS is the negative half: the guard must +// not reject the two schemes discovery legitimately uses. +func TestRequireHTTPSchemeAcceptsHTTPAndHTTPS(t *testing.T) { + for _, raw := range []string{ + "http://localhost:8080/.well-known/oauth-protected-resource", + "https://example.com/.well-known/openid-configuration", + } { + if err := requireHTTPScheme(raw); err != nil { + t.Fatalf("requireHTTPScheme(%q) = %v, want nil", raw, err) + } + } +} diff --git a/internal/sched/handlers/handlers.go b/internal/sched/handlers/handlers.go index d05a5859..e0d60a5f 100644 --- a/internal/sched/handlers/handlers.go +++ b/internal/sched/handlers/handlers.go @@ -547,7 +547,8 @@ func (h *Handlers) CreateTask(w http.ResponseWriter, r *http.Request) { return } - log.Printf("Task created: %s (prompt: %.50s...)", task.ID, task.Prompt) + //nolint:gosec // G706: untrusted fields are sanitized via logSafe (strips CR/LF); gosec's taint tracker cannot see through the helper. task.ID is a uuid.UUID. + log.Printf("Task created: %s (prompt: %.50s...)", task.ID, logSafe(task.Prompt)) localizeTask(task) writeJSON(w, http.StatusOK, task) } @@ -1720,6 +1721,18 @@ func (h *Handlers) UpdateTask(w http.ResponseWriter, r *http.Request) { return } + // Own-rows authorization (#1082 model): PermissionCreateTask above admits + // the principal to the surface; this decides WHICH task. Without it any + // client-role principal could rewrite a teammate's pending run — prompt, + // model, mcp_selection and credential_allowlist included — even though the + // read path guarding the same row was narrowed to own rows. GetTask is + // unscoped by design (the read surfaces filter above it), so the check + // belongs here. + if !taskWritableByPrincipal(p, task) { + writeError(w, http.StatusForbidden, "Only the task creator or an admin can edit this task") + return + } + // Only allow editing tasks that haven't started if task.Status != models.TaskStatusPending && task.Status != models.TaskStatusScheduled { writeError(w, http.StatusBadRequest, "Only pending or scheduled tasks can be edited") @@ -1954,6 +1967,13 @@ func (h *Handlers) UpdateTaskTags(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusNotFound, "Task not found") return } + // Own-rows authorization, same helper and same reason as UpdateTask. Tags + // drive filtering and routing, so retagging a teammate's task is a write to + // their work, not a cosmetic change. + if !taskWritableByPrincipal(p, task) { + writeError(w, http.StatusForbidden, "Only the task creator or an admin can retag this task") + return + } var body tagMutation if err := readJSON(r, &body); err != nil { writeError(w, http.StatusBadRequest, "Invalid request body") @@ -2345,7 +2365,8 @@ func (h *Handlers) CreateAPIKey(w http.ResponseWriter, r *http.Request) { } } - log.Printf("Created API key: %s (%s)", key.KeyID, key.Name) + //nolint:gosec // G706: key.Name is unvalidated body text sanitized via logSafe (strips CR/LF), matching the sibling key handlers; key.KeyID is server-minted. + log.Printf("Created API key: %s (%s)", key.KeyID, logSafe(key.Name)) resp := key.ToResponse() writeJSON(w, http.StatusOK, models.APIKeyCreated{ diff --git a/internal/sched/handlers/learned_instructions.go b/internal/sched/handlers/learned_instructions.go index 36a3344f..f3007275 100644 --- a/internal/sched/handlers/learned_instructions.go +++ b/internal/sched/handlers/learned_instructions.go @@ -54,6 +54,15 @@ func (h *Handlers) SubmitFeedback(w http.ResponseWriter, r *http.Request) { if !ok { return } + // taskFromPath is lookup only (see its doc) — the authorization decision is + // the caller's. Own-rows: without this any `client`-role principal could + // down-vote a teammate's task with an attacker-authored critique, which + // maybeDistill then feeds — together with the victim's prompt — into an LLM + // to mint a proposal on their task, at unmetered model spend. + if !taskVisibleToPrincipal(p, task) { + writeError(w, http.StatusNotFound, "Task not found") + return + } var req feedbackRequest if err := readJSON(r, &req); err != nil { writeError(w, http.StatusBadRequest, "Invalid JSON: "+err.Error()) @@ -136,6 +145,14 @@ func (h *Handlers) LearnedInstructions(w http.ResponseWriter, r *http.Request) { if !ok { return } + // Own-rows, same reason as SubmitFeedback: the GET branch below discloses + // another principal's learned instructions, which are distilled from their + // task's prompt and critiques. 404 rather than 403 so the surface does not + // confirm that an unowned task id exists. + if !taskVisibleToPrincipal(p, task) { + writeError(w, http.StatusNotFound, "Task not found") + return + } versionStr := chi.URLParam(r, "version") switch { diff --git a/internal/sched/handlers/pause.go b/internal/sched/handlers/pause.go index 01da8496..eb73ce3d 100644 --- a/internal/sched/handlers/pause.go +++ b/internal/sched/handlers/pause.go @@ -116,6 +116,12 @@ func (h *Handlers) ListPausedTasks(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusInternalServerError, "Failed to list paused tasks") return } + // Own-rows visibility (#1082): ListPausedTasks selects by status alone, with + // no principal predicate in SQL, and the projection carries each task's + // prompt — so it has to be scoped here like GET /tasks, /tasks/export and + // /tasks/upcoming. Without this a `client`-role principal read every + // principal's paused prompts, and learned their task UUIDs besides. + tasks = visibleTasks(p, tasks) if tasks == nil { tasks = []*models.Task{} } diff --git a/internal/sched/handlers/principal_authz_test.go b/internal/sched/handlers/principal_authz_test.go index fc32508e..3655f643 100644 --- a/internal/sched/handlers/principal_authz_test.go +++ b/internal/sched/handlers/principal_authz_test.go @@ -83,13 +83,46 @@ func setupAuthzHandler(t *testing.T) (*storage.Storage, *apikeys.Manager, *chi.M } func mustCreateRoleKey(t *testing.T, keyMgr *apikeys.Manager, role string) string { + t.Helper() + _, raw := mustCreateRoleKeyWithID(t, keyMgr, role) + return raw +} + +// mustCreateRoleKeyWithID also returns the key's KeyID, so a test can attribute +// a task to the key (task.CreatedByKeyID) and exercise own-rows authorization +// on an API-key principal rather than only on a user principal. +func mustCreateRoleKeyWithID(t *testing.T, keyMgr *apikeys.Manager, role string) (string, string) { t.Helper() r := role - _, raw, err := keyMgr.CreateKey("test-"+role, nil, &r, 0, nil, "") + key, raw, err := keyMgr.CreateKey("test-"+role+"-"+uuid.NewString(), nil, &r, 0, nil, "") if err != nil { t.Fatalf("create key: %v", err) } - return raw + return key.KeyID, raw +} + +// addTaskCreatedByKey inserts a task attributed to the given API key. The +// column is written on insert (taskColumnRegistry), so it is set before AddTask. +func addTaskCreatedByKey(t *testing.T, store *storage.Storage, prompt, keyID string) *models.Task { + t.Helper() + return addTaskCreatedByKeyWithStatus(t, store, prompt, keyID, models.TaskStatusPending) +} + +// addTaskCreatedByKeyWithStatus is the same, at a chosen status — the paused +// queue selects on status alone, so its tests need rows already paused. +func addTaskCreatedByKeyWithStatus(t *testing.T, store *storage.Storage, prompt, keyID string, status models.TaskStatus) *models.Task { + t.Helper() + task := &models.Task{ + ID: uuid.New(), + Prompt: prompt, + Status: status, + CreatedAt: time.Now().UTC(), + CreatedByKeyID: &keyID, + } + if _, err := store.AddTask(task); err != nil { + t.Fatalf("add task: %v", err) + } + return task } func addTask(t *testing.T, store *storage.Storage, prompt string) *models.Task { @@ -160,20 +193,46 @@ func TestScopedAPIKeyAuthorization(t *testing.T) { } }) - t.Run("client key can edit an editable task", func(t *testing.T) { - // The client role carries create_task (which gates editing) but not - // cancel_task, so editing is the right op to test mutating authorization - // on a call a scoped key is actually permitted to make. - clientKey := mustCreateRoleKey(t, keyMgr, "client") + // Editing is own-rows, not merely permission-gated (taskWritableByPrincipal). + // The client role carries create_task (which admits it to the edit surface) + // but not cancel_task, so it is the right role to test WHICH task a scoped + // key may mutate. + t.Run("client key can edit a task it created", func(t *testing.T) { + keyID, clientKey := mustCreateRoleKeyWithID(t, keyMgr, "client") + + // Attributed to this key — the scoped-intake-app case that must keep working. + own := addTaskCreatedByKey(t, store, "a task this key created", keyID) body, _ := json.Marshal(models.TaskCreate{Prompt: "edited prompt that is sufficiently long"}) - req := httptest.NewRequest("PUT", "/tasks/"+taskA.ID.String(), bytes.NewReader(body)) + req := httptest.NewRequest("PUT", "/tasks/"+own.ID.String(), bytes.NewReader(body)) req.Header.Set("X-API-Key", clientKey) req.Header.Set("Content-Type", "application/json") w := httptest.NewRecorder() r.ServeHTTP(w, req) if w.Code != http.StatusOK { - t.Fatalf("client key edit should be 200, got %d: %s", w.Code, w.Body.String()) + t.Fatalf("client key editing its OWN task should be 200, got %d: %s", w.Code, w.Body.String()) + } + }) + + // The regression this pair exists for: PUT /tasks/{id} authorized on + // PermissionCreateTask alone, so a scoped key could rewrite a task it did + // not create — prompt, model, mcp_selection, credential_allowlist — while + // the READ path for the same row was already narrowed to own rows (#1082). + t.Run("client key cannot edit a task it did not create", func(t *testing.T) { + clientKey := mustCreateRoleKey(t, keyMgr, "client") + + body, _ := json.Marshal(models.TaskCreate{Prompt: "hijacked prompt that is long enough"}) + req := httptest.NewRequest("PUT", "/tasks/"+taskA.ID.String(), bytes.NewReader(body)) + req.Header.Set("X-API-Key", clientKey) + req.Header.Set("Content-Type", "application/json") + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + if w.Code != http.StatusForbidden { + t.Fatalf("client key must not edit an unowned task; got %d: %s", w.Code, w.Body.String()) + } + after, _ := store.GetTask(taskA.ID) + if after == nil || after.Prompt != "task A" { + t.Fatalf("a refused edit must leave the prompt alone, got %q", after.Prompt) } }) } diff --git a/internal/sched/handlers/task_authz.go b/internal/sched/handlers/task_authz.go index aeceea3e..85b982d8 100644 --- a/internal/sched/handlers/task_authz.go +++ b/internal/sched/handlers/task_authz.go @@ -60,3 +60,27 @@ func visibleTasks(p principal, tasks []*models.Task) []*models.Task { } return out } + +// taskWritableByPrincipal reports whether the principal may MUTATE the given +// task's definition (edit, retag). The mutating permission (PermissionCreateTask, +// checked by the caller) admits it to the surface; this decides WHICH task. +// +// Same own-rows model as taskVisibleToPrincipal, and deliberately the same +// helper pair: a write surface must be no looser than the read surface guarding +// the same row. Before this existed, PUT /tasks/{id} and POST /tasks/{id}/tags +// authorized on PermissionCreateTask alone, so any client-role user or scoped +// task key could rewrite ANY task on the box — prompt, model, mcp_selection and +// credential_allowlist included — while the read path had already been narrowed +// to own rows by #1082. That asymmetry was the hole. +// +// Note this is NOT principal.ownsTask, which resolves ownership through +// ownerID() and therefore returns false for every API-key principal. Using it +// here would deny a scoped intake-app key the right to edit the task it just +// created. taskCreatedByPrincipal matches a creating user OR a creating key +// (task.CreatedByKeyID), which is the model #980/#1082 established. +func taskWritableByPrincipal(p principal, task *models.Task) bool { + if p.fleetWideTaskVisibility() { + return true + } + return taskCreatedByPrincipal(p, task) +} diff --git a/internal/sched/handlers/task_write_authz_test.go b/internal/sched/handlers/task_write_authz_test.go new file mode 100644 index 00000000..2b572054 --- /dev/null +++ b/internal/sched/handlers/task_write_authz_test.go @@ -0,0 +1,176 @@ +// Copyright (c) 2025 ElcanoTek +// SPDX-License-Identifier: MIT + +package handlers + +import ( + "bytes" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + + "github.com/go-chi/chi/v5" + + "github.com/ElcanoTek/fleet/internal/sched/models" +) + +// Own-rows regression tests for three surfaces that authorized on a permission +// alone while the read path guarding the same rows had already been narrowed to +// own rows (#980/#1082): +// +// - GET /tasks/paused — returned every principal's rows +// - POST /tasks/{id}/feedback — wrote to any principal's task +// - GET /tasks/{id}/learned-instructions — read any principal's instructions +// +// PUT /tasks/{id} and POST /tasks/{id}/tags are covered by +// TestScopedAPIKeyAuthorization in principal_authz_test.go. + +// taskAuthzRouter wires only the routes under test, behind the same middleware +// the server uses, so each request carries a real principal. +func taskAuthzRouter(t *testing.T) (*chi.Mux, *Handlers, func()) { + t.Helper() + store, keyMgr, _, cleanup := setupAuthzHandler(t) + h := New(Config{ + DefaultTaskModel: "test/model", + OrchestratorURL: "http://localhost:8000", + AdminAPIKey: "test-admin-key", + Version: "0.1.0", + }, store, keyMgr) + r := chi.NewRouter() + r.Group(func(r chi.Router) { + r.Use(h.AdminOrUserAuthMiddleware) + r.Get("/tasks/paused", h.ListPausedTasks) + r.Post("/tasks/{task_id}/feedback", h.SubmitFeedback) + r.Get("/tasks/{task_id}/learned-instructions", h.LearnedInstructions) + }) + return r, h, cleanup +} + +func decodeTasks(t *testing.T, body []byte) []*models.Task { + t.Helper() + var got struct { + Tasks []*models.Task `json:"tasks"` + } + if err := json.Unmarshal(body, &got); err != nil { + t.Fatalf("decode: %v", err) + } + return got.Tasks +} + +// A paused task carries its prompt in the projection, and ListPausedTasks +// selects on status alone with no principal predicate in SQL. Before the fix a +// client-role key read every principal's paused prompts — and their task UUIDs. +func TestListPausedTasksIsScopedToOwnRows(t *testing.T) { + r, h, cleanup := taskAuthzRouter(t) + defer cleanup() + + keyID, rawKey := mustCreateRoleKeyWithID(t, h.apiKeys, "client") + mine := addTaskCreatedByKeyWithStatus(t, h.storage, "my own paused prompt", keyID, models.TaskStatusPausedAwaitingInput) + + otherKeyID, _ := mustCreateRoleKeyWithID(t, h.apiKeys, "client") + theirs := addTaskCreatedByKeyWithStatus(t, h.storage, "SOMEONE ELSE secret prompt", otherKeyID, models.TaskStatusPausedAwaitingInput) + + req := httptest.NewRequest("GET", "/tasks/paused", nil) + req.Header.Set("X-API-Key", rawKey) + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + if w.Code != http.StatusOK { + t.Fatalf("GET /tasks/paused = %d, want 200: %s", w.Code, w.Body.String()) + } + + tasks := decodeTasks(t, w.Body.Bytes()) + var sawMine bool + for _, task := range tasks { + if task.ID == theirs.ID { + t.Fatalf("another principal's paused task leaked into the queue: %q", task.Prompt) + } + if task.ID == mine.ID { + sawMine = true + } + } + if !sawMine { + t.Fatal("the principal's OWN paused task must still be listed — otherwise this asserts nothing") + } +} + +// An admin must still see the whole queue: a fleet-wide view is the point of the +// "needs a human answer" surface. +func TestListPausedTasksAdminSeesEveryRow(t *testing.T) { + r, h, cleanup := taskAuthzRouter(t) + defer cleanup() + + otherKeyID, _ := mustCreateRoleKeyWithID(t, h.apiKeys, "client") + theirs := addTaskCreatedByKeyWithStatus(t, h.storage, "someone's paused prompt", otherKeyID, models.TaskStatusPausedAwaitingInput) + + req := httptest.NewRequest("GET", "/tasks/paused", nil) + req.Header.Set("X-API-Key", "test-admin-key") + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + if w.Code != http.StatusOK { + t.Fatalf("admin GET /tasks/paused = %d, want 200: %s", w.Code, w.Body.String()) + } + for _, task := range decodeTasks(t, w.Body.Bytes()) { + if task.ID == theirs.ID { + return + } + } + t.Fatal("an admin must see other principals' paused tasks") +} + +// Feedback writes to the task and can trigger LLM distillation against the +// victim's prompt, so it is own-rows. 404 rather than 403 so the surface does +// not confirm that an unowned task id exists. +func TestFeedbackAndLearnedInstructionsAreScopedToOwnRows(t *testing.T) { + r, h, cleanup := taskAuthzRouter(t) + defer cleanup() + + _, rawKey := mustCreateRoleKeyWithID(t, h.apiKeys, "client") + otherKeyID, _ := mustCreateRoleKeyWithID(t, h.apiKeys, "client") + theirs := addTaskCreatedByKey(t, h.storage, "someone else's prompt", otherKeyID) + + body, _ := json.Marshal(map[string]string{"rating": models.FeedbackDown, "critique": "attacker-authored critique"}) + req := httptest.NewRequest("POST", "/tasks/"+theirs.ID.String()+"/feedback", bytes.NewReader(body)) + req.Header.Set("X-API-Key", rawKey) + req.Header.Set("Content-Type", "application/json") + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + if w.Code != http.StatusNotFound { + t.Fatalf("feedback on an unowned task = %d, want 404: %s", w.Code, w.Body.String()) + } + + req = httptest.NewRequest("GET", "/tasks/"+theirs.ID.String()+"/learned-instructions", nil) + req.Header.Set("X-API-Key", rawKey) + w = httptest.NewRecorder() + r.ServeHTTP(w, req) + if w.Code != http.StatusNotFound { + t.Fatalf("learned-instructions on an unowned task = %d, want 404: %s", w.Code, w.Body.String()) + } +} + +// The owning principal must still be able to use both surfaces. +func TestFeedbackOnOwnTaskIsAllowed(t *testing.T) { + r, h, cleanup := taskAuthzRouter(t) + defer cleanup() + + keyID, rawKey := mustCreateRoleKeyWithID(t, h.apiKeys, "client") + mine := addTaskCreatedByKey(t, h.storage, "my own prompt", keyID) + + body, _ := json.Marshal(map[string]string{"rating": models.FeedbackUp}) + req := httptest.NewRequest("POST", "/tasks/"+mine.ID.String()+"/feedback", bytes.NewReader(body)) + req.Header.Set("X-API-Key", rawKey) + req.Header.Set("Content-Type", "application/json") + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + if w.Code != http.StatusOK { + t.Fatalf("feedback on own task = %d, want 200: %s", w.Code, w.Body.String()) + } + + req = httptest.NewRequest("GET", "/tasks/"+mine.ID.String()+"/learned-instructions", nil) + req.Header.Set("X-API-Key", rawKey) + w = httptest.NewRecorder() + r.ServeHTTP(w, req) + if w.Code != http.StatusOK { + t.Fatalf("learned-instructions on own task = %d, want 200: %s", w.Code, w.Body.String()) + } +} diff --git a/internal/sched/handlers/upload.go b/internal/sched/handlers/upload.go index 5a36d564..9ce7ee26 100644 --- a/internal/sched/handlers/upload.go +++ b/internal/sched/handlers/upload.go @@ -194,10 +194,12 @@ func (h *Handlers) HandleUpload(w http.ResponseWriter, r *http.Request) { checksumPath := filepath.Join(tempDir, ".checksums", filename+".sha256") if err := os.WriteFile(checksumPath, []byte(checksum), 0600); err != nil { // Non-critical error, just log it - log.Printf("Failed to save checksum sidecar for %s: %v", filename, err) + //nolint:gosec // G706: filename is sanitized via logSafe (strips CR/LF) and already passed sanitizeFilename; gosec's taint tracker cannot see through the helper. + log.Printf("Failed to save checksum sidecar for %s: %v", logSafe(filename), err) } - log.Printf("File uploaded: %s (size: %d, checksum: %s)", filename, size, checksum) + //nolint:gosec // G706: filename is sanitized via logSafe (strips CR/LF) and already passed sanitizeFilename; size is an int and checksum is hex. + log.Printf("File uploaded: %s (size: %d, checksum: %s)", logSafe(filename), size, checksum) writeJSON(w, http.StatusOK, map[string]interface{}{ "filename": filename, diff --git a/internal/sched/models/models.go b/internal/sched/models/models.go index a40e1840..86ab35ed 100644 --- a/internal/sched/models/models.go +++ b/internal/sched/models/models.go @@ -297,6 +297,25 @@ func (wc *WorktreeConfig) Validate() error { strings.Contains(wc.BranchPrefix, ".lock") { return fmt.Errorf("branch_prefix is not a valid git ref-name fragment") } + // BaseBranch is the trailing positional of `git worktree add -b + // `, and that invocation carries no "--" end-of-options + // separator, so a value beginning with "-" would be parsed by git as an + // option rather than a commit-ish. Reject that shape here — worktree_config + // is settable by any task creator (unlike run_if, which is admin-only), so + // this is the boundary. The ref-name checks mirror BranchPrefix above; git + // still makes the authoritative check at run time. + if base := strings.TrimSpace(wc.BaseBranch); base != "" { + if strings.HasPrefix(base, "-") { + return fmt.Errorf("base_branch may not begin with '-'") + } + if strings.ContainsAny(base, " ~^:?*[\\") || + strings.Contains(base, "@{") || + strings.Contains(base, "..") || + strings.Contains(base, "//") || + strings.Contains(base, ".lock") { + return fmt.Errorf("base_branch is not a valid git ref-name fragment") + } + } return nil } diff --git a/web/e2e/test-auth-key.ts b/web/e2e/test-auth-key.ts index cb2aedf6..d6de78b7 100644 --- a/web/e2e/test-auth-key.ts +++ b/web/e2e/test-auth-key.ts @@ -48,8 +48,22 @@ export function generateTestAuthKey(): TestAuthKeyMaterial { }; // Atomic write so a worker reading concurrently never sees a half-written // file: write to a temp sibling, then rename. - const tmp = `${KEY_FILE}.${process.pid}.tmp`; - fs.writeFileSync(tmp, JSON.stringify(material), { encoding: "utf8" }); + // + // The sibling name carries random bytes, and the write is `wx` (O_CREAT|O_EXCL) + // at mode 0600. os.tmpdir() is world-writable and the old name was + // `${KEY_FILE}.${pid}.tmp` — fully predictable, so a local user could + // pre-create it as a symlink and turn this into an arbitrary-file write as the + // test user, and the default 0644 left the private half world-readable. + // O_EXCL refuses to follow or clobber a pre-planted path; rename(2) acts on the + // link itself, so a hostile KEY_FILE symlink is replaced rather than written + // through. The key is throwaway and protects nothing real — this is hygiene on + // a private-key write, not a fix for a reachable compromise. + const tmp = `${KEY_FILE}.${process.pid}.${crypto.randomBytes(6).toString("hex")}.tmp`; + fs.writeFileSync(tmp, JSON.stringify(material), { + encoding: "utf8", + mode: 0o600, + flag: "wx", + }); fs.renameSync(tmp, KEY_FILE); return material; } From e90fc99b005d434c2cfd4cf157f04dfa416ec62b Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 15:55:50 +0000 Subject: [PATCH 17/34] CodeQL: gate on High-and-above plus a reviewed accepted-findings register MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate #1246 shipped blocked every push to dev and main. Its threshold was "any finding at any severity", justified by a measured zero across all four languages — but that measurement came from Dev CI run 525, a `pull_request` event, and on pull_request events the CodeQL action runs DIFF-INFORMED: it builds the full database, evaluates every query, then reports only results located inside the PR's diff. Run 525's own log says both halves out loud ("Persisted 204 diff range(s) across 43 file(s)", and "file coverage information is only enabled when analyzing the default branch and protected branches"). The Go database held all 428 files and TaintedPath/RequestForgery/LogInjection/ WeakSensitiveDataHashing all ran; the SARIF was empty because results outside the 43 changed files were dropped. So the first full-tree evaluation was the push that merged #1246 — run 527 — which reported 38 Go and 17 javascript-typescript findings and turned Dev gate red, with no PR-shaped way out: a PR into dev is scanned diff-informed and stays green while dev itself stays red. The generalizable lesson, now written into the file and the ADR: a PR-event CodeQL run certifies a diff, not a tree. Any "the scanners are green, therefore the tree is clean" claim resting on one is unsound, and that is permanent behavior, not a bug. New threshold: a finding blocks when its SARIF level is error/warning or its rule carries security-severity >= 7.0, unless it is waived. Below that band findings are printed and go to the Security tab as advisory. Severity alone is not sufficient to separate the false positives, which is why the register exists: go/request-forgery is 9.1 and fires on web_fetch.go, a deliberate user-facing fetch tool behind netguard's resolve-then-dial SSRF guard; go/weak-sensitive-data-hashing is 7.5 and fires on SHA-256 used as a lookup index over a 32-byte crypto/rand token, which is the recommended construction. .github/codeql-accepted-findings.json registers accepted (rule, file) pairs with a mandatory written reason. Per-FILE, not per-rule, and that is the point of preferring it to a query-filters exclude: an exclude switches a 9.1 query off repo-wide, while the register waives it in the two files that were adjudicated and leaves it live everywhere else. A synthetic SARIF carrying a fresh go/request-forgery in an unregistered file fails the gate — verified while developing the jq, along with the clean, notes-only, mixed-severity, in-source-suppressed, missing-SARIF, malformed-SARIF and missing-register cases. The gate fails closed twice over: a missing register and an unevaluatable jq both refuse to report the scan clean, rather than reading as zero. Anti-rot controls, because a register nobody re-reads is worse than none: - scripts/check_codeql_register_test.go (in make test) requires every entry to name a file that exists, carry a substantive reason, use a plausible rule id, and be unique — and asserts codeql.yml still references the register, so the two cannot be silently decoupled. Mutation-tested: a bogus rule id, a missing file and a one-word reason all fail it. - The log and step summary print three tiers — BLOCKING, ACCEPTED and ADVISORY — so every waiver appears in ordinary CI output instead of only in a file someone has to think to open. Also corrects the claims this file made about itself, which an auditor reads as documentation: - the "reports ZERO ... a finding here is new" premise (false); - "They cannot be part of CI gate" two paragraphs after correctly explaining that the calling job is in the gate's needs; - "whether a red check BLOCKS a merge is branch protection's call", which contradicted the header — and now records the real caveat, that the dev ruleset requires no status checks so Dev gate is red-but-not-required there; - the summarize step's "gating on findings is merge protection's job, not this step's", directly above the step that gates; - dev-ci.yml's "makes green mean clean, not just ran", now qualified for pull_request events. ADR-0048 records the decision, what gets worse (a note-level regression no longer fails the build — gosec's G706 still covers the log-injection class through golangci-lint, which does block), and the sharpest edge ( the register keys on rule+file, not rule+file+line, so a second bad instance in an already-waived file would not block; line keys churn on every edit and a register that fails on unrelated refactors is one people delete). Signed-off-by: Brad Flaugher --- .github/codeql-accepted-findings.json | 90 +++++++++++ .github/workflows/codeql.yml | 189 +++++++++++++++++------- .github/workflows/dev-ci.yml | 10 +- docs/adr/0048-codeql-severity-gating.md | 157 ++++++++++++++++++++ scripts/check_codeql_register_test.go | 137 +++++++++++++++++ 5 files changed, 527 insertions(+), 56 deletions(-) create mode 100644 .github/codeql-accepted-findings.json create mode 100644 docs/adr/0048-codeql-severity-gating.md create mode 100644 scripts/check_codeql_register_test.go diff --git a/.github/codeql-accepted-findings.json b/.github/codeql-accepted-findings.json new file mode 100644 index 00000000..f2fac66d --- /dev/null +++ b/.github/codeql-accepted-findings.json @@ -0,0 +1,90 @@ +{ + "$schema-note": [ + "Register of CodeQL findings that are accepted as false positives in fleet's", + "threat model. Consumed by .github/workflows/codeql.yml's `Fail on findings`", + "step: a finding whose (rule, file) pair appears here does not block the", + "build. Everything else at level error/warning, or security-severity >= 7.0,", + "does.", + "", + "WHY A REGISTER AND NOT query-filters OR AN IGNORED PATH:", + "a query-filter `exclude` switches the rule off for the whole repository, so", + "a genuine future instance of go/request-forgery (security-severity 9.1)", + "would never be reported again. An entry here waives ONE rule in ONE file and", + "leaves the query live everywhere else — including elsewhere in the same", + "package. The findings still upload to the Security tab either way; this only", + "governs whether CI blocks.", + "", + "RULES FOR EDITING:", + " - One entry per (rule, file). `reason` is mandatory and must say why the", + " finding cannot be exploited HERE, not that the rule is noisy.", + " - Widening this file is a security decision. It belongs in the PR diff and", + " the reviewer is expected to check the reason against the code.", + " - An entry is not a permanent waiver. The weekly scheduled scan reports", + " entries that no longer match any finding so a stale waiver gets removed", + " rather than quietly widening coverage loss.", + " - Fixing the code is always preferred to adding an entry.", + "", + "PROVENANCE: every reason below was derived by reading the flagged code during", + "the audit recorded in docs/adr/0048-codeql-severity-gating.md. The 55", + "findings that the first full-tree scan surfaced (Dev CI run 527) were triaged", + "individually; the four that were reachable were FIXED in code, not accepted." + ], + + "accepted": [ + { + "rule": "go/request-forgery", + "file": "internal/tools/web_fetch.go", + "reason": "FetchURLForContext is the deliberate @url composer-handle fetch — a user-requested outbound GET is the feature, so the taint is by design. It dials through newSSRFGuardedDialer(), whose net.Dialer.Control hook runs after DNS resolution on EVERY dial and refuses netguard.IsBlockedIP: loopback, RFC1918, ULA, link-local (incl. 169.254.169.254), multicast, unspecified, RFC 6598 CGNAT (the Alibaba/Oracle 100.100.100.x metadata range), TEST-NET, RFC 2544 and 240.0.0.0/4. IPv4-mapped IPv6 is normalized first and a nil IP fails closed. Because the check is per-dial rather than per-save, DNS rebinding is closed too, and redirect hops re-dial through the same hook. http.Transport refuses any scheme but http/https, Go's 10-redirect cap applies, and the body is capped at 5 MiB. internal/netguard is the single source of truth with a 24-case regression matrix in netguard_test.go." + }, + { + "rule": "go/request-forgery", + "file": "internal/mcpoauth/discovery.go", + "reason": "These two URLs are an operator-typed MCP server URL and the remote-derived pointers reached from it (a WWW-Authenticate resource_metadata= parameter, a PRM-declared issuer). Every request uses mcpoauth.SafeHTTPClient (wired at remotemcp/service.go), whose safeDialContext resolves, rejects blocked IPs, then dials the exact validated IP — closing the resolve-to-connect TOCTOU — and whose CheckRedirect hard-fails so a 30x can never relay a bearer to a new origin. CanonicalResourceURI rejects a non-http(s) scheme, embedded userinfo, or a hostless URL before this point, fetchJSON now refuses a non-http(s) scheme by name, and maxMetadataBytes caps the body at 1 MiB. Mix-up defenses are downstream: verifyAuthServer rejects a missing or mismatched issuer and refuses a non-S256 PKCE downgrade, and Discover adopts a PRM-declared resource only when sameOrigin." + }, + { + "rule": "go/path-injection", + "file": "internal/agent/session.go", + "reason": "The value is sanitized one frame up and CodeQL loses the sanitizer across a struct-field and package boundary (chatAttachment -> agent.ImageAttachment -> TurnInput). httpapi/chat.go calls validateAttachments, which filepath.Abs+Clean's the client path, takes filepath.Rel(root, abs), rejects it unless filepath.IsLocal(rel), and then REBUILDS the path as filepath.Join(root, rel), storing only that. attachments.go is the only construction site of agent.ImageAttachment in the tree, so no unvalidated path can reach these os.Stat/os.ReadFile calls. The caller contract is documented at loadImageAttachments because the guard lives in the producer, not here." + }, + { + "rule": "go/weak-sensitive-data-hashing", + "file": "internal/sched/apikeys/apikeys.go", + "reason": "SHA-256 is a lookup INDEX over a full-entropy random token, not a password hash. generateKey mints the key from 32 crypto/rand bytes (\"sk-\" + base64url), so there is no guessable preimage to iterate and a KDF would add per-request cost without adding security. This is the standard construction for bearer-token storage." + }, + { + "rule": "go/weak-sensitive-data-hashing", + "file": "internal/sched/handlers/handlers.go", + "reason": "Both call sites hash only to equalize length before subtle.ConstantTimeCompare — the digests are compared in memory and never stored. That is the standard defense against deducing secret length from comparison timing, and the handler already fails closed when AdminAPIKey is unset." + }, + { + "rule": "go/weak-sensitive-data-hashing", + "file": "internal/store/users.go", + "reason": "The digest is taken over the BCRYPT HASH, not the password, to derive an 8-byte session-revocation epoch. The password is bcrypt'd elsewhere; this input already carries bcrypt's 128-bit random salt, and the epoch is a generation counter that /auth/verify would never accept as a credential. The reasoning is written out at the call site and sessionEpochExpr pins the SQL twin." + }, + { + "rule": "go/clear-text-logging", + "file": "cmd/fleet/main.go", + "reason": "Field-insensitive taint through ProviderConfig, the same misattribution already recorded in the //nolint:gosec at main.go:1276. All three sinks log only a wrapped boot error. The plausible flow (a decrypted admin-managed ProviderConfig.APIKey reaching resolver.go's fmt.Errorf) is unreachable: anthropic.New and openai.New in charm.land/fantasy always return a nil error, so the only errors that arm can produce are buildProvider's own literal strings. The MCP-broker and reload sinks return only clientconfig/store errors, whose bundle-config messages quote variable NAMES, never values, per the manifest doctrine in AGENTS.md." + }, + { + "rule": "go/clear-text-logging", + "file": "internal/admincli/import.go", + "reason": "Field-insensitive taint through the legacy-export struct, which happens to carry a password_hash field. stats.warnings is populated by ten warnf call sites and not one touches a secret — they carry conversation IDs, MCP server names, persona names, roles, timezones and recurrence strings. u.PasswordHash is read at exactly one place, which prints only u.Username and u.ID. The sink is the operator's own terminal." + }, + { + "rule": "go/clear-text-logging", + "file": "internal/agent/scheduled.go", + "reason": "Logs an internal run error, and agentcore's boundary errors are deliberately opaque — containedBoundaryError surfaces only the incident ID, never the recovered value or a stack. The string is additionally passed through agentcore.RedactSecrets before both this log and the persisted transcript." + }, + { + "rule": "js/remote-property-injection", + "file": "web/src/app/chat/ui/useTurnStream.ts", + "reason": "Two independent reasons, either sufficient. (1) Every flagged sink is keyed by a conversation slot id (ctx.target or convId) and by nothing else; no model-authored payload field ever reaches a key position. That id space is server-minted uuid.NewString() from store.CreateConversation — a client-supplied conversation_id is never inserted, it must resolve to an existing row owned by the caller or the request 404s — so neither a client nor the model can choose the key. (2) The sink shapes cannot reach Object.prototype anyway: an object-literal computed key performs CreateDataPropertyOrThrow, producing an OWN \"__proto__\" property and leaving the prototype untouched, and bracket assignment rebinds at most the one local record object. Worst achievable impact, given an operator-imported hostile id via the admin-CLI import path, is self-inflicted state confusion in one browser tab." + }, + { + "rule": "js/insecure-temporary-file", + "file": "web/e2e/test-auth-key.ts", + "reason": "Test-only, and the reported defect is fixed as far as it can be without changing the cross-process rendezvous contract: the write is now O_EXCL (flag \"wx\") at mode 0600 with crypto random bytes in the sibling name, so it cannot follow or clobber a pre-planted symlink and does not leave the private half world-readable. The query recognizes only mkdtemp as safe, but the fixed path is a deliberate rendezvous — playwright.config.ts is loaded in the main process AND re-imported in every worker, which must all read the same throwaway keypair. The key is generated per run, protects nothing real, and is never committed." + } + ] +} diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 00651bec..f95bbf06 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -63,12 +63,14 @@ # Grype step, govulncheck-scheduled.yml, grype-scheduled.yml). Those push their # own SARIF to the Security tab and never depended on CodeQL being configured. # -# Merge gating, in two parts — both now closed: +# Merge gating, in two parts — both closed: # -# 1. Does a finding turn the CHECK red? YES. The `Fail on findings` step below -# fails the job on any finding. Without it the analyze step exits 0 whether -# it found nothing or a hundred alerts, so a red check could only ever mean -# "the scanner broke". +# 1. Does a finding turn the CHECK red? YES, for a finding in the blocking +# band — see the `Fail on findings` step for the threshold and +# docs/adr/0048-codeql-severity-gating.md for why it is that band and not +# "any finding". Without that step the analyze step exits 0 whether it found +# nothing or a hundred alerts, so a red check could only ever mean "the +# scanner broke". # 2. Does a red check BLOCK a merge? YES, through the EXISTING required check. # This is a REUSABLE workflow (`on: workflow_call`): ci.yml and dev-ci.yml # each call it as a job, and a job that calls a reusable workflow can sit in @@ -78,13 +80,17 @@ # `needs` cannot cross workflow FILES, but a workflow_call brings the jobs # into the caller's file. # -# They cannot be part of `CI gate`: a job's `needs` cannot reach across workflow -# files. So this file carries its own aggregate `CodeQL gate` job at the bottom, -# for the same reason ci.yml and dev-ci.yml carry theirs — it is the ONE check to -# name in branch protection if CodeQL should ever become blocking, instead of -# four per-language checks that would have to be re-pointed by hand every time -# the matrix changes. Adding it here does not make it required; that is a -# repo-settings decision, deliberately not expressible from this file. +# CAVEAT, and it is load-bearing: that routing only blocks a merge where the +# aggregate gate is a REQUIRED status check. On `main` it is (`CI gate`). +# On `dev` the ruleset requires no status checks at all, so `Dev gate` is +# red-but-not-required there — see docs/SCANNING.md ("Known gaps"). +# +# The per-language `analyze` legs cannot be named directly in `CI gate`'s +# `needs` — a job's `needs` cannot reach across workflow files — but the CALLING +# job can be, and is. So the aggregate `CodeQL gate` job at the bottom of this +# file exists for the standalone schedule/dispatch runs (one legible verdict per +# weekly re-scan instead of four boxes) and as a stable single check name to put +# in a ruleset, not because the workflow_call path needs it. # See docs/CODEQL.md ("Merge gating"). name: CodeQL @@ -129,10 +135,13 @@ jobs: # supported for Go — only `autobuild` or `manual` — so the toolchain # has to be right rather than skipped. # security-extended everywhere: the broader security suite (more - # queries, lower average precision than the default). Measured before - # adoption like every other gate here — the run's own findings - # summary is the measurement, and the Fail-on-findings step means - # anything it surfaces must be fixed or reasoned away, not accrued. + # queries, lower average precision than the default). The adoption + # measurement was taken on a `pull_request` run, where CodeQL is + # DIFF-INFORMED and reports only results inside the PR's diff — so it + # measured the diff, not the tree, and read as zero when the tree held + # 55 findings. Do not re-derive a tree-wide claim from a PR run; the + # full-tree numbers come from push/schedule runs. See + # docs/adr/0048-codeql-severity-gating.md. - language: go build-mode: autobuild queries: security-extended @@ -201,15 +210,16 @@ jobs: # reading CI output, to `gh run view`, and to any automation that has # the log but not the code-scanning API. # - # This mirrors what govulncheck-scheduled.yml already does for its own - # SARIF: jq a per-rule count into the job log AND the step summary, so - # the run is readable without leaving Actions. It is reporting only and - # never fails the job — gating on findings is code scanning merge - # protection's job (see docs/CODEQL.md), not this step's. + # The listing is split into the SAME three tiers the next step gates on, + # so the log says not just what was found but which part of it blocks — + # and, importantly, prints the ACCEPTED tier by name. A waiver that is + # invisible in CI output is a waiver nobody re-reads. Reporting only; the + # `Fail on findings` step below is what fails the job. if: ${{ !cancelled() }} env: SARIF_DIR: ${{ runner.temp }}/codeql-sarif LANGUAGE: ${{ matrix.language }} + ACCEPTED_FILE: ${{ github.workspace }}/.github/codeql-accepted-findings.json # The database directory is named after the EXTRACTOR, which is not # always the matrix language: javascript-typescript -> javascript. CODEQL_DB: ${{ matrix.language == 'javascript-typescript' && 'javascript' || matrix.language }} @@ -229,19 +239,42 @@ jobs: echo '```' # `-s` slurps every SARIF doc into one array, so a language that # emits more than one file is still counted once, in total. - jq -rs ' - [ .[] | .runs[]? | .results[]? ] as $res - | if ($res | length) == 0 then "No findings." - else - ( $res - | map( - "[\(.level // "note")] \(.ruleId) " - + ((.locations[0].physicalLocation // {}) as $l - | "\($l.artifactLocation.uri // "?"):\($l.region.startLine // "?")") - ) - | sort | join("\n") - ) + "\n--\ntotal findings: \($res | length)" - end + # + # security-severity lives on the RULE, not the result, so each + # result is joined back to its rule in the same run's + # tool.driver.rules[] to recover it. Absent -> 0, which is + # correct: a query with no security-severity is not a High. + jq -rs --slurpfile reg "$ACCEPTED_FILE" ' + ( reduce ($reg[0].accepted[]? | "\(.rule) \(.file)") as $k ({}; .[$k] = true) ) as $waived + | [ .[] | .runs[]? + | ( [ .tool.driver.rules[]? ] ) as $rules + | .results[]? + | . as $r + | ( [ $rules[] | select(.id == $r.ruleId) + | .properties["security-severity"] ][0] // "" ) as $sev + | ( ($r.locations[0].physicalLocation.artifactLocation.uri) // "" ) as $file + | ( ($r.level) // "note" ) as $level + | { level: $level, + sev: (($sev | tonumber? // 0)), + rule: $r.ruleId, + loc: "\($file):\(($r.locations[0].physicalLocation.region.startLine) // "?")", + waived: ( (($waived | has("\($r.ruleId) \($file)"))) + or ((($r.suppressions // []) | length) > 0) ), + high: ( $level == "error" or $level == "warning" or (($sev | tonumber? // 0) >= 7.0) ) } + ] as $all + | ( $all | map(select(.high and (.waived | not))) ) as $b + | ( $all | map(select(.high and .waived)) ) as $w + | ( $all | map(select(.high | not)) ) as $n + | "BLOCKING — High+ and not registered (\($b|length)):", + ( if ($b|length) == 0 then " none" else ($b | sort_by(.rule,.loc) | .[] | " [\(.level)] sec-sev=\(.sev) \(.rule) \(.loc)") end ), + "", + "ACCEPTED — High+ waived in codeql-accepted-findings.json or in-source (\($w|length)):", + ( if ($w|length) == 0 then " none" else ($w | sort_by(.rule,.loc) | .[] | " [\(.level)] sec-sev=\(.sev) \(.rule) \(.loc)") end ), + "", + "ADVISORY — below High, triage in the Security tab (\($n|length)):", + ( if ($n|length) == 0 then " none" else ($n | sort_by(.rule,.loc) | .[] | " [\(.level)] sec-sev=\(.sev) \(.rule) \(.loc)") end ), + "", + "totals: \($all|length) finding(s) — \($b|length) blocking, \($w|length) accepted, \($n|length) advisory" ' "${files[@]}" echo '```' # COVERAGE, not just the verdict: "No findings." alone cannot be @@ -268,41 +301,91 @@ jobs: # code has a problem", which is precisely how the Go toolchain break sat # unnoticed behind a red-but-not-required check for weeks. # - # Threshold is ANY finding, deliberately. The security suite currently - # reports ZERO across go/python/javascript-typescript/actions, so there - # is no backlog to grandfather and no severity line to argue about — a - # finding here is new. Switching a gate on over an existing backlog is - # how a gate becomes something people route around. + # THRESHOLD, in three parts: + # 1. A finding blocks when its SARIF level is `error`/`warning`, or its + # rule carries security-severity >= 7.0 (CodeQL's High/Critical + # band). Note-level findings below High are printed and uploaded to + # the Security tab; they do not block. + # 2. A finding whose (rule, file) pair appears in + # .github/codeql-accepted-findings.json does not block. Each entry + # carries a written reason. The waiver is per-FILE, so the rule + # stays live everywhere else — a new go/request-forgery in a + # different file still fails the build. + # 3. An in-source `// codeql[rule-id]` suppression is honored (CodeQL + # emits it as a `suppressions` array on the result). # - # Runs after the summary so the log leads with WHAT was found. + # WHY NOT "any finding" — that was tried in #1246, and it deadlocked the + # repo. That gate was armed on a measurement of ZERO across all four + # languages, but the measurement came from a `pull_request` run (Dev CI + # run 525), and on pull_request events the CodeQL action runs + # DIFF-INFORMED: it builds the full database and evaluates every query, + # then reports only results located inside the PR's diff. Run 525's own + # log says so — "Persisted 204 diff range(s) across 43 file(s)", and + # "file coverage information is only enabled when analyzing the default + # branch and protected branches". + # + # So the first full-tree scan was the PUSH that merged it (Dev CI run + # 527), which reported 38 Go and 17 javascript-typescript findings and + # turned `Dev gate` red with no PR to fix it through. The any-finding + # threshold was never armed over a clean tree; it was armed over a + # 55-finding backlog nobody had measured yet. # - # NOTE ON SCOPE: this makes the CHECK red. Whether a red check BLOCKS a - # merge is branch protection's call — `CodeQL gate` has to be a required - # status check for that, which is a repo-settings action a workflow file - # cannot perform. See docs/CODEQL.md ("Merge gating"). + # Those 55 were triaged individually. Four were reachable and are FIXED + # in code, not accepted — the task-create log line whose update-path twin + # was already sanitized, two pre-validation client paths on reject + # branches, a client-echoed attachment name, and a world-readable + # private-key write in the e2e harness. The rest are false positives in + # fleet's threat model and are registered with their reasons. + # + # Runs after the summary so the log leads with WHAT was found. + # See docs/CODEQL.md ("Merge gating") and + # docs/adr/0048-codeql-severity-gating.md. if: ${{ !cancelled() }} env: SARIF_DIR: ${{ runner.temp }}/codeql-sarif LANGUAGE: ${{ matrix.language }} + # GITHUB_WORKSPACE, not ${{ github.workspace }} interpolated into the + # run: block — same reason the summary step uses RUNNER_TEMP. + ACCEPTED_FILE: ${{ github.workspace }}/.github/codeql-accepted-findings.json run: | set -uo pipefail shopt -s nullglob files=("$SARIF_DIR"/*.sarif) if [ ${#files[@]} -eq 0 ]; then - # No SARIF means the analysis did not produce results to judge. Fail - # loudly rather than reporting a clean scan that never happened. echo "::error::No SARIF written for $LANGUAGE — cannot conclude the scan was clean." exit 1 fi - count=$(jq -rs '[ .[] | .runs[]? | .results[]? ] | length' "${files[@]}") - if [ "$count" != "0" ]; then - echo "::error::CodeQL found ${count} finding(s) for ${LANGUAGE} — see the summary above." - echo "Fix it, or if it is a false positive dismiss the alert in the" - echo "Security tab with a reason, or add a query filter with a comment" - echo "saying why. Silently raising the threshold is not one of the options." + if [ ! -f "$ACCEPTED_FILE" ]; then + echo "::error::accepted-findings register $ACCEPTED_FILE is missing — refusing to gate without it." + exit 1 + fi + blocking=$(jq -rs --slurpfile reg "$ACCEPTED_FILE" ' + ( reduce ($reg[0].accepted[]? | "\(.rule) \(.file)") as $k ({}; .[$k] = true) ) as $waived + | [ .[] | .runs[]? + | ( [ .tool.driver.rules[]? ] ) as $rules + | .results[]? + | . as $r + | ( [ $rules[] | select(.id == $r.ruleId) + | .properties["security-severity"] ][0] // "" ) as $sev + | ( ($r.locations[0].physicalLocation.artifactLocation.uri) // "" ) as $file + | ( ($r.level) // "note" ) as $level + | ( "\($r.ruleId) \($file)" ) as $key + | select( (($r.suppressions // []) | length) == 0 ) + | select( ($waived | has($key)) | not ) + | select( $level == "error" or $level == "warning" or (($sev | tonumber? // 0) >= 7.0) ) + | " [\($level)] sec-sev=\($sev) \($r.ruleId) \($file):\(($r.locations[0].physicalLocation.region.startLine) // "?")" + ] | .[]' "${files[@]}") + rc=$? + if [ $rc -ne 0 ]; then + echo "::error::could not evaluate the SARIF for $LANGUAGE — refusing to report it clean." + exit 1 + fi + if [ -n "$blocking" ]; then + echo "::error::CodeQL found blocking finding(s) for ${LANGUAGE}:" + echo "$blocking" exit 1 fi - echo "CodeQL ($LANGUAGE): 0 findings." + echo "CodeQL ($LANGUAGE): 0 blocking findings." codeql-gate: name: CodeQL gate diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index b4153e6f..6ef818a9 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -169,11 +169,15 @@ jobs: codeql: # Reusable-workflow call (see codeql.yml's header): puts the CodeQL jobs in - # THIS graph so `Dev gate` blocks on them. A finding fails the gate — the - # `Fail on findings` step inside makes green mean "clean", not just "ran". + # THIS graph so `Dev gate` blocks on them. A finding in the blocking band + # fails the gate, so green means "no unwaived High-and-above finding", not + # just "ran" — and on a `pull_request` event it means that of the DIFF only, + # because CodeQL is diff-informed there (docs/adr/0048-codeql-severity-gating.md). + # The tree-wide verdict comes from the push runs. # Unconditional (no docs-only detection in the fast lane, and Dev gate # demands strict success, so a skip would fail it); running on dev pushes - # too also covers any direct push that bypassed a PR. + # too also covers any direct push that bypassed a PR — and is the only event + # shape that scans the whole tree. permissions: contents: read security-events: write diff --git a/docs/adr/0048-codeql-severity-gating.md b/docs/adr/0048-codeql-severity-gating.md new file mode 100644 index 00000000..e4866b32 --- /dev/null +++ b/docs/adr/0048-codeql-severity-gating.md @@ -0,0 +1,157 @@ +# ADR-0048: CodeQL gates on High-and-above plus a reviewed accepted-findings register + +- **Status:** Accepted +- **Date:** 2026-08-22 +- **Deciders:** fleet maintainers +- **Amends:** the gating decision shipped in #1246 (`docs/CODEQL.md`, + `docs/SCANNING.md`) — the CodeQL threshold changes from *any finding* to + *High-and-above, minus a reviewed register*. No other scanner's threshold + changes. + +## Context + +#1246 restored CodeQL as **advanced setup** running `security-extended` over +go / python / javascript-typescript / actions, added a `Fail on findings` step, +and routed the result into `ci-gate` / `Dev gate` through `workflow_call`. All of +that was right and none of it is revisited here. + +The **threshold** was wrong, and it was wrong for an instructive reason. + +The step failed the job on *any* finding at any severity. Its own comment +recorded the justification: + +> Threshold is ANY finding, deliberately. The security suite currently reports +> ZERO across go/python/javascript-typescript/actions, so there is no backlog to +> grandfather and no severity line to argue about — a finding here is new. + +That zero was real, and it was measured — on Dev CI run 525, a `pull_request` +event. On `pull_request` events the CodeQL action runs **diff-informed**: it +builds the full database and evaluates every query, then reports only results +whose location falls inside the PR's diff. Run 525's own log says both halves +out loud: + +``` +Computing PR diff ranges... +Persisted 204 diff range(s) across 43 file(s). +Successfully created diff range extension pack at .../pr-diff-range +codeql database run-queries ... --extension-packs=codeql-action/pr-diff-range +``` +``` +To speed up pull request analysis, file coverage information is only enabled +when analyzing the default branch and protected branches. +``` + +The Go database held all 428 files and the queries that later fired did run — +`LogInjection.ql`, `TaintedPath.ql`, `RequestForgery.ql`, +`WeakSensitiveDataHashing.ql` are all listed as "Interpreted" in that run. The +SARIF was empty because the results were filtered to the PR's 43 changed files. + +So the first full-tree evaluation of `security-extended` against this repository +was the **push** that merged #1246: Dev CI run 527, which reported **38 Go and 17 +javascript-typescript findings** and turned `Dev gate` red. The gate then blocked +every subsequent push to `dev` — including any push that would have fixed it — +with no PR-shaped path out, because a PR into `dev` is scanned diff-informed and +therefore green while `dev` itself stays red. + +Two conclusions, and the second is the one that generalises: + +1. The any-finding threshold was never armed over a clean tree. It was armed over + a 55-finding backlog that no one had measured, because the only measurement + available at PR time is structurally incapable of showing it. +2. **A PR-event CodeQL run cannot certify a tree.** It certifies a diff. Any + claim of the form "the scanners are green, therefore the tree is clean" that + rests on a `pull_request` run is unsound, and that is a permanent property of + diff-informed analysis, not a bug to be fixed. + +The 55 were then triaged individually against the code. Four were reachable: + +- `internal/sched/handlers/handlers.go` logged `task.Prompt` unsanitized on the + task-create path, while the **update** path's twin line was already wrapped in + `logSafe`. `POST /tasks` is reachable by a scoped `create_task` key, so this + was genuine log forgery — and demonstrably so. +- `internal/httpapi/attachments.go` logged the raw client attachment path with + `%s` on the two branches where the containment guard had just *failed*, i.e. + precisely where the value is hostile by construction. +- `internal/agent/session.go` logged the client-echoed attachment `Name`, which + — unlike `Path` — is never re-sanitized on the `/chat` path. +- `web/e2e/test-auth-key.ts` wrote an Ed25519 private key to a fully predictable + path in the world-writable temp dir at default `0644`. + +Those four are fixed in code. The remaining 51 are false positives in fleet's +threat model, and the interesting part is that **severity alone does not separate +them**: `go/request-forgery` carries security-severity 9.1 and fires on +`web_fetch.go`, which is a deliberate user-facing fetch tool sitting behind +`internal/netguard`'s resolve-then-dial SSRF guard. `go/weak-sensitive-data-hashing` +carries 7.5 and fires on SHA-256 used as a lookup index over a 32-byte +`crypto/rand` token — the recommended construction. A pure severity line would +block both. + +## Decision + +**CodeQL blocks on a finding that is (a) at SARIF level `error`/`warning`, or has +security-severity >= 7.0, and (b) is not waived.** Findings below that band are +printed and uploaded to the Security tab as advisory. Waivers come from two +places: + +1. `.github/codeql-accepted-findings.json` — a register of accepted + `(rule, file)` pairs, each with a mandatory written reason. +2. An in-source `// codeql[rule-id]` comment, which CodeQL emits as a + `suppressions` array on the result. (Both `go` and `javascript` ship an + `AlertSuppression.ql`; the comment must sit on its own line and covers the + line immediately below it.) + +The register is **per-file, not per-rule**, and that is the whole point of +preferring it to a `query-filters` exclude. A `query-filters: exclude: {id: +go/request-forgery}` switches a security-severity 9.1 query off for the entire +repository; the register waives it in `internal/tools/web_fetch.go` and +`internal/mcpoauth/discovery.go` and leaves it live everywhere else, including +elsewhere in those same packages. This is asserted, not asserted-and-hoped: a +synthetic SARIF carrying a fresh `go/request-forgery` in an unregistered file +fails the gate, and that case is exercised as part of validating the jq. + +Three anti-rot controls, because a waiver register that nobody re-reads is worse +than no register: + +- `scripts/check_codeql_register_test.go` (in `make test`) requires every entry + to name a file that exists, carry a substantive reason, use a plausible rule + id, and be unique — and asserts that `codeql.yml` still references the register + at all, so the two cannot be silently decoupled. +- The gate **fails closed** if the register is missing, and fails closed if the + jq cannot be evaluated. A scan that could not be judged is never reported clean. +- The job log and step summary print the **ACCEPTED tier by name**, alongside + BLOCKING and ADVISORY, so every waiver is visible in ordinary CI output rather + than only in a file somebody has to think to open. + +## Consequences + +**What gets better.** `dev` and `main` are unblocked, and for the first time the +push-event runs report a verdict that means something: the High-and-above band is +enforced tree-wide, on every push, with a reviewed exception list. The +`security-extended` suite keeps running in full — nothing is filtered out of the +Security tab — so the 51 advisory/accepted findings remain visible for triage. +The specific claim "a green CodeQL check means the tree is clean" is now +false-by-construction only for PR events, and the docs say so instead of implying +otherwise. + +**What gets worse.** A note-level regression no longer fails the build. If +someone adds a 23rd `go/log-injection` sink on genuinely untrusted input, CI will +not stop them; it will appear in the advisory tier and in the Security tab. This +is a deliberate trade: the alternative, as demonstrated above, is a gate that +blocks every push and therefore gets routed around or switched off. `gosec`'s +G706 covers the same log-injection class in `golangci-lint`, which **does** block +via `ci-gate`, and carries 77 reviewed per-site annotations — so this class is +not unguarded, it is guarded by the instrument that was already there. + +**What is now load-bearing.** Widening the register is a security decision that +shows up in a PR diff, and reviewers are expected to check the reason against the +code rather than the reason's existence. That is a process control, and process +controls decay; the tests above are what make the decay visible. + +**Known limitation, stated rather than fixed.** The 621 `_test.go` files remain +outside the Go database (autobuild builds packages, not tests) — unchanged from +default setup and from #1246. And the register keys on `(rule, file)` rather than +`(rule, file, line)` deliberately: line numbers churn on every edit, and a +register that fails on unrelated refactors is a register people delete. The cost +is that a *second*, genuinely bad instance of an already-waived rule in an +already-waived file would not block. That is the sharpest edge here, and it is +the reason the reason-strings name the specific call sites and their guards. diff --git a/scripts/check_codeql_register_test.go b/scripts/check_codeql_register_test.go new file mode 100644 index 00000000..fc514f88 --- /dev/null +++ b/scripts/check_codeql_register_test.go @@ -0,0 +1,137 @@ +// Copyright (c) 2025 ElcanoTek +// SPDX-License-Identifier: MIT + +package scripts + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" +) + +// .github/codeql-accepted-findings.json waives specific CodeQL findings from the +// blocking gate in .github/workflows/codeql.yml. A waiver register is only worth +// anything if it cannot rot, so these tests are the anti-rot controls an auditor +// would ask for: +// +// - every entry names a real file, so a waiver cannot outlive the code it was +// written about (a renamed or deleted file silently widens coverage loss — +// the gate would stop matching, but nobody would notice the dead entry); +// - every entry carries a substantive reason, so "why is this accepted?" is +// answerable from the repo rather than from a PR conversation; +// - no duplicate (rule, file) pairs, so there is exactly one reviewed reason +// per waiver rather than two that can disagree; +// - the rule ids look like CodeQL rule ids, so a typo fails here instead of +// silently never matching (a waiver that matches nothing is indistinguishable +// from a waiver that works, until the day it was supposed to fire). + +type acceptedFinding struct { + Rule string `json:"rule"` + File string `json:"file"` + Reason string `json:"reason"` +} + +type acceptedRegister struct { + Accepted []acceptedFinding `json:"accepted"` +} + +func loadRegister(t *testing.T) (acceptedRegister, string) { + t.Helper() + root := repoRoot(t) + path := filepath.Join(root, ".github", "codeql-accepted-findings.json") + raw, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + var reg acceptedRegister + if err := json.Unmarshal(raw, ®); err != nil { + t.Fatalf("%s is not valid JSON: %v", path, err) + } + return reg, root +} + +// The register is consumed by jq in the workflow, which fails the build if the +// file is missing but cannot judge whether an entry still makes sense. +func TestCodeQLRegisterEntriesAreWellFormed(t *testing.T) { + reg, _ := loadRegister(t) + if len(reg.Accepted) == 0 { + // Not an error in principle — an empty register means nothing is waived, + // which is the goal state. Say so rather than asserting a count that + // would have to be edited every time a finding is genuinely fixed. + t.Log("register is empty: no CodeQL findings are currently waived") + return + } + for _, e := range reg.Accepted { + if strings.TrimSpace(e.Rule) == "" { + t.Errorf("entry with file %q has no rule", e.File) + continue + } + // CodeQL rule ids are "/" — e.g. go/request-forgery. + if !strings.Contains(e.Rule, "/") || strings.ContainsAny(e.Rule, " \t") { + t.Errorf("rule %q does not look like a CodeQL rule id (want /)", e.Rule) + } + if strings.TrimSpace(e.File) == "" { + t.Errorf("entry for rule %q has no file", e.Rule) + } + // A reason has to actually say something. The gate cannot check this and + // a reviewer skimming a diff might not either. + if len(strings.TrimSpace(e.Reason)) < 80 { + t.Errorf("rule %q file %q: reason is too short to be a justification (%d chars) — say why the finding cannot be exploited HERE", + e.Rule, e.File, len(strings.TrimSpace(e.Reason))) + } + if strings.Contains(strings.ToLower(e.Reason), "false positive") && + len(strings.TrimSpace(e.Reason)) < 160 { + t.Errorf("rule %q file %q: %q is an assertion, not a justification", + e.Rule, e.File, e.Reason) + } + } +} + +// A waiver that names a file which no longer exists matches nothing, so the gate +// would silently start blocking (or, worse, the finding moved to a file that is +// NOT waived and nobody connected the two). Either way the entry is stale. +func TestCodeQLRegisterFilesExist(t *testing.T) { + reg, root := loadRegister(t) + for _, e := range reg.Accepted { + if strings.TrimSpace(e.File) == "" { + continue + } + // The register stores repo-relative, forward-slash SARIF URIs. + p := filepath.Join(root, filepath.FromSlash(e.File)) + if _, err := os.Stat(p); err != nil { + t.Errorf("rule %q waives %q, which does not exist — remove the entry or repoint it: %v", + e.Rule, e.File, err) + } + } +} + +// Two entries for the same (rule, file) means two reasons that can drift apart, +// and the gate would honor whichever it saw first. +func TestCodeQLRegisterHasNoDuplicates(t *testing.T) { + reg, _ := loadRegister(t) + seen := make(map[string]bool, len(reg.Accepted)) + for _, e := range reg.Accepted { + key := e.Rule + " " + e.File + if seen[key] { + t.Errorf("duplicate register entry for %q", key) + } + seen[key] = true + } +} + +// The workflow reads the register by an exact path. If that path moves, the gate +// fails closed (it refuses to run without the file) — but only at CI time, on +// whatever PR happens to move it. Assert the coupling here instead. +func TestCodeQLWorkflowReferencesTheRegister(t *testing.T) { + root := repoRoot(t) + raw, err := os.ReadFile(filepath.Join(root, ".github", "workflows", "codeql.yml")) + if err != nil { + t.Fatalf("read codeql.yml: %v", err) + } + const want = ".github/codeql-accepted-findings.json" + if !strings.Contains(string(raw), want) { + t.Fatalf("codeql.yml no longer references %s — the gate and the register have been decoupled", want) + } +} From 9a573e8aec64c62c1314c84fb0104e369e1709fe Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 16:06:01 +0000 Subject: [PATCH 18/34] Fix a vacuous CodeQL gate, and harden the CI supply chain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate I added in the previous commit was VACUOUS, and the first full-tree run proved it: run 32583247659 reported "0 blocking" over a tree holding 30 findings, with sec-sev=0 on every one of them — including go/request-forgery, whose real security-severity is 9.1. Cause: CodeQL writes query metadata into runs[].tool.extensions[].rules[] (one extension per query pack), NOT runs[].tool.driver.rules[]. The driver is the CLI itself. Reading only the driver resolved nothing, so every finding scored 0 and nothing could ever reach the High band. For the same reason a result's `level` is usually absent from the result: SARIF falls back to the rule's defaultConfiguration.level, which was also unreachable. Three changes, in order of importance: 1. The classifier now reads driver.rules AND extensions[].rules, and resolves level from the rule when the result omits it. Verified against a fixture built to the real SARIF shape. 2. A VACUITY CHECK that would have caught this: if a scan produced findings but resolved zero rule metadata, the job fails instead of reporting clean. "Findings but no metadata" means the lookup is broken and the gate is evaluating nothing — the green-but-vacuous outcome this workflow exists to rule out, which I then walked straight into. 3. The banding is now security-severity only, with level as the fallback for a rule that publishes no security-severity. Once metadata resolved, banding on level as well put all 23 go/log-injection findings (security-severity 6.1) into the blocking tier, because nearly every CodeQL security query is @problem.severity error — level carries no severity information for them. That would have reproduced the any-finding deadlock by a different route. The filter now lives in .github/codeql-gate.jq and is used by both the summary and the gate via `jq -f`, so the thing that reports and the thing that blocks cannot disagree about what "blocking" means — and it can be exercised against fixture SARIF with the exact bytes CI runs. Exercised end-to-end from the YAML: the real-shape fixture (0 blocking, 8 accepted, 2 advisory), a fresh unregistered go/request-forgery in an unwaived file (blocks — the property that makes the per-file register different from a query-filters exclude), the vacuity fixture (fails), missing register, missing filter, and unparseable SARIF. CI supply chain, from the same audit: - Repin github/codeql-action (5 refs) and golangci/golangci-lint-action (2 refs). Both were pinned to the ANNOTATED TAG OBJECT of a MUTABLE major tag, not to a commit: `refs/tags/v4` -> 4c0873ef but `refs/tags/v4^{}` -> db488dde. A tag object is immutable but only reachable while that tag points at it, so the day upstream moves v4 — which codeql-action does on essentially every release — the object is unreferenced and Actions can no longer resolve the ref. A self-inflicted CI outage with no attacker involved, armed in seven places. Verified with `git ls-remote --tags` and repinned to the peeled commits, with exact `# vX.Y.Z` comments (they read `# v4 (4.37.8)` and `# v9`, which dependabot-core parses as "4" and "9"). scripts/check_action_pins_test.go now enforces the shape across all 53 third-party refs. - build-sandbox-image.yml / publish-sandbox-image.yml: replace the fleet_ref deny-list with an allow-list. The deny-list had two holes. GITHUB_OUTPUT newline injection: a workflow_call string input may contain newlines and the value was printf'd unsanitized, so fleet_ref="main\nresolved=refs/pull/1/head" matched no deny pattern, exited 0, and emitted two `resolved=` lines — last-wins handed the attacker the ref, and the same primitive forges any step output. And a raw commit SHA: "every ref here is collaborator-written except refs/pull/*" is true of named refs and false of reachable commits, since GitHub keeps fork-PR commits in the base repo's object store and actions/checkout will fetch a bare SHA. Both matter because these workflows EXECUTE the checked-out build script, and the publish twin holds packages: write with a live GHCR login. Tested: 5 legitimate refs pass, 9 attack shapes fail, including the injection payload. - ci.yml docs-only classifier: `*.md` matched at any depth and `docs/*` matched everything under docs/, so a PR touching only internal/clientconfig/builtin_skills/*/SKILL.md (go:embed'd and asserted by three test files), config/default/system_prompts/*.md (the shipped prompts docs/PROMPT-CACHE-CONTRACT.md exists to protect), or docs/openapi.yaml (asserted by openapi_drift_test.go) was classified docs-only — and every job skipped while CI gate reported green. Narrowed to a prose allow-list. - ci.yml ci-gate: a `skipped` job passed the gate unconditionally. Now a skip is only accepted when the classifier actually said docs-only; otherwise the gate refuses to pass over a suite that did not run. Same rot pattern as red-but-not-required, colours inverted. Gate logic tested in six directions. - scripts/check_gate_needs_test.go: assert every job in ci.yml and dev-ci.yml is in its aggregate gate's `needs`. Both are complete today (11/11 and 7/7); nothing asserted it, and adding a job without extending needs is a silent one-line regression that produces a red-but-not-required check — the exact failure #1246 was written to stop recurring. Signed-off-by: Brad Flaugher --- .github/codeql-gate.jq | 74 ++++++ .github/workflows/build-sandbox-image.yml | 33 ++- .github/workflows/ci.yml | 46 +++- .github/workflows/codeql.yml | 263 +++++++++----------- .github/workflows/dev-ci.yml | 2 +- .github/workflows/govulncheck-scheduled.yml | 2 +- .github/workflows/grype-scheduled.yml | 2 +- .github/workflows/publish-sandbox-image.yml | 33 ++- scripts/check_action_pins_test.go | 93 +++++++ scripts/check_gate_needs_test.go | 96 +++++++ 10 files changed, 485 insertions(+), 159 deletions(-) create mode 100644 .github/codeql-gate.jq create mode 100644 scripts/check_action_pins_test.go create mode 100644 scripts/check_gate_needs_test.go diff --git a/.github/codeql-gate.jq b/.github/codeql-gate.jq new file mode 100644 index 00000000..0ae963d6 --- /dev/null +++ b/.github/codeql-gate.jq @@ -0,0 +1,74 @@ +# Shared CodeQL SARIF classifier. Used by BOTH steps in +# .github/workflows/codeql.yml (the summary and the gate) via `jq -f`, so the +# thing that reports and the thing that blocks can never drift apart — and so it +# can be exercised against fixture SARIF locally with the exact file CI runs. +# +# Input: `jq -rs --slurpfile reg .github/codeql-accepted-findings.json -f this` +# over one or more CodeQL SARIF files. +# Output: one JSON object, `{blocking: [...], accepted: [...], advisory: [...], +# ruleMetaCount: N, total: N}`. The caller formats it. +# +# WHY THE RULE LOOKUP IS THE WAY IT IS — this is the subtle part, and getting it +# wrong makes the gate silently vacuous rather than loudly broken: +# +# CodeQL writes query metadata into `runs[].tool.extensions[].rules[]` (one +# extension per query pack), NOT into `runs[].tool.driver.rules[]`. The driver is +# the CodeQL CLI itself. A first cut of this filter read only driver.rules, found +# nothing, and therefore scored EVERY finding at security-severity 0 — including +# go/request-forgery, whose real value is 9.1. The gate passed with "0 blocking" +# on a tree holding 30 findings, which is exactly the green-but-vacuous outcome +# the workflow exists to rule out. Verified against the actual SARIF from run +# 32583247659. +# +# For the same reason, a result's SEVERITY LEVEL usually is not on the result at +# all: SARIF says an omitted `level` falls back to the rule's +# `defaultConfiguration.level`, and CodeQL relies on that. So the level is +# resolved from the rule too, with the result's own `level` winning when present. +# +# `ruleMetaCount` is returned so the caller can fail closed when results exist +# but no rule metadata resolved — i.e. when this lookup has broken again. + +# Every rule object anywhere in the tool description, keyed by id. +( [ .[] | .runs[]? + | ( [ .tool.driver.rules[]? ] + [ .tool.extensions[]?.rules[]? ] )[] + | select(.id != null) + ] ) as $ruleList +| ( reduce $ruleList[] as $r ({}; .[$r.id] = $r) ) as $rules +| ( reduce ($reg[0].accepted[]? | "\(.rule) \(.file)") as $k ({}; .[$k] = true) ) as $waived +| ( [ .[] | .runs[]? | .results[]? + | . as $res + | ( $rules[$res.ruleId] // {} ) as $rule + | ( ($rule.properties["security-severity"]) // "" ) as $sevRaw + | ( $sevRaw | tonumber? // 0 ) as $sev + | ( ($sevRaw | tonumber? | type == "number") // false ) as $hasSev + | ( ($res.level) // ($rule.defaultConfiguration.level) // "note" ) as $level + | ( ($res.locations[0].physicalLocation.artifactLocation.uri) // "" ) as $file + | ( ($res.locations[0].physicalLocation.region.startLine) // "?" ) as $line + | { rule: $res.ruleId, + file: $file, + line: $line, + level: $level, + sev: $sev, + # An in-source `// codeql[rule-id]` comment lands here. + suppressed: ((($res.suppressions // []) | length) > 0), + waived: ($waived | has("\($res.ruleId) \($file)")), + hasSev: $hasSev, + # HIGH BAND. security-severity is the dimension that carries severity + # information; CodeQL's own High/Critical cut is 7.0, and that is what + # GitHub's code-scanning merge protection bands on. + # + # `level` (i.e. @problem.severity) is NOT a severity signal for a + # security query — almost every one of them is `error`, including + # go/log-injection at security-severity 6.1. Banding on level as well + # would put all 23 log-injection findings in the blocking tier and + # reproduce the any-finding deadlock this replaced. So level is used + # ONLY as the fallback for a rule that publishes no security-severity + # at all (a non-security query), where it is the only signal there is. + high: (if $hasSev then $sev >= 7.0 + else ($level == "error" or $level == "warning") end) } + ] ) as $all +| { total: ($all | length), + ruleMetaCount: ($ruleList | length), + blocking: [ $all[] | select(.high and (.waived | not) and (.suppressed | not)) ], + accepted: [ $all[] | select(.high and (.waived or .suppressed)) ], + advisory: [ $all[] | select(.high | not) ] } diff --git a/.github/workflows/build-sandbox-image.yml b/.github/workflows/build-sandbox-image.yml index b44ada1c..98ace253 100644 --- a/.github/workflows/build-sandbox-image.yml +++ b/.github/workflows/build-sandbox-image.yml @@ -118,11 +118,38 @@ jobs: REQUESTED: ${{ inputs.fleet_ref || 'main' }} run: | set -euo pipefail + # ALLOW-LIST, not a deny-list. The previous deny-list (`refs/pull/*|pull/*|-*`) + # had two holes, both of which this closes by construction: + # + # 1. GITHUB_OUTPUT newline injection. A workflow_call string input may contain + # newlines, and `printf 'resolved=%s\n'` wrote it unsanitized, so + # `fleet_ref: "main\nresolved=refs/pull/1/head"` matched no deny pattern + # (the string starts "main"), exited 0, and emitted TWO `resolved=` lines — + # last-wins gave the attacker the ref. The same primitive could forge any + # step output. The character class below admits no newline, so the value + # cannot carry a second assignment. + # 2. A raw commit SHA. "Every ref in this repo is collaborator-written except + # refs/pull/*" is true of NAMED REFS and false of reachable COMMITS: GitHub + # keeps fork-PR commits in the base repo's object store and + # actions/checkout will happily fetch a bare SHA. So a bare hex SHA is + # refused too — pass a branch or tag name, which only a collaborator can + # create. case "$REQUESTED" in - refs/pull/*|pull/*|-*) - echo "::error::fleet_ref '$REQUESTED' is refused: pull-request refs carry fork-controlled code, and this workflow executes the checked-out build script." - exit 1 ;; + "") + echo "::error::fleet_ref is empty"; exit 1 ;; + *[!a-zA-Z0-9._/-]*) + echo "::error::fleet_ref contains a character outside [A-Za-z0-9._/-] (newline, space or shell metacharacter). Refused before checkout."; exit 1 ;; + -*|*..*|*//*) + echo "::error::fleet_ref '$REQUESTED' is not a plausible ref name"; exit 1 ;; + refs/pull/*|pull/*) + echo "::error::fleet_ref '$REQUESTED' is refused: pull-request refs carry fork-controlled code, and this workflow executes the checked-out build script."; exit 1 ;; esac + # Refuse a bare commit SHA (see 2 above). A ref name that happens to be all-hex + # and >=7 chars must be given in its refs/heads/ or refs/tags/ form. + if [ -z "$(printf '%s' "$REQUESTED" | tr -d '0-9a-fA-F')" ] && [ "${#REQUESTED}" -ge 7 ]; then + echo "::error::fleet_ref '$REQUESTED' looks like a raw commit SHA. Fork pull-request commits are reachable by SHA from this repository, so only named refs are accepted (e.g. main, refs/heads/x, v1.2.3)." + exit 1 + fi printf 'resolved=%s\n' "$REQUESTED" >> "$GITHUB_OUTPUT" - name: Checkout fleet (build script) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 99bf8d60..241bd38e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -64,8 +64,31 @@ jobs: while IFS= read -r f; do [ -z "$f" ] && continue # `*` in a shell case pattern spans '/', so these match at any depth. + # + # This list is deliberately an ALLOW-LIST of prose, and deliberately + # NOT a bare `*.md` or `docs/*`. Both of those were wrong, and both + # made `CI gate` report green over a suite that never ran: + # + # - `*.md` matched at any depth, so it swallowed 15 markdown files + # that are COMPILED PRODUCT CONTENT — every + # internal/clientconfig/builtin_skills/*/SKILL.md (pulled in by + # `//go:embed all:builtin_skills` and asserted by + # builtin_skills_browserbase_test.go, clientconfig_test.go and + # validate_config_test.go), plus config/default/system_prompts/ + # {default,chat}.md, which ARE the shipped system prompts that + # docs/PROMPT-CACHE-CONTRACT.md exists to protect. + # - `docs/*` swallowed docs/openapi.yaml, which + # cmd/fleet/openapi_drift_test.go asserts against the Go models, + # and docs/scripts/*.py + docs/img/*.py, which are inside the + # ruff, Semgrep p/python and CodeQL python scopes. + # + # So a PR touching only a shipped prompt or the OpenAPI spec skipped + # the very tests that validate it. Keep this list prose-only; when in + # doubt, leave a path OFF it and run the full suite. case "$f" in - *.md|docs/*|LICENSE) : ;; # documentation: no compiled/runtime signal + docs/*.md|LICENSE) : ;; + README.md|CHANGELOG.md|CONTRIBUTING.md|SECURITY.md) : ;; + CODE_OF_CONDUCT.md|AGENTS.md|CLAUDE.md|ONBOARDING.md) : ;; *) docs_only=false ;; esac done <<< "$files" @@ -231,7 +254,7 @@ jobs: run: go vet -tags fleet_host_executor ./... - name: golangci-lint - uses: golangci/golangci-lint-action@db9de0fc1a667e1a49d2291a1a042dff081d78f6 # v9 + uses: golangci/golangci-lint-action@ba0d7d2ec06a0ea1cb5fa41b2e4a3ab91d21278a # v9.3.0 with: # Repo .golangci.yml is the v2 schema (version: "2"). It no longer # pins run.go: golangci-lint's documented default is "use Go version @@ -791,7 +814,7 @@ jobs: # affected packages, and fix versions rather than only a red CI job. The # hashFiles guard skips this step (instead of erroring "file not found") # when an EARLIER step failed before grype could write the SARIF. - uses: github/codeql-action/upload-sarif@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) + uses: github/codeql-action/upload-sarif@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8 if: ${{ !cancelled() && hashFiles('grype-results.sarif') != '' }} with: sarif_file: 'grype-results.sarif' @@ -814,12 +837,29 @@ jobs: env: # needs.*.result is one of: success | failure | cancelled | skipped. RESULTS: ${{ join(needs.*.result, ',') }} + DOCS_ONLY: ${{ needs.changes.outputs.docs_only }} run: | set -euo pipefail echo "Upstream job results: ${RESULTS}" + echo "docs_only: ${DOCS_ONLY}" case ",${RESULTS}," in *,failure,*|*,cancelled,*) echo "::error::A required upstream CI job failed or was cancelled." exit 1 ;; esac + # A `skipped` job passes this gate only because the docs-only + # classifier above is allowed to skip the suite. If nothing was + # classified docs-only, a skip means a required job did not run and + # this gate would otherwise report green over it — the + # red-but-not-required rot pattern with the colours inverted. So trust + # the classifier for the one case it exists for, and refuse a skip in + # every other case rather than assuming the `if:` that produced it was + # correct. + if [ "${DOCS_ONLY}" != "true" ]; then + case ",${RESULTS}," in + *,skipped,*) + echo "::error::A required upstream job was SKIPPED on a change that is not docs-only. CI gate does not pass over a suite that did not run." + exit 1 ;; + esac + fi echo "All required CI jobs passed (or were cleanly skipped for a docs-only change)." diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index f95bbf06..c1417224 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -170,7 +170,7 @@ jobs: cache: true - name: Initialize CodeQL - uses: github/codeql-action/init@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) + uses: github/codeql-action/init@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8 with: languages: ${{ matrix.language }} build-mode: ${{ matrix.build-mode }} @@ -180,7 +180,7 @@ jobs: - name: Autobuild if: matrix.build-mode == 'autobuild' - uses: github/codeql-action/autobuild@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) + uses: github/codeql-action/autobuild@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8 env: # Same build tag ci.yml and dev-ci.yml pass to `go vet` and `go test`, # and for the same reason: internal/sandbox/host.go — the UNSANDBOXED @@ -194,7 +194,7 @@ jobs: GOFLAGS: -tags=fleet_host_executor - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) + uses: github/codeql-action/analyze@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8 with: category: /language:${{ matrix.language }} # Also write the SARIF to disk so the step below can summarize it. The @@ -202,6 +202,54 @@ jobs: # this only adds a local copy. output: ${{ runner.temp }}/codeql-sarif + - name: Classify findings (shared filter) + # ONE classifier, in .github/codeql-gate.jq, consumed by both the summary + # and the gate below. Two copies of a SARIF filter is two copies that can + # disagree about what "blocking" means, and the report disagreeing with + # the gate is worse than either being wrong alone. Keeping it in a file + # also means it can be exercised against fixture SARIF locally with the + # exact bytes CI runs. + if: ${{ !cancelled() }} + env: + SARIF_DIR: ${{ runner.temp }}/codeql-sarif + LANGUAGE: ${{ matrix.language }} + ACCEPTED_FILE: ${{ github.workspace }}/.github/codeql-accepted-findings.json + GATE_FILTER: ${{ github.workspace }}/.github/codeql-gate.jq + CLASSIFIED: ${{ runner.temp }}/codeql-classified.json + run: | + set -uo pipefail + shopt -s nullglob + files=("$SARIF_DIR"/*.sarif) + if [ ${#files[@]} -eq 0 ]; then + echo "::error::No SARIF written for $LANGUAGE — cannot conclude the scan was clean." + exit 1 + fi + for f in "$ACCEPTED_FILE" "$GATE_FILTER"; do + if [ ! -f "$f" ]; then + echo "::error::$f is missing — refusing to gate without it." + exit 1 + fi + done + if ! jq -rs --slurpfile reg "$ACCEPTED_FILE" -f "$GATE_FILTER" \ + "${files[@]}" > "$CLASSIFIED"; then + echo "::error::could not classify the SARIF for $LANGUAGE — refusing to report it clean." + exit 1 + fi + total=$(jq -r '.total' "$CLASSIFIED") + meta=$(jq -r '.ruleMetaCount' "$CLASSIFIED") + echo "$LANGUAGE: $total finding(s), $meta rule metadata entr(ies)" + # VACUITY CHECK. Severity banding depends on resolving each result's + # rule metadata, and CodeQL puts that in tool.extensions[].rules[] — + # not tool.driver.rules[]. A first cut of the filter read only the + # driver, resolved nothing, scored every finding at security-severity 0 + # and reported "0 blocking" over a tree holding 30 findings. The check + # below is what would have caught that: findings but no rule metadata + # means the lookup is broken and the gate is not evaluating anything. + if [ "$total" != "0" ] && [ "$meta" = "0" ]; then + echo "::error::$LANGUAGE: $total finding(s) but ZERO rule metadata resolved — severity banding is inoperative, so this gate would pass vacuously. Fix the rule lookup in .github/codeql-gate.jq." + exit 1 + fi + - name: Summarize findings to the job log # WHY THIS EXISTS: a CodeQL run reports nothing about what it FOUND to # its own log. It writes SARIF, uploads it, and exits 0 — with findings @@ -210,86 +258,50 @@ jobs: # reading CI output, to `gh run view`, and to any automation that has # the log but not the code-scanning API. # - # The listing is split into the SAME three tiers the next step gates on, - # so the log says not just what was found but which part of it blocks — - # and, importantly, prints the ACCEPTED tier by name. A waiver that is - # invisible in CI output is a waiver nobody re-reads. Reporting only; the - # `Fail on findings` step below is what fails the job. + # The listing is split into the SAME three tiers the gate uses, from the + # same classifier, and prints the ACCEPTED tier BY NAME — a waiver that + # is invisible in CI output is a waiver nobody re-reads. if: ${{ !cancelled() }} env: - SARIF_DIR: ${{ runner.temp }}/codeql-sarif LANGUAGE: ${{ matrix.language }} - ACCEPTED_FILE: ${{ github.workspace }}/.github/codeql-accepted-findings.json + CLASSIFIED: ${{ runner.temp }}/codeql-classified.json # The database directory is named after the EXTRACTOR, which is not # always the matrix language: javascript-typescript -> javascript. CODEQL_DB: ${{ matrix.language == 'javascript-typescript' && 'javascript' || matrix.language }} run: | set -uo pipefail - shopt -s nullglob - files=("$SARIF_DIR"/*.sarif) { echo "### CodeQL findings — $LANGUAGE" - if [ ${#files[@]} -eq 0 ]; then - # Not a failure: the analyze step is what fails on a broken run. - # Say it plainly rather than printing "No findings." for a scan - # that never produced a file — reporting a clean result you did - # not observe is the error this repo keeps writing down. - echo 'No SARIF file was written — nothing to summarize (see the analyze step).' - else - echo '```' - # `-s` slurps every SARIF doc into one array, so a language that - # emits more than one file is still counted once, in total. - # - # security-severity lives on the RULE, not the result, so each - # result is joined back to its rule in the same run's - # tool.driver.rules[] to recover it. Absent -> 0, which is - # correct: a query with no security-severity is not a High. - jq -rs --slurpfile reg "$ACCEPTED_FILE" ' - ( reduce ($reg[0].accepted[]? | "\(.rule) \(.file)") as $k ({}; .[$k] = true) ) as $waived - | [ .[] | .runs[]? - | ( [ .tool.driver.rules[]? ] ) as $rules - | .results[]? - | . as $r - | ( [ $rules[] | select(.id == $r.ruleId) - | .properties["security-severity"] ][0] // "" ) as $sev - | ( ($r.locations[0].physicalLocation.artifactLocation.uri) // "" ) as $file - | ( ($r.level) // "note" ) as $level - | { level: $level, - sev: (($sev | tonumber? // 0)), - rule: $r.ruleId, - loc: "\($file):\(($r.locations[0].physicalLocation.region.startLine) // "?")", - waived: ( (($waived | has("\($r.ruleId) \($file)"))) - or ((($r.suppressions // []) | length) > 0) ), - high: ( $level == "error" or $level == "warning" or (($sev | tonumber? // 0) >= 7.0) ) } - ] as $all - | ( $all | map(select(.high and (.waived | not))) ) as $b - | ( $all | map(select(.high and .waived)) ) as $w - | ( $all | map(select(.high | not)) ) as $n - | "BLOCKING — High+ and not registered (\($b|length)):", - ( if ($b|length) == 0 then " none" else ($b | sort_by(.rule,.loc) | .[] | " [\(.level)] sec-sev=\(.sev) \(.rule) \(.loc)") end ), - "", - "ACCEPTED — High+ waived in codeql-accepted-findings.json or in-source (\($w|length)):", - ( if ($w|length) == 0 then " none" else ($w | sort_by(.rule,.loc) | .[] | " [\(.level)] sec-sev=\(.sev) \(.rule) \(.loc)") end ), - "", - "ADVISORY — below High, triage in the Security tab (\($n|length)):", - ( if ($n|length) == 0 then " none" else ($n | sort_by(.rule,.loc) | .[] | " [\(.level)] sec-sev=\(.sev) \(.rule) \(.loc)") end ), - "", - "totals: \($all|length) finding(s) — \($b|length) blocking, \($w|length) accepted, \($n|length) advisory" - ' "${files[@]}" - echo '```' - # COVERAGE, not just the verdict: "No findings." alone cannot be - # told apart from "analyzed nothing", which is the exact - # green-but-vacuous outcome this workflow exists to rule out. - # The source archive is the file set the database was built from. - # RUNNER_TEMP (the env var), not ${{ runner.temp }}: interpolating a - # GitHub expression straight into a run: block is the shape - # semgrep's gha-curl-pipe-shell / curl-eval rules flag, and it also - # breaks their bash sub-parser — which silently costs coverage on - # this very file. The env var is equivalent and parses. - src="$RUNNER_TEMP/codeql_databases/$CODEQL_DB/src.zip" - if [ -f "$src" ]; then - echo "files in the $LANGUAGE database: $(unzip -Z1 "$src" 2>/dev/null | grep -vc '/$' || echo '?')" - fi + echo '```' + jq -r ' + def fmt: " [\(.level)] sec-sev=\(if .hasSev then .sev else "n/a" end) \(.rule) \(.file):\(.line)" + + (if .suppressed then " (in-source suppression)" else "" end); + def tier($label; $rows): + "\($label) (\($rows | length)):", + (if ($rows | length) == 0 then " none" + else ($rows | sort_by(.rule, .file, .line) | .[] | fmt) end); + tier("BLOCKING — High band (security-severity >= 7.0), not waived"; .blocking), + "", + tier("ACCEPTED — High band, waived in codeql-accepted-findings.json or in-source"; .accepted), + "", + tier("ADVISORY — below the High band; triage in the Security tab"; .advisory), + "", + "totals: \(.total) finding(s) — \(.blocking | length) blocking, \(.accepted | length) accepted, \(.advisory | length) advisory", + "rule metadata resolved: \(.ruleMetaCount)" + ' "$CLASSIFIED" + echo '```' + # COVERAGE, not just the verdict: "No findings." alone cannot be + # told apart from "analyzed nothing", which is the exact + # green-but-vacuous outcome this workflow exists to rule out. + # The source archive is the file set the database was built from. + # RUNNER_TEMP (the env var), not ${{ runner.temp }}: interpolating a + # GitHub expression straight into a run: block is the shape + # semgrep's gha-curl-pipe-shell / curl-eval rules flag, and it also + # breaks their bash sub-parser — which silently costs coverage on + # this very file. The env var is equivalent and parses. + src="$RUNNER_TEMP/codeql_databases/$CODEQL_DB/src.zip" + if [ -f "$src" ]; then + echo "files in the $LANGUAGE database: $(unzip -Z1 "$src" 2>/dev/null | grep -vc '/$' || echo '?')" fi } | tee -a "$GITHUB_STEP_SUMMARY" @@ -301,88 +313,45 @@ jobs: # code has a problem", which is precisely how the Go toolchain break sat # unnoticed behind a red-but-not-required check for weeks. # - # THRESHOLD, in three parts: - # 1. A finding blocks when its SARIF level is `error`/`warning`, or its - # rule carries security-severity >= 7.0 (CodeQL's High/Critical - # band). Note-level findings below High are printed and uploaded to - # the Security tab; they do not block. - # 2. A finding whose (rule, file) pair appears in - # .github/codeql-accepted-findings.json does not block. Each entry - # carries a written reason. The waiver is per-FILE, so the rule - # stays live everywhere else — a new go/request-forgery in a - # different file still fails the build. - # 3. An in-source `// codeql[rule-id]` suppression is honored (CodeQL - # emits it as a `suppressions` array on the result). + # THRESHOLD (see .github/codeql-gate.jq for the mechanics): + # - security-severity >= 7.0 blocks. That is CodeQL's own High/Critical + # cut and what GitHub's code-scanning merge protection bands on. + # `level` / @problem.severity is NOT used for a rule that publishes a + # security-severity: nearly every security query is `error`, + # go/log-injection at 6.1 included, so banding on it would block all + # 23 log-injection findings and reproduce the deadlock this replaced. + # - a rule with no security-severity falls back to level error/warning. + # - a (rule, file) pair in .github/codeql-accepted-findings.json, or an + # in-source `// codeql[rule-id]` comment, moves a High-band finding + # to ACCEPTED. The register is per-FILE, so the rule stays live + # everywhere else. # - # WHY NOT "any finding" — that was tried in #1246, and it deadlocked the - # repo. That gate was armed on a measurement of ZERO across all four - # languages, but the measurement came from a `pull_request` run (Dev CI - # run 525), and on pull_request events the CodeQL action runs - # DIFF-INFORMED: it builds the full database and evaluates every query, - # then reports only results located inside the PR's diff. Run 525's own - # log says so — "Persisted 204 diff range(s) across 43 file(s)", and - # "file coverage information is only enabled when analyzing the default - # branch and protected branches". - # - # So the first full-tree scan was the PUSH that merged it (Dev CI run - # 527), which reported 38 Go and 17 javascript-typescript findings and - # turned `Dev gate` red with no PR to fix it through. The any-finding - # threshold was never armed over a clean tree; it was armed over a - # 55-finding backlog nobody had measured yet. - # - # Those 55 were triaged individually. Four were reachable and are FIXED - # in code, not accepted — the task-create log line whose update-path twin - # was already sanitized, two pre-validation client paths on reject - # branches, a client-echoed attachment name, and a world-readable - # private-key write in the e2e harness. The rest are false positives in - # fleet's threat model and are registered with their reasons. - # - # Runs after the summary so the log leads with WHAT was found. - # See docs/CODEQL.md ("Merge gating") and - # docs/adr/0048-codeql-severity-gating.md. + # WHY NOT "any finding" — that was tried in #1246 and it deadlocked the + # repo. See docs/adr/0048-codeql-severity-gating.md: the zero it was armed + # on came from a `pull_request` run, where CodeQL is DIFF-INFORMED and + # reports only results inside the PR's diff, so it measured the diff and + # not the tree. if: ${{ !cancelled() }} env: - SARIF_DIR: ${{ runner.temp }}/codeql-sarif LANGUAGE: ${{ matrix.language }} - # GITHUB_WORKSPACE, not ${{ github.workspace }} interpolated into the - # run: block — same reason the summary step uses RUNNER_TEMP. - ACCEPTED_FILE: ${{ github.workspace }}/.github/codeql-accepted-findings.json + CLASSIFIED: ${{ runner.temp }}/codeql-classified.json run: | set -uo pipefail - shopt -s nullglob - files=("$SARIF_DIR"/*.sarif) - if [ ${#files[@]} -eq 0 ]; then - echo "::error::No SARIF written for $LANGUAGE — cannot conclude the scan was clean." - exit 1 - fi - if [ ! -f "$ACCEPTED_FILE" ]; then - echo "::error::accepted-findings register $ACCEPTED_FILE is missing — refusing to gate without it." - exit 1 - fi - blocking=$(jq -rs --slurpfile reg "$ACCEPTED_FILE" ' - ( reduce ($reg[0].accepted[]? | "\(.rule) \(.file)") as $k ({}; .[$k] = true) ) as $waived - | [ .[] | .runs[]? - | ( [ .tool.driver.rules[]? ] ) as $rules - | .results[]? - | . as $r - | ( [ $rules[] | select(.id == $r.ruleId) - | .properties["security-severity"] ][0] // "" ) as $sev - | ( ($r.locations[0].physicalLocation.artifactLocation.uri) // "" ) as $file - | ( ($r.level) // "note" ) as $level - | ( "\($r.ruleId) \($file)" ) as $key - | select( (($r.suppressions // []) | length) == 0 ) - | select( ($waived | has($key)) | not ) - | select( $level == "error" or $level == "warning" or (($sev | tonumber? // 0) >= 7.0) ) - | " [\($level)] sec-sev=\($sev) \($r.ruleId) \($file):\(($r.locations[0].physicalLocation.region.startLine) // "?")" - ] | .[]' "${files[@]}") - rc=$? - if [ $rc -ne 0 ]; then - echo "::error::could not evaluate the SARIF for $LANGUAGE — refusing to report it clean." + count=$(jq -r '.blocking | length' "$CLASSIFIED") + if [ -z "$count" ]; then + echo "::error::could not read the classification for $LANGUAGE — refusing to report it clean." exit 1 fi - if [ -n "$blocking" ]; then - echo "::error::CodeQL found blocking finding(s) for ${LANGUAGE}:" - echo "$blocking" + if [ "$count" != "0" ]; then + echo "::error::CodeQL found ${count} blocking finding(s) for ${LANGUAGE} — see the summary above." + jq -r '.blocking[] | " \(.rule) \(.file):\(.line) (security-severity \(.sev))"' "$CLASSIFIED" + echo "" + echo "Fix it. If it is a false positive the honest options are a code" + echo "change that removes the sink, an in-source // codeql[rule-id]" + echo "comment, or an entry in .github/codeql-accepted-findings.json" + echo "with a written reason. Note that dismissing the alert in the" + echo "Security tab will NOT turn this check green: this step reads the" + echo "run's own SARIF and never consults the code-scanning API." exit 1 fi echo "CodeQL ($LANGUAGE): 0 blocking findings." diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index 6ef818a9..caa9d480 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -107,7 +107,7 @@ jobs: run: go vet -tags fleet_host_executor ./... - name: golangci-lint - uses: golangci/golangci-lint-action@db9de0fc1a667e1a49d2291a1a042dff081d78f6 # v9 + uses: golangci/golangci-lint-action@ba0d7d2ec06a0ea1cb5fa41b2e4a3ab91d21278a # v9.3.0 with: # Keep pinned in lockstep with ci.yml + .golangci.yml (see the note # there) so the fast lane and the full gate never disagree. diff --git a/.github/workflows/govulncheck-scheduled.yml b/.github/workflows/govulncheck-scheduled.yml index 0e4a09e1..d29b3f06 100644 --- a/.github/workflows/govulncheck-scheduled.yml +++ b/.github/workflows/govulncheck-scheduled.yml @@ -92,7 +92,7 @@ jobs: echo '```' >> "$GITHUB_STEP_SUMMARY" - name: Upload scan results - uses: github/codeql-action/upload-sarif@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) + uses: github/codeql-action/upload-sarif@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8 # Skip cleanly (rather than error "file not found") if the scan step # failed before govulncheck could write the SARIF. if: ${{ !cancelled() && hashFiles('govulncheck.sarif') != '' }} diff --git a/.github/workflows/grype-scheduled.yml b/.github/workflows/grype-scheduled.yml index b2357c52..d48f4d6e 100644 --- a/.github/workflows/grype-scheduled.yml +++ b/.github/workflows/grype-scheduled.yml @@ -80,7 +80,7 @@ jobs: --output sarif=grype-results.sarif - name: Upload weekly scan results - uses: github/codeql-action/upload-sarif@4c0873ef8656cb3c50b3f42fb63bc1ade0cfa827 # v4 (4.37.8) + uses: github/codeql-action/upload-sarif@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8 # Skip cleanly (rather than error "file not found") if an earlier step # failed before grype could write the SARIF. if: ${{ !cancelled() && hashFiles('grype-results.sarif') != '' }} diff --git a/.github/workflows/publish-sandbox-image.yml b/.github/workflows/publish-sandbox-image.yml index 8be79186..1335e287 100644 --- a/.github/workflows/publish-sandbox-image.yml +++ b/.github/workflows/publish-sandbox-image.yml @@ -235,11 +235,38 @@ jobs: REQUESTED: ${{ inputs.fleet_ref || 'main' }} run: | set -euo pipefail + # ALLOW-LIST, not a deny-list. The previous deny-list (`refs/pull/*|pull/*|-*`) + # had two holes, both of which this closes by construction: + # + # 1. GITHUB_OUTPUT newline injection. A workflow_call string input may contain + # newlines, and `printf 'resolved=%s\n'` wrote it unsanitized, so + # `fleet_ref: "main\nresolved=refs/pull/1/head"` matched no deny pattern + # (the string starts "main"), exited 0, and emitted TWO `resolved=` lines — + # last-wins gave the attacker the ref. The same primitive could forge any + # step output. The character class below admits no newline, so the value + # cannot carry a second assignment. + # 2. A raw commit SHA. "Every ref in this repo is collaborator-written except + # refs/pull/*" is true of NAMED REFS and false of reachable COMMITS: GitHub + # keeps fork-PR commits in the base repo's object store and + # actions/checkout will happily fetch a bare SHA. So a bare hex SHA is + # refused too — pass a branch or tag name, which only a collaborator can + # create. case "$REQUESTED" in - refs/pull/*|pull/*|-*) - echo "::error::fleet_ref '$REQUESTED' is refused: pull-request refs carry fork-controlled code, and this workflow executes the checked-out build script." - exit 1 ;; + "") + echo "::error::fleet_ref is empty"; exit 1 ;; + *[!a-zA-Z0-9._/-]*) + echo "::error::fleet_ref contains a character outside [A-Za-z0-9._/-] (newline, space or shell metacharacter). Refused before checkout."; exit 1 ;; + -*|*..*|*//*) + echo "::error::fleet_ref '$REQUESTED' is not a plausible ref name"; exit 1 ;; + refs/pull/*|pull/*) + echo "::error::fleet_ref '$REQUESTED' is refused: pull-request refs carry fork-controlled code, and this workflow executes the checked-out build script."; exit 1 ;; esac + # Refuse a bare commit SHA (see 2 above). A ref name that happens to be all-hex + # and >=7 chars must be given in its refs/heads/ or refs/tags/ form. + if [ -z "$(printf '%s' "$REQUESTED" | tr -d '0-9a-fA-F')" ] && [ "${#REQUESTED}" -ge 7 ]; then + echo "::error::fleet_ref '$REQUESTED' looks like a raw commit SHA. Fork pull-request commits are reachable by SHA from this repository, so only named refs are accepted (e.g. main, refs/heads/x, v1.2.3)." + exit 1 + fi printf 'resolved=%s\n' "$REQUESTED" >> "$GITHUB_OUTPUT" - name: Checkout fleet (build script) diff --git a/scripts/check_action_pins_test.go b/scripts/check_action_pins_test.go new file mode 100644 index 00000000..498b8c5d --- /dev/null +++ b/scripts/check_action_pins_test.go @@ -0,0 +1,93 @@ +// Copyright (c) 2025 ElcanoTek +// SPDX-License-Identifier: MIT + +package scripts + +import ( + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// Every third-party `uses:` in .github/workflows must be pinned to a 40-hex +// commit SHA with a trailing `# vX.Y.Z` comment. Two separate properties, and +// the repo has been bitten by the second one: +// +// 1. A mutable tag (`@v4`) lets whoever controls the upstream repository change +// what fleet's CI executes, which is the whole reason #1246 converted 53 +// refs to SHAs. +// 2. The comment has to name an EXACT version. Two refs shipped as +// `# v4 (4.37.8)` and `# v9`, and dependabot-core parses the leading version +// token — so it read those as "4" and "9". Worse, both SHAs were the +// ANNOTATED TAG OBJECT of the mutable major tag rather than the commit it +// pointed at (verified with `git ls-remote --tags`: `refs/tags/v4` -> +// 4c0873ef, `refs/tags/v4^{}` -> db488dde). A tag object is immutable, but +// it is only reachable while that tag still points at it — the moment +// upstream moves `v4`, the object is unreferenced and Actions can no longer +// resolve the ref. That is a self-inflicted CI outage with no bad actor +// involved, and it was armed in six places. +// +// This test cannot tell a tag object from a commit offline (that needs the +// network). It enforces the shape, which is what makes the drift reviewable: +// an exact version comment is what lets a human or Dependabot check the SHA +// against a release. + +var ( + usesLine = regexp.MustCompile(`(?m)^\s*(?:-\s+)?uses:\s*(\S+)\s*(#.*)?$`) + shaRef = regexp.MustCompile(`^[0-9a-f]{40}$`) + exactVer = regexp.MustCompile(`^#\s*v\d+\.\d+\.\d+\b`) +) + +func TestWorkflowsPinActionsBySHAWithExactVersionComment(t *testing.T) { + root := repoRoot(t) + dir := filepath.Join(root, ".github", "workflows") + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatalf("read %s: %v", dir, err) + } + var checked int + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".yml") { + continue + } + raw, err := os.ReadFile(filepath.Join(dir, e.Name())) + if err != nil { + t.Fatalf("read %s: %v", e.Name(), err) + } + for _, m := range usesLine.FindAllStringSubmatch(string(raw), -1) { + ref, comment := m[1], strings.TrimSpace(m[2]) + // A local reusable-workflow call is a path, not a versioned action; + // it is this repository's own code at this repository's own commit. + if strings.HasPrefix(ref, "./") { + continue + } + at := strings.LastIndex(ref, "@") + if at < 0 { + t.Errorf("%s: `uses: %s` has no version at all", e.Name(), ref) + continue + } + checked++ + action, version := ref[:at], ref[at+1:] + if !shaRef.MatchString(version) { + t.Errorf("%s: `uses: %s` is pinned to %q, not a 40-hex commit SHA — a mutable ref lets upstream change what CI runs", + e.Name(), action, version) + continue + } + if comment == "" { + t.Errorf("%s: `uses: %s@%s` has no version comment — add `# vX.Y.Z` so the pin is reviewable and Dependabot can bump it", + e.Name(), action, version[:8]) + continue + } + if !exactVer.MatchString(comment) { + t.Errorf("%s: `uses: %s@%s` is commented %q — must name an EXACT version (`# vX.Y.Z`); a bare major reads as that major to Dependabot and hides which release the SHA is", + e.Name(), action, version[:8], comment) + } + } + } + if checked == 0 { + t.Fatal("no third-party `uses:` refs found — this test would pass vacuously") + } + t.Logf("checked %d third-party action references", checked) +} diff --git a/scripts/check_gate_needs_test.go b/scripts/check_gate_needs_test.go new file mode 100644 index 00000000..ec8cd525 --- /dev/null +++ b/scripts/check_gate_needs_test.go @@ -0,0 +1,96 @@ +// Copyright (c) 2025 ElcanoTek +// SPDX-License-Identifier: MIT + +package scripts + +import ( + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" +) + +// ci.yml and dev-ci.yml each end in an aggregate gate job (`CI gate` / +// `Dev gate`) whose `needs` is the list branch protection is pointed at. A job +// that runs but is NOT in that list is red-but-not-required — it can fail +// forever behind a green gate, which is exactly how the CodeQL Go extraction +// break sat unnoticed for weeks and the reason #1246 was written. +// +// Nothing asserted the list was complete. Adding a job and forgetting to extend +// `needs` is a silent, one-line regression with no failing test, so this is that +// test: every job in the file except the gate itself must be in the gate's +// needs. +// +// Deliberately a hand-rolled scan rather than a YAML dependency: the `scripts` +// package has no non-test Go files and no imports beyond the standard library, +// and adding a YAML parser to the module for one assertion is a worse trade than +// a regexp over a file whose shape this repo controls. + +var ( + // A top-level job key: exactly two spaces of indent, then `name:`. + jobKeyRe = regexp.MustCompile(`(?m)^ ([a-zA-Z0-9_-]+):$`) + needsRe = regexp.MustCompile(`(?m)^ needs:\s*\[([^\]]*)\]`) +) + +func TestAggregateGateNeedsEveryJob(t *testing.T) { + root := repoRoot(t) + for _, tc := range []struct{ file, gate string }{ + {"ci.yml", "ci-gate"}, + {"dev-ci.yml", "dev-gate"}, + } { + raw, err := os.ReadFile(filepath.Join(root, ".github", "workflows", tc.file)) + if err != nil { + t.Fatalf("read %s: %v", tc.file, err) + } + text := string(raw) + + // Only look below `jobs:` so a top-level key like `permissions:` is not + // mistaken for a job. + jobsAt := strings.Index(text, "\njobs:\n") + if jobsAt < 0 { + t.Fatalf("%s: no top-level `jobs:` block", tc.file) + } + jobsBlock := text[jobsAt:] + + var jobs []string + for _, m := range jobKeyRe.FindAllStringSubmatch(jobsBlock, -1) { + jobs = append(jobs, m[1]) + } + if len(jobs) < 2 { + t.Fatalf("%s: found %d jobs — the scan is broken, not the workflow", tc.file, len(jobs)) + } + + gateAt := strings.Index(jobsBlock, "\n "+tc.gate+":\n") + if gateAt < 0 { + t.Fatalf("%s: no `%s` job — if the gate was renamed, update this test AND the branch ruleset", tc.file, tc.gate) + } + m := needsRe.FindStringSubmatch(jobsBlock[gateAt:]) + if m == nil { + t.Fatalf("%s: `%s` has no inline `needs: [...]` — this test only understands the inline form", tc.file, tc.gate) + } + needs := map[string]bool{} + for _, n := range strings.Split(m[1], ",") { + if n = strings.TrimSpace(n); n != "" { + needs[n] = true + } + } + + var missing []string + for _, j := range jobs { + if j == tc.gate { + continue + } + if !needs[j] { + missing = append(missing, j) + } + } + sort.Strings(missing) + if len(missing) > 0 { + t.Errorf("%s: job(s) %v run but are not in `%s`'s needs — they are red-but-not-required. Add them, or the gate reports green while they fail.", + tc.file, missing, tc.gate) + } + t.Logf("%s: %s covers %d/%d jobs", tc.file, tc.gate, len(needs), len(jobs)-1) + } +} From 9340efd4e063e297e5ce0e81b837934fcdd5a45a Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 16:16:15 +0000 Subject: [PATCH 19/34] Remove dead code, close the CI permission gaps, settle two stale claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Dead code is audit surface: an auditor asks "why is this here, is it reachable, is it maintained?" and every unreachable identifier costs a round of that. Each deletion below was verified with `git grep -w ` across all tracked files — every hit was the declaration and its own doc comment, zero call sites, zero test references, and no coupling to docs/openapi.yaml or schemaModelRegistry (so TestOpenAPISchemaDrift cannot regress). - internal/mcpoauth/errors.go: IsInvalidClient. The odd one out of a four-predicate set — IsInvalidGrant, IsInvalidTarget and IsInvalidScope all have callers. Its job is already done inline: IsTerminalRefreshError and ReauthDetail both switch on the literal "invalid_client" rather than calling it. - internal/sched/apikeys/apikeys.go: Manager.LogAction. A nine-parameter exported wrapper over the private m.logAudit that nothing calls. Worth naming precisely because it is API-key AUDIT logging: a reviewer grepping for audit surface lands here first and has to work out that callers use logAudit directly. Deleting removes the ambiguity. - internal/sched/models/models.go: TaskAssignment and LogSubmission — the two halves of the retired v1 remote-worker protocol (OrchestratorURL, Files/FileChecksums, the worker log POST), superseded when the platform consolidated into one process. The live log path uses models.LogSession directly. - internal/sched/models/models.go: MaxLogSubmissionSize. This one is more than clutter — it declared a 24 MB body cap that NOTHING enforced. The cap actually applied is MaxJSONBodySize = 1 MB (internal/sched/handlers/middleware.go, wired via BodySizeLimitMiddleware). So the real posture was 24x stricter than the constant claimed, and an auditor reading models.go would have concluded fleet accepts 24 MB bodies. An unenforced-limit claim is exactly the kind of thing that becomes a finding. - internal/config/config.go: DefaultFromEmail. Its doc comment called it "the fallback From address for outgoing mail", but no code path consumes it — an operator who sets neither SENDGRID_FROM_EMAIL nor MAILBUX_FROM_EMAIL does not get this fallback. A documented capability that does not exist is a violation of this repo's own honesty-in-docs invariant, so the constant goes rather than the claim being left standing. - internal/tools/task_tracker.go: the only commented-out code block in the tree, plus the inProgressCount it was the sole reader of (the counter was incremented and never read once the comment is gone). Replaced with a sentence saying why there is deliberately no "more than one in_progress" check. `golangci-lint` with `default: standard` already includes `unused` and is a full gate, so unexported dead code is structurally zero — which is why everything above is an EXPORTED identifier in internal/, the class `unused` deliberately does not report. Confirmed independently with `deadcode -test -tags fleet_host_executor ./...`, which now reports nothing. CI permissions and the alarm: - scan-cron-alarm.yml only fired on `conclusion == 'failure'`. That ignores startup_failure — which is the exact failure this file's own header describes as the incident that motivated it (an in-job alarm variant failed a whole Dev CI run that way, so NO scanning ran on that head) — and timed_out, which matters given codeql.yml caps at 30 minutes and semgrep.yml at 15. Now alarms on any conclusion that is not success or skipped. Also added the daily real-model canary to the watched list: it had no alarm at all, and a silently red daily canary is the rot pattern this file exists to prevent. Noted that the watcher matches on workflow DISPLAY NAME, so renaming `name:` disarms it. - ci.yml carried `pull-requests: read` at WORKFLOW level for golangci-lint-action's only-new-issues, which is explicitly `false`. So the scope had no consumer while still reaching every job that does not override it — including web, playwright and e2e-live, which npm-install and execute thousands of third-party packages. Removed. - screenshots.yml held `contents: write` at workflow level for a single job that runs `npm ci` and `playwright install`. It bought nothing: the push it existed for cannot succeed, because the main ruleset carries a pull_request rule with current_user_can_bypass: never and no bypass actors. So it was a repo-writable token handed to third-party code on every run, in exchange for a guaranteed-failing push whose commit message also carried [skip ci]. Dropped to read, with the shape a real implementation would take written down. - auto-merge-dependabot.yml: the header asserted "CI is the approval signal and it is never bypassed", because `gh pr merge --auto` holds the merge until every REQUIRED check passes. That is only true where something is required. The dev ruleset requires no status checks at all, and dependabot.yml points every version update at dev — so there was nothing holding the merge. Excluded `github_actions` from auto-merge (that ecosystem's "dependency" IS the CI definition, and cooldown is not even available for it — Dependabot supports cooldown for gomod and npm only), added a `branches: [main, dev]` filter so this can never silently apply to an unprotected branch, and moved the write scopes from workflow level onto the one job that needs them. Getting `Dev gate` into the dev ruleset is the real fix and is a repo-settings action; it is flagged for the owner. - codeql.yml / dev-ci.yml: moved the two remaining `${{ }}` expressions out of `run:` blocks and into `env:`. The values come from {success, failure, cancelled, skipped} so nothing attacker-controlled reached the shell, but this is the shape the two sites fixed in #1246 were fixed away FROM, and it breaks semgrep's bash sub-parser, which silently costs coverage on the very files it appears in. Two stale claims settled rather than left for an auditor to find: - docs/adr/0012: `cmd/fleet-admin` was to be "a deprecation shim for ONE release ... removed next release". That clock never started — `git tag` returns nothing, VERSION is 0.0.0, and CHANGELOG.md has only an [Unreleased] heading, so "next release" is not a date. The shim also turns out to be load-bearing rather than vestigial: the Makefile, bootstrap.sh, update.sh and fleet-upgrade.sh all build or install it, the last two hard-fail without it, and scripts_dryrun_test.go asserts the "would install fleet + fleet-admin" string. Amended with a concrete trigger — removed in the first release after 1.0.0 — and the note that it forks no logic (it shares internal/admincli.Run). docs/EVENT-TRIGGERS.md and docs/openapi.yaml were still teaching `fleet-admin sched trigger …` as the primary command for HMAC-secret rotation; those are security procedures, so they now say `fleet`. - migration 022 carried the tree's only TODO(security) — and every auditor greps for that string. Two problems beyond the deferred work: it sat in an APPLIED migration, so it was parked where nobody can close it in place, and it pointed at a source file in an unrelated external codebase (a dangling cross-repo pointer in a security note). Rewritten to state the fact plainly (the column holds account NAMES, never credential values, which are brokered host-side per ADR-0003 / ADR-0042), to say that whether account names are themselves in scope is an open threat-model question for the owner, and to point at SECURITY.md as where that gets answered. golang-migrate tracks by version with no checksum, so editing the comment cannot re-run or invalidate the applied DDL. The tree now has zero TODO/FIXME/XXX/HACK markers. - .golangci.yml's noctx exclusion said "the one production noctx (cmd/fleet-admin bootstrap) is fixed in code via exec.CommandContext". cmd/fleet-admin has held no exec call since the CLI was unified in #461; the real call sites are in cmd/fleet and internal/admincli. Repointed, because a lint suppression whose stated reason names dead code is a suppression nobody can re-verify. - scripts/generate-icons.py declared web/src/app/favicon.ico among its outputs. That file has never been committed and is not gitignored either, so it existed only on whoever last ran the script — while every other declared output IS committed. The App Router serves icon.svg and apple-icon.png, with favicon-16/32.png under public/, so the .ico had no consumer. Dropped, and the docstring now records that the outputs are committed and when to regenerate. Signed-off-by: Brad Flaugher --- .github/workflows/auto-merge-dependabot.yml | 57 ++++++++++++++----- .github/workflows/ci.yml | 8 ++- .github/workflows/codeql.yml | 9 ++- .github/workflows/dev-ci.yml | 9 ++- .github/workflows/scan-cron-alarm.yml | 26 +++++++-- .github/workflows/screenshots.yml | 27 ++++++++- .golangci.yml | 7 ++- docs/EVENT-TRIGGERS.md | 8 +-- docs/adr/0012-unified-fleet-cli.md | 30 ++++++++-- docs/openapi.yaml | 2 +- internal/config/config.go | 4 -- internal/mcpoauth/errors.go | 12 ---- internal/sched/apikeys/apikeys.go | 5 -- .../022_add_task_credential_allowlist.up.sql | 19 +++++-- internal/sched/models/models.go | 25 -------- internal/tools/task_tracker.go | 16 ++---- scripts/generate-icons.py | 22 ++++--- 17 files changed, 178 insertions(+), 108 deletions(-) diff --git a/.github/workflows/auto-merge-dependabot.yml b/.github/workflows/auto-merge-dependabot.yml index 810b2e00..00b60e27 100644 --- a/.github/workflows/auto-merge-dependabot.yml +++ b/.github/workflows/auto-merge-dependabot.yml @@ -2,11 +2,31 @@ # but every one — even a routine patch — currently waits on a human to merge. # This workflow lets PATCH-level bumps merge themselves once the full CI gate # (build / vet / lint / test / -race / govulncheck, web lint+test+build, -# Playwright mocked + live, and the gitleaks secret scan) is green. CI is the -# correct approval signal for a patch; it is never bypassed — `gh pr merge -# --auto` only enables auto-merge, so GitHub still holds the merge until every -# required check passes. Minor and major bumps are intentionally left for a -# human, where an API change or a transitive surprise is more likely. +# Playwright mocked + live, and the gitleaks secret scan) is green. Minor and +# major bumps are intentionally left for a human, where an API change or a +# transitive surprise is more likely. +# +# TWO LIMITS THAT ARE LOAD-BEARING, both learned the hard way: +# +# 1. "CI is the approval signal, and it is never bypassed" IS ONLY TRUE WHERE +# THE GATE IS A REQUIRED CHECK. `gh pr merge --auto` asks GitHub to hold the +# merge until every REQUIRED check passes — so on a branch whose ruleset +# requires nothing, there is nothing to hold it and the PR merges as soon as +# it is mergeable. The `dev` ruleset currently requires no status checks at +# all (only `deletion` and `non_fast_forward`), and .github/dependabot.yml +# points every version update at `dev`. So the `branches:` filter below is +# not cosmetic: it keeps this workflow from applying to a branch where its +# central assumption does not hold. Getting `Dev gate` into the dev ruleset +# is the real fix and is a repo-settings action; see docs/SCANNING.md +# ("Known gaps"). +# +# 2. A `github-actions` bump IS A REWRITE OF .github/workflows/*. It changes +# what CI executes, on a surface where the cooldown that protects gomod and +# npm is not even available (Dependabot supports `cooldown` for those two +# ecosystems only), so a freshly published action version can be proposed +# the same day. That combination — self-modifying CI, no cooldown, no +# required check on the target branch — is not something to auto-merge, so +# that ecosystem is excluded below and takes a human. # # Requires "Allow auto-merge" to be enabled on the repository (Settings → # General → Pull Requests). This is the pattern documented in GitHub's @@ -15,18 +35,26 @@ name: Auto-merge Dependabot patch PRs on: pull_request: + # See limit 1 in the header: this workflow's safety rests on the target + # branch having required checks. Naming the branches explicitly means it can + # never silently start applying to one nobody protected. + branches: [main, dev] -# Dependabot-triggered runs get a read-only GITHUB_TOKEN by default; these -# elevated permissions are honored only for the dependabot[bot] actor, and the -# job guard below makes sure nothing else can reach the merge step. -permissions: - contents: write - pull-requests: write +# What actually confines these scopes is the `if: github.actor == +# 'dependabot[bot]'` guard on the job below — a `permissions:` block is honored +# for whatever run reaches it, regardless of actor. github.actor is not +# spoofable, so the guard holds; the scopes are declared on the JOB rather than +# the workflow so a second job added here later does not inherit write access it +# never asked for. +permissions: {} jobs: auto-merge: if: ${{ github.actor == 'dependabot[bot]' }} runs-on: ubuntu-latest + permissions: + contents: write + pull-requests: write steps: - name: Fetch Dependabot metadata id: meta @@ -35,8 +63,11 @@ jobs: github-token: ${{ secrets.GITHUB_TOKEN }} - name: Enable auto-merge for patch updates - # Only patch bumps auto-merge; minor and major get human review. - if: ${{ steps.meta.outputs.update-type == 'version-update:semver-patch' }} + # Only patch bumps auto-merge; minor and major get human review. And + # never github-actions, whatever the bump level — see limit 2 in the + # header: that ecosystem's "dependency" is the CI definition itself. + if: ${{ steps.meta.outputs.update-type == 'version-update:semver-patch' + && steps.meta.outputs.package-ecosystem != 'github_actions' }} run: gh pr merge --auto --squash "$PR_URL" env: PR_URL: ${{ github.event.pull_request.html_url }} diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 241bd38e..612edae3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -7,9 +7,13 @@ on: branches: [main] permissions: + # Least privilege at the workflow level, so nothing is granted to a job that + # did not ask. `pull-requests: read` used to sit here for + # golangci-lint-action's only-new-issues — which is now explicitly `false` + # (see the note at that step), so the scope had no consumer while still + # reaching every job that does not override it, including web / playwright / + # e2e-live, which npm-install and run thousands of third-party packages. contents: read - # Required by golangci-lint-action's only-new-issues option (reads the PR diff). - pull-requests: read jobs: changes: diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index c1417224..60c69232 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -371,8 +371,15 @@ jobs: runs-on: ubuntu-latest steps: - name: Fail if any CodeQL analysis did not succeed + # RESULTS via env, not interpolated into the run: block. The values are + # drawn from {success, failure, cancelled, skipped} so nothing + # attacker-controlled reaches the shell — but this is the exact shape the + # two sites fixed in #1246 were fixed away FROM, and it breaks semgrep's + # bash sub-parser, which silently costs coverage on this very file. + env: + RESULTS: ${{ join(needs.*.result, ' ') }} run: | - results='${{ join(needs.*.result, ' ') }}' + results="$RESULTS" echo "job results: $results" for r in $results; do [ "$r" = "success" ] || { echo "a CodeQL analysis did not succeed"; exit 1; } diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index caa9d480..027cdcbf 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -313,8 +313,15 @@ jobs: runs-on: ubuntu-latest steps: - name: Fail if any fast-lane job failed + # RESULTS via env, not interpolated into the run: block. The values are + # drawn from {success, failure, cancelled, skipped} so nothing + # attacker-controlled reaches the shell — but this is the exact shape the + # two sites fixed in #1246 were fixed away FROM, and it breaks semgrep's + # bash sub-parser, which silently costs coverage on this very file. + env: + RESULTS: ${{ join(needs.*.result, ' ') }} run: | - results='${{ join(needs.*.result, ' ') }}' + results="$RESULTS" echo "job results: $results" for r in $results; do [ "$r" = "success" ] || { echo "a fast-lane job did not succeed"; exit 1; } diff --git a/.github/workflows/scan-cron-alarm.yml b/.github/workflows/scan-cron-alarm.yml index ac12b94a..07b62a4e 100644 --- a/.github/workflows/scan-cron-alarm.yml +++ b/.github/workflows/scan-cron-alarm.yml @@ -25,7 +25,10 @@ name: Scan cron alarm on: workflow_run: - workflows: [CodeQL, Semgrep] + # Matched by workflow DISPLAY NAME (the `name:` in each file), not by path — + # renaming `name:` in codeql.yml, semgrep.yml or e2e-canary.yml silently + # disarms this alarm with no error anywhere. Keep those three in sync. + workflows: [CodeQL, Semgrep, "E2E canary (real model)"] types: [completed] permissions: @@ -36,14 +39,29 @@ jobs: name: File an issue so a red cron cannot rot silently # Scheduled failures only: a red workflow_call run already reddens the # calling gate on a PR, and a red manual dispatch has a human watching it. + # + # NOT `conclusion == 'failure'`. workflow_run.conclusion is also + # startup_failure, timed_out, cancelled or action_required — and + # startup_failure is the one this file exists for: the in-job alarm variant + # this replaced failed an entire Dev CI run that way (see the header), which + # meant NO scanning ran on that head at all. A weekly cron that dies on + # malformed YAML or a permissions change is exactly the invisible breakage + # the alarm is for, and `== 'failure'` ignored it. timed_out matters too: + # codeql.yml caps at 30 minutes and semgrep.yml at 15. + # + # `skipped` and `success` are the only conclusions that are not an alarm. if: >- - github.event.workflow_run.conclusion == 'failure' && - github.event.workflow_run.event == 'schedule' + github.event.workflow_run.event == 'schedule' && + github.event.workflow_run.conclusion != 'success' && + github.event.workflow_run.conclusion != 'skipped' runs-on: ubuntu-latest steps: - name: File or update the alarm issue # Body mirrors the in-job alarm steps in govulncheck-scheduled.yml and - # grype-scheduled.yml — keep the three in sync. + # grype-scheduled.yml — keep the three in sync. Those two still carry + # their own in-job step because they can request `issues: write` + # directly; CodeQL and Semgrep cannot, because a called workflow may not + # request a permission its caller did not grant (see the header). env: GH_TOKEN: ${{ github.token }} WORKFLOW_NAME: ${{ github.event.workflow_run.name }} diff --git a/.github/workflows/screenshots.yml b/.github/workflows/screenshots.yml index 7eef6e50..729d4184 100644 --- a/.github/workflows/screenshots.yml +++ b/.github/workflows/screenshots.yml @@ -28,8 +28,21 @@ concurrency: group: screenshots-${{ github.ref }} cancel-in-progress: true +# Read-only, deliberately. This workflow has ONE job, and that job npm-installs +# and runs Playwright — thousands of third-party packages — so `contents: write` +# here was a repo-writable token handed to third-party code on every run. +# +# It bought nothing: the push it existed for cannot succeed. The `main` ruleset +# carries a pull_request rule with current_user_can_bypass: never and no bypass +# actors, so a direct push to main is refused regardless of token scope. The push +# step is already written to warn rather than fail, so dropping the scope changes +# the warning's wording and nothing else. +# +# If refreshing screenshots from CI is ever wanted for real, the shape is a +# second job that needs: [screenshots], holds `contents: write` alone, runs no +# third-party code, and opens a PR — not a direct push. permissions: - contents: write + contents: read jobs: screenshots: @@ -97,6 +110,14 @@ jobs: git config user.email "41898282+github-actions[bot]@users.noreply.github.com" git add docs/screenshots git commit -m "docs(screenshots): refresh GUI + TUI screenshots [skip ci]" - # main may require PRs; a rejected push warns rather than failing the run. + # The `main` ruleset carries a pull_request rule with + # current_user_can_bypass: never and no bypass actors, so a direct push + # to main CANNOT succeed — this is a permanent no-op, not an + # occasional one, and the [skip ci] in the commit message above means + # anything it did land would bypass CI on main. The workflow token is + # also read-only now (see the permissions note at the top), so this + # fails on scope first. Kept as a warning rather than an error so the + # capture half stays useful; do not "fix" it by widening either the + # ruleset or the token — refresh via a PR instead. git push origin HEAD:main \ - || echo "::warning::could not push refreshed screenshots (is main protected against direct pushes? allow the github-actions bot, or refresh via PR). The committed baseline still serves." + || echo "::warning::could not push refreshed screenshots — the main ruleset requires a pull request, so this push is expected to fail. Refresh via PR; the committed baseline still serves." diff --git a/.golangci.yml b/.golangci.yml index 31f4ebf2..13337e81 100644 --- a/.golangci.yml +++ b/.golangci.yml @@ -111,8 +111,11 @@ linters: # noctx: httptest.NewRequest (vs NewRequestWithContext) is idiomatic # in tests — these are synthetic requests with no cancellation # semantics, so threading a context through every one is pure churn. - # The one production noctx (cmd/fleet-admin bootstrap) is fixed in - # code via exec.CommandContext. + # Production call sites use exec.CommandContext rather than + # exec.Command — cmd/fleet/{mcp_broker_runtime,health,validate_config}.go + # and internal/admincli/backup.go. (This note used to name + # cmd/fleet-admin, which has held no exec call since the CLI was + # unified in #461.) - noctx # Exclude shadow checking for err variables (common pattern) - text: 'shadow: declaration of "err"' diff --git a/docs/EVENT-TRIGGERS.md b/docs/EVENT-TRIGGERS.md index e2472d1f..90e9a597 100644 --- a/docs/EVENT-TRIGGERS.md +++ b/docs/EVENT-TRIGGERS.md @@ -140,7 +140,7 @@ is a no-op when SMTP isn't configured. # The template task must exist first (create it with trigger_type=webhook so the # cron engine never runs it; set allow_event_triggers=true to let event runs use # its connectors). -fleet-admin sched trigger create \ +fleet sched trigger create \ --task \ --slug weekly-deploy \ --kind email \ @@ -150,9 +150,9 @@ fleet-admin sched trigger create \ --max-attachments 3 --max-attachment-bytes 1048576 \ [--template prompt.tmpl] # optional Go text/template over {{.From}} {{.Subject}} {{.Text}} {{.HTML}} {{.To}} -fleet-admin sched trigger list # shows id, kind, slug, task -fleet-admin sched trigger rotate # rotate the HMAC secret -fleet-admin sched trigger delete +fleet sched trigger list # shows id, kind, slug, task +fleet sched trigger rotate # rotate the HMAC secret +fleet sched trigger delete ``` The rendered prompt is what the spawned run receives. With no `--template`, a diff --git a/docs/adr/0012-unified-fleet-cli.md b/docs/adr/0012-unified-fleet-cli.md index be6ba577..a1669387 100644 --- a/docs/adr/0012-unified-fleet-cli.md +++ b/docs/adr/0012-unified-fleet-cli.md @@ -30,9 +30,26 @@ There is **one `fleet` binary** (`cmd/fleet`) with subcommand dispatch - Every other verb (`update`, `status`, `bootstrap`, `chat`, `sched`, `task`, `mcp`, `notes`, `worktree`, `backup`, `restore`, `motd`, …) routes to `internal/admincli.Run`. -- `cmd/fleet-admin` is reduced to a **deprecation shim** for ONE release: it - prints a one-line notice and forwards to the same `admincli.Run`, so existing - scripts and the in-place upgrade path keep working. It is removed next release. +- `cmd/fleet-admin` is reduced to a **deprecation shim**: it prints a one-line + notice and forwards to the same `admincli.Run`, so existing scripts and the + in-place upgrade path keep working. + + **Amended 2026-08-22 (enterprise security audit).** This originally said "for + ONE release ... removed next release". That clock never started: `git tag` + returns nothing, `VERSION` is `0.0.0`, and `CHANGELOG.md` has only an + `[Unreleased]` heading — there has never been a release, so "next release" is + not a date and "one release" is not a window. The shim also turns out to be + load-bearing rather than vestigial: `Makefile` (`bins`, `install`), + `scripts/bootstrap.sh`, `scripts/update.sh` and `scripts/fleet-upgrade.sh` all + build or install it, the last two *hard-fail* if the binary is missing, and + `internal/admincli/scripts_dryrun_test.go` asserts the "would install fleet + + fleet-admin" string. So removal is a coordinated change across four scripts and + two test assertions, not a deletion. + + The concrete trigger, replacing the unanchored one: **the shim is removed in + the first release after 1.0.0.** Until then it stays, and it is 20 lines that + fork no logic — it shares `internal/admincli.Run` with `fleet`, so it adds no + second governance path. `make install` puts `fleet` (and the shim) on `PATH` — the actual fix for "isn't installed" on a dev box. The systemd unit is **not** force-migrated to @@ -51,11 +68,12 @@ installed" on a dev box. The systemd unit is **not** force-migrated to ## Consequences - Operators get the unified `fleet` they asked for; muscle memory (`fleet-admin - `) still works for one release with a deprecation warning. + `) still works, with a deprecation warning, until the removal trigger + above. - The daemon artifact stays named `fleet`, so the highest-blast-radius references (systemd unit + bootstrap on a *running* box) barely move. -- Two binaries still build for one release (the shim), so the existing - build/upgrade scripts that expect both `fleet` and `fleet-admin` are unchanged. +- Two binaries still build (the shim), so the existing build/upgrade scripts + that expect both `fleet` and `fleet-admin` are unchanged. - A future release deletes the shim and may flip bare `fleet` to print help (requiring explicit `serve`); by then every deployed unit says `fleet serve`. diff --git a/docs/openapi.yaml b/docs/openapi.yaml index 24fd1665..612c46ba 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -2716,7 +2716,7 @@ paths: equalized HMAC work) to prevent slug enumeration. A leaked secret lets a caller inject the spawned run's prompt under the template's seat (network/MCP/files); the sandbox remains the execution boundary. Rotate - with `fleet-admin sched trigger rotate`. + with `fleet sched trigger rotate`. security: [] requestBody: content: diff --git a/internal/config/config.go b/internal/config/config.go index 4c8f91b1..9669ddba 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -57,10 +57,6 @@ const ( // Mirrors the frontend's DEFAULT_MODEL (the recommended everyday pick). const DefaultTitleModel = "google/gemini-3.7-flash" -// DefaultFromEmail is the fallback From address for outgoing mail. Neutral by -// default; a deployment overrides via SENDGRID_FROM_EMAIL / MAILBUX_FROM_EMAIL. -const DefaultFromEmail = "noreply@example.com" - // Sub-agent caps (#175, tightened for delegation #264): deliberately SMALL // defaults. Depth bounds recursion; fan-out bounds how many children one parent // may spawn; the budget fraction bounds each child's slice of the parent's diff --git a/internal/mcpoauth/errors.go b/internal/mcpoauth/errors.go index 1fbfb960..cbe8c1a1 100644 --- a/internal/mcpoauth/errors.go +++ b/internal/mcpoauth/errors.go @@ -41,18 +41,6 @@ func IsInvalidTarget(err error) bool { return errors.As(err, &oe) && oe.Code == "invalid_target" } -// IsInvalidClient reports whether err is an OAuthError with code invalid_client -// — the authorization server no longer recognizes our client credentials. For a -// DCR-registered client this usually means the registration was pruned or -// expired server-side; for a BYO client it means the id/secret is wrong or was -// rotated. Either way the stored registration is unusable, so this is terminal -// for refresh (see IsTerminalRefreshError): the connection is marked needs-reauth -// and reconnecting re-runs registration through the normal connect flow. -func IsInvalidClient(err error) bool { - var oe *OAuthError - return errors.As(err, &oe) && oe.Code == "invalid_client" -} - // IsInvalidScope reports whether err is an OAuthError with code invalid_scope — // the authorization server rejected the requested scope. On refresh this is // recoverable: RFC 6749 §6 makes `scope` OPTIONAL and defines its omission as diff --git a/internal/sched/apikeys/apikeys.go b/internal/sched/apikeys/apikeys.go index 61bdcbc8..2bf627a0 100644 --- a/internal/sched/apikeys/apikeys.go +++ b/internal/sched/apikeys/apikeys.go @@ -782,11 +782,6 @@ func (m *Manager) SetMaxPriority(keyID string, ceiling *int) error { return m.save() } -// LogAction logs an action performed with an API key. -func (m *Manager) LogAction(keyID, action, resourceType string, resourceID *string, details map[string]interface{}, ipAddress, userAgent *string, success bool, errorMessage *string) { - m.logAudit(AuditLogEntry{KeyID: keyID, Action: action, ResourceType: resourceType, ResourceID: resourceID, Details: details, IPAddress: ipAddress, UserAgent: userAgent, Success: success, ErrorMessage: errorMessage}) -} - // GetKey gets a key by ID. func (m *Manager) GetKey(keyID string) *APIKey { m.mu.RLock() diff --git a/internal/sched/db/migrations/022_add_task_credential_allowlist.up.sql b/internal/sched/db/migrations/022_add_task_credential_allowlist.up.sql index 29f1617e..159cbb2e 100644 --- a/internal/sched/db/migrations/022_add_task_credential_allowlist.up.sql +++ b/internal/sched/db/migrations/022_add_task_credential_allowlist.up.sql @@ -4,9 +4,18 @@ -- unchanged. A non-null (possibly empty) array enforces least-privilege: an MCP -- call to a pair not on the list is denied before it executes. -- --- TODO(security): credential_allowlist stores (server, account) pair NAMES only, --- not credential values. The values themselves never enter the database (they --- live in the process env file; see internal/creds). If account names are --- themselves sensitive, encrypt this column with the AES-256-GCM pattern used --- for project secrets (cf. Suna's apps/api/src/projects/secrets.ts). +-- SECURITY NOTE, and an open question that is deliberately NOT settled here: +-- credential_allowlist stores (server, account) pair NAMES only, never +-- credential values. The values do not enter the database at all — they live in +-- the process env file and are brokered host-side (internal/creds, ADR-0003, +-- ADR-0042). So this column is not a secret store. +-- +-- What is unsettled is whether ACCOUNT NAMES are themselves in scope. If a +-- deployment treats them as sensitive, this column wants encryption at rest. +-- That is a threat-model decision for the repo owner, not something a migration +-- can decide, and it is recorded in SECURITY.md rather than here: a migration is +-- applied history, so a question parked in one is a question nobody can close in +-- place. (An earlier version of this note was an untracked security to-do that +-- pointed at a source file in an unrelated external codebase; the pointer was +-- dangling and nothing tracked the item.) ALTER TABLE tasks ADD COLUMN IF NOT EXISTS credential_allowlist JSONB; diff --git a/internal/sched/models/models.go b/internal/sched/models/models.go index 86ab35ed..0bd6be1c 100644 --- a/internal/sched/models/models.go +++ b/internal/sched/models/models.go @@ -1968,22 +1968,6 @@ type TaskArtifact struct { Size int64 `json:"size"` // bytes at publish time } -// TaskAssignment is the task assignment carried to the worker. -type TaskAssignment struct { - TaskID uuid.UUID `json:"task_id"` - Prompt string `json:"prompt"` - Model *string `json:"model,omitempty"` - FallbackModel *string `json:"fallback_model,omitempty"` - MaxIterations *int `json:"max_iterations,omitempty"` - MCPSelection MCPSelection `json:"mcp_selection,omitempty"` - CredentialAllowlist CredentialAllowlist `json:"credential_allowlist"` - InstructionSelfImprove bool `json:"instruction_self_improve,omitempty"` - OrchestratorURL string `json:"orchestrator_url"` - Files []string `json:"files,omitempty"` - FileNames []string `json:"file_names,omitempty"` - FileChecksums []string `json:"file_checksums,omitempty"` -} - // DashboardStats contains statistics for the dashboard. type DashboardStats struct { PendingTasks int `json:"pending_tasks"` @@ -2083,15 +2067,6 @@ func (ls LogSession) MarshalJSON() ([]byte, error) { }) } -// LogSubmission is a log submission for a task. -type LogSubmission struct { - TaskID uuid.UUID `json:"task_id"` - Session LogSession `json:"session"` -} - -// MaxLogSubmissionSize is the maximum size of a log submission in bytes (24MB). -const MaxLogSubmissionSize = 24 * 1024 * 1024 - // APIKeyCreate is the request model for creating an API key. type APIKeyCreate struct { Name string `json:"name"` diff --git a/internal/tools/task_tracker.go b/internal/tools/task_tracker.go index 66a2ef3d..b39e79cf 100644 --- a/internal/tools/task_tracker.go +++ b/internal/tools/task_tracker.go @@ -185,7 +185,6 @@ func (t *taskTracker) validateTasks(tasks []Task) error { // Check for duplicate IDs seenIDs := make(map[string]bool) - inProgressCount := 0 for i, task := range tasks { if task.ID == "" { @@ -207,18 +206,13 @@ func (t *taskTracker) validateTasks(tasks []Task) error { return fmt.Errorf("duplicate task ID: %s", task.ID) } seenIDs[task.ID] = true - - // Count in_progress tasks - if task.Status == StatusInProgress { - inProgressCount++ - } } - // Warn if multiple tasks are in_progress (but don't error) - // if inProgressCount > 1 { - // // This is just a warning in the description, not enforced - // } - + // Deliberately no "more than one in_progress" check: the tool description + // asks for one in-progress task at a time, but it is guidance, not a + // validation rule, and rejecting the call would strand a model mid-plan. + // A counter and a commented-out `if` used to sit here saying so; the comment + // is the whole content, so it is a comment. return nil } diff --git a/scripts/generate-icons.py b/scripts/generate-icons.py index 57de4d06..258960d7 100755 --- a/scripts/generate-icons.py +++ b/scripts/generate-icons.py @@ -5,8 +5,20 @@ Requires: pip install cairosvg pillow Master: web/public/logos/fleet-mark.svg (single flattened path, 512 viewBox) -Outputs: web/src/app/ favicon.ico, icon.svg, apple-icon.png +Outputs: web/src/app/ icon.svg, apple-icon.png web/public/app-icons/ favicon-16/32, icon-192/512, maskable-icon-512 + +Every output above IS committed, so this script only needs running when the +master mark changes — regenerate, then commit the diff alongside it. Nothing +calls it from CI or the Makefile on purpose: the icons are assets, not build +products, and a generator in the build path would make every CI run depend on +cairosvg. + +It used to also write web/src/app/favicon.ico, which was never committed and is +not gitignored either — an output that existed only on whoever last ran this. The +App Router serves icon.svg (crisp at any DPI) and apple-icon.png, with +favicon-16/32.png under public/ for anything that wants a raster, so the .ico +had no consumer. Dropped rather than left as drift. """ import io @@ -48,14 +60,6 @@ def render(px: int, scale: float = 1.0, bg: str | None = None) -> Image.Image: # --- classic favicons -------------------------------------------------------- render(16).save(PUB / "favicon-16.png") render(32).save(PUB / "favicon-32.png") -ico = [render(s) for s in (48, 32, 16)] -ico[0].save( - APP / "favicon.ico", - format="ICO", - append_images=ico[1:], - sizes=[(48, 48), (32, 32), (16, 16)], -) - # --- apple touch icon (iOS Add to Home Screen) ------------------------------- # Must be opaque: iOS fills transparency with black. iOS applies its own # corner mask, so this is a full-bleed square with the glyph inset. From e2ff6a66ae1929ca4f2fee544150832002ad1d9f Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 16:19:59 +0000 Subject: [PATCH 20/34] Attribute the fixtures that the own-rows edit gate now requires MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three tests broke on taskWritableByPrincipal, and all three for the same reason: their fixture tasks were inserted with neither CreatedBy nor CreatedByKeyID, so an unattributed row was being edited by a client-role key. That is precisely the shape the new gate refuses. None of them is about ownership: - TestUpdateTaskRunIfPersistsAndNormalizes and TestUpdateTaskKeepsGatedTaskOnSchedulerPath are about the run_if privilege boundary and the dispatch-state recompute. - TestTypedKeyRouteScope is about the #190 middleware type-scope gate. So the fix is the fixture, not the gate: each task is now attributed to the key that acts on it, which is also the realistic shape — a task a client key created really does carry that key's CreatedByKeyID. Left unattributed, these tests would have been asserting run_if and type-scope behavior through a path that the ownership check short-circuits first, which is a worse test than either intent. mustCreateTypedKeyWithID joins mustCreateRoleKeyWithID as the helper that returns the KeyID alongside the secret, with a note saying why a fixture wants it. Full Go suite green against a real Postgres 16 (`make test`, -p 1). Signed-off-by: Brad Flaugher --- internal/sched/handlers/run_if_authz_test.go | 20 ++++++++++++---- .../sched/handlers/typed_key_scope_test.go | 24 +++++++++++++++---- 2 files changed, 35 insertions(+), 9 deletions(-) diff --git a/internal/sched/handlers/run_if_authz_test.go b/internal/sched/handlers/run_if_authz_test.go index 01ebcf95..28222aa8 100644 --- a/internal/sched/handlers/run_if_authz_test.go +++ b/internal/sched/handlers/run_if_authz_test.go @@ -154,19 +154,25 @@ func TestCreateTaskRunIfRequiresAdminPermission(t *testing.T) { func TestUpdateTaskRunIfPersistsAndNormalizes(t *testing.T) { r, keyMgr, store := setupRunIfAuthz(t) - clientKey := mustCreateRoleKey(t, keyMgr, "client") + clientKeyID, clientKey := mustCreateRoleKeyWithID(t, keyMgr, "client") // The gated tasks are seeded SCHEDULED: that is where a gate normally lives // (models.RunIf's enforcement contract parks every gated task on the // scheduler path), and gate changes on a pending task are refused outright // — the pending refusal has its own subtests below. + // + // They are also seeded ATTRIBUTED to clientKey. Editing a task is own-rows + // (taskWritableByPrincipal), so an unattributed fixture would be refused by + // the ownership check before reaching the run_if logic this test is about. addGatedWithStatus := func(t *testing.T, status models.TaskStatus) *models.Task { t.Helper() future := time.Now().UTC().Add(time.Hour) + keyID := clientKeyID task := &models.Task{ ID: uuid.New(), Prompt: "a gated task prompt that is long enough", Status: status, CreatedAt: time.Now().UTC(), ScheduledFor: &future, - RunIf: &models.RunIf{Command: "test -f /tmp/ready", ExitCodeIs: 2, TimeoutSeconds: 30}, + CreatedByKeyID: &keyID, + RunIf: &models.RunIf{Command: "test -f /tmp/ready", ExitCodeIs: 2, TimeoutSeconds: 30}, } if _, err := store.AddTask(task); err != nil { t.Fatalf("add task: %v", err) @@ -353,7 +359,10 @@ func TestUpdateTaskRunIfPersistsAndNormalizes(t *testing.T) { func TestUpdateTaskKeepsGatedTaskOnSchedulerPath(t *testing.T) { r, keyMgr, store := setupRunIfAuthz(t) - clientKey := mustCreateRoleKey(t, keyMgr, "client") + // Attributed fixtures throughout: editing is own-rows + // (taskWritableByPrincipal), and this test is about the dispatch-state + // recompute, not about ownership. + clientKeyID, clientKey := mustCreateRoleKeyWithID(t, keyMgr, "client") put := func(taskID uuid.UUID, tc models.TaskCreate) *httptest.ResponseRecorder { body, _ := json.Marshal(tc) req := httptest.NewRequest("PUT", "/tasks/"+taskID.String(), bytes.NewReader(body)) @@ -373,10 +382,11 @@ func TestUpdateTaskKeepsGatedTaskOnSchedulerPath(t *testing.T) { // Seeded the way NewTask parks an immediate gated create: scheduled, // with the parked timestamp in the past by the time the edit lands. past := time.Now().UTC().Add(-time.Minute) + keyID := clientKeyID task := &models.Task{ ID: uuid.New(), Prompt: "a gated task prompt that is long enough", Status: models.TaskStatusScheduled, CreatedAt: time.Now().UTC(), - ScheduledFor: &past, RunIf: gate, + ScheduledFor: &past, RunIf: gate, CreatedByKeyID: &keyID, } if _, err := store.AddTask(task); err != nil { t.Fatalf("add task: %v", err) @@ -411,6 +421,8 @@ func TestUpdateTaskKeepsGatedTaskOnSchedulerPath(t *testing.T) { TriggerType: models.TriggerTypeWebhook, RunIf: gate, }) + keyID := clientKeyID + template.CreatedByKeyID = &keyID if _, err := store.AddTask(template); err != nil { t.Fatalf("add template: %v", err) } diff --git a/internal/sched/handlers/typed_key_scope_test.go b/internal/sched/handlers/typed_key_scope_test.go index 9791cee6..7875b98e 100644 --- a/internal/sched/handlers/typed_key_scope_test.go +++ b/internal/sched/handlers/typed_key_scope_test.go @@ -19,11 +19,21 @@ import ( func mustCreateTypedKey(t *testing.T, keyMgr *apikeys.Manager, kt apikeys.KeyType, slugs []string) string { t.Helper() - _, raw, err := keyMgr.CreateTypedKey("test-"+string(kt), kt, slugs, 0, nil, "") + _, raw := mustCreateTypedKeyWithID(t, keyMgr, kt, slugs) + return raw +} + +// mustCreateTypedKeyWithID also returns the KeyID, so a test can attribute a +// fixture task to the key that will act on it (task.CreatedByKeyID). Mutating a +// task is own-rows (taskWritableByPrincipal), so an UNATTRIBUTED fixture now +// exercises the ownership check rather than whatever the test meant to assert. +func mustCreateTypedKeyWithID(t *testing.T, keyMgr *apikeys.Manager, kt apikeys.KeyType, slugs []string) (string, string) { + t.Helper() + key, raw, err := keyMgr.CreateTypedKey("test-"+string(kt), kt, slugs, 0, nil, "") if err != nil { t.Fatalf("create typed key: %v", err) } - return raw + return key.KeyID, raw } // TestTypedKeyRouteScope verifies the #190 middleware type-scope gate on the @@ -33,11 +43,15 @@ func TestTypedKeyRouteScope(t *testing.T) { store, keyMgr, r, cleanup := setupAuthzHandler(t) defer cleanup() - taskA := addTask(t, store, "task A") - readonlyKey := mustCreateTypedKey(t, keyMgr, apikeys.KeyTypeReadonly, nil) webhookKey := mustCreateTypedKey(t, keyMgr, apikeys.KeyTypeWebhook, []string{"pr-review"}) - taskKey := mustCreateTypedKey(t, keyMgr, apikeys.KeyTypeTask, nil) + taskKeyID, taskKey := mustCreateTypedKeyWithID(t, keyMgr, apikeys.KeyTypeTask, nil) + + // Attributed to the task key: this test is about the #190 type-scope gate, + // not about ownership, so the fixture is the realistic shape (a task the + // acting key created) rather than an unattributed row that would additionally + // trip the own-rows edit check. + taskA := addTaskCreatedByKey(t, store, "task A", taskKeyID) t.Run("readonly key may GET tasks", func(t *testing.T) { req := httptest.NewRequest("GET", "/tasks", nil) From 054e9f0f3a4f4823be01e388f4250e9f9bd99c37 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 16:21:29 +0000 Subject: [PATCH 21/34] Trim the unlinked planning scratchpad to its one live plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/implementation-plans-enhancements.md had zero inbound links from any markdown, Go file, workflow or script, and three of its four plans had shipped. The shipped copies are the problem, not the file: a plan sitting next to the authoritative record is the worse of the two, and this one was phrased in the present tense. The #167 entry is why this matters for an audit rather than being tidy-up. It carried a section headed "OAuth control-plane tokens parent-readable" whose resolution — accepted for v1, threat model written down (parent compromise implies stored remote-MCP tokens; agent runs stay child-side per ADR-0040) — was already recorded in SECURITY.md and docs/MCP-BROKER-SCOPES.md. A second copy, phrased as an open decision, in an unlinked file, reads like an unresolved security gap to anyone who greps their way into it. It is not one. What remains is #984 (the Fleet <-> Buzz bridge), which genuinely has not shipped, under a header that says so — and that says explicitly that its unchecked acceptance boxes (including "Bot token not logged; secrets in env only") are criteria the feature must meet BEFORE it ships, on code that does not exist yet, not a list of open findings. That distinction is invisible in a bare checklist and is exactly what an auditor would otherwise have to ask about. Signed-off-by: Brad Flaugher --- docs/implementation-plans-enhancements.md | 132 ++++------------------ 1 file changed, 23 insertions(+), 109 deletions(-) diff --git a/docs/implementation-plans-enhancements.md b/docs/implementation-plans-enhancements.md index 28b21345..da73857d 100644 --- a/docs/implementation-plans-enhancements.md +++ b/docs/implementation-plans-enhancements.md @@ -1,89 +1,29 @@ -# Implementation plans for open enhancements +# Implementation plan: #984 — Fleet ↔ Buzz bridge -Working notes for implementers. Prefer the linked issue comment/body when present. +**Status: NOT SHIPPED.** This is a forward-looking design note for one open +enhancement. Read it as a proposal, not as a description of fleet's behavior, and +in particular do not read the acceptance checklist below as a list of open +security findings — the unchecked boxes are criteria this feature must MEET +BEFORE it ships, on code that does not exist yet. -| Issue | Plan location | +Everything else that used to live in this file has been removed because it had +shipped, and a shipped plan sitting next to the authoritative record is the worse +of the two copies: + +| was | now recorded in | | --- | --- | -| #989 | [comment](https://github.com/ElcanoTek/fleet/issues/989#issuecomment-5198861451) | -| #988 | [issue body](https://github.com/ElcanoTek/fleet/issues/988) | -| #987 | [comment](https://github.com/ElcanoTek/fleet/issues/987#issuecomment-5198925257) — **shipped**; see `internal/clientconfig/builtin_skills/browserbase/`, `internal/tools/browserbase_live_view.go`, `docs/BROWSERBASE.md` | -| #986 | [issue body](https://github.com/ElcanoTek/fleet/issues/986) | -| #985 | Full plan below — **shipped**; see `internal/clientconfig/builtin_skills/bento-slides/`, `docs/SKILLS.md`, `docs/FEATURE-NOTES.md` | -| #984 | Full plan below (pending issue comment) | -| #167 | Full plan below — **all three residuals resolved**; see `docs/MCP-BROKER-SCOPES.md`, ADR-0042, `SECURITY.md` | - ---- - -## #985 — Bento built-in skill (good first issue, size S) - -[Bento](https://github.com/nyblnet/bento) decks are a **single HTML file** (viewer + editor + slides). Agent edits HTML in workspace → downloadable deck **without Gamma or any external API**. - -### Approach - -1. **Built-in skill** in `internal/clientconfig/builtin_skills/`: - - `bento-slides/SKILL.md` — when to use; copy template; structure slides; what not to break. - - `bento-slides/templates/starter.bento.html` — minimal legal template. -2. **Agent workflow:** copy template → `workspace/decks/.bento.html` → edit via file tools → user downloads and opens in browser. -3. **License / attribution:** confirm redistribution allowed; attribute in skill + NOTICE. -4. **Validation:** `ValidateSkills` frontmatter; optional eval "Create a 5-slide deck about X". -5. **Docs:** one line in `docs/SKILLS.md`. No new HTTP APIs. - -### Non-goals - -PPTX export; hosted collab editing; PowerPoint animation parity. - -### Acceptance — met - -- [x] Skill shows in Settings → Skills as Built-in — no code change needed; - `httpapi.skillSource` derives `builtin` from absence in the bundle dir. Asserted - in `web/e2e/live/skills-connections.spec.ts`. -- [x] `/bento-slides` loads instructions — `matchSkillInvocation` resolves any - roster name, so this came for free. -- [x] Agent produces openable `.bento.html` — via the bundled - `scripts/bento_doc.py`; round-trip, escaping and shell byte-identity are - covered by `internal/clientconfig/builtin_skills_bento_test.go`. -- [x] License/attribution settled — Bento is MIT (© 2026 The Bento authors). - Recorded in `templates/NOTICE.md` (pack-local; **no** root - `THIRD_PARTY_NOTICES.md` was added), and the shell carries upstream's own - `NOTICE` comment internally so it travels with every deck. -- [x] Works offline except model provider — the app is vendored and embedded, so - nothing is fetched at turn time, nothing is fetched to render a deck, and a - deck `new` creates makes **no** network request when opened — no update check - and no live collaboration. Multiplayer is off by construction: a CSP - `connect-src 'none'` meta the browser enforces, upstream's own offline switch, - and `set` refusing to write a `collab` block (which is not inert — carrying one - joins a live session on load). The vendored template stays byte-identical, and a - deck the user brought is reported by `validate` rather than rewritten. See - `templates/NOTICE.md` for the layer-by-layer rationale and the Chromium - verification matrix. - -### Deviations from the approach above - -1. **`templates/starter.bento.html` → `templates/Bento_Slides.bento.html`, the - full upstream v1.0.18 release artifact vendored unmodified (689KB, sha256 - pinned).** There is no "minimal legal template": a Bento deck's shell *is* the - application, so anything smaller would not open. -2. **The agent does not edit the HTML with file tools.** It uses a bundled - stdlib-only `scripts/bento_doc.py` (`new`/`get`/`set`/`validate`). The document - block sits at byte 6322 of a minified bundle, so `view_file` would spend ~125KB - of context reaching it; and the block's `<`-escaping rule fails silently rather - than loudly. The helper also makes `collab` private-key redaction and `docId` - preservation mechanical instead of instructions the model must remember. -3. **`ValidateSkills` does not cover this pack** — it reads - `Bundle.BundleSkillsDir`, i.e. the bundle's own skills, not the embedded pack. - The real gate is `TestBuiltinSkillsPackWellFormed` plus the new bento tests. -4. **No eval case.** Evals do not run in CI, need a live model plus podman, and - `evals.Case` has no skill field — the Go tests are the honest gate instead. - -### Scope discovered while shipping - -Bundle skills are **interactive-chat-only**: `internal/scheduledrun` emits no -bundle-skill roster, so scheduled tasks and `fleet task run` cannot discover this -(or any) bundle skill, even though the merged dir is bind-mounted for them. -`docs/SKILLS.md` previously implied taskrun picked the pack up unchanged; that -claim is now corrected there. - ---- +| #987 Browserbase skill | `internal/clientconfig/builtin_skills/browserbase/`, `internal/tools/browserbase_live_view.go`, `docs/BROWSERBASE.md` | +| #985 Bento built-in skill | `internal/clientconfig/builtin_skills/bento-slides/`, `docs/SKILLS.md`, `docs/FEATURE-NOTES.md` | +| #167 three residual decisions | `docs/MCP-BROKER-SCOPES.md`, [ADR-0042](adr/0042-child-side-mcp-scope-authorization.md), [ADR-0040](adr/0040-child-owned-remote-mcp-runtime.md), `SECURITY.md` | + +The #167 entry mattered most: it carried a section headed "OAuth control-plane +tokens parent-readable" whose resolution ("accepted for v1, threat model +documented — parent compromise implies stored remote-MCP tokens; agent runs stay +child-side") was already written down in `SECURITY.md` and +`docs/MCP-BROKER-SCOPES.md`. A second copy phrased as an open decision, in an +unlinked file, read like an unresolved gap. It is not one. + +For the current plan-of-record on anything else, prefer the GitHub issue. ## #984 — Fleet ↔ Buzz bridge @@ -127,29 +67,3 @@ Fleet hosting Buzz relay; full tool UI parity; every Buzz user → fleet user ma - [ ] Bot token not logged; secrets in env only Size: **M** if ACP external command is clean; **L** if deep Buzz harness embed needed. - ---- - -## #167 — Three residual decisions - -Delivered broker work (can't-read) is solid. Explicit decisions: - -### 1. Child-side authorization → **Implement** - -Parent-only gating is insufficient (Gate-2 proof). On `OpenScope`, pass policy snapshot; child enforces allowlists on every CallTool/discovery; restrict unscoped shared client for agent paths; tests for refused disallowed tools. Update `docs/MCP-BROKER-SCOPES.md`. - -### 2. Approval execution seat → **Persist staged scope** - -Preserve `{server, account}` at staging; reopen scope on approve; fail closed if account revoked; show account in UI. Unblocks #988. Tests: stage with B, approve later, assert B used. - -### 3. OAuth control-plane tokens parent-readable → **Accept v1 + document** - -Accept connect/callback/CRUD parent-side for v1; document threat model (parent compromise ⇒ remote MCP tokens). Agent runs stay child-side (ADR-0040). Optional v2: full control-plane behind child as separate issue. - -### Closing criteria — resolved - -| Residual | Resolution | -| --- | --- | -| 1 Child auth | **Implemented.** `cmd/fleet/mcp_broker_authz.go`; bundle-derived Gate-2 floor, `ScopeSpec.Policy` narrowing, child-side Gate-3, filtered scope catalogs, restricted unscoped client. ADR-0042; tests in `cmd/fleet/mcp_broker_authz_test.go`. | -| 2 Approval seat | **Implemented.** Migration 048 (`approvals.mcp_server` / `mcp_account`), `BindTurnMCPScope` at staging, `OpenApprovalMCPScope` at execution, fail-closed on a revoked seat, account badge on the card. Tests in `internal/httpapi/approvals_seat_test.go`, `internal/store/approval_seat_test.go`, `web/.../ApprovalCards.seat.test.tsx`. | -| 3 OAuth parent-readable | **Accepted + documented** (2026-08-14). `SECURITY.md` and `docs/MCP-BROKER-SCOPES.md` state the threat model: parent compromise ⇒ stored remote-MCP tokens. Agent runs stay child-side (ADR-0040). Full control-plane isolation would be a separate change. | From 68da69137fa98619f9413c059eb1e4a637905b8b Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 16:29:06 +0000 Subject: [PATCH 22/34] Give the tool-output redactor the connector secrets it never had MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tool-output scrubber's literal set was empty of connector credentials in every production build, so its own doc comment ("OPENROUTER_API_KEY, connector credentials, …") described coverage it did not have. The mechanism is an ordering accident, not a missing feature. agentcore's redactor is built lazily behind a sync.Once that fires on the first tool output, and it seeds itself from os.Environ(). But the MCP broker's boot path calls scrubParentConnectorState, which os.Unsetenv's every connector environment key, and that runs during broker startup — before the interactive engine is even constructed, let alone before any turn produces output. So by the time the snapshot is taken those values are gone, and RegisterEnvLiterals registered only what survives the scrub. That divestment is the entire point of the broker boundary and is not touched here. What it left uncovered is the return path: a connector that echoes its OWN credential back in a tool result or error string was scrubbed only if the value happened to match one of internal/redact's shape patterns (sk-*, ghp_*, AKIA*, Authorization:, marker=value). A novel bare token would have reached the model context, the SSE stream and the session log. internal/tools/browserbase_live_view.go is written as though the literal set were richer than it was — it survives only because it independently checks strings.Contains(liveView, apiKey) and refuses. Two wirings close it: - scrubParentConnectorState now hands each value to agentcore.RegisterSecretLiteral immediately BEFORE unsetting it, so the scrubber learns it at the last moment it is knowable. - cmd/fleet/main.go wires the PARENT's remotemcp service to the same hook, mirroring what the credential-owning child already does via mcpbroker.RegisterSecretLiteral (#1124). The parent unseals per-user api_key secrets and mints/refreshes OAuth bearers for its control plane — browserbase_live_view among them — and those are acquired at runtime, so no boot-time env snapshot could ever know them. That SetSecretObserver call was simply absent. RegisterSecretLiteral buffers values offered before the redactor exists and drains them at construction, both under the same mutex that guards the sharedRedactor nil-check — so a value offered concurrently with construction is either buffered and drained or added directly, never dropped between the two. Publication happens under that lock for the same reason; the unlocked read in toolRedactor is safe on sync.Once's happens-before. Verified with `go test -race` on internal/agentcore and internal/redact. Neither change widens what the parent can READ. A registered value is held solely as a redaction target and is never emitted, logged or persisted — and this is a backstop for what comes back across the broker boundary, not a substitute for the boundary. The test uses a token that matches none of the shape patterns, so it passes only if literal registration genuinely reached the redactor, and registers it before anything forces construction — the real boot order. Mutation-tested: neutering RegisterSecretLiteral makes it fail with the secret visible in the output. It also covers post-construction registration (the runtime-acquired case) and asserts that an empty registration does not turn the scrubber into a match-everything. Signed-off-by: Brad Flaugher --- cmd/fleet/main.go | 8 +++ cmd/fleet/mcp_broker_runtime.go | 14 ++++ internal/agentcore/redact.go | 67 ++++++++++++++++++- internal/agentcore/redact_integration_test.go | 44 ++++++++++++ 4 files changed, 131 insertions(+), 2 deletions(-) diff --git a/cmd/fleet/main.go b/cmd/fleet/main.go index 015f7ae3..94751cf3 100644 --- a/cmd/fleet/main.go +++ b/cmd/fleet/main.go @@ -2575,6 +2575,14 @@ func setupRemoteMCP(cfg *config.Config, chatStore *store.Store) *remotemcp.Servi PublicBaseURL: cfg.PublicBaseURL, AllowInsecureHTTP: cfg.RemoteMCPAllowInsecureHTTP, }) + // Same wiring the credential-owning child does (see mcp_broker.go, #1124), + // for this process's own redactor. The parent's control-plane service + // unseals per-user api_key secrets and mints/refreshes OAuth bearers — + // e.g. for browserbase_live_view — and those are acquired at RUNTIME, so + // the boot-time env snapshot cannot know them. Without this the parent's + // literal set never learned them, and a connector echoing its own bare + // token back was scrubbed only if it happened to match a shape pattern. + svc.SetSecretObserver(agentcore.RegisterSecretLiteral) // Abandoned OAuth flow rows are reclaimed by the maintenance loop (see // runMaintenancePass), not by a daemon of their own. This used to be a // `for range ticker.C` goroutine with a context.Background() per sweep — diff --git a/cmd/fleet/mcp_broker_runtime.go b/cmd/fleet/mcp_broker_runtime.go index 736248e5..6e13dc11 100644 --- a/cmd/fleet/mcp_broker_runtime.go +++ b/cmd/fleet/mcp_broker_runtime.go @@ -465,6 +465,20 @@ func scrubParentConnectorState(bundle *clientconfig.Bundle, cfg *config.Config, keys := bundle.ConnectorEnvironmentKeys(os.Environ()) var errs []error for _, key := range keys { + // Hand the VALUE to the tool-output scrubber before it becomes + // unreachable. agentcore's redactor snapshots os.Environ() lazily, on the + // first tool output, which is long after this loop — so without this the + // literal set held no connector secret at all, and a connector echoing + // its own credential back in a tool result was caught only if the value + // happened to match one of internal/redact's shape patterns. A novel bare + // token would have reached the model context, the SSE stream and the + // session log. + // + // This does not weaken the divestment below or re-expose anything: the + // value is kept solely as a redaction target and is never emitted. It is + // the backstop for what comes BACK across the broker boundary, which the + // boundary itself cannot police. + agentcore.RegisterSecretLiteral(os.Getenv(key)) if err := os.Unsetenv(key); err != nil { errs = append(errs, fmt.Errorf("unset connector environment %s: %w", key, err)) } diff --git a/internal/agentcore/redact.go b/internal/agentcore/redact.go index d7aa7439..ec713bb3 100644 --- a/internal/agentcore/redact.go +++ b/internal/agentcore/redact.go @@ -10,18 +10,81 @@ import ( // toolRedactor returns the process-wide secret scrubber applied to tool output // (in the tool wrappers + stream sink) and to the persisted session log. Built // once: the canonical pattern set plus literal redaction of secret-named env -// values (OPENROUTER_API_KEY, connector credentials, …) so a novel key format -// is still scrubbed by value. See internal/redact. +// values so a novel key format is still scrubbed by value. See internal/redact. +// +// NOTE ON WHAT THE ENV SNAPSHOT DOES AND DOES NOT COVER. This is lazy — the +// Once fires on the first tool output — and by then the parent has already +// divested its connector credentials: the MCP broker's boot path +// os.Unsetenv's every connector environment key (scrubParentConnectorState) +// long before any turn runs. So os.Environ() here no longer contains connector +// values, and this call alone registers only what survives the scrub, e.g. +// OPENROUTER_API_KEY. +// +// That divestment is the point of the broker boundary and is not being undone. +// But it also meant the literal set was EMPTY of connector secrets, so a +// connector echoing its own credential back in a tool result was caught only if +// the value happened to match one of internal/redact's shape patterns (sk-*, +// ghp_*, AKIA*, …) — a novel bare token would have reached the model context, +// the SSE stream and the session log. RegisterSecretLiteral below is how the +// boot path hands those values over BEFORE unsetting them, so defense-in-depth +// against an upstream echoing a credential does not depend on its format. func toolRedactor() *redact.Redactor { redactorOnce.Do(func() { r := redact.NewRedactor(nil) r.RegisterEnvLiterals(os.Environ()) + // Publish under pendingMu, and drain under the same lock: that is what + // makes RegisterSecretLiteral's nil-check safe against a racing + // construction, so a value offered concurrently is either buffered here + // and drained, or added directly — never dropped between the two. + pendingMu.Lock() + for _, v := range pendingLiterals { + r.AddLiteral(v) + } + pendingLiterals = nil sharedRedactor = r + pendingMu.Unlock() }) + // Safe unlocked: sync.Once establishes happens-before for every caller that + // returns from Do, so the write above is visible here. return sharedRedactor } +// RegisterSecretLiteral adds one secret VALUE to the process-wide tool-output +// scrubber, so it is redacted by exact match regardless of format. +// +// Call this with a value that is about to become unreachable — the boot path +// uses it for each connector credential immediately before os.Unsetenv removes +// it from the environment. Safe before or after the redactor is built: earlier +// calls are buffered and drained at construction, later ones go straight in +// (redact.Redactor.AddLiteral is mutex-guarded). Values shorter than the +// redactor's floor are ignored by AddLiteral, so a short or empty setting cannot +// turn the scrubber into a match-everything. +// +// This never widens what the parent can READ — the value is stored only as a +// scrub target and is never emitted. It is not a substitute for the broker +// boundary; it is the backstop for output that comes back from the other side of +// it. +func RegisterSecretLiteral(value string) { + if value == "" { + return + } + pendingMu.Lock() + if sharedRedactor == nil { + pendingLiterals = append(pendingLiterals, value) + pendingMu.Unlock() + return + } + pendingMu.Unlock() + sharedRedactor.AddLiteral(value) +} + var ( redactorOnce sync.Once sharedRedactor *redact.Redactor + + // pendingMu guards literals registered before the redactor is built. It + // also guards the sharedRedactor nil-check in RegisterSecretLiteral so a + // value cannot be dropped by racing construction. + pendingMu sync.Mutex + pendingLiterals []string ) diff --git a/internal/agentcore/redact_integration_test.go b/internal/agentcore/redact_integration_test.go index bc51299b..82abda17 100644 --- a/internal/agentcore/redact_integration_test.go +++ b/internal/agentcore/redact_integration_test.go @@ -37,3 +37,47 @@ func TestPolicyGuardedTool_RedactsToolOutput(t *testing.T) { t.Errorf("redaction ate surrounding output: %q", resp.Content) } } + +// TestRegisterSecretLiteralScrubsNovelFormat is the regression test for the gap +// that made the parent's literal set empty of connector secrets. +// +// The tool-output redactor snapshots os.Environ() lazily, on first use. The MCP +// broker's boot path unsets every connector environment key long before that, so +// the snapshot never saw those values, and a connector echoing its own +// credential back was scrubbed only if the value happened to match one of +// internal/redact's shape patterns. The token below deliberately matches NONE of +// them — no sk-/ghp_/AKIA prefix, no "key=value" shape — so it is scrubbed only +// if literal registration actually reached the redactor. +// +// Ordering matters and is the thing under test: RegisterSecretLiteral is called +// BEFORE anything forces the redactor into existence, which is the real boot +// order (scrubParentConnectorState runs during broker startup, the first tool +// output comes much later). +func TestRegisterSecretLiteralScrubsNovelFormat(t *testing.T) { + const novel = "Zq7Z2pLmVnT4rWxK9dCbYeHgJ1sAuF6o" + + RegisterSecretLiteral(novel) + + got := RedactSecrets("upstream said: token " + novel + " was rejected") + if strings.Contains(got, novel) { + t.Fatalf("novel-format secret survived redaction: %q", got) + } + if !strings.Contains(got, "upstream said") { + t.Fatalf("redaction ate the surrounding text: %q", got) + } + + // Registering after construction must work too — runtime-acquired + // credentials (OAuth bearers, unsealed api_keys) arrive mid-process via + // Service.SetSecretObserver, i.e. long after the redactor exists. + const later = "Rr8Y3qMnWoU5sXyL0eDcZfIhK2tBvG7p" + RegisterSecretLiteral(later) + if out := RedactSecrets("bearer " + later); strings.Contains(out, later) { + t.Fatalf("post-construction secret survived redaction: %q", out) + } + + // An empty registration must not turn the scrubber into a match-everything. + RegisterSecretLiteral("") + if out := RedactSecrets("ordinary text"); out != "ordinary text" { + t.Fatalf("empty literal corrupted redaction: %q", out) + } +} From 5003949d11944edf4c1737b5e6a9b770a272a70f Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 16:41:27 +0000 Subject: [PATCH 23/34] Correct every false and stale claim the audit found in the docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An auditor reads the docs and then checks the code, so a doc that overstates is worse than one that says less. This repo has an explicit "Honesty in docs" invariant; these are the places it had drifted. The load-bearing corrections: - AGENTS.md said "Everything is at zero findings today; keeping it there is the point." False, and it was the one line in the agent-facing operating guide an auditor would hold up. The thresholds now differ by scanner, and the difference is stated: Semgrep, ruff and npm audit gate on any finding; CodeQL gates on the High band minus the reviewed register, with everything below it advisory. Also records the two facts an agent must not get wrong — a pull_request CodeQL run certifies a DIFF and never a tree, and `Dev gate` is not a required check on `dev`. - docs/CODEQL.md carried a whole Triggers section describing a workflow that no longer exists (it showed `push: [main]` and `pull_request: [main, dev]`; the file has only workflow_call, workflow_dispatch and schedule), plus the assertion that a push to dev "would re-analyze identical content". That reasoning is exactly the trap that broke dev — the push run is full-tree and the PR run is diff-informed, so they are not identical content — and it is now preserved as the error rather than the rule. Also removed "a CodeQL job with a hundred open alerts still exits 0 and reports green" (false since the Fail-on-findings step) and the sample log format that never matched what the workflow prints. - docs/SCANNING.md claimed the extended suite "reports zero findings on this tree (verified in CI across all four languages on Dev CI run 525)" and that a green check means "clean tree". Run 525 was a pull_request event. Corrected, with the run-527 numbers, and the known-gaps section rewritten — the dev-ruleset gap is now the FIRST gap, since it is the one that makes several other statements in the file conditional. - SECURITY.md had no SAST section at all, though CodeQL and Semgrep are the controls an enterprise auditor asks about by name. It now has one. Its Grype paragraph claimed the gate covers "the image's RPM or Python packages" at a fixable CRITICAL; the policy script selects `.artifact.type == "rpm"` and fires on CRITICAL *and* HIGH, so Python dist-info records are reported and deliberately do not gate — both halves were wrong in the direction that overstates coverage. Its supply-chain section omitted the npm CVE gate entirely. And "CI runs gitleaks on every push" is not true of a feature branch, which runs nothing. - docs/TESTING.md's lane table said the fast lane SKIPS CodeQL. It runs both scanners. The table also omitted four lanes that now block. - CONTRIBUTING.md contradicted itself inside one sentence: "fails the build on a fixable CRITICAL or HIGH CVE ... (HIGH and below are reported, not blocking)". - CHANGELOG.md's [Unreleased] section carried five overlapping entries from #1246 that contradicted each other — one said the scanners gate through ci-gate, another said "CodeQL and Semgrep stay advisory"; one said `ruff format` is reported but not gated, another said it gates; one said Semgrep ships only p/github-actions, another all four packs; one said the code-quality suite was restored when it was dropped. A reader could not tell which was current. Collapsed into one entry describing the shipped end state, and extended with this PR's work. - ruff.toml's "<- what we gate on" marker pointed at the default-only rule line while `select` includes B, SIM and S. - Makefile's .PHONY omitted lint-python. - README.md's documentation table did not list SCANNING.md or CODEQL.md — the two most audit-relevant docs, unreachable from the README — and its layout tree was abridged without saying so. ADR housekeeping, both mine: - ADR-0036 presents its host-side exception list as exhaustive, and an auditor reads it that way, so it has to be. It still named `fastio_upload` as a host-read exception; there is no such native tool any more (Fast.io is an MCP server behind the broker), so the ADR was claiming a hole that does not exist. Two real classes were missing: host `git worktree` management on the scheduled-run path, and the admin-gated host `podman` build for the rampart install. Neither is model-authored and neither weakens the invariant — but "is this exception in the ADR?" should have a reliable answer, and now does. - ADR-0048's two counts were off and are now measured, not recalled: 625 `_test.go` files (not 621) and 81 `//nolint:gosec // G706` sites (not 77 — four of the increase are this branch's own). Verification: make build, make lint (golangci-lint v2.13.1 + ruff + migration DDL lint) and make test all clean; web oxlint, tsc, vitest (1104 tests) and next build all clean. Every markdown link target in every file touched here was checked to resolve on disk. Two lint findings from my own earlier commits fixed here rather than left standing: four `//nolint:gosec // G706` directives I had added were UNUSED — once the value goes through logSafe, gosec stops flagging the line, so nolintlint was right and an unused suppression is worse than none — and a prealloc nit in the new gate-needs test. Signed-off-by: Brad Flaugher --- AGENTS.md | 49 +- CHANGELOG.md | 504 ++++++++++++------ CONTRIBUTING.md | 45 +- Makefile | 6 +- README.md | 14 +- SECURITY.md | 194 ++++++- docs/BENTO-PDF-EXPORT.md | 8 +- docs/CODEQL.md | 444 ++++++++++----- docs/SCANNING.md | 269 ++++++++-- docs/TESTING.md | 111 +++- ...boxed-file-tools-and-host-io-exceptions.md | 47 +- docs/adr/0048-codeql-severity-gating.md | 5 +- internal/sched/handlers/handlers.go | 2 - internal/sched/handlers/upload.go | 2 - ruff.toml | 16 +- scripts/check_gate_needs_test.go | 5 +- web/next-env.d.ts | 1 + 17 files changed, 1291 insertions(+), 431 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5466eec8..aaa52782 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -46,13 +46,29 @@ gitleaks secret scan. **Every job must be green before merge.** Tests are deterministic without a live model: use the fake-LLM seam (`internal/fakellm` via `OPENROUTER_BASE_URL`), never a real key. -CodeQL (security queries) and Semgrep (Go/JS/Python SAST + Actions supply chain) -also run per PR, **fail on any finding**, and are **inside `ci-gate` and -`Dev gate`** — both are reusable workflows that ci.yml/dev-ci.yml call as jobs, -so a finding blocks the merge through the existing required check. `npm audit` -gates the web and rampart-service dependency trees the same way. Everything is -at zero findings today; keeping it there is the point. See -[`docs/SCANNING.md`](docs/SCANNING.md). +CodeQL (security queries, `security-extended`) and Semgrep (Go/JS/Python SAST + +Actions supply chain) also run per PR and are **inside `ci-gate` and `Dev gate`** +— both are reusable workflows that ci.yml/dev-ci.yml call as jobs. `npm audit` +(both npm trees, lockfile-only, any severity) and ruff (`check` **and** +`format --check`) gate the same way. + +Their thresholds differ, and the difference is load-bearing: + +- **Semgrep, ruff and `npm audit`: zero findings, gating on any finding.** +- **CodeQL: zero *blocking* findings.** It gates on the **High band** + (`security-severity >= 7.0`, or level `error`/`warning` for a rule publishing + no security-severity), minus a reviewed register of accepted `(rule, file)` + pairs in `.github/codeql-accepted-findings.json` — each with a written reason. + Below the band is **advisory**: printed in the job log and uploaded to the + Security tab, not blocking. So "CodeQL is green" means "no unwaived High-band + finding", not "no findings". The reasoning is [ADR-0048](docs/adr/0048-codeql-severity-gating.md). + +Two facts an agent must not get wrong here. **A `pull_request` CodeQL run is +diff-informed** — it evaluates every query over the full database, then reports +only results inside the PR's diff — so it certifies a *diff*, never a tree; only +push and scheduled runs give a tree-wide verdict. And **`Dev gate` is not a +required check on `dev`**, so on that branch a scanner failure is a red X beside a +mergeable PR. See [`docs/SCANNING.md`](docs/SCANNING.md) ("Known gaps"). ## Repository map @@ -124,8 +140,10 @@ same PR. `codecov.yml` were removed because the repo has no `CODECOV_TOKEN`, so the upload only ever produced a missing-token warning. Treat coverage as a quality signal, not a gate: add tests that catch real behavior, not to chase a - number. (The merge gates are build/vet/lint, the test suites, the - `-race` lane, govulncheck, Grype, the migration linter, and gitleaks.) + number. (The merge gates are build/vet/lint, ruff — `check` and + `format --check` — the test suites, the `-race` lane, govulncheck, Grype, + `npm audit` + `scripts/check-npm-overrides.sh`, CodeQL, Semgrep, the migration + linter, and gitleaks.) - **Match the surrounding code:** naming, idioms, and comment density. The `internal/agentcore` package comments explain *why* each governance invariant holds — preserve that level of explanation when you extend it. @@ -170,13 +188,16 @@ same PR. - **Testing strategy** (unit / fake-LLM / mocked + live Playwright / canary): [`docs/TESTING.md`](docs/TESTING.md) - **The scanning stack** (who checks what, why ruff owns Python lint, why - Semgrep is scoped to Actions supply chain after its broad packs scored 0/6, - what blocks vs what reports, and the known gaps): + Semgrep runs all four registry packs — `p/github-actions`, `p/golang`, + `p/javascript`, `p/python` — and blocks with 6 false positives waived at the + line, what blocks vs what only reports, and the known gaps, chief among them + that the `dev` ruleset requires no status checks): [`docs/SCANNING.md`](docs/SCANNING.md) - **CodeQL** (why default setup was replaced by an advanced-setup workflow, how - the Go toolchain is resolved, why it runs security queries only, and the - difference between a required status check and code scanning merge protection): - [`docs/CODEQL.md`](docs/CODEQL.md) + the Go toolchain is resolved, why it runs security queries only, why a + `pull_request` run certifies a diff and not a tree, and the High-band threshold + plus accepted-findings register): [`docs/CODEQL.md`](docs/CODEQL.md) + + [ADR-0048](docs/adr/0048-codeql-severity-gating.md) - **HTTP API versioning** (the `/v1` prefix + `X-Fleet-API-Version` + `/api-info` discovery + deprecation contract): [`docs/api-versioning.md`](docs/api-versioning.md) - **Database migrations** (the two runners, safe-DDL patterns, the migration DDL diff --git a/CHANGELOG.md b/CHANGELOG.md index 61ac4943..70759210 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,233 @@ prior versions are listed because none have shipped. ### Fixed -- **Every remaining scanner follow-up closed: rule families widened and fixed, +- **Three own-rows authorization holes on the task surface.** The read path for + task rows was narrowed to own rows in #1082 and run logs in #980; three + surfaces never got the same treatment and authorized on a *permission* alone, + so any client-role principal reached every principal's rows: + + - `GET /tasks/paused` — `ListPausedTasks` selects on status alone with no + principal predicate in SQL, and the projection carries each task's prompt. + Its siblings (`/tasks/export`, `/tasks/upcoming`) both call `visibleTasks`; + this one did not. It leaked other principals' paused prompts and their task + UUIDs. + - `PUT /tasks/{id}` and `POST /tasks/{id}/tags` — loaded the task with the + unscoped `GetTask` and never checked ownership, so a client-role principal + could rewrite a teammate's pending run: `prompt`, `model`, `mcp_selection` + and `credential_allowlist` included. Only `run_if` was gated (admin-only). + - `POST /tasks/{id}/feedback` and `GET /tasks/{id}/learned-instructions` — + `taskFromPath` is lookup-only by contract ("a handler that needs an + authorization decision makes it on the returned task") and neither caller + made one. A down-vote with an attacker-authored critique fed `maybeDistill`, + which mints a proposal from the victim's prompt at unmetered model spend; + the GET disclosed their learned instructions. + + The write gate is a new `taskWritableByPrincipal`, deliberately **not** + `principal.ownsTask`: `ownsTask` resolves through `ownerID()`, which is nil for + every API-key principal, so it would deny a scoped intake-app key the right to + edit the task it just created. `taskCreatedByPrincipal` matches a creating user + **or** a creating key (`CreatedByKeyID`) — the model #980/#1082 established. A + write surface must be no looser than the read surface guarding the same row. + `TestScopedAPIKeyAuthorization` previously asserted "client key can edit an + editable task" against an unattributed row, which *was* the vulnerable + behaviour; it is split into the owned case (must keep working — the intake-app + path) and the unowned case (must 403). Every fix is mutation-tested: stripped, + the tests fail with the exploit visible. + +- **Secret material and untrusted text reaching error strings, logs and the + persisted transcript**, from the same audit sweep: + + - `internal/config/config.go`: `ValidateScheduled` interpolated the first 6 + bytes of `OPENROUTER_API_KEY` into a validation error — the only place in the + tree where secret material reached an error string. Removed. Its doc comment + also claimed "Called at startup" and has no production caller; corrected. + - `internal/agent/scheduled.go`: run-error strings now pass through + `agentcore.RedactSecrets` before both the log and the **persisted + transcript**. Tool output, the stream sink, hooks and the session log were + already scrubbed; run errors were the one path that skipped it, and the + transcript is the larger surface. + - Log-injection sinks carrying genuinely untrusted text are now `logSafe`/`%q`, + matching each line's already-sanitized sibling: the task-create log + (`task.Prompt`), the pre-validation client attachment path on the reject + branches, the client-echoed attachment `Name`, the upload filename, and the + API-key name. + - `web/e2e/test-auth-key.ts`: the Ed25519 private key was written to a fully + predictable path in the world-writable temp dir at default `0644`. Now + `O_EXCL` at `0600` with random bytes in the sibling name. + +- **Two input-validation gaps with a shell/URL surface.** + `internal/mcpoauth/discovery.go` now refuses a non-`http(s)` scheme on the + remote-derived discovery URLs (a `WWW-Authenticate` `resource_metadata` + pointer, a PRM-declared issuer) *before* the request — already contained by + `SafeHTTPClient` and the transport, so this makes the argument explicit rather + than dependent on transport behaviour. And `internal/sched/models/models.go` + validates `WorktreeConfig.BaseBranch`: it is the trailing positional of + `git worktree add -b ` with no `--` separator, so a + leading-dash value was parsed by git as an option — and `worktree_config` is + settable by any task creator, unlike `run_if`. + +- **Two stale claims that an auditor would have read as capabilities.** + `models.MaxLogSubmissionSize` declared a 24 MB body cap that **nothing + enforced** — the cap actually applied is `MaxJSONBodySize` (1 MB, wired through + `BodySizeLimitMiddleware`), so the real posture was 24× stricter than the + constant claimed. `config.DefaultFromEmail`'s doc comment called it "the + fallback From address for outgoing mail" with no code path consuming it. Both + deleted rather than left standing, along with four other exported-but-unreached + identifiers (`mcpoauth.IsInvalidClient`, `apikeys.Manager.LogAction` — worth + naming because it reads as API-key *audit* surface — and the two halves of the + retired v1 remote-worker protocol, `TaskAssignment`/`LogSubmission`) and the + tree's only commented-out code block. `golangci-lint`'s `unused` already makes + unexported dead code structurally zero, which is why every deletion here is an + exported identifier in `internal/` — the class `unused` deliberately does not + report. Confirmed with `deadcode -test -tags fleet_host_executor ./...`. + +- **CI permission gaps and an alarm that ignored the failure it was built for.** + `scan-cron-alarm.yml` only fired on `conclusion == 'failure'`, which ignores + `startup_failure` — the exact failure its own header describes as the + motivating incident, and the one where *no scanning ran at all* — and + `timed_out`, which matters given codeql.yml caps at 30 minutes and semgrep.yml + at 15. It now alarms on any conclusion that is not `success` or `skipped`, and + the daily real-model canary joins the watched list (it had no alarm at all). + Noted in the file: the watcher matches on workflow **display name**, so + renaming `name:` disarms it. Separately, `ci.yml` carried `pull-requests: read` + at *workflow* level for golangci-lint-action's `only-new-issues`, which is + explicitly `false` — a scope with no consumer that nonetheless reached every + job not overriding it, including `web`, `playwright` and `e2e-live`, which + npm-install and run thousands of third-party packages. +- **The CodeQL gate was armed on a measurement that could not mean what it was + read to mean, and it deadlocked `dev`.** The `Fail on findings` step shipped + with a threshold of *any finding at any severity*, justified by the security + suite reporting zero across all four languages. That zero was real and it was + measured — on **Dev CI run 525, a `pull_request` event**. On PR events the + CodeQL action runs **diff-informed**: it builds the full database, evaluates + every query, and then reports only results whose location falls inside the PR's + diff. Run 525's own log says both halves (`Persisted 204 diff range(s) across + 43 file(s)`; `file coverage information is only enabled when analyzing the + default branch and protected branches`). It measured the **diff**, not the tree. + + The first full-tree evaluation was therefore the **push** that merged that work: + **Dev CI run 527**, which reported **38 Go and 17 javascript-typescript + findings** and turned `Dev gate` red — with no PR-shaped way out, since a PR + into `dev` is scanned diff-informed and comes back green while `dev` itself + stays red. The generalisable rule, now written into the docs: **a PR-event + CodeQL run certifies a diff, not a tree.** + + All 55 were triaged individually against the code. **Four were reachable and are + fixed in code, not waived:** + + - `internal/sched/handlers/handlers.go` logged `task.Prompt` unsanitized on the + task-**create** path while the **update** path's twin line was already wrapped + in `logSafe`. `POST /tasks` is reachable with a scoped `create_task` key, so + this was genuine log forgery. + - `internal/httpapi/attachments.go` logged the raw, pre-validation client + attachment path with `%s` on the two branches where the containment guard had + just *failed* — precisely where the value is hostile by construction. + - `internal/agent/session.go` logged the client-echoed attachment `Name`, which, + unlike `Path`, is never re-sanitized on the `/chat` path. + - `web/e2e/test-auth-key.ts` wrote an Ed25519 private key to a fully predictable + path in the world-writable temp dir at default `0644`. + + **The gate was then redesigned rather than switched off** + ([ADR-0048](docs/adr/0048-codeql-severity-gating.md)). A finding blocks when it + is unwaived and either its rule publishes `security-severity >= 7.0` (CodeQL's + own High/Critical cut) or, for a rule publishing no security-severity, its SARIF + level is `error`/`warning`. Level is deliberately **not** used for rules that do + publish a security-severity: nearly every CodeQL security query is + `@problem.severity error` — `go/log-injection` at 6.1 included — so banding on + level would block on all 23 log-injection findings and reproduce the deadlock. + Below the band is **advisory**: printed and uploaded to the Security tab, not + blocking. + + The remaining 51 findings live in `.github/codeql-accepted-findings.json`, a + register of accepted `(rule, file)` pairs each with a mandatory written reason. + Per-**file** is the point: a `query-filters` exclude would switch a + security-severity 9.1 query (`go/request-forgery`) off repo-wide, while a + register entry leaves it live everywhere else. Severity alone does not separate + the true from the false positives here — that 9.1 fires on the deliberate `@url` + fetch tool behind `internal/netguard`'s resolve-then-dial SSRF guard, and a 7.5 + `go/weak-sensitive-data-hashing` fires on SHA-256 used as a lookup index over a + 32-byte `crypto/rand` token — which is exactly why the band ships *with* a + reviewed register rather than instead of one. An in-source `// codeql[rule-id]` + comment waives too. + + One classifier, `.github/codeql-gate.jq`, is consumed by **both** the summary + and the gate, so the report and the block can never disagree about what + "blocking" means; the job log prints three tiers (BLOCKING / ACCEPTED by name / + ADVISORY). It **fails closed** on a missing register, a missing filter file, + unparseable SARIF, and — a vacuity check — findings present with **zero rule + metadata resolved**. That last one is not hypothetical: CodeQL writes query + metadata to `tool.extensions[].rules[]`, not `tool.driver.rules[]`, and a first + cut of the filter read only the driver, resolved nothing, scored every finding + at severity 0 and reported "0 blocking" over a tree that was not clean. + `scripts/check_codeql_register_test.go` keeps the register honest in `make test`. + +- **Two action pins were the annotated tag object of a mutable major tag, not a + commit.** `git ls-remote` returns the *tag object's* SHA for `refs/tags/v4` on a + repository that publishes annotated tags — for `github/codeql-action` that is + `4c0873ef…`, while the commit (`refs/tags/v4^{}`) is `db488dde…`. A pin taken + from the unpeeled form is a 40-hex string that passes every "is it a SHA" check + and still resolves to a **moving major tag**, which is the exact defect the + pinning exercise existed to remove. Two distinct pins were in that state across + 7 usages; both are repinned to the peeled commits with exact `# vX.Y.Z` + comments, and `scripts/check_action_pins_test.go` now asserts the shape so the + next pin cannot be taken from the wrong ref. Count for the record: **13** + workflow files, 12 of which reference an action, **53** third-party action + references, all SHA-pinned. + +- **`ci.yml`'s docs-only classifier skipped the suite over compiled product + content, and `CI gate` reported green over it.** The classifier matched `*.md` + at any depth plus all of `docs/*`. `*` spans `/` in a shell `case` pattern, so + that swallowed the `go:embed`'d `internal/clientconfig/builtin_skills/*/SKILL.md` + files (asserted by three test files), the shipped + `config/default/system_prompts/{default,chat}.md` — which *are* the system + prompts `docs/PROMPT-CACHE-CONTRACT.md` exists to protect — and + `docs/openapi.yaml`, which `cmd/fleet/openapi_drift_test.go` asserts against the + Go models, plus `docs/scripts/*.py` and `docs/img/*.py`, which are inside the + ruff, Semgrep `p/python` and CodeQL python scopes. A PR touching only a shipped + prompt or the OpenAPI spec therefore skipped the tests that validate it while + the gate went green. Narrowed to an explicit prose allow-list, and **`ci-gate` + now refuses to pass over a `skipped` job unless the classifier actually said + docs-only** — previously a skip from any cause was read as the docs-only case. + +- **Dependabot could rewrite CI on a branch with no required checks.** + `.github/dependabot.yml` targets `dev` for the `github-actions` ecosystem daily + with **no `cooldown`** (Dependabot supports `cooldown` for gomod and npm only), + `auto-merge-dependabot.yml` auto-merged patch bumps, and a `github-actions` + bump *is a rewrite of `.github/workflows/*`*. Since `gh pr merge --auto` only + holds a merge for checks that are **required**, and the `dev` ruleset requires + none, a same-day third-party action patch could land on `dev` with no CI and no + review. Mitigated on the workflow side: the `github_actions` ecosystem is now + **excluded from auto-merge at any bump level**, the workflow carries an explicit + `branches: [main, dev]` filter so its central assumption cannot silently stop + holding, and its write scopes moved to the job. **The remaining fix is a + repo-settings action nobody can perform from a PR** — adding `Dev gate` to the + `dev` ruleset's required status checks — and it is now documented as an open + item in [`docs/SCANNING.md`](docs/SCANNING.md) ("Known gaps") rather than + implied away. + +- **`fleet_ref` is validated against an allow-list shape before checkout** in + `build-sandbox-image.yml` and `publish-sandbox-image.yml`. The `pin` step + refuses an empty value, any character outside `[A-Za-z0-9._/-]` (so a newline + cannot smuggle a second step-output line), an implausible ref name, + `refs/pull/*` / `pull/*`, and a raw commit SHA — fork PR commits are reachable + by SHA from this repository, so only named refs are accepted. Both workflows + execute the checked-out build script and one of them holds `packages: write`. + +- **Documentation corrected against the shipped workflows** for an + enterprise-security review. The scanner docs had accumulated claims that were + true of an intermediate state and false of the merged one: CodeQL "fails on any + finding" and "zero findings across all four languages"; `docs/CODEQL.md`'s + entire Triggers section (it documented `push`/`pull_request` triggers the + workflow does not have, and justified the missing `push` on `dev` with the + reasoning that the PR run analyzes "identical content" — the very trap that + broke `dev`); `docs/TESTING.md` claiming the fast lane *skips* CodeQL when it in + fact runs both scanners; the Grype threshold in three places; "12 workflows"; + and `ruff.toml`'s own gate marker contradicting its `select`. `SECURITY.md` + gained the SAST section it had never had, plus the npm CVE gate it had omitted, + and the `dev`-ruleset gap is now stated wherever a doc claims something + "blocks". + +- **Scanner follow-ups from the same effort: rule families widened and fixed, CodeQL at `security-extended`, grype tightened, override canary, cron alarms, and one reasoned rejection.** @@ -34,33 +260,32 @@ prior versions are listed because none have shipped. intent stated at each. The one `subprocess.Popen` carries a reasoned `# noqa: S603` (argv is `sys.executable` plus internal literals; mutation-tested — stripping the noqa re-fires the rule). Full Go suite - green on the result; the sandbox fileops and bridge behavior is covered by - its tests. - - - **CodeQL widened to the `security-extended` suite** on all four languages, - adopted the same way everything else was: the default suite measured zero, - so the broader set starts from a clean baseline and its findings on this - PR's own run are the measurement. That measurement found exactly one thing - — and it was real: `actions/untrusted-checkout/medium` on - `build-sandbox-image.yml`'s `fleet_ref`-fed checkout. Fixed, not waived - (the `actions` language has no `AlertSuppression.ql`, so a comment waiver - does not even exist): the workflow now **refuses `refs/pull/*` refs before - checkout** — a fork-PR ref would put fork-controlled code into a workflow - that executes the checked-out build script — and the identical hardening - went into `publish-sandbox-image.yml`, the *unflagged* twin that holds - `packages: write` and escaped the name-heuristic query only because its - ref plumbing was named differently. Extended suite then verified clean in - CI on all four languages (Dev CI run 525). The CodeQL/Semgrep log summaries - also now print **`file:line` per finding** (plus a database file count as - the coverage line), and the override canary is invoked via - `$GITHUB_WORKSPACE` so it survives the job's `working-directory: web`. + green on the result. + + - **CodeQL widened to the `security-extended` suite** on all four languages. + Its one `actions`-language finding was real: + `actions/untrusted-checkout/medium` on `build-sandbox-image.yml`'s + `fleet_ref`-fed checkout. Fixed, not waived (the `actions` language has no + `AlertSuppression.ql`, so a comment waiver does not even exist) — see the + `fleet_ref` entry above, and note that the identical hardening went into + `publish-sandbox-image.yml`, the *unflagged* twin that holds `packages: write` + and escaped the name-heuristic query only because its ref plumbing was named + differently. The suite's no-findings result in CI (Dev CI run 525) was a + `pull_request` run and is therefore a statement about that diff, not the tree + — see the first entry above for what the tree actually held. The + CodeQL/Semgrep log summaries also now print **`file:line` per finding** plus a + database file count as the coverage line, and the override canary is invoked + via `$GITHUB_WORKSPACE` so it survives the job's `working-directory: web`. - **Grype gate tightened to fixable CRITICAL + HIGH**, after measuring: the - published sandbox image carries zero fixable Critical/High RPM findings - (its only fixable findings are two Medium openssh advisories, which the - next routine image rebuild picks up). Policy change mutation-tested in - three directions: real scan passes, injected fixable High fails, injected - fixable Medium still passes. + published sandbox image carries zero fixable Critical/High RPM findings (its + only fixable findings are two Medium openssh advisories, which the next + routine image rebuild picks up). The policy stays **RPM-only** + (`.artifact.type == "rpm"`), because the Python `dist-info` Grype catalogs + alongside Fedora's RPMs carries upstream versions and advisories — gating on + it produced pip wheels layered over distro-owned files. Those records still + upload to SARIF. Policy change mutation-tested in three directions: real scan + passes, injected fixable High fails, injected fixable Medium still passes. - **`scripts/check-npm-overrides.sh`**: the rampart sharp/adm-zip overrides are forks of upstream's intent, correct only while upstream is broken — so @@ -85,38 +310,74 @@ prior versions are listed because none have shipped. this public MIT repo would be redistribution. The binary stays pinned; the rules stay registry-fetched with the failure mode documented. -- **The scanners gate through `ci-gate`/`Dev gate` themselves, npm dependencies - are audited, and the whole Python tree is ruff-formatted — with every finding - fixed, none deferred.** - - - **Gate wiring, corrected.** The previous entry said making CodeQL/Semgrep - merge-blocking needed a branch-protection click, reasoning from "`needs` - cannot cross workflow files". Incomplete: `codeql.yml` and `semgrep.yml` are - now **reusable workflows** (`on: workflow_call`) that ci.yml and dev-ci.yml - call as jobs, and those jobs sit in `ci-gate`'s / `Dev gate`'s `needs` — so a - scanner finding blocks a merge through the one existing required check, no - settings change anywhere. Their own push/pull_request triggers are removed - (nothing runs twice); the weekly re-scan crons and a workflow_dispatch stay. - - - **`npm audit` is a new blocking gate** for both npm trees, lockfile-only and - failing on any severity — the npm counterpart of the govulncheck gate. - `web/` was already clean. `scripts/rampart-service` **had no lockfile at - all**, and generating one exposed **5 high-severity vulnerabilities** it had - been hiding: `sharp <0.35.0` (four libvips CVEs) and `adm-zip <0.6.0` - (GHSA-xcpc-8h2w-3j85) via `onnxruntime-node`. No upstream release fixes - either — latest `@huggingface/transformers` still pins `sharp ^0.34.5`, and - npm's suggested "fix" was a breaking transformers downgrade — so the - package now carries `overrides` to `sharp ^0.35.3` and `adm-zip ^0.6.0`, - each the release immediately after the vulnerable line. The overridden - stack was installed and load-tested, not just resolved: sharp renders a PNG - through the new libvips, transformers loads on it, rampart exports its API, - adm-zip round-trips a zip. Both trees now audit at 0. - - - **`ruff format` applied and gated.** 9 of 13 Python files reformatted - (~3.7k lines), `ruff format --check` now blocks in both CI lanes and in - `make lint`. Validated by the full Go suite (the bento/fileops golden tests - exercise the reformatted scripts), byte-compilation of every file, and a - re-scan showing the fileops `nosemgrep` waiver survived the reformat. +- **The scanning stack, as shipped.** This entry supersedes four earlier ones that + described intermediate states of the same work and contradicted each other on + every threshold that matters — whether the scanners gate through `ci-gate` or + stay advisory, whether `ruff format` gates, whether Semgrep ships one pack or + four, and whether CodeQL's code-quality suite was restored or dropped. The + shipped end state, stated once: + + - **Python had no linter at all, and ruff is now a blocking gate.** Go had + golangci-lint and the web tier had oxlint; the tree's 13 Python files — the + sandbox FileOp helper, the python bridge, the bento-slides and data-profiler + skill scripts, MCP test servers, icon/doc generators — had nothing. Rule + selection is narrow on purpose and `ruff.toml` records the numbers: + `E4,E7,E9,F` found 3 real findings (an unused import, a lambda assignment, and + a **byte-identical duplicate `has_guard` definition** in `bento_doc.py` where + the second copy silently shadowed the first); `B`/`SIM`/`S` found 21 more, all + fixed and those families then enabled; a broad selection finds 333, of which + 176 are `%`-format style and 43 are magic values, so the style tiers stay out. + **`ruff format --check` also gates** — the whole tree was ruff-formatted in one + dedicated commit (9 of the 13 files, ~3.7k lines, validated by the full Go + suite, since the bento/fileops golden tests exercise these scripts). `F401` is + waived for `internal/mcp/testdata/*.py` and `cmd/fleet/testdata/*.py`, where an + unused import can be the point of the fixture. + + - **CodeQL narrowed to security queries only — the code-quality suite was + enabled, measured, and dropped.** It produced 32 findings, every one + note-level: for Go and the web tier it duplicated golangci-lint and oxlint, + which already block; 28 of the 32 were Python, now ruff's job at a fraction of + the runtime and with autofix; and 3 were false positives on correct code + (`value != value`, the idiomatic NaN test). CodeQL keeps the thing nothing else + here can do — interprocedural taint, which is the actual shape of "a credential + must not reach a log sink". + + - **Semgrep runs all four packs and blocks** — `p/github-actions`, `p/golang`, + `p/javascript`, `p/python`, with `--error` and no `continue-on-error`. The 6 + non-Actions findings are false positives, suppressed at the line with + `nosemgrep: ` plus a reason: three were *already* triaged and + suppressed for gosec, and one (`0o644` for a sandbox directory) would have been + a security regression if followed. Every suppression was mutation-tested — + removing it makes the finding reappear, so a green scan means the waivers work + rather than the rules having silently stopped matching. `p/github-actions` also + found the one real class nothing else here checks: **51 actions referenced by a + mutable tag**, all now SHA-pinned (see the pin correction above). + + - **`npm audit` is a blocking gate** for both npm trees, lockfile-only and + failing on any severity — the npm counterpart of the govulncheck gate. `web/` + was already clean. `scripts/rampart-service` **had no lockfile at all**, and + generating one exposed **5 high-severity vulnerabilities** it had been hiding: + `sharp <0.35.0` (four libvips CVEs) and `adm-zip <0.6.0` + (GHSA-xcpc-8h2w-3j85) via `onnxruntime-node`. No upstream release fixes either + — latest `@huggingface/transformers` still pins `sharp ^0.34.5`, and npm's + suggested "fix" was a breaking transformers downgrade — so the package carries + `overrides` to `sharp ^0.35.3` and `adm-zip ^0.6.0`, each the release + immediately after the vulnerable line. The overridden stack was installed and + load-tested, not just resolved: sharp renders a PNG through the new libvips, + transformers loads on it, rampart exports its API, adm-zip round-trips a zip. + Both trees now audit at 0. + + - **Gate wiring.** `codeql.yml` and `semgrep.yml` are **reusable workflows** + (`on: workflow_call`) that ci.yml and dev-ci.yml call as jobs, and those jobs + sit in `ci-gate`'s / `Dev gate`'s `needs` — so a scanner finding reaches the + aggregate check with no settings change. Their own push/pull_request triggers + are removed (nothing runs twice); the weekly re-scan crons and a + `workflow_dispatch` stay. An earlier entry here claimed this needed a + branch-protection click, reasoning from "`needs` cannot cross workflow files": + true of a job's `needs`, but a `workflow_call` brings the called jobs into the + caller's file. **Whether that red check blocks a merge is a separate, + branch-dependent fact** — `CI gate` is required on `main`; the `dev` ruleset + requires no status checks at all. - **All three semgrep parse errors fixed**, so no file is partially covered: `${{ steps.build.outcome }}` interpolated into a `run:` script in @@ -124,94 +385,15 @@ prior versions are listed because none have shipped. `${tag:-(…)}` expansion default whose bare paren choked the bash sub-parser (hoisted to a plain assignment), and an inline `import("@playwright/test")` type in fixtures.ts (now a named `import type`; web lint, tsc and all 1104 - vitest tests pass on it). The scanners' coverage lines now read - **0 parse/scan errors** alongside 0 findings. - -- **The scanners now block, and the repo passes them.** Turning a gate on over an - unfixed backlog is how a gate becomes something people route around, so - everything they reported was fixed or adjudicated first. - - - **All 53 action references pinned to commit SHAs.** Semgrep's - `github-actions-mutable-action-tag` found 51 instances of actions referenced - by a mutable tag (`actions/checkout@v7`); if a tag moves, - attacker-controlled code runs with this repo's `GITHUB_TOKEN`. Every `uses:` - across all 12 workflows is now `@<40-hex-sha> # ` — the form - Dependabot updates, and `.github/dependabot.yml` already watches the - `github-actions` ecosystem. Each SHA is the commit the previously-used tag - resolved to at pin time, so the pin does not smuggle in a version bump. - - - **Semgrep blocks over all four packs** (`p/github-actions`, `p/golang`, - `p/javascript`, `p/python`) with `--error` and no `continue-on-error`. The 6 - false positives are suppressed at the line with `nosemgrep: ` plus a - reason — three of them were *already* triaged and suppressed for gosec, and - one (`0o644` for a sandbox directory) would have been a security regression - if followed. Every suppression was mutation-tested: removing it makes the - finding reappear, so a green scan means the waivers work rather than the - rules having silently stopped matching. - - - **CodeQL fails on findings.** Previously the analyze step exited 0 whether it - found nothing or a hundred alerts, so a red check could only ever mean "the - scanner broke" — which is exactly how the Go toolchain break hid for weeks. - Threshold is any finding, safe because the security suite reports zero across - all four languages. - - Both scanners report as their own checks (`CodeQL gate`, `Semgrep scan`) rather - than through `ci-gate`, because a job's `needs` cannot reach across workflow - files. **Making a red check actually block a merge still requires adding those - two checks to the branch ruleset** — a workflow file cannot make itself - required. - - Two knock-on fixes found while doing this: SHA pinning broke two regexes in - `scripts/check_versions_test.go` that matched `golangci-lint-action@v\d+`, and - they fail *open* by skipping — so they were widened to tolerate a pinned ref - plus its trailing version comment, and mutation-tested to confirm they still - bite. And a standalone `nosemgrep` comment inside a Go import block breaks - `goimports`, so that one waiver is a trailing comment instead. - -- **Python had no linter, and two scanners were pointed at ground already - covered.** Reshaped the scanning stack so each tool owns one job - ([`docs/SCANNING.md`](docs/SCANNING.md)): - - - **ruff is new, and it blocks.** fleet ships 13 Python files — the sandbox - FileOp helper, the python bridge, the bento-slides and data-profiler skill - scripts, MCP test servers — and *nothing* linted any of them. Go had - golangci-lint, the web tier had oxlint, Python had neither. Rule selection is - narrow on purpose and `ruff.toml` records why: the default rules find 3 - findings on this tree, a broad selection finds 333, of which 176 are - `%`-format style and 43 are magic values. Three real findings were fixed to - make the gate clean on day one — an unused import, a lambda assignment, and a - **byte-identical duplicate `has_guard` definition** in `bento_doc.py` where - the second copy silently shadowed the first. `ruff format` is reported but - not gated (the tree has never been ruff-formatted). - - - **CodeQL narrowed to security queries only.** Its code-quality suite was - enabled, measured, and dropped: 32 findings, every one note-level, zero - security findings. For Go and the web tier it duplicated golangci-lint and - oxlint, which already block; 28 of the 32 were Python, now ruff's job; and 3 - were false positives on correct code (`value != value`, the idiomatic NaN - test). CodeQL keeps the thing nothing else here can do — interprocedural - taint, which is the actual shape of "a credential must not reach a log sink". - - - **Semgrep is new, scoped, and advisory.** The obvious move — point it at - `p/golang`/`p/javascript`/`p/python` — was measured and rejected: 6 of 6 - non-Actions findings were false positives, three of them *already* triaged - and suppressed for gosec, and one (`0o644` for a sandbox directory) would - have been a security regression if followed. What ships is - `p/github-actions`, which found 51 instances of one real issue nothing else - checks: actions pinned to mutable tags rather than commit SHAs. Advisory - because all 51 are real and repinning is its own PR, not because they are - doubted. - - - **Both scanners now print findings to the job log** and the step summary, and - Semgrep uploads raw JSON as an artifact. A CodeQL run otherwise reports - nothing about what it found to its own log — it writes SARIF, uploads it, and - exits 0 either way — which made outcomes invisible to `gh run view` and to - any agent holding the log but not the code-scanning API. - - Also added: an aggregate `CodeQL gate` job, so making CodeQL blocking later is - one required check rather than four per-language checks needing manual - re-pointing whenever the matrix changes. Nothing here is wired into `ci-gate` - beyond ruff; CodeQL and Semgrep stay advisory. + vitest tests pass on it). The scanners' coverage lines read **0 parse/scan + errors**. + + - Two knock-on fixes found while doing this: SHA pinning broke two regexes in + `scripts/check_versions_test.go` that matched `golangci-lint-action@v\d+`, and + they fail *open* by skipping — so they were widened to tolerate a pinned ref + plus its trailing version comment, and mutation-tested to confirm they still + bite. And a standalone `nosemgrep` comment inside a Go import block breaks + `goimports`, so that one waiver is a trailing comment instead. - **CodeQL had stopped analyzing the repo's Go code, and then stopped analyzing anything.** Default setup's Go analysis failed on every main-targeting PR from @@ -232,26 +414,26 @@ prior versions are listed because none have shipped. at all in the interim. Replaced with an advanced-setup workflow, `.github/workflows/codeql.yml`, which - restores security analysis over go, python, javascript-typescript and actions - plus the code-quality query suite over the first three, and resolves Go's - interpreter from `go.mod` via `actions/setup-go` — never a literal version, the - bug class #1240 and #1241 already fixed twice for node. - - Two things the first cut got wrong, both of which ran **green**: `analysis-kinds` - turns out to be GitHub-internal and unusable in a custom workflow (it logged - `##[error]` and silently continued with security only), and Go extraction - missed exactly one file — `internal/sandbox/host.go`, the unsandboxed host - executor, invisible to the default build behind `//go:build - fleet_host_executor`. Fixed with `queries: code-quality` and - `GOFLAGS: -tags=fleet_host_executor`, the same tag `ci.yml` and `dev-ci.yml` - already pass to `go vet` and `go test`. + restores security analysis over go, python, javascript-typescript and actions, + and resolves Go's interpreter from `go.mod` via `actions/setup-go` — never a + literal version, the bug class #1240 and #1241 already fixed twice for node. + (The code-quality suite was also restored at this point, then measured and + dropped — see "The scanning stack, as shipped" above for where that landed.) + + Two things the first cut got wrong, both of which ran **green**: + `analysis-kinds` turns out to be GitHub-internal and unusable in a custom + workflow (it logged `##[error]` and silently continued with security only), and + Go extraction missed exactly one file — `internal/sandbox/host.go`, the + unsandboxed host executor, invisible to the default build behind + `//go:build fleet_host_executor`. Fixed with `queries: code-quality` (at the + time) and `GOFLAGS: -tags=fleet_host_executor`, the same tag `ci.yml` and + `dev-ci.yml` already pass to `go vet` and `go test`. Verified from the extractor's own output rather than the check mark: `extraction succeeded for all 2 discovered project(s)`, 916 packages, 426 `.go` files including `host.go`, and distinct queries evaluated rising from 72→116 (go), 90→292 (python) and 178→374 (javascript-typescript) as the quality suite - came in, with `actions` unchanged at 36 by design. CodeQL remains advisory — - these jobs are deliberately not wired into `ci-gate`. See + came in, with `actions` unchanged at 36 by design. See [`docs/CODEQL.md`](docs/CODEQL.md). - **`fleet update` built the web tier on the node it had just refused.** Every diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8bb50ef1..0d16cd11 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -98,19 +98,50 @@ each runs, and the `make` targets that mirror them locally (`make ci-go`, Every pull request must be green before merge. CI runs: - **Go** — `go build`, `go vet`, `golangci-lint` (full gate — fails on any - finding), and `go test`. -- **Web** — `npm run lint`, vitest, and `npm run build`. + finding), `go test`, and a `-race` lane. +- **Python** — `ruff check` **and** `ruff format --check` over the tree's Python + files (`make lint-python` runs both, and skips loudly if ruff is not installed). +- **Web** — `npm run lint` (oxlint), `npm run typecheck` (`tsc --noEmit`), vitest, + and `npm run build`. - **Playwright** — the mocked suite, plus a live suite against a real backend with a stubbed LLM (no OpenRouter spend). - **Secret scan (gitleaks)** — fails the build on any new, un-ignored secret. -- **Container image scan (Grype)** — fails the build on a fixable CRITICAL or HIGH CVE in - the sandbox image built from `config/default/sandbox/Containerfile` (HIGH and - below are reported, not blocking). Findings upload to GitHub Security → Code - scanning. A separate weekly scheduled scan (non-blocking) catches new CVEs - against the existing image between PRs. (Grype, not Trivy: the image's +- **SAST (CodeQL and Semgrep)** — both are reusable workflows called by `ci.yml` + and `dev-ci.yml`, so they sit inside the aggregate gate. **Semgrep** fails on + any unsuppressed finding across `p/github-actions`, `p/golang`, `p/javascript` + and `p/python`; a false positive is waived at the line with + `nosemgrep: ` plus a reason. **CodeQL** (`security-extended` over go / + python / javascript-typescript / actions) fails on an unwaived finding in the + **High band** — `security-severity >= 7.0`, or level `error`/`warning` for a + rule that publishes no security-severity — with lower-severity findings + reported as advisory. A false positive is waived either by an in-source + `// codeql[rule-id]` comment or by an entry in + `.github/codeql-accepted-findings.json` **with a written reason**; both are + reviewable in the diff, and fixing the code is always preferred. See + [`docs/SCANNING.md`](docs/SCANNING.md), [`docs/CODEQL.md`](docs/CODEQL.md) and + [ADR-0048](docs/adr/0048-codeql-severity-gating.md). +- **Dependency CVEs** — `govulncheck` for the Go module, and + `npm audit --audit-level=low` (lockfile-only, **any** severity) for both + `web/` and `scripts/rampart-service`, alongside + `scripts/check-npm-overrides.sh`, which fails once upstream ships fixes that + make the pinned security `overrides` droppable. +- **Container image scan (Grype)** — fails the build on a fixable **CRITICAL or + HIGH** CVE in an **RPM** of the sandbox image built from + `config/default/sandbox/Containerfile`. MEDIUM and below are reported, not + blocking, and the Python `dist-info` records Grype catalogs alongside the RPMs + never block whatever their severity (they are uploaded to SARIF — the rationale + is in [`docs/TESTING.md`](docs/TESTING.md)). Findings upload to GitHub Security + → Code scanning. A separate weekly scheduled scan (non-blocking) catches new + CVEs against the existing image between PRs. (Grype, not Trivy: the image's `fedora-minimal` base has no Trivy advisory feed, so Trivy would scan none of its packages; Grype matches its RPM + Python packages against NVD/GHSA.) +One qualifier on "must be green", because it differs by branch: `CI gate` is a +**required** status check on `main`, so a red lane genuinely blocks the merge +there. The `dev` ruleset requires no status checks, so on `dev` a red `Dev gate` +is a signal rather than a block — please treat it as one anyway. See +[`docs/SCANNING.md`](docs/SCANNING.md) ("Known gaps"). + If golangci-lint flags something, either fix it or add a `//nolint` with a reason (the `nolintlint` linter requires the reason). The lint backlog is at zero — please keep it there. diff --git a/Makefile b/Makefile index 4a53dfaa..6686d860 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: all build compile bins fleet-bench install test test-race test-cover lint lint-go lint-migrations fmt tidy clean help \ +.PHONY: all build compile bins fleet-bench install test test-race test-cover lint lint-go lint-python lint-migrations fmt tidy clean help \ govulncheck ci-go ci-web ci-e2e-mocked ci-local # GOTOOLCHAIN=auto — the operator does NOT have to hand-install the pinned Go. @@ -35,7 +35,9 @@ help: @echo " make test run the Go test suite" @echo " make test-race run the Go test suite with the race detector" @echo " make test-cover run the Go test suite with coverage (writes coverage.out)" - @echo " make lint run golangci-lint + the migration DDL linter" + @echo " make lint run golangci-lint + ruff (check & format) + the migration DDL linter" + @echo " make lint-go golangci-lint only" + @echo " make lint-python ruff check + ruff format --check (skips loudly if ruff is absent)" @echo " make lint-migrations reject dangerous DDL in changed migration files (#256)" @echo " make fleet-bench build the load-testing tool (cmd/fleet-bench, #296)" @echo " make fmt gofmt the tree" diff --git a/README.md b/README.md index 20a112fe..9c66d7ba 100644 --- a/README.md +++ b/README.md @@ -195,6 +195,11 @@ and status codes remain documentary. ## Repository layout +Abridged — the load-bearing directories, not every package. `internal/` alone +holds roughly forty packages; `cmd/` also carries the test/bench helpers +(`fake-llm`, `fleet-bench`), and `scripts/` and `.github/` hold the operator +scripts and the CI definition. + ``` cmd/ fleet/ the one unified binary — server (`fleet serve`: chat HTTP/SSE + orchestrator HTTP + scheduler + worker pool) AND operator CLI (every other verb) @@ -214,6 +219,9 @@ internal/ sched/ orchestrator/scheduler (was moc) + its migrations httpapi/ chat HTTP/SSE/auth layer config/ unified configuration (env loading; the MCP catalog comes from the bundle) + ... (~30 more: agentcore's neighbours, netguard, mcpoauth, observability, ...) +scripts/ bootstrap / update / doctor, the sandbox image build, and the CI policy checks +.github/ workflows (the CI + SAST gates), CODEOWNERS, dependabot, the CodeQL gate filter web/ one Next.js app: /chat and /orchestrator config/default/ the GENERIC client bundle baked into the repo (runs bare), including config/default/sandbox/Containerfile — the sandbox @@ -353,6 +361,8 @@ Deep references live in [`docs/`](docs/) so this README stays an orientation, no | [`docs/SERVER-STATS.md`](docs/SERVER-STATS.md) | Admin Server tab — lightweight CPU, memory, disk, network, and uptime status | | [`docs/BACKUP_RESTORE.md`](docs/BACKUP_RESTORE.md) | Disaster recovery — backup + restore of both databases | | [`docs/WEBHOOK-SIGNING.md`](docs/WEBHOOK-SIGNING.md) · [`docs/TESTING.md`](docs/TESTING.md) | Webhook HMAC signing · the test suite + fake-LLM seam | +| [`docs/SCANNING.md`](docs/SCANNING.md) | The scanning stack — which of golangci-lint / ruff / govulncheck / Grype / gitleaks / npm audit / CodeQL / Semgrep owns what, what actually blocks a merge, and the known gaps | +| [`docs/CODEQL.md`](docs/CODEQL.md) | CodeQL specifics — advanced setup, the four-language matrix, the High-band gate + accepted-findings register, and why a PR-event run certifies a diff rather than a tree | | [`docs/BUILDING-ON-FLEET.md`](docs/BUILDING-ON-FLEET.md) | The HTTP API as an automation substrate — keys, kicking off jobs, consuming structured output | | [`docs/MCP-CATALOG.md`](docs/MCP-CATALOG.md) | The connector catalog — bundled vs third-party trust classes | | [`docs/adr/`](docs/adr/) | Architecture Decision Records — the *why* behind the non-negotiable invariants | @@ -417,7 +427,9 @@ standards. Our thanks to the teams and communities behind them: Python data stack installed as **signed Fedora RPMs** instead of `pip` at runtime — one audited supply chain, not a thousand PyPI tarballs. fleet deliberately tracks the rolling tag so every on-box rebuild picks up the - current patches, and per-PR + weekly Grype scans keep the claim honest. + current patches, and Grype scans keep the claim honest — on every main-targeting + PR that is not docs-only, plus a weekly scheduled re-scan of the existing image + (PRs into `dev` get no image scan; it runs at the dev→main promotion). - **[Model Context Protocol](https://modelcontextprotocol.io)** and its SDKs — the open standard fleet speaks (stdio + HTTP) to reach tools and data through a credential-brokered MCP catalog. diff --git a/SECURITY.md b/SECURITY.md index 14525ae0..a3ee7176 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -39,17 +39,114 @@ yet. Please reproduce against current `main` before reporting. ## Secret scanning -CI runs [gitleaks](https://github.com/gitleaks/gitleaks) on every push and pull -request and fails the build on any new, un-ignored secret. If you are -contributing, never commit real credentials — the generic `config/default` -bundle ships with no connector secrets, and all deployment secrets live in an -operator-managed `0600` env file outside the repo (see the README). +CI runs [gitleaks](https://github.com/gitleaks/gitleaks) — `gitleaks dir . +--redact --exit-code 1`, over the whole working tree, not just the diff — and +fails the build on any new, un-ignored secret. It is the one lane that is +deliberately **not** gated on the docs-only classifier, because a secret can be +pasted into a markdown file. + +To be precise about coverage, since "on every push" would overstate it: the scan +runs on **pull requests into `dev` and `main`, and on pushes to `dev` and +`main`**. Those are the only events either CI workflow subscribes to, so a push +to a personal feature branch runs no CI at all — including no secret scan — until +a PR is opened against `dev`. Treat pre-PR local hygiene accordingly. + +If you are contributing, never commit real credentials — the generic +`config/default` bundle ships with no connector secrets, and all deployment +secrets live in an operator-managed `0600` env file outside the repo (see the +README). + +## Static analysis (SAST) + +Two static analyzers gate merges, and an auditor will want them by name. The full +design notes are [`docs/SCANNING.md`](docs/SCANNING.md) (who checks what, and what +actually gates) and [`docs/CODEQL.md`](docs/CODEQL.md); the threshold decision is +[ADR-0048](docs/adr/0048-codeql-severity-gating.md). + +**CodeQL — advanced setup, four languages, `security-extended`.** +`.github/workflows/codeql.yml` analyzes `go`, `python`, `javascript-typescript` +and `actions`, each with the `security-extended` query suite (the broader security +set, not the default one). Go builds via `autobuild` with +`GOFLAGS=-tags=fleet_host_executor`, so `internal/sandbox/host.go` — the +unsandboxed host executor, fenced out of the default build — is inside the +database rather than the one file the analysis cannot see. It is a **reusable** +workflow: `ci.yml` (main) and `dev-ci.yml` (dev) call it as a job, so its result +lands in the caller's aggregate gate. There is also a weekly schedule, because a +CodeQL verdict is a function of the query pack as well as the commit. + +Four properties of that gate matter for an audit, and each is a deliberate +limit rather than an oversight: + +- **The threshold is the High band, not "any finding".** A finding blocks when its + rule publishes `security-severity >= 7.0` (CodeQL's own High/Critical cut), or — + for a rule that publishes no security-severity — when its SARIF level is + `error`/`warning`. Level is deliberately not used for rules that *do* publish a + security-severity: nearly every CodeQL security query is + `@problem.severity error`, `go/log-injection` at severity 6.1 included, so + banding on level would block on everything. Findings below the band are + **advisory** — printed in the job log and uploaded to the Security tab, not + blocking. +- **Accepted findings are a reviewed register, not a disabled query.** + `.github/codeql-accepted-findings.json` holds accepted `(rule, file)` pairs, + each with a mandatory written reason that must say why the finding is not + exploitable *there*. It is per-**file** on purpose: a `query-filters` exclude + would switch a security-severity 9.1 query off repo-wide, whereas a register + entry leaves it live everywhere else. An in-source `// codeql[rule-id]` comment + is the second waiver route. Widening the register appears in the PR diff, and + `scripts/check_codeql_register_test.go` fails the test suite on an entry that + names a missing file, lacks a reason, or has become decoupled from the workflow. +- **A `pull_request` run certifies a diff, not a tree.** On PR events the CodeQL + action runs **diff-informed**: it builds the full database and evaluates every + query, then reports only results inside the PR's diff. Tree-wide verdicts come + only from the push and scheduled runs. Any "the scanners are green, therefore + the tree is clean" claim resting on a PR run is unsound — this repo learned that + the expensive way, and ADR-0048 records it. +- **`_test.go` files are outside the Go database.** `autobuild` builds packages, + not tests, so every `_test.go` file in the tree (625 at the time of writing) is + unanalyzed. Unchanged from GitHub's default setup; bringing them in would + require `build-mode: manual`. + +The gate **fails closed**: a missing register, a missing filter file, unparseable +SARIF, or findings present with zero rule metadata resolved all fail the job +rather than reporting clean. One shared classifier (`.github/codeql-gate.jq`) +feeds both the report and the gate, so the two cannot disagree, and the job log +prints three tiers — BLOCKING, ACCEPTED (by name), ADVISORY. Note that +**dismissing an alert in the Security tab does not turn the check green**: the +gate reads the run's own SARIF and never consults the code-scanning API. + +**Semgrep — all four registry packs, blocking on any finding.** +`.github/workflows/semgrep.yml` runs `p/github-actions`, `p/golang`, +`p/javascript` and `p/python` with `--error` and no `continue-on-error`, also as a +reusable workflow inside both gates, also with a weekly schedule. The tree is at +zero unsuppressed findings; six false positives are waived at the line with +`nosemgrep: ` plus a reason, and each waiver was mutation-tested +(removing it makes the finding reappear, so a green scan means the waivers work +rather than the rules having silently stopped matching). One honest limitation: +the rule packs are fetched from the registry at scan time and cannot be vendored — +the Semgrep Rules License v1.0 forbids redistribution, and this is a public MIT +repo — so a registry-side rule addition can turn the lane red with no commit to +blame. The binary version is pinned; the rules are not. + +**All 53 third-party action references across the 13 workflow files are pinned to +40-hex commit SHAs** with the version in a trailing comment, which is also the +form Dependabot updates. Two of those pins were subtly wrong — taken from the +*annotated tag object* of a mutable major tag rather than the commit it points at, +so they looked like commit pins while resolving to a moving tag — and both are now +the peeled commit, with `scripts/check_action_pins_test.go` asserting the shape. + +**Where enforcement actually lands.** All of the above is wired into the branches' +aggregate gate jobs, but a gate job only blocks a merge where it is a *required* +status check. `CI gate` is required on `main`. The `dev` ruleset requires no +status checks at all, so `Dev gate` — and every scanner inside it — is +red-but-not-required on `dev`. That gap, and what it interacts with, is written up +under "Known gaps" in [`docs/SCANNING.md`](docs/SCANNING.md). ## Supply-chain security (dependencies) -Fleet pulls third-party code from two ecosystems — Go modules at the repo root -and npm packages under `web/` — and relies on several deliberate controls to -keep a compromised or fresh-and-unvetted release from reaching `main`: +Fleet pulls third-party code from three ecosystems — Go modules at the repo root, +npm packages under `web/` and under `scripts/rampart-service`, and Fedora RPMs +inside the sandbox image — and relies on several deliberate controls to keep a +compromised or fresh-and-unvetted release from reaching `main`: - **Go module integrity is verified, with the defaults intact.** The repo commits a complete `go.sum`, and the build does **not** set any of `GOFLAGS`, `GOPROXY`, @@ -64,29 +161,72 @@ keep a compromised or fresh-and-unvetted release from reaching `main`: - **Dependency-CVE scanning.** CI runs `govulncheck` against the Go module on every PR (the `govulncheck` job in `.github/workflows/ci.yml`), failing the build on a known-vulnerable dependency that fleet actually calls into. +- **npm dependency-CVE scanning.** `npm audit --audit-level=low` runs in the + `web` job of **both** CI lanes, lockfile-only (before the install, so a + vulnerable lockfile fails fast) and against **both** npm trees — `web/` and + `scripts/rampart-service` — failing the build on **any** severity. Like + govulncheck, its verdict is a function of the clock as well as the commit: a + newly published advisory can redden an unchanged tree, which is the point. + + Two `overrides` in `scripts/rampart-service/package.json` are load-bearing and + worth disclosing: `sharp ^0.35.3` and `adm-zip ^0.6.0`, each the release + immediately after a vulnerable range that **no upstream release yet fixes** + (`@huggingface/transformers` still pins `sharp ^0.34.5`; `adm-zip` arrives + under `onnxruntime-node`). An override is a fork of upstream's intent, correct + only while upstream is broken — so `scripts/check-npm-overrides.sh` runs beside + the audit in both lanes and **fails the build with removal instructions the day + upstream's own ranges reach the patched lines**. A registry flake skips with a + notice rather than delivering a verdict; the audit above is the CVE gate. - **Container-image CVE scanning.** CI also scans the rootless-Podman sandbox - image (built from `config/default/sandbox/Containerfile`) with Grype on every - non-docs PR (the `grype-scan` job), failing the build on a *fixable* CRITICAL - CVE in the image's RPM or Python packages — a surface `govulncheck` (Go modules - only) cannot see. Findings upload to GitHub Security → Code scanning, and a - weekly scheduled scan (`.github/workflows/grype-scheduled.yml`) catches - newly-disclosed CVEs against the existing image between PRs. -- **Release cooldown.** `.github/dependabot.yml` applies a `cooldown` to the gomod - and npm surfaces so Dependabot waits a few days (3 for patch, 7 for minor, 14 - for major) before proposing a freshly published release. This blunts fast - typosquat / account-takeover attacks, where a malicious version is published and - then yanked once the ecosystem flags it. It matters most for **patch** bumps, - which `.github/workflows/auto-merge-dependabot.yml` auto-merges once the full CI - gate is green: without a cooldown a minutes-old patch could be proposed and - auto-merged before any scrutiny. Cooldown applies to version updates only — - Dependabot **security** updates are never delayed, so urgent CVE fixes still - flow immediately. + image (built from `config/default/sandbox/Containerfile`) with Grype in the + `grype-scan` job, a surface `govulncheck` (Go modules only) cannot see. + `scripts/check-grype-policy.sh` fails the build on a *fixable* **CRITICAL or + HIGH** CVE — and only in the image's **RPM** packages. Be precise about that + restriction, because it is a real, deliberate limit: Grype also catalogs the + Python `dist-info` that Fedora RPMs ship as independent PyPI artifacts, and + those records use upstream versions and advisories, so one can claim a fix + exists when Fedora has already backported it or has not published an RPM + update. Those language records are **uploaded to SARIF but do not gate**; + treating them as a merge gate previously produced hand-maintained pip + replacements layered over a coherent distro package set. MEDIUM and below are + reported, not blocking. Findings upload to GitHub Security → Code scanning + (category `grype-sandbox-image`), and a weekly scheduled scan + (`.github/workflows/grype-scheduled.yml`) catches newly-disclosed CVEs against + the existing image between PRs. + + Scope note: `grype-scan` lives only in `ci.yml`, so it runs on main-targeting, + non-docs-only events. PRs into `dev` get no image scan; the image is scanned at + the dev→main promotion and weekly. +- **Release cooldown — on the two ecosystems that support it.** + `.github/dependabot.yml` applies a `cooldown` to the gomod and npm surfaces so + Dependabot waits a few days (3 for patch, 7 for minor, 14 for major) before + proposing a freshly published release. This blunts fast typosquat / + account-takeover attacks, where a malicious version is published and then yanked + once the ecosystem flags it. It matters most for **patch** bumps, which + `.github/workflows/auto-merge-dependabot.yml` auto-merges once the CI gate is + green: without a cooldown a minutes-old patch could be proposed and auto-merged + before any scrutiny. Cooldown applies to version updates only — Dependabot + **security** updates are never delayed, so urgent CVE fixes still flow + immediately. + + **The `github-actions` ecosystem is the exception, and it is the one where a + cooldown would matter most.** Dependabot supports `cooldown` for gomod and npm + only, so the one ecosystem whose "dependency" is *the CI definition itself* — + a `github-actions` bump rewrites `.github/workflows/*` and therefore changes + what CI executes — cannot be made to wait, and it is configured daily against + `dev`. Because `dev` additionally has no required status checks (see "Static + analysis" above), that combination is not something to auto-merge, so + `auto-merge-dependabot.yml` **excludes `github_actions` at any bump level** and + those PRs take a human. The workflow also carries an explicit + `branches: [main, dev]` filter, so it can never silently begin applying to some + other branch, and declares its write scopes on the job rather than the workflow. The cooldown reduces the window for a fast attack but is **not** a guarantee: a patient attacker who waits out the cooldown, or a compromise the ecosystem -never flags, would still slip through. The committed `go.sum` + checksum DB and -`govulncheck` are the stronger, always-on controls; the cooldown is -defense-in-depth on top of the auto-merge path. +never flags, would still slip through. The committed `go.sum` + checksum DB, +`govulncheck` and `npm audit` are the stronger, always-on controls; the cooldown +is defense-in-depth on top of the auto-merge path, and it does not cover +`github-actions` at all. ## CSRF protection (cookie-authenticated routes) diff --git a/docs/BENTO-PDF-EXPORT.md b/docs/BENTO-PDF-EXPORT.md index 03349e48..f44db03c 100644 --- a/docs/BENTO-PDF-EXPORT.md +++ b/docs/BENTO-PDF-EXPORT.md @@ -34,9 +34,11 @@ rejected: - **~400MB** added to an image built on `fedora-minimal`. - **The Grype gate.** `scripts/check-grype-policy.sh` fails CI on any fixable - CRITICAL Fedora RPM in the sandbox image. Chromium is the most CVE-heavy RPM in - any distro, so this would become a recurring gate that blocks every merge in - the repository, not just Bento work. + CRITICAL **or HIGH** Fedora RPM in the sandbox image. Chromium is the most + CVE-heavy RPM in any distro, so this would become a recurring gate that blocks + every merge in the repository, not just Bento work — and at the HIGH threshold + the argument is stronger than it was when this was written against CRITICAL + alone. - **A driver.** `--print-to-pdf` prints the page, not the app's print DOM, so driving the real export needs CDP. There is no Node in the sandbox, so that means a hand-rolled WebSocket/CDP client — more moving parts than the renderer diff --git a/docs/CODEQL.md b/docs/CODEQL.md index 8c7cc8b1..a085d066 100644 --- a/docs/CODEQL.md +++ b/docs/CODEQL.md @@ -87,9 +87,12 @@ enough", not "unset the pin". Neither `env: GOTOOLCHAIN: auto` nor **Security queries only — at the `security-extended` tier.** The code-quality suite was enabled, measured, and deliberately removed (see "Why code quality -was dropped" below); the *security* side was then widened from the default -suite to `security-extended` once the default measured clean, so the broader -set also started from a zero baseline. +was dropped" below); the *security* side was then widened from the default suite +to `security-extended`. The widening was done on the belief that the default +suite had "measured clean" and that the broader set therefore also started from a +zero baseline. **That belief was an artifact of measuring on a PR run** — see +"The threshold, and the measurement that was misread" below. The suite choice +still stands; the baseline claim did not. Adopting the extended suite was a measurement, and it produced exactly **one finding across all four languages**: `actions/untrusted-checkout/medium` on the @@ -108,10 +111,12 @@ that is strictly more dangerous (it holds `packages: write`) but escaped the query because its ref plumbing was named differently. A heuristic query's silence is not evidence of safety; the flagged file just pointed at the class. -The extended suite then verified **clean in CI on all four languages** — Dev CI -run 525 (`32580031374`), the same run that exercises the hardened `actions` -lane — so the fail-on-findings gate holds at the extended tier, not just the -default one. +The extended suite then reported **no findings in CI on all four languages** — +Dev CI run 525 (`32580031374`), the same run that exercises the hardened +`actions` lane. Read that sentence narrowly: run 525 was a `pull_request` event, +so what it establishes is that the extended suite found nothing **inside that +PR's diff**. The tree-wide numbers came later and were not zero. The full account +is in "The threshold, and the measurement that was misread". `build-mode: none` is [not supported for Go](https://docs.github.com/en/code-security/reference/code-scanning/codeql/build-options-for-compiled-languages) @@ -140,27 +145,44 @@ $ go test -count=1 -run TestWorkflowsDeclareVersionsByFile ./scripts ### Triggers +`codeql.yml` has **no `push` and no `pull_request` trigger of its own.** It is a +reusable workflow: + ```yaml -push: branches: [main] -pull_request: branches: [main, dev] -schedule: - cron: '0 10 * * 1' +on: + workflow_call: # ci.yml (main) and dev-ci.yml (dev) each call it as a job + workflow_dispatch: # manual re-run + schedule: + - cron: '0 10 * * 1' # Monday 10:00 UTC ``` -`push` on `main` mirrors `ci.yml` and produces the alert set of record for the -default branch. `pull_request` on `main` matches what default setup covered. - -`dev` on `pull_request` is the **one place this covers more than default setup -did**, and the expansion is deliberate. Every change lands on `dev` first; `main` -only ever receives a promote merge. Scanning `main` alone means a finding -surfaces for the first time on a promote commit — the same complaint -`dev-ci.yml`'s own header already makes about compilation ("a branch whose job is -to integrate should not be where compilation is first attempted"). It is also -what made this change provable before merge: with main-only triggers, the first -real run of a workflow written to fix a silent-failure bug would have happened -*after* it merged. - -There is no `push` trigger on `dev`: a push to `dev` is the merge of a PR that -was just scanned, so it would re-analyze identical content. +Per-change runs therefore arrive through `workflow_call`: `ci.yml` fires on +push/PR against `main`, `dev-ci.yml` on push/PR against `dev`, and each calls +this workflow as a job. Every branch event is covered exactly once, and — because +a called workflow's jobs land in the *caller's* graph — the result feeds +`CI gate` / `Dev gate` directly. Scanning `dev` PRs at all is the one place this +exceeds old default setup, which never ran on them. + +**An earlier revision of this section was wrong in a way worth preserving, +because the error is instructive.** It described a `push: [main]` / +`pull_request: [main, dev]` trigger set and justified omitting `push` on `dev` +like this: *"a push to `dev` is the merge of a PR that was just scanned, so it +would re-analyze identical content."* + +Both halves are false, and the second is the exact trap that later broke `dev`. +The PR run and the push run **do not analyze identical content**: on a +`pull_request` event the CodeQL action runs **diff-informed** — it builds the +full database and evaluates every query, then reports only results whose location +falls inside the PR's diff. A push run has no diff to scope to and reports the +whole tree. So the two events differ in the most consequential way an analysis +can differ: one certifies a diff, the other certifies a tree. Treating them as +interchangeable is how an any-finding gate got armed on a zero that had never +seen the tree. See [ADR-0048](adr/0048-codeql-severity-gating.md), and "The +threshold, and the measurement that was misread" below. + +The current shape has no such hole: `dev-ci.yml` calls this workflow on **pushes +to `dev` as well as PRs into it**, so `dev` gets a full-tree verdict on every +merge, and any direct push that bypassed a PR is covered too. The weekly cron exists because a CodeQL verdict is a function of the query pack as well as the commit — new queries ship continuously, and without a schedule the @@ -253,6 +275,13 @@ until you look at what they found. **What it found: 32 findings, every one note-level, and zero security findings.** +(The "zero security findings" half of that sentence was measured on `pull_request` +runs and is therefore a statement about those diffs, not about the tree — see +"The threshold, and the measurement that was misread". It does not change the +drop decision, which rests on the 32 quality findings and where they were: those +were *reported* results, and the argument against them is that other blocking +tools already cover the same ground.) + | language | findings | what they were | | --- | --- | --- | | `go` | 2 | `go/useless-assignment-to-field` | @@ -351,26 +380,31 @@ with go1.25.1 and cannot lint the tree. ## What was NOT verified, and what is deliberately out of scope -- **No push-on-`main` or scheduled run has executed.** Both triggers are - unexercised until this merges and is promoted. They are ordinary trigger - syntax, and the `pull_request` path shares every step with them, but the cron - expression itself has not fired. It is a weekly cron, so its first real proof - is up to a week after promotion. -- **`_test.go` files are not analyzed.** 621 test files are outside the - database, because `autobuild` builds packages, not tests. Default setup did - not analyze them either, so this is not a regression — it is an unchanged - limit, stated because "CodeQL covers the Go code" would otherwise overclaim. - Bringing tests in would need `build-mode: manual`. +- **No push-on-`main` or scheduled run had executed when this section was first + written.** That is no longer true of the push path: `dev-ci.yml` calls this + workflow on pushes to `dev`, and the first such run (Dev CI run 527) is the + full-tree measurement described below — it is also what proved the section + above wrong. The **weekly cron** is still the one trigger whose own first proof + is up to a week away; the `schedule` path shares every step with the others, so + what is unexercised is the cron expression, not the analysis. +- **`_test.go` files are not analyzed.** Every `_test.go` file is outside the + database (625 in this tree; the count moves with the suite), because + `autobuild` builds packages, not tests. Default setup did not analyze them + either, so this is not a regression — it is an unchanged limit, stated because + "CodeQL covers the Go code" would otherwise overclaim. Bringing tests in would + need `build-mode: manual`. - **The lines-of-code metric value was not read.** `Summary/LinesOfCode.ql` evaluates, but CodeQL does not print the number to the job log; it lands in a `.bqrs`. File and package counts are what was actually observed, so they are what is reported here. No line count is claimed. -- **Alert counts on `main` are not claimed.** The zero-security-findings result - above was measured on a PR run of this branch. PR-run file-coverage detail is - suppressed by CodeQL ("To speed up pull request analysis, file coverage +- **Alert counts on `main` are still not claimed, and the reason turned out to + matter far more than expected.** The no-findings result above was measured on a + **PR run**, where CodeQL is diff-informed and additionally suppresses + file-coverage detail ("To speed up pull request analysis, file coverage information is only enabled when analyzing the default branch and protected - branches"), so the default-branch alert set is not established until this - merges and a promote lands on `main`. + branches"). So it never established a tree-wide baseline for `dev`, let alone + for `main`. The `dev` tree-wide numbers now exist (run 527, below); the + default-branch alert set is established when a promote lands on `main`. - **`build-mode: manual` was not built.** `autobuild` works, so the more complex option was not needed. If `autobuild` regresses, manual mode plus the repo's own `go build ./...` is the fallback — and it is also the route to @@ -380,90 +414,214 @@ with go1.25.1 and cannot lint the tree. own SARIF independently of CodeQL configuration; breaking them would silently drop CVE findings from the Security tab. +## The threshold, and the measurement that was misread + +This is the most load-bearing correction in this document, and it generalises +beyond CodeQL. The decision is recorded as +[ADR-0048](adr/0048-codeql-severity-gating.md); what follows is the short form. + +The `Fail on findings` step originally failed the job on **any finding at any +severity**, with this justification written into the workflow: + +> Threshold is ANY finding, deliberately. The security suite currently reports +> ZERO across go/python/javascript-typescript/actions, so there is no backlog to +> grandfather and no severity line to argue about — a finding here is new. + +That zero was real, and it was measured — **on a `pull_request` event** (Dev CI +run 525). On `pull_request` events the CodeQL action runs **diff-informed**: it +builds the full database and evaluates every query, then reports only results +whose location falls inside the PR's diff. Run 525's own log says both halves: + +``` +Computing PR diff ranges... +Persisted 204 diff range(s) across 43 file(s). +codeql database run-queries ... --extension-packs=codeql-action/pr-diff-range +``` +``` +To speed up pull request analysis, file coverage information is only enabled +when analyzing the default branch and protected branches. +``` + +The database held every file and the queries that later fired did run — +`LogInjection.ql`, `TaintedPath.ql`, `RequestForgery.ql` and +`WeakSensitiveDataHashing.ql` are all listed as "Interpreted" in that run. The +SARIF was empty because the results were filtered to the PR's 43 changed files. + +So the first full-tree evaluation of `security-extended` against this repository +was the **push** that merged that work: **Dev CI run 527**, which reported **38 Go +and 17 javascript-typescript findings** and turned `Dev gate` red. The gate then +blocked every subsequent push to `dev` — including a push that would have fixed +it — with no PR-shaped way out, because a PR into `dev` is scanned diff-informed +and comes back green while `dev` itself stays red. + +**The rule to carry away: a PR-event CodeQL run certifies a diff, not a tree.** +Any claim of the form "the scanners are green, therefore the tree is clean" that +rests on a `pull_request` run is unsound, and that is a permanent property of +diff-informed analysis rather than a bug awaiting a fix. Tree-wide verdicts come +from push and scheduled runs. + +### What the threshold is now + +A finding **blocks** when it is not waived and either: + +- its rule publishes `security-severity >= 7.0` — CodeQL's own High/Critical cut, + and what GitHub's code-scanning merge protection bands on; or +- its rule publishes **no** security-severity at all, in which case the fallback + is the SARIF level `error` or `warning`. + +Level is deliberately **not** consulted for a rule that does publish a +security-severity. Nearly every CodeQL security query is +`@problem.severity error` — `go/log-injection` is `error` at security-severity +**6.1** — so banding on level would put all 23 log-injection findings in the +blocking tier and reproduce the deadlock. For orientation, the severities that +actually appear on this tree: + +| rule | security-severity | tier | +| --- | --- | --- | +| `go/request-forgery` | 9.1 | High band (waived per-file) | +| `go/clear-text-logging` | 7.5 | High band (waived per-file) | +| `go/path-injection` | 7.5 | High band (waived per-file) | +| `go/weak-sensitive-data-hashing` | 7.5 | High band (waived per-file) | +| `js/remote-property-injection` | 7.5 | High band (waived per-file) | +| `js/insecure-temporary-file` | 7.0 | High band (fixed, then waived) | +| `go/log-injection` | 6.1 | advisory | +| `js/client-side-request-forgery` | 5.0 | advisory | + +Everything below the band is **advisory**: printed in the job log and the step +summary, uploaded to the Security tab, not blocking. + +Note what that table shows: **severity alone does not separate the true positives +from the false ones.** The 9.1 `go/request-forgery` fires on `web_fetch.go`, the +deliberate `@url` fetch tool, which dials through `internal/netguard`'s +resolve-then-dial SSRF guard. The 7.5 `go/weak-sensitive-data-hashing` fires on +SHA-256 used as a lookup index over a 32-byte `crypto/rand` bearer token — the +recommended construction. A pure severity line would block both. That is why the +band comes with a register rather than instead of one. + +### The register, and why it is per-file + +`.github/codeql-accepted-findings.json` lists accepted `(rule, file)` pairs, each +with a **mandatory written reason** that must say why the finding cannot be +exploited *there* — not that the rule is noisy. + +Per-**file** is the whole point of preferring it to a `query-filters` exclude. An +`exclude: {id: go/request-forgery}` switches a security-severity 9.1 query off +for the entire repository; a register entry waives it in +`internal/tools/web_fetch.go` and `internal/mcpoauth/discovery.go` and leaves the +query live everywhere else, including elsewhere in those same packages. An +in-source `// codeql[rule-id]` comment is the second waiver route — CodeQL emits +it as a `suppressions` array on the result; the comment must sit on its own line +and covers the line immediately below it. + +Of the 55 findings run 527 surfaced, **four were reachable and were fixed in +code**: an unsanitized `task.Prompt` in the task-create log (its update-path twin +was already wrapped in `logSafe`), the raw pre-validation client attachment path +logged on the two branches where containment had just failed, the client-echoed +attachment `Name` on the `/chat` path, and an Ed25519 private key written to a +predictable world-writable temp path at `0644` in `web/e2e/test-auth-key.ts`. The +other 51 are in the register. + +### One classifier, three tiers, and it fails closed + +`.github/codeql-gate.jq` does the banding and the waiver lookup, and **both** the +summary step and the gate step run it through `jq -f`. Two copies of a SARIF +filter is two copies that can disagree about what "blocking" means, and the +report disagreeing with the gate is worse than either being wrong alone. + +The job log prints three tiers from that single classification — **BLOCKING**, +**ACCEPTED** (by name, because a waiver that is invisible in CI output is a waiver +nobody re-reads) and **ADVISORY**. + +It fails the job rather than reporting clean when: the register is missing, the +filter file is missing, the SARIF will not parse, or — the subtle one — findings +exist but **zero rule metadata resolved**. That last is a vacuity check with real +provenance: CodeQL writes query metadata into `tool.extensions[].rules[]`, not +`tool.driver.rules[]`. A first cut of the filter read only the driver, resolved +nothing, scored every finding at security-severity 0 — `go/request-forgery`'s 9.1 +included — and reported "0 blocking" over a tree that was not clean. That is the +green-but-vacuous outcome this entire workflow exists to rule out, so it is now +an explicit failure mode rather than a silent one. + +Three anti-rot controls sit on top: `scripts/check_codeql_register_test.go` (in +`make test`) requires every entry to name a file that exists, carry a substantive +reason, use a plausible rule id, and be unique — and asserts that `codeql.yml` +still references the register at all; the weekly scheduled scan surfaces entries +that no longer match any finding, so a stale waiver gets removed rather than +quietly widening coverage loss; and widening the register shows up in a PR diff +where the reviewer is expected to check the reason against the code. + ## Two different things can gate, and they are not the same lever -This distinction is the one most worth internalizing, because a status check on -the CodeQL job does **not** gate on findings: +This distinction is still worth internalizing, because the two mechanisms answer +different questions: | you want to block a merge when… | the mechanism | where it lives | | --- | --- | --- | -| the analysis **failed or did not run** | a required status check on `CodeQL gate` | branch protection / ruleset | -| CodeQL **found alerts** at/above a severity | **code scanning merge protection** | ruleset → "Code scanning" rule | - -The second is the one people mean by "gate on CodeQL", and the first does not -give it to you. **A CodeQL job with a hundred open alerts still exits 0 and -reports green** — the job's success only says extraction and query evaluation -worked. That is exactly why the toolchain break was able to hide for weeks behind -a red-but-not-required check, and equally why a green check is not evidence of a -clean codebase. - -fleet is a **public** repository, so code scanning merge protection is available -at no cost (on private repos it requires GitHub Advanced Security). To turn it -on: Settings → Rules → the "Main" ruleset → add the **Code scanning** rule → -add tool **CodeQL** → set the alert thresholds. Two independent knobs there: -*Security alerts* (the CWE/security queries — the only ones this workflow runs) -and *Alerts* (everything else, which would be where a code-quality suite landed -if one were enabled; it is not). Since the security suite currently reports zero -findings on this tree, a **High or higher** security threshold can go on without -inheriting a backlog. - -## Merge gating today — unchanged, with the lever put within reach - -**A finding now turns the check red.** A `Fail on findings` step fails the job on -any finding, at a threshold of *any* — safe to set because the security suite -currently reports zero on this tree, so there is no backlog to grandfather. -Without that step the analyze step exits 0 whether it found nothing or a hundred -alerts, so a red check could only ever mean "the scanner broke" — which is -exactly how the toolchain break hid for weeks. - -**A red check now blocks the merge too**, and through the *existing* required -check rather than a new one: `codeql.yml` is a reusable workflow -(`on: workflow_call`) that `ci.yml` and `dev-ci.yml` call as a job, and that -calling job sits in `ci-gate`'s / `Dev gate`'s `needs`. A correction worth -keeping: an earlier revision claimed this half needed a repo-settings click, -reasoning from "`needs` cannot cross workflow files" — true, but a -`workflow_call` brings the jobs into the caller's file, which is the standard -mechanism and what ships. - -It *cannot* be folded into `ci-gate`: a job's `needs` cannot reach across -workflow files. So `codeql.yml` carries its own aggregate **`CodeQL gate`** job, -mirroring `ci.yml`'s `CI gate` and `dev-ci.yml`'s `Dev gate`. That job is the one -deliberate piece of forward work here, and it is worth being clear that it -changes nothing on its own: - -- It does **not** make CodeQL required. Requiring a check is a repo-settings - action, deliberately not expressible from a workflow file. -- What it buys is that **flipping the switch later is one check, not four.** - Naming `Analyze (go)`, `Analyze (python)`, `Analyze (javascript-typescript)` - and `Analyze (actions)` individually in branch protection would mean - re-pointing branch protection by hand every time the matrix gains or loses a - language — and the failure mode of getting that wrong is the dangerous - direction: a required check that never reports again blocks every PR, or a - removed one silently stops gating. One aggregate check has neither problem. - -No ruleset action is required for any of this: the gate wiring above is entirely -in the workflow files. (`CodeQL gate` still exists as the aggregate job — the -weekly scheduled run's single verdict — and could additionally be named in the -ruleset as belt-and-braces, but nothing depends on that.) - -**On sequencing:** Not out of -caution for its own sake — because of this specific incident. The analysis spent -weeks red for a toolchain reason unrelated to any diff, and a required check in -that state blocks *every* merge, including the promote PR that would carry the -fix. Requiring it also means a `dev`-PR CodeQL failure blocks `dev`, a heavier -posture than that lane's stated "does it compile, lint, and pass tests" job. The -sequence with the least chance of self-inflicted deadlock is: merge this, watch a -few promotions go green, then add `CodeQL gate` to the ruleset. - -What makes that sequencing *safer than it was*: with code quality dropped, the -security suite is all that runs, and it currently reports **zero findings** on -this tree (see "Why code quality was dropped"). So there is no pre-existing -backlog for a required gate to trip over — which is the usual reason turning one -on hurts. The remaining risk is the one this whole document is about: a toolchain -or extractor regression going red for reasons unrelated to any diff. - -No repo-settings or API change to code-scanning configuration was attempted as -part of this change. +| the analysis **failed, did not run, or found something in the blocking band** | the `Fail on findings` step, reaching a required aggregate check | this repo's workflow files | +| CodeQL **found alerts** at/above a severity, judged from the Security tab's alert set | **code scanning merge protection** | ruleset → "Code scanning" rule | + +The first row is what ships, and the wording has been corrected: an earlier +revision of this document said **"a CodeQL job with a hundred open alerts still +exits 0 and reports green"**, and that is no longer true. It was true of the +`analyze` step alone, and it is precisely why the `Fail on findings` step exists. +The step reads the run's **own SARIF** and never consults the code-scanning API, +which has one consequence worth stating plainly: **dismissing an alert in the +Security tab does not turn this check green.** The honest routes are a code +change, an in-source `// codeql[rule-id]` comment, or a register entry with a +reason. + +The second row remains available and nothing depends on it. fleet is a **public** +repository, so code scanning merge protection is free (on private repos it needs +GitHub Advanced Security): Settings → Rules → the "Main" ruleset → add the **Code +scanning** rule → add tool **CodeQL** → set the thresholds. Two independent knobs +there: *Security alerts* (the CWE/security queries — the only ones this workflow +runs) and *Alerts* (everything else, which is where a code-quality suite would +land if one were enabled; it is not). Note that a **High or higher** threshold +there would *not* start from a clean slate: the tree carries High-band findings +that are accepted in the register, and merge protection has no view of that +register. It bands on the Security tab's alert set, so those alerts would need +dismissing individually in the UI. + +## Merge gating today + +**A finding in the blocking band turns the check red.** The `Fail on findings` +step is what makes that true; the threshold is the High band described above, not +"any finding". Without the step, `analyze` exits 0 whether it found nothing or a +hundred alerts, so a red check could only ever mean "the scanner broke" — which +is exactly how the toolchain break hid for weeks. + +**A red check reaches the aggregate gate**, through the *existing* check rather +than a new one: `codeql.yml` is a reusable workflow (`on: workflow_call`) that +`ci.yml` and `dev-ci.yml` call as a job, and that calling job sits in +`ci-gate`'s / `Dev gate`'s `needs`. A correction worth keeping: an earlier +revision claimed this half needed a repo-settings click, reasoning from "`needs` +cannot cross workflow files" — true of a job's `needs`, but a `workflow_call` +brings the called jobs *into* the caller's file, which is the standard mechanism +and what ships. (An even earlier revision of this section then went on to repeat +the original error two paragraphs later. Both are corrected here.) + +**Whether a red check blocks a merge is a separate, branch-dependent fact, and on +`dev` the answer is no.** `CI gate` is a required status check on `main`, so +there the routing closes. The `dev` ruleset requires **no status checks at all** — +its only rules are `deletion` and `non_fast_forward` — so `Dev gate` is +red-but-not-required, and a CodeQL failure on `dev` is a red X beside a mergeable +PR. Adding `Dev gate` to the `dev` ruleset is a repo-settings action that no pull +request can perform; it is tracked as an open item in +[`SCANNING.md`](SCANNING.md) ("Known gaps"). + +`codeql.yml` also carries its own aggregate **`CodeQL gate`** job. In the +`workflow_call` path it is redundant — the caller's `needs: codeql` already rolls +up every matrix leg — so what it is for is the standalone schedule/dispatch runs +(one legible verdict per weekly re-scan instead of four boxes) and as a stable +single check name should anyone want to name one in a ruleset. Naming +`Analyze (go)`, `Analyze (python)`, `Analyze (javascript-typescript)` and +`Analyze (actions)` individually would mean re-pointing branch protection by hand +every time the matrix gains or loses a language, and the failure mode of getting +that wrong runs in the dangerous direction: a required check that never reports +again blocks every PR, and a removed one silently stops gating. + +No repo-settings or API change to code-scanning configuration was made as part of +this work. ## Where findings appear — and why the job log now says @@ -479,19 +637,41 @@ API. So the analyze step now also writes SARIF locally (`output:`) and a following step jq-summarizes it into both the job log and the step summary — the same thing `govulncheck-scheduled.yml` already does with its SARIF: +The counts and line numbers below are placeholders — they move with every commit. +What is fixed is the format: + ``` ### CodeQL findings — go -2 [error] go/clear-text-logging -1 [warning] go/incomplete-hostname-regexp -1 [note] go/redundant-assignment --- -total findings: 4 +BLOCKING — High band (security-severity >= 7.0), not waived (): + none + +ACCEPTED — High band, waived in codeql-accepted-findings.json or in-source (): + [error] sec-sev=9.1 go/request-forgery internal/tools/web_fetch.go: + [error] sec-sev=7.5 go/clear-text-logging cmd/fleet/main.go: + ... + +ADVISORY — below the High band; triage in the Security tab (): + [error] sec-sev=6.1 go/log-injection : + ... + +totals: finding(s) — blocking, accepted, advisory +rule metadata resolved: +files in the go database: ``` -It is reporting only and never fails the job; blocking on findings is merge -protection's job, above. When no SARIF was written it says so explicitly rather -than printing "No findings." — reporting a clean result you did not observe is -the error this repo keeps having to write down. +Three properties of that listing are deliberate. Every line carries the +finding's **`file:line`**, so an agent reading the log can go straight to the +site. The **ACCEPTED tier is printed by name**, because a waiver invisible in CI +output is a waiver nobody re-reads. And the two trailing counts are coverage +lines rather than verdicts: `rule metadata resolved` is what the vacuity check +reads, and `files in the … database` is what distinguishes "no findings" from +"analyzed nothing". + +The summary step is reporting only — a **separate** `Fail on findings` step does +the blocking, from the same classification (see "The threshold" above), so the +report and the gate cannot disagree. When no SARIF was written the step says so +and **fails**, rather than printing "No findings." — reporting a clean result you +did not observe is the error this repo keeps having to write down. `security-events: write` plus the analyze step's upload is the code-scanning ingestion path, so results also land in the repo's **Security → Code scanning**. @@ -508,9 +688,11 @@ Analysis upload status is complete. Two practical consequences worth stating, because they explain an empty-looking Security tab rather than a broken one: -- **The Security tab's alert list is the DEFAULT BRANCH's.** This workflow's only - `push` trigger is `main`, so that list refreshes when a promote merge lands on - `main` — not when a PR is scanned. +- **The Security tab's alert list is the DEFAULT BRANCH's.** This workflow has no + `push` trigger of its own at all; push-event runs reach it through `ci.yml` + (on `main`) and `dev-ci.yml` (on `dev`). So the default-branch list refreshes + when a promote merge lands on `main` — not when a PR is scanned, and not when + `dev` moves. - **PR runs report on the PR**, not into the default-branch alert list, and CodeQL additionally suppresses file-coverage detail there: *"To speed up pull request analysis, file coverage information is only enabled when analyzing the diff --git a/docs/SCANNING.md b/docs/SCANNING.md index 93064ba8..397b3275 100644 --- a/docs/SCANNING.md +++ b/docs/SCANNING.md @@ -13,15 +13,22 @@ security queries only) and [`TESTING.md`](TESTING.md) (the rest of the ladder). | `oxlint` + `tsc` | web tier lint + types | ~5s | **blocks** (`ci-gate`) | job log | | **`ruff`** | **Python lint** | **~1s** | **blocks** (`ci-gate`) | job log | | `govulncheck` | Go dependency CVEs (called symbols) | ~30s | **blocks** (`ci-gate`) | job log + Security tab | -| `grype` | sandbox image CVEs (fixable **CRITICAL + HIGH**) | ~1m | **blocks** (`ci-gate`) | job log + Security tab | +| `grype` | sandbox image CVEs (fixable **CRITICAL + HIGH**, **RPMs only**) | ~1m | **blocks** (`ci-gate`) | job log + Security tab | | `gitleaks` | secrets, every branch | ~10s | **blocks** (`ci-gate`) | job log | | **`npm audit`** | npm dependency CVEs (web + rampart-service) | ~5s | **blocks** (`ci-gate`) | job log | -| CodeQL | **interprocedural taint / `security-extended`** | ~2m | **blocks** (`ci-gate`/`Dev gate` via workflow_call) | job log + Security tab | -| **Semgrep** | **Go/JS/Python SAST + Actions supply chain** | ~40s | **blocks** (`ci-gate`/`Dev gate` via workflow_call) | job log + artifact | +| CodeQL | **interprocedural taint / `security-extended`** | ~2m | **blocks** on an unwaived High-band finding (`ci-gate`/`Dev gate` via workflow_call) | job log + Security tab | +| **Semgrep** | **Go/JS/Python SAST + Actions supply chain** | ~40s | **blocks** on any unsuppressed finding (`ci-gate`/`Dev gate` via workflow_call) | job log + artifact | Two things were added here (**ruff**, **Semgrep**) and one was narrowed (**CodeQL**, to security queries only). +**Read "blocks" with one caveat, and it is a big one.** Every lane above reaches +its branch's aggregate gate job — but a gate job only *blocks a merge* where it +is a **required status check**. On `main` it is (`CI gate`). On `dev` the ruleset +requires no status checks at all, so `Dev gate` is red-but-not-required there. +The "gates?" column describes the wiring, which is real; the enforcement half is +branch-dependent. See ["Known gaps"](#known-gaps-deliberately-not-closed-here). + ## Why each tool is where it is **The design rule: one owner per job.** A second tool over ground an existing @@ -62,12 +69,20 @@ violation is a regression rather than noise in a backlog: the only finding here that was arguably a latent bug. - `bento_pdf.py` — a lambda assigned to a name (`E731`), rewritten as a `def`. +One narrowing is worth naming, because the rule set is otherwise uniform across +all 13 files: `ruff.toml`'s `[lint.per-file-ignores]` waives **`F401` (unused +import) for `internal/mcp/testdata/*.py` and `cmd/fleet/testdata/*.py`**. Those +are deliberately minimal MCP stand-ins that exist to be spawned and to misbehave +in specific ways, so an import that nothing uses can be the point of the +fixture. Nothing else is waived anywhere, and the waiver is per-path and +per-rule — `F` still bites everywhere else in those directories. + `ruff format --check` is **also gated** (CI and `make lint`): the whole tree was ruff-formatted in one dedicated commit (9 files, ~3.7k lines, validated against the full Go suite — the bento/fileops golden tests exercise these scripts), so the gate started clean and a failure means one new file. -### CodeQL owns interprocedural taint (narrowed, fails on findings) +### CodeQL owns interprocedural taint (narrowed, gates on the High band) CodeQL is the only tool in this stack that does cross-function dataflow, and that is exactly the shape of fleet's headline invariants: *a credential must not reach @@ -78,21 +93,86 @@ So CodeQL keeps its security queries and gives up everything else — the qualit suite duplicated `golangci-lint`/`oxlint` for Go and JS, and ruff is a better fit for Python. Full reasoning and measurements in [`CODEQL.md`](CODEQL.md). -It runs the **`security-extended`** suite — the broader security set, adopted -after the default suite measured clean — and reports **zero findings** on this -tree (verified in CI across all four languages on Dev CI run 525), which is -what makes it safe to gate: a `Fail on findings` step now fails the job on any -finding, so a red `Analyze (…)` check means the *code* has a problem rather than -just "the scanner broke". That distinction is the whole reason the Go toolchain -break sat unnoticed for weeks. - -Getting the extended suite to zero was itself a fix, not a rubber stamp: its -one finding across all four languages was `actions/untrusted-checkout/medium` -on `build-sandbox-image.yml`'s `fleet_ref`-fed checkout. Rather than waive it -(the `actions` language has no `AlertSuppression.ql`, so there is no in-code -waiver anyway), the workflow now **refuses `refs/pull/*` refs** before checking -out — a fork-PR ref would put fork-controlled code into a workflow that runs -the checked-out build script — and the identical hardening went into +It runs the **`security-extended`** suite — the broader security set — and a +`Fail on findings` step fails the job on an unwaived finding in the **High band** +(`security-severity >= 7.0`; for a rule that publishes no security-severity, the +fallback is SARIF level `error`/`warning`). So a red `Analyze (…)` check means +the *code* has a problem rather than just "the scanner broke", which is the +distinction the Go toolchain break survived weeks inside of. Findings below the +band are **advisory**: printed in the job log and step summary, uploaded to the +Security tab, not blocking. The threshold and the reasoning behind it are +[ADR-0048](adr/0048-codeql-severity-gating.md). + +**The threshold used to be "any finding", and correcting that is the most +instructive thing in this document.** The any-finding gate was armed on a +measured zero — Dev CI run 525, across all four languages. That run was a +`pull_request` event, and on `pull_request` events the CodeQL action runs +**diff-informed**: it builds the full database and evaluates every query, then +reports only results whose location falls inside the PR's diff. Run 525's own +log says both halves out loud — `Persisted 204 diff range(s) across 43 file(s)` +and `file coverage information is only enabled when analyzing the default branch +and protected branches`. The zero measured the **diff**, not the tree. + +The first full-tree evaluation was therefore the **push** that merged that work: +Dev CI run 527, which reported **38 Go and 17 javascript-typescript findings** and +turned `Dev gate` red — with no PR-shaped way out, because a PR into `dev` is +scanned diff-informed and comes back green while `dev` itself stays red. + +The generalisable rule, worth carrying to any scanner that supports diff-scoped +analysis: **a PR-event CodeQL run certifies a diff, not a tree.** Any claim of +the form "the scanners are green, therefore the tree is clean" that rests on a +`pull_request` run is unsound. Tree-wide verdicts come from the push and +scheduled runs. + +Of those 55, **four were reachable and were fixed in code, not waived**: an +unsanitized `task.Prompt` in the task-create log (its update-path twin was +already wrapped in `logSafe`), the raw pre-validation client attachment path +logged on the two branches where the containment guard had just *failed*, the +client-echoed attachment `Name` on the `/chat` path, and an Ed25519 private key +written to a predictable world-writable temp path at `0644` in +`web/e2e/test-auth-key.ts`. The remaining 51 are false positives in fleet's +threat model, and severity alone does not separate them — `go/request-forgery` +is 9.1 and fires on the deliberate `@url` fetch tool behind `internal/netguard`'s +resolve-then-dial SSRF guard; `go/weak-sensitive-data-hashing` is 7.5 and fires +on SHA-256 used as a lookup index over a 32-byte `crypto/rand` token, which is +the recommended construction. + +Those 51 live in **`.github/codeql-accepted-findings.json`**, a register of +accepted `(rule, file)` pairs each carrying a mandatory written reason. It is +per-**file**, not per-rule, and that is the whole point of preferring it to a +`query-filters` exclude: excluding `go/request-forgery` would switch a +security-severity 9.1 query off for the entire repository, whereas a register +entry waives it in the two files that were read and leaves the query live +everywhere else. An in-source `// codeql[rule-id]` comment waives too (CodeQL +emits it as a `suppressions` array on the result; the comment must sit on its own +line and covers the line below it). Widening the register is a security decision +that appears in the PR diff, and `scripts/check_codeql_register_test.go` fails +`make test` on an entry naming a file that does not exist, a missing reason, or a +register that `codeql.yml` has stopped referencing. + +**One classifier, two consumers.** `.github/codeql-gate.jq` does the banding and +the waiver lookup, and both the summary step and the gate step run it via +`jq -f` — two copies of a SARIF filter is two copies that can disagree about +what "blocking" means, and the report disagreeing with the gate is worse than +either being wrong alone. The job log prints **three tiers** from that one +classification: BLOCKING, ACCEPTED (by name — a waiver nobody re-reads is worse +than no waiver) and ADVISORY. + +**It fails closed.** A missing register, a missing filter file, SARIF that will +not parse, and — the subtle one — findings present with **zero rule metadata +resolved** all fail the job rather than reporting clean. That last is a vacuity +check with a real provenance: CodeQL puts query metadata in +`tool.extensions[].rules[]`, not `tool.driver.rules[]`, and a first cut of the +filter read only the driver, resolved nothing, scored every finding at +security-severity 0 and reported "0 blocking" over a tree holding findings. + +Getting the extended suite adopted was itself a fix, not a rubber stamp: the one +`actions`-language finding was `actions/untrusted-checkout/medium` on +`build-sandbox-image.yml`'s `fleet_ref`-fed checkout. Rather than waive it (the +`actions` language has no `AlertSuppression.ql`, so there is no in-code waiver +anyway), the workflow now **refuses `refs/pull/*` refs** before checking out — a +fork-PR ref would put fork-controlled code into a workflow that runs the +checked-out build script — and the identical hardening went into `publish-sandbox-image.yml`, the *unflagged* twin that holds `packages: write` and only escaped the (name-heuristic) query because its plumbing was named differently. Details in [`CODEQL.md`](CODEQL.md). @@ -112,8 +192,9 @@ fixing every real finding and adjudicating every false one. `p/github-actions` found one issue class nothing else in this repo checks — actions referenced by a **mutable tag** (`actions/checkout@v7`) instead of an immutable commit SHA. If a tag moves, attacker-controlled code runs with this -repo's `GITHUB_TOKEN`. Every one of the **53** action references across all 12 -workflows is now pinned: +repo's `GITHUB_TOKEN`. There are **13** workflow files, **12** of which reference +an action at all (`scan-cron-alarm.yml` has no `uses:`), and every one of the +**53** third-party action references across them is now pinned: ```yaml uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -125,6 +206,18 @@ not smuggle in a version bump. The trailing version comment is also the form Dependabot reads and updates, and `.github/dependabot.yml` already watches the `github-actions` ecosystem, so these stay current without hand-editing. +**Two of those pins were not what they looked like, and the failure is silent.** +`git ls-remote` on a repository that publishes *annotated* tags returns the tag +**object's** SHA for `refs/tags/v4`, not the commit it points at — for +`github/codeql-action` that is `4c0873ef…` for `refs/tags/v4` and `db488dde…` +for `refs/tags/v4^{}`. A pin taken from the unpeeled form is a 40-hex string +that looks exactly like a commit pin, satisfies every "is it a SHA" check, and +resolves to a **mutable major tag** — the precise defect the pinning exercise +existed to remove. Two distinct pins were in that state, across 7 usages; both +are now the peeled commit with an exact `# vX.Y.Z` comment, and +`scripts/check_action_pins_test.go` asserts the shape so the next pin cannot be +taken from the wrong ref. + Two `uses:` lines are deliberately left on `@main`: both are inside **comments** in `build-sandbox-image.yml` / `publish-sandbox-image.yml`, documenting how a downstream bundle repo calls fleet's reusable workflows. `@main` is the right @@ -202,19 +295,41 @@ wrong; exit 127 on the first CI run taught that one.) Both scanners print a per-rule summary into the job log **and** the step summary: +The shape of it — the counts and line numbers below are placeholders, since they +move with every commit; what is fixed is the format: + ``` -### CodeQL findings — actions -[warning] actions/untrusted-checkout/medium .github/workflows/build-sandbox-image.yml:106 --- -total findings: 1 -files in the actions database: 13 +### CodeQL findings — go +BLOCKING — High band (security-severity >= 7.0), not waived (): + none +ACCEPTED — High band, waived in codeql-accepted-findings.json or in-source (): + [error] sec-sev=9.1 go/request-forgery internal/tools/web_fetch.go: + [error] sec-sev=7.5 go/clear-text-logging cmd/fleet/main.go: + ... +ADVISORY — below the High band; triage in the Security tab (): + [error] sec-sev=6.1 go/log-injection : + ... + +totals: finding(s) — blocking, accepted, advisory +rule metadata resolved: +files in the go database: ``` -Each line carries the **`file:line`** of the finding — an agent reading the log -can go straight to the site — and the `files in the … database` count is the -coverage line: "No findings." over an empty database is the green-but-vacuous -outcome this workflow exists to rule out, and the two are indistinguishable -without it. +Three properties of that listing are deliberate. Each line carries the +**`file:line`** of the finding, so an agent reading the log can go straight to +the site. The **ACCEPTED tier is printed by name**, because a waiver that is +invisible in CI output is a waiver nobody re-reads. And the two trailing counts +are the coverage lines: `rule metadata resolved` is what the gate's vacuity check +reads (findings present with zero metadata resolved fails the job), and +`files in the … database` distinguishes "no findings" from "analyzed nothing" — +the green-but-vacuous outcome this whole stack exists to rule out. + +For the one measurement that is worth quoting rather than illustrating: the Go +database holds **426** of the tree's 427 non-test `.go` files, the missing one +being `host_disabled.go` — `host.go` and `host_disabled.go` carry mutually +exclusive build tags, and the `fleet_host_executor` tag passed to autobuild +deliberately trades which of the two is analyzed in favour of the real +unsandboxed-execution logic. See [`CODEQL.md`](CODEQL.md). This exists because a CodeQL run reports **nothing** about what it found to its own log — it writes SARIF, uploads it, exits 0, findings or not. Verified by @@ -236,13 +351,16 @@ paren choked the bash sub-parser (hoisted to a plain assignment), and an inline `import("@playwright/test")` type in `fixtures.ts` (a named `import type`, validated by `tsc`). -## What gates — everything, through the gates that already exist +## What gates — the wiring, and where enforcement actually lands Every lane in the table reaches the branch's aggregate gate: - `ci-gate` (the single required status check on `main`) `needs` the lint, test and build jobs — **and the two scanners**. -- `Dev gate` does the same on `dev`. +- `Dev gate` `needs` the same set on `dev` — but nothing in the `dev` ruleset + requires `Dev gate` to be green, so on that branch it is a red check rather + than a closed gate. That gap is the first item under "Known gaps" and it is + the single most important qualifier on this whole document. The scanners get there because `codeql.yml` and `semgrep.yml` are **reusable workflows** (`on: workflow_call`): `ci.yml` and `dev-ci.yml` each call them as a @@ -258,13 +376,29 @@ ships now. The scanners' own `push`/`pull_request` triggers were removed so nothing runs twice; each keeps its weekly `schedule` (new queries/rules against unchanged code) and a `workflow_dispatch`. -**Both scanners fail their job on any finding.** That is what makes a green -check mean "clean tree" rather than "the scanner ran" — the analyze step alone -exits 0 whether it found nothing or a hundred alerts, which is how the Go -toolchain break survived weeks behind a red-but-not-required check. Failing on -*any* finding is only defensible because the tree is at zero unsuppressed -findings everywhere — verified before the switch was flipped. A gate turned on -over an existing backlog is a gate people route around. +**Both scanners fail their job on a finding, but not on the same threshold, and +the difference is deliberate.** + +- **Semgrep: any unsuppressed finding.** `--error`, no `continue-on-error`. That + is defensible because the tree is at zero unsuppressed findings across all four + packs, with the 6 false positives waived at the line and mutation-tested. +- **CodeQL: an unwaived finding in the High band** (`security-severity >= 7.0`, + or level `error`/`warning` for a rule that publishes no security-severity), + with the accepted-findings register applied. Below the band is advisory. It was + "any finding" for exactly one merge, and [ADR-0048](adr/0048-codeql-severity-gating.md) + records why that could not hold: nearly every CodeQL security query is + `@problem.severity error` — `go/log-injection` is `error` at security-severity + 6.1 — so banding on level would block on all 23 log-injection findings, and the + zero the any-finding gate was armed on came from a diff-informed PR run. + +What both thresholds buy is the same thing: a green check that means something +about the *code*, not just that the scanner ran. The analyze step alone exits 0 +whether it found nothing or a hundred alerts, which is how the Go toolchain break +survived weeks behind a red-but-not-required check. + +What neither buys is enforcement on a branch whose ruleset requires no checks. +Wiring and enforcement are two different levers, and only one of them lives in +this repo. (Code scanning merge protection — the ruleset's alert-severity rule — remains available on top as a belt-and-braces option, but nothing depends on it now.) @@ -273,8 +407,59 @@ available on top as a belt-and-braces option, but nothing depends on it now.) Stated rather than left for rediscovery: -- **`_test.go` files are outside CodeQL's database** (621 files) — `autobuild` - builds packages, not tests. Unchanged from default setup. +- **Nothing in `dev-ci.yml` is a required check on `dev`, so every job in it — + CodeQL and Semgrep included — is red-but-not-required there.** This is the + largest gap on the page and it cannot be closed from a pull request, so it is + written down rather than implied away. + + The `dev` ruleset's only rules are `deletion` and `non_fast_forward`. There is + no `pull_request` rule and no `required_status_checks` block, so there is no + status check for GitHub to hold a merge on. `main` is the branch that does + require one (`CI gate`). Every sentence in this document about a scanner + "blocking" describes wiring that is genuinely in place — the `workflow_call` + jobs really do sit in `Dev gate`'s `needs` — and on `dev` that wiring produces + a red X beside a mergeable PR. + + Two things compound it, and together they are the actual risk: + + 1. `.github/dependabot.yml` points the `github-actions` ecosystem at `dev` on a + **daily** interval with **no `cooldown`** — Dependabot supports `cooldown` + for `gomod` and `npm` only, so the one ecosystem whose "dependency" is the + CI definition itself is also the one that cannot be made to wait. + 2. A `github-actions` bump **is a rewrite of `.github/workflows/*`**: it + changes what CI executes. + + So the pre-existing shape was: a same-day patch bump to a third-party action, + auto-merged into a branch with no required checks, rewriting the workflows that + are supposed to check it. Three workflow-side mitigations ship alongside this + document — `auto-merge-dependabot.yml` now **excludes the `github_actions` + ecosystem** whatever the bump level, carries an explicit + `branches: [main, dev]` filter so it can never silently start applying to an + unprotected branch, and declares its write scopes on the job rather than the + workflow. Those narrow the blast radius; they do not make `Dev gate` required. + + **The remaining fix is a repo-settings action and belongs to the owner:** add + `Dev gate` to the `dev` ruleset's required status checks. Nothing in a workflow + file can make itself required, so no PR can close this item. + +- **`_test.go` files are outside CodeQL's database** (625 files in this tree — + the count moves with the suite) — `autobuild` builds packages, not tests. + Unchanged from default setup, and stated because "CodeQL covers the Go code" + would otherwise overclaim. +- **The accepted-findings register keys on `(rule, file)`, not + `(rule, file, line)`.** Deliberate: line numbers churn on every edit, and a + register that fails on unrelated refactors is a register people delete. The + cost is that a *second*, genuinely bad instance of an already-waived rule in an + already-waived file would not block. That is the sharpest edge in the CodeQL + gate, and it is why each reason string names the specific call sites and the + guard that makes them safe. See [ADR-0048](adr/0048-codeql-severity-gating.md). +- **A note-level regression no longer fails the build.** A 24th + `go/log-injection` sink on genuinely untrusted input would appear in the + advisory tier and the Security tab, not in a red check. The class is not + unguarded — `gosec`'s G706 covers it inside `golangci-lint`, which *does* block + through `ci-gate`, and carries a reviewed `//nolint:gosec` annotation at each + of the ~80 sites where it was adjudicated — but the CodeQL lane is not what + would stop it. - **Semgrep's rule packs are registry-fetched and cannot be pinned by vendoring** — investigated and rejected on license grounds, not neglect. The Semgrep Rules License v1.0 grants use for "your own internal business diff --git a/docs/TESTING.md b/docs/TESTING.md index dca9c52e..83336744 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -9,13 +9,23 @@ delegate to the same commands the workflows run, so "make it green locally" and "make CI green" are the same act. The source of truth is, and remains, the workflow files themselves: -- [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) — the PR gates - (every job must be green to merge). +- [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) — the full gate on + `main` (every job must be green to merge; `CI gate` is the required check). +- [`.github/workflows/dev-ci.yml`](../.github/workflows/dev-ci.yml) — the fast + lane on `dev`. Same shape, fewer lanes — and its aggregate `Dev gate` is **not** + a required check, see "Which lanes run where" below. +- [`.github/workflows/codeql.yml`](../.github/workflows/codeql.yml) and + [`.github/workflows/semgrep.yml`](../.github/workflows/semgrep.yml) — the two + SAST lanes. Both are **reusable** workflows (`on: workflow_call`) with no + push/PR triggers of their own: `ci.yml` and `dev-ci.yml` call them as jobs, so + they land in the caller's gate. Each also keeps a weekly `schedule`. - [`.github/workflows/e2e-canary.yml`](../.github/workflows/e2e-canary.yml) — the nightly real-model canary (never a PR gate). - [`.github/workflows/grype-scheduled.yml`](../.github/workflows/grype-scheduled.yml) - — a weekly, non-blocking container-image vulnerability scan (never a PR - gate). + and + [`.github/workflows/govulncheck-scheduled.yml`](../.github/workflows/govulncheck-scheduled.yml) + — scheduled, non-blocking re-scans of unchanged code (never PR gates), because + a CVE/advisory verdict is a function of the clock as well as the commit. If a command here ever disagrees with those files, the workflow wins — please fix this doc (and the `make` targets) to match. @@ -27,13 +37,17 @@ fix this doc (and the `make` targets) to match. | Secret scan | `gitleaks` | No secrets committed | `gitleaks dir . --redact --exit-code 1` | | Go build | `go` | Release binary compiles (host executor fenced out) | `make compile` | | Go vet | `go` | `go vet` clean (tagged) | part of `make ci-go` | -| Go lint | `go` | `golangci-lint` full gate (zero findings) | `make lint` | +| Go lint | `go` | `golangci-lint` full gate (zero findings) | `make lint-go` | +| Python lint | `python` | `ruff check` **and** `ruff format --check` over the 13 Python files | `make lint-python` | | Go test | `go` | Unit + integration suites + coverage profile (needs Postgres) | `make test` | | Go coverage | `go` | Coverage profile summarised in the log + job summary (advisory, no threshold) | `make test-cover` | | Go test -race | `go` | Race detector on the same suites | `make test-race` | | govulncheck | `go` | Dependency CVEs reachable from fleet | `make govulncheck` | -| Grype (image) | `grype-scan` | CVEs in the sandbox container image (fail on a fixable CRITICAL or HIGH) | see below | -| Web lint/test/build | `web` | ESLint + vitest + `next build` | `make ci-web` | +| Grype (image) | `grype-scan` | CVEs in the sandbox container image (fail on a fixable CRITICAL or HIGH **Fedora RPM**) | see below | +| CodeQL | `codeql` (called workflow) | `security-extended` taint analysis over go / python / javascript-typescript / actions; fails on an unwaived **High-band** finding | not wrapped (see [`CODEQL.md`](CODEQL.md)) | +| Semgrep | `semgrep` (called workflow) | `p/github-actions` + `p/golang` + `p/javascript` + `p/python`; fails on **any** unsuppressed finding | `semgrep scan --config …` (see [`SCANNING.md`](SCANNING.md)) | +| npm CVE audit | `web` | `npm audit --audit-level=low`, lockfile-only, over `web/` **and** `scripts/rampart-service` — fails on any severity; plus `scripts/check-npm-overrides.sh` | `npm audit --audit-level=low` in each tree | +| Web lint/test/build | `web` | oxlint + `tsc --noEmit` + vitest + `next build` | `make ci-web` | | Playwright (mocked) | `playwright` | Deterministic browser e2e, no backend | `make ci-e2e-mocked` | | Playwright (live) | `e2e-live` | Real stack + rootless-Podman sandbox, fake LLM | `npm run test:e2e:live` | | Playwright (canary) | `canary` (nightly) | Real cheap OpenRouter model, drift detection | `npm run test:e2e:canary` | @@ -53,13 +67,44 @@ divide the work like this: | | `dev-ci.yml` (fast lane) | `ci.yml` (full gate) | | --- | --- | --- | | **Fires on** | PRs into `dev`, and pushes to `dev` | PRs into `main`, and pushes to `main` — in practice, the dev→main promotion PR | -| **Runs** | Go compile / vet / lint / test (with Postgres), web lint / test / build, migration DDL lint, gitleaks | everything in the table above | -| **Skips** | `-race`, govulncheck, Grype, both Playwright suites, CodeQL | nothing | +| **Runs** | Go compile / vet / lint / test (with Postgres), Python lint (ruff check + format), **CodeQL**, **Semgrep**, web lint / typecheck / test / build **plus the npm CVE audit and the override canary**, migration DDL lint, gitleaks | everything in the table above | +| **Skips** | `-race`, govulncheck, the Grype image scan, both Playwright suites | nothing | | **Aggregate check** | `Dev gate` | `CI gate` | - -The split is "does it compile, lint, and pass tests" on `dev`; "is it safe to -ship" on the promotion. The skipped lanes are the slow ones, and none of them is -what a routine change breaks. +| **Is that aggregate a *required* check?** | **No** — see the caveat below | Yes | + +The split is "does it compile, lint, pass tests, and pass the SAST scanners" on +`dev`; "is it safe to ship" on the promotion. The skipped lanes are the slow ones, +and none of them is what a routine change breaks. + +**CodeQL and Semgrep used to be on that skipped list. They are not any more** — +both are reusable workflows that `dev-ci.yml` calls as jobs, so they sit in +`Dev gate`'s `needs` and run on every push to `dev` and every PR into it. An +earlier revision of this table said the fast lane skipped CodeQL, which was the +opposite of what shipped. + +> **The caveat that qualifies this whole section: `Dev gate` is not a required +> status check.** The `dev` ruleset's only rules are `deletion` and +> `non_fast_forward` — there is no `pull_request` rule and no +> `required_status_checks` — so every job in `dev-ci.yml`, the two scanners +> included, is *red-but-not-required* on `dev`. A failing fast lane produces a red +> X beside a mergeable PR. `main` is the branch that genuinely gates, on +> `CI gate`. Making `Dev gate` required is a repo-settings action that no pull +> request can perform; it is tracked as an open item in +> [`SCANNING.md`](SCANNING.md) ("Known gaps"). + +**One more thing worth knowing about `ci.yml`, because it decides whether the +suite runs at all:** a `changes` job classifies each push/PR as docs-only, and the +heavy jobs (`go`, `python`, `codeql`, `semgrep`, `web`, both Playwright lanes, +`grype-scan`) skip when it says yes. That classifier used to match `*.md` at any +depth plus all of `docs/*`, which swallowed compiled product content — the +`go:embed`'d `builtin_skills/*/SKILL.md` files, the shipped +`config/default/system_prompts/*.md`, and `docs/openapi.yaml` (asserted by +`cmd/fleet/openapi_drift_test.go`) — so a PR touching only a shipped system +prompt or the OpenAPI spec skipped the very tests that validate it while +`CI gate` reported green. It is now an explicit prose allow-list, and `ci-gate` +additionally **refuses to pass over a `skipped` job unless the classifier +actually said docs-only**, so a skip produced by any other cause fails the gate +instead of passing silently. > **Both triggers on the fast lane matter.** The `pull_request` trigger was added > after a period when PRs into `dev` were gated by nothing but CodeQL, which made @@ -443,14 +488,25 @@ supply chain matters as much as what it scans. The per-PR scan collects and uploads **all** findings, including unfixed and non-blocking language-package records. A separate repository-owned policy -(`scripts/check-grype-policy.sh`) fails only on a **CRITICAL Fedora RPM** with a -non-empty fix version. This distinction is intentional: Fedora RPMs sometimes -also expose Python `dist-info`, which Grype catalogs as a second PyPI artifact; -an upstream PyPI fix does not mean Fedora has published an installable RPM. The -generic image follows Fedora latest, so an actionable failure should be fixed by -rebuilding/updating the RPM rather than by layering a pip wheel over files owned -by the distro. The weekly scan uses the same complete reporting model (see -below). Narrow, reviewed suppressions live in [`.grype.yaml`](../.grype.yaml) +(`scripts/check-grype-policy.sh`) fails on a **CRITICAL *or* HIGH Fedora RPM** +with a non-empty fix version — that is, `severity in {critical, high}` **and** +`.artifact.type == "rpm"` **and** a non-empty `fix.versions`. MEDIUM and below are +reported, not blocking, and a non-RPM record never blocks whatever its severity. + +Both halves of that filter are deliberate. HIGH was added to the gate *after* +measuring rather than before: the published image at the time carried zero fixable +Critical or High RPM findings (its only fixable findings were two Medium openssh +advisories), so the tightened gate started clean instead of arming over a backlog. +And the RPM restriction is there because Fedora RPMs also ship Python +`dist-info`, which Grype catalogs as a second, independent PyPI artifact using +upstream versions and advisories — so such a record can claim a fix exists when +Fedora has already backported it or has not published an RPM update yet. Treating +those language records as a merge gate previously led to hand-maintained pip +replacements layered over a coherent distro package set. They are still uploaded +to SARIF; they just do not gate. The generic image follows Fedora latest, so an +actionable failure should be fixed by rebuilding/updating the RPM rather than by +layering a pip wheel over files owned by the distro. The weekly scan uses the same +complete reporting model (see below). Narrow, reviewed suppressions live in [`.grype.yaml`](../.grype.yaml) (one `ignore:` entry per CVE, with a rationale comment); Grype auto-reads it from the repository root. @@ -468,10 +524,15 @@ SARIF. Reproduce locally (needs podman + the Grype binary): # Build the same image the job scans, and export it to a docker-archive tarball. IMAGE_NAME=localhost/fleet-sandbox scripts/build-sandbox-image.sh latest podman save --format docker-archive -o sandbox-image.tar localhost/fleet-sandbox:latest -# Install grype first (see .github/workflows/ci.yml for the pinned version+sha), -# then scan exactly as the gate does: -grype docker-archive:sandbox-image.tar --only-fixed --fail-on critical \ - --output table --output sarif=grype-results.sarif +# Install grype first (see .github/workflows/ci.yml for the pinned version+sha). +# The CI job scans with NO --fail-on and NO --only-fixed: it reports everything, +# then hands the JSON to the policy script, which is where the gate lives. +grype docker-archive:sandbox-image.tar \ + --output table \ + --output json=grype-results.json \ + --output sarif=grype-results.sarif +# The gate itself — fixable CRITICAL/HIGH Fedora RPMs only: +scripts/check-grype-policy.sh grype-results.json ``` There is no `make` target for this lane because it boots a podman image build; diff --git a/docs/adr/0036-sandboxed-file-tools-and-host-io-exceptions.md b/docs/adr/0036-sandboxed-file-tools-and-host-io-exceptions.md index 18c68c54..ff8a7195 100644 --- a/docs/adr/0036-sandboxed-file-tools-and-host-io-exceptions.md +++ b/docs/adr/0036-sandboxed-file-tools-and-host-io-exceptions.md @@ -76,8 +76,7 @@ host-brokered credentials/network that by invariant never enter the sandbox: - **Host network / brokered fetch**: `web_fetch`, `web_search`, `tavily_search`, `smart_search`, `download_url` (HTTP fetch), - `generate_image` (provider API), `fastio_upload` / Fast.io find, - `browserbase_live_view` (#987 — one authenticated GET to a fixed public + `generate_image` (provider API), `browserbase_live_view` (#987 — one authenticated GET to a fixed public vendor host that converts a hosted browser session id into a live-view URL for a HUMAN; it drives no browser, so ADR-0044's "browser automation is a connector" stands. Registered per turn only when a credential is actually @@ -85,11 +84,10 @@ host-brokered credentials/network that by invariant never enter the sandbox: `BROWSERBASE_API_KEY`; see `docs/BROWSERBASE.md`). These use host-side credentials and the egress-proxy/allowlist posture; running them in the sandbox would either leak credentials in or lose the host broker. -- **Host workspace staging** (path-validated legacy exceptions): - `fastio_upload` reads bytes for an outbound upload; `publish_artifact` stats - a confined path and records a pointer rather than opening arbitrary content. - Neither invokes a shell, dynamic import, or template executor; all - model-selected paths pass the workspace/pathsec allowlist. This class +- **Host workspace staging** (path-validated legacy exception): + `publish_artifact` stats a confined path and records a pointer rather than + opening arbitrary content. It invokes no shell, dynamic import, or template + executor; every model-selected path passes the workspace/pathsec allowlist. This class originally also covered `download_url` (writing fetched bytes), `generate_image` (reading reference images, writing provider output), and `xlsx` (a host zip read/rewrite) — those three were migrated in #1083: they @@ -121,6 +119,41 @@ host-brokered credentials/network that by invariant never enter the sandbox: spills and agent-history overflow breadcrumbs are removed; governed recovery bytes are written only through the bound sandbox FileOp capability. +**Amended 2026-08-22 (enterprise security audit).** Three corrections to the +enumeration above, because it presents itself as exhaustive and an auditor will +read it that way: + +- **`fastio_upload` is gone from both lists.** There is no native Go + `fastio_upload` tool any more — Fast.io is an MCP server, gated by + `internal/agentcore/mcp_fastio_guard.go` and reached through the broker like + any other connector. Leaving it enumerated as a host-read exception claimed a + hole that does not exist, which is its own kind of inaccuracy. (The "Deferred" + section still names it as a migration candidate; that entry is historical.) + +- **Host `git` worktree management** was not enumerated and should have been. + `internal/scheduledrun/worktree.go` and `internal/worktree/worktree.go` run + `git worktree add/remove` and `git branch -D` on the host via + `exec.CommandContext`. This is not model-authored: the argv is + fleet-constructed, no shell is involved, and the only externally-influenced + component is `WorktreeConfig.BranchPrefix`/`BaseBranch`, both of which are now + validated as git ref-name fragments with a leading-dash refusal + (`models.WorktreeConfig.Validate`) — `BaseBranch` reaches `git worktree add` as + a trailing positional with no `--` separator, so a leading dash would + otherwise have been parsed as an option. It belongs in the control-plane class, + named rather than implicit. + +- **Admin-triggered host `podman` build/run** was not enumerated either. + `internal/rampartinstall/installer.go` shells out to `podman` with fixed + arguments behind `POST /admin/pii-redaction/install`, which is admin-gated + (`internal/httpapi/routes.go`) and not model-callable. Same reasoning: fixed + argv, no shell, operator-initiated — a control-plane operation, but one this + ADR should have listed. + +Neither addition weakens the invariant: the sandbox is still mandatory for every +agent tool call's data-plane execution, and neither of these is an agent tool. +What changes is that the enumeration is now actually complete, so "is this +exception in the ADR?" is a question with a reliable answer. + ## Consequences ADR-0002 now states the enforceable boundary precisely: general model-authored diff --git a/docs/adr/0048-codeql-severity-gating.md b/docs/adr/0048-codeql-severity-gating.md index e4866b32..ea16bd9e 100644 --- a/docs/adr/0048-codeql-severity-gating.md +++ b/docs/adr/0048-codeql-severity-gating.md @@ -139,7 +139,8 @@ not stop them; it will appear in the advisory tier and in the Security tab. This is a deliberate trade: the alternative, as demonstrated above, is a gate that blocks every push and therefore gets routed around or switched off. `gosec`'s G706 covers the same log-injection class in `golangci-lint`, which **does** block -via `ci-gate`, and carries 77 reviewed per-site annotations — so this class is +via `ci-gate`, and carries 81 reviewed per-site `//nolint:gosec // G706` +annotations at the time of writing — so this class is not unguarded, it is guarded by the instrument that was already there. **What is now load-bearing.** Widening the register is a security decision that @@ -147,7 +148,7 @@ shows up in a PR diff, and reviewers are expected to check the reason against th code rather than the reason's existence. That is a process control, and process controls decay; the tests above are what make the decay visible. -**Known limitation, stated rather than fixed.** The 621 `_test.go` files remain +**Known limitation, stated rather than fixed.** The 625 `_test.go` files remain outside the Go database (autobuild builds packages, not tests) — unchanged from default setup and from #1246. And the register keys on `(rule, file)` rather than `(rule, file, line)` deliberately: line numbers churn on every edit, and a diff --git a/internal/sched/handlers/handlers.go b/internal/sched/handlers/handlers.go index e0d60a5f..5e5c21cd 100644 --- a/internal/sched/handlers/handlers.go +++ b/internal/sched/handlers/handlers.go @@ -547,7 +547,6 @@ func (h *Handlers) CreateTask(w http.ResponseWriter, r *http.Request) { return } - //nolint:gosec // G706: untrusted fields are sanitized via logSafe (strips CR/LF); gosec's taint tracker cannot see through the helper. task.ID is a uuid.UUID. log.Printf("Task created: %s (prompt: %.50s...)", task.ID, logSafe(task.Prompt)) localizeTask(task) writeJSON(w, http.StatusOK, task) @@ -2365,7 +2364,6 @@ func (h *Handlers) CreateAPIKey(w http.ResponseWriter, r *http.Request) { } } - //nolint:gosec // G706: key.Name is unvalidated body text sanitized via logSafe (strips CR/LF), matching the sibling key handlers; key.KeyID is server-minted. log.Printf("Created API key: %s (%s)", key.KeyID, logSafe(key.Name)) resp := key.ToResponse() diff --git a/internal/sched/handlers/upload.go b/internal/sched/handlers/upload.go index 9ce7ee26..06994e21 100644 --- a/internal/sched/handlers/upload.go +++ b/internal/sched/handlers/upload.go @@ -194,11 +194,9 @@ func (h *Handlers) HandleUpload(w http.ResponseWriter, r *http.Request) { checksumPath := filepath.Join(tempDir, ".checksums", filename+".sha256") if err := os.WriteFile(checksumPath, []byte(checksum), 0600); err != nil { // Non-critical error, just log it - //nolint:gosec // G706: filename is sanitized via logSafe (strips CR/LF) and already passed sanitizeFilename; gosec's taint tracker cannot see through the helper. log.Printf("Failed to save checksum sidecar for %s: %v", logSafe(filename), err) } - //nolint:gosec // G706: filename is sanitized via logSafe (strips CR/LF) and already passed sanitizeFilename; size is an int and checksum is hex. log.Printf("File uploaded: %s (size: %d, checksum: %s)", logSafe(filename), size, checksum) writeJSON(w, http.StatusOK, map[string]interface{}{ diff --git a/ruff.toml b/ruff.toml index ed18d40f..bd87d8c1 100644 --- a/ruff.toml +++ b/ruff.toml @@ -15,9 +15,14 @@ # RULE SELECTION IS DELIBERATELY NARROW, and the numbers behind that are worth # recording so nobody widens it by accident. Measured on this tree: # -# default (E4,E7,E9,F) -> 3 findings <- what we gate on +# default (E4,E7,E9,F) -> 3 findings +# + B,SIM,S (bandit) -> 21 more, all fixed <- the gate today # E,F,W,I,UP,B,SIM,ISC,PLR,PLW,S -> 333 findings # +# What we gate on is the [lint] `select` at the bottom of this file: +# E4, E7, E9, F, B, SIM, S. The middle line is the current gate; the third line +# is the measurement that keeps the style tiers out. +# # The 330-finding difference is almost entirely style churn, not defects: # 176 UP031 (%-format vs f-string), 43 PLR2004 (magic values), 35 E501 (line # length). Gating on that would mean a 300-commit reformat for no correctness @@ -69,8 +74,13 @@ exclude = [ [lint] # The default rule set: pycodestyle errors (E4 imports, E7 statements, # E9 syntax/IO) plus Pyflakes (F — undefined names, unused imports, redefined -# names, unused locals). This is the "is it actually broken" tier; see the -# header for why the style tiers are not enabled. +# names, unused locals). This is the "is it actually broken" tier. +# +# Plus three families that were measured, fixed, and then ENABLED rather than +# left as a documented backlog (see the header): B (flake8-bugbear), SIM +# (flake8-simplify) and S (bandit — the security tier, which is why an S finding +# on a new line is a real question to answer rather than pre-existing noise). +# The style tiers (E501, UP, PLR, …) stay out; the header records why. select = ["E4", "E7", "E9", "F", "B", "SIM", "S"] [lint.per-file-ignores] diff --git a/scripts/check_gate_needs_test.go b/scripts/check_gate_needs_test.go index ec8cd525..3a681070 100644 --- a/scripts/check_gate_needs_test.go +++ b/scripts/check_gate_needs_test.go @@ -54,8 +54,9 @@ func TestAggregateGateNeedsEveryJob(t *testing.T) { } jobsBlock := text[jobsAt:] - var jobs []string - for _, m := range jobKeyRe.FindAllStringSubmatch(jobsBlock, -1) { + matches := jobKeyRe.FindAllStringSubmatch(jobsBlock, -1) + jobs := make([]string, 0, len(matches)) + for _, m := range matches { jobs = append(jobs, m[1]) } if len(jobs) < 2 { diff --git a/web/next-env.d.ts b/web/next-env.d.ts index 9edff1c7..ce4e94a6 100644 --- a/web/next-env.d.ts +++ b/web/next-env.d.ts @@ -1,6 +1,7 @@ /// /// import "./.next/types/routes.d.ts"; +import "./.next/types/root-params.d.ts"; // NOTE: This file should not be edited // see https://nextjs.org/docs/app/api-reference/config/typescript for more information. From 8e17eba7477f74a44593aa36a99be6b5c3f85bd2 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 16:42:21 +0000 Subject: [PATCH 24/34] Revert generated next-env.d.ts churn MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Next.js rewrites web/next-env.d.ts on every `next build`, and my verification run swept the added root-params reference into the docs commit. The file's own header says it should not be edited, and this branch changes nothing that would legitimately alter it — so it goes back to dev's version rather than carrying a build artifact through review. Signed-off-by: Brad Flaugher --- web/next-env.d.ts | 1 - 1 file changed, 1 deletion(-) diff --git a/web/next-env.d.ts b/web/next-env.d.ts index ce4e94a6..9edff1c7 100644 --- a/web/next-env.d.ts +++ b/web/next-env.d.ts @@ -1,7 +1,6 @@ /// /// import "./.next/types/routes.d.ts"; -import "./.next/types/root-params.d.ts"; // NOTE: This file should not be edited // see https://nextjs.org/docs/app/api-reference/config/typescript for more information. From a88dd9d9b9a70d5787d015983c23436d407a2088 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sat, 22 Aug 2026 16:45:02 +0000 Subject: [PATCH 25/34] Record why the forced final summary is exempt from the ceiling guard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit streamForceFinalSummary does not apply in.GuardStep while its sibling streamLeakedToolCallRetry does, and the asymmetry reads as an oversight — it was flagged as one during the audit. It is deliberate, so the reason now sits next to the code instead of in someone's head: the retry runs WITH tools and can buy an unbounded number of paid completions, so it must be held to the run's ceilings. The forced summary is a single tool-less completion bounded by tc.MaxTokens, it is metered via in.RecordUsage in OnStepFinish like any other call, and it only runs on the canFinish path — a run stopped by ErrCostCeilingExceeded never reaches Finalize, so it cannot be entered after a ceiling has already tripped mid-run. Worst case is one bounded, accounted completion of overshoot when a ceiling is reached on the final step, which is the price of returning a usable answer rather than a truncated one. The comment says explicitly that this stops being true if the function ever grows tools or a loop. No behavior change. Signed-off-by: Brad Flaugher --- internal/agent/interactive.go | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/internal/agent/interactive.go b/internal/agent/interactive.go index fdb1ff6d..5027d954 100644 --- a/internal/agent/interactive.go +++ b/internal/agent/interactive.go @@ -391,6 +391,23 @@ func streamLeakedToolCallRetry(ctx context.Context, tc TurnConfig, in agentcore. // here, but production never populated TurnHistory, so this recovery saw prior // turns only and fabricated from stale context — #1117. The loop's own message // slice is the single source of truth; TurnConfig no longer duplicates it.) +// NO in.GuardStep HERE, AND THAT IS DELIBERATE — unlike +// streamLeakedToolCallRetry above, which needs it. Recording the reasoning +// because the asymmetry looks like an oversight and was flagged as one during +// the enterprise security audit: +// +// - The retry runs WITH tools and can therefore buy an unbounded number of +// paid completions, so it has to be held to the run's ceilings. +// - This one is a SINGLE tool-less completion, bounded by tc.MaxTokens, and it +// is still metered (RecordUsage), so it lands in the ledger like any other +// call. It also only runs on the canFinish path: a run stopped by +// ErrCostCeilingExceeded never reaches Finalize (agentcore/run.go), so this +// cannot be reached after a ceiling has already tripped mid-run. +// +// The worst case is therefore one bounded, accounted completion of overshoot +// when a ceiling is reached on the final step — which is the price of returning +// a usable answer instead of a truncated one. If this ever grows tools or a +// loop, it needs the guard. func streamForceFinalSummary(ctx context.Context, tc TurnConfig, in agentcore.FinalizeInput) (string, error) { convo := append(append([]fantasy.Message{}, in.Messages...), fantasy.NewUserMessage(interactiveForceFinalSummaryNudge)) agent := fantasy.NewAgent(tc.Model, From a8e21b3067a8abdf9d3f6805fa5814832f216c8f Mon Sep 17 00:00:00 2001 From: Brad Flaugher <16511019+bradflaugher@users.noreply.github.com> Date: Sat, 22 Aug 2026 19:59:53 -0400 Subject: [PATCH 26/34] Kubernetes as a first-class deployment: pluggable sandbox backend + Helm chart (#989) (#1249) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Kubernetes as a first-class deployment: pluggable sandbox backend + Helm chart (#989) Ship the enterprise path in one pass per the issue plan: the fleet control plane split from execution runners, with a pluggable sandbox backend selected by FLEET_SANDBOX_BACKEND=podman|kubernetes (env over manifest sandbox.backend, mirroring sandbox.runtime; unrecognized values refuse to boot). - internal/sandbox: a third backend (k8sImpl) behind the same impl interface as podman/host — one ephemeral pod per sandbox, bash as one-shot execs over the apiserver's v4.channel.k8s.io WebSocket protocol, the python bridge as a held exec session, file ops running the same embedded fileops.py. Pods are read-only-rootfs, non-root, capabilities dropped, seccomp RuntimeDefault/Localhost, automountServiceAccountToken=false; the workspace is a shared RWX PVC mounted same-path. #796 poison-and-retire carries over (cancel/timeout deletes the pod with zero grace); orphaned pods are swept at boot like podman containers. No client-go: a minimal hand-rolled REST + WebSocket-exec client (gorilla/websocket was already in the tree; zero new modules), kubeconfig support deliberately narrow (token/client-cert; exec plugins and insecure-skip-tls-verify refused). - Fail-closed boot preflight (and fleet validate-config): apiserver + credentials, exact RBAC verbs, workspace claim, sealed-egress NetworkPolicy object, RuntimeClass when set. Podman-only knobs are refused, not ignored (FLEET_SANDBOX_RUNTIME, FLEET_SANDBOX_SECCOMP_PROFILE, allowlisted egress). - deploy/helm/fleet: single-replica control-plane Deployment (Recreate, no replica knob), runner RBAC, workspace/data PVCs, deny-all NetworkPolicy for egress=none pods + optional open-egress shaping, optional eval Postgres, optional web/Ingress. Linted + template-rendered by a new helm CI job in both gates. - Docs: docs/DEPLOYMENT-KUBERNETES.md (kind walkthrough + production checklist + honest deviations), DEPLOYMENT.md/EKS-DEPLOYMENT.md/ SANDBOX-RUNTIMES.md/TIMERS.md updates, README + AGENTS.md index rows, CHANGELOG. ADR-0049 amends ADR-0004: the single-box podman install stays the default and unchanged; only the no-k8s-artifacts enforcement clause is superseded. Closes #989 Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_0119Ygu6B16k5wBR3vj471VW Signed-off-by: Brad Flaugher * Drop postgres.password from chart values; the Secret is the only password home CodeQL (High band) flagged the empty-string password default in deploy/helm/fleet/values.yaml. The real fix, not a waiver: no password belongs in a values file at all. The eval-Postgres template now always reuses the -postgres Secret's password when one exists (an operator pre-creates it to choose their own) and generates a random one otherwise — same lookup logic as before, minus the values seam. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_0119Ygu6B16k5wBR3vj471VW Signed-off-by: Brad Flaugher * Clear the two blocking CodeQL findings on the k8s-backend PR - go/allocation-size-overflow (k8s_exec.go writeStdin): drop the manual `make([]byte, 1+n)` size arithmetic and let append size the backing array — the same shape podmanArgs already uses for the same rule. - go/command-injection (host.go runBash): accepted-findings register entry with the reason. The sink is the component's documented contract: the unsandboxed TEST/DEV-ONLY executor behind the fleet_host_executor build tag; a release build ships the fail-closed stub, so the sink cannot reach production (ADR-0002 enforcement). It surfaced on this PR only because the diff-informed run intersected the flow's path. The 12 medium log-injection findings are below the High band — advisory per ADR-0048, triaged in the Security tab, not gated. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_0119Ygu6B16k5wBR3vj471VW Signed-off-by: Brad Flaugher * Retire the EKS privileged-pod recipe; make the k8s guide the one reference docs/EKS-DEPLOYMENT.md documented running the whole single-box model — rootless Podman included — inside one privileged pod. With the first-class path landed, keeping it as a parallel track would imply support it never had (it was explicitly hand-verified, not CI-exercised). Removed; its durable content is folded into docs/DEPLOYMENT-KUBERNETES.md, which grows into the full reference: architecture diagram, both image builds (the control-plane Containerfile's FROM golang: stage is now the copy scripts/check_versions_test.go pins to go.mod, taking over the EKS doc's slot in that drift test), provider notes (EKS/GKE/AKS/bare metal), day-2 operations (CronJob equivalents of the systemd timers, upgrade story, metrics), troubleshooting, and a migration note for deployments built from the old recipe. All references repointed (DEPLOYMENT.md callout, ADR-0049, chart values, publish-sandbox-image.yml comment); CHANGELOG gains a Removed entry. Historical changelog entries are left as history. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_0119Ygu6B16k5wBR3vj471VW Signed-off-by: Brad Flaugher * k8s backend polish: dedicated runner pools, explicit pull policy, no migration prose - Sandbox pods can pin to a dedicated node pool: FLEET_SANDBOX_K8S_NODE_SELECTOR (k=v,k=v) + FLEET_SANDBOX_K8S_TOLERATIONS (JSON array), or the manifest's structured node_selector/tolerations (env wins; bundle values are canonicalized into the env string forms at boot so the pool build parses one source). Malformed values refuse to boot — a typo'd selector must not silently schedule sandboxes onto the wrong nodes. Chart values sandbox.kubernetes.nodeSelector/tolerations render to the env vars. - Sandbox pods set imagePullPolicy: IfNotPresent explicitly: the API default for a :latest tag is Always, which breaks side-loaded kind images and re-pulls a mutable tag mid-run. - Drop the old-EKS-recipe migration prose everywhere (guide, DEPLOYMENT.md callout, ADR-0049, CHANGELOG) — nobody deployed from it. - Document /metrics scraping honestly in the k8s guide: it is admin-API-key gated (X-API-Key), which stock Prometheus cannot send — use Alloy/vmagent or a header-injecting sidecar. (A ServiceMonitor was considered and deliberately NOT shipped: it would 401.) Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_0119Ygu6B16k5wBR3vj471VW Signed-off-by: Brad Flaugher * CodeQL green end-to-end: in-source suppression + cluster-text log sanitization Two changes so the Security tab, the SARIF, and CI tell one story with no manual alert-dismissal step: - host.go command-injection: the waiver moves INTO the source as a `// codeql[go/command-injection]` suppression on the sink line (running the caller's shell is this test-only executor's documented contract; the fail-closed stub ships in release builds). The Go CodeQL analysis now also runs the standard pack's AlertSuppression query so the annotation is honored end-to-end: the SARIF result carries suppressions[], the repo gate classifies it ACCEPTED, and code scanning closes the Security-tab alert — which is what turns GitHub's app-side CodeQL check green without a human dismissing anything. The register entry stays for this commit as a belt while CI proves the suppression parses; it is removed once confirmed. - go/log-injection (12 medium advisories): fixed at the source instead of waived. The kubernetes backend embeds cluster-API/pod-derived text (status messages, exec status, stderr snippets, upgrade-response bodies) into error strings that reach log.Printf sites across the codebase; a pod printing a crafted line to stderr could forge journal entries. sanitizeClusterText (newline stripping, the sanitizer shape CodeQL models) is applied at every point remote text enters an error: status errors at construction, pod phase/waiting messages, exec status and close-reason text, stderr snippets, list-returned pod names. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_0119Ygu6B16k5wBR3vj471VW Signed-off-by: Brad Flaugher * codeql: '+' prefix so the AlertSuppression pack actually runs Without the additive prefix the packs input is treated as a replacement set that the queries input then overrides — verified on the previous run: rule-metadata count stayed at 35 (no suppression query) and the gate's accepted tier showed the register waiver, not '(in-source suppression)'. The sanitization half of the previous commit did land: go/log-injection dropped from 12 findings to 9 (the error-string flows). Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_0119Ygu6B16k5wBR3vj471VW Signed-off-by: Brad Flaugher * codeql: inline config so security-extended + AlertSuppression combine Third and definitive form: the separate queries:/packs: inputs override one another (verified on this repo with and without the '+' prefix — the suppression query never ran either way); an inline config block is where a suite and an extra pack have documented combine semantics. Every language keeps security-extended; go adds AlertSuppression so the in-source // codeql[rule-id] comment on host.go's test-only executor is honored end-to-end (SARIF suppressions -> gate ACCEPTED tier -> the Security-tab alert closes). Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_0119Ygu6B16k5wBR3vj471VW Signed-off-by: Brad Flaugher * codeql: settle on the register as the one waiver mechanism (measured) In-source // codeql[rule-id] suppressions do not work with this pipeline. Three forms were tried on this PR (packs input, packs with the '+' additive prefix, inline config combining security-extended with codeql/go-queries:AlertSuppression.ql); in every case the uploaded SARIF carried no suppressions on the annotated result, the gate classified the host.go waiver from the register, and the Security-tab alert stayed open. The analyze action's interpret step is not configurable enough to change that, so: revert the workflow to plain security-extended, record the measurement in codeql.yml so nobody re-tries it blind, drop the inert suppression annotation from host.go (the explanatory comment and the register entry — the mechanism that demonstrably gates — stay), and close the deliberately-waived Security-tab alert with a one-time human dismissal, which persists across analyses. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_0119Ygu6B16k5wBR3vj471VW Signed-off-by: Brad Flaugher --------- Signed-off-by: Brad Flaugher Co-authored-by: Brad Flaugher Co-authored-by: Claude --- .github/codeql-accepted-findings.json | 5 + .github/workflows/ci.yml | 35 +- .github/workflows/codeql.yml | 22 + .github/workflows/dev-ci.yml | 31 +- .github/workflows/publish-sandbox-image.yml | 2 +- AGENTS.md | 6 + CHANGELOG.md | 75 + README.md | 9 + cmd/fleet/main.go | 141 +- cmd/fleet/validate_config.go | 111 ++ config/default/manifest.yaml | 18 + deploy/helm/fleet/Chart.yaml | 17 + deploy/helm/fleet/README.md | 47 + deploy/helm/fleet/templates/NOTES.txt | 22 + deploy/helm/fleet/templates/_helpers.tpl | 42 + deploy/helm/fleet/templates/deployment.yaml | 174 +++ deploy/helm/fleet/templates/ingress.yaml | 38 + .../helm/fleet/templates/networkpolicy.yaml | 57 + deploy/helm/fleet/templates/postgres.yaml | 119 ++ deploy/helm/fleet/templates/rbac.yaml | 70 + deploy/helm/fleet/templates/service.yaml | 18 + .../helm/fleet/templates/serviceaccount.yaml | 23 + deploy/helm/fleet/templates/storage.yaml | 50 + deploy/helm/fleet/templates/web.yaml | 67 + deploy/helm/fleet/values.yaml | 191 +++ docs/DEPLOYMENT-KUBERNETES.md | 336 +++++ docs/DEPLOYMENT.md | 18 +- docs/EKS-DEPLOYMENT.md | 1334 ----------------- docs/SANDBOX-RUNTIMES.md | 11 + docs/TIMERS.md | 4 +- .../0004-single-box-vm-native-deployment.md | 6 +- ...-kubernetes-backend-split-control-plane.md | 131 ++ docs/adr/README.md | 3 +- go.mod | 2 +- internal/agent/manager.go | 92 ++ internal/clientconfig/clientconfig.go | 76 +- internal/config/config.go | 51 +- internal/sandbox/container.go | 13 +- internal/sandbox/host.go | 9 + internal/sandbox/k8s_backend.go | 1052 +++++++++++++ internal/sandbox/k8s_backend_test.go | 562 +++++++ internal/sandbox/k8s_client.go | 493 ++++++ internal/sandbox/k8s_exec.go | 283 ++++ internal/sandbox/k8s_fake_test.go | 481 ++++++ internal/sandbox/k8s_kubeconfig.go | 227 +++ internal/sandbox/k8s_preflight.go | 99 ++ internal/sandbox/k8s_preflight_test.go | 174 +++ internal/sandbox/pool.go | 83 +- internal/sandbox/sandbox.go | 14 +- scripts/check_versions_test.go | 18 +- 50 files changed, 5557 insertions(+), 1405 deletions(-) create mode 100644 deploy/helm/fleet/Chart.yaml create mode 100644 deploy/helm/fleet/README.md create mode 100644 deploy/helm/fleet/templates/NOTES.txt create mode 100644 deploy/helm/fleet/templates/_helpers.tpl create mode 100644 deploy/helm/fleet/templates/deployment.yaml create mode 100644 deploy/helm/fleet/templates/ingress.yaml create mode 100644 deploy/helm/fleet/templates/networkpolicy.yaml create mode 100644 deploy/helm/fleet/templates/postgres.yaml create mode 100644 deploy/helm/fleet/templates/rbac.yaml create mode 100644 deploy/helm/fleet/templates/service.yaml create mode 100644 deploy/helm/fleet/templates/serviceaccount.yaml create mode 100644 deploy/helm/fleet/templates/storage.yaml create mode 100644 deploy/helm/fleet/templates/web.yaml create mode 100644 deploy/helm/fleet/values.yaml create mode 100644 docs/DEPLOYMENT-KUBERNETES.md delete mode 100644 docs/EKS-DEPLOYMENT.md create mode 100644 docs/adr/0049-kubernetes-backend-split-control-plane.md create mode 100644 internal/sandbox/k8s_backend.go create mode 100644 internal/sandbox/k8s_backend_test.go create mode 100644 internal/sandbox/k8s_client.go create mode 100644 internal/sandbox/k8s_exec.go create mode 100644 internal/sandbox/k8s_fake_test.go create mode 100644 internal/sandbox/k8s_kubeconfig.go create mode 100644 internal/sandbox/k8s_preflight.go create mode 100644 internal/sandbox/k8s_preflight_test.go diff --git a/.github/codeql-accepted-findings.json b/.github/codeql-accepted-findings.json index f2fac66d..9ed176e9 100644 --- a/.github/codeql-accepted-findings.json +++ b/.github/codeql-accepted-findings.json @@ -85,6 +85,11 @@ "rule": "js/insecure-temporary-file", "file": "web/e2e/test-auth-key.ts", "reason": "Test-only, and the reported defect is fixed as far as it can be without changing the cross-process rendezvous contract: the write is now O_EXCL (flag \"wx\") at mode 0600 with crypto random bytes in the sibling name, so it cannot follow or clobber a pre-planted symlink and does not leave the private half world-readable. The query recognizes only mkdtemp as safe, but the fixed path is a deliberate rendezvous — playwright.config.ts is loaded in the main process AND re-imported in every worker, which must all read the same throwaway keypair. The key is generated per run, protects nothing real, and is never committed." + }, + { + "rule": "go/command-injection", + "file": "internal/sandbox/host.go", + "reason": "hostImpl.runBash executing caller-supplied shell (`bash -c req.Command`) IS the component's contract: it is the unsandboxed TEST/DEV-ONLY executor, compiled solely behind the fleet_host_executor build tag (#159) — a release `go build ./...` contains the fail-closed stub in host_disabled.go and MockMode refuses to boot without the tag, so this sink cannot ship in a production binary (ADR-0002's enforcement). Every production tool call runs through the container or kubernetes backend instead. The same line already carries the equivalent gosec waiver ('shell execution is the purpose of this tool'). The finding surfaced on #1249 because the diff-informed PR run intersected the flow's path (sandbox.go/pool.go edits), not because a new source reached the sink." } ] } diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 612edae3..13db2c3d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -132,6 +132,39 @@ jobs: # allowlist in .gitleaks.toml is just local-run robustness. run: gitleaks dir . --redact --exit-code 1 + helm: + # Lint + render the fleet Helm chart (#989) so a values/template drift + # fails here, not at an operator's install. Fast (<15s) and not gated on + # the docs-only classifier — chart files are product, not prose. helm + # ships preinstalled on the ubuntu-latest runner image. + name: Helm chart lint (#989) + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Lint and render the fleet chart + run: | + set -euo pipefail + helm version --short + helm lint deploy/helm/fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci + # Minimal render (the defaults every install starts from) … + helm template ci deploy/helm/fleet --namespace fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci >/dev/null + # … and the everything-on render so every optional template compiles. + helm template ci deploy/helm/fleet --namespace fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci \ + --set postgres.enabled=true \ + --set web.enabled=true --set web.image=registry.invalid/fleet-web:ci \ + --set ingress.enabled=true --set ingress.host=fleet.example.com \ + --set networkPolicies.openEgress.create=true \ + --set 'networkPolicies.openEgress.blockedCIDRs={10.0.0.0/8}' \ + --set sandbox.kubernetes.runtimeClass=kata >/dev/null + migrations: name: Migration DDL lint (#256) runs-on: ubuntu-latest @@ -834,7 +867,7 @@ jobs: # allowed (docs-only), but any failure or cancellation fails the gate. name: CI gate if: ${{ always() }} - needs: [changes, gitleaks, migrations, go, python, codeql, semgrep, web, playwright, e2e-live, grype-scan] + needs: [changes, gitleaks, migrations, helm, go, python, codeql, semgrep, web, playwright, e2e-live, grype-scan] runs-on: ubuntu-latest steps: - name: Require all upstream jobs to have succeeded or been skipped diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 60c69232..1b4f8659 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -142,6 +142,28 @@ jobs: # 55 findings. Do not re-derive a tree-wide claim from a PR run; the # full-tree numbers come from push/schedule runs. See # docs/adr/0048-codeql-severity-gating.md. + # Go additionally runs the standard pack's AlertSuppression query so + # an in-source `// codeql[rule-id]` comment is honored end-to-end: + # the analyze step stamps the SARIF result's `suppressions` array, + # the gate below classifies it into the ACCEPTED tier, and code + # scanning closes the Security-tab alert — the one waiver mechanism + # that keeps CI, the SARIF, and the Security tab telling the same + # story (an accepted-findings.json entry, by contrast, is invisible + # to the Security tab). The pack is resolved from the bundled + # standard library — no network pull, no `packages: read`. + # NOTE on in-source `// codeql[rule-id]` suppressions: they do NOT + # work with this pipeline, and this was measured, not assumed. Three + # forms were tried on PR #1249 (the `packs:` input, `packs:` with + # the documented "+" additive prefix, and an inline `config:` block + # combining security-extended with codeql/go-queries' + # AlertSuppression.ql) — in every case the uploaded SARIF carried no + # `suppressions` on the annotated result, the gate below kept + # classifying the waiver from the register, and the Security-tab + # alert stayed open. The analyze action's interpret step is not + # configurable enough to change that, so the accepted-findings + # register stays the ONE waiver mechanism for the CI gate, and a + # deliberately-waived alert is closed in the Security tab by a + # one-time human dismissal there (which persists across analyses). - language: go build-mode: autobuild queries: security-extended diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index 027cdcbf..20c9a34b 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -255,6 +255,35 @@ jobs: # dependency bump breaking the app. run: npm run build + helm: + # Same chart lint as ci.yml (#989): lint + minimal and everything-on + # renders. helm is preinstalled on the runner image. + name: Helm chart lint + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Lint and render the fleet chart + run: | + set -euo pipefail + helm version --short + helm lint deploy/helm/fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci + helm template ci deploy/helm/fleet --namespace fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci >/dev/null + helm template ci deploy/helm/fleet --namespace fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci \ + --set postgres.enabled=true \ + --set web.enabled=true --set web.image=registry.invalid/fleet-web:ci \ + --set ingress.enabled=true --set ingress.host=fleet.example.com \ + --set networkPolicies.openEgress.create=true \ + --set 'networkPolicies.openEgress.blockedCIDRs={10.0.0.0/8}' \ + --set sandbox.kubernetes.runtimeClass=kata >/dev/null + migrations: name: Migration DDL lint runs-on: ubuntu-latest @@ -309,7 +338,7 @@ jobs: # Aggregate check for branch protection: passes only when every fast-lane # job succeeded (mirrors ci.yml's `CI gate`). if: always() - needs: [go, python, codeql, semgrep, web, migrations, gitleaks] + needs: [go, python, codeql, semgrep, web, migrations, gitleaks, helm] runs-on: ubuntu-latest steps: - name: Fail if any fast-lane job failed diff --git a/.github/workflows/publish-sandbox-image.yml b/.github/workflows/publish-sandbox-image.yml index 1335e287..790221cf 100644 --- a/.github/workflows/publish-sandbox-image.yml +++ b/.github/workflows/publish-sandbox-image.yml @@ -98,7 +98,7 @@ name: Publish sandbox image (reusable) # # NO image_name NEEDED. It is derived from the bundle itself — see "WHERE THE # IMAGE NAME COMES FROM" below. Pass it only to publish somewhere other than -# GHCR under the caller's own owner (e.g. an ECR ref, per docs/EKS-DEPLOYMENT.md). +# GHCR under the caller's own owner (e.g. an ECR ref, per docs/DEPLOYMENT-KUBERNETES.md). # # WHERE THE IMAGE NAME COMES FROM # diff --git a/AGENTS.md b/AGENTS.md index aaa52782..70131b55 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -237,6 +237,12 @@ same PR. - **Installing the backup/maintenance timers on an existing box** (`fleet timers install`, the `fleet update` offer + `--no-timers` opt-out, the non-systemd/Kubernetes posture): [`docs/TIMERS.md`](docs/TIMERS.md) +- **Kubernetes as a first-class deployment** (the `deploy/helm/fleet` chart, + the pluggable sandbox backend — `FLEET_SANDBOX_BACKEND=podman|kubernetes`, + sandboxes as ephemeral pods, the fail-closed cluster preflight, and the + honest deviations from the podman backend): + [`docs/DEPLOYMENT-KUBERNETES.md`](docs/DEPLOYMENT-KUBERNETES.md) + + [ADR-0049](docs/adr/0049-kubernetes-backend-split-control-plane.md) - **Load testing & benchmarks** (`fleet-bench` HTTP chat load via the fake-LLM seam + subsystem throughput benchmarks): [`docs/LOAD-TESTING.md`](docs/LOAD-TESTING.md) - **Prompt-cache prefix-stability contract** (what must stay byte-stable in the diff --git a/CHANGELOG.md b/CHANGELOG.md index 70759210..9314ed86 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,81 @@ prior versions are listed because none have shipped. ## [Unreleased] +### Added + +- **Kubernetes as a first-class deployment (#989 / ADR-0049):** the fleet + control plane can now run in a cluster with agent sandboxes as **ephemeral + pods**, selected by one knob — `FLEET_SANDBOX_BACKEND=podman|kubernetes` + (env overrides the bundle manifest's `sandbox.backend`, mirroring + `sandbox.runtime`'s precedence; an unrecognized value refuses to boot). + What landed, in one pass per the issue's plan: + + - **A third sandbox backend** in `internal/sandbox` (`k8sImpl`) behind the + same internal interface as the podman and host executors: one pod per + sandbox (`sleep infinity` + exec over the apiserver's WebSocket channel + protocol), bash as one-shot execs, the python bridge as a held exec + session, and file ops running the same embedded `fileops.py`. Pods are + read-only-rootfs, non-root (uid 1000), all capabilities dropped, seccomp + RuntimeDefault (or an operator-installed Localhost profile), + `automountServiceAccountToken=false` — and the workspace is a shared + ReadWriteMany PVC mounted at the **same absolute path** as the control + plane, preserving the same-path invariant. The #796 poison-and-retire + containment carries over: a cancelled/timed-out call deletes the pod with + zero grace and retires the sandbox. A boot-time orphan-pod sweep mirrors + the podman container prune. No client-go: a minimal hand-rolled REST + + WebSocket-exec client (gorilla/websocket was already in the tree — zero + new modules), with kubeconfig support deliberately narrow (token / + client-cert; exec plugins and `insecure-skip-tls-verify` refused). + - **Fail-closed preflight** when the backend is selected, at boot and in + `fleet validate-config`: apiserver reachability + credentials, the exact + RBAC verbs (pods create/get/list/delete, pods/exec create), the workspace + claim, the sealed-egress NetworkPolicy object, and the RuntimeClass when + configured. Podman-only knobs are refused rather than silently ignored + (`FLEET_SANDBOX_RUNTIME` → use `FLEET_SANDBOX_K8S_RUNTIME_CLASS`; + `FLEET_SANDBOX_SECCOMP_PROFILE` → `FLEET_SANDBOX_K8S_SECCOMP_PROFILE`; + `FLEET_DEFAULT_NETWORK_MODE=allowlisted` is unsupported — the host egress + proxy is unreachable from pods). + - **Dedicated runner pools**: sandbox pods can be pinned to their own node + pool with `FLEET_SANDBOX_K8S_NODE_SELECTOR` ("k=v,k=v") and + `FLEET_SANDBOX_K8S_TOLERATIONS` (a JSON array), or the manifest's + structured `sandbox.kubernetes.node_selector` / `.tolerations` — fleet's + horizontal scaling story made concrete (more runner capacity = a bigger + pool, never more fleet replicas). Malformed values refuse to boot. Sandbox + pods also pin `imagePullPolicy: IfNotPresent` explicitly (the API's + `Always`-for-`:latest` default breaks side-loaded kind images and re-pulls + a mutable tag mid-run). + - **A Helm chart** (`deploy/helm/fleet`): single-replica control-plane + Deployment (strategy Recreate, deliberately no replica knob — the + scheduler is single-owner), the runner RBAC Role/Binding, workspace/data + PVCs, the `fleet-sandbox-deny-all` NetworkPolicy (selecting pods labeled + `fleet.elcanotek.com/egress=none`), optional egress shaping for open + pods, optional evaluation Postgres, optional web tier + Ingress. Linted + and template-rendered in CI (new `helm` job inside both gates). + - **Docs**: `docs/DEPLOYMENT-KUBERNETES.md` — the one Kubernetes reference: + 15-minute kind path, the two image builds (the control-plane + Containerfile's `FROM golang:` stage is now pinned to go.mod by + `scripts/check_versions_test.go`), production checklist, provider notes + (EKS/GKE/AKS), day-2 operations (the CronJob equivalents of the systemd + timers), troubleshooting, and an explicit honest-deviations list + (NetworkPolicy enforcement belongs to the CNI, no per-pod pids limit, no + #263 resource telemetry, no bundled-seccomp/supporting-doc mounts). Plus + updates to `DEPLOYMENT.md` and `SANDBOX-RUNTIMES.md` + (`FLEET_SANDBOX_BACKEND` documented next to `FLEET_SANDBOX_RUNTIME`). + [ADR-0049](docs/adr/0049-kubernetes-backend-split-control-plane.md) + amends ADR-0004: the single-box podman install **stays the default and is + unchanged**; only the no-k8s-artifacts enforcement clause is superseded. + +### Removed + +- **`docs/EKS-DEPLOYMENT.md`** — the hand-verified recipe for running the + whole single-box model (rootless Podman included) inside one privileged pod + on one large EKS node. Retired in favor of the first-class path above + rather than kept as a parallel track: an unmaintained privileged-pod recipe + beside a supported unprivileged one would imply support it never had (it + was explicitly "hand-verified, not CI-exercised"). Its durable content — + EFS/RWX storage, ECR/IRSA, NetworkPolicy-enforcement caveats, the backup + CronJob, day-2 mappings — was folded into `docs/DEPLOYMENT-KUBERNETES.md`. + ### Fixed - **Three own-rows authorization holes on the task surface.** The read path for diff --git a/README.md b/README.md index 9c66d7ba..d550cb81 100644 --- a/README.md +++ b/README.md @@ -324,6 +324,14 @@ sudo bash /opt/fleet/src/scripts/bootstrap.sh --postgres=local --enable-service **→ Full deployment guide** — host sizing, the one-command web + Caddy/TLS stack, the env file, and every option: **[`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md)**. +**Kubernetes shop?** fleet also ships a first-class cluster path +([ADR-0049](docs/adr/0049-kubernetes-backend-split-control-plane.md)): a Helm +chart (`deploy/helm/fleet`) for the single-replica control plane, with agent +sandboxes running as **ephemeral pods** via +`FLEET_SANDBOX_BACKEND=kubernetes` — same loop, same security model, one +backend switch. See +**[`docs/DEPLOYMENT-KUBERNETES.md`](docs/DEPLOYMENT-KUBERNETES.md)**. + ## Operating fleet The operator lifecycle is **bootstrap → update → status**, one box. The server @@ -354,6 +362,7 @@ Deep references live in [`docs/`](docs/) so this README stays an orientation, no | Doc | What it covers | |---|---| | [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | Full deployment guide — host sizing, the one-command web + Caddy/TLS stack, options | +| [`docs/DEPLOYMENT-KUBERNETES.md`](docs/DEPLOYMENT-KUBERNETES.md) | Kubernetes as a first-class path — the Helm chart, the `kubernetes` sandbox backend (agent sandboxes as ephemeral pods), kind walkthrough + production checklist | | [`docs/OPERATORS.md`](docs/OPERATORS.md) | Operator runbook — the env file, the client-config checkout, every lifecycle verb | | [`docs/AGENT-RUNTIME.md`](docs/AGENT-RUNTIME.md) | Agent runtime mechanics — per-turn sandbox, ceilings, compaction, verifier, artifacts | | [`docs/SANDBOX-RUNTIMES.md`](docs/SANDBOX-RUNTIMES.md) | Sandbox OCI runtimes — `runc` / Kata / libkrun isolation tiers | diff --git a/cmd/fleet/main.go b/cmd/fleet/main.go index 94751cf3..78879d1d 100644 --- a/cmd/fleet/main.go +++ b/cmd/fleet/main.go @@ -321,21 +321,11 @@ func run() error { // MCP catalog. Empty in the generic bundle. cfg.HTTPTools = bundle.HTTPToolConfigs() - // The sandbox image is a per-client bundle artifact: resolve it from the - // bundle manifest (sandbox.image when set — the opt-in prebuilt/registry - // path — else sandbox.tag, the build-on-box default). An explicit - // FLEET_SANDBOX_IMAGE / CHAT_SANDBOX_IMAGE in the process env still wins - // (config.Load already populated cfg.SandboxImage from it). fleet does NOT - // build the image here — bootstrap / scripts/build-sandbox-image.sh does; - // this only feeds the resolved ref to the consuming sandbox pool. - if strings.TrimSpace(cfg.SandboxImage) == "" { - if ref := bundle.Sandbox().ResolvedImageRef(); ref != "" { - cfg.SandboxImage = ref - log.Printf("sandbox: image resolved from bundle = %s", ref) - } + // Resolve the sandbox image, OCI runtime, and backend from env + bundle + // (env wins). Fail-closed on an unrecognized backend value (#989). + if err := resolveSandboxInto(cfg, bundle); err != nil { + return err } - - resolveSandboxRuntimeInto(cfg, bundle) // Sandbox egress allowlist (#211): the bundle manifest supplies the default // allowed domains for allowlisted network mode (operator-authored deployment // config, like the runtime + image). FLEET_DEFAULT_NETWORK_MODE selects the @@ -858,13 +848,7 @@ func run() error { // skips containers carrying this process's own instance label in every // state — otherwise a warm container caught in "created" state would be // force-removed by its own process. See sandbox.PruneOrphanedContainers. - pruneCtx, pruneCancel := context.WithTimeout(context.Background(), 30*time.Second) - if n, err := sandbox.PruneOrphanedContainers(pruneCtx, "podman"); err != nil { - log.Printf("startup: prune orphaned sandbox containers: %v", err) - } else if n > 0 { - log.Printf("startup: pruned %d orphaned sandbox container(s) from a prior run", n) - } - pruneCancel() + pruneOrphanedSandboxes(mgr.SandboxPool()) // Share the process shutdown grace with the pool so its in-flight-task drain // uses the same budget as the chat-turn drain (#278). A non-positive grace @@ -1988,6 +1972,121 @@ func resolveSandboxRuntimeInto(cfg *config.Config, bundle *clientconfig.Bundle) cfg.SandboxRuntime = resolved } +// pruneOrphanedSandboxes reclaims sandboxes orphaned by a PRIOR crash, +// routed by the active backend: pods under the kubernetes backend (#989), +// podman containers otherwise. Best-effort — log and continue. Extracted from +// run() to keep it within the cyclomatic budget; see the call site's comment +// for why this must run only AFTER the manager (and thus the pool) exists. +func pruneOrphanedSandboxes(pool *sandbox.Pool) { + pruneCtx, pruneCancel := context.WithTimeout(context.Background(), 30*time.Second) + defer pruneCancel() + if kb := pool.KubernetesBackend(); kb != nil { + // Kubernetes backend (#989): orphans are pods, not podman containers. + if n, err := kb.PruneOrphanedPods(pruneCtx); err != nil { + log.Printf("startup: prune orphaned sandbox pods: %v", err) + } else if n > 0 { + //nolint:gosec // G706: n is an integer count of deleted pods — no attacker-controllable text reaches the log. + log.Printf("startup: pruned %d orphaned sandbox pod(s) from a prior run", n) + } + return + } + if n, err := sandbox.PruneOrphanedContainers(pruneCtx, "podman"); err != nil { + log.Printf("startup: prune orphaned sandbox containers: %v", err) + } else if n > 0 { + log.Printf("startup: pruned %d orphaned sandbox container(s) from a prior run", n) + } +} + +// resolveSandboxInto resolves the sandbox image, OCI runtime, and backend +// from env + bundle, in that order. Extracted from run() to keep it within +// the cyclomatic budget. +// +// The image is a per-client bundle artifact: the bundle manifest's +// sandbox.image (the opt-in prebuilt/registry path — else sandbox.tag, the +// build-on-box default) fills it when no explicit FLEET_SANDBOX_IMAGE / +// CHAT_SANDBOX_IMAGE env is set (config.Load already populated +// cfg.SandboxImage from those). fleet does NOT build the image here — +// bootstrap / scripts/build-sandbox-image.sh does; this only feeds the +// resolved ref to the consuming sandbox pool. +func resolveSandboxInto(cfg *config.Config, bundle *clientconfig.Bundle) error { + if strings.TrimSpace(cfg.SandboxImage) == "" { + if ref := bundle.Sandbox().ResolvedImageRef(); ref != "" { + cfg.SandboxImage = ref + log.Printf("sandbox: image resolved from bundle = %s", ref) + } + } + resolveSandboxRuntimeInto(cfg, bundle) + return resolveSandboxBackendInto(cfg, bundle) +} + +// resolveSandboxBackendInto resolves the sandbox backend (#989) into +// cfg.SandboxBackend with the SAME env-wins-else-bundle precedence as the +// image and runtime (sandbox.ResolveBackend is the one shared resolver, so +// boot and validate-config cannot drift), then fills each kubernetes setting +// from the bundle's sandbox.kubernetes block when the corresponding +// FLEET_SANDBOX_K8S_* env var is empty. An unrecognized backend value is a +// boot error, never a silent fallback to podman. +func resolveSandboxBackendInto(cfg *config.Config, bundle *clientconfig.Bundle) error { + sb := bundle.Sandbox() + resolved, err := sandbox.ResolveBackend(cfg.SandboxBackend, sb.Backend) + if err != nil { + return err + } + cfg.SandboxBackend = resolved + if resolved != sandbox.BackendKubernetes { + return nil + } + fill := func(dst *string, bundleVal string) { + if strings.TrimSpace(*dst) == "" { + *dst = strings.TrimSpace(bundleVal) + } + } + k := sb.Kubernetes + fill(&cfg.SandboxK8sNamespace, k.Namespace) + fill(&cfg.SandboxK8sWorkspaceClaim, k.WorkspaceClaim) + fill(&cfg.SandboxK8sServiceAccount, k.ServiceAccount) + fill(&cfg.SandboxK8sImagePullSecret, k.ImagePullSecret) + fill(&cfg.SandboxK8sRuntimeClass, k.RuntimeClass) + fill(&cfg.SandboxK8sSeccompProfile, k.SeccompProfile) + fill(&cfg.SandboxK8sKubeconfig, k.Kubeconfig) + fill(&cfg.SandboxK8sNetworkPolicy, k.NetworkPolicy) + // The scheduling knobs are structured in the manifest; canonicalize them + // into the same string forms the env vars use so the pool build has ONE + // source to parse (env wins, like every other field). + if strings.TrimSpace(cfg.SandboxK8sNodeSelector) == "" && len(k.NodeSelector) > 0 { + keys := make([]string, 0, len(k.NodeSelector)) + for key := range k.NodeSelector { + keys = append(keys, key) + } + sort.Strings(keys) + pairs := make([]string, 0, len(keys)) + for _, key := range keys { + pairs = append(pairs, key+"="+k.NodeSelector[key]) + } + cfg.SandboxK8sNodeSelector = strings.Join(pairs, ",") + } + if strings.TrimSpace(cfg.SandboxK8sTolerations) == "" && len(k.Tolerations) > 0 { + raw, err := json.Marshal(k.Tolerations) + if err != nil { + return fmt.Errorf("encode sandbox.kubernetes.tolerations: %w", err) + } + cfg.SandboxK8sTolerations = string(raw) + } + //nolint:gosec // G706: backend + namespace are operator config (FLEET_SANDBOX_* / bundle manifest), quoted with %q — not request input. + log.Printf("sandbox: backend resolved to %q (control plane and runners split — pods in namespace %q)", + resolved, defaultString(cfg.SandboxK8sNamespace, "fleet-sandboxes")) + return nil +} + +// defaultString returns v, or fallback when v is empty — a log-formatting +// helper for resolveSandboxBackendInto. +func defaultString(v, fallback string) string { + if strings.TrimSpace(v) == "" { + return fallback + } + return v +} + // registerRuntimeMetrics wires the pull-at-scrape gauges (#176): in-flight turn // counts (interactive/scheduled), warm sandbox depth, host disk headroom, and // the Go runtime's goroutine/heap counters. Extracted from run() to keep it diff --git a/cmd/fleet/validate_config.go b/cmd/fleet/validate_config.go index f3c545ba..ba63804f 100644 --- a/cmd/fleet/validate_config.go +++ b/cmd/fleet/validate_config.go @@ -700,6 +700,20 @@ func checkSandbox(ctx context.Context, cfg *config.Config, bundle *clientconfig. return res } + // Kubernetes backend (#989): none of the podman checks apply — validate + // the backend selection and run the same fail-closed cluster preflight the + // boot path runs (apiserver reachability, RBAC, workspace claim, the + // sealed-egress NetworkPolicy, the RuntimeClass when one is configured). + backend, err := resolveValidateSandboxBackend(cfg, bundle) + if err != nil { + res.Status = statusFail + res.Detail = err.Error() + return res + } + if backend == sandbox.BackendKubernetes { + return checkKubernetesSandbox(ctx, res, cfg, bundle) + } + const podmanBin = "podman" if _, err := exec.LookPath(podmanBin); err != nil { res.Status = statusFail @@ -766,6 +780,103 @@ func checkSandbox(ctx context.Context, cfg *config.Config, bundle *clientconfig. return res } +// resolveValidateSandboxBackend resolves the sandbox backend the same way the +// boot path does (sandbox.ResolveBackend: env FLEET_SANDBOX_BACKEND wins, else +// the bundle manifest's sandbox.backend, else podman; unrecognized = error). +func resolveValidateSandboxBackend(cfg *config.Config, bundle *clientconfig.Bundle) (string, error) { + envBackend := "" + if cfg != nil { + envBackend = cfg.SandboxBackend + } + bundleBackend := "" + if bundle != nil { + bundleBackend = bundle.Sandbox().Backend + } + return sandbox.ResolveBackend(envBackend, bundleBackend) +} + +// checkKubernetesSandbox validates the kubernetes sandbox backend: the image +// ref resolves, the podman-only knobs are unset, and the boot preflight's +// cluster checks pass. Image PRESENCE is not checked — pulls happen on the +// sandbox nodes' kubelets, which this process cannot see; a bad ref fails +// fast at the first pod start instead. +func checkKubernetesSandbox(ctx context.Context, res checkResult, cfg *config.Config, bundle *clientconfig.Bundle) checkResult { + if rt := resolveSandboxRuntime(cfg, bundle); rt != "" { + res.Status = statusFail + res.Detail = fmt.Sprintf("FLEET_SANDBOX_RUNTIME=%q has no effect under the kubernetes backend — use FLEET_SANDBOX_K8S_RUNTIME_CLASS", rt) + return res + } + if cfg.DefaultNetworkMode == sandbox.NetworkModeAllowlisted { + res.Status = statusFail + res.Detail = "FLEET_DEFAULT_NETWORK_MODE=allowlisted is not supported under the kubernetes backend (the host egress proxy is unreachable from pods) — use lockdown or open" + return res + } + image := resolveSandboxImage(cfg, bundle) + if image == "" { + res.Status = statusFail + res.Detail = "no sandbox image resolved (set FLEET_SANDBOX_IMAGE or the bundle manifest's sandbox.image — kubernetes nodes cannot consume a build-on-box tag)" + return res + } + // Same env-wins-else-bundle resolution and fail-closed parse as the boot + // path: an env value is parsed from its string form; with no env value + // the bundle's structured knobs apply directly. + k8s := bundle.Sandbox().Kubernetes + nodeSelector := k8s.NodeSelector + if strings.TrimSpace(cfg.SandboxK8sNodeSelector) != "" { + parsed, err := sandbox.ParseK8sNodeSelector(cfg.SandboxK8sNodeSelector) + if err != nil { + res.Status = statusFail + res.Detail = "FLEET_SANDBOX_K8S_NODE_SELECTOR: " + err.Error() + return res + } + nodeSelector = parsed + } + var tolerations []sandbox.K8sToleration + for _, tol := range k8s.Tolerations { + tolerations = append(tolerations, sandbox.K8sToleration(tol)) + } + if strings.TrimSpace(cfg.SandboxK8sTolerations) != "" { + parsed, err := sandbox.ParseK8sTolerations(cfg.SandboxK8sTolerations) + if err != nil { + res.Status = statusFail + res.Detail = "FLEET_SANDBOX_K8S_TOLERATIONS: " + err.Error() + return res + } + tolerations = parsed + } + fill := func(env, bundleVal string) string { + if strings.TrimSpace(env) != "" { + return strings.TrimSpace(env) + } + return strings.TrimSpace(bundleVal) + } + backend, err := sandbox.NewKubernetesBackend(sandbox.KubernetesConfig{ + Namespace: fill(cfg.SandboxK8sNamespace, k8s.Namespace), + WorkspaceClaim: fill(cfg.SandboxK8sWorkspaceClaim, k8s.WorkspaceClaim), + ServiceAccount: fill(cfg.SandboxK8sServiceAccount, k8s.ServiceAccount), + ImagePullSecret: fill(cfg.SandboxK8sImagePullSecret, k8s.ImagePullSecret), + RuntimeClassName: fill(cfg.SandboxK8sRuntimeClass, k8s.RuntimeClass), + SeccompLocalhostProfile: fill(cfg.SandboxK8sSeccompProfile, k8s.SeccompProfile), + KubeconfigPath: fill(cfg.SandboxK8sKubeconfig, k8s.Kubeconfig), + NetworkPolicyName: fill(cfg.SandboxK8sNetworkPolicy, k8s.NetworkPolicy), + NodeSelector: nodeSelector, + Tolerations: tolerations, + }) + if err != nil { + res.Status = statusFail + res.Detail = err.Error() + return res + } + if err := backend.Preflight(ctx); err != nil { + res.Status = statusFail + res.Detail = err.Error() + return res + } + res.Status = statusOK + res.Detail = fmt.Sprintf("kubernetes backend ok; image %q, sandbox namespace %q (image pullability is checked at first pod start)", image, backend.Namespace()) + return res +} + // sandboxIsContainerBacked reports whether this binary will run agent tool calls // in a container (the only sandbox path that needs podman). True for a release // build (host executor not compiled in). When the host executor IS compiled in, diff --git a/config/default/manifest.yaml b/config/default/manifest.yaml index 79e275e5..4637ff58 100644 --- a/config/default/manifest.yaml +++ b/config/default/manifest.yaml @@ -28,6 +28,24 @@ sandbox: # fail-closed boot preflight. # See docs/SANDBOX-RUNTIMES.md. FLEET_SANDBOX_RUNTIME overrides this. runtime: "${FLEET_SANDBOX_RUNTIME:-}" + # WHERE sandboxes run (#989 / ADR-0049): "podman" (default — co-located + # rootless containers) or "kubernetes" (each sandbox an ephemeral pod; the + # split control-plane/runner enterprise path — see + # docs/DEPLOYMENT-KUBERNETES.md). FLEET_SANDBOX_BACKEND overrides this, and + # each sandbox.kubernetes.* field below is overridden by the matching + # FLEET_SANDBOX_K8S_* env var. Uncomment to opt a bundle in: + # backend: kubernetes + # kubernetes: + # namespace: "" # default: the control plane's own namespace + # workspace_claim: fleet-workspace # required: shared RWX PVC + # service_account: fleet-sandbox + # image_pull_secret: "" + # runtime_class: "" # e.g. kata (preflighted fail-closed) + # seccomp_profile: "" # node-local Localhost profile; empty = RuntimeDefault + # kubeconfig: "" # out-of-cluster auth; empty = in-cluster + # network_policy: fleet-sandbox-deny-all + # node_selector: {} # pin sandbox pods to a dedicated runner pool + # tolerations: [] # [{key,operator,value,effect}] for a tainted pool branding: app_name: "Fleet" diff --git a/deploy/helm/fleet/Chart.yaml b/deploy/helm/fleet/Chart.yaml new file mode 100644 index 00000000..305bbfc4 --- /dev/null +++ b/deploy/helm/fleet/Chart.yaml @@ -0,0 +1,17 @@ +apiVersion: v2 +name: fleet +description: >- + fleet control plane on Kubernetes with pluggable sandbox runners (issue + #989 / ADR-0049): a single-replica control-plane Deployment plus RBAC, + workspace storage, and NetworkPolicies for the kubernetes sandbox backend + (FLEET_SANDBOX_BACKEND=kubernetes). The single-box podman install remains + the default fleet deployment; this chart is the enterprise split path. +type: application +# Chart version tracks chart changes; appVersion is informational — operators +# build and pin their own fleet images (no public fleet image is published). +version: 0.1.0 +appVersion: "unreleased" +kubeVersion: ">=1.29.0-0" +home: https://github.com/ElcanoTek/fleet +sources: + - https://github.com/ElcanoTek/fleet diff --git a/deploy/helm/fleet/README.md b/deploy/helm/fleet/README.md new file mode 100644 index 00000000..b9b8654e --- /dev/null +++ b/deploy/helm/fleet/README.md @@ -0,0 +1,47 @@ +# fleet Helm chart + +> The enterprise deployment path (issue #989 / [ADR-0049](../../../docs/adr/0049-kubernetes-backend-split-control-plane.md)): +> the fleet control plane as a single-replica Deployment, with agent sandboxes +> running as ephemeral pods via `FLEET_SANDBOX_BACKEND=kubernetes`. The +> single-box podman install ([`docs/DEPLOYMENT.md`](../../../docs/DEPLOYMENT.md)) +> remains the default fleet deployment; use this chart when Kubernetes is your +> platform standard. + +Full walkthrough — a 15-minute kind path and the production checklist — lives +in [`docs/DEPLOYMENT-KUBERNETES.md`](../../../docs/DEPLOYMENT-KUBERNETES.md). + +## What it installs + +| Piece | Object | Notes | +| --- | --- | --- | +| Control plane | Deployment (1 replica, Recreate) | chat :8080 + orchestrator :8000; no replica knob — the scheduler is single-owner | +| Sandbox RBAC | Role/RoleBinding | exactly the verbs the boot preflight checks: pods create/get/list/delete, pods/exec create, PVC + NetworkPolicy get | +| Workspace | RWX PVC | mounted at the SAME path in the control plane and every sandbox pod | +| Sealed egress | NetworkPolicy `fleet-sandbox-deny-all` | selects pods labeled `fleet.elcanotek.com/egress=none`; the preflight requires it to exist | +| Postgres | optional StatefulSet | evaluation only — production points at a managed database | +| Web / Ingress | optional | you build the web image from `web/` | + +## Minimum install + +```sh +helm install fleet deploy/helm/fleet \ + --namespace fleet --create-namespace \ + --set image.repository=REGISTRY/fleet --set image.tag=v1 \ + --set sandbox.image=REGISTRY/fleet-sandbox:v1 \ + --set postgres.enabled=true \ + --set config.existingSecret=fleet-secrets # OPENROUTER_API_KEY etc. +``` + +You build both images yourself — fleet publishes none. See the deployment +guide for the two Containerfiles. + +## Honest scope + +- **NetworkPolicy enforcement is the CNI's job.** The chart ships the deny-all + object and fleet's preflight verifies it exists; on a CNI without + NetworkPolicy support it seals nothing. Verify enforcement (the guide's + checklist shows how). +- **One control-plane replica.** No HPA, no `replicas`. Scale work with more + sandbox capacity, the control plane with bigger requests. +- The chart is linted in CI but not exercised against a live cluster there; + the kind walkthrough in the guide is the verified path. diff --git a/deploy/helm/fleet/templates/NOTES.txt b/deploy/helm/fleet/templates/NOTES.txt new file mode 100644 index 00000000..2997000d --- /dev/null +++ b/deploy/helm/fleet/templates/NOTES.txt @@ -0,0 +1,22 @@ +fleet {{ .Chart.Version }} installed as {{ .Release.Name }} in {{ .Release.Namespace }}. + +Control plane : one replica (by design — single-owner scheduler; do not scale it) +Sandbox pods : namespace {{ include "fleet.sandboxNamespace" . }}, image {{ .Values.sandbox.image }} +Workspace : PVC {{ .Values.workspace.claimName }} mounted at {{ .Values.workspaceRoot }} (control plane + every sandbox pod) + +Check boot (the kubernetes sandbox preflight runs before serving; a missing +RBAC grant, workspace claim, or NetworkPolicy aborts start with the reason): + + kubectl -n {{ .Release.Namespace }} logs deploy/{{ include "fleet.fullname" . }} | grep -E 'sandbox|preflight' + +Reach the chat API: + + kubectl -n {{ .Release.Namespace }} port-forward svc/{{ include "fleet.fullname" . }} 8080:8080 + curl -s localhost:8080/healthz + +Verify a turn executes in a sandbox pod: + + kubectl -n {{ include "fleet.sandboxNamespace" . }} get pods -l app.kubernetes.io/name=fleet-sandbox -w + +Full guide (15-minute kind path + production checklist): + docs/DEPLOYMENT-KUBERNETES.md in the fleet repository. diff --git a/deploy/helm/fleet/templates/_helpers.tpl b/deploy/helm/fleet/templates/_helpers.tpl new file mode 100644 index 00000000..0d0ef242 --- /dev/null +++ b/deploy/helm/fleet/templates/_helpers.tpl @@ -0,0 +1,42 @@ +{{/* Chart name + fullname, standard helpers. */}} +{{- define "fleet.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{- define "fleet.fullname" -}} +{{- if .Values.fullnameOverride -}} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- printf "%s" .Release.Name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +{{- end -}} + +{{- define "fleet.labels" -}} +app.kubernetes.io/name: {{ include "fleet.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version }} +{{- end -}} + +{{- define "fleet.selectorLabels" -}} +app.kubernetes.io/name: {{ include "fleet.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "fleet.serviceAccountName" -}} +{{- if .Values.serviceAccount.create -}} +{{- default (include "fleet.fullname" .) .Values.serviceAccount.name -}} +{{- else -}} +{{- required "serviceAccount.name is required when serviceAccount.create=false" .Values.serviceAccount.name -}} +{{- end -}} +{{- end -}} + +{{/* Namespace sandbox pods run in: explicit value or the release namespace. */}} +{{- define "fleet.sandboxNamespace" -}} +{{- default .Release.Namespace .Values.sandbox.kubernetes.namespace -}} +{{- end -}} + +{{/* In-cluster Postgres endpoints (postgres.enabled=true only). */}} +{{- define "fleet.postgresHost" -}} +{{- printf "%s-postgres" (include "fleet.fullname" .) -}} +{{- end -}} diff --git a/deploy/helm/fleet/templates/deployment.yaml b/deploy/helm/fleet/templates/deployment.yaml new file mode 100644 index 00000000..b77f3e0b --- /dev/null +++ b/deploy/helm/fleet/templates/deployment.yaml @@ -0,0 +1,174 @@ +# The fleet control plane. ONE replica, forever: scheduler leases and the +# worker semaphore are single-owner (ADR-0004 / ADR-0049) — two replicas +# against one database pair is a correctness bug. There is deliberately no +# replicas knob; horizontal scale of work = more sandbox pods. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "fleet.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + replicas: 1 + strategy: + # Recreate, not RollingUpdate: a rolling update would run two control + # planes side by side for the overlap window — the exact multi-writer + # state the single-owner invariant forbids. + type: Recreate + selector: + matchLabels: + {{- include "fleet.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "fleet.selectorLabels" . | nindent 8 }} + {{- with .Values.podLabels }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + serviceAccountName: {{ include "fleet.serviceAccountName" . }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: fleet + image: "{{ required "image.repository is required (you build the fleet image yourself — see docs/DEPLOYMENT-KUBERNETES.md)" .Values.image.repository }}:{{ required "image.tag is required" .Values.image.tag }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + args: ["serve"] + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + ports: + - name: chat + containerPort: 8080 + - name: orchestrator + containerPort: 8000 + env: + # Listeners bind all pod interfaces; the Service/Ingress is the + # exposure boundary (the single-box default is loopback + Caddy). + - name: FLEET_SERVER_ADDR + value: "0.0.0.0:8080" + - name: FLEET_ORCHESTRATOR_ADDR + value: "0.0.0.0:8000" + - name: FLEET_DATA_DIR + value: {{ .Values.dataDir | quote }} + - name: FLEET_WORKSPACE_ROOT + value: {{ .Values.workspaceRoot | quote }} + # ── sandbox backend (#989) ── + - name: FLEET_SANDBOX_BACKEND + value: {{ .Values.sandbox.backend | quote }} + - name: FLEET_SANDBOX_IMAGE + value: {{ required "sandbox.image is required (a registry ref sandbox nodes can pull)" .Values.sandbox.image | quote }} + - name: FLEET_SANDBOX_K8S_NAMESPACE + value: {{ include "fleet.sandboxNamespace" . | quote }} + - name: FLEET_SANDBOX_K8S_WORKSPACE_CLAIM + value: {{ .Values.workspace.claimName | quote }} + - name: FLEET_SANDBOX_K8S_NETWORK_POLICY + value: {{ .Values.networkPolicies.denyAll.name | quote }} + {{- if .Values.sandbox.kubernetes.podServiceAccount.name }} + - name: FLEET_SANDBOX_K8S_SERVICE_ACCOUNT + value: {{ .Values.sandbox.kubernetes.podServiceAccount.name | quote }} + {{- end }} + {{- with .Values.sandbox.kubernetes.imagePullSecret }} + - name: FLEET_SANDBOX_K8S_IMAGE_PULL_SECRET + value: {{ . | quote }} + {{- end }} + {{- with .Values.sandbox.kubernetes.runtimeClass }} + - name: FLEET_SANDBOX_K8S_RUNTIME_CLASS + value: {{ . | quote }} + {{- end }} + {{- with .Values.sandbox.kubernetes.seccompProfile }} + - name: FLEET_SANDBOX_K8S_SECCOMP_PROFILE + value: {{ . | quote }} + {{- end }} + {{- with .Values.sandbox.kubernetes.nodeSelector }} + - name: FLEET_SANDBOX_K8S_NODE_SELECTOR + value: {{ $pairs := list }}{{- range $k, $v := . }}{{- $pairs = append $pairs (printf "%s=%s" $k $v) }}{{- end }}{{ join "," $pairs | quote }} + {{- end }} + {{- with .Values.sandbox.kubernetes.tolerations }} + - name: FLEET_SANDBOX_K8S_TOLERATIONS + value: {{ toJson . | quote }} + {{- end }} + - name: FLEET_SANDBOX_MEMORY + value: {{ .Values.sandbox.memory | quote }} + - name: FLEET_SANDBOX_CPUS + value: {{ .Values.sandbox.cpus | quote }} + - name: FLEET_SANDBOX_DISK_GB + value: {{ .Values.sandbox.diskGB | quote }} + {{- if gt (int .Values.sandbox.warmSize) 0 }} + - name: FLEET_SANDBOX_WARM_SIZE + value: {{ .Values.sandbox.warmSize | quote }} + {{- end }} + {{- if .Values.postgres.enabled }} + - name: FLEET_CHAT_DATABASE_URL + valueFrom: + secretKeyRef: + name: {{ include "fleet.fullname" . }}-postgres + key: chat-url + - name: FLEET_SCHED_DATABASE_URL + valueFrom: + secretKeyRef: + name: {{ include "fleet.fullname" . }}-postgres + key: sched-url + {{- end }} + {{- range $k, $v := .Values.config.env }} + - name: {{ $k }} + value: {{ $v | quote }} + {{- end }} + {{- with .Values.config.existingSecret }} + envFrom: + - secretRef: + name: {{ . }} + {{- end }} + volumeMounts: + - name: workspace + mountPath: {{ .Values.workspaceRoot }} + - name: data + mountPath: {{ .Values.dataDir }} + readinessProbe: + httpGet: + path: /healthz + port: chat + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: + path: /healthz + port: chat + initialDelaySeconds: 30 + periodSeconds: 30 + resources: + {{- toYaml .Values.resources | nindent 12 }} + volumes: + - name: workspace + persistentVolumeClaim: + claimName: {{ .Values.workspace.claimName }} + - name: data + persistentVolumeClaim: + claimName: {{ .Values.data.claimName }} + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} diff --git a/deploy/helm/fleet/templates/ingress.yaml b/deploy/helm/fleet/templates/ingress.yaml new file mode 100644 index 00000000..5b539f3d --- /dev/null +++ b/deploy/helm/fleet/templates/ingress.yaml @@ -0,0 +1,38 @@ +{{- if .Values.ingress.enabled }} +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: {{ include "fleet.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} + {{- with .Values.ingress.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + {{- with .Values.ingress.className }} + ingressClassName: {{ . }} + {{- end }} + {{- with .Values.ingress.tls }} + tls: + {{- toYaml . | nindent 4 }} + {{- end }} + rules: + - host: {{ required "ingress.host is required when ingress.enabled" .Values.ingress.host }} + http: + paths: + - path: / + pathType: Prefix + backend: + service: + {{- if .Values.web.enabled }} + name: {{ include "fleet.fullname" . }}-web + port: + name: http + {{- else }} + name: {{ include "fleet.fullname" . }} + port: + name: chat + {{- end }} +{{- end }} diff --git a/deploy/helm/fleet/templates/networkpolicy.yaml b/deploy/helm/fleet/templates/networkpolicy.yaml new file mode 100644 index 00000000..8f79eac0 --- /dev/null +++ b/deploy/helm/fleet/templates/networkpolicy.yaml @@ -0,0 +1,57 @@ +{{- if .Values.networkPolicies.denyAll.create }} +# The sealed-egress policy for sandbox pods labeled +# fleet.elcanotek.com/egress=none (lockdown turns, sealed scheduled runs). +# fleet's boot preflight REQUIRES this object to exist before it will start +# with the kubernetes backend. HONEST LIMIT: a NetworkPolicy object is +# enforced by the cluster CNI — on a CNI without NetworkPolicy support this +# object exists but seals nothing. Verify your CNI enforces it (the +# production checklist in docs/DEPLOYMENT-KUBERNETES.md shows how). +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ .Values.networkPolicies.denyAll.name }} + namespace: {{ include "fleet.sandboxNamespace" . }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: fleet-sandbox + fleet.elcanotek.com/egress: "none" + policyTypes: ["Ingress", "Egress"] + # No ingress/egress rules: everything denied, both directions. +{{- end }} +{{- if .Values.networkPolicies.openEgress.create }} +--- +# Egress shaping for OPEN sandbox pods: allow DNS + everything EXCEPT the +# blocked CIDRs (your cluster/node ranges), so `pip install` works but the +# sandbox cannot reach in-cluster Services. Ingress stays denied. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "fleet.fullname" . }}-sandbox-open-egress + namespace: {{ include "fleet.sandboxNamespace" . }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: fleet-sandbox + fleet.elcanotek.com/egress: "open" + policyTypes: ["Ingress", "Egress"] + egress: + - to: + - ipBlock: + cidr: 0.0.0.0/0 + {{- with .Values.networkPolicies.openEgress.blockedCIDRs }} + except: + {{- toYaml . | nindent 14 }} + {{- end }} + # DNS (kube-dns lives inside the cluster CIDRs excluded above). + - to: [] + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 +{{- end }} diff --git a/deploy/helm/fleet/templates/postgres.yaml b/deploy/helm/fleet/templates/postgres.yaml new file mode 100644 index 00000000..07871145 --- /dev/null +++ b/deploy/helm/fleet/templates/postgres.yaml @@ -0,0 +1,119 @@ +{{- if .Values.postgres.enabled }} +# Optional in-cluster PostgreSQL for evaluation and small installs: one +# replica, one PVC, both fleet databases (chat + sched) in one server. +# Production should use a managed database and config.existingSecret URLs. +{{- /* The password never comes from values: reuse the existing Secret's + (operator-pre-created or a prior install's) or generate a random one. */ -}} +{{- $password := "" }} +{{- $existing := lookup "v1" "Secret" .Release.Namespace (printf "%s-postgres" (include "fleet.fullname" .)) }} +{{- if $existing }} +{{- $password = index $existing.data "password" | b64dec }} +{{- else }} +{{- $password = randAlphaNum 32 }} +{{- end }} +apiVersion: v1 +kind: Secret +metadata: + name: {{ include "fleet.fullname" . }}-postgres + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +type: Opaque +stringData: + password: {{ $password | quote }} + chat-url: "postgres://fleet:{{ $password }}@{{ include "fleet.postgresHost" . }}:5432/chat?sslmode=disable" + sched-url: "postgres://fleet:{{ $password }}@{{ include "fleet.postgresHost" . }}:5432/sched?sslmode=disable" +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "fleet.fullname" . }}-postgres-init + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +data: + init.sql: | + CREATE DATABASE chat; + CREATE DATABASE sched; +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: {{ include "fleet.fullname" . }}-postgres + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + serviceName: {{ include "fleet.postgresHost" . }} + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: {{ include "fleet.name" . }}-postgres + app.kubernetes.io/instance: {{ .Release.Name }} + template: + metadata: + labels: + app.kubernetes.io/name: {{ include "fleet.name" . }}-postgres + app.kubernetes.io/instance: {{ .Release.Name }} + spec: + securityContext: + fsGroup: 999 + seccompProfile: + type: RuntimeDefault + containers: + - name: postgres + image: {{ .Values.postgres.image }} + env: + - name: POSTGRES_USER + value: fleet + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: {{ include "fleet.fullname" . }}-postgres + key: password + - name: PGDATA + value: /var/lib/postgresql/data/pgdata + ports: + - name: postgres + containerPort: 5432 + volumeMounts: + - name: pgdata + mountPath: /var/lib/postgresql/data + - name: init + mountPath: /docker-entrypoint-initdb.d + readinessProbe: + exec: + command: ["pg_isready", "-U", "fleet"] + periodSeconds: 5 + volumes: + - name: init + configMap: + name: {{ include "fleet.fullname" . }}-postgres-init + volumeClaimTemplates: + - metadata: + name: pgdata + spec: + accessModes: ["ReadWriteOnce"] + {{- with .Values.postgres.storageClassName }} + storageClassName: {{ . }} + {{- end }} + resources: + requests: + storage: {{ .Values.postgres.size }} +--- +apiVersion: v1 +kind: Service +metadata: + name: {{ include "fleet.postgresHost" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + clusterIP: None + selector: + app.kubernetes.io/name: {{ include "fleet.name" . }}-postgres + app.kubernetes.io/instance: {{ .Release.Name }} + ports: + - name: postgres + port: 5432 +{{- end }} diff --git a/deploy/helm/fleet/templates/rbac.yaml b/deploy/helm/fleet/templates/rbac.yaml new file mode 100644 index 00000000..e9d018dc --- /dev/null +++ b/deploy/helm/fleet/templates/rbac.yaml @@ -0,0 +1,70 @@ +# RBAC for the kubernetes sandbox backend: exactly the verbs the backend's +# boot preflight checks (internal/sandbox/k8s_preflight.go) — pod lifecycle + +# exec in the sandbox namespace, plus read access to the two objects the +# preflight verifies (the workspace claim and the deny-all NetworkPolicy). +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "fleet.fullname" . }}-runner + namespace: {{ include "fleet.sandboxNamespace" . }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["create", "get", "list", "delete"] + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] + - apiGroups: [""] + resources: ["persistentvolumeclaims"] + verbs: ["get"] + - apiGroups: ["networking.k8s.io"] + resources: ["networkpolicies"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "fleet.fullname" . }}-runner + namespace: {{ include "fleet.sandboxNamespace" . }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "fleet.fullname" . }}-runner +subjects: + - kind: ServiceAccount + name: {{ include "fleet.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- if .Values.sandbox.kubernetes.runtimeClass }} +--- +# RuntimeClass objects are cluster-scoped; the preflight GETs the configured +# one to fail closed when it does not exist (ADR-0010 posture). +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "fleet.fullname" . }}-runtimeclass-reader + labels: + {{- include "fleet.labels" . | nindent 4 }} +rules: + - apiGroups: ["node.k8s.io"] + resources: ["runtimeclasses"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "fleet.fullname" . }}-runtimeclass-reader + labels: + {{- include "fleet.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "fleet.fullname" . }}-runtimeclass-reader +subjects: + - kind: ServiceAccount + name: {{ include "fleet.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/deploy/helm/fleet/templates/service.yaml b/deploy/helm/fleet/templates/service.yaml new file mode 100644 index 00000000..7f4b87d1 --- /dev/null +++ b/deploy/helm/fleet/templates/service.yaml @@ -0,0 +1,18 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ include "fleet.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + type: {{ .Values.service.type }} + selector: + {{- include "fleet.selectorLabels" . | nindent 4 }} + ports: + - name: chat + port: 8080 + targetPort: chat + - name: orchestrator + port: 8000 + targetPort: orchestrator diff --git a/deploy/helm/fleet/templates/serviceaccount.yaml b/deploy/helm/fleet/templates/serviceaccount.yaml new file mode 100644 index 00000000..a0ded8eb --- /dev/null +++ b/deploy/helm/fleet/templates/serviceaccount.yaml @@ -0,0 +1,23 @@ +{{- if .Values.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "fleet.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +{{- end }} +{{- if .Values.sandbox.kubernetes.podServiceAccount.create }} +--- +# Identity stamped on sandbox pods so admission policies can key on it. +# No RBAC is bound to it and no token is ever mounted +# (automountServiceAccountToken=false on every sandbox pod). +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.sandbox.kubernetes.podServiceAccount.name }} + namespace: {{ include "fleet.sandboxNamespace" . }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +automountServiceAccountToken: false +{{- end }} diff --git a/deploy/helm/fleet/templates/storage.yaml b/deploy/helm/fleet/templates/storage.yaml new file mode 100644 index 00000000..8c055d2a --- /dev/null +++ b/deploy/helm/fleet/templates/storage.yaml @@ -0,0 +1,50 @@ +{{- if .Values.workspace.create }} +{{- if ne (include "fleet.sandboxNamespace" .) .Release.Namespace }} +{{- fail "workspace.create=true only supports sandbox pods in the release namespace: a PVC cannot be mounted across namespaces. For a separate sandbox namespace, set workspace.create=false and provision a claim with the same name in EACH namespace, both bound to the same ReadWriteMany export (static NFS/EFS PVs)." }} +{{- end }} +# The shared workspace claim: mounted by the control plane AND every sandbox +# pod at {{ .Values.workspaceRoot }}. ReadWriteMany in production (EFS, NFS, +# CephFS); ReadWriteOnce only works on a single-node cluster (kind). +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: {{ .Values.workspace.claimName }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} + annotations: + # Keep the workspace when the release is deleted/pruned — it holds user + # data (conversation files, task outputs). + helm.sh/resource-policy: keep +spec: + accessModes: + {{- toYaml .Values.workspace.accessModes | nindent 4 }} + {{- with .Values.workspace.storageClassName }} + storageClassName: {{ . }} + {{- end }} + resources: + requests: + storage: {{ .Values.workspace.size }} +{{- end }} +{{- if .Values.data.create }} +--- +# The fleet data dir (journals, logs, caches). Control plane only. +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: {{ .Values.data.claimName }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} + annotations: + helm.sh/resource-policy: keep +spec: + accessModes: + {{- toYaml .Values.data.accessModes | nindent 4 }} + {{- with .Values.data.storageClassName }} + storageClassName: {{ . }} + {{- end }} + resources: + requests: + storage: {{ .Values.data.size }} +{{- end }} diff --git a/deploy/helm/fleet/templates/web.yaml b/deploy/helm/fleet/templates/web.yaml new file mode 100644 index 00000000..7d938990 --- /dev/null +++ b/deploy/helm/fleet/templates/web.yaml @@ -0,0 +1,67 @@ +{{- if .Values.web.enabled }} +# Optional web tier (the Next.js app you build from web/). It talks to the +# control-plane Service; expose it via the Ingress below or your own. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "fleet.fullname" . }}-web + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: {{ include "fleet.name" . }}-web + app.kubernetes.io/instance: {{ .Release.Name }} + template: + metadata: + labels: + app.kubernetes.io/name: {{ include "fleet.name" . }}-web + app.kubernetes.io/instance: {{ .Release.Name }} + spec: + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + containers: + - name: web + image: {{ required "web.image is required when web.enabled" .Values.web.image }} + ports: + - name: http + containerPort: {{ .Values.web.port }} + env: + - name: CHAT_SERVER_URL + value: "http://{{ include "fleet.fullname" . }}:8080" + - name: ORCHESTRATOR_URL + value: "http://{{ include "fleet.fullname" . }}:8000" + {{- range $k, $v := .Values.web.env }} + - name: {{ $k }} + value: {{ $v | quote }} + {{- end }} + {{- with .Values.config.existingSecret }} + envFrom: + - secretRef: + name: {{ . }} + {{- end }} +--- +apiVersion: v1 +kind: Service +metadata: + name: {{ include "fleet.fullname" . }}-web + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + selector: + app.kubernetes.io/name: {{ include "fleet.name" . }}-web + app.kubernetes.io/instance: {{ .Release.Name }} + ports: + - name: http + port: {{ .Values.web.port }} + targetPort: http +{{- end }} diff --git a/deploy/helm/fleet/values.yaml b/deploy/helm/fleet/values.yaml new file mode 100644 index 00000000..a8550ad5 --- /dev/null +++ b/deploy/helm/fleet/values.yaml @@ -0,0 +1,191 @@ +# Default values for the fleet chart (issue #989 / ADR-0049). +# +# READ FIRST — the two invariants this chart encodes: +# +# 1. ONE control-plane replica, forever. fleet's scheduler leases and worker +# semaphore are single-owner; two replicas against one database pair is a +# correctness bug, not a capacity increase. The Deployment pins +# replicas: 1 with strategy Recreate and exposes no replica knob. +# Horizontal scale of WORK = more sandbox pods (bigger node pool), never +# more fleet processes. +# 2. Sandboxes never hold credentials. Sandbox pods are created by the +# control plane with automountServiceAccountToken=false, no env, no +# secret mounts — only the shared workspace claim. MCP credentials stay +# in the control-plane process (ADR-0003). +# +# You build the images yourself — fleet does not publish images: +# control plane: a container image with the `fleet` binary + the client +# bundle (see docs/DEPLOYMENT-KUBERNETES.md §images) +# sandbox: scripts/build-sandbox-image.sh, pushed to your registry + +image: + # REQUIRED: your fleet control-plane image (binary + bundle baked in or + # mounted). Example: 123456789.dkr.ecr.us-east-1.amazonaws.com/fleet:v42 + repository: "" + tag: "" + pullPolicy: IfNotPresent + +imagePullSecrets: [] +# - name: regcred + +serviceAccount: + # Control-plane ServiceAccount (holds the RBAC grant for sandbox pods). + create: true + name: "" + +sandbox: + # REQUIRED: the sandbox image sandbox pods run + # (FLEET_SANDBOX_IMAGE). Must live in a registry the sandbox nodes can + # pull from — a localhost/ build-on-box tag cannot work here. + image: "" + # Backend is what this chart exists for; podman-in-a-pod is NOT a supported + # chart topology (the podman backend's supported home is a VM — + # docs/DEPLOYMENT.md). + backend: kubernetes + kubernetes: + # Namespace for sandbox pods. Empty = the release namespace (the default, + # because the shared workspace PVC cannot be mounted across namespaces — + # use a separate namespace only with storage that supports two PVs on one + # export, e.g. static NFS/EFS PVs). + namespace: "" + # ServiceAccount stamped on sandbox pods (identity for admission + # policies; no token is ever mounted). Created when create=true. + podServiceAccount: + create: true + name: fleet-sandbox + # Pull secret for the sandbox image, when the registry is private. + imagePullSecret: "" + # RuntimeClass for hypervisor-isolated sandboxes (e.g. kata). Preflighted + # fail-closed at boot when set. + runtimeClass: "" + # Node-local seccomp profile applied as a Localhost profile (path + # relative to the kubelet seccomp root). Empty = RuntimeDefault. + seccompProfile: "" + # Pin sandbox pods to a DEDICATED runner node pool: label the pool and + # select it here; taint it and add the matching toleration so nothing + # else lands on it. This is fleet's horizontal scaling story — more + # runner capacity is a bigger pool, never more fleet replicas. + nodeSelector: {} + # fleet.elcanotek.com/pool: sandboxes + tolerations: [] + # - key: fleet.elcanotek.com/sandbox + # operator: Exists + # effect: NoSchedule + # Per-sandbox ceilings, podman-flag format (converted by fleet): + memory: "512m" + cpus: "1.0" + diskGB: 5 + # Warm pool depth (FLEET_SANDBOX_WARM_SIZE; 0 = derive from + # FLEET_MAX_CONCURRENT_AGENTS). + warmSize: 0 + +networkPolicies: + # Ship the deny-all egress policy for sandbox pods labeled + # fleet.elcanotek.com/egress=none. The boot preflight REQUIRES this object + # to exist; disable only if you manage an equivalent policy yourself (and + # then set its name below so the preflight finds it). + denyAll: + create: true + name: fleet-sandbox-deny-all + # Optional egress shaping for OPEN sandbox pods (egress=open): allow DNS + + # anything except the listed CIDRs (typically your cluster + node CIDRs), + # so an open sandbox can reach PyPI but not your Services. Off by default + # because the CIDRs are cluster-specific. + openEgress: + create: false + # CIDRs an open sandbox must NOT reach (cluster Pod/Service/node ranges). + blockedCIDRs: [] + # - 10.0.0.0/8 + # - 172.16.0.0/12 + # - 192.168.0.0/16 + +workspace: + # The shared workspace claim: mounted by the control plane AND every + # sandbox pod at the same absolute path (workspaceRoot below). + create: true + claimName: fleet-workspace + size: 20Gi + # MUST be a ReadWriteMany-capable class in production (EFS, NFS, CephFS). + # On a single-node cluster (kind), a ReadWriteOnce class works because + # every pod lands on the one node — fine for evaluation, not production. + storageClassName: "" + accessModes: + - ReadWriteMany + +data: + # The fleet data dir (FLEET_DATA_DIR): journals, logs, caches. Control + # plane only — sandbox pods never mount it. + create: true + claimName: fleet-data + size: 10Gi + storageClassName: "" + accessModes: + - ReadWriteOnce + +# Absolute path the workspace claim is mounted at in BOTH the control plane +# and every sandbox pod. Same-path mounting is what keeps absolute workspace +# paths meaningful across the control plane, MCP brokers, and sandboxes. +workspaceRoot: /var/lib/fleet/workspace +dataDir: /var/lib/fleet + +config: + # Name of an EXISTING Secret carrying the sensitive env (recommended): + # OPENROUTER_API_KEY, FLEET_CHAT_DATABASE_URL, FLEET_SCHED_DATABASE_URL, + # FLEET_SERVER_TOKEN, and any SMTP/webhook secrets. + # When postgres.enabled=true the DB URLs are wired automatically and may be + # omitted from the secret. + existingSecret: "" + # Extra non-secret env for the control plane, verbatim. + env: {} + # FLEET_MAX_CONCURRENT_AGENTS: "8" + # FLEET_DEFAULT_NETWORK_MODE: "lockdown" + # FLEET_TIMEZONE: "UTC" + +postgres: + # Optional in-cluster PostgreSQL (single replica, one PVC) for evaluation + # and small installs. Production should point config at a managed database + # (RDS, Cloud SQL) via the existingSecret DB URLs instead. + # + # The database password is NOT a values entry: the chart auto-generates a + # random one into the -postgres Secret on first install and reuses + # it on upgrades (helm lookup). To pick your own, pre-create that Secret + # with a `password` key before installing. Rotating it is your job — this + # is the evaluation path. + enabled: false + image: docker.io/library/postgres:17 + size: 10Gi + storageClassName: "" + +web: + # Optional Next.js web tier (you build web/ into an image yourself). + enabled: false + image: "" + port: 3000 + env: {} + +service: + # ClusterIP service exposing chat (8080) and orchestrator (8000). + type: ClusterIP + +ingress: + enabled: false + className: "" + annotations: {} + host: "" + tls: [] + +resources: + # Control-plane resources. Size for your FLEET_MAX_CONCURRENT_AGENTS — the + # sandboxes themselves run as separate pods with their own limits. + requests: + cpu: "1" + memory: 2Gi + limits: + memory: 4Gi + +nodeSelector: {} +tolerations: [] +affinity: {} + +podAnnotations: {} +podLabels: {} diff --git a/docs/DEPLOYMENT-KUBERNETES.md b/docs/DEPLOYMENT-KUBERNETES.md new file mode 100644 index 00000000..42a97acd --- /dev/null +++ b/docs/DEPLOYMENT-KUBERNETES.md @@ -0,0 +1,336 @@ +# Deploying fleet on Kubernetes + +> The first-class Kubernetes path (issue #989 / +> [ADR-0049](adr/0049-kubernetes-backend-split-control-plane.md)): the fleet +> control plane as a single-replica Deployment, with agent sandboxes running as +> **ephemeral pods** via the pluggable sandbox backend +> (`FLEET_SANDBOX_BACKEND=kubernetes`). The single-box podman install +> ([`DEPLOYMENT.md`](DEPLOYMENT.md)) remains the default and an equally +> supported path; come here when Kubernetes is your platform standard. + +## The model + +Same agent loop, same security model, one backend switch: + +| Piece | Where it runs | +| --- | --- | +| fleet control plane (chat + orchestrator + MCP broker) | one Deployment replica — **never more**; the scheduler leases and worker semaphore are single-owner | +| Agent sandboxes (bash, run_python, file ops) | **ephemeral pods**, one per turn / sealed run / persistent-REPL conversation, created and exec'd by the control plane over the apiserver | +| MCP credentials | the control-plane process, always (ADR-0003) — sandbox pods carry no env, no secrets, no service-account token | +| Workspace | one **ReadWriteMany** PVC mounted at the *same absolute path* in the control plane and every sandbox pod | + +``` + browser ──TLS──▶ Ingress ──▶ web (optional) ──▶ fleet Service + │ chat :8080 + │ orchestrator :8000 + ┌────────────────────────────────────────────────────┴──────────┐ + │ fleet control plane pod (1 replica, Recreate) │ + │ agent loop · scheduler · MCP broker (credentials stay here) │ + └───────┬──────────────────────────────┬────────────────────────┘ + │ pods/exec (WebSocket) │ Postgres (managed, or the + ▼ ▼ chart's eval StatefulSet) + fleet-sandbox- pods (ephemeral) chat + sched databases + read-only rootfs · non-root · no caps + no ServiceAccount token · egress by label + │ + └── workspace PVC (RWX) — same path as the control plane +``` + +The backend is selected by `FLEET_SANDBOX_BACKEND` (`podman`, the default, or +`kubernetes`), overriding the bundle manifest's `sandbox.backend` — exactly the +precedence `FLEET_SANDBOX_RUNTIME` / `sandbox.runtime` uses +([SANDBOX-RUNTIMES.md](SANDBOX-RUNTIMES.md)). An unrecognized value refuses to +boot; there is no silent fallback. + +**Fail-closed preflight.** With `kubernetes` selected, fleet refuses to start +unless, at boot: the apiserver is reachable with valid credentials; RBAC grants +`create/get/list/delete pods` and `create pods/exec` in the sandbox namespace; +the workspace claim exists; the sealed-egress NetworkPolicy object exists; and +the RuntimeClass exists when one is configured. `fleet validate-config` runs +the same checks. + +## Build the two images + +fleet publishes no images — you build both and push them to a registry your +nodes can pull from (ECR, GAR, ACR, GHCR, …). A `localhost/` build-on-box tag +cannot work outside a single-node kind cluster. + +**Sandbox image** — the bundle artifact the sandboxes run, unchanged from the +single-box install: + +```sh +scripts/build-sandbox-image.sh +podman tag localhost/fleet-sandbox:latest REGISTRY/fleet-sandbox:v1 +podman push REGISTRY/fleet-sandbox:v1 +``` + +(Or let CI publish it — see `.github/workflows/publish-sandbox-image.yml`.) + +**Control-plane image** — the `fleet` binary plus your client bundle. A +reproducible multi-stage Containerfile (build it from the repo root; the +builder stage's Go minor is pinned to `go.mod` by +`scripts/check_versions_test.go`, so a stale copy of this stage fails CI): + +```dockerfile +# ── build stage ── +FROM docker.io/library/golang:1.27 AS build +WORKDIR /src +COPY go.mod go.sum ./ +RUN go mod download +COPY . . +RUN CGO_ENABLED=0 go build -ldflags "-X github.com/ElcanoTek/fleet/internal/version.version=$(cat VERSION)" -o /out/fleet ./cmd/fleet + +# ── runtime stage ── +FROM registry.fedoraproject.org/fedora-minimal:latest +RUN microdnf install -y git ca-certificates tzdata && microdnf clean all +COPY --from=build /out/fleet /usr/local/bin/fleet +# Bake the client bundle in (the generic one here; substitute your own). +COPY config/default /opt/fleet/client +ENV FLEET_CLIENT_CONFIG_DIR=/opt/fleet/client +RUN mkdir -p /var/lib/fleet && chown 1000:1000 /var/lib/fleet +USER 1000 +ENTRYPOINT ["/usr/local/bin/fleet"] +``` + +An out-of-repo client bundle can be baked into your image the same way, or +mounted from a ConfigMap/volume at `FLEET_CLIENT_CONFIG_DIR` — either satisfies +the engine/bundle split (ADR-0006). + +**Web image** (optional) — build `web/` with its own `next build` stage and set +`web.image` in the chart; the chart wires `CHAT_SERVER_URL` / +`ORCHESTRATOR_URL` at the fleet Service automatically. + +## 15-minute path (kind) + +Prereqs: `kind`, `kubectl`, `helm`, `podman` or `docker` to build images, and +an OpenRouter API key. + +```sh +# 1. A cluster. +kind create cluster --name fleet + +# 2. Build both images and load them into kind. Save the control-plane +# Containerfile from "Build the two images" above as Containerfile.fleet. +scripts/build-sandbox-image.sh +podman save localhost/fleet-sandbox:latest -o /tmp/sandbox.tar +kind load image-archive /tmp/sandbox.tar --name fleet +podman build -t localhost/fleet:dev -f Containerfile.fleet . +podman save localhost/fleet:dev -o /tmp/fleet.tar +kind load image-archive /tmp/fleet.tar --name fleet + +# 3. Secrets (the API key; DB URLs come from the chart's eval Postgres). +kubectl create namespace fleet +kubectl -n fleet create secret generic fleet-secrets \ + --from-literal=OPENROUTER_API_KEY=sk-or-... + +# 4. Install. kind is single-node, so the default (ReadWriteOnce) storage +# class works for the shared workspace — every pod lands on the one node. +helm install fleet deploy/helm/fleet --namespace fleet \ + --set image.repository=localhost/fleet --set image.tag=dev \ + --set image.pullPolicy=Never \ + --set sandbox.image=localhost/fleet-sandbox:latest \ + --set 'workspace.accessModes={ReadWriteOnce}' \ + --set postgres.enabled=true \ + --set config.existingSecret=fleet-secrets + +# 5. Watch boot — the sandbox preflight logs its verdict before serving. +kubectl -n fleet logs deploy/fleet -f | grep -E 'sandbox|preflight' + +# 6. Talk to it, and watch a sandbox pod appear for the turn. +kubectl -n fleet port-forward svc/fleet 8080:8080 & +kubectl -n fleet get pods -l app.kubernetes.io/name=fleet-sandbox -w +``` + +A chat turn that runs bash or python creates a `fleet-sandbox-` pod, +execs into it, and deletes it when the turn ends. Cancelling a turn deletes +the pod immediately (zero grace) — the same poison-and-retire containment the +podman backend guarantees (#796). + +## Production checklist + +1. **Storage: the workspace claim must be ReadWriteMany** (EFS, NFS, CephFS, + Azure Files). A ReadWriteOnce class only works when every pod shares one + node (kind). Verify: + `kubectl -n fleet get pvc fleet-workspace -o jsonpath='{.spec.accessModes}'`. +2. **Database: managed Postgres.** Put `FLEET_CHAT_DATABASE_URL` and + `FLEET_SCHED_DATABASE_URL` in your `config.existingSecret` and leave + `postgres.enabled=false`. The chart's Postgres is an evaluation + convenience: one replica, one PVC, no backups. +3. **NetworkPolicy enforcement is your CNI's job.** fleet verifies the + deny-all policy *object* exists; only a CNI that implements NetworkPolicy + (Calico, Cilium, the EKS VPC CNI's network-policy agent, GKE Dataplane V2, + Azure CNI with policy) makes it real. Verify from a sealed sandbox: + ```sh + kubectl -n fleet run seal-test --restart=Never --rm -it \ + --labels=app.kubernetes.io/name=fleet-sandbox,fleet.elcanotek.com/egress=none \ + --image=busybox -- wget -T 5 -q -O- https://example.com && echo "NOT SEALED" + ``` + A CNI that enforces the policy times that request out. +4. **Shape open-sandbox egress.** Non-lockdown sandboxes are labeled + `egress=open` and unrestricted by default (they need PyPI etc.). Set + `networkPolicies.openEgress.create=true` with your cluster/node CIDRs in + `blockedCIDRs` so an open sandbox can reach the internet but not your + Services. +5. **Registry, not build-on-box.** Both images in a registry the nodes pull + from; set `sandbox.kubernetes.imagePullSecret` for private registries (on + EKS, node-role ECR access covers sandbox pulls without a secret). +6. **Hypervisor isolation** (optional): install Kata Containers on the sandbox + nodes, create a `kata` RuntimeClass, set + `sandbox.kubernetes.runtimeClass=kata`. Preflighted fail-closed, mirroring + `FLEET_SANDBOX_RUNTIME` (ADR-0010). Note `FLEET_SANDBOX_RUNTIME` itself is + a podman knob and is **refused** under this backend. +7. **One replica.** Do not add an HPA or `replicas: 2` for the control plane. + Scale work by raising `FLEET_MAX_CONCURRENT_AGENTS` and giving the sandbox + namespace more node capacity; scale the control plane vertically + (`resources` in values). Size for peak: the control plane runs the agent + loop + brokers; the sandboxes' cost lives in their own pods, so warm-pool + pods hold their requests while parked — size `FLEET_SANDBOX_WARM_SIZE` + accordingly. +8. **Give runners their own node pool.** Label (and usually taint) a dedicated + pool, then set `sandbox.kubernetes.nodeSelector` + `.tolerations` in the + chart — sandbox pods pin there and autoscale the pool, while the control + plane stays on your general nodes. This is the horizontal scaling story: + more runner capacity is a bigger pool, never a second fleet. +9. **Run `fleet validate-config`** (`kubectl -n fleet exec deploy/fleet -- + fleet validate-config`) after any config change: it runs the same + fail-closed preflight boot does, plus everything else the verb checks. + +## Day-2 operations + +The systemd timers and host scripts (`bootstrap.sh`, `fleet update`, +`scripts/doctor.sh`, `fleet timers install`) are single-box tooling and do not +apply here ([TIMERS.md](TIMERS.md)). Their cluster equivalents: + +| Single-box | Kubernetes | +| --- | --- | +| `fleet update` | build + push a new control-plane image, `helm upgrade` (strategy Recreate = a brief restart; in-flight turns drain per `FLEET_SHUTDOWN_GRACE_SECONDS`) | +| `fleet-backup.timer` | a CronJob running `fleet backup --db=all --prune` — or skip it entirely by using managed-database backups (RDS/Cloud SQL snapshots), the recommended posture | +| `fleet-maintenance.timer` | a CronJob running `fleet cleanup` daily | +| journald | `kubectl logs` / your log stack; set `FLEET_LOG_FILE` only if you also mount somewhere rotatable | +| Grafana node dashboards | scrape the control plane's `/metrics` (orchestrator port). NOTE it is **admin-API-key gated** (`X-API-Key`) — cost/token data must not be public — and stock Prometheus cannot send custom headers, so use a scraper that can (Grafana Alloy, vmagent) or a small header-injecting sidecar. Sandbox pods are ordinary pods your cluster metrics already see | + +Minimal backup CronJob (only needed when you run the eval Postgres or want +`fleet backup`'s application-level dumps next to managed snapshots): + +```yaml +apiVersion: batch/v1 +kind: CronJob +metadata: {name: fleet-backup, namespace: fleet} +spec: + schedule: "0 2 * * *" + jobTemplate: + spec: + template: + spec: + restartPolicy: Never + containers: + - name: backup + image: REGISTRY/fleet:v1 # the control-plane image + args: ["backup", "--db=all", "--prune"] + envFrom: [{secretRef: {name: fleet-secrets}}] + volumeMounts: [{name: data, mountPath: /var/lib/fleet}] + volumes: + - name: data + persistentVolumeClaim: {claimName: fleet-data} +``` + +## Provider notes + +- **EKS**: EFS (via the EFS CSI driver) is the standard RWX workspace class; + ECR for both images (node-role pull, no secret needed); enable the VPC CNI + network-policy agent or run Calico/Cilium so the deny-all policy is + enforced; ALB via the AWS Load Balancer Controller for `ingress`. RDS for + Postgres. Kata needs a bare-metal (`*.metal`) node group for `/dev/kvm`. +- **GKE**: Filestore CSI for RWX; Dataplane V2 enforces NetworkPolicy natively; + Artifact Registry with Workload Identity; Cloud SQL. +- **AKS**: Azure Files (NFS) for RWX; enable Azure Network Policy or Cilium; + ACR with the kubelet identity; Azure Database for PostgreSQL. +- **Bare metal / on-prem**: any NFS/CephFS class for RWX; Calico or Cilium for + policy; your own registry. + +## Configuration reference + +Every knob can come from env (the chart sets these) or the bundle manifest's +`sandbox:` block (env wins, field by field): + +| Env | Manifest | Meaning | +| --- | --- | --- | +| `FLEET_SANDBOX_BACKEND` | `sandbox.backend` | `podman` (default) or `kubernetes` | +| `FLEET_SANDBOX_K8S_NAMESPACE` | `sandbox.kubernetes.namespace` | sandbox pod namespace (default: the control plane's own, else `fleet-sandboxes`) | +| `FLEET_SANDBOX_K8S_WORKSPACE_CLAIM` | `…workspace_claim` | **required** — the shared RWX PVC name | +| `FLEET_SANDBOX_K8S_SERVICE_ACCOUNT` | `…service_account` | identity stamped on sandbox pods (no token is mounted) | +| `FLEET_SANDBOX_K8S_IMAGE_PULL_SECRET` | `…image_pull_secret` | pull secret for the sandbox image | +| `FLEET_SANDBOX_K8S_RUNTIME_CLASS` | `…runtime_class` | hypervisor isolation (kata); preflighted | +| `FLEET_SANDBOX_K8S_SECCOMP_PROFILE` | `…seccomp_profile` | node-local Localhost seccomp profile; empty = RuntimeDefault | +| `FLEET_SANDBOX_K8S_KUBECONFIG` | `…kubeconfig` | out-of-cluster auth (token / client-cert kubeconfigs only); empty = in-cluster | +| `FLEET_SANDBOX_K8S_NETWORK_POLICY` | `…network_policy` | deny-all policy name the preflight requires (default `fleet-sandbox-deny-all`) | +| `FLEET_SANDBOX_K8S_NODE_SELECTOR` | `…node_selector` | pin sandbox pods to a dedicated runner pool — env form `"pool=sandboxes,arch=amd64"`, manifest form a map; a malformed value refuses to boot | +| `FLEET_SANDBOX_K8S_TOLERATIONS` | `…tolerations` | tolerations for a tainted runner pool — env form a JSON array of `{key,operator,value,effect}`, manifest form a YAML list | + +The shared sandbox knobs apply to both backends: `FLEET_SANDBOX_IMAGE`, +`FLEET_SANDBOX_MEMORY` / `_CPUS` (converted to pod resource limits), +`FLEET_SANDBOX_DISK_GB` (the pod's ephemeral-storage limit), +`FLEET_SANDBOX_WARM_SIZE` / `_WARM_TTL` (the warm pool holds pre-started +pods), and the python REPL knobs. + +## Troubleshooting + +- **Boot fails with "kubernetes sandbox preflight"** — the message names the + exact missing piece (RBAC verb, claim, NetworkPolicy, RuntimeClass). The + chart's `fleet-runner` Role carries every needed verb; if you wrote your own + RBAC, diff it against `deploy/helm/fleet/templates/rbac.yaml`. +- **First turn fails with `ErrImagePull` / `ImagePullBackOff`** — the sandbox + image ref isn't pullable *from the nodes* (fleet fails the pod fast with the + kubelet's reason instead of burning the start timeout). Check the ref and + `sandbox.kubernetes.imagePullSecret`. +- **`sandbox pod … not ready before start timeout`** — usually scheduling + (no node fits the sandbox requests) or a slow first pull; `kubectl describe + pod fleet-sandbox-…` shows which. +- **Workspace files owned by the wrong uid** — the claim's storage class must + honor `fsGroup` (1000) or be provisioned world-writable at the root; both + the control plane and sandbox pods run uid/gid 1000. +- **A sealed turn can still reach the network** — your CNI is not enforcing + NetworkPolicy (checklist item 3). The policy *object* existing is not + enforcement. +- **Turn cancelled but you want proof nothing survived** — cancellation + deletes the pod with zero grace; `kubectl get pods -l + app.kubernetes.io/name=fleet-sandbox` should not show the pod after the + cancel completes. A pod that lingers past a crash is reclaimed by the + boot-time orphan sweep on the next control-plane start. + +## Honest scope — what the kubernetes backend does differently + +Recorded here so nobody discovers them in production: + +- **Egress sealing is delegated.** Podman's `--network=none` is a kernel + namespace with no interface; the k8s equivalent is a label + (`fleet.elcanotek.com/egress=none`) matched by a deny-all NetworkPolicy. + fleet verifies the object exists — it cannot verify the CNI enforces it. +- **`FLEET_DEFAULT_NETWORK_MODE=allowlisted` is refused** at boot: the + host-side egress proxy (ADR-0012) is unreachable from pods. Use `lockdown` + or `open` + NetworkPolicy shaping. +- **No per-pod pids limit.** `FLEET_SANDBOX_PIDS` has no Pod-spec equivalent; + runaway process counts are bounded by pod memory/CPU limits and node + `podPidsLimit` if you configure the kubelet. +- **No per-sandbox resource telemetry (#263).** `podman stats` has no + in-process counterpart here; task resource summaries are absent. Use your + cluster's metrics stack on the `fleet-sandbox` pods. +- **The bundled seccomp profile does not apply.** Pods run `RuntimeDefault`, + or a profile you install on the nodes yourself via + `FLEET_SANDBOX_K8S_SECCOMP_PROFILE`. Setting the podman + `FLEET_SANDBOX_SECCOMP_PROFILE` under this backend refuses to boot rather + than being silently ignored. +- **Supporting-doc bind mounts don't apply.** The podman backend bind-mounts + persona/protocol dirs same-path into containers; a pod only mounts the + workspace claim. In-sandbox reads of those host paths degrade exactly like + the podman missing-dir case. +- **Disk quota is per-pod ephemeral storage**, which caps the writable layer + and scratch emptyDirs — a *stronger* cap than podman's per-file ulimit — but + the workspace claim is still unbounded by it, same as the bind mount is + under podman: many files still add up. +- **Warm-pool pods hold cluster resources while parked.** Requests equal + limits; size `FLEET_SANDBOX_WARM_SIZE` accordingly. +- **kind e2e is a documented walkthrough, not a CI job.** CI lints and + template-renders the chart (`helm` job) and unit-tests the backend against a + fake apiserver (including exec streaming and the poison path); it does not + stand up a cluster. diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 177c20b3..28e49102 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -15,13 +15,17 @@ proxies, server-side over loopback, to the two Go backends the single process boots (chat on `127.0.0.1:8080`, orchestrator on `127.0.0.1:8000`). Caddy fronts the web app with TLS; the backends stay loopback-only. -> **Your platform standard is Kubernetes?** fleet's shipped target is this -> single-VM/systemd model ([ADR-0004](adr/0004-single-box-vm-native-deployment.md)), -> and no chart or manifest lives in the tree. For an operator recipe that keeps -> the one-process/one-node model intact inside EKS — one pod on one big node, -> Podman running *inside* it, nothing split across worker nodes — see -> [`docs/EKS-DEPLOYMENT.md`](EKS-DEPLOYMENT.md). It is hand-verified, not -> CI-exercised. +> **Your platform standard is Kubernetes?** fleet's default install remains +> this single-VM/systemd model +> ([ADR-0004](adr/0004-single-box-vm-native-deployment.md)), but Kubernetes is +> now a first-class path +> ([ADR-0049](adr/0049-kubernetes-backend-split-control-plane.md)): a Helm +> chart (`deploy/helm/fleet`) runs the control plane as a single-replica +> Deployment, and `FLEET_SANDBOX_BACKEND=kubernetes` runs every agent sandbox +> as an ephemeral pod — no Podman on the node, no privileged pod. See +> [`docs/DEPLOYMENT-KUBERNETES.md`](DEPLOYMENT-KUBERNETES.md): a 15-minute +> kind path, the production checklist, provider notes (EKS/GKE/AKS), and +> day-2 operations. > **Single-host by design.** Scheduled-task crash recovery uses single-owner > database leases and the worker-pool concurrency cap is a per-process semaphore — diff --git a/docs/EKS-DEPLOYMENT.md b/docs/EKS-DEPLOYMENT.md deleted file mode 100644 index 5b2de940..00000000 --- a/docs/EKS-DEPLOYMENT.md +++ /dev/null @@ -1,1334 +0,0 @@ -# Deploying fleet on Amazon EKS (one pod, one big node) - -> Operator recipe for organizations whose platform standard is Kubernetes. It -> keeps fleet's single-process, single-node model intact — one pod on one large -> node, scaled vertically — and does **not** try to spread the Podman sandboxes -> across worker nodes. For the supported single-host install see -> [`docs/DEPLOYMENT.md`](DEPLOYMENT.md). - -## Read this first (scope, honesty, and what is not shipped) - -- **fleet's shipped deployment target is a single VM under systemd** - ([ADR-0004](adr/0004-single-box-vm-native-deployment.md)). That ADR stands. - There is no Helm chart, no operator, and no k8s manifest in this repo, and - **CI does not exercise this path** — the CI matrix builds and tests the - systemd/single-host model. Everything below is a hand-verified recipe you own - and must validate on your own cluster. -- **No fleet container image ships either.** `deploy/` contains systemd units, not - images. You build two images yourself (§3): the fleet runtime image (Go binary - **+ Podman inside it**) and the Next.js web image. The sandbox image stays what - it already is — a per-client *bundle* artifact. -- **One pod. One replica. Forever.** Scheduled-task crash recovery uses - single-owner database leases and the concurrency cap is a per-process - semaphore. Two fleet pods against one pair of databases is a **correctness - bug**, not a capacity increase. No `Deployment` with rolling updates, no HPA, - no `replicas: 2`. -- **The sandbox stays local to the process.** Every agent tool call's data plane - runs in a rootless-Podman container that `agentcore` starts and `podman exec`s - into on the same host as the run loop - ([ADR-0002](adr/0002-mandatory-rootless-podman-sandbox.md)); the remote worker - registry was deliberately removed - ([ADR-0011](adr/0011-remove-worker-node-registry.md)). There is no seam that - dispatches a sandbox to another node, so "put the runners on their own node - group" is not a configuration — it would be a rewrite. This guide runs Podman - **inside the fleet pod**, which is why the pod needs the privileges in §2. -- **What you gain** by doing this at all: your existing ECR/IRSA/ALB/Secrets - Manager/observability plumbing, one node group to patch, and node-failure - rescheduling. **What you give up** versus the systemd path: `bootstrap.sh`, - `fleet update`, and `scripts/doctor.sh` all assume a systemd host — you replace - them with image rebuilds and `kubectl exec` (§9). - -## The objections a Kubernetes-native reviewer will raise - -Answer these before the design review, not during it. Each links to the section -that implements it. - -| "This isn't Kubernetes-native because…" | Answer | -|---|---| -| "…there's no Helm chart / it's not GitOps" | Package the §7 manifests as Kustomize or a thin Helm chart and sync with Argo CD or Flux — [§7 GitOps](#packaging-these-manifests-for-gitops). Two Argo-specific gotchas are called out there. | -| "…a privileged pod will never pass admission" | It won't under `restricted`/`baseline` Pod Security Standards. You need a labelled namespace and a scoped policy exception — [§7 admission control](#namespace-admission-control-and-identity). If your org forbids privileged pods outright, [§2](#if-your-policy-forbids-privileged-pods) is the unprivileged variant and its costs. | -| "…one replica isn't highly available" | Correct, and it cannot be: single-owner task leases + a per-process semaphore. HA here means fast, *graceful* recovery, not zero downtime — [§6 availability](#az-pinning-node-loss-and-what-ha-means-here) states the RTO/RPO plainly. | -| "…we can't autoscale it" | Scale vertically: raise `FLEET_MAX_CONCURRENT_AGENTS` and the pod resources together ([§6](#resource-requests-count-the-sandboxes)). HPA and VPA are both actively harmful here ([§8](#cluster-integration-gotchas)). | -| "…the workloads are invisible to the cluster" | True and worth naming: sandboxes are Podman containers inside the pod, so they never appear in `kubectl get pods` or cAdvisor. Where to see them instead: [§8](#cluster-integration-gotchas). | -| "…it pins itself to one AZ" | It does — `ReadWriteOnce` EBS. Make the node group single-AZ deliberately rather than discovering it during an incident ([§6](#az-pinning-node-loss-and-what-ha-means-here)). | -| "…NetworkPolicy can't govern what the agent runs" | It can. Sandbox egress traverses the pod's network namespace via the rootless network helper, so pod-level NetworkPolicy applies to agent-executed code too ([§7](#networkpolicy), [§8](#cluster-integration-gotchas)). | -| "…secrets are in a `Secret`" | Swap in External Secrets Operator or the Secrets Store CSI driver ([§7](#secrets)). fleet's own guarantee is stronger than either: MCP credentials are brokered host-side and never enter a sandbox. | -| "…nothing here is CI-tested" | Also true. Run [§10](#10-verification-checklist) as an acceptance gate in your own pipeline; that is the substitute. | - -## 1. Topology - -Everything that was a process on the single box becomes a container in **one -pod**, so the loopback wiring the code expects still holds (containers in a pod -share a network namespace, so `127.0.0.1:8080` from the web container reaches the -chat listener): - -``` - ┌──────────────── EKS node (dedicated, one big instance) ───────────────┐ - Internet ─TLS─▶ ALB ──┼─▶ Service :3000 ─▶ pod │ - (ACM cert) │ ┌───────────────────────────────────────────────────────────┐ │ - │ │ container: web Next.js, 0.0.0.0:3000 │ │ - │ │ │ server-side proxy over loopback │ │ - │ │ ├─▶ 127.0.0.1:8080 chat ┐ │ │ - │ │ └─▶ 127.0.0.1:8000 orchestr. ┘ container: fleet │ │ - │ │ (one process: │ │ - │ │ chat + orchestrator │ │ - │ │ + scheduler + pool) │ │ - │ │ │ podman (rootless, │ │ - │ │ │ in-container) │ │ - │ │ ├─▶ sandbox ctr 1 │ │ - │ │ ├─▶ sandbox ctr 2 │ │ - │ │ └─▶ … up to │ │ - │ │ FLEET_MAX_CONCURRENT_AGENTS│ │ - │ └───────────────────────────────────────────────────────────┘ │ - └──────────────────────────────────────────────────────────────────────┘ - │ - └─▶ RDS PostgreSQL (two databases: chat + sched) -``` - -The Go listeners stay **loopback-only**. The orchestrator in particular is -impersonation-load-bearing and must remain on `127.0.0.1` — do not bind it to the -pod IP (see §8 for how metrics scraping works without breaking that). - -## 2. The hard part: rootless Podman inside a pod - -The sandbox is mandatory and fails closed, so the pod must be able to run -rootless Podman. Concretely fleet shells out to -`podman run --userns=keep-id:uid=1000,gid=1000 --read-only --cap-drop=ALL ---security-opt=no-new-privileges --security-opt seccomp=… --memory=… --cpus=… ---pids-limit=… …` and then `podman exec`s each tool call into it. Network posture -is per-turn: normal turns pass **no** `--network` flag (podman's rootless default -— pasta on ≥ 5.0, slirp4netns before it), lockdown and scheduled runs get -`--network=none`, and the allowlisted-egress posture explicitly requests -`--network=slirp4netns:allow_host_loopback=true`. That needs, inside the fleet -container: - -| Requirement | Why | How | -|---|---|---| -| `/etc/subuid` + `/etc/subgid` ranges for the container's user | `--userns=keep-id` maps uids into the range; without it Podman fails with a `newuidmap` mapping error | baked into the image (§3) | -| `newuidmap`/`newgidmap` with their file capabilities intact | performs the uid/gid mapping | `shadow-utils` in the image **and** `allowPrivilegeEscalation: true` (file caps are neutralized by `NoNewPrivileges` — the same reason `deploy/fleet.service` sets `NoNewPrivileges=no`) | -| a writable, **persistent** graph root (`$HOME/.local/share/containers`) | holds the ~1.5 GB sandbox image + per-container writable layers | the PVC mounted at `/var/lib/fleet` (§5) | -| an overlay-capable storage driver | `vfs` copies the whole ~1.5 GB image per container start — fatal for a per-turn warm pool | native `overlay` (privileged) or `fuse-overlayfs` + `/dev/fuse` | -| `/dev/net/tun` | the rootless network helper: **pasta** on Podman ≥ 5.0 (podman's own default, used by normal turns), or **slirp4netns**, which the allowlisted-egress posture specifically requires | privileged, or a device plugin | -| a **writable cgroup subtree** | otherwise Podman silently ignores `--memory`/`--cpus`, so the per-sandbox caps and per-task `sandbox_limits` **do not bind** | privileged (rw `/sys/fs/cgroup`); the analogue of `Delegate=yes` in the systemd unit | -| cgroup **v2** on the node | project-quota/limit behavior above | Amazon Linux 2023 nodes default to cgroup v2 | - -### Recommendation: run the fleet container privileged on a dedicated node - -```yaml -securityContext: - privileged: true - allowPrivilegeEscalation: true - runAsUser: 1000 # the image's fleet user, NOT root - runAsGroup: 1000 -``` - -**The container must run as uid 1000, not root — even privileged.** Podman -running as real root is *rootful*, and rootful Podman **ignores** -`--userns=keep-id`. The sandbox's uid 1000 then no longer maps to the process -that owns the workspace directory, so the agent can neither `chdir` into its -per-conversation workspace nor write files there — the failure the -`keep-id`/same-path invariant tests in `internal/sandbox` exist to catch. Keep -`runAsUser: 1000` and give the image a fixed uid-1000 user with subuid ranges -(§3b). - -This is the configuration that reliably satisfies all six rows above. It is also -the honest trade: **a privileged container is not a security boundary**, so the -security model becomes "the *node* is the blast radius, and the pod owns it." -That is materially weaker than the systemd deployment, where the fleet process is -an unprivileged system user. Mitigate deliberately: - -- **Dedicate the node group to fleet** — taint it and schedule nothing else there - (§6). Never co-schedule other tenants' workloads. -- **Block IMDS from pods** on that node group - (`--metadata-options http-put-response-hop-limit=1`) so agent-executed code - cannot mint the node role's credentials. Give fleet its own IRSA role with only - what it needs (ECR pull; Secrets Manager read if you use it). -- **Give the node role the minimum**, and keep the cluster's own secrets out of - the namespace. -- **NetworkPolicy** on the namespace: ingress only from the ALB target group, - egress only to RDS, your model provider, and the MCP endpoints you intend. -- The **inner** hardening is unchanged and still does the real work per turn: - read-only rootfs, `--cap-drop=ALL`, no-new-privileges, the default-deny seccomp - profile, `--network=none` for lockdown/scheduled runs, per-container - memory/CPU/pid caps, and the credential broker keeping MCP secrets host-side - (they never enter a sandbox — [ADR-0003](adr/0003-host-side-mcp-credential-brokering.md)). - -### If your policy forbids privileged pods - -The unprivileged variant needs, at minimum, `SYS_ADMIN` plus device access to -`/dev/fuse` and `/dev/net/tun` (a device plugin such as smarter-device-manager, -because containerd's default device cgroup denies both), `fuse-overlayfs` as the -driver, and a `RuntimeDefault` seccomp profile that permits `unshare`/`clone` -with `CLONE_NEWUSER`. Expect to fight the cgroup-delegation row above — and -**verify that `--memory` actually binds** (§10) rather than assuming it, because -if it doesn't, a `pandas` job takes the whole pod down instead of one sandbox. - -`kata`/`libkrun` microVM runtimes ([`docs/SANDBOX-RUNTIMES.md`](SANDBOX-RUNTIMES.md)) -need read-write `/dev/kvm` inside the pod. On EC2 that means a `.metal` instance -(nested KVM is not exposed on normal instance types), plus device access. Fleet's -boot preflight is fail-closed, so a missing `/dev/kvm` aborts startup rather than -silently downgrading to a shared kernel. Leave `sandbox.runtime` at the default -unless you have committed to metal nodes. - -## 3. Build the images - -### 3a. Sandbox image → ECR - -Unchanged from the single-host path: the Containerfile is a **bundle** artifact -(`/sandbox/Containerfile`), and fleet **never builds it at startup**. -Build and push it in CI — the repo already ships the reusable workflow -`.github/workflows/publish-sandbox-image.yml` (`workflow_call`) for exactly this, -which builds with `scripts/build-sandbox-image.sh` and pushes an immutable -`{git-sha}` tag. It exposes the pushed `image_ref` and `image_digest` as workflow -outputs, so a deploy job can consume the exact digest this section wants without -re-deriving it. Point it at ECR instead of GHCR, or mirror the GHCR tag into ECR. - -**Pull credentials depend on the package's visibility**, which in this org -follows the publishing repo (measured 2026-08-20): the images from the public -`fleet` and `example-config` repos are anonymously pullable, so a cluster needs -no `imagePullSecret` for them; the client-bundle images come from private repos -and do. GitHub's docs describe a private-by-default that these packages did not -follow, so verify a new package's visibility rather than assuming — an image you -expect to pull anonymously failing with a 403 at pod start is the symptom. - -The workflow publishes but does **not** pin: adoption is the explicit step -below. (It used to open a PR pinning `sandbox.image` in the client repo; that -step never once succeeded and was removed on 2026-08-20 — see the reusable -workflow's header.) - -Then set `sandbox.image` in the bundle's `manifest.yaml` to the immutable ref, or -override it per deployment with `FLEET_SANDBOX_IMAGE` -(`.dkr.ecr..amazonaws.com/fleet-sandbox@sha256:…`). Pin by digest — -`:latest` in a rebuilt-nightly registry means a turn's execution environment can -change under you. - -### 3b. fleet runtime image - -Podman lives in this image. A Fedora base keeps you on the same `crun`/ -`slirp4netns`/`fuse-overlayfs` versions the project develops against: - -```dockerfile -# Containerfile.fleet — build with: podman build -f Containerfile.fleet -t /fleet: . -FROM golang:1.27 AS build -WORKDIR /src -COPY go.mod go.sum ./ -RUN go mod download -COPY . . -RUN make build # → ./fleet and ./fleet-admin - -FROM fedora:44 -# podman + the rootless stack fleet actually invokes; curl for the exec probes (§7). -# Install BOTH rootless network helpers: passt/pasta is podman >= 5.0's default -# (normal turns), and slirp4netns is required by the allowlisted-egress posture — -# where a missing binary now aborts boot with a fail-closed preflight rather than -# erroring on every turn. awscli2 is for the ECR-login init container (§7); curl -# is for the exec probes. -RUN dnf install -y --setopt=install_weak_deps=False \ - podman crun conmon passt slirp4netns fuse-overlayfs containers-common \ - shadow-utils catatonit iptables-nft git curl ca-certificates awscli2 \ - && dnf clean all -# Fixed unprivileged user, uid 1000 — matches --userns=keep-id:uid=1000 and the -# sandbox image's USER, so bind-mounted workspace files line up from both sides. -RUN useradd --uid 1000 --home-dir /var/lib/fleet --shell /sbin/nologin fleet \ - && echo 'fleet:100000:65536' > /etc/subuid \ - && echo 'fleet:100000:65536' > /etc/subgid -# Same rootless-Podman settings scripts/bootstrap.sh writes for the service user: -# cgroupfs avoids needing a systemd user D-Bus session; the file events logger -# avoids journald permissions. fleet also passes --cgroup-manager=cgroupfs itself. -RUN install -d -o fleet -g fleet -m 0755 /etc/containers \ - && printf '[engine]\ncgroup_manager = "cgroupfs"\nevents_logger = "file"\n' \ - > /etc/containers/containers.conf -COPY --from=build /src/fleet /usr/local/bin/fleet -# The client config bundle. Bake it in for an immutable deploy (recommended) or -# mount it from a PVC/initContainer git clone; either way it must be WRITABLE by -# uid 1000, because the sandbox bind-mounts bundle dirs with SELinux relabeling. -COPY --chown=fleet:fleet config/default /opt/fleet/client -USER fleet -ENV HOME=/var/lib/fleet \ - XDG_RUNTIME_DIR=/var/lib/fleet/run \ - FLEET_CLIENT_CONFIG_DIR=/opt/fleet/client -WORKDIR /var/lib/fleet -# `fleet serve` is the explicit server verb (bare `fleet` also serves). -ENTRYPOINT ["/usr/local/bin/fleet", "serve"] -``` - -Notes that matter: - -- **`XDG_RUNTIME_DIR` must be writable and on the PVC-or-emptyDir**, not on the - read-only image layer — it holds per-container runtime state. -- **`FLEET_ENV_FILE` is a real choice, not a leftover.** With config injected as - pod env (the manifests below), leave it unset — but know the consequence: - config hot-reload re-reads the **env file** and honors boot's - process-env-over-file precedence, so anything pinned in the pod's environment - is **fixed until the pod restarts**, and `SIGUSR2` / - `POST /admin/reload-config` will report it under `skipped`. That is the right - trade for immutable-config deployments. If you want the reload path - ([`docs/CONFIG-RELOAD.md`](CONFIG-RELOAD.md)) to work, mount the credential - file from a Secret instead (e.g. `/etc/fleet/fleet.env`), point - `FLEET_ENV_FILE` at it, and do **not** also inject those keys as env vars. -- **Leave `FLEET_LOG_FILE` unset** so the process log goes to stdout/stderr for - your normal cluster log pipeline. -- Keep the bundle **writable by uid 1000** — the sandbox mounts `protocols/`, - `personas/`, `skills/`, and `system_prompts/` with `:Z`, which needs to write - the `security.selinux` xattr. This is the same reason `deploy/fleet.service` - lists `/opt/fleet/client` in `ReadWritePaths`. - -### 3c. web image - -```dockerfile -FROM node:24 AS build -WORKDIR /app -COPY web/package*.json ./ -RUN npm ci -COPY web/ . -RUN npm run build -FROM node:24-slim -WORKDIR /app -COPY --from=build /app ./ -ENV NODE_ENV=production PORT=3000 -USER node -CMD ["npm", "run", "start"] -``` - -## 4. PostgreSQL - -Use RDS (or Aurora PostgreSQL). fleet needs **two databases** in the same -cluster — chat and sched are deliberately separate -([ADR-0005](adr/0005-separate-chat-and-sched-databases.md)) — and each service -**self-migrates on first start**, so create empty databases and roles only: - -```sql -CREATE ROLE chat LOGIN PASSWORD '…'; CREATE DATABASE chat OWNER chat; -CREATE ROLE sched LOGIN PASSWORD '…'; CREATE DATABASE sched OWNER sched; -``` - -``` -FLEET_CHAT_DATABASE_URL=postgres://chat:…@:5432/chat?sslmode=require -FLEET_SCHED_DATABASE_URL=postgres://sched:…@:5432/sched?sslmode=require -``` - -Use `sslmode=require` or stricter (`verify-full` with the RDS CA bundle mounted). -Both pools are **critical readiness checks** — if either is down, `/readyz` -returns 503, which is exactly the signal you want the ALB to see. Tune -`FLEET_CHAT_DB_MAX_CONNS` / `FLEET_SCHED_DB_MAX_CONNS` against the instance -class's connection limit. `fleet migrate status` (via `kubectl exec`) reports -applied vs pending migrations; see [`docs/MIGRATIONS.md`](MIGRATIONS.md). - -Running Postgres in-cluster works but buys you a second stateful single-writer -workload to babysit; managed is the better trade here, and it lowers the pod's -base footprint. - -## 5. Storage - -One `ReadWriteOnce` EBS volume mounted at `/var/lib/fleet` carries **everything -stateful in the pod**: the rootless Podman graph root, the per-conversation -workspaces, the data dir (attachments/uploads, audit), and `XDG_RUNTIME_DIR`. - -```yaml -apiVersion: storage.k8s.io/v1 -kind: StorageClass -metadata: - name: fleet-gp3-xfs -provisioner: ebs.csi.aws.com -parameters: - type: gp3 - iops: "6000" - throughput: "500" - fsType: xfs # xfs + prjquota is what makes --storage-opt size work -mountOptions: - - prjquota -allowVolumeExpansion: true -volumeBindingMode: WaitForFirstConsumer -``` - -**Why `xfs` + `prjquota`:** the two disk caps are **layered, not either/or**. A -per-file `--ulimit fsize` cap is applied on every container regardless of -filesystem, and it is what bounds writes to the workspace bind mount. On top of -that, fleet adds `--storage-opt size=…` — a hard **total** cap on the writable -layer — but Podman only accepts that on a quota-capable driver (overlay+xfs with -pquota, btrfs, zfs — **not** overlay+ext4, and not vfs). fleet probes this once at -boot; where it isn't supported, the total-size cap is simply omitted and an agent -can still fill the writable layer with many individually-legal files. The omission -is logged at startup. Making the probe succeed is the point of this StorageClass. - -Sizing: the sandbox image (~1.5 GB) + one writable layer per concurrent sandbox -(`FLEET_SANDBOX_DISK_GB`, default 5 GiB each) + persistent workspaces + uploads -(`FLEET_UPLOAD_MAX_BYTES`, default 1 GiB per file). Start from the disk column of -the sizing table in [`docs/DEPLOYMENT.md`](DEPLOYMENT.md#choosing-a-host-sizing) -and add your workspace retention. `allowVolumeExpansion` matters — a full volume -is an outage. - -*Optional:* on an instance with local NVMe you can put the graph root on the -instance store (a `hostPath` plus a `storage.conf` `graphroot`) and keep only -workspaces/data on the PVC. Images and warm-container layers are reconstructible, -so ephemeral is fine for them, and you get much faster container starts. Verify -the quota probe still passes on that filesystem. - -## 6. Node group and scheduling - -One dedicated managed node group, one instance, nothing else on it: - -``` -eksctl create nodegroup --cluster --name fleet \ - --node-type m7i.12xlarge --nodes 1 --nodes-min 1 --nodes-max 1 \ - --node-zones \ - --node-taints dedicated=fleet:NoSchedule \ - --node-labels workload=fleet \ - --node-volume-size 200 --node-volume-type gp3 \ - --metadata-options httpPutResponseHopLimit=1 -``` - -- **AMI: Amazon Linux 2023** (cgroup v2 by default, ordinary writable - containerd). Bottlerocket and other minimal/immutable AMIs are **untested for - nested Podman** here — if you must, prove out §10's checks first. If your nodes - enforce SELinux, confirm the `:z`/`:Z` relabels the sandbox performs actually - succeed; AL2023's permissive default is what this recipe assumes. -- **Instance family:** memory-per-vCPU is the binding constraint (~1.5–3 GB of - RAM per concurrent agent for `pandas`/`matplotlib` work), so `r7i`/`r7a` beats - `c7i`. Size from the arithmetic in the next subsection, not from the vCPU - count: the 32-agent worked example used below and in the appendix needs - ~36 vCPU / 72 GiB once the base and web tier are counted, so **`m7i.12xlarge` - (48 vCPU / 192 GiB)** fits it with headroom while a 32-vCPU instance is already - short. Step up to `r7i.12xlarge` (48/384) if you intend to raise - `FLEET_SANDBOX_MEMORY` to 4–8 GiB for heavy `pandas`/`matplotlib` work, and to - `r7i.24xlarge` (96/768) for `FLEET_MAX_CONCURRENT_AGENTS=64` at those per-agent - sizes. Don't buy the memory before you've raised the per-sandbox cap that would - use it — the default is 512 MiB. -- **`maxPods`:** the sandboxes are Podman containers *inside* the pod, so they - don't consume pod IPs or count against `maxPods`. Only fleet's own pod does. -- **Karpenter:** annotate the pod `karpenter.sh/do-not-disrupt: "true"`. Node - consolidation on a single stateful pod means unplanned restarts. -- **Cluster Autoscaler / HPA:** neither applies. Do not attach an HPA. -- **PodDisruptionBudget:** don't set a blocking one (`minAvailable: 1` on a - single-replica workload blocks node drains indefinitely). A single-pod - deployment means node replacement is a **planned downtime window** — the honest - consequence of the single-writer design, same as rebooting the single box. - -### AZ pinning, node loss, and what "HA" means here - -Decide this deliberately — it is the question your reviewer will press hardest on. - -- **The pod is pinned to one Availability Zone.** A `ReadWriteOnce` EBS volume - exists in exactly one AZ, and `WaitForFirstConsumer` binds it where the pod - first scheduled. If your node group spans AZs, a replacement node in a - different AZ **cannot** mount the volume and the pod stays `Pending` with a - volume-node-affinity conflict. Make the node group **single-AZ on purpose** so - a replacement node always lands where the volume is. (EFS as an alternative - gets you cross-AZ at the cost of a network filesystem under the Podman graph - root and per-conversation workspaces — don't.) -- **Node loss does not self-heal quickly.** When a node goes `NotReady`, a - StatefulSet pod is *not* recreated until the old one is confirmed gone — - Kubernetes will not risk two writers, which is the same invariant fleet needs. - Recovery is: the node object is deleted (or you `kubectl delete pod --force`), - the EBS volume detaches, and the new pod attaches it on a fresh node in the - same AZ. Budget minutes, and prefer letting the node group replace the instance - over force-deleting by hand. -- **What HA actually means for this workload:** RTO is one pod restart plus - volume reattach; RPO for conversations, tasks, and run history is your RDS - backup window (in-flight turns are lost, and the graceful drain is what keeps - that number near zero for planned restarts). There is no zero-downtime rolling - upgrade, on EKS or on the single box — that is a property of the single-writer - design, not of this recipe. -- **PodDisruptionBudget:** leave it unset, or `maxUnavailable: 1`. A - `minAvailable: 1` PDB on a one-replica workload blocks every node drain - indefinitely and will page someone at 3am during a routine AMI upgrade. - -### Resource requests: count the sandboxes - -The sandbox containers' cgroups nest **under the pod's cgroup**, so their memory -and CPU count against the pod's limits. Size the pod, not just the process: - -``` -pod limit ≈ base (2 vCPU / 6 GB: Go process + Next app) - + FLEET_MAX_CONCURRENT_AGENTS × (FLEET_SANDBOX_CPUS, FLEET_SANDBOX_MEMORY) - + headroom -``` - -With `FLEET_MAX_CONCURRENT_AGENTS=32` and per-sandbox caps of `2g`/`1.0` CPU: -≈ 34 vCPU and ≈ 70 GB. Set **`requests == limits`** (Guaranteed QoS) and leave -real headroom: if the pod cgroup hits its memory limit, the kernel OOM killer -picks the biggest process in the cgroup — which can be the fleet process itself, -turning one runaway sandbox into a full restart. Raise the per-sandbox ceilings -(`FLEET_SANDBOX_MEMORY`, `FLEET_SANDBOX_CPUS`, `FLEET_SANDBOX_PIDS`, and the -operator maxima `FLEET_SANDBOX_{MEMORY_MAX_MB,CPUS_MAX,PIDS_MAX}`) and the pod -limits **together** — the defaults are 512 MiB / 1.0 CPU / 128 pids per sandbox, -and heavy analysis workloads get OOM-killed against that default long before your -node runs out of RAM. - -## 7. Manifests - -### Namespace, admission control, and identity - -**A privileged pod is rejected outright under the `baseline` or `restricted` Pod -Security Standards.** This is the single most likely reason a first deploy fails -in a governed cluster, and it fails at admission with no pod to debug. Label the -namespace so PSA permits it, and keep the exception scoped to this one namespace: - -```yaml -apiVersion: v1 -kind: Namespace -metadata: - name: fleet - labels: - # Required for the privileged fleet container (§2). Scope the exception to - # THIS namespace; do not relax the cluster-wide default. - pod-security.kubernetes.io/enforce: privileged - pod-security.kubernetes.io/enforce-version: latest - # Keep the warnings visible so you can see exactly which controls you gave up. - pod-security.kubernetes.io/audit: baseline - pod-security.kubernetes.io/warn: baseline - # AWS Load Balancer Controller: hold the pod un-Ready until the ALB target is - # registered, so a restart doesn't briefly 5xx (§7 ingress). - elbv2.k8s.aws/pod-readiness-gate-inject: enabled -``` - -If you run **Kyverno** or **Gatekeeper** as well, PSA labels are not enough — -those policies evaluate independently. Add a narrowly-scoped exception (namespace -`fleet`, the `fleet` StatefulSet, the specific rules: privileged, -`allowPrivilegeEscalation`, host devices) rather than a blanket exemption, and -write the §2 mitigations into the exception's justification field so the next -auditor finds the reasoning instead of just the hole. - -Identity: **fleet needs no Kubernetes API access at all** — nothing in the -process talks to the API server. Its ServiceAccount exists only to carry an AWS -role for ECR pulls (and Secrets Manager, if you use it), so it gets **no Role or -RoleBinding**, which is a useful thing to be able to say in review: - -```yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: fleet - namespace: fleet - annotations: - # IRSA. EKS Pod Identity (eks-pod-identity-agent + a PodIdentityAssociation) - # is the newer equivalent and avoids the OIDC-trust-policy boilerplate; use - # whichever your platform standardizes on. Neither needs IMDS, which is why - # the §6 hop-limit-1 hardening is safe. - eks.amazonaws.com/role-arn: arn:aws:iam:::role/fleet -# No RBAC Role/RoleBinding: fleet makes zero Kubernetes API calls. -automountServiceAccountToken: false -``` - -The attached IAM policy needs only `ecr:GetAuthorizationToken`, -`ecr:BatchGetImage`, `ecr:GetDownloadUrlForLayer`, and -`ecr:BatchCheckLayerAvailability` on the two repositories — plus -`secretsmanager:GetSecretValue` on your specific secret ARNs if you use External -Secrets with this role. - -### Secrets - -The literal `Secret` below is the minimum. For a GitOps repo, replace it with an -`ExternalSecret` (External Secrets Operator) or a `SecretProviderClass` (Secrets -Store CSI driver) pointing at Secrets Manager or Parameter Store — the pod spec -is unchanged either way, since both project a normal `Secret`. Note that rotating -these takes effect on **pod restart** unless you use the env-file mount described -in §3b. - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: fleet-env - namespace: fleet -stringData: - OPENROUTER_API_KEY: "…" - FLEET_CHAT_DATABASE_URL: "postgres://chat:…@rds:5432/chat?sslmode=require" - FLEET_SCHED_DATABASE_URL: "postgres://sched:…@rds:5432/sched?sslmode=require" - FLEET_SERVER_TOKEN: "…" # web container's CHAT_SERVER_TOKEN must match - ADMIN_API_KEY: "…" # orchestrator admin key - APP_SESSION_SECRET: "…" # signs the web session cookie - # plus the MCP connector credentials the bundle's manifest names -``` - -The workload — a `StatefulSet`, because it gives at-most-one-pod semantics with -an `RWO` volume (a `Deployment`'s rolling update would briefly run two fleet -processes against one database pair, which the single-owner leases forbid): - -```yaml -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: fleet - namespace: fleet -spec: - replicas: 1 # never raise this - serviceName: fleet - podManagementPolicy: OrderedReady - updateStrategy: - type: RollingUpdate # for a single replica: terminate, then create - selector: - matchLabels: { app: fleet } - template: - metadata: - labels: { app: fleet } - annotations: - karpenter.sh/do-not-disrupt: "true" - spec: - serviceAccountName: fleet - # A freshly provisioned EBS volume is root-owned; without fsGroup the - # uid-1000 process cannot create the Podman graph root, the workspace, or - # XDG_RUNTIME_DIR, and the pod crash-loops on startup. OnRootMismatch keeps - # restarts fast once the volume is large (no full recursive rechown). - securityContext: - fsGroup: 1000 - fsGroupChangePolicy: OnRootMismatch - nodeSelector: { workload: fleet } - tolerations: - - key: dedicated - value: fleet - effect: NoSchedule - # Must exceed FLEET_SHUTDOWN_GRACE_SECONDS: on SIGTERM fleet stops - # admitting work, flips /healthz + /readyz to 503 so the ALB drains it, - # then drains in-flight chat turns AND scheduled tasks before exiting. - terminationGracePeriodSeconds: 90 - - initContainers: - # Pre-pull the sandbox image into the rootless store on the PVC. fleet - # never builds it and this keeps the first turn off a 1.5 GB download - # (and keeps a registry outage from surfacing as a failed turn). - - name: pull-sandbox - image: .dkr.ecr..amazonaws.com/fleet: - command: ["/bin/sh", "-c"] - args: - - | - set -e - aws ecr get-login-password --region "$AWS_REGION" \ - | podman login --username AWS --password-stdin "${ECR_REGISTRY}" - podman pull "$FLEET_SANDBOX_IMAGE" - env: - - { name: AWS_REGION, value: "" } - - { name: ECR_REGISTRY, value: ".dkr.ecr..amazonaws.com" } - - { name: FLEET_SANDBOX_IMAGE, value: ".dkr.ecr..amazonaws.com/fleet-sandbox@sha256:…" } - - { name: HOME, value: /var/lib/fleet } - - { name: XDG_RUNTIME_DIR, value: /var/lib/fleet/run } - securityContext: - privileged: true # same reasons as the main container (§2) - runAsUser: 1000 - volumeMounts: - - { name: state, mountPath: /var/lib/fleet } - - containers: - - name: fleet - image: .dkr.ecr..amazonaws.com/fleet: - envFrom: - - secretRef: { name: fleet-env } - env: - # Listeners stay loopback — the web container reaches them through - # the shared pod network namespace. The orchestrator MUST stay on - # 127.0.0.1 (it is impersonation-load-bearing). - - { name: FLEET_SERVER_ADDR, value: "127.0.0.1:8080" } - - { name: FLEET_ORCHESTRATOR_ADDR, value: "127.0.0.1:8000" } - - { name: FLEET_CLIENT_CONFIG_DIR, value: "/opt/fleet/client" } - # Absolute, not CWD-relative: don't depend on WORKDIR surviving an - # image refactor. - - { name: FLEET_DATA_DIR, value: "/var/lib/fleet/data" } - - { name: FLEET_WORKSPACE_ROOT, value: "/var/lib/fleet/workspace" } - - { name: HOME, value: "/var/lib/fleet" } - - { name: XDG_RUNTIME_DIR, value: "/var/lib/fleet/run" } - - { name: FLEET_SANDBOX_IMAGE, value: ".dkr.ecr..amazonaws.com/fleet-sandbox@sha256:…" } - # Sizing knobs — keep in lockstep with the pod resources below (§6). - - { name: FLEET_MAX_CONCURRENT_AGENTS, value: "32" } - - { name: FLEET_SANDBOX_MEMORY, value: "2g" } - - { name: FLEET_SANDBOX_CPUS, value: "1.0" } - - { name: FLEET_SANDBOX_WARM_SIZE, value: "4" } - - { name: FLEET_SHUTDOWN_GRACE_SECONDS, value: "60" } - - { name: FLEET_TIMEZONE, value: "UTC" } - # Trust the ALB's X-Forwarded-For only from in-pod/in-VPC hops. - - { name: FLEET_TRUSTED_PROXIES, value: "127.0.0.1,::1" } - securityContext: - privileged: true - allowPrivilegeEscalation: true # newuidmap/newgidmap file caps - runAsUser: 1000 - runAsGroup: 1000 - resources: - requests: { cpu: "34", memory: "70Gi" } - limits: { cpu: "34", memory: "70Gi" } - # Probes are exec, not httpGet: kubelet dials the POD IP, which cannot - # reach a 127.0.0.1-only listener. curl is in the image for this. - startupProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/readyz"] } - periodSeconds: 10 - failureThreshold: 30 # DB self-migration + warm-pool fill - livenessProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/livez"] } - periodSeconds: 30 - failureThreshold: 4 - readinessProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/readyz"] } - periodSeconds: 10 - volumeMounts: - - { name: state, mountPath: /var/lib/fleet } - - - name: web - image: .dkr.ecr..amazonaws.com/fleet-web: - ports: - - { name: http, containerPort: 3000 } - env: - - { name: CHAT_SERVER_URL, value: "http://127.0.0.1:8080" } - - { name: ORCHESTRATOR_SERVER_URL, value: "http://127.0.0.1:8000" } - - { name: CHAT_SERVER_TOKEN, valueFrom: { secretKeyRef: { name: fleet-env, key: FLEET_SERVER_TOKEN } } } - - { name: ORCHESTRATOR_SERVER_TOKEN, valueFrom: { secretKeyRef: { name: fleet-env, key: ADMIN_API_KEY } } } - - { name: APP_SESSION_SECRET, valueFrom: { secretKeyRef: { name: fleet-env, key: APP_SESSION_SECRET } } } - securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: false # Next writes .next/cache at runtime - capabilities: { drop: ["ALL"] } - resources: - requests: { cpu: "500m", memory: "1Gi" } - limits: { cpu: "2", memory: "2Gi" } - readinessProbe: - httpGet: { path: /, port: 3000 } - periodSeconds: 10 - # SIGTERM reaches every container at once, so without this the public - # tier can die while fleet is still draining a turn — the browser sees a - # dropped stream instead of a finished answer. Sleep past the ALB's - # deregistration delay, then let Next exit. - lifecycle: - preStop: - exec: { command: ["sleep", "20"] } - - volumeClaimTemplates: - - metadata: { name: state } - spec: - accessModes: ["ReadWriteOnce"] - storageClassName: fleet-gp3-xfs - resources: { requests: { storage: 400Gi } } -``` - -Service + ALB ingress (TLS terminates at the ALB with an ACM cert, so no Caddy -container is needed — the Next app remains the only public entrypoint): - -```yaml -apiVersion: v1 -kind: Service -metadata: { name: fleet, namespace: fleet } -spec: - selector: { app: fleet } - ports: [{ name: http, port: 3000, targetPort: 3000 }] ---- -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: fleet - namespace: fleet - annotations: - alb.ingress.kubernetes.io/scheme: internet-facing - alb.ingress.kubernetes.io/target-type: ip - alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' - alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:… - alb.ingress.kubernetes.io/ssl-redirect: "443" - # SSE: agent turns stream for minutes. The default 60s idle timeout cuts them off. - alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=1800 - alb.ingress.kubernetes.io/healthcheck-path: / -spec: - ingressClassName: alb - rules: - - host: fleet.example.com - http: - paths: - - path: / - pathType: Prefix - backend: { service: { name: fleet, port: { number: 3000 } } } -``` - -**The ALB idle timeout is the one setting people get wrong.** Chat responses are -SSE streams that can run for many minutes; the ALB's 60-second default idle -timeout will sever them mid-turn. Raise it (1800s above) — the equivalent of -`flush_interval -1` + `read_timeout 30m` in `deploy/Caddyfile`. - -Set `FLEET_PUBLIC_URL` / `FLEET_PUBLIC_BASE_URL` to the public origin so -notification links and share URLs resolve. Login works exactly as on the single -box (email + password, optional magic-link, optional OIDC SSO — all in the Next -layer); see [`docs/DEPLOYMENT.md`](DEPLOYMENT.md) for the login model. - -### NetworkPolicy - -Worth stating explicitly because it answers a real objection: **agent-executed -code is covered by pod-level NetworkPolicy.** Sandbox containers have no pod IP of -their own — their egress is NAT'd through the fleet pod's network namespace by the -rootless network helper (pasta on Podman ≥ 5.0, slirp4netns before it) — so a -policy on this pod governs what the model's `bash` and `run_python` can reach. This composes with, and does not replace, fleet's own -egress controls: `--network=none` for lockdown and scheduled runs is the hard -seal, and the allowlisted-egress proxy mode is -[ADR-0012](adr/0012-sandbox-egress-allowlist.md) / -[ADR-0031](adr/0031-chat-sandbox-egress.md). - -Requires a policy-enforcing CNI — the **VPC CNI enforces NetworkPolicy** only -with `enableNetworkPolicy: true` (EKS 1.25+); otherwise use Calico or Cilium. - -```yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: { name: fleet, namespace: fleet } -spec: - podSelector: { matchLabels: { app: fleet } } - policyTypes: ["Ingress", "Egress"] - ingress: - - from: [{ ipBlock: { cidr: } }] # ALB target-type: ip - ports: [{ port: 3000, protocol: TCP }] - egress: - - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }, - podSelector: { matchLabels: { k8s-app: kube-dns } } }] - ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }] - - to: [{ ipBlock: { cidr: } }] - ports: [{ port: 5432, protocol: TCP }] - # Model provider, ECR, and the MCP endpoints you intend. Narrow this as far as - # your provider's addressing allows; it is the boundary on agent egress. - - to: [{ ipBlock: { cidr: 0.0.0.0/0, except: [, 169.254.169.254/32] } }] - ports: [{ port: 443, protocol: TCP }] -``` - -Excluding `169.254.169.254/32` is belt-and-braces alongside the §6 IMDS hop -limit: two independent controls stopping agent code from reaching instance -credentials. - -### Metrics scrape sidecar - -`/metrics` is on the orchestrator's loopback listener and is admin-key gated -(§8). Rather than binding that listener to the pod IP — which would break the -impersonation boundary — add a tiny proxy that exposes only `GET /metrics` on the -pod IP and injects the key, then point a `ServiceMonitor`/`PodMonitor` (Prometheus -Operator) or your scrape config at port 9090: - -```yaml - - name: metrics-proxy - image: nginx:1.30-alpine - ports: [{ name: metrics, containerPort: 9090 }] - # nginx.conf (mount from a ConfigMap): - # server { listen 9090; - # location = /metrics { - # proxy_pass http://127.0.0.1:8000/metrics; - # proxy_set_header Authorization "Bearer "; - # } - # location / { return 404; } } - # Render the key in via envsubst on an nginx.conf.template at startup — - # don't bake it into the ConfigMap. - securityContext: - allowPrivilegeEscalation: false - capabilities: { drop: ["ALL"] } - resources: - requests: { cpu: "50m", memory: "64Mi" } - limits: { cpu: "200m", memory: "128Mi" } -``` - -The orchestrator authenticates admin reads with `Authorization: Bearer -`. Keep the proxy's `location /` a 404 so the sidecar cannot become -a general-purpose hole into the orchestrator, and keep it off the Service that -backs the Ingress. - -### Packaging these manifests for GitOps - -Nothing above needs templating to be managed declaratively. A Kustomize base with -per-environment overlays is the smaller-footprint option; a thin Helm chart is -fine if charts are your standard: - -``` -deploy/k8s/ - base/ namespace.yaml serviceaccount.yaml statefulset.yaml service.yaml - ingress.yaml networkpolicy.yaml storageclass.yaml kustomization.yaml - overlays/prod/ kustomization.yaml (images, replicas:1, resources, host, ARNs) -``` - -Two things bite in **Argo CD** specifically: - -1. **`volumeClaimTemplates` are immutable.** Any change to them makes the - StatefulSet un-patchable, and Argo reports a permanently `OutOfSync` app. To - resize, patch the **PVC** directly (`allowVolumeExpansion: true`, §5) and leave - the template alone; for a template change, delete the StatefulSet with - `--cascade=orphan` and re-apply. -2. **Set `Replace=false` and avoid auto-prune on the PVC.** An automated sync that - prunes the volume claim destroys workspaces, uploads, and the audit dir. Add - `argocd.argoproj.io/sync-options: Prune=false` on the PVC, or exclude - PersistentVolumeClaims from the app's prune scope. - -Also keep **automated sync from being an upgrade mechanism you didn't intend**: -this workload restarts (with downtime, §6) on every pod-spec change, so pin -digests in the overlay and let a human promote them. - -## 8. Observability and cluster integration - -- **Logs** go to stdout/stderr → your CloudWatch/Fluent Bit pipeline. Leave - `FLEET_LOG_FILE` unset; the file sink exists for hosts without a log collector - and would only duplicate lines onto the PVC. -- **Metrics:** `/metrics` (Prometheus text format) is served by the - **orchestrator** on `127.0.0.1:8000` and is **admin-API-key gated** — cost and - token data must not be public. Because that listener must stay loopback, give - the pod a tiny reverse-proxy sidecar that listens on the pod IP, forwards only - `GET /metrics` to `127.0.0.1:8000`, and injects the admin key; point your - scraper at the sidecar. Do not "solve" this by binding the orchestrator to the - pod IP. -- **Useful series** for this deployment: `fleet_sandbox_memory_usage_bytes` / - `fleet_sandbox_memory_limit_bytes` (right-size the per-sandbox caps), - `fleet_sandbox_pids_peak`, and the sandbox pool gauge. Per-run peaks come from - read-only `podman stats` sampling — observability only, never affecting - isolation ([`docs/DEPLOYMENT.md`](DEPLOYMENT.md)). -- **Tracing:** `FLEET_OTEL_ENDPOINT` + `FLEET_OTEL_SAMPLE_RATIO` if you run a - collector. - -### Cluster integration gotchas - -Each of these is something a Kubernetes-native environment does by default that -either breaks this pod or silently misleads you about it. - -- **The sandboxes are invisible to the Kubernetes API.** They are Podman - containers inside the pod: no entry in `kubectl get pods`, no cAdvisor - container metrics, no kubelet events, no k8s audit records for them. Where to - look instead: `kubectl exec … -- podman ps`, the `fleet_sandbox_*` metrics, the - per-task resource telemetry, and the per-run logs / audit dir. Say this out - loud in review — a platform team that expects pod-level visibility into agent - workloads will otherwise assume it exists. -- **NodeLocal DNSCache breaks DNS inside sandboxes.** If the node's - `/etc/resolv.conf` points at a link-local or loopback address (`169.254.20.10`, - `127.0.0.1`), that address means something different inside the sandbox's - network namespace under either helper, and name resolution fails for every - outbound HTTP tool — while the fleet process itself resolves fine, so it looks - like a model problem, not a DNS problem. Pin explicit resolvers for Podman in - the image's `containers.conf`: - - ```ini - [containers] - dns_servers = ["172.20.0.10"] # your cluster's kube-dns Service IP, or a VPC resolver - ``` - -- **`ResourceQuota` / `LimitRange` in the namespace will reject the pod.** A - 34-vCPU/70-GiB request trips inherited defaults, and a `LimitRange` with a - low `max` silently caps it. Give the namespace its own quota sized to the node, - or none. -- **VPA in `Auto` mode is destructive here** — it restarts the pod to resize it. - If you run VPA cluster-wide, exclude this workload or set `updateMode: "Off"` - and use its recommendations to hand-tune §6. -- **`automountServiceAccountToken: false`** is safe and recommended: fleet makes - no API calls, and the pod runs code the model wrote. IRSA/Pod Identity project - their own token separately and keep working. -- **Runtime security tooling** (Falco, GuardDuty Runtime Monitoring, Aqua/Sysdig) - will see nested container creation, user-namespace clones, and `newuidmap` from - a privileged pod, and will alert on all of it. Baseline those signatures for - this namespace *before* go-live — otherwise fleet's normal operation reads as an - ongoing container-escape attempt, and the noise trains everyone to ignore the - detector. -- **Node AMI upgrades are planned outages** (§6), so exclude this node group from - any automatic AMI-refresh schedule and drain it deliberately. - -## 9. Day-2 operations (what replaces bootstrap/update/doctor) - -| Single-host | On EKS | -|---|---| -| `scripts/bootstrap.sh` | build images (§3) + `kubectl apply` | -| `fleet update` | build a new image tag, `kubectl set image` / re-apply, pod restarts | -| `fleet restart` | `kubectl rollout restart statefulset/fleet` | -| `scripts/doctor.sh` (systemd-specific) | `kubectl exec … -- fleet validate-config` plus §10 — and the in-process Doctor panel still works, see below | -| `fleet admin add ` | `kubectl exec -it sts/fleet -c fleet -- fleet admin add ` | -| `fleet mcp account set …` | same, via `kubectl exec` | -| journald | `kubectl logs sts/fleet -c fleet` | - -`fleet validate-config` is the portable check — it verifies the bundle, podman -reachability, the sandbox image's presence, and the runtime preflight. - -**Settings → Admin → Doctor works here too**, and degrades honestly: its -container-portable checks (chat and sched databases, model API key, -subuid/subgid ranges, rootless podman, sandbox image) all run normally, while the -systemd-dependent ones (sibling unit state, the scheduled-backup timer) report -`skip` with "systemctl not on PATH (no systemd)" rather than inventing advisories -about units that were never meant to exist here. Note the consequence, though: a -`skip` on scheduled backups is *not* reassurance — it is the gap you closed by -hand above. - -**Config and bundle changes.** With the bundle baked into the image, a bundle -change is an image rebuild + pod restart. If you instead mount the bundle from a -PVC or clone it in an init container, MCP server definitions can be reloaded live -with `fleet mcp reload` / SIGHUP / the admin endpoint -([`docs/MCP-RELOAD.md`](MCP-RELOAD.md)). Reloadable env ceilings need the env-file -setup described in §3b; otherwise change them by editing the manifest and -restarting. - -**Backups — read this one carefully.** Two things are stateful: the databases and -the PVC (EBS snapshots via the CSI `VolumeSnapshot` API — workspaces, uploads, -audit). The Podman image store on the PVC is reconstructible; don't optimize -backups for it. - -The trap: fleet now ships `deploy/fleet-backup.service` + `fleet-backup.timer`, -which `bootstrap.sh --enable-service` installs and enables **by default**, and -`fleet doctor` reports on. **None of that exists here** — those are systemd units, -`bootstrap.sh` never runs on this deployment, and nothing in the pod will tell you -backups aren't happening. That gap is precisely the failure the timer was added to -fix (#966: a box reporting "38 ok, 0 advisories" while holding no backups at all, -for five days, with live client data). So pick one deliberately and write it down: - -- **RDS automated backups + snapshots** (simplest, and what this guide assumes) — - covers exactly the loss of a host or volume that a same-host `pg_dump` does not. -- **A Kubernetes `CronJob`** running `fleet backup` or `pg_dump` on a schedule, if - you want the logical dump the timer would have produced (recoverable from a bad - migration or an accidental delete). Give it its own ServiceAccount and write to - S3, not to the PVC — a dump beside the data it protects is not a backup. - -Either way, note that neither captures attachment/upload files, which live on the -PVC — those need the `VolumeSnapshot` schedule. See -[`docs/BACKUP_RESTORE.md`](BACKUP_RESTORE.md) for what a dump does and does not -cover. - -**Upgrades and node patching** are downtime windows. Sequence: cordon nothing, -just `kubectl delete pod` / `rollout restart` and let the drain budget run — -fleet flips `/readyz` to 503, the ALB stops sending traffic, in-flight turns and -scheduled tasks drain within `FLEET_SHUTDOWN_GRACE_SECONDS`, and the new pod -re-attaches the same PVC. - -## 10. Verification checklist - -Run these in the fleet container (`kubectl exec -it sts/fleet -c fleet -- bash`) -before you call the deployment done. Each maps to a row in §2 that fails -*silently* if you skip it. - -```sh -# 1. Rootless podman works at all, with the expected driver. -podman info --format '{{.Host.CgroupsVersion}} {{.Store.GraphDriverName}} {{.Host.Security.Rootless}}' -# want: v2 true (NOT vfs) - -# 2. The uid mapping fleet actually uses. -podman run --rm --userns=keep-id:uid=1000,gid=1000 "$FLEET_SANDBOX_IMAGE" id -# want: uid=1000 gid=1000 - -# 3. Memory limits BIND. If this prints "max", --memory is being ignored and -# every per-sandbox and per-task cap is fiction. -podman run --rm --memory=64m "$FLEET_SANDBOX_IMAGE" cat /sys/fs/cgroup/memory.max -# want: 67108864 - -# 4. All three network postures (each needs /dev/net/tun for its helper). -# a) normal turns — podman's rootless default (pasta on >= 5.0): -podman run --rm "$FLEET_SANDBOX_IMAGE" \ - python3 -c 'import socket;socket.create_connection(("1.1.1.1",443),5);print("default egress ok")' -# b) allowlisted-egress posture — needs the slirp4netns binary specifically. -# A missing binary now aborts BOOT with a fail-closed preflight, so check it -# here if you plan to enable that mode: -podman run --rm --network=slirp4netns:allow_host_loopback=true \ - "$FLEET_SANDBOX_IMAGE" python3 -c 'import socket;socket.create_connection(("1.1.1.1",443),5);print("slirp egress ok")' -# c) lockdown / scheduled runs — the hard seal: -podman run --rm --network=none "$FLEET_SANDBOX_IMAGE" true && echo "sealed mode ok" - -# 5. The TOTAL writable-layer cap (the per-file ulimit applies either way, §5). -podman run --rm --storage-opt size=1g "$FLEET_SANDBOX_IMAGE" true \ - && echo "total-size cap available" || echo "per-file cap only — total layer size unbounded" - -# 6. DNS inside a sandbox — the NodeLocal DNSCache trap (§8). Resolution can -# fail here while the fleet process itself resolves fine. -podman run --rm --network=slirp4netns:allow_host_loopback=true \ - "$FLEET_SANDBOX_IMAGE" python3 -c 'import socket;print(socket.gethostbyname("api.openai.com"))' - -# 7. The volume is actually writable by uid 1000 (fsGroup, §7). -touch /var/lib/fleet/.write-probe && rm /var/lib/fleet/.write-probe && echo "volume writable" - -# 8. fleet's own preflight: bundle, podman, image, runtime. -fleet validate-config - -# 9. Health + drain semantics. -curl -fsS http://127.0.0.1:8080/readyz; curl -fsS http://127.0.0.1:8080/livez -``` - -From outside the pod, confirm the cluster-side wiring: - -```sh -# Admission actually permits the pod (fails at admission, with no pod to debug). -kubectl -n fleet get statefulset fleet -o jsonpath='{.status.readyReplicas}' -kubectl -n fleet describe statefulset fleet | grep -iA3 'FailedCreate\|forbidden' - -# The volume landed in the AZ the node group lives in (§6). -kubectl -n fleet get pvc state-fleet-0 -o jsonpath='{.spec.volumeName}' \ - | xargs -I{} kubectl get pv {} -o jsonpath='{.spec.nodeAffinity}' - -# SSE survives the ALB: this must stream for minutes, not cut off at 60s. -curl -N https://fleet.example.com/… # any streaming chat turn - -# Graceful drain: /readyz flips to 503 and in-flight work finishes, no SIGKILL. -kubectl -n fleet delete pod fleet-0 --wait=true -``` - -Then, from the UI, run one interactive turn that executes `run_python` and one -scheduled task, and confirm in `kubectl logs` that no line reports the -`--storage-opt` fallback or a warm-pool cold-start failure. - -## Appendix: the complete manifest set - -The sections above explain each piece; this is all of it assembled in apply -order, so nothing gets missed in transcription. It is the same content — if the -two ever disagree, the numbered sections are the explanation and this is the -transcription. - -**Fill these in first.** Every placeholder appears in angle brackets: - -| Placeholder | Where it comes from | -|---|---| -| ``, `` | your AWS account ID and region | -| `` | the image tags you built in §3 | -| `` | `sha256:…` of the sandbox image pushed in §3a — pin by digest | -| `` | the IRSA/Pod Identity role from §7 | -| `` | the ACM certificate for your hostname | -| ``, `` | the RDS endpoint and its subnet range (§4) | -| `` | the cluster VPC range, for the ALB ingress rule (§7) | -| `` | `kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'` | -| `` | the public hostname, e.g. `fleet.example.com` | -| secret values | §7; prefer External Secrets over literals | - -Sizing below is the worked 32-concurrent-agent example (`m7i.12xlarge`): raise -`FLEET_MAX_CONCURRENT_AGENTS`, the per-sandbox caps, the pod resources, and the -instance type **together** — see [§6](#resource-requests-count-the-sandboxes). - -```yaml -# 1 ── Namespace. The PSA labels are what let the privileged pod be admitted (§7). -apiVersion: v1 -kind: Namespace -metadata: - name: fleet - labels: - pod-security.kubernetes.io/enforce: privileged - pod-security.kubernetes.io/enforce-version: latest - pod-security.kubernetes.io/audit: baseline - pod-security.kubernetes.io/warn: baseline - elbv2.k8s.aws/pod-readiness-gate-inject: enabled ---- -# 2 ── StorageClass. xfs + prjquota is what makes the sandbox disk quota a HARD -# cap on top of the per-file ulimit that applies regardless (§5). -apiVersion: storage.k8s.io/v1 -kind: StorageClass -metadata: - name: fleet-gp3-xfs -provisioner: ebs.csi.aws.com -parameters: - type: gp3 - iops: "6000" - throughput: "500" - fsType: xfs -mountOptions: ["prjquota"] -allowVolumeExpansion: true -volumeBindingMode: WaitForFirstConsumer ---- -# 3 ── Identity. No Role/RoleBinding: fleet makes zero Kubernetes API calls (§7). -apiVersion: v1 -kind: ServiceAccount -metadata: - name: fleet - namespace: fleet - annotations: - eks.amazonaws.com/role-arn: -automountServiceAccountToken: false ---- -# 4 ── Secrets. Replace with an ExternalSecret / SecretProviderClass in a GitOps -# repo — the pod spec below is identical either way (§7). -apiVersion: v1 -kind: Secret -metadata: - name: fleet-env - namespace: fleet -stringData: - OPENROUTER_API_KEY: "" - FLEET_CHAT_DATABASE_URL: "postgres://chat:@:5432/chat?sslmode=require" - FLEET_SCHED_DATABASE_URL: "postgres://sched:@:5432/sched?sslmode=require" - FLEET_SERVER_TOKEN: "" - ADMIN_API_KEY: "" - APP_SESSION_SECRET: "" - # plus every MCP connector credential the bundle's manifest.yaml names ---- -# 5 ── The workload. -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: fleet - namespace: fleet -spec: - replicas: 1 # NEVER raise: single-owner leases + per-process semaphore - serviceName: fleet - podManagementPolicy: OrderedReady - updateStrategy: { type: RollingUpdate } - selector: - matchLabels: { app: fleet } - template: - metadata: - labels: { app: fleet } - annotations: - karpenter.sh/do-not-disrupt: "true" - spec: - serviceAccountName: fleet - # Without fsGroup, uid 1000 cannot write the fresh EBS volume and the pod - # crash-loops before it ever starts podman (§7). - securityContext: - fsGroup: 1000 - fsGroupChangePolicy: OnRootMismatch - nodeSelector: { workload: fleet } - tolerations: - - { key: dedicated, value: fleet, effect: NoSchedule } - terminationGracePeriodSeconds: 90 # > FLEET_SHUTDOWN_GRACE_SECONDS - - initContainers: - - name: pull-sandbox - image: .dkr.ecr..amazonaws.com/fleet: - command: ["/bin/sh", "-c"] - args: - - | - set -e - aws ecr get-login-password --region "$AWS_REGION" \ - | podman login --username AWS --password-stdin "$ECR_REGISTRY" - podman pull "$FLEET_SANDBOX_IMAGE" - env: - - { name: AWS_REGION, value: "" } - - { name: ECR_REGISTRY, value: ".dkr.ecr..amazonaws.com" } - - { name: FLEET_SANDBOX_IMAGE, value: ".dkr.ecr..amazonaws.com/fleet-sandbox@" } - - { name: HOME, value: "/var/lib/fleet" } - - { name: XDG_RUNTIME_DIR, value: "/var/lib/fleet/run" } - securityContext: { privileged: true, runAsUser: 1000 } - volumeMounts: - - { name: state, mountPath: /var/lib/fleet } - - containers: - - name: fleet - image: .dkr.ecr..amazonaws.com/fleet: - envFrom: - - secretRef: { name: fleet-env } - env: - - { name: FLEET_SERVER_ADDR, value: "127.0.0.1:8080" } - - { name: FLEET_ORCHESTRATOR_ADDR, value: "127.0.0.1:8000" } # must stay loopback - - { name: FLEET_CLIENT_CONFIG_DIR, value: "/opt/fleet/client" } - - { name: FLEET_DATA_DIR, value: "/var/lib/fleet/data" } - - { name: FLEET_WORKSPACE_ROOT, value: "/var/lib/fleet/workspace" } - - { name: HOME, value: "/var/lib/fleet" } - - { name: XDG_RUNTIME_DIR, value: "/var/lib/fleet/run" } - - { name: FLEET_SANDBOX_IMAGE, value: ".dkr.ecr..amazonaws.com/fleet-sandbox@" } - - { name: FLEET_PUBLIC_URL, value: "https://" } - - { name: FLEET_MAX_CONCURRENT_AGENTS, value: "32" } - - { name: FLEET_SANDBOX_MEMORY, value: "2g" } - - { name: FLEET_SANDBOX_CPUS, value: "1.0" } - - { name: FLEET_SANDBOX_WARM_SIZE, value: "4" } - - { name: FLEET_SHUTDOWN_GRACE_SECONDS, value: "60" } - - { name: FLEET_TIMEZONE, value: "UTC" } - - { name: FLEET_TRUSTED_PROXIES, value: "127.0.0.1,::1" } - securityContext: - privileged: true # see §2 for what this buys and costs - allowPrivilegeEscalation: true # newuidmap/newgidmap file caps - runAsUser: 1000 # NOT root — rootful podman ignores keep-id - runAsGroup: 1000 - resources: - requests: { cpu: "34", memory: "70Gi" } # base + 32 × per-sandbox cap - limits: { cpu: "34", memory: "70Gi" } - # exec, not httpGet: kubelet dials the pod IP and cannot reach loopback. - startupProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/readyz"] } - periodSeconds: 10 - failureThreshold: 30 - livenessProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/livez"] } - periodSeconds: 30 - failureThreshold: 4 - readinessProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/readyz"] } - periodSeconds: 10 - volumeMounts: - - { name: state, mountPath: /var/lib/fleet } - - - name: web - image: .dkr.ecr..amazonaws.com/fleet-web: - ports: - - { name: http, containerPort: 3000 } - env: - - { name: CHAT_SERVER_URL, value: "http://127.0.0.1:8080" } - - { name: ORCHESTRATOR_SERVER_URL, value: "http://127.0.0.1:8000" } - - { name: CHAT_SERVER_TOKEN, valueFrom: { secretKeyRef: { name: fleet-env, key: FLEET_SERVER_TOKEN } } } - - { name: ORCHESTRATOR_SERVER_TOKEN, valueFrom: { secretKeyRef: { name: fleet-env, key: ADMIN_API_KEY } } } - - { name: APP_SESSION_SECRET, valueFrom: { secretKeyRef: { name: fleet-env, key: APP_SESSION_SECRET } } } - securityContext: - allowPrivilegeEscalation: false - capabilities: { drop: ["ALL"] } - resources: - requests: { cpu: "500m", memory: "1Gi" } - limits: { cpu: "2", memory: "2Gi" } - readinessProbe: - httpGet: { path: /, port: 3000 } - periodSeconds: 10 - lifecycle: - preStop: - exec: { command: ["sleep", "20"] } # outlive ALB deregistration - - volumeClaimTemplates: # immutable — see the Argo notes in §7 - - metadata: { name: state } - spec: - accessModes: ["ReadWriteOnce"] - storageClassName: fleet-gp3-xfs - resources: { requests: { storage: 400Gi } } ---- -# 6 ── Service (the web tier is the only exposed port). -apiVersion: v1 -kind: Service -metadata: { name: fleet, namespace: fleet } -spec: - selector: { app: fleet } - ports: [{ name: http, port: 3000, targetPort: 3000 }] ---- -# 7 ── Ingress. The idle timeout is what keeps SSE turns from being severed. -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: fleet - namespace: fleet - annotations: - alb.ingress.kubernetes.io/scheme: internet-facing - alb.ingress.kubernetes.io/target-type: ip - alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' - alb.ingress.kubernetes.io/certificate-arn: - alb.ingress.kubernetes.io/ssl-redirect: "443" - alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=1800 - alb.ingress.kubernetes.io/healthcheck-path: / -spec: - ingressClassName: alb - rules: - - host: - http: - paths: - - path: / - pathType: Prefix - backend: { service: { name: fleet, port: { number: 3000 } } } ---- -# 8 ── NetworkPolicy. This governs agent-executed code too: sandbox egress NATs -# through the pod's netns (§7). Needs a policy-enforcing CNI. -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: { name: fleet, namespace: fleet } -spec: - podSelector: { matchLabels: { app: fleet } } - policyTypes: ["Ingress", "Egress"] - ingress: - - from: [{ ipBlock: { cidr: } }] - ports: [{ port: 3000, protocol: TCP }] - egress: - - to: - - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } - podSelector: { matchLabels: { k8s-app: kube-dns } } - ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }] - - to: [{ ipBlock: { cidr: } }] - ports: [{ port: 5432, protocol: TCP }] - - to: - - ipBlock: - cidr: 0.0.0.0/0 - except: [, 169.254.169.254/32] - ports: [{ port: 443, protocol: TCP }] -``` - -Then, in order: create the node group (§6), apply the above, `kubectl exec` in and -run the §10 checklist, add an admin (`fleet admin add `), and log in. - -Not included above, deliberately: the metrics scrape sidecar and its ConfigMap -(optional, §7 — add it once Prometheus is wired up), and the `containers.conf` -`dns_servers` pin, which belongs in the **image** rather than the manifest (§8). - -## What this deployment does not change - -- One governed run loop (`agentcore.Run`) — policy, cost/token ceilings, audit - ([ADR-0001](adr/0001-one-governed-run-loop.md)). -- The mandatory sandbox for every tool call's data plane, with no host-execution - fallback ([ADR-0002](adr/0002-mandatory-rootless-podman-sandbox.md), - [ADR-0036](adr/0036-sandboxed-file-tools-and-host-io-exceptions.md)). -- Host-side MCP credential brokering — secrets never enter a sandbox - ([ADR-0003](adr/0003-host-side-mcp-credential-brokering.md)). -- Client content stays in an out-of-repo bundle - ([ADR-0006](adr/0006-external-client-config-bundle.md)). - -What it *does* change is the outer boundary: on the single box the fleet process -is an unprivileged system user, and here it is a privileged container on a -dedicated node. Treat the node as the trust boundary and size the isolation you -add around it (§2) accordingly. diff --git a/docs/SANDBOX-RUNTIMES.md b/docs/SANDBOX-RUNTIMES.md index f25c1b5a..e5558be9 100644 --- a/docs/SANDBOX-RUNTIMES.md +++ b/docs/SANDBOX-RUNTIMES.md @@ -21,6 +21,17 @@ through to `podman run --runtime=`; fleet adds a fail-closed boot preflight and, for Kata, a guest-memory adjustment. See [ADR-0010](adr/0010-microvm-sandbox-runtimes.md) for the design rationale. +> **Sibling knob — `FLEET_SANDBOX_BACKEND`** (manifest `sandbox.backend`, +> same env-wins precedence): while the *runtime* picks the isolation posture +> of a podman sandbox, the *backend* picks WHERE sandboxes run at all — +> `podman` (this page's co-located default) or `kubernetes` (each sandbox an +> ephemeral pod; the split enterprise deployment, +> [ADR-0049](adr/0049-kubernetes-backend-split-control-plane.md)). Everything +> below applies to the podman backend; under the kubernetes backend +> `FLEET_SANDBOX_RUNTIME` is refused (use a cluster RuntimeClass via +> `FLEET_SANDBOX_K8S_RUNTIME_CLASS` instead) — see +> [DEPLOYMENT-KUBERNETES.md](DEPLOYMENT-KUBERNETES.md). + ## The three tiers | | **runc / crun** (default) | **Kata Containers** | **libkrun** | diff --git a/docs/TIMERS.md b/docs/TIMERS.md index 854f3ff0..e494dcca 100644 --- a/docs/TIMERS.md +++ b/docs/TIMERS.md @@ -66,7 +66,9 @@ the command does not pretend: it explains that the equivalent jobs belong to the platform's scheduler — daily `fleet backup --db=all --prune` and daily `fleet cleanup` (cron, a Kubernetes CronJob) — and exits non-zero. `fleet update`'s offer and doctor's advisories are likewise skipped entirely where -there is no systemd. +there is no systemd. For the first-class Kubernetes deployment, the CronJob +equivalents are part of the production checklist in +[`DEPLOYMENT-KUBERNETES.md`](DEPLOYMENT-KUBERNETES.md). ## Honest scope / deliberately not done diff --git a/docs/adr/0004-single-box-vm-native-deployment.md b/docs/adr/0004-single-box-vm-native-deployment.md index e4e5e562..2cac6540 100644 --- a/docs/adr/0004-single-box-vm-native-deployment.md +++ b/docs/adr/0004-single-box-vm-native-deployment.md @@ -1,6 +1,10 @@ # ADR-0004: Single-box, VM-native deployment (no Kubernetes) -- **Status:** Accepted +- **Status:** Accepted; amended by [ADR-0049](0049-kubernetes-backend-split-control-plane.md) + (the single-box default install stands; the "no k8s manifest, Helm chart, or + operator in the tree" enforcement clause and the cluster-work-is-out-of-scope + consequence are superseded — `deploy/helm/fleet` and the kubernetes sandbox + backend are the sanctioned enterprise path) - **Date:** 2026-06-28 (documents a decision that predates this record) - **Deciders:** fleet maintainers diff --git a/docs/adr/0049-kubernetes-backend-split-control-plane.md b/docs/adr/0049-kubernetes-backend-split-control-plane.md new file mode 100644 index 00000000..f1a26942 --- /dev/null +++ b/docs/adr/0049-kubernetes-backend-split-control-plane.md @@ -0,0 +1,131 @@ +# ADR-0049: Kubernetes as a first-class deployment — split control plane, pluggable sandbox backend + +- **Status:** Accepted +- **Date:** 2026-08-22 +- **Deciders:** fleet maintainers +- **Amends:** [ADR-0004](0004-single-box-vm-native-deployment.md) (supersedes + its "no k8s manifest, Helm chart, or operator in the tree" enforcement + clause and its cluster-work-is-out-of-scope consequence; the single-box + default install it decides **stands**) + +## Context + +ADR-0004 made fleet VM-native on one box: systemd, Caddy, rootless Podman +co-located with the process. That remains the right default install for +individuals and small teams. But Kubernetes-native organizations were left +with no supported path (issue #989): `deploy/` shipped only systemd units, the +sandbox was **always** co-located with the fleet process, and the only k8s +document was a hand-verified EKS recipe (`docs/EKS-DEPLOYMENT.md`, since +removed) that ran rootless Podman inside a privileged pod — an operator +workaround, not a product. + +The owner decision on #989: ship the **enterprise path in one pass** — the +fleet control plane separate from execution runners, with a pluggable sandbox +backend — and do **not** build a co-located "fleet pod + Podman on a +privileged node" packaging track as a stepping stone. + +## Decision + +1. **The sandbox backend is pluggable, selected by one knob.** The internal + per-sandbox interface (`internal/sandbox`'s `impl`) gains a third + implementation: alongside the rootless-Podman backend (`containerImpl`) and + the test-only host executor, `k8sImpl` runs each sandbox as an **ephemeral + Kubernetes Pod** exec'd over the apiserver. `FLEET_SANDBOX_BACKEND` + (overriding the bundle manifest's `sandbox.backend`) selects + `podman` (default) or `kubernetes`, mirroring `sandbox.runtime`'s + precedence exactly (ADR-0010). An unrecognized value refuses to boot. +2. **The kubernetes backend fails closed at boot.** Selecting it triggers a + preflight — apiserver reachable, RBAC verbs present (pods CRUD + + `pods/exec`), the shared workspace claim exists, the sealed-egress + NetworkPolicy object exists, the RuntimeClass exists when configured — and + any failure aborts boot. There is no fallback to podman or host execution + (the ADR-0010 no-degrade posture, applied to backends). +3. **The workspace is a shared ReadWriteMany claim, mounted same-path.** The + control plane and every sandbox pod mount the same PVC at the same absolute + path, preserving the invariant (ADR-0036 territory) that an absolute + workspace path means the same thing to the process, host-side brokers, and + sandboxed bash/python. +4. **Sealing is expressed as labels + a required NetworkPolicy.** Sandbox pods + carry `fleet.elcanotek.com/egress=none|open`; the Helm chart ships a + deny-all policy selecting `none`. fleet verifies the policy **object** + exists; enforcement is the CNI's, and the docs say so plainly rather than + implying a seal fleet cannot provide (the podman `--network=none` namespace + seal has no per-pod apiserver equivalent). +5. **Enterprise packaging is one Helm chart** (`deploy/helm/fleet`): + single-replica control-plane Deployment (strategy Recreate, no replica + knob), the runner RBAC, workspace storage, the NetworkPolicies, optional + in-cluster Postgres / web / Ingress. No operator, no CRDs in v1. +6. **The API client is hand-rolled, not client-go.** The backend needs five + verbs plus WebSocket exec streaming; client-go would add dozens of modules + to a tree gated by govulncheck and image CVE scans. `internal/sandbox` + speaks plain REST via net/http and `v4.channel.k8s.io` exec framing via + gorilla/websocket (already a dependency). If the backend ever needs + watches/informers or exotic auth, revisit client-go rather than growing the + hand-rolled client. Kubeconfig support is deliberately narrow — token, + token-file, client-cert; exec plugins and `insecure-skip-tls-verify` are + refused. + +## What does not change + +- **The single-box podman install stays the default** and its story is + untouched: bootstrap, systemd units, Caddy, `fleet timers`. ADR-0004's + decision section stands for that install. +- **The sandbox is still mandatory** (ADR-0002): the kubernetes backend is a + different *where*, not a weaker *whether*. Pods run read-only-rootfs, + non-root, all capabilities dropped, seccomp RuntimeDefault (or an + operator-installed Localhost profile), `automountServiceAccountToken=false`. +- **Credentials stay in the control plane** (ADR-0003): sandbox pods get no + env, no secrets, no service-account token — only the workspace mount. The + MCP broker never moves. +- **One governed loop** (ADR-0001): the backend swap is entirely below + `agentcore`; no second governance path exists. +- **Single-owner control plane:** one fleet replica, ever. Horizontal scale of + *work* is more sandbox pods / bigger node pools, not more fleet processes. +- **Poison-and-retire (#796)** carries over: a cancelled or timed-out call + deletes the whole pod with zero grace — destroying its PID namespace and + every straggler — and retires the sandbox. + +## Explicit non-goals (v1) + +- Co-located "fleet + Podman in a privileged pod" as the supported enterprise + story. The EKS recipe that documented it is **removed** rather than kept as + a parallel path — an unmaintained privileged-pod recipe beside a first-class + unprivileged one would imply support it does not have. +- Multi-replica / active-active fleet; a Kubernetes operator or CRDs. +- The **allowlisted** egress mode under the kubernetes backend: the host-side + egress proxy is unreachable from pods, so the mode is refused at boot + (fail-closed) instead of silently granting open egress. Cluster-side egress + shaping via NetworkPolicy is the replacement. +- Per-pod pids limits (not expressible in a Pod spec), the bundled seccomp + JSON (nodes take a Localhost profile instead), `podman stats` resource + telemetry (#263), and same-path supporting-doc bind mounts — each recorded + as an honest deviation in `docs/DEPLOYMENT-KUBERNETES.md`. + +## Consequences + +- Kubernetes-native organizations get a supported, preflighted, CI-linted + path: `helm install` + two images they build. The EKS privileged-pod recipe + is retired in its favor. +- `deploy/` now contains cluster artifacts, so ADR-0004's enforcement clause + ("no k8s manifest, Helm chart, or operator in the tree") is superseded; its + index row and status note point here. +- A second execution substrate must be kept honest: the backend seam + (`sandbox.Backend`-shaped `impl`) is now a contract two production backends + implement, and behavior-affecting changes must land in both or say why not. +- The chart is linted and template-rendered in CI (`helm` job in ci.yml / + dev-ci.yml) but not exercised against a live cluster there; the kind + walkthrough in `docs/DEPLOYMENT-KUBERNETES.md` is the verified end-to-end + path. + +## Alternatives considered + +- **client-go.** Rejected for dependency weight against a five-verb surface; + recorded above as the explicit revisit trigger. +- **Chart-only first, backend later.** Rejected by the issue itself: a chart + that still requires privileged Podman-in-pod would enshrine the workaround. +- **A Kubernetes operator/CRD.** Unnecessary for v1 — Helm + RBAC covers + install; fleet's own scheduler owns runtime orchestration. +- **Running sandbox pods in a dedicated namespace by default.** The RBAC story + is marginally nicer, but a PVC cannot be mounted across namespaces, so the + default topology shares the release namespace; a split namespace remains + possible with static same-export PVs and is documented. diff --git a/docs/adr/README.md b/docs/adr/README.md index 18bec27c..1e597386 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -29,7 +29,7 @@ reviewable, and citable. Each record names the file or test that enforces it. | [0001](0001-one-governed-run-loop.md) | One governed agent run loop | Accepted | | [0002](0002-mandatory-rootless-podman-sandbox.md) | Mandatory rootless-Podman sandbox; host executor never ships | Accepted | | [0003](0003-host-side-mcp-credential-brokering.md) | Host-side MCP credential brokering | Accepted | -| [0004](0004-single-box-vm-native-deployment.md) | Single-box, VM-native deployment (no Kubernetes) | Accepted | +| [0004](0004-single-box-vm-native-deployment.md) | Single-box, VM-native deployment (no Kubernetes) | Accepted; amended by ADR-0049 | | [0005](0005-separate-chat-and-sched-databases.md) | Separate Postgres databases for chat and sched | Accepted | | [0006](0006-external-client-config-bundle.md) | Client content lives in an external config bundle | Accepted | | [0007](0007-governed-sub-agents.md) | Governed sub-agents spawn only through the one run loop | Accepted | @@ -57,3 +57,4 @@ reviewable, and citable. Each record names the file or test that enforces it. | [0045](0045-remove-node-name-scopes.md) | Remove node-name scopes; a principal's authority is its permission set | Accepted | | [0046](0046-remove-per-key-spending-caps.md) | Remove per-API-key spending caps; rolling budgets are the one spend gate | Accepted | | [0047](0047-self-serve-team-membership.md) | Self-serve team membership — create/leave is yours, joining is granted | Accepted | +| [0049](0049-kubernetes-backend-split-control-plane.md) | Kubernetes as a first-class deployment — split control plane, pluggable sandbox backend | Accepted | diff --git a/go.mod b/go.mod index 30af55f3..cbb5a990 100644 --- a/go.mod +++ b/go.mod @@ -15,6 +15,7 @@ require ( github.com/goccy/go-yaml v1.19.2 github.com/golang-migrate/migrate/v4 v4.19.1 github.com/google/uuid v1.6.0 + github.com/gorilla/websocket v1.5.3 github.com/itchyny/gojq v0.12.19 github.com/jackc/pgx/v5 v5.10.0 github.com/robfig/cron/v3 v3.0.1 @@ -84,7 +85,6 @@ require ( github.com/googleapis/enterprise-certificate-proxy v0.3.20 // indirect github.com/googleapis/gax-go/v2 v2.23.0 // indirect github.com/gorilla/css v1.0.1 // indirect - github.com/gorilla/websocket v1.5.3 // indirect github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect github.com/invopop/jsonschema v0.14.0 // indirect github.com/itchyny/timefmt-go v0.1.8 // indirect diff --git a/internal/agent/manager.go b/internal/agent/manager.go index 27865646..553d83b2 100644 --- a/internal/agent/manager.go +++ b/internal/agent/manager.go @@ -581,6 +581,13 @@ func buildSandboxPool(cfg *config.Config, personasDir, protocolsDir, systemPromp BridgeDir: filepath.Join(filepath.Dir(workspaceRoot), "data", "sandbox-bridge"), ReadOnlyMounts: absSupportingDocs(personasDir, protocolsDir, systemPromptsDir, skillsDir, uploadsRoot), } + // Kubernetes backend (#989): sandboxes are ephemeral pods in a cluster + // instead of co-located podman containers. All podman-specific boot work + // below (bridge-file prune, OCI-runtime preflight, egress proxy) is + // replaced by the backend's own fail-closed cluster preflight. + if cfg.SandboxBackend == sandbox.BackendKubernetes { + return buildKubernetesSandboxPool(cfg, poolCfg, sandboxRuntime) + } // Reclaim bridge-script/seccomp temp files orphaned by a PRIOR crash: only // the graceful close path removes them, so without this sweep every // non-graceful exit leaks them into BridgeDir permanently. Age-bounded and @@ -646,6 +653,91 @@ func buildSandboxPool(cfg *config.Config, personasDir, protocolsDir, systemPromp return sandbox.NewPool(poolCfg), nil } +// buildKubernetesSandboxPool finishes pool construction for the kubernetes +// backend (#989): it refuses podman-only knobs that would otherwise be +// silently ignored (a configured-but-inert security knob is the failure mode +// ADR-0010's no-degrade rule exists for), builds the backend handle, and runs +// the fail-closed cluster preflight before the warm pool spawns its first pod. +func buildKubernetesSandboxPool(cfg *config.Config, poolCfg sandbox.PoolConfig, sandboxRuntime string) (*sandbox.Pool, error) { + if sandboxRuntime != "" { + return nil, fmt.Errorf( + "FLEET_SANDBOX_RUNTIME=%q is a podman OCI-runtime knob and has no effect under FLEET_SANDBOX_BACKEND=kubernetes; "+ + "select hypervisor isolation with FLEET_SANDBOX_K8S_RUNTIME_CLASS (a cluster RuntimeClass, e.g. kata) instead (fail-closed)", sandboxRuntime) + } + if v := strings.TrimSpace(os.Getenv("FLEET_SANDBOX_SECCOMP_PROFILE")); v != "" { + return nil, fmt.Errorf( + "FLEET_SANDBOX_SECCOMP_PROFILE=%q is a podman knob and has no effect under FLEET_SANDBOX_BACKEND=kubernetes; "+ + "install the profile on the sandbox nodes and set FLEET_SANDBOX_K8S_SECCOMP_PROFILE (a kubelet-relative Localhost profile) instead (fail-closed)", v) + } + if cfg.DefaultNetworkMode == sandbox.NetworkModeAllowlisted { + return nil, fmt.Errorf( + "FLEET_DEFAULT_NETWORK_MODE=allowlisted is not supported under FLEET_SANDBOX_BACKEND=kubernetes: the host-side egress proxy " + + "is unreachable from sandbox pods. Use lockdown (sealed by the deny-all NetworkPolicy) or open, and shape egress with cluster NetworkPolicies (fail-closed)") + } + // Supporting-doc mounts are same-path HOST bind mounts; a pod has no host + // filesystem to bind them from, and leaving them in the config would make + // the fileop anchor logic trust paths that are not actually mounted. + // Bundles relying on in-sandbox persona/protocol reads degrade exactly + // like the podman missing-dir case (skipped; host-path view_file still + // works via the workspace when the bundle lives under it). + if len(poolCfg.Container.ReadOnlyMounts) > 0 { + log.Printf("sandbox: kubernetes backend — supporting-doc bind mounts do not apply (pods mount only the workspace claim); in-sandbox reads of %d host dir(s) will not resolve", len(poolCfg.Container.ReadOnlyMounts)) + poolCfg.Container.ReadOnlyMounts = nil + } + poolCfg.Container.Runtime = "" + + // Scheduling knobs fail closed on a malformed value: a typo'd selector + // must not silently schedule sandboxes onto the wrong (untainted, + // unlabeled) nodes. + nodeSelector, err := sandbox.ParseK8sNodeSelector(cfg.SandboxK8sNodeSelector) + if err != nil { + return nil, fmt.Errorf("FLEET_SANDBOX_K8S_NODE_SELECTOR / sandbox.kubernetes.node_selector: %w", err) + } + tolerations, err := sandbox.ParseK8sTolerations(cfg.SandboxK8sTolerations) + if err != nil { + return nil, fmt.Errorf("FLEET_SANDBOX_K8S_TOLERATIONS / sandbox.kubernetes.tolerations: %w", err) + } + backend, err := sandbox.NewKubernetesBackend(sandbox.KubernetesConfig{ + Namespace: cfg.SandboxK8sNamespace, + WorkspaceClaim: cfg.SandboxK8sWorkspaceClaim, + ServiceAccount: cfg.SandboxK8sServiceAccount, + ImagePullSecret: cfg.SandboxK8sImagePullSecret, + RuntimeClassName: cfg.SandboxK8sRuntimeClass, + SeccompLocalhostProfile: cfg.SandboxK8sSeccompProfile, + KubeconfigPath: cfg.SandboxK8sKubeconfig, + NetworkPolicyName: cfg.SandboxK8sNetworkPolicy, + NodeSelector: nodeSelector, + Tolerations: tolerations, + }) + if err != nil { + return nil, err + } + // Fail closed BEFORE the warm pool spawns its first pod: a cluster that + // cannot run sandboxes (unreachable apiserver, missing RBAC, absent + // workspace claim or sealed-egress policy) must abort boot, never + // silently fall back to podman or host execution. + if err := backend.Preflight(context.Background()); err != nil { + return nil, fmt.Errorf("kubernetes sandbox preflight failed (fail-closed): %w", err) + } + poolCfg.Mode = sandbox.ModeKubernetes + poolCfg.KubernetesBackend = backend + + poolCfg.DefaultNetworkMode = cfg.DefaultNetworkMode + poolCfg.DefaultEgressAllowlist = nil + log.Printf("sandbox: kubernetes backend — image=%s, pool=%d, workspace=%s, namespace=%s, runtime_class=%s", + poolCfg.Container.Image, poolCfg.Size, poolCfg.Container.WorkspaceHostDir, backend.Namespace(), defaultIfEmpty(cfg.SandboxK8sRuntimeClass, "cluster default")) + if poolCfg.PersistentREPL { + log.Printf("sandbox: run_python REPL mode=persistent — one kernel per conversation survives across turns (idle TTL %s, max %d sessions)", + poolCfg.PersistentIdleTTL, cfg.PythonREPLMaxSessions) + } else { + log.Printf("sandbox: run_python REPL mode=per-turn — kernel is fresh each turn (the default)") + } + if cfg.DefaultNetworkMode == sandbox.NetworkModeLockdown { + log.Printf("sandbox: network mode=lockdown — every sandbox pod is labeled %s=none for the deny-all NetworkPolicy (enforcement is the cluster CNI's job — see docs/DEPLOYMENT-KUBERNETES.md)", "fleet.elcanotek.com/egress") + } + return sandbox.NewPool(poolCfg), nil +} + // absSupportingDocs absolutizes the persona/protocol/skill/system-prompt dirs // (plus the uploads root) and drops empties so they can be passed as // ContainerConfig.ReadOnlyMounts. The container backend bind-mounts each at the diff --git a/internal/clientconfig/clientconfig.go b/internal/clientconfig/clientconfig.go index 32652e65..e61c8929 100644 --- a/internal/clientconfig/clientconfig.go +++ b/internal/clientconfig/clientconfig.go @@ -416,6 +416,64 @@ type Sandbox struct { // manifest sandbox.network_allowlist. Empty in allowlisted mode = deny all // egress (best-effort — see ADR-0012). NetworkAllowlist []string + + // Backend selects WHERE sandboxes run (#989 / ADR-0049): "" or "podman" + // for the co-located rootless-Podman backend (the single-box default), or + // "kubernetes" for ephemeral pods in a cluster (the split + // control-plane/runner enterprise path). Stored VERBATIM; the consuming + // layer (cmd/fleet) validates fail-closed and an explicit + // FLEET_SANDBOX_BACKEND env var wins, mirroring sandbox.runtime. + Backend string + + // Kubernetes carries the kubernetes-backend settings (manifest + // sandbox.kubernetes). Meaningful only when the resolved backend is + // "kubernetes"; each FLEET_SANDBOX_K8S_* env var overrides its field. + Kubernetes KubernetesSandbox +} + +// KubernetesSandbox is the resolved sandbox.kubernetes block: where sandbox +// pods run and what they mount. All fields are trusted operator config, same +// authority tier as sandbox.image / sandbox.runtime. +type KubernetesSandbox struct { + // Namespace for sandbox pods (default applied at consume time: + // "fleet-sandboxes" — kept separate from the control plane's namespace so + // RBAC and the deny-all NetworkPolicy stay narrowly scoped). + Namespace string `yaml:"namespace"` + // WorkspaceClaim is the ReadWriteMany PVC (in Namespace) holding the + // workspace root, mounted into every sandbox pod at the same absolute + // path the control plane mounts it. Required for the kubernetes backend. + WorkspaceClaim string `yaml:"workspace_claim"` + // ServiceAccount stamped on sandbox pods (identity only — the token is + // never mounted). + ServiceAccount string `yaml:"service_account"` + // ImagePullSecret for private sandbox-image registries. + ImagePullSecret string `yaml:"image_pull_secret"` + // RuntimeClass selects hypervisor isolation (e.g. kata) — the kubernetes + // counterpart of sandbox.runtime, preflighted fail-closed (ADR-0010). + RuntimeClass string `yaml:"runtime_class"` + // SeccompProfile is a node-local Localhost seccomp profile path (relative + // to the kubelet seccomp root); empty = RuntimeDefault. + SeccompProfile string `yaml:"seccomp_profile"` + // Kubeconfig selects out-of-cluster auth; empty = in-cluster. + Kubeconfig string `yaml:"kubeconfig"` + // NetworkPolicy is the deny-all NetworkPolicy name the boot preflight + // requires to exist (default "fleet-sandbox-deny-all"). + NetworkPolicy string `yaml:"network_policy"` + // NodeSelector pins sandbox pods to labeled nodes (a dedicated runner + // pool). FLEET_SANDBOX_K8S_NODE_SELECTOR ("k=v,k=v") overrides it. + NodeSelector map[string]string `yaml:"node_selector"` + // Tolerations let sandbox pods schedule onto a tainted runner pool. + // FLEET_SANDBOX_K8S_TOLERATIONS (a JSON array) overrides it. + Tolerations []KubernetesToleration `yaml:"tolerations"` +} + +// KubernetesToleration is the manifest shape of one sandbox-pod toleration +// (the four core/v1 fields fleet forwards). +type KubernetesToleration struct { + Key string `yaml:"key" json:"key,omitempty"` + Operator string `yaml:"operator" json:"operator,omitempty"` + Value string `yaml:"value" json:"value,omitempty"` + Effect string `yaml:"effect" json:"effect,omitempty"` } // ResolvedImageRef returns the image reference the fleet process should consume: @@ -429,11 +487,13 @@ func (s Sandbox) ResolvedImageRef() string { // sandboxManifest is the on-disk YAML shape of the manifest's sandbox: block. type sandboxManifest struct { - Containerfile string `yaml:"containerfile"` - Tag string `yaml:"tag"` - Image string `yaml:"image"` - Runtime string `yaml:"runtime"` - NetworkAllowlist []string `yaml:"network_allowlist"` + Containerfile string `yaml:"containerfile"` + Tag string `yaml:"tag"` + Image string `yaml:"image"` + Runtime string `yaml:"runtime"` + NetworkAllowlist []string `yaml:"network_allowlist"` + Backend string `yaml:"backend"` + Kubernetes *KubernetesSandbox `yaml:"kubernetes"` } // Branding carries the white-label strings surfaced in the web UI + login. @@ -1345,12 +1405,18 @@ func resolveSandbox(sm *sandboxManifest, bundleDir string) Sandbox { allowlist = append(allowlist, d) } } + var k8s KubernetesSandbox + if raw.Kubernetes != nil { + k8s = *raw.Kubernetes + } return Sandbox{ ContainerfileAbsPath: filepath.Join(bundleDir, cf), Tag: tag, Image: strings.TrimSpace(raw.Image), Runtime: strings.TrimSpace(raw.Runtime), NetworkAllowlist: allowlist, + Backend: strings.ToLower(strings.TrimSpace(raw.Backend)), + Kubernetes: k8s, } } diff --git a/internal/config/config.go b/internal/config/config.go index 9669ddba..c79c2271 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -383,6 +383,17 @@ var allowedEnvVars = map[string]bool{ "CHAT_WORKSPACE_ROOT": true, "FLEET_SANDBOX_IMAGE": true, "FLEET_SANDBOX_RUNTIME": true, + "FLEET_SANDBOX_BACKEND": true, + "FLEET_SANDBOX_K8S_NAMESPACE": true, + "FLEET_SANDBOX_K8S_WORKSPACE_CLAIM": true, + "FLEET_SANDBOX_K8S_SERVICE_ACCOUNT": true, + "FLEET_SANDBOX_K8S_IMAGE_PULL_SECRET": true, + "FLEET_SANDBOX_K8S_RUNTIME_CLASS": true, + "FLEET_SANDBOX_K8S_SECCOMP_PROFILE": true, + "FLEET_SANDBOX_K8S_KUBECONFIG": true, + "FLEET_SANDBOX_K8S_NETWORK_POLICY": true, + "FLEET_SANDBOX_K8S_NODE_SELECTOR": true, + "FLEET_SANDBOX_K8S_TOLERATIONS": true, "FLEET_DEFAULT_NETWORK_MODE": true, "FLEET_PII_REDACTION_ENABLED": true, "FLEET_PII_REDACTION_MODE": true, @@ -1011,6 +1022,27 @@ type Config struct { // ── sandbox ── SandboxImage string SandboxRuntime string + // SandboxBackend selects WHERE sandboxes run (#989): "" / "podman" — the + // co-located rootless-Podman backend (the single-box default) — or + // "kubernetes" — ephemeral pods in a cluster, for the split + // control-plane/runner enterprise deployment. FLEET_SANDBOX_BACKEND + // overrides the bundle manifest's sandbox.backend (same precedence as + // sandbox.runtime). Anything else refuses to boot (fail-closed, #1119 + // posture: an unrecognized value must never silently mean "podman"). + SandboxBackend string + // SandboxK8s* configure the kubernetes backend; ignored (and refused if + // set, to catch dead config) under the podman backend. Env values override + // the bundle manifest's sandbox.kubernetes block field-by-field. + SandboxK8sNamespace string // FLEET_SANDBOX_K8S_NAMESPACE, default "fleet-sandboxes" + SandboxK8sWorkspaceClaim string // FLEET_SANDBOX_K8S_WORKSPACE_CLAIM — required RWX PVC name + SandboxK8sServiceAccount string // FLEET_SANDBOX_K8S_SERVICE_ACCOUNT — pod identity (no token is ever mounted) + SandboxK8sImagePullSecret string // FLEET_SANDBOX_K8S_IMAGE_PULL_SECRET + SandboxK8sRuntimeClass string // FLEET_SANDBOX_K8S_RUNTIME_CLASS — hypervisor isolation (kata), preflighted + SandboxK8sSeccompProfile string // FLEET_SANDBOX_K8S_SECCOMP_PROFILE — node-local Localhost profile; empty = RuntimeDefault + SandboxK8sKubeconfig string // FLEET_SANDBOX_K8S_KUBECONFIG — out-of-cluster auth; empty = in-cluster + SandboxK8sNetworkPolicy string // FLEET_SANDBOX_K8S_NETWORK_POLICY — deny-all policy the preflight requires; default "fleet-sandbox-deny-all" + SandboxK8sNodeSelector string // FLEET_SANDBOX_K8S_NODE_SELECTOR — "key=value,key=value" pinning sandbox pods to a runner pool + SandboxK8sTolerations string // FLEET_SANDBOX_K8S_TOLERATIONS — JSON array of {key,operator,value,effect} for a tainted runner pool // PIIRedactionEnabled gates the OPTIONAL PII redaction pass (#450) applied to // tool output before it enters the model context. FLEET_PII_REDACTION_ENABLED, // default false (byte-for-byte unchanged when off). Provider-neutral; the @@ -1521,9 +1553,22 @@ func Load(envFile string) (*Config, error) { AdminEmails: splitEmails(os.Getenv("ADMIN_EMAILS")), // ── sandbox ── - SandboxImage: getenvFleet("SANDBOX_IMAGE"), - SandboxRuntime: getenvFleet("SANDBOX_RUNTIME"), - DefaultNetworkMode: strings.ToLower(strings.TrimSpace(getenvFleet("DEFAULT_NETWORK_MODE"))), + SandboxImage: getenvFleet("SANDBOX_IMAGE"), + SandboxRuntime: getenvFleet("SANDBOX_RUNTIME"), + // Sandbox backend (#989). Lower-cased here; validated fail-closed at + // boot (resolveSandboxBackend in cmd/fleet) against the bundle value. + SandboxBackend: strings.ToLower(strings.TrimSpace(getenvFleet("SANDBOX_BACKEND"))), + SandboxK8sNamespace: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NAMESPACE")), + SandboxK8sWorkspaceClaim: strings.TrimSpace(getenvFleet("SANDBOX_K8S_WORKSPACE_CLAIM")), + SandboxK8sServiceAccount: strings.TrimSpace(getenvFleet("SANDBOX_K8S_SERVICE_ACCOUNT")), + SandboxK8sImagePullSecret: strings.TrimSpace(getenvFleet("SANDBOX_K8S_IMAGE_PULL_SECRET")), + SandboxK8sRuntimeClass: strings.TrimSpace(getenvFleet("SANDBOX_K8S_RUNTIME_CLASS")), + SandboxK8sSeccompProfile: strings.TrimSpace(getenvFleet("SANDBOX_K8S_SECCOMP_PROFILE")), + SandboxK8sKubeconfig: strings.TrimSpace(getenvFleet("SANDBOX_K8S_KUBECONFIG")), + SandboxK8sNetworkPolicy: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NETWORK_POLICY")), + SandboxK8sNodeSelector: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NODE_SELECTOR")), + SandboxK8sTolerations: strings.TrimSpace(getenvFleet("SANDBOX_K8S_TOLERATIONS")), + DefaultNetworkMode: strings.ToLower(strings.TrimSpace(getenvFleet("DEFAULT_NETWORK_MODE"))), // PII redaction (#450) — optional, default off. PIIRedactionEnabled: lp.getenvFleetBool("PII_REDACTION_ENABLED", false), diff --git a/internal/sandbox/container.go b/internal/sandbox/container.go index 8eaa850a..679f2729 100644 --- a/internal/sandbox/container.go +++ b/internal/sandbox/container.go @@ -1011,13 +1011,20 @@ func (c *containerImpl) executeFileOp(ctx context.Context, req FileOpRequest, an // it using directory descriptors. A model-controlled Root can therefore never // turn the shared workspace mount into authority over a sibling conversation. func (c *containerImpl) fileOpAnchor(root string) (anchor string, readOnly bool, err error) { + return fileOpAnchorFor(c.cfg.WorkspaceHostDir, c.cfg.ReadOnlyMounts, root) +} + +// fileOpAnchorFor is the backend-shared anchor resolution (see fileOpAnchor's +// doc): the kubernetes backend applies the identical policy over its own +// mount set, so anchor semantics cannot drift between backends. +func fileOpAnchorFor(workspaceDir string, readOnlyMounts []string, root string) (anchor string, readOnly bool, err error) { type mount struct { path string readOnly bool } - candidates := make([]mount, 0, len(c.cfg.ReadOnlyMounts)+1) - candidates = append(candidates, mount{path: c.cfg.WorkspaceHostDir}) - for _, path := range c.cfg.ReadOnlyMounts { + candidates := make([]mount, 0, len(readOnlyMounts)+1) + candidates = append(candidates, mount{path: workspaceDir}) + for _, path := range readOnlyMounts { candidates = append(candidates, mount{path: path, readOnly: true}) } best := "" diff --git a/internal/sandbox/host.go b/internal/sandbox/host.go index 707a2981..fcf9d86a 100644 --- a/internal/sandbox/host.go +++ b/internal/sandbox/host.go @@ -81,6 +81,15 @@ func (h *hostImpl) runBash(ctx context.Context, req BashRequest) (BashResult, er cmdCtx, cancel := context.WithTimeout(ctx, timeout) defer cancel() + // Running the caller's shell IS this component's contract: it is the + // unsandboxed TEST/DEV-ONLY executor behind the fleet_host_executor build + // tag (#159) — a release build gets the fail-closed stub in + // host_disabled.go, so this sink cannot ship (ADR-0002 enforcement). The + // CodeQL go/command-injection finding here is waived in + // .github/codeql-accepted-findings.json for the same reason (the CodeQL + // workflow deliberately compiles this file into the database so it is + // scanned rather than being a coverage hole; in-source codeql[] + // suppressions do not work with that pipeline — see codeql.yml). //nolint:gosec // shell execution is the purpose of this tool cmd := exec.CommandContext(cmdCtx, "bash", "-c", req.Command) if req.WorkingDir != "" { diff --git a/internal/sandbox/k8s_backend.go b/internal/sandbox/k8s_backend.go new file mode 100644 index 00000000..c48633dc --- /dev/null +++ b/internal/sandbox/k8s_backend.go @@ -0,0 +1,1052 @@ +// Copyright (c) 2026 ElcanoTek +// SPDX-License-Identifier: MIT + +package sandbox + +// k8s_backend.go is the Kubernetes sandbox backend (#989): the same per-turn +// execution boundary as the rootless-Podman backend, delivered as an +// ephemeral Pod per sandbox instead of a local container. One Sandbox = one +// Pod running `sleep infinity`; bash is a one-shot exec, the python bridge is +// a held exec session, and file operations run the same embedded fileops.py — +// all over the apiserver's exec subresource, so the fleet control plane never +// shares a kernel with model-authored execution. +// +// What carries over from the podman backend unchanged: the workspace is +// mounted at the SAME absolute path as the control plane sees it (a shared +// RWX PersistentVolumeClaim replaces the bind mount), the rootfs is +// read-only with tmpfs-equivalent emptyDirs for scratch, all capabilities are +// dropped, and a cancelled/timed-out call poisons the sandbox and destroys +// the whole PID namespace — here by deleting the Pod with zero grace (#796). +// +// What is honestly different (documented in docs/DEPLOYMENT-KUBERNETES.md and +// ADR-0049): egress sealing is delegated to a NetworkPolicy the chart ships +// (verified to exist at boot, but ENFORCED by the cluster CNI, not by fleet); +// the per-pod pids limit is not expressible in a Pod spec; the "allowlisted" +// egress mode is unsupported (fail-closed at boot); resource telemetry (#263) +// is not collected; and seccomp is RuntimeDefault or an operator-installed +// Localhost profile rather than the bundled JSON. +// +// MCP credentials keep their ADR-0003 posture automatically: the broker runs +// in the control-plane process, and nothing in a sandbox Pod's spec, env, or +// mounts carries a credential — automountServiceAccountToken is explicitly +// false so a sandbox cannot even talk to the apiserver that created it. + +import ( + "bufio" + "bytes" + "context" + "crypto/rand" + "encoding/base64" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "log" + "os" + "strconv" + "strings" + "sync" + "sync/atomic" + "time" + + "github.com/ElcanoTek/fleet/internal/safe" +) + +// Sandbox backend names (#989): where every sandbox runs. The knob mirrors +// sandbox.runtime's precedence (FLEET_SANDBOX_BACKEND env wins, else the +// bundle manifest's sandbox.backend, else podman). +const ( + // BackendPodman is the co-located rootless-Podman backend — the + // single-box default and the only backend before #989. + BackendPodman = "podman" + // BackendKubernetes runs each sandbox as an ephemeral pod in a cluster: + // the split control-plane/runner enterprise deployment. + BackendKubernetes = "kubernetes" +) + +// ResolveBackend applies the sandbox-backend precedence in ONE place so every +// entrypoint (fleet boot, `fleet validate-config`) resolves identically: an +// explicit env value (FLEET_SANDBOX_BACKEND) wins, else the bundle manifest's +// sandbox.backend, else podman. An unrecognized value is an ERROR, never a +// silent fallback to podman — the #1119 posture: a typo'd security-relevant +// knob must refuse to boot rather than quietly mean something else. +func ResolveBackend(envBackend, bundleBackend string) (string, error) { + raw := strings.ToLower(strings.TrimSpace(envBackend)) + if raw == "" { + raw = strings.ToLower(strings.TrimSpace(bundleBackend)) + } + switch raw { + case "", BackendPodman: + return BackendPodman, nil + case BackendKubernetes: + return BackendKubernetes, nil + default: + return "", fmt.Errorf("unrecognized sandbox backend %q (FLEET_SANDBOX_BACKEND / manifest sandbox.backend): want %q or %q — refusing to guess (fail-closed)", raw, BackendPodman, BackendKubernetes) + } +} + +// KubernetesConfig configures the kubernetes sandbox backend. It is trusted +// operator config, same authority tier as ContainerConfig / sandbox.runtime. +type KubernetesConfig struct { + // Namespace is where sandbox Pods are created. Defaults to + // "fleet-sandboxes"; in-cluster deployments may point it at any namespace + // the RBAC grant covers. Keeping it SEPARATE from the control plane's own + // namespace is what lets the RBAC grant be pod-scoped and narrow. + Namespace string + + // WorkspaceClaim is the name of the ReadWriteMany PersistentVolumeClaim + // (in Namespace) that holds the workspace root. It is mounted into every + // sandbox Pod at ContainerConfig.WorkspaceHostDir — the same absolute path + // the control plane mounts it at — preserving the same-path invariant that + // keeps MCP-returned paths usable inside bash/run_python. Required. + WorkspaceClaim string + + // ServiceAccount, when set, is stamped as the Pod's serviceAccountName. + // The token is never mounted either way (automountServiceAccountToken is + // forced false); this exists so admission policies can key on identity. + ServiceAccount string + + // ImagePullSecret, when set, is attached for pulling the sandbox image + // from a private registry. + ImagePullSecret string + + // RuntimeClassName, when set, selects a hypervisor-isolated runtime class + // (e.g. kata) for sandbox Pods — the k8s counterpart of sandbox.runtime. + // Preflighted to exist, fail-closed, mirroring ADR-0010. + RuntimeClassName string + + // SeccompLocalhostProfile, when set, is a node-local profile path + // (relative to the kubelet's seccomp root) applied as a Localhost seccomp + // profile. Empty means RuntimeDefault. + SeccompLocalhostProfile string + + // KubeconfigPath selects out-of-cluster auth. Empty means in-cluster + // (the standard service-account mount). + KubeconfigPath string + + // NetworkPolicyName is the deny-all NetworkPolicy the boot preflight + // requires to exist in Namespace — the object that seals egress for Pods + // labeled fleet.elcanotek.com/egress=none. Defaults to + // "fleet-sandbox-deny-all". The preflight verifies the OBJECT exists; + // enforcement is the CNI's job and the docs say so plainly. + NetworkPolicyName string + + // NodeSelector pins sandbox pods to labeled nodes — the standard way to + // give runners a DEDICATED node pool, which is the issue's scaling story + // (more runner capacity = a bigger pool, never more fleet replicas). + NodeSelector map[string]string + + // Tolerations let sandbox pods schedule onto a tainted runner pool, the + // usual companion to NodeSelector for a pool nothing else may land on. + Tolerations []K8sToleration + + // StartTimeout caps pod schedule+pull+start. Zero defaults to 2 minutes + // (image pulls make the podman default of 30s unrealistic). + StartTimeout time.Duration +} + +// K8sToleration mirrors the four core/v1 Toleration fields sandbox pods need +// (tolerationSeconds is a drain concern that does not apply to pods fleet +// deletes itself). +type K8sToleration struct { + Key string `json:"key,omitempty"` + Operator string `json:"operator,omitempty"` + Value string `json:"value,omitempty"` + Effect string `json:"effect,omitempty"` +} + +// ParseK8sNodeSelector parses the FLEET_SANDBOX_K8S_NODE_SELECTOR form — +// comma-separated key=value pairs ("pool=fleet-sandboxes,arch=amd64") — into +// the map the pod spec takes. Empty input is a nil map; a malformed pair is +// an error so a typo'd selector refuses to boot instead of silently +// scheduling sandboxes onto the wrong nodes. +func ParseK8sNodeSelector(s string) (map[string]string, error) { + s = strings.TrimSpace(s) + if s == "" { + return nil, nil + } + out := make(map[string]string) + for _, pair := range strings.Split(s, ",") { + k, v, ok := strings.Cut(strings.TrimSpace(pair), "=") + k, v = strings.TrimSpace(k), strings.TrimSpace(v) + if !ok || k == "" || v == "" { + return nil, fmt.Errorf("invalid node selector pair %q (want key=value, comma-separated)", pair) + } + out[k] = v + } + return out, nil +} + +// ParseK8sTolerations parses the FLEET_SANDBOX_K8S_TOLERATIONS form — a JSON +// array of {key, operator, value, effect} objects. Empty input is nil; +// malformed JSON or an unknown field is an error (fail-closed, strict +// decoding, matching the additive-first schema posture). +func ParseK8sTolerations(s string) ([]K8sToleration, error) { + s = strings.TrimSpace(s) + if s == "" { + return nil, nil + } + dec := json.NewDecoder(strings.NewReader(s)) + dec.DisallowUnknownFields() + var out []K8sToleration + if err := dec.Decode(&out); err != nil { + return nil, fmt.Errorf("invalid tolerations JSON (want an array of {key,operator,value,effect}): %w", err) + } + return out, nil +} + +// defaultK8sNamespace / defaultK8sNetworkPolicy are the conventions the Helm +// chart ships; the backend defaults match so a chart install needs no extra +// wiring. +const ( + defaultK8sNamespace = "fleet-sandboxes" + defaultK8sNetworkPolicy = "fleet-sandbox-deny-all" + defaultK8sStartTimeout = 2 * time.Minute +) + +// sandboxContainerName is the single container in every sandbox Pod. +const sandboxContainerName = "sandbox" + +// k8sBridgeDir is the writable emptyDir where the bridge + fileops scripts +// are uploaded at pod start (the k8s counterpart of the /opt/bridge bind +// mount — there is no host filesystem to bind from). +const ( + k8sBridgeDir = "/opt/fleet-bridge" + k8sBridgePath = k8sBridgeDir + "/bridge.py" + k8sFileOpsPath = k8sBridgeDir + "/fileops.py" + k8sPodNamePrefix = "fleet-sandbox-" +) + +// Pod labels. app.kubernetes.io/* follow the k8s recommended-label +// convention; the fleet.elcanotek.com/* pair carries the ownership identity +// the orphan sweep keys on (the k8s counterpart of the podman +// fleet.instance label) and the egress posture the chart's NetworkPolicies +// select on. +const ( + k8sLabelName = "app.kubernetes.io/name" + k8sLabelManagedBy = "app.kubernetes.io/managed-by" + k8sLabelInstance = "fleet.elcanotek.com/instance" + k8sLabelEgress = "fleet.elcanotek.com/egress" + k8sLabelNameValue = "fleet-sandbox" + k8sLabelManagedVal = "fleet" +) + +// k8sInstanceLabel is thisInstanceLabel ("@") re-encoded to the +// label-safe form "p-t" ('@' is not a legal label character). +var k8sInstanceLabel = func() string { + pid, start, _ := instanceLabelOwner(thisInstanceLabel) + return fmt.Sprintf("p%d-t%d", pid, start) +}() + +// parseK8sInstanceLabel inverts k8sInstanceLabel's encoding. ok is false for +// anything unparseable — callers treat that as "ownership unknown". +func parseK8sInstanceLabel(label string) (pid int, startedAt int64, ok bool) { + rest, found := strings.CutPrefix(label, "p") + if !found { + return 0, 0, false + } + pidStr, startStr, found := strings.Cut(rest, "-t") + if !found { + return 0, 0, false + } + pid, err := strconv.Atoi(pidStr) + if err != nil || pid <= 0 { + return 0, 0, false + } + startedAt, err = strconv.ParseInt(startStr, 10, 64) + if err != nil || startedAt <= 0 { + return pid, 0, true + } + return pid, startedAt, true +} + +// KubernetesBackend is the boot-built handle for the kubernetes sandbox +// backend: one API client plus the resolved config, shared by the pool, the +// preflight, and the orphan sweep. Construct with NewKubernetesBackend. +type KubernetesBackend struct { + cfg KubernetesConfig + client *k8sClient +} + +// NewKubernetesBackend resolves credentials (in-cluster unless a kubeconfig +// is configured) and defaults, returning the backend handle. It performs no +// network I/O — Preflight does the fail-closed cluster checks. +func NewKubernetesBackend(cfg KubernetesConfig) (*KubernetesBackend, error) { + var ( + client *k8sClient + kubeconfigNS string + err error + ) + if cfg.KubeconfigPath != "" { + client, kubeconfigNS, err = newKubeconfigClient(cfg.KubeconfigPath) + } else { + client, err = newInClusterClient() + } + if err != nil { + return nil, fmt.Errorf("kubernetes sandbox backend: %w", err) + } + if cfg.Namespace == "" { + // Precedence: explicit config, else the kubeconfig context's + // namespace, else (in-cluster) the control plane's own namespace — + // the Helm chart's default topology, because a PersistentVolumeClaim + // cannot be mounted across namespaces and the workspace claim is + // shared with the control plane — else the shipped default name. + cfg.Namespace = kubeconfigNS + if cfg.Namespace == "" && cfg.KubeconfigPath == "" { + cfg.Namespace = inClusterNamespace() + } + if cfg.Namespace == "" { + cfg.Namespace = defaultK8sNamespace + } + } + if cfg.NetworkPolicyName == "" { + cfg.NetworkPolicyName = defaultK8sNetworkPolicy + } + if cfg.StartTimeout <= 0 { + cfg.StartTimeout = defaultK8sStartTimeout + } + return &KubernetesBackend{cfg: cfg, client: client}, nil +} + +// Namespace reports the resolved sandbox namespace (for logs and doctor +// output). +func (b *KubernetesBackend) Namespace() string { return b.cfg.Namespace } + +// StartTimeout reports the resolved pod start ceiling; the pool derives its +// outer construction contexts from it (mirroring resolveStartTimeout). +func (b *KubernetesBackend) StartTimeout() time.Duration { return b.cfg.StartTimeout } + +// newSandbox starts one sandbox Pod and returns the wrapping handle. cfg +// carries the backend-shared knobs (image, workspace path, limits, network +// posture); the pool routes here from the same take paths that call +// NewContainer for podman. +func (b *KubernetesBackend) newSandbox(ctx context.Context, cfg ContainerConfig) (*Sandbox, error) { + if cfg.Image == "" { + return nil, fmt.Errorf("sandbox: ContainerConfig.Image required") + } + if cfg.WorkspaceHostDir == "" { + return nil, fmt.Errorf("sandbox: ContainerConfig.WorkspaceHostDir required") + } + if cfg.BridgeScript == nil { + return nil, fmt.Errorf("sandbox: ContainerConfig.BridgeScript required") + } + if cfg.ProxyURL != "" { + // The allowlisted egress proxy binds to the control-plane host's + // loopback; a pod on another node cannot reach it, and pretending + // otherwise would grant open egress under an "allowlisted" banner. + return nil, errors.New("sandbox: allowlisted egress mode is not supported by the kubernetes backend (fail-closed)") + } + cfg = applyContainerDefaults(cfg) + k := &k8sImpl{backend: b, cfg: cfg} + if err := k.start(ctx); err != nil { + k.close() + return nil, err + } + return &Sandbox{mode: ModeKubernetes, impl: k}, nil +} + +// k8sImpl is the kubernetes impl: one Pod, exec'd into for every operation. +// The struct mirrors containerImpl's locking discipline — podMu guards the +// pod name (cleared by close, snapshotted by cross-goroutine readers), mu +// guards the bridge session and is held for a whole run_python cell. +type k8sImpl struct { + backend *KubernetesBackend + cfg ContainerConfig + + podMu sync.Mutex + podName string + + mu sync.Mutex + bridge *k8sExecSession + bridgeStdout *bufio.Reader + bridgeStderr *syncBuffer + bridgeStarted bool + + execPoisoned atomic.Bool +} + +// generatePodName mirrors generateContainerName with the pod prefix. +func generatePodName() string { + var buf [8]byte + _, _ = rand.Read(buf[:]) + return k8sPodNamePrefix + hex.EncodeToString(buf[:]) +} + +// k8sQuantityFromPodmanMemory converts a podman --memory value ("512m", +// "2g", bare bytes) into a Kubernetes resource quantity (plain bytes — +// unambiguous and accepted everywhere a quantity is). +func k8sQuantityFromPodmanMemory(limit string) (string, error) { + b, err := parseMemoryToBytes(limit) + if err != nil { + return "", err + } + return strconv.FormatInt(b, 10), nil +} + +// k8sQuantityFromPodmanCPU converts a podman --cpus value ("1.0", "2.50") +// into a Kubernetes CPU quantity in millicores. +func k8sQuantityFromPodmanCPU(limit string) (string, error) { + f, err := strconv.ParseFloat(strings.TrimSpace(limit), 64) + if err != nil || f <= 0 { + return "", fmt.Errorf("invalid cpu limit %q", limit) + } + return strconv.FormatInt(int64(f*1000), 10) + "m", nil +} + +// buildSandboxPod is the pure pod-spec builder — the k8s counterpart of the +// `podman run` argument list in containerImpl.start, kept side-effect-free so +// the hardening posture is pinned by unit tests the way podman_args_test.go +// pins the flag list. +func buildSandboxPod(cfg ContainerConfig, kcfg KubernetesConfig, name string) (*k8sPod, error) { + memory, err := k8sQuantityFromPodmanMemory(cfg.MemoryLimit) + if err != nil { + return nil, fmt.Errorf("sandbox pod memory limit: %w", err) + } + cpu, err := k8sQuantityFromPodmanCPU(cfg.CPULimit) + if err != nil { + return nil, fmt.Errorf("sandbox pod cpu limit: %w", err) + } + + egress := "open" + if cfg.NoNetwork { + // Sealed posture: selected by the deny-all NetworkPolicy the preflight + // verified. The label is the contract; enforcement is the CNI's. + egress = "none" + } + + boolPtr := func(v bool) *bool { return &v } + int64Ptr := func(v int64) *int64 { return &v } + + limits := map[string]string{"memory": memory, "cpu": cpu} + requests := map[string]string{"memory": memory, "cpu": cpu} + if cfg.DiskLimitGB > 0 { + // ephemeral-storage caps the pod's writable layer AND its emptyDirs — + // a strictly stronger surface than podman's per-file ulimit + layer + // quota. The workspace PVC is still outside it, exactly like the bind + // mount is outside podman's quota; the docs carry the same honest + // limit statement. + limits["ephemeral-storage"] = fmt.Sprintf("%dGi", cfg.DiskLimitGB) + } + + seccomp := &k8sSeccompProfile{Type: "RuntimeDefault"} + if kcfg.SeccompLocalhostProfile != "" { + profile := kcfg.SeccompLocalhostProfile + seccomp = &k8sSeccompProfile{Type: "Localhost", LocalhostProfile: &profile} + } + + // emptyDir scratch mounts mirror the podman --tmpfs set (sizes included) + // so a --read-only-rootfs image behaves identically in both backends, + // plus the bridge dir the scripts are uploaded into. + mounts := []k8sVolumeMount{ + {Name: "workspace", MountPath: cfg.WorkspaceHostDir}, + {Name: "bridge", MountPath: k8sBridgeDir}, + {Name: "tmp", MountPath: "/tmp"}, + {Name: "ipython", MountPath: "/home/sandbox/.ipython"}, + {Name: "cache", MountPath: "/home/sandbox/.cache"}, + {Name: "config", MountPath: "/home/sandbox/.config"}, + } + volumes := []k8sVolume{ + {Name: "workspace", PersistentVolumeClaim: &k8sPVCVolSource{ClaimName: kcfg.WorkspaceClaim}}, + {Name: "bridge", EmptyDir: &k8sEmptyDir{SizeLimit: "8Mi"}}, + {Name: "tmp", EmptyDir: &k8sEmptyDir{SizeLimit: "128Mi"}}, + {Name: "ipython", EmptyDir: &k8sEmptyDir{SizeLimit: "32Mi"}}, + {Name: "cache", EmptyDir: &k8sEmptyDir{SizeLimit: "32Mi"}}, + {Name: "config", EmptyDir: &k8sEmptyDir{SizeLimit: "8Mi"}}, + } + + spec := k8sPodSpec{ + RestartPolicy: "Never", + // The sandbox must not be able to reach the apiserver that made it: + // no token, no service links, ever. + AutomountServiceAccountToken: boolPtr(false), + EnableServiceLinks: boolPtr(false), + TerminationGracePeriodSeconds: int64Ptr(5), + ServiceAccountName: kcfg.ServiceAccount, + SecurityContext: &k8sPodSecurityCtx{ + RunAsNonRoot: boolPtr(true), + // uid/gid 1000 matches the image's USER sandbox and the podman + // keep-id mapping, so workspace files are owned consistently across + // backends. fsGroup makes the PVC group-writable for that gid. + RunAsUser: int64Ptr(1000), + RunAsGroup: int64Ptr(1000), + FSGroup: int64Ptr(1000), + SeccompProfile: seccomp, + }, + Containers: []k8sContainer{{ + Name: sandboxContainerName, + Image: cfg.Image, + // Explicit IfNotPresent: the API default for a :latest tag is + // Always, which breaks side-loaded images (kind) and re-pulls a + // mutable tag mid-fleet-run — sandbox image freshness is a deploy + // concern, not a per-pod one. + ImagePullPolicy: "IfNotPresent", + // PID 1: a do-nothing process to keep the pod alive; every real + // operation execs into it — same shape as the podman backend. + Command: []string{"sleep", "infinity"}, + WorkingDir: cfg.WorkspaceHostDir, + SecurityContext: &k8sContainerSecCtx{ + AllowPrivilegeEscalation: boolPtr(false), + ReadOnlyRootFilesystem: boolPtr(true), + Capabilities: &k8sCapabilities{Drop: []string{"ALL"}}, + }, + Resources: &k8sResources{Limits: limits, Requests: requests}, + VolumeMounts: mounts, + }}, + Volumes: volumes, + } + if kcfg.RuntimeClassName != "" { + rc := kcfg.RuntimeClassName + spec.RuntimeClassName = &rc + } + if kcfg.ImagePullSecret != "" { + spec.ImagePullSecrets = []k8sLocalObjRef{{Name: kcfg.ImagePullSecret}} + } + // Dedicated runner pool: selector + taints, when configured. + if len(kcfg.NodeSelector) > 0 { + spec.NodeSelector = kcfg.NodeSelector + } + for _, tol := range kcfg.Tolerations { + spec.Tolerations = append(spec.Tolerations, k8sToleration(tol)) + } + + return &k8sPod{ + Metadata: k8sObjectMeta{ + Name: name, + Namespace: kcfg.Namespace, + Labels: map[string]string{ + k8sLabelName: k8sLabelNameValue, + k8sLabelManagedBy: k8sLabelManagedVal, + k8sLabelInstance: k8sInstanceLabel, + k8sLabelEgress: egress, + }, + }, + Spec: spec, + }, nil +} + +// k8sPodPollInterval is how often start() re-reads the pod while waiting for +// Running. +const k8sPodPollInterval = 500 * time.Millisecond + +func (k *k8sImpl) start(ctx context.Context) error { + name := generatePodName() + k.podMu.Lock() + k.podName = name + k.podMu.Unlock() + + pod, err := buildSandboxPod(k.cfg, k.backend.cfg, name) + if err != nil { + return err + } + startCtx, cancel := context.WithTimeout(ctx, k.backend.cfg.StartTimeout) + defer cancel() + if err := k.backend.client.createPod(startCtx, k.backend.cfg.Namespace, pod); err != nil { + return fmt.Errorf("create sandbox pod: %w", err) + } + if err := k.waitForRunning(startCtx, name); err != nil { + return err + } + // Upload the bridge + fileops scripts into the pod's writable bridge + // emptyDir. There is no host filesystem to bind-mount them from; the + // upload verifies byte counts so a truncated transfer fails loudly here + // rather than as an opaque bridge error mid-turn. + if err := k.uploadFile(startCtx, k8sBridgePath, k.cfg.BridgeScript); err != nil { + return fmt.Errorf("upload bridge script: %w", err) + } + if err := k.uploadFile(startCtx, k8sFileOpsPath, fileOpsScript); err != nil { + return fmt.Errorf("upload fileops script: %w", err) + } + return nil +} + +func (k *k8sImpl) waitForRunning(ctx context.Context, name string) error { + ticker := time.NewTicker(k8sPodPollInterval) + defer ticker.Stop() + var lastState string + for { + pod, err := k.backend.client.getPod(ctx, k.backend.cfg.Namespace, name) + if err == nil { + switch pod.Status.Phase { + case "Running": + for _, cs := range pod.Status.ContainerStatuses { + if cs.Name == sandboxContainerName && cs.Ready { + return nil + } + } + case "Failed", "Succeeded": + // Status text is cluster-derived — sanitized before it enters + // an error that upstream code logs (go/log-injection). + return fmt.Errorf("sandbox pod %s entered terminal phase %s before becoming ready: %s", name, sanitizeClusterText(pod.Status.Phase), sanitizeClusterText(pod.Status.Message)) + } + for _, cs := range pod.Status.ContainerStatuses { + if cs.State.Waiting != nil { + lastState = sanitizeClusterText(cs.State.Waiting.Reason) + // Pull failures never self-heal within a start timeout — + // fail fast with the reason instead of burning the window. + if lastState == "ErrImagePull" || lastState == "ImagePullBackOff" || lastState == "InvalidImageName" { + return fmt.Errorf("sandbox pod %s cannot pull image %s (%s): %s", name, k.cfg.Image, lastState, sanitizeClusterText(cs.State.Waiting.Message)) + } + } + } + } + select { + case <-ctx.Done(): + if lastState != "" { + return fmt.Errorf("sandbox pod %s not ready before start timeout (last container state: %s): %w", name, lastState, ctx.Err()) + } + return fmt.Errorf("sandbox pod %s not ready before start timeout: %w", name, ctx.Err()) + case <-ticker.C: + } + } +} + +// uploadFile writes data to path inside the pod via a one-shot exec. The v4 +// exec protocol cannot half-close stdin, so the reader is bounded with +// `head -c `; the write is verified by byte count. +func (k *k8sImpl) uploadFile(ctx context.Context, path string, data []byte) error { + podName := k.currentPodName() + if podName == "" { + return ErrClosed + } + // path is one of the two fixed k8sBridgeDir constants — never + // model-supplied — so embedding it in the shell line is safe. + script := fmt.Sprintf("head -c %d > %s && wc -c < %s", len(data), path, path) + var stdout, stderr bytes.Buffer + code, err := k.backend.client.runOneShotExec(ctx, k.backend.cfg.Namespace, podName, sandboxContainerName, + []string{"/bin/sh", "-c", script}, data, &stdout, &stderr) + if err != nil { + return fmt.Errorf("upload %s: %w", path, err) + } + if code != 0 { + // stderr is pod output — sanitized like all cluster-derived text. + return fmt.Errorf("upload %s: exit %d (%.200s)", path, code, sanitizeClusterText(stderr.String())) + } + if got := sanitizeClusterText(strings.TrimSpace(stdout.String())); got != strconv.Itoa(len(data)) { + return fmt.Errorf("upload %s: wrote %s of %d bytes", path, got, len(data)) + } + return nil +} + +// currentPodName snapshots the pod name under podMu — same discipline as +// containerImpl.currentContainerID; "" means already torn down. +func (k *k8sImpl) currentPodName() string { + k.podMu.Lock() + defer k.podMu.Unlock() + return k.podName +} + +func (k *k8sImpl) poisoned() bool { return k.execPoisoned.Load() } + +// deletePodNow removes the pod immediately on a fresh context — the #796 +// containment: destroying the pod destroys its PID namespace and every +// straggler in it. Reports whether the deletion (or prior disappearance) was +// confirmed. Mirrors killContainerNow, including taking the name as a +// parameter so it never touches k.mu. +func (k *k8sImpl) deletePodNow(podName string) bool { + if podName == "" { + return true // already torn down by close() + } + delCtx, cancel := context.WithTimeout(context.Background(), execReapTimeout) + defer cancel() + if err := k.backend.client.deletePod(delCtx, k.backend.cfg.Namespace, podName); err != nil { + if isK8sNotFound(err) { + return true + } + log.Printf("sandbox: cancelled-exec pod delete unconfirmed (%s): %v", podName, err) + return false + } + return true +} + +func (k *k8sImpl) runBash(ctx context.Context, req BashRequest) (BashResult, error) { + podName := k.currentPodName() + if podName == "" { + return BashResult{}, fmt.Errorf("run bash: %w", ErrClosed) + } + timeout := req.Timeout + if timeout <= 0 { + timeout = 5 * time.Minute + } + cmdCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + + // The exec API has no --workdir; a positional-parameter wrapper applies + // the cwd without any quoting of the user command or the directory. + command := []string{"bash", "-c", req.Command} + if req.WorkingDir != "" { + command = []string{"/bin/sh", "-c", `cd -- "$1" || exit 126; shift; exec bash -c "$1"`, "fleet-bash", req.WorkingDir, req.Command} + } + + stdoutBuf := &cappedBuffer{cap: BashOutputCaptureCap} + stderrBuf := &cappedBuffer{cap: BashOutputCaptureCap} + code, execErr := k.backend.client.runOneShotExec(cmdCtx, k.backend.cfg.Namespace, podName, sandboxContainerName, + command, nil, stdoutBuf, stderrBuf) + + res := BashResult{ + ExitCode: code, + Stdout: stdoutBuf.buf.Bytes(), + Stderr: stderrBuf.buf.Bytes(), + StdoutDiscarded: stdoutBuf.discarded, + StderrDiscarded: stderrBuf.discarded, + } + if cmdCtx.Err() != nil { + // Cancellation/timeout only tore down the exec CONNECTION; the shell + // and its descendants keep running in the pod (#796). Delete the pod + // synchronously — destroying the PID namespace is the one guaranteed + // containment — and poison the sandbox so it is retired, exactly like + // the podman backend. + res.TimedOut = errors.Is(cmdCtx.Err(), context.DeadlineExceeded) + res.Cancelled = !res.TimedOut + k.execPoisoned.Store(true) + res.CleanupConfirmed = k.deletePodNow(podName) + res.SandboxRetired = true + return res, nil + } + if execErr != nil { + return res, fmt.Errorf("pod exec bash: %w", execErr) + } + return res, nil +} + +func (k *k8sImpl) runFileOp(ctx context.Context, req FileOpRequest) (FileOpResult, error) { + anchor, readOnly, err := fileOpAnchorFor(k.cfg.WorkspaceHostDir, k.cfg.ReadOnlyMounts, req.Root) + if err != nil { + return FileOpResult{}, err + } + if readOnly && req.Op != FileOpRead { + return FileOpResult{}, fmt.Errorf("fileop write requested beneath a read-only mount: %w", ErrFileOpUnsafePath) + } + if !readOnly && !req.rootBound { + return FileOpResult{}, fmt.Errorf("writable fileop root was not bound before the turn: %w", ErrFileOpUnsafePath) + } + return k.executeFileOp(ctx, req, anchor) +} + +func (k *k8sImpl) bindFileOpRoot(ctx context.Context, root string) (FileOpRootIdentity, error) { + anchor, readOnly, err := fileOpAnchorFor(k.cfg.WorkspaceHostDir, k.cfg.ReadOnlyMounts, root) + if err != nil { + return FileOpRootIdentity{}, err + } + if readOnly { + return FileOpRootIdentity{}, fmt.Errorf("cannot bind a read-only mount as the writable fileop root: %w", ErrFileOpUnsafePath) + } + res, err := k.executeFileOp(ctx, FileOpRequest{Op: fileOpBindRoot, Path: root, Root: root}, anchor) + if err != nil { + return FileOpRootIdentity{}, err + } + return res.rootIdentity, nil +} + +func (k *k8sImpl) executeFileOp(ctx context.Context, req FileOpRequest, anchor string) (FileOpResult, error) { + podName := k.currentPodName() + if podName == "" { + return FileOpResult{}, fmt.Errorf("fileop %s: %w", req.Op, ErrClosed) + } + cmdCtx, cancel := context.WithTimeout(ctx, fileOpTimeout) + defer cancel() + + reqJSON, err := encodeFileOpWire(req, anchor) + if err != nil { + return FileOpResult{}, err + } + + // fileops.py reads stdin to EOF; v4 exec cannot half-close stdin, so the + // read is bounded with `head -c ` — EOF arrives when head exits. + script := fmt.Sprintf("head -c %d | python3 %s", len(reqJSON), k8sFileOpsPath) + var stdout, stderr bytes.Buffer + code, execErr := k.backend.client.runOneShotExec(cmdCtx, k.backend.cfg.Namespace, podName, sandboxContainerName, + []string{"/bin/sh", "-c", script}, reqJSON, &stdout, &stderr) + if execErr != nil || cmdCtx.Err() != nil { + if cmdCtx.Err() != nil { + // Same containment as bash: the helper may still be alive inside a + // persistent pod and could complete a rename after the turn stopped. + k.execPoisoned.Store(true) + _ = k.deletePodNow(podName) + return FileOpResult{}, fmt.Errorf("fileop %s interrupted (%w); sandbox retired: %w", req.Op, cmdCtx.Err(), ErrPoisoned) + } + return FileOpResult{}, fmt.Errorf("fileop %s exec: %w (%.200s)", req.Op, execErr, sanitizeClusterText(stderr.String())) + } + if code != 0 { + return FileOpResult{}, fmt.Errorf("fileop %s: helper exit %d (%.200s)", req.Op, code, sanitizeClusterText(stderr.String())) + } + return decodeFileOpResponse(stdout.Bytes()) +} + +// encodeFileOpWire builds the JSON request fileops.py reads — shared shape +// with the podman/host backends (their inline wire-building predates this +// helper; the k8s backend uses it so the three cannot drift further). +func encodeFileOpWire(req FileOpRequest, anchor string) ([]byte, error) { + wire := map[string]any{ + "op": string(req.Op), + "path": req.Path, + "root": req.Root, + "anchor": anchor, + } + switch req.Op { + case FileOpRead: + wire["offset"] = req.Offset + wire["limit"] = req.Limit + case FileOpWrite: + wire["data_b64"] = base64.StdEncoding.EncodeToString(req.Data) + case FileOpEdit: + wire["old_b64"] = base64.StdEncoding.EncodeToString([]byte(req.OldText)) + wire["new_b64"] = base64.StdEncoding.EncodeToString([]byte(req.NewText)) + wire["replace_all"] = req.ReplaceAll + if req.ExpectedSHA256 != "" { + wire["expected_sha256"] = req.ExpectedSHA256 + } + case fileOpBindRoot: + // Root + anchor are the complete request. + default: + return nil, fmt.Errorf("unknown fileop %q", req.Op) + } + if req.testPause > 0 { + wire["test_pause_ms"] = req.testPause.Milliseconds() + wire["test_ready_name"] = req.testReadyName + } + if req.rootBound { + wire["expected_dev"] = req.expectedDev + wire["expected_ino"] = req.expectedIno + } + out, err := json.Marshal(wire) + if err != nil { + return nil, fmt.Errorf("marshal fileop: %w", err) + } + return out, nil +} + +func (k *k8sImpl) runPython(ctx context.Context, req PythonRequest) (PythonResult, error) { + timeout := req.Timeout + if timeout <= 0 { + timeout = 5 * time.Minute + } + if err := k.ensureBridge(); err != nil { + return PythonResult{}, fmt.Errorf("start python bridge in pod: %w", err) + } + + wireReq := bridgeRequest{ + Code: req.Code, + ReturnVars: req.ReturnVars, + TimeoutSeconds: int(timeout.Seconds()), + WorkspaceDir: req.WorkspaceDir, + ResetKernel: req.ResetKernel, + } + reqBytes, err := json.Marshal(wireReq) + if err != nil { + return PythonResult{}, fmt.Errorf("marshal bridge request: %w", err) + } + + k.mu.Lock() + defer k.mu.Unlock() + + // Re-validate under the lock (mirrors containerImpl): a concurrent close() + // nils the bridge fields between ensureBridge and this re-acquire. + if k.bridge == nil || k.bridgeStdout == nil { + return PythonResult{}, fmt.Errorf("send bridge request: %w", ErrClosed) + } + if err := k.bridge.writeStdin(append(reqBytes, '\n')); err != nil { + k.terminateBridgeLocked() + return PythonResult{}, fmt.Errorf("send bridge request: %w%s", err, k.bridgeStderrSuffix()) + } + + type readResult struct { + data []byte + discarded int64 + err error + } + ch := make(chan readResult, 1) + // Snapshot the reader before launching the goroutine — the cancel/timeout + // arms nil the field via terminateBridgeLocked (#583's lesson, upheld). + stdout := k.bridgeStdout + go func() { + defer safe.Recover("sandbox.k8s.bridge_read", func(any) { + ch <- readResult{err: fmt.Errorf("bridge reader panicked")} + }) + data, discarded, err := readCappedLine(stdout, bridgeResponseCaptureCap) + ch <- readResult{data: data, discarded: discarded, err: err} + }() + timer := time.NewTimer(timeout) + defer timer.Stop() + select { + case <-ctx.Done(): + // The cell keeps executing in the pod until its PID namespace goes + // away — delete the pod and poison, exactly the podman #796 handling. + k.execPoisoned.Store(true) + _ = k.deletePodNow(k.currentPodName()) + k.terminateBridgeLocked() + return PythonResult{}, fmt.Errorf("python execution cancelled (%w); sandbox retired: %w", ctx.Err(), ErrPoisoned) + case <-timer.C: + k.execPoisoned.Store(true) + _ = k.deletePodNow(k.currentPodName()) + k.terminateBridgeLocked() + return PythonResult{}, fmt.Errorf("python execution timed out after %v; sandbox retired: %w", timeout, ErrPoisoned) + case r := <-ch: + if r.err != nil { + // Session dead (pod-side bridge exited, connection dropped). The pod + // itself is intact — reset the bridge so the next call boots fresh. + k.terminateBridgeLocked() + return PythonResult{}, fmt.Errorf("bridge closed unexpectedly: %w%s", r.err, k.bridgeStderrSuffix()) + } + if r.discarded > 0 { + return PythonResult{}, fmt.Errorf("bridge response exceeded %d bytes (%d bytes discarded) and was dropped — return large results by writing them to a workspace file instead", bridgeResponseCaptureCap, r.discarded) + } + return parseBridgeResponse(r.data) + } +} + +// ensureBridge starts the bridge exec session on first use, mirroring +// containerImpl.ensureBridge (including the post-start settle delay outside +// the lock). +func (k *k8sImpl) ensureBridge() error { + started, err := k.startBridgeIfNeeded() + if err != nil || !started { + return err + } + time.Sleep(100 * time.Millisecond) + return nil +} + +func (k *k8sImpl) startBridgeIfNeeded() (started bool, err error) { + k.mu.Lock() + defer k.mu.Unlock() + + if k.bridgeStarted && k.bridge != nil { + select { + case <-k.bridge.done: + // Session ended behind our back (pod-side exit) — fall through and + // start a fresh one. + default: + return false, nil + } + } + + podName := k.currentPodName() + if podName == "" { + return false, fmt.Errorf("start bridge: %w", ErrClosed) + } + + // The bridge reads JSON-per-line from stdin and writes JSON-per-line to + // stdout; the demux loop feeds a pipe the reader side wraps in bufio. + pr, pw := io.Pipe() + stderrBuf := &syncBuffer{} + // The bridge intentionally outlives any single request ctx and is torn + // down in close() / terminateBridgeLocked — dial under a handshake-bounded + // background context, mirroring the podman backend's noctx bridge exec. + session, err := k.backend.client.execPod(context.Background(), k.backend.cfg.Namespace, podName, sandboxContainerName, + []string{"python3", k8sBridgePath}, true, pw, io.MultiWriter(os.Stderr, stderrBuf)) + if err != nil { + _ = pw.Close() + _ = pr.Close() + return false, fmt.Errorf("start bridge exec: %w", err) + } + // Unblock any reader waiting on the pipe once the session ends — the + // demux loop does not own the pipe writer. + go func() { + <-session.done + _ = pw.CloseWithError(io.EOF) + }() + k.bridge = session + k.bridgeStdout = bufio.NewReader(pr) + k.bridgeStderr = stderrBuf + k.bridgeStarted = true + return true, nil +} + +// terminateBridgeLocked tears the bridge session down and clears the state so +// the next ensureBridge starts fresh. Caller must hold k.mu. Closing the +// websocket ends the server-side exec streams; an orphaned kernel inside a +// still-live pod is reaped by the next bridge start (reap_stale_kernels in +// python_bridge.py), the same recovery story as the podman backend. +func (k *k8sImpl) terminateBridgeLocked() { + if k.bridge != nil { + k.bridge.close() + } + k.bridge = nil + k.bridgeStdout = nil + k.bridgeStarted = false +} + +func (k *k8sImpl) bridgeStderrSuffix() string { + if k.bridgeStderr == nil { + return "" + } + // Pod output — sanitized like all cluster-derived text before it joins an + // error string that upstream code logs. + stderr := sanitizeClusterText(strings.TrimSpace(k.bridgeStderr.Snapshot())) + if stderr == "" { + return "" + } + const maxLen = 1024 + if len(stderr) > maxLen { + stderr = stderr[len(stderr)-maxLen:] + } + return " (bridge stderr: " + stderr + ")" +} + +// resourceUsage reports no telemetry: the k8s backend has no `podman stats` +// counterpart wired up (kubelet metrics need a different collection path). +// Recorded as an honest deviation in docs/DEPLOYMENT-KUBERNETES.md. +func (k *k8sImpl) resourceUsage() (ResourceUsageSummary, bool) { + return ResourceUsageSummary{}, false +} + +func (k *k8sImpl) close() { + k.mu.Lock() + k.terminateBridgeLocked() + k.bridgeStderr = nil + k.mu.Unlock() + + k.podMu.Lock() + podName := k.podName + k.podName = "" + k.podMu.Unlock() + + if podName != "" { + delCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + if err := k.backend.client.deletePod(delCtx, k.backend.cfg.Namespace, podName); err != nil && !isK8sNotFound(err) { + log.Printf("sandbox: close-time pod delete unconfirmed (%s): %v — the pod may linger until the boot-time orphan prune", podName, err) + } + } +} + +// PruneOrphanedPods removes leftover sandbox pods from a prior fleet process +// (a crash never runs close(), so in-flight + warm pods outlive it). The +// kubernetes counterpart of PruneOrphanedContainers, with the same ownership +// discipline: pods carrying THIS process's instance label are never touched +// (the warm pool is already filling when the sweep runs); other instances' +// pods are removed only when their labeled owner pid is provably not running +// here anymore — and since a restarted control plane is a fresh process (in +// k8s, usually a fresh container), a prior incarnation's pods always qualify. +func (b *KubernetesBackend) PruneOrphanedPods(ctx context.Context) (int, error) { + selector := k8sLabelName + "=" + k8sLabelNameValue + "," + k8sLabelManagedBy + "=" + k8sLabelManagedVal + list, err := b.client.listPods(ctx, b.cfg.Namespace, selector) + if err != nil { + return 0, fmt.Errorf("list orphaned sandbox pods: %w", err) + } + removed := 0 + for _, pod := range list.Items { + label := pod.Metadata.Labels[k8sLabelInstance] + if label == k8sInstanceLabel { + continue + } + pid, startedAt, ok := parseK8sInstanceLabel(label) + if ok && labeledOwnerStillRunning(pid, startedAt) { + // A live process in THIS pid namespace owns it (a sibling fleet + // process sharing the namespace) — leave it alone. Leaking a pod is + // recoverable; deleting a live sibling's sandbox mid-turn is not. + continue + } + if err := b.client.deletePod(ctx, b.cfg.Namespace, pod.Metadata.Name); err != nil { + if isK8sNotFound(err) { + continue + } + // The name came back from the API list — sanitized like all + // cluster-derived text before it enters a logged error. + return removed, fmt.Errorf("remove orphaned sandbox pod %s: %w", sanitizeClusterText(pod.Metadata.Name), err) + } + removed++ + } + return removed, nil +} diff --git a/internal/sandbox/k8s_backend_test.go b/internal/sandbox/k8s_backend_test.go new file mode 100644 index 00000000..c315d4c4 --- /dev/null +++ b/internal/sandbox/k8s_backend_test.go @@ -0,0 +1,562 @@ +package sandbox + +// Tests for the kubernetes sandbox backend (#989): pod lifecycle, exec +// transport, the #796 poison-and-retire containment, fileops through the real +// executor, the pool routing, and the orphan sweep — all against the fake +// apiserver in k8s_fake_test.go. + +import ( + "context" + "errors" + "io" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/gorilla/websocket" +) + +func testContainerConfig(t *testing.T) ContainerConfig { + t.Helper() + return ContainerConfig{ + Image: "registry.example/fleet-sandbox:test", + WorkspaceHostDir: t.TempDir(), + BridgeScript: []byte("# fake bridge\n"), + } +} + +func TestK8sResolveBackend(t *testing.T) { + cases := []struct { + env, bundle, want string + wantErr bool + }{ + {"", "", BackendPodman, false}, + {"podman", "kubernetes", BackendPodman, false}, + {"", "kubernetes", BackendKubernetes, false}, + {"KUBERNETES", "", BackendKubernetes, false}, + {" kubernetes ", "", BackendKubernetes, false}, + {"", "docker", "", true}, + {"k8s", "", "", true}, + } + for _, tc := range cases { + got, err := ResolveBackend(tc.env, tc.bundle) + if tc.wantErr { + if err == nil { + t.Errorf("ResolveBackend(%q,%q): want error, got %q", tc.env, tc.bundle, got) + } + continue + } + if err != nil || got != tc.want { + t.Errorf("ResolveBackend(%q,%q) = %q, %v; want %q", tc.env, tc.bundle, got, err, tc.want) + } + } +} + +func TestK8sPodSpecHardening(t *testing.T) { + cfg := testContainerConfig(t) + cfg.NoNetwork = true + kcfg := KubernetesConfig{ + Namespace: "fleet-sandboxes", + WorkspaceClaim: "fleet-workspace", + ServiceAccount: "fleet-sandbox", + ImagePullSecret: "regcred", + } + pod, err := buildSandboxPod(applyContainerDefaults(cfg), kcfg, "fleet-sandbox-abc") + if err != nil { + t.Fatalf("buildSandboxPod: %v", err) + } + + // These pins are the k8s counterpart of podman_args_test.go: a change that + // weakens the pod hardening must show up as a failing assertion, not slip + // through a spec refactor. + spec := pod.Spec + if spec.AutomountServiceAccountToken == nil || *spec.AutomountServiceAccountToken { + t.Error("automountServiceAccountToken must be explicitly false — a sandbox must never hold apiserver credentials") + } + if spec.SecurityContext == nil || spec.SecurityContext.RunAsNonRoot == nil || !*spec.SecurityContext.RunAsNonRoot { + t.Error("runAsNonRoot must be true") + } + if spec.SecurityContext.SeccompProfile == nil || spec.SecurityContext.SeccompProfile.Type != "RuntimeDefault" { + t.Error("seccompProfile must default to RuntimeDefault") + } + c := spec.Containers[0] + if c.SecurityContext == nil || c.SecurityContext.ReadOnlyRootFilesystem == nil || !*c.SecurityContext.ReadOnlyRootFilesystem { + t.Error("readOnlyRootFilesystem must be true") + } + if c.SecurityContext.AllowPrivilegeEscalation == nil || *c.SecurityContext.AllowPrivilegeEscalation { + t.Error("allowPrivilegeEscalation must be false") + } + if c.SecurityContext.Capabilities == nil || len(c.SecurityContext.Capabilities.Drop) != 1 || c.SecurityContext.Capabilities.Drop[0] != "ALL" { + t.Error("capabilities must drop ALL") + } + if c.WorkingDir != cfg.WorkspaceHostDir { + t.Errorf("workingDir = %q, want the workspace root %q", c.WorkingDir, cfg.WorkspaceHostDir) + } + // Same-path workspace mount — the invariant that keeps MCP paths valid. + foundWorkspace := false + for _, m := range c.VolumeMounts { + if m.Name == "workspace" { + foundWorkspace = true + if m.MountPath != cfg.WorkspaceHostDir { + t.Errorf("workspace mounted at %q, want same-path %q", m.MountPath, cfg.WorkspaceHostDir) + } + } + } + if !foundWorkspace { + t.Error("workspace volume mount missing") + } + // Limit conversions: 512m podman → bytes; 1.0 cpus → 1000m; disk 5 → 5Gi. + if got := c.Resources.Limits["memory"]; got != "536870912" { + t.Errorf("memory limit = %q, want 536870912 (512 MiB in bytes)", got) + } + if got := c.Resources.Limits["cpu"]; got != "1000m" { + t.Errorf("cpu limit = %q, want 1000m", got) + } + if got := c.Resources.Limits["ephemeral-storage"]; got != "5Gi" { + t.Errorf("ephemeral-storage limit = %q, want 5Gi", got) + } + // Sealed posture label + ownership labels. + if got := pod.Metadata.Labels[k8sLabelEgress]; got != "none" { + t.Errorf("egress label = %q, want none for NoNetwork", got) + } + if got := pod.Metadata.Labels[k8sLabelInstance]; got != k8sInstanceLabel { + t.Errorf("instance label = %q, want %q", got, k8sInstanceLabel) + } + if len(spec.ImagePullSecrets) != 1 || spec.ImagePullSecrets[0].Name != "regcred" { + t.Error("imagePullSecrets not carried") + } + if spec.ServiceAccountName != "fleet-sandbox" { + t.Error("serviceAccountName not carried") + } +} + +func TestK8sPodSpecVariants(t *testing.T) { + cfg := applyContainerDefaults(testContainerConfig(t)) + kcfg := KubernetesConfig{Namespace: "ns", WorkspaceClaim: "ws", RuntimeClassName: "kata", SeccompLocalhostProfile: "profiles/fleet.json"} + pod, err := buildSandboxPod(cfg, kcfg, "fleet-sandbox-x") + if err != nil { + t.Fatalf("buildSandboxPod: %v", err) + } + if got := pod.Metadata.Labels[k8sLabelEgress]; got != "open" { + t.Errorf("egress label = %q, want open without NoNetwork", got) + } + if pod.Spec.RuntimeClassName == nil || *pod.Spec.RuntimeClassName != "kata" { + t.Error("runtimeClassName not carried") + } + sp := pod.Spec.SecurityContext.SeccompProfile + if sp.Type != "Localhost" || sp.LocalhostProfile == nil || *sp.LocalhostProfile != "profiles/fleet.json" { + t.Errorf("seccomp profile = %+v, want Localhost profiles/fleet.json", sp) + } + + // A negative disk limit disables the ephemeral-storage cap. + cfg.DiskLimitGB = -1 + pod, err = buildSandboxPod(cfg, kcfg, "fleet-sandbox-y") + if err != nil { + t.Fatalf("buildSandboxPod: %v", err) + } + if _, ok := pod.Spec.Containers[0].Resources.Limits["ephemeral-storage"]; ok { + t.Error("negative DiskLimitGB must not emit an ephemeral-storage limit") + } +} + +func TestK8sSchedulingKnobs(t *testing.T) { + // Parse helpers fail closed on malformed input. + if _, err := ParseK8sNodeSelector("pool"); err == nil { + t.Error("bare key without =value must error") + } + if _, err := ParseK8sNodeSelector("=v"); err == nil { + t.Error("empty key must error") + } + sel, err := ParseK8sNodeSelector(" pool=sandboxes, arch=amd64 ") + if err != nil || sel["pool"] != "sandboxes" || sel["arch"] != "amd64" { + t.Errorf("ParseK8sNodeSelector = %v, %v", sel, err) + } + if _, err := ParseK8sTolerations(`[{"unknown":"field"}]`); err == nil { + t.Error("unknown toleration field must error (strict decoding)") + } + tols, err := ParseK8sTolerations(`[{"key":"fleet.elcanotek.com/sandbox","operator":"Exists","effect":"NoSchedule"}]`) + if err != nil || len(tols) != 1 || tols[0].Key != "fleet.elcanotek.com/sandbox" { + t.Errorf("ParseK8sTolerations = %+v, %v", tols, err) + } + + // They reach the pod spec, and the pull policy is explicit (the API + // default for a :latest tag is Always, which breaks side-loaded images). + cfg := applyContainerDefaults(testContainerConfig(t)) + pod, err := buildSandboxPod(cfg, KubernetesConfig{ + Namespace: "ns", WorkspaceClaim: "ws", + NodeSelector: sel, Tolerations: tols, + }, "fleet-sandbox-sched") + if err != nil { + t.Fatalf("buildSandboxPod: %v", err) + } + if pod.Spec.NodeSelector["pool"] != "sandboxes" { + t.Errorf("nodeSelector not carried: %v", pod.Spec.NodeSelector) + } + if len(pod.Spec.Tolerations) != 1 || pod.Spec.Tolerations[0].Effect != "NoSchedule" { + t.Errorf("tolerations not carried: %+v", pod.Spec.Tolerations) + } + if got := pod.Spec.Containers[0].ImagePullPolicy; got != "IfNotPresent" { + t.Errorf("imagePullPolicy = %q, want explicit IfNotPresent", got) + } +} + +func TestK8sSanitizeClusterText(t *testing.T) { + // Cluster/pod-derived text is newline-stripped before it can enter a + // logged error — a pod printing "\nFAKE LOG LINE" to stderr must not be + // able to forge journal entries (go/log-injection). + got := sanitizeClusterText("line one\r\nFAKE LOG LINE\ntail") + if strings.ContainsAny(got, "\r\n") { + t.Errorf("sanitizeClusterText left line breaks in %q", got) + } + if got != "line one FAKE LOG LINE tail" { + t.Errorf("sanitizeClusterText = %q", got) + } +} + +func TestK8sQuantityConversions(t *testing.T) { + if _, err := k8sQuantityFromPodmanMemory("512x"); err == nil { + t.Error("bad memory suffix must error") + } + if got, _ := k8sQuantityFromPodmanMemory("2g"); got != "2147483648" { + t.Errorf("2g = %q", got) + } + if got, _ := k8sQuantityFromPodmanCPU("2.50"); got != "2500m" { + t.Errorf("2.50 cpus = %q", got) + } + if _, err := k8sQuantityFromPodmanCPU("zero"); err == nil { + t.Error("bad cpu must error") + } +} + +func TestK8sInstanceLabelRoundTrip(t *testing.T) { + pid, start, ok := parseK8sInstanceLabel(k8sInstanceLabel) + if !ok || pid != os.Getpid() || start <= 0 { + t.Fatalf("parseK8sInstanceLabel(%q) = %d, %d, %v", k8sInstanceLabel, pid, start, ok) + } + if _, _, ok := parseK8sInstanceLabel("garbage"); ok { + t.Error("garbage label must not parse") + } +} + +func TestK8sBashExec(t *testing.T) { + fake := newFakeKube(t) + fake.bashBehaviors["echo hi"] = func(_ string, stdout, _ io.Writer, _ *websocket.Conn) int { + _, _ = stdout.Write([]byte("hi\n")) + return 0 + } + fake.bashBehaviors["exit 3"] = func(_ string, _, stderr io.Writer, _ *websocket.Conn) int { + _, _ = stderr.Write([]byte("boom")) + return 3 + } + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + sb, err := backend.newSandbox(context.Background(), testContainerConfig(t)) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + if got := sb.ModeName(); got != "kubernetes" { + t.Errorf("ModeName = %q", got) + } + + res, err := sb.RunBash(context.Background(), BashRequest{Command: "echo hi"}) + if err != nil { + t.Fatalf("RunBash: %v", err) + } + if res.ExitCode != 0 || string(res.Stdout) != "hi\n" { + t.Errorf("RunBash = exit %d stdout %q", res.ExitCode, res.Stdout) + } + + res, err = sb.RunBash(context.Background(), BashRequest{Command: "exit 3", WorkingDir: "/some/dir"}) + if err != nil { + t.Fatalf("RunBash exit 3: %v", err) + } + if res.ExitCode != 3 || string(res.Stderr) != "boom" { + t.Errorf("RunBash exit3 = exit %d stderr %q", res.ExitCode, res.Stderr) + } + fake.mu.Lock() + workdir := fake.lastBashWorkdir + fake.mu.Unlock() + if workdir != "/some/dir" { + t.Errorf("workdir wrapper carried %q, want /some/dir", workdir) + } +} + +func TestK8sBashCancelPoisonsAndDeletesPod(t *testing.T) { + fake := newFakeKube(t) + started := make(chan struct{}, 1) + fake.bashBehaviors["block"] = func(_ string, _, _ io.Writer, conn *websocket.Conn) int { + started <- struct{}{} + // Simulate a process that never exits: hold until the client closes. + for { + if _, _, err := conn.ReadMessage(); err != nil { + return 0 + } + } + } + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + sb, err := backend.newSandbox(context.Background(), testContainerConfig(t)) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + + ctx, cancel := context.WithCancel(context.Background()) + go func() { + <-started + cancel() + }() + res, err := sb.RunBash(ctx, BashRequest{Command: "block", Timeout: time.Minute}) + if err != nil { + t.Fatalf("RunBash: %v", err) + } + if !res.Cancelled || res.TimedOut { + t.Errorf("want Cancelled, got %+v", res) + } + if !res.SandboxRetired || !res.CleanupConfirmed { + t.Errorf("want SandboxRetired+CleanupConfirmed, got %+v", res) + } + if !sb.Poisoned() { + t.Error("sandbox must be poisoned after a cancelled bash call") + } + fake.mu.Lock() + deleted := len(fake.deleted) + fake.mu.Unlock() + if deleted == 0 { + t.Error("cancelled bash must delete the pod (the #796 containment)") + } + if _, err := sb.RunBash(context.Background(), BashRequest{Command: "echo hi"}); !errors.Is(err, ErrPoisoned) { + t.Errorf("post-poison RunBash err = %v, want ErrPoisoned", err) + } +} + +func TestK8sRunPythonBridge(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + sb, err := backend.newSandbox(context.Background(), testContainerConfig(t)) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + + res, err := sb.RunPython(context.Background(), PythonRequest{Code: "1+1"}) + if err != nil { + t.Fatalf("RunPython: %v", err) + } + if res.Status != "ok" || res.Result != "ran: 1+1" { + t.Errorf("RunPython = %+v", res) + } + // Second call reuses the session. + res, err = sb.RunPython(context.Background(), PythonRequest{Code: "2+2"}) + if err != nil { + t.Fatalf("RunPython second: %v", err) + } + if res.Result != "ran: 2+2" { + t.Errorf("RunPython second = %+v", res) + } +} + +func TestK8sFileOpsThroughRealExecutor(t *testing.T) { + if !pythonAvailable() { + t.Skip("python3 not available on the test host") + } + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + cfg := testContainerConfig(t) + cfg.BridgeScript = []byte("# unused\n") + sb, err := backend.newSandbox(context.Background(), cfg) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + + // The fake runs the UPLOADED fileops.py on the host, so the workspace dir + // is a real host directory here. + root := filepath.Join(cfg.WorkspaceHostDir, "conv1") + if err := os.MkdirAll(root, 0o755); err != nil { + t.Fatal(err) + } + if err := sb.BindFileOpRoot(context.Background(), root); err != nil { + t.Fatalf("BindFileOpRoot: %v", err) + } + target := filepath.Join(root, "hello.txt") + if _, err := sb.RunFileOp(context.Background(), FileOpRequest{Op: FileOpWrite, Path: target, Root: root, Data: []byte("hello k8s\n")}); err != nil { + t.Fatalf("write: %v", err) + } + res, err := sb.RunFileOp(context.Background(), FileOpRequest{Op: FileOpRead, Path: target, Root: root, Limit: 1024}) + if err != nil { + t.Fatalf("read: %v", err) + } + if string(res.Data) != "hello k8s\n" { + t.Errorf("read back %q", res.Data) + } + // A root outside every mount is refused before any exec happens. + if _, err := sb.RunFileOp(context.Background(), FileOpRequest{Op: FileOpRead, Path: "/etc/passwd", Root: "/etc"}); !errors.Is(err, ErrFileOpUnsafePath) { + t.Errorf("outside-mount fileop err = %v, want ErrFileOpUnsafePath", err) + } +} + +func TestK8sPoolRouting(t *testing.T) { + fake := newFakeKube(t) + fake.bashBehaviors["echo pool"] = func(_ string, stdout, _ io.Writer, _ *websocket.Conn) int { + _, _ = stdout.Write([]byte("pool\n")) + return 0 + } + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + pool := NewPool(PoolConfig{ + Mode: ModeKubernetes, + KubernetesBackend: backend, + BridgeScript: []byte("# bridge\n"), + Container: testContainerConfig(t), + }) + defer pool.Close() + + sb, cleanup, err := pool.Take(context.Background()) + if err != nil { + t.Fatalf("Take: %v", err) + } + res, err := sb.RunBash(context.Background(), BashRequest{Command: "echo pool"}) + if err != nil || res.ExitCode != 0 { + t.Fatalf("RunBash via pool: %v, %+v", err, res) + } + cleanup() + + // Lockdown take: sealed pods come from the same backend. + sb, cleanup, err = pool.TakeContainer(context.Background()) + if err != nil { + t.Fatalf("TakeContainer: %v", err) + } + _ = sb + cleanup() + + // Allowlisted egress is refused, fail-closed. + if _, _, err := pool.TakeContainerWithEgress(context.Background(), ResourceOverride{}, []string{"example.com"}); err == nil { + t.Error("TakeContainerWithEgress must fail closed under the kubernetes backend") + } +} + +func TestK8sResourceOverridesReachPodSpec(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + pool := NewPool(PoolConfig{ + Mode: ModeKubernetes, + KubernetesBackend: backend, + BridgeScript: []byte("# bridge\n"), + Container: testContainerConfig(t), + }) + defer pool.Close() + + sb, cleanup, err := pool.TakeContainerWithOverrides(context.Background(), ResourceOverride{MemoryLimit: "1024m", CPULimit: "2.00"}, true) + if err != nil { + t.Fatalf("TakeContainerWithOverrides: %v", err) + } + defer cleanup() + _ = sb + + fake.mu.Lock() + defer fake.mu.Unlock() + if len(fake.pods) != 1 { + t.Fatalf("want 1 pod, have %d", len(fake.pods)) + } + for _, pod := range fake.pods { + limits := pod.Spec.Containers[0].Resources.Limits + if limits["memory"] != "1073741824" { + t.Errorf("override memory = %q, want 1073741824", limits["memory"]) + } + if limits["cpu"] != "2000m" { + t.Errorf("override cpu = %q, want 2000m", limits["cpu"]) + } + if pod.Metadata.Labels[k8sLabelEgress] != "none" { + t.Errorf("sealed take must label egress=none, got %q", pod.Metadata.Labels[k8sLabelEgress]) + } + } +} + +func TestK8sCloseDeletesPod(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + sb, err := backend.newSandbox(context.Background(), testContainerConfig(t)) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + sb.Close() + fake.mu.Lock() + defer fake.mu.Unlock() + if len(fake.pods) != 0 || len(fake.deleted) != 1 { + t.Errorf("Close must delete the pod: %d live, %d deleted", len(fake.pods), len(fake.deleted)) + } +} + +func TestK8sPruneOrphanedPods(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + + addPod := func(name, instance string) { + fake.mu.Lock() + fake.pods[name] = &k8sPod{Metadata: k8sObjectMeta{ + Name: name, + Labels: map[string]string{ + k8sLabelName: k8sLabelNameValue, + k8sLabelManagedBy: k8sLabelManagedVal, + k8sLabelInstance: instance, + }, + }} + fake.mu.Unlock() + } + addPod("fleet-sandbox-own", k8sInstanceLabel) // this process — never touched + addPod("fleet-sandbox-dead", "p999999999-t12345") // dead owner — pruned + // Unlabeled/unparseable ownership: fails "still running" (unparseable pid), + // so it is pruned — in k8s a pod without a live owner in THIS process tree + // is a leftover by construction (single-replica invariant). + addPod("fleet-sandbox-mystery", "not-a-label") + + n, err := backend.PruneOrphanedPods(context.Background()) + if err != nil { + t.Fatalf("PruneOrphanedPods: %v", err) + } + if n != 2 { + t.Errorf("pruned %d, want 2", n) + } + fake.mu.Lock() + defer fake.mu.Unlock() + if _, ok := fake.pods["fleet-sandbox-own"]; !ok { + t.Error("own pod must never be pruned") + } + if _, ok := fake.pods["fleet-sandbox-dead"]; ok { + t.Error("dead-owner pod must be pruned") + } +} + +func TestK8sProxyURLRefused(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + cfg := testContainerConfig(t) + cfg.ProxyURL = "http://127.0.0.1:9999" + if _, err := backend.newSandbox(context.Background(), cfg); err == nil { + t.Error("a ProxyURL (allowlisted egress) must be refused by the kubernetes backend") + } +} + +func TestK8sBridgeUploadVerified(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + cfg := testContainerConfig(t) + sb, err := backend.newSandbox(context.Background(), cfg) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + fake.mu.Lock() + defer fake.mu.Unlock() + var podName string + for name := range fake.pods { + podName = name + } + if got := fake.files[podName+":"+k8sBridgePath]; string(got) != string(cfg.BridgeScript) { + t.Errorf("bridge upload = %q, want %q", got, cfg.BridgeScript) + } + if got := fake.files[podName+":"+k8sFileOpsPath]; len(got) == 0 || !strings.Contains(string(got), "fileops.py") { + t.Errorf("fileops upload missing or wrong (%d bytes)", len(got)) + } +} diff --git a/internal/sandbox/k8s_client.go b/internal/sandbox/k8s_client.go new file mode 100644 index 00000000..ed0d76d0 --- /dev/null +++ b/internal/sandbox/k8s_client.go @@ -0,0 +1,493 @@ +// Copyright (c) 2026 ElcanoTek +// SPDX-License-Identifier: MIT + +package sandbox + +// k8s_client.go is the minimal Kubernetes API client the kubernetes sandbox +// backend (#989) uses to create, inspect, delete, and exec into sandbox Pods. +// +// It is deliberately NOT client-go. The backend needs exactly five verbs — +// create/get/delete/list on pods, plus the exec subresource — and client-go +// would add several dozen modules to a dependency tree that is gated by +// govulncheck and a container CVE scan. Everything here is plain net/http +// against the well-versioned core/v1 REST surface, plus gorilla/websocket +// (already in the tree) for exec streaming. The trade is accepted and +// recorded in the ADR: if the backend ever needs watches, informers, or +// exotic auth, revisit client-go rather than growing this file into one. +// +// Auth is loaded from the standard in-cluster mount +// (/var/run/secrets/kubernetes.io/serviceaccount) when no kubeconfig is +// configured, else from a kubeconfig file supporting token and client-cert +// credentials. exec-plugin and auth-provider kubeconfigs are refused with an +// actionable error — the control plane runs unattended, so credentials that +// shell out to an interactive helper cannot work anyway. Fail closed, never +// guess. + +import ( + "bytes" + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "net/http" + "net/url" + "os" + "path" + "strings" + "sync" + "time" +) + +// inClusterTokenFile / inClusterCAFile / inClusterNamespaceFile are the +// standard projected service-account mount every Pod gets. The token file is +// re-read per request (see bearerToken) because bound tokens rotate — a +// long-lived fleet process holding the boot-time token would start getting +// 401s about an hour in. +const ( + inClusterTokenFile = "/var/run/secrets/kubernetes.io/serviceaccount/token" //nolint:gosec // well-known mount path, not a credential + inClusterCAFile = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" + inClusterNamespaceFile = "/var/run/secrets/kubernetes.io/serviceaccount/namespace" +) + +// k8sRequestTimeout bounds any single non-streaming API request. Generous for +// a loaded apiserver; short enough that a dead one fails a turn promptly +// instead of hanging it. +const k8sRequestTimeout = 30 * time.Second + +// k8sClient is the minimal REST client. Safe for concurrent use. +type k8sClient struct { + baseURL *url.URL + httpc *http.Client + // tlsConfig is retained for the websocket dialer (exec), which cannot + // share http.Transport. + tlsConfig *tls.Config + + // Exactly one of the following is set. staticToken is a kubeconfig token; + // tokenFile is re-read per request so rotated bound tokens keep working. + // Client-cert auth lives inside tlsConfig and needs neither. + staticToken string + tokenFile string + + tokenMu sync.Mutex + cachedToken string + tokenRead time.Time +} + +// tokenRefreshInterval is how long a token-file read is trusted before the +// file is consulted again. Kubernetes rotates bound tokens well before their +// ~1h expiry, so a 1-minute cache never serves a stale token while keeping +// the common path free of file I/O. +const tokenRefreshInterval = time.Minute + +// bearerToken returns the Authorization bearer value for a request, or "" +// when client-cert auth is in use. +func (c *k8sClient) bearerToken() (string, error) { + if c.staticToken != "" { + return c.staticToken, nil + } + if c.tokenFile == "" { + return "", nil + } + c.tokenMu.Lock() + defer c.tokenMu.Unlock() + if c.cachedToken != "" && time.Since(c.tokenRead) < tokenRefreshInterval { + return c.cachedToken, nil + } + raw, err := os.ReadFile(c.tokenFile) + if err != nil { + return "", fmt.Errorf("read service-account token: %w", err) + } + c.cachedToken = strings.TrimSpace(string(raw)) + c.tokenRead = time.Now() + return c.cachedToken, nil +} + +// newInClusterClient builds a client from the standard service-account mount. +func newInClusterClient() (*k8sClient, error) { + host, port := os.Getenv("KUBERNETES_SERVICE_HOST"), os.Getenv("KUBERNETES_SERVICE_PORT") + if host == "" || port == "" { + return nil, fmt.Errorf("not running in a cluster (KUBERNETES_SERVICE_HOST/PORT unset) and no kubeconfig configured") + } + caPEM, err := os.ReadFile(inClusterCAFile) + if err != nil { + return nil, fmt.Errorf("read in-cluster CA: %w", err) + } + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(caPEM) { + return nil, fmt.Errorf("in-cluster CA at %s contains no usable certificates", inClusterCAFile) + } + if _, err := os.Stat(inClusterTokenFile); err != nil { + return nil, fmt.Errorf("in-cluster service-account token: %w", err) + } + base, err := url.Parse("https://" + net.JoinHostPort(host, port)) + if err != nil { + return nil, fmt.Errorf("parse in-cluster apiserver address: %w", err) + } + tlsCfg := &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12} + return &k8sClient{ + baseURL: base, + tlsConfig: tlsCfg, + tokenFile: inClusterTokenFile, + httpc: &http.Client{ + Transport: &http.Transport{TLSClientConfig: tlsCfg}, + }, + }, nil +} + +// inClusterNamespace reads the namespace the control plane itself runs in, +// used only as a default when the sandbox namespace is not configured. +func inClusterNamespace() string { + raw, err := os.ReadFile(inClusterNamespaceFile) + if err != nil { + return "" + } + return strings.TrimSpace(string(raw)) +} + +// sanitizeClusterText strips newlines from text that originated in the +// cluster API or a pod (status messages, exec status, stderr snippets) before +// it is embedded in an error or log line. Everything remote-derived leaves +// this package through error strings that end up in log.Printf sites all over +// the codebase, and a forged newline in that text is a log-injection vector +// (CodeQL go/log-injection): without this, a pod that prints a crafted line +// to stderr could fabricate whole log entries in the operator's journal. +func sanitizeClusterText(s string) string { + s = strings.ReplaceAll(s, "\r", " ") + s = strings.ReplaceAll(s, "\n", " ") + return s +} + +// k8sStatusError is a non-2xx API response, carrying enough of the +// metav1.Status body to be actionable in logs and boot errors. +type k8sStatusError struct { + Code int + Reason string + Message string +} + +func (e *k8sStatusError) Error() string { + if e.Message != "" { + return fmt.Sprintf("kubernetes API error %d (%s): %s", e.Code, e.Reason, e.Message) + } + return fmt.Sprintf("kubernetes API error %d (%s)", e.Code, e.Reason) +} + +// isK8sNotFound reports whether err is a 404 from the API — the pod-already- +// gone case teardown paths treat as success, mirroring containerAlreadyGone. +func isK8sNotFound(err error) bool { + var se *k8sStatusError + return errors.As(err, &se) && se.Code == http.StatusNotFound +} + +// do performs one JSON API request. body may be nil. A non-2xx response is +// returned as *k8sStatusError with the server's status message decoded. +func (c *k8sClient) do(ctx context.Context, method, apiPath string, query url.Values, body []byte) ([]byte, error) { + reqCtx, cancel := context.WithTimeout(ctx, k8sRequestTimeout) + defer cancel() + + u := *c.baseURL + u.Path = path.Join(u.Path, apiPath) + if query != nil { + u.RawQuery = query.Encode() + } + var rdr io.Reader + if body != nil { + rdr = bytes.NewReader(body) + } + req, err := http.NewRequestWithContext(reqCtx, method, u.String(), rdr) + if err != nil { + return nil, fmt.Errorf("build %s %s: %w", method, apiPath, err) + } + if body != nil { + req.Header.Set("Content-Type", "application/json") + } + req.Header.Set("Accept", "application/json") + token, err := c.bearerToken() + if err != nil { + return nil, err + } + if token != "" { + req.Header.Set("Authorization", "Bearer "+token) + } + resp, err := c.httpc.Do(req) + if err != nil { + return nil, fmt.Errorf("%s %s: %w", method, apiPath, err) + } + defer func() { _ = resp.Body.Close() }() + // Responses are bounded reads: pod objects are a few KB; even a large list + // stays far under this. Guards against a misbehaving endpoint, not real use. + data, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20)) + if err != nil { + return nil, fmt.Errorf("%s %s: read response: %w", method, apiPath, err) + } + if resp.StatusCode < 200 || resp.StatusCode > 299 { + var status struct { + Reason string `json:"reason"` + Message string `json:"message"` + } + _ = json.Unmarshal(data, &status) + // Sanitized at construction so every path that logs this error — + // boot preflights, pool fill, the prune sweep — is covered at once. + return nil, &k8sStatusError{ + Code: resp.StatusCode, + Reason: sanitizeClusterText(status.Reason), + Message: sanitizeClusterText(status.Message), + } + } + return data, nil +} + +// ── typed pod surface (the narrow slice of core/v1 the backend touches) ── + +type k8sPod struct { + Metadata k8sObjectMeta `json:"metadata"` + Spec k8sPodSpec `json:"spec"` + Status k8sPodStatus `json:"status,omitempty"` + + // APIVersion/Kind are emitted on create; ignored on read. + APIVersion string `json:"apiVersion,omitempty"` + Kind string `json:"kind,omitempty"` +} + +type k8sObjectMeta struct { + Name string `json:"name"` + Namespace string `json:"namespace,omitempty"` + Labels map[string]string `json:"labels,omitempty"` +} + +type k8sPodSpec struct { + RestartPolicy string `json:"restartPolicy,omitempty"` + AutomountServiceAccountToken *bool `json:"automountServiceAccountToken,omitempty"` + EnableServiceLinks *bool `json:"enableServiceLinks,omitempty"` + TerminationGracePeriodSeconds *int64 `json:"terminationGracePeriodSeconds,omitempty"` + ServiceAccountName string `json:"serviceAccountName,omitempty"` + RuntimeClassName *string `json:"runtimeClassName,omitempty"` + ImagePullSecrets []k8sLocalObjRef `json:"imagePullSecrets,omitempty"` + SecurityContext *k8sPodSecurityCtx `json:"securityContext,omitempty"` + Containers []k8sContainer `json:"containers"` + Volumes []k8sVolume `json:"volumes,omitempty"` + NodeSelector map[string]string `json:"nodeSelector,omitempty"` + Tolerations []k8sToleration `json:"tolerations,omitempty"` +} + +type k8sLocalObjRef struct { + Name string `json:"name"` +} + +type k8sToleration struct { + Key string `json:"key,omitempty"` + Operator string `json:"operator,omitempty"` + Value string `json:"value,omitempty"` + Effect string `json:"effect,omitempty"` +} + +type k8sPodSecurityCtx struct { + RunAsNonRoot *bool `json:"runAsNonRoot,omitempty"` + RunAsUser *int64 `json:"runAsUser,omitempty"` + RunAsGroup *int64 `json:"runAsGroup,omitempty"` + FSGroup *int64 `json:"fsGroup,omitempty"` + SeccompProfile *k8sSeccompProfile `json:"seccompProfile,omitempty"` +} + +type k8sSeccompProfile struct { + Type string `json:"type"` + LocalhostProfile *string `json:"localhostProfile,omitempty"` +} + +type k8sContainer struct { + Name string `json:"name"` + Image string `json:"image"` + ImagePullPolicy string `json:"imagePullPolicy,omitempty"` + Command []string `json:"command,omitempty"` + WorkingDir string `json:"workingDir,omitempty"` + SecurityContext *k8sContainerSecCtx `json:"securityContext,omitempty"` + Resources *k8sResources `json:"resources,omitempty"` + VolumeMounts []k8sVolumeMount `json:"volumeMounts,omitempty"` +} + +type k8sContainerSecCtx struct { + AllowPrivilegeEscalation *bool `json:"allowPrivilegeEscalation,omitempty"` + ReadOnlyRootFilesystem *bool `json:"readOnlyRootFilesystem,omitempty"` + Capabilities *k8sCapabilities `json:"capabilities,omitempty"` +} + +type k8sCapabilities struct { + Drop []string `json:"drop,omitempty"` +} + +type k8sResources struct { + Limits map[string]string `json:"limits,omitempty"` + Requests map[string]string `json:"requests,omitempty"` +} + +type k8sVolumeMount struct { + Name string `json:"name"` + MountPath string `json:"mountPath"` + ReadOnly bool `json:"readOnly,omitempty"` + SubPath string `json:"subPath,omitempty"` +} + +type k8sVolume struct { + Name string `json:"name"` + EmptyDir *k8sEmptyDir `json:"emptyDir,omitempty"` + PersistentVolumeClaim *k8sPVCVolSource `json:"persistentVolumeClaim,omitempty"` +} + +type k8sEmptyDir struct { + SizeLimit string `json:"sizeLimit,omitempty"` + Medium string `json:"medium,omitempty"` +} + +type k8sPVCVolSource struct { + ClaimName string `json:"claimName"` + ReadOnly bool `json:"readOnly,omitempty"` +} + +type k8sPodStatus struct { + Phase string `json:"phase,omitempty"` + Reason string `json:"reason,omitempty"` + Message string `json:"message,omitempty"` + ContainerStatuses []k8sContainerStatus `json:"containerStatuses,omitempty"` +} + +type k8sContainerStatus struct { + Name string `json:"name"` + Ready bool `json:"ready"` + State struct { + Waiting *struct { + Reason string `json:"reason,omitempty"` + Message string `json:"message,omitempty"` + } `json:"waiting,omitempty"` + } `json:"state,omitempty"` +} + +func (c *k8sClient) createPod(ctx context.Context, namespace string, pod *k8sPod) error { + pod.APIVersion, pod.Kind = "v1", "Pod" + body, err := json.Marshal(pod) + if err != nil { + return fmt.Errorf("marshal pod: %w", err) + } + _, err = c.do(ctx, http.MethodPost, "/api/v1/namespaces/"+namespace+"/pods", nil, body) + return err +} + +func (c *k8sClient) getPod(ctx context.Context, namespace, name string) (*k8sPod, error) { + data, err := c.do(ctx, http.MethodGet, "/api/v1/namespaces/"+namespace+"/pods/"+name, nil, nil) + if err != nil { + return nil, err + } + var pod k8sPod + if err := json.Unmarshal(data, &pod); err != nil { + return nil, fmt.Errorf("decode pod: %w", err) + } + return &pod, nil +} + +// deletePod removes a pod immediately (gracePeriodSeconds=0). The sandbox +// image's PID 1 is `sleep`, which has no state worth a graceful drain, and +// the #796 poison path NEEDS the hard kill: a straggler must not get grace +// time to finish a side effect. +func (c *k8sClient) deletePod(ctx context.Context, namespace, name string) error { + body := []byte(`{"apiVersion":"v1","kind":"DeleteOptions","gracePeriodSeconds":0,"propagationPolicy":"Background"}`) + _, err := c.do(ctx, http.MethodDelete, "/api/v1/namespaces/"+namespace+"/pods/"+name, nil, body) + return err +} + +type k8sPodList struct { + Items []k8sPod `json:"items"` +} + +func (c *k8sClient) listPods(ctx context.Context, namespace, labelSelector string) (*k8sPodList, error) { + q := url.Values{} + if labelSelector != "" { + q.Set("labelSelector", labelSelector) + } + data, err := c.do(ctx, http.MethodGet, "/api/v1/namespaces/"+namespace+"/pods", q, nil) + if err != nil { + return nil, err + } + var list k8sPodList + if err := json.Unmarshal(data, &list); err != nil { + return nil, fmt.Errorf("decode pod list: %w", err) + } + return &list, nil +} + +// getNetworkPolicy fetches one networking.k8s.io/v1 NetworkPolicy, used only +// by the boot preflight to verify the sealed-egress policy object exists. +func (c *k8sClient) getNetworkPolicy(ctx context.Context, namespace, name string) error { + _, err := c.do(ctx, http.MethodGet, "/apis/networking.k8s.io/v1/namespaces/"+namespace+"/networkpolicies/"+name, nil, nil) + return err +} + +// getPVC fetches one PersistentVolumeClaim, used only by the boot preflight to +// verify the shared workspace claim exists before the first pod references it. +func (c *k8sClient) getPVC(ctx context.Context, namespace, name string) error { + _, err := c.do(ctx, http.MethodGet, "/api/v1/namespaces/"+namespace+"/persistentvolumeclaims/"+name, nil, nil) + return err +} + +// getRuntimeClass fetches one node.k8s.io/v1 RuntimeClass (cluster-scoped), +// used only by the boot preflight when a runtime class is configured. +func (c *k8sClient) getRuntimeClass(ctx context.Context, name string) error { + _, err := c.do(ctx, http.MethodGet, "/apis/node.k8s.io/v1/runtimeclasses/"+name, nil, nil) + return err +} + +// selfSubjectAccessReview asks the apiserver whether the client's identity can +// perform verb on resource (optionally subresource) in namespace. Used by the +// boot preflight so a missing RBAC grant fails at start with a precise message +// instead of at the first turn. +func (c *k8sClient) selfSubjectAccessReview(ctx context.Context, namespace, verb, resource, subresource string) (bool, error) { + review := map[string]any{ + "apiVersion": "authorization.k8s.io/v1", + "kind": "SelfSubjectAccessReview", + "spec": map[string]any{ + "resourceAttributes": map[string]any{ + "namespace": namespace, + "verb": verb, + "resource": resource, + "subresource": subresource, + "group": "", + }, + }, + } + body, err := json.Marshal(review) + if err != nil { + return false, fmt.Errorf("marshal access review: %w", err) + } + data, err := c.do(ctx, http.MethodPost, "/apis/authorization.k8s.io/v1/selfsubjectaccessreviews", nil, body) + if err != nil { + return false, err + } + var resp struct { + Status struct { + Allowed bool `json:"allowed"` + Reason string `json:"reason,omitempty"` + } `json:"status"` + } + if err := json.Unmarshal(data, &resp); err != nil { + return false, fmt.Errorf("decode access review: %w", err) + } + return resp.Status.Allowed, nil +} + +// serverVersion fetches /version — the cheapest authenticated "is the +// apiserver reachable and are my credentials valid" probe the preflight runs. +func (c *k8sClient) serverVersion(ctx context.Context) (string, error) { + data, err := c.do(ctx, http.MethodGet, "/version", nil, nil) + if err != nil { + return "", err + } + var v struct { + GitVersion string `json:"gitVersion"` + } + if err := json.Unmarshal(data, &v); err != nil { + return "", fmt.Errorf("decode /version: %w", err) + } + return sanitizeClusterText(v.GitVersion), nil +} diff --git a/internal/sandbox/k8s_exec.go b/internal/sandbox/k8s_exec.go new file mode 100644 index 00000000..1a0a72e2 --- /dev/null +++ b/internal/sandbox/k8s_exec.go @@ -0,0 +1,283 @@ +// Copyright (c) 2026 ElcanoTek +// SPDX-License-Identifier: MIT + +package sandbox + +// k8s_exec.go streams pod exec sessions for the kubernetes sandbox backend +// over the apiserver's WebSocket channel protocol (v4.channel.k8s.io): each +// binary frame's first byte names a channel — 0 stdin (client→server), +// 1 stdout, 2 stderr, 3 error/status (server→client) — and the server closes +// the connection when the exec'd process exits, after publishing a +// metav1.Status on channel 3 carrying the exit code. +// +// v4 is the oldest protocol every supported apiserver speaks; it cannot +// half-close stdin, so an exec whose process reads stdin TO EOF must bound +// the read itself — the backend wraps such commands in `head -c ` (see +// k8s_backend.go) rather than depending on the newer v5 close channel. + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "path" + "strconv" + "sync" + "time" + + "github.com/gorilla/websocket" +) + +const ( + k8sExecProtocolV4 = "v4.channel.k8s.io" + + k8sChannelStdin = 0 + k8sChannelStdout = 1 + k8sChannelStderr = 2 + k8sChannelError = 3 + + // k8sExecHandshakeTimeout bounds the WebSocket dial+upgrade. + k8sExecHandshakeTimeout = 15 * time.Second + + // k8sStdinChunk bounds a single stdin frame. Large writes (a bridge + // request embedding a big code cell, a fileop write payload) are split so + // no single frame approaches server message-size limits. + k8sStdinChunk = 512 * 1024 +) + +// k8sExecSession is one live exec connection. Writes go to the process's +// stdin; the background read loop demultiplexes stdout/stderr/status frames +// into the sinks given at dial time. done is closed when the read loop ends; +// result() then reports the exec's outcome. +type k8sExecSession struct { + conn *websocket.Conn + + writeMu sync.Mutex + + done chan struct{} + + mu sync.Mutex + exitCode int + execErr error +} + +// execPod dials the exec subresource for the named pod and starts the +// background demux loop. stdout/stderr sinks must be goroutine-safe or owned +// solely by the loop. withStdin controls whether the server keeps a stdin +// channel open (a stdin-less exec gives the process an immediately-EOF stdin, +// matching the podman backend's unset cmd.Stdin). +func (c *k8sClient) execPod(ctx context.Context, namespace, pod, container string, command []string, withStdin bool, stdout, stderr io.Writer) (*k8sExecSession, error) { + q := url.Values{} + q.Set("container", container) + q.Set("stdout", "true") + q.Set("stderr", "true") + q.Set("tty", "false") + q.Set("stdin", strconv.FormatBool(withStdin)) + for _, arg := range command { + q.Add("command", arg) + } + u := *c.baseURL + switch u.Scheme { + case "https": + u.Scheme = "wss" + case "http": + u.Scheme = "ws" + } + u.Path = path.Join(u.Path, "/api/v1/namespaces/"+namespace+"/pods/"+pod+"/exec") + u.RawQuery = q.Encode() + + header := http.Header{} + token, err := c.bearerToken() + if err != nil { + return nil, err + } + if token != "" { + header.Set("Authorization", "Bearer "+token) + } + dialer := &websocket.Dialer{ + TLSClientConfig: c.tlsConfig, + Subprotocols: []string{k8sExecProtocolV4}, + HandshakeTimeout: k8sExecHandshakeTimeout, + } + conn, resp, err := dialer.DialContext(ctx, u.String(), header) + if err != nil { + if resp != nil { + // The upgrade response body carries the apiserver's status message + // (RBAC denial, container not found) — surface it, bounded and + // newline-sanitized (cluster-derived text ends up in logs). + body, _ := io.ReadAll(io.LimitReader(resp.Body, 2048)) + _ = resp.Body.Close() + return nil, fmt.Errorf("pod exec dial: %w (HTTP %d: %.500s)", err, resp.StatusCode, sanitizeClusterText(string(body))) + } + return nil, fmt.Errorf("pod exec dial: %w", err) + } + + s := &k8sExecSession{conn: conn, done: make(chan struct{}), exitCode: -1} + go s.readLoop(stdout, stderr) + return s, nil +} + +// readLoop demultiplexes server frames until the connection ends, then +// records the outcome. A clean close after a Success status yields exit 0; a +// NonZeroExitCode status yields the process's code; a connection that ends +// without any status is an error (the process outcome is unknown). +func (s *k8sExecSession) readLoop(stdout, stderr io.Writer) { + defer close(s.done) + var status []byte + sawStatus := false + for { + msgType, data, err := s.conn.ReadMessage() + if err != nil { + code, execErr := parseExecStatus(status, sawStatus, err) + s.mu.Lock() + s.exitCode, s.execErr = code, execErr + s.mu.Unlock() + return + } + if msgType != websocket.BinaryMessage && msgType != websocket.TextMessage { + continue + } + if len(data) == 0 { + continue + } + payload := data[1:] + switch data[0] { + case k8sChannelStdout: + if stdout != nil && len(payload) > 0 { + _, _ = stdout.Write(payload) + } + case k8sChannelStderr: + if stderr != nil && len(payload) > 0 { + _, _ = stderr.Write(payload) + } + case k8sChannelError: + sawStatus = true + status = append(status, payload...) + } + } +} + +// parseExecStatus turns the channel-3 metav1.Status (if any) plus the read +// error that ended the loop into (exitCode, err). Only a normal-closure / +// EOF-family end with a parsed status is a trustworthy outcome. +func parseExecStatus(status []byte, sawStatus bool, readErr error) (int, error) { + if !sawStatus { + if websocket.IsCloseError(readErr, websocket.CloseNormalClosure) { + // Some proxies drop the status frame on a zero-exit process; treat a + // clean close without status as success — the failure directions + // (non-zero exit, kill, RBAC) all DO produce a status or an abnormal + // close, so this cannot mask them. + return 0, nil + } + // %s of the sanitized text, not %w: a close error's reason text is + // server-supplied (remote), and nothing upstream matches on the + // wrapped type — losing the chain costs nothing here. + return -1, fmt.Errorf("pod exec ended without a status frame: %s", sanitizeClusterText(readErr.Error())) + } + var st struct { + Status string `json:"status"` + Reason string `json:"reason"` + Message string `json:"message"` + Details struct { + Causes []struct { + Reason string `json:"reason"` + Message string `json:"message"` + } `json:"causes"` + } `json:"details"` + } + if err := json.Unmarshal(status, &st); err != nil { + return -1, fmt.Errorf("parse pod exec status: %w (raw: %.200s)", err, sanitizeClusterText(string(status))) + } + // Every message below is cluster-derived text that ends up in logged + // errors — sanitized like everything else that leaves this package. + switch { + case st.Status == "Success": + return 0, nil + case st.Reason == "NonZeroExitCode": + for _, cause := range st.Details.Causes { + if cause.Reason == "ExitCode" { + code, err := strconv.Atoi(cause.Message) + if err != nil { + return -1, fmt.Errorf("parse pod exec exit code %q: %w", sanitizeClusterText(cause.Message), err) + } + return code, nil + } + } + return -1, fmt.Errorf("pod exec reported NonZeroExitCode without an ExitCode cause: %s", sanitizeClusterText(st.Message)) + default: + // A failure that is not an exit code: the exec itself failed (command + // not found in a way the shell couldn't report, container gone, …). + return -1, fmt.Errorf("pod exec failed: %s", sanitizeClusterText(st.Message)) + } +} + +// writeStdin sends bytes to the process's stdin, chunked. Goroutine-safe. +func (s *k8sExecSession) writeStdin(p []byte) error { + s.writeMu.Lock() + defer s.writeMu.Unlock() + for len(p) > 0 { + n := len(p) + if n > k8sStdinChunk { + n = k8sStdinChunk + } + // No manual size arithmetic (`1+n` trips CodeQL's + // allocation-size-overflow check, exactly like podmanArgs' old + // `len(rest)+1`); append sizes the backing array itself. + frame := append([]byte{k8sChannelStdin}, p[:n]...) + if err := s.conn.WriteMessage(websocket.BinaryMessage, frame); err != nil { + return fmt.Errorf("write pod exec stdin: %w", err) + } + p = p[n:] + } + return nil +} + +// wait blocks until the exec ends or ctx is done. On ctx expiry the +// connection is torn down (which unblocks the read loop) and ctx's error is +// returned — the PROCESS inside the pod may still be running; the caller owns +// the #796 containment (delete the pod, poison the sandbox). +func (s *k8sExecSession) wait(ctx context.Context) (int, error) { + select { + case <-ctx.Done(): + _ = s.conn.Close() + <-s.done + return -1, ctx.Err() + case <-s.done: + s.mu.Lock() + defer s.mu.Unlock() + return s.exitCode, s.execErr + } +} + +// close tears the connection down and waits for the read loop to exit. +func (s *k8sExecSession) close() { + _ = s.conn.Close() + <-s.done +} + +// runOneShotExec execs command in the pod, optionally feeding stdin, and +// waits for it to finish. ctx bounds the whole call. The command must +// consume a bounded stdin (v4 cannot signal stdin EOF): callers wrap +// stdin-to-EOF readers in `head -c `. +func (c *k8sClient) runOneShotExec(ctx context.Context, namespace, pod, container string, command []string, stdin []byte, stdout, stderr io.Writer) (int, error) { + session, err := c.execPod(ctx, namespace, pod, container, command, len(stdin) > 0, stdout, stderr) + if err != nil { + return -1, err + } + defer session.close() + if len(stdin) > 0 { + if err := session.writeStdin(stdin); err != nil { + // The write can fail because the process already exited (its outcome + // frame may still be in flight) — fall through to wait, which reports + // the authoritative result; surface the write error only if the exec + // outcome is itself unusable. + if code, werr := session.wait(ctx); werr == nil { + return code, nil + } + return -1, err + } + } + return session.wait(ctx) +} diff --git a/internal/sandbox/k8s_fake_test.go b/internal/sandbox/k8s_fake_test.go new file mode 100644 index 00000000..0cde74f7 --- /dev/null +++ b/internal/sandbox/k8s_fake_test.go @@ -0,0 +1,481 @@ +package sandbox + +// k8s_fake_test.go is the fake Kubernetes apiserver the kubernetes-backend +// tests run against: enough of the core/v1 REST surface (pods CRUD, access +// reviews, the preflight objects) plus a v4.channel.k8s.io WebSocket exec +// endpoint whose "processes" are Go handlers. File uploads store bytes; the +// fileops exec pipes them through a REAL host python3 running the uploaded +// script, so the k8s transport is tested against the genuine executor. + +import ( + "bytes" + "encoding/base64" + "encoding/json" + "encoding/pem" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "regexp" + "strconv" + "strings" + "sync" + "testing" + "time" + + "github.com/gorilla/websocket" +) + +const fakeKubeToken = "test-token" + +type fakeKube struct { + t *testing.T + srv *httptest.Server + + mu sync.Mutex + pods map[string]*k8sPod // name → object (status stamped Running/ready) + files map[string][]byte // ":" → uploaded bytes + deleted []string + + // Failure injection for preflight tests. + denied map[string]bool // " [/]" → deny + noPVC bool + noNetpol bool + noRuntimeClass bool + + // bashBehaviors maps a bash command string to its fake process. The + // handler receives the parsed workdir ("" when the call carried none). + bashBehaviors map[string]func(workdir string, stdout, stderr io.Writer, conn *websocket.Conn) int + + // lastBashWorkdir records the workdir the most recent bash exec carried. + lastBashWorkdir string +} + +func newFakeKube(t *testing.T) *fakeKube { + t.Helper() + f := &fakeKube{ + t: t, + pods: make(map[string]*k8sPod), + files: make(map[string][]byte), + denied: make(map[string]bool), + bashBehaviors: make(map[string]func(string, io.Writer, io.Writer, *websocket.Conn) int), + } + f.srv = httptest.NewTLSServer(http.HandlerFunc(f.handle)) + t.Cleanup(f.srv.Close) + return f +} + +// kubeconfigPath writes a kubeconfig pointing at the fake server (token auth, +// CA pinned to the httptest certificate) and returns its path. +func (f *fakeKube) kubeconfigPath(t *testing.T) string { + t.Helper() + caPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: f.srv.Certificate().Raw}) + dir := t.TempDir() + path := filepath.Join(dir, "kubeconfig") + content := fmt.Sprintf(`apiVersion: v1 +kind: Config +current-context: fake +contexts: + - name: fake + context: + cluster: fake + user: fake +clusters: + - name: fake + cluster: + server: %s + certificate-authority-data: %s +users: + - name: fake + user: + token: %s +`, f.srv.URL, base64.StdEncoding.EncodeToString(caPEM), fakeKubeToken) + if err := os.WriteFile(path, []byte(content), 0o600); err != nil { + t.Fatalf("write kubeconfig: %v", err) + } + return path +} + +// backend builds a KubernetesBackend against the fake server. +func (f *fakeKube) backend(t *testing.T, cfg KubernetesConfig) *KubernetesBackend { + t.Helper() + cfg.KubeconfigPath = f.kubeconfigPath(t) + if cfg.WorkspaceClaim == "" { + cfg.WorkspaceClaim = "fleet-workspace" + } + b, err := NewKubernetesBackend(cfg) + if err != nil { + t.Fatalf("NewKubernetesBackend: %v", err) + } + return b +} + +func (f *fakeKube) authorized(r *http.Request) bool { + return r.Header.Get("Authorization") == "Bearer "+fakeKubeToken +} + +var ( + podPathRe = regexp.MustCompile(`^/api/v1/namespaces/([^/]+)/pods/([^/]+)$`) + podExecRe = regexp.MustCompile(`^/api/v1/namespaces/([^/]+)/pods/([^/]+)/exec$`) + podListRe = regexp.MustCompile(`^/api/v1/namespaces/([^/]+)/pods$`) + pvcPathRe = regexp.MustCompile(`^/api/v1/namespaces/([^/]+)/persistentvolumeclaims/([^/]+)$`) + netpolRe = regexp.MustCompile(`^/apis/networking\.k8s\.io/v1/namespaces/([^/]+)/networkpolicies/([^/]+)$`) + runtimeClsRe = regexp.MustCompile(`^/apis/node\.k8s\.io/v1/runtimeclasses/([^/]+)$`) +) + +func writeK8sStatus(w http.ResponseWriter, code int, reason, msg string) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(code) + _ = json.NewEncoder(w).Encode(map[string]any{ + "kind": "Status", "apiVersion": "v1", "reason": reason, "message": msg, "code": code, + }) +} + +func (f *fakeKube) handle(w http.ResponseWriter, r *http.Request) { + if !f.authorized(r) { + writeK8sStatus(w, http.StatusUnauthorized, "Unauthorized", "bad token") + return + } + path := r.URL.Path + switch { + case path == "/version": + _, _ = w.Write([]byte(`{"gitVersion":"v1.31.0-fake"}`)) + case path == "/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": + f.handleAccessReview(w, r) + case podExecRe.MatchString(path): + f.handleExec(w, r) + case podPathRe.MatchString(path): + f.handlePod(w, r) + case podListRe.MatchString(path): + if r.Method == http.MethodPost { + f.handleCreatePod(w, r) + return + } + f.handleListPods(w, r) + case pvcPathRe.MatchString(path): + if f.noPVC { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "pvc not found") + return + } + _, _ = w.Write([]byte(`{"kind":"PersistentVolumeClaim"}`)) + case netpolRe.MatchString(path): + if f.noNetpol { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "networkpolicy not found") + return + } + _, _ = w.Write([]byte(`{"kind":"NetworkPolicy"}`)) + case runtimeClsRe.MatchString(path): + if f.noRuntimeClass { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "runtimeclass not found") + return + } + _, _ = w.Write([]byte(`{"kind":"RuntimeClass"}`)) + default: + writeK8sStatus(w, http.StatusNotFound, "NotFound", "no fake route for "+path) + } +} + +func (f *fakeKube) handleAccessReview(w http.ResponseWriter, r *http.Request) { + var review struct { + Spec struct { + ResourceAttributes struct { + Verb string `json:"verb"` + Resource string `json:"resource"` + Subresource string `json:"subresource"` + } `json:"resourceAttributes"` + } `json:"spec"` + } + _ = json.NewDecoder(r.Body).Decode(&review) + key := review.Spec.ResourceAttributes.Verb + " " + review.Spec.ResourceAttributes.Resource + if review.Spec.ResourceAttributes.Subresource != "" { + key += "/" + review.Spec.ResourceAttributes.Subresource + } + f.mu.Lock() + allowed := !f.denied[key] + f.mu.Unlock() + _ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"allowed": allowed}}) +} + +func (f *fakeKube) handleCreatePod(w http.ResponseWriter, r *http.Request) { + var pod k8sPod + if err := json.NewDecoder(r.Body).Decode(&pod); err != nil { + writeK8sStatus(w, http.StatusBadRequest, "BadRequest", err.Error()) + return + } + pod.Status = k8sPodStatus{ + Phase: "Running", + ContainerStatuses: []k8sContainerStatus{ + {Name: sandboxContainerName, Ready: true}, + }, + } + f.mu.Lock() + f.pods[pod.Metadata.Name] = &pod + f.mu.Unlock() + w.WriteHeader(http.StatusCreated) + _ = json.NewEncoder(w).Encode(&pod) +} + +func (f *fakeKube) handlePod(w http.ResponseWriter, r *http.Request) { + m := podPathRe.FindStringSubmatch(r.URL.Path) + name := m[2] + f.mu.Lock() + pod, ok := f.pods[name] + f.mu.Unlock() + switch r.Method { + case http.MethodGet: + if !ok { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "pod not found") + return + } + _ = json.NewEncoder(w).Encode(pod) + case http.MethodDelete: + if !ok { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "pod not found") + return + } + f.mu.Lock() + delete(f.pods, name) + f.deleted = append(f.deleted, name) + f.mu.Unlock() + _ = json.NewEncoder(w).Encode(map[string]any{"kind": "Status", "status": "Success"}) + default: + writeK8sStatus(w, http.StatusMethodNotAllowed, "MethodNotAllowed", r.Method) + } +} + +func (f *fakeKube) handleListPods(w http.ResponseWriter, r *http.Request) { + selector := r.URL.Query().Get("labelSelector") + var want [][2]string + if selector != "" { + for _, kv := range strings.Split(selector, ",") { + k, v, _ := strings.Cut(kv, "=") + want = append(want, [2]string{k, v}) + } + } + list := k8sPodList{Items: []k8sPod{}} + f.mu.Lock() + for _, pod := range f.pods { + match := true + for _, kv := range want { + if pod.Metadata.Labels[kv[0]] != kv[1] { + match = false + break + } + } + if match { + list.Items = append(list.Items, *pod) + } + } + f.mu.Unlock() + _ = json.NewEncoder(w).Encode(&list) +} + +// ── exec ── + +var ( + uploadRe = regexp.MustCompile(`^head -c (\d+) > (\S+) && wc -c < (\S+)$`) + fileOpsRe = regexp.MustCompile(`^head -c (\d+) \| python3 (\S+)$`) +) + +var execUpgrader = websocket.Upgrader{Subprotocols: []string{k8sExecProtocolV4}} + +// execConn wraps the server side of one exec connection. +type execConn struct { + conn *websocket.Conn + writeMu sync.Mutex +} + +func (e *execConn) send(channel byte, data []byte) { + e.writeMu.Lock() + defer e.writeMu.Unlock() + _ = e.conn.WriteMessage(websocket.BinaryMessage, append([]byte{channel}, data...)) +} + +func (e *execConn) finish(exitCode int) { + var status []byte + if exitCode == 0 { + status = []byte(`{"metadata":{},"status":"Success"}`) + } else { + status = []byte(fmt.Sprintf(`{"metadata":{},"status":"Failure","reason":"NonZeroExitCode","details":{"causes":[{"reason":"ExitCode","message":"%d"}]}}`, exitCode)) + } + e.send(k8sChannelError, status) + e.writeMu.Lock() + _ = e.conn.WriteControl(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""), timeNowPlusSecond()) + e.writeMu.Unlock() + _ = e.conn.Close() +} + +// readStdin reads stdin frames until n bytes have arrived (or the conn ends). +func (e *execConn) readStdin(n int) []byte { + var buf bytes.Buffer + for buf.Len() < n { + _, data, err := e.conn.ReadMessage() + if err != nil { + break + } + if len(data) > 0 && data[0] == k8sChannelStdin { + buf.Write(data[1:]) + } + } + return buf.Bytes() +} + +func (f *fakeKube) handleExec(w http.ResponseWriter, r *http.Request) { + m := podExecRe.FindStringSubmatch(r.URL.Path) + podName := m[2] + f.mu.Lock() + _, podExists := f.pods[podName] + f.mu.Unlock() + if !podExists { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "pod not found") + return + } + command := r.URL.Query()["command"] + conn, err := execUpgrader.Upgrade(w, r, nil) + if err != nil { + return + } + e := &execConn{conn: conn} + f.runFakeProcess(podName, command, e) +} + +//nolint:gocognit // test fixture dispatch over the fake process kinds +func (f *fakeKube) runFakeProcess(podName string, command []string, e *execConn) { + // Bridge session: read JSON lines, respond with canned bridge responses. + if len(command) == 2 && command[0] == "python3" && command[1] == k8sBridgePath { + f.runFakeBridge(podName, e) + return + } + + // Shell one-shots. + if len(command) >= 3 && (command[0] == "/bin/sh" || command[0] == "sh") && command[1] == "-c" { + script := command[2] + if um := uploadRe.FindStringSubmatch(script); um != nil { + n, _ := strconv.Atoi(um[1]) + data := e.readStdin(n) + f.mu.Lock() + f.files[podName+":"+um[2]] = data + f.mu.Unlock() + e.send(k8sChannelStdout, []byte(strconv.Itoa(len(data))+"\n")) + e.finish(0) + return + } + if fm := fileOpsRe.FindStringSubmatch(script); fm != nil { + n, _ := strconv.Atoi(fm[1]) + req := e.readStdin(n) + f.mu.Lock() + script := f.files[podName+":"+fm[2]] + f.mu.Unlock() + if script == nil { + e.send(k8sChannelStderr, []byte("fileops script not uploaded")) + e.finish(1) + return + } + // Run the REAL uploaded fileops.py on the test host so the k8s + // transport is exercised against the genuine executor. + cmd := exec.Command("python3", "-c", string(script)) + cmd.Stdin = bytes.NewReader(req) + var stdout, stderr bytes.Buffer + cmd.Stdout, cmd.Stderr = &stdout, &stderr + code := 0 + if err := cmd.Run(); err != nil { + var ee *exec.ExitError + if errors.As(err, &ee) { + code = ee.ExitCode() + } else { + e.send(k8sChannelStderr, []byte(err.Error())) + e.finish(127) + return + } + } + e.send(k8sChannelStdout, stdout.Bytes()) + if stderr.Len() > 0 { + e.send(k8sChannelStderr, stderr.Bytes()) + } + e.finish(code) + return + } + // Workdir-wrapped bash: ["/bin/sh","-c",script,"fleet-bash",dir,cmd] + if len(command) == 6 && command[3] == "fleet-bash" { + f.dispatchBash(command[5], command[4], e) + return + } + } + if len(command) == 3 && command[0] == "bash" && command[1] == "-c" { + f.dispatchBash(command[2], "", e) + return + } + e.send(k8sChannelStderr, []byte(fmt.Sprintf("fake apiserver: unhandled command %q", command))) + e.finish(127) +} + +func (f *fakeKube) dispatchBash(cmd, workdir string, e *execConn) { + f.mu.Lock() + f.lastBashWorkdir = workdir + behavior := f.bashBehaviors[cmd] + f.mu.Unlock() + if behavior == nil { + e.send(k8sChannelStderr, []byte("fake apiserver: no behavior for bash command "+cmd)) + e.finish(127) + return + } + stdout := &channelWriter{e: e, channel: k8sChannelStdout} + stderr := &channelWriter{e: e, channel: k8sChannelStderr} + e.finish(behavior(workdir, stdout, stderr, e.conn)) +} + +// runFakeBridge speaks the bridge line protocol: each request line gets a +// canned success response echoing the code back in `result`. +func (f *fakeKube) runFakeBridge(_ string, e *execConn) { + var pending bytes.Buffer + for { + _, data, err := e.conn.ReadMessage() + if err != nil { + _ = e.conn.Close() + return + } + if len(data) == 0 || data[0] != k8sChannelStdin { + continue + } + pending.Write(data[1:]) + for { + line, rest, found := bytes.Cut(pending.Bytes(), []byte("\n")) + if !found { + break + } + var req bridgeRequest + if err := json.Unmarshal(line, &req); err != nil { + e.send(k8sChannelStderr, []byte("bad bridge request: "+err.Error())) + pending = *bytes.NewBuffer(append([]byte(nil), rest...)) + continue + } + resp, _ := json.Marshal(bridgeResponse{Status: "ok", Result: "ran: " + req.Code}) + e.send(k8sChannelStdout, append(resp, '\n')) + pending = *bytes.NewBuffer(append([]byte(nil), rest...)) + } + } +} + +type channelWriter struct { + e *execConn + channel byte +} + +func (c *channelWriter) Write(p []byte) (int, error) { + c.e.send(c.channel, p) + return len(p), nil +} + +// pythonAvailable reports whether the test host has python3 (the fileops +// integration tests need it; CI always does — it runs the host-executor +// suite). +func pythonAvailable() bool { + _, err := exec.LookPath("python3") + return err == nil +} + +func timeNowPlusSecond() time.Time { return time.Now().Add(time.Second) } diff --git a/internal/sandbox/k8s_kubeconfig.go b/internal/sandbox/k8s_kubeconfig.go new file mode 100644 index 00000000..fcfbe14c --- /dev/null +++ b/internal/sandbox/k8s_kubeconfig.go @@ -0,0 +1,227 @@ +// Copyright (c) 2026 ElcanoTek +// SPDX-License-Identifier: MIT + +package sandbox + +// k8s_kubeconfig.go loads client credentials for the kubernetes sandbox +// backend from a kubeconfig file — the out-of-cluster path used when the +// fleet control plane runs OUTSIDE the cluster that hosts its sandbox pods +// (a dev box pointed at kind, or a single-box install delegating runners to +// a cluster). In-cluster service-account auth (the production Helm path) +// lives in newInClusterClient (k8s_client.go). +// +// Deliberately minimal: current-context resolution, token / token-file / +// client-certificate credentials, CA bundles inline or by path. exec +// plugins and auth-providers are REFUSED with an actionable error rather +// than half-supported — the fleet process runs unattended, and shelling out +// to an interactive credential helper on every token expiry is exactly the +// kind of silent-degradation surface the fail-closed posture forbids. + +import ( + "crypto/tls" + "crypto/x509" + "encoding/base64" + "fmt" + "net/http" + "net/url" + "os" + "path/filepath" + + "github.com/goccy/go-yaml" +) + +// kubeconfigFile mirrors the subset of the kubeconfig v1 schema the loader +// reads. Unknown fields are ignored by the YAML decoder, EXCEPT the exec / +// auth-provider blocks, which are decoded precisely so they can be refused. +type kubeconfigFile struct { + CurrentContext string `yaml:"current-context"` + Contexts []struct { + Name string `yaml:"name"` + Context struct { + Cluster string `yaml:"cluster"` + User string `yaml:"user"` + Namespace string `yaml:"namespace"` + } `yaml:"context"` + } `yaml:"contexts"` + Clusters []struct { + Name string `yaml:"name"` + Cluster struct { + Server string `yaml:"server"` + CertificateAuthority string `yaml:"certificate-authority"` + CertificateAuthorityData string `yaml:"certificate-authority-data"` + InsecureSkipTLSVerify bool `yaml:"insecure-skip-tls-verify"` + } `yaml:"cluster"` + } `yaml:"clusters"` + Users []struct { + Name string `yaml:"name"` + User struct { + ClientCertificate string `yaml:"client-certificate"` + ClientCertificateData string `yaml:"client-certificate-data"` + ClientKey string `yaml:"client-key"` + ClientKeyData string `yaml:"client-key-data"` + Token string `yaml:"token"` + TokenFile string `yaml:"tokenFile"` + Exec map[string]any `yaml:"exec"` + AuthProvider map[string]any `yaml:"auth-provider"` + } `yaml:"user"` + } `yaml:"users"` +} + +// newKubeconfigClient builds a k8sClient from the kubeconfig at path, +// following its current-context. The returned namespace is the context's +// default namespace ("" when the context sets none). +func newKubeconfigClient(path string) (*k8sClient, string, error) { + raw, err := os.ReadFile(path) //nolint:gosec // operator-configured kubeconfig path + if err != nil { + return nil, "", fmt.Errorf("read kubeconfig: %w", err) + } + var kc kubeconfigFile + if err := yaml.Unmarshal(raw, &kc); err != nil { + return nil, "", fmt.Errorf("parse kubeconfig %s: %w", path, err) + } + if kc.CurrentContext == "" { + return nil, "", fmt.Errorf("kubeconfig %s has no current-context", path) + } + var clusterName, userName, namespace string + for _, c := range kc.Contexts { + if c.Name == kc.CurrentContext { + clusterName, userName, namespace = c.Context.Cluster, c.Context.User, c.Context.Namespace + break + } + } + if clusterName == "" { + return nil, "", fmt.Errorf("kubeconfig %s: current-context %q not found", path, kc.CurrentContext) + } + + // Relative CA / cert / key paths in a kubeconfig are relative to the FILE, + // not the process cwd — kubectl's convention, kept so the same file works. + baseDir := filepath.Dir(path) + resolve := func(p string) string { + if p == "" || filepath.IsAbs(p) { + return p + } + return filepath.Join(baseDir, p) + } + + tlsCfg := &tls.Config{MinVersion: tls.VersionTLS12} + server, err := applyKubeconfigCluster(&kc, clusterName, path, resolve, tlsCfg) + if err != nil { + return nil, "", err + } + + client := &k8sClient{tlsConfig: tlsCfg} + if err := applyKubeconfigUser(&kc, userName, path, resolve, tlsCfg, client); err != nil { + return nil, "", err + } + + base, err := url.Parse(server) + if err != nil { + return nil, "", fmt.Errorf("kubeconfig %s: parse server URL %q: %w", path, server, err) + } + if base.Scheme != "https" { + return nil, "", fmt.Errorf("kubeconfig %s: server %q is not https — the sandbox control channel requires TLS (fail-closed)", path, server) + } + client.baseURL = base + client.httpc = &http.Client{Transport: &http.Transport{TLSClientConfig: tlsCfg}} + return client, namespace, nil +} + +// applyKubeconfigCluster resolves the named cluster's server URL and installs +// its CA bundle into tlsCfg. insecure-skip-tls-verify is refused rather than +// honored: a sandbox control channel that skips server verification can be +// MITM'd into running tool calls on an attacker's cluster. +func applyKubeconfigCluster(kc *kubeconfigFile, clusterName, path string, resolve func(string) string, tlsCfg *tls.Config) (server string, err error) { + for _, c := range kc.Clusters { + if c.Name != clusterName { + continue + } + server = c.Cluster.Server + if c.Cluster.InsecureSkipTLSVerify { + return "", fmt.Errorf("kubeconfig %s: cluster %q sets insecure-skip-tls-verify, which the sandbox backend refuses (fail-closed) — use a CA bundle", path, clusterName) + } + caPEM := []byte(nil) + switch { + case c.Cluster.CertificateAuthorityData != "": + caPEM, err = base64.StdEncoding.DecodeString(c.Cluster.CertificateAuthorityData) + if err != nil { + return "", fmt.Errorf("kubeconfig %s: decode certificate-authority-data: %w", path, err) + } + case c.Cluster.CertificateAuthority != "": + caPEM, err = os.ReadFile(resolve(c.Cluster.CertificateAuthority)) + if err != nil { + return "", fmt.Errorf("kubeconfig %s: read certificate-authority: %w", path, err) + } + } + if caPEM != nil { + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(caPEM) { + return "", fmt.Errorf("kubeconfig %s: cluster %q CA bundle contains no usable certificates", path, clusterName) + } + tlsCfg.RootCAs = pool + } + break + } + if server == "" { + return "", fmt.Errorf("kubeconfig %s: cluster %q not found or has no server", path, clusterName) + } + return server, nil +} + +// applyKubeconfigUser resolves the named user's credentials into the client +// (token / token-file) or tlsCfg (client certificate). exec plugins and +// auth-providers are refused — the fleet process runs unattended. +func applyKubeconfigUser(kc *kubeconfigFile, userName, path string, resolve func(string) string, tlsCfg *tls.Config, client *k8sClient) error { + for _, u := range kc.Users { + if u.Name != userName { + continue + } + if u.User.Exec != nil || u.User.AuthProvider != nil { + return fmt.Errorf("kubeconfig %s: user %q uses an exec plugin / auth-provider, which the sandbox backend does not support — "+ + "create a ServiceAccount token or client certificate for fleet instead", path, userName) + } + certPEM, err := kubeconfigPEM(u.User.ClientCertificateData, resolve(u.User.ClientCertificate)) + if err != nil { + return fmt.Errorf("kubeconfig %s: client-certificate: %w", path, err) + } + keyPEM, err := kubeconfigPEM(u.User.ClientKeyData, resolve(u.User.ClientKey)) + if err != nil { + return fmt.Errorf("kubeconfig %s: client-key: %w", path, err) + } + switch { + case certPEM != nil && keyPEM != nil: + cert, cerr := tls.X509KeyPair(certPEM, keyPEM) + if cerr != nil { + return fmt.Errorf("kubeconfig %s: load client certificate: %w", path, cerr) + } + tlsCfg.Certificates = []tls.Certificate{cert} + case u.User.Token != "": + client.staticToken = u.User.Token + case u.User.TokenFile != "": + client.tokenFile = resolve(u.User.TokenFile) + default: + return fmt.Errorf("kubeconfig %s: user %q has no supported credentials (token, tokenFile, or client certificate)", path, userName) + } + return nil + } + return fmt.Errorf("kubeconfig %s: user %q not found", path, userName) +} + +// kubeconfigPEM loads a PEM blob from inline base64 data (preferred) or a +// file path; both empty returns (nil, nil). +func kubeconfigPEM(inlineB64, filePath string) ([]byte, error) { + if inlineB64 != "" { + data, err := base64.StdEncoding.DecodeString(inlineB64) + if err != nil { + return nil, fmt.Errorf("decode inline data: %w", err) + } + return data, nil + } + if filePath == "" { + return nil, nil + } + data, err := os.ReadFile(filePath) //nolint:gosec // path from an operator-configured kubeconfig, not request input + if err != nil { + return nil, err + } + return data, nil +} diff --git a/internal/sandbox/k8s_preflight.go b/internal/sandbox/k8s_preflight.go new file mode 100644 index 00000000..498dbda6 --- /dev/null +++ b/internal/sandbox/k8s_preflight.go @@ -0,0 +1,99 @@ +// Copyright (c) 2026 ElcanoTek +// SPDX-License-Identifier: MIT + +package sandbox + +// Boot-time preflight for the kubernetes sandbox backend (#989 / ADR-0049). +// +// When FLEET_SANDBOX_BACKEND=kubernetes is selected, a cluster that cannot +// actually run sandbox pods must abort boot — never silently fall back to +// podman or host execution (the same no-degrade posture as PreflightRuntime +// and PreflightAllowlistedNetwork). The checks run in failure-likelihood +// order so the first error an operator sees is the most actionable one: +// +// 1. apiserver reachable + credentials valid (GET /version) +// 2. RBAC: create/get/list/delete pods and create pods/exec in the sandbox +// namespace (SelfSubjectAccessReview — precise "which verb is missing") +// 3. the shared workspace PVC exists +// 4. the sealed-egress NetworkPolicy object exists (the chart ships it; +// the OBJECT check cannot prove the CNI enforces it, and the docs say so) +// 5. the RuntimeClass exists, when one is configured +// +// Deliberately NOT checked: image pullability (imagePullSecrets are resolved +// by the kubelet per node — the only faithful probe is running a pod, which +// the first turn does, failing fast on ErrImagePull in waitForRunning) and +// PVC access mode (RWX vs RWO is advisory in the API; a wrong mode surfaces +// as a scheduling error the docs' checklist covers). + +import ( + "context" + "fmt" + "log" + "time" +) + +// k8sPreflightTimeout bounds the whole preflight sequence. Generous for a +// cold connection; short enough that a dead apiserver fails boot promptly. +const k8sPreflightTimeout = 30 * time.Second + +// k8sRBACChecks are the (verb, resource, subresource) grants the backend +// needs. Listed as data so the error message names exactly what is missing. +var k8sRBACChecks = []struct { + verb, resource, subresource string +}{ + {"create", "pods", ""}, + {"get", "pods", ""}, + {"list", "pods", ""}, + {"delete", "pods", ""}, + {"create", "pods", "exec"}, +} + +// Preflight verifies, fail-closed, that the cluster can deliver what the +// kubernetes sandbox backend promises. Called from the single production +// pool-construction path (agent.buildSandboxPool) and from +// `fleet validate-config` — mirroring PreflightRuntime's contract. Callers +// must treat any error as fatal to boot; there is no degraded mode. +func (b *KubernetesBackend) Preflight(ctx context.Context) error { + if b.cfg.WorkspaceClaim == "" { + return fmt.Errorf("kubernetes sandbox preflight: no workspace PVC configured — set FLEET_SANDBOX_K8S_WORKSPACE_CLAIM (or the bundle manifest's sandbox.kubernetes.workspace_claim) to the ReadWriteMany claim shared with the control plane") + } + preCtx, cancel := context.WithTimeout(ctx, k8sPreflightTimeout) + defer cancel() + + version, err := b.client.serverVersion(preCtx) + if err != nil { + return fmt.Errorf("kubernetes sandbox preflight: apiserver unreachable or credentials rejected: %w", err) + } + + for _, check := range k8sRBACChecks { + allowed, err := b.client.selfSubjectAccessReview(preCtx, b.cfg.Namespace, check.verb, check.resource, check.subresource) + if err != nil { + return fmt.Errorf("kubernetes sandbox preflight: access review for %s %s/%s failed: %w", check.verb, check.resource, check.subresource, err) + } + if !allowed { + target := check.resource + if check.subresource != "" { + target += "/" + check.subresource + } + return fmt.Errorf("kubernetes sandbox preflight: the fleet service account may not %s %s in namespace %q — grant the fleet-runner Role from the Helm chart (deploy/helm/fleet) or an equivalent RoleBinding", check.verb, target, b.cfg.Namespace) + } + } + + if err := b.client.getPVC(preCtx, b.cfg.Namespace, b.cfg.WorkspaceClaim); err != nil { + return fmt.Errorf("kubernetes sandbox preflight: workspace PVC %q not readable in namespace %q (it must be a ReadWriteMany claim mounted by the control plane at the same path): %w", b.cfg.WorkspaceClaim, b.cfg.Namespace, err) + } + + if err := b.client.getNetworkPolicy(preCtx, b.cfg.Namespace, b.cfg.NetworkPolicyName); err != nil { + return fmt.Errorf("kubernetes sandbox preflight: sealed-egress NetworkPolicy %q not found in namespace %q — the deny-all policy for pods labeled %s=none must exist before sealed sandboxes can be trusted (the Helm chart ships it): %w", + b.cfg.NetworkPolicyName, b.cfg.Namespace, k8sLabelEgress, err) + } + + if b.cfg.RuntimeClassName != "" { + if err := b.client.getRuntimeClass(preCtx, b.cfg.RuntimeClassName); err != nil { + return fmt.Errorf("kubernetes sandbox preflight: RuntimeClass %q not found — a hypervisor runtime that cannot be verified must abort boot, never degrade to the default runtime (ADR-0010 posture): %w", b.cfg.RuntimeClassName, err) + } + } + + log.Printf("sandbox: kubernetes backend preflight OK — apiserver %s, sandbox namespace %q", version, b.cfg.Namespace) + return nil +} diff --git a/internal/sandbox/k8s_preflight_test.go b/internal/sandbox/k8s_preflight_test.go new file mode 100644 index 00000000..0c7cd9ba --- /dev/null +++ b/internal/sandbox/k8s_preflight_test.go @@ -0,0 +1,174 @@ +package sandbox + +// Preflight + kubeconfig fail-closed tests for the kubernetes backend (#989). + +import ( + "context" + "encoding/base64" + "encoding/pem" + "fmt" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestK8sPreflightHappyPath(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes", RuntimeClassName: "kata"}) + if err := backend.Preflight(context.Background()); err != nil { + t.Fatalf("Preflight: %v", err) + } +} + +func TestK8sPreflightFailClosed(t *testing.T) { + t.Run("missing workspace claim", func(t *testing.T) { + fake := newFakeKube(t) + b, err := NewKubernetesBackend(KubernetesConfig{KubeconfigPath: fake.kubeconfigPath(t)}) + if err != nil { + t.Fatal(err) + } + if err := b.Preflight(context.Background()); err == nil || !strings.Contains(err.Error(), "workspace PVC") { + t.Errorf("want workspace-claim error, got %v", err) + } + }) + t.Run("rbac denial names the verb", func(t *testing.T) { + fake := newFakeKube(t) + fake.denied["create pods/exec"] = true + backend := fake.backend(t, KubernetesConfig{}) + err := backend.Preflight(context.Background()) + if err == nil || !strings.Contains(err.Error(), "create pods/exec") { + t.Errorf("want create pods/exec denial, got %v", err) + } + }) + t.Run("missing pvc", func(t *testing.T) { + fake := newFakeKube(t) + fake.noPVC = true + backend := fake.backend(t, KubernetesConfig{}) + if err := backend.Preflight(context.Background()); err == nil || !strings.Contains(err.Error(), "workspace PVC") { + t.Errorf("want PVC error, got %v", err) + } + }) + t.Run("missing networkpolicy", func(t *testing.T) { + fake := newFakeKube(t) + fake.noNetpol = true + backend := fake.backend(t, KubernetesConfig{}) + if err := backend.Preflight(context.Background()); err == nil || !strings.Contains(err.Error(), "NetworkPolicy") { + t.Errorf("want NetworkPolicy error, got %v", err) + } + }) + t.Run("missing runtimeclass", func(t *testing.T) { + fake := newFakeKube(t) + fake.noRuntimeClass = true + backend := fake.backend(t, KubernetesConfig{RuntimeClassName: "kata"}) + if err := backend.Preflight(context.Background()); err == nil || !strings.Contains(err.Error(), "RuntimeClass") { + t.Errorf("want RuntimeClass error, got %v", err) + } + }) + t.Run("bad credentials", func(t *testing.T) { + fake := newFakeKube(t) + path := fake.kubeconfigPath(t) + raw, _ := os.ReadFile(path) + bad := strings.ReplaceAll(string(raw), fakeKubeToken, "wrong-token") + if err := os.WriteFile(path, []byte(bad), 0o600); err != nil { + t.Fatal(err) + } + b, err := NewKubernetesBackend(KubernetesConfig{KubeconfigPath: path, WorkspaceClaim: "ws"}) + if err != nil { + t.Fatal(err) + } + if err := b.Preflight(context.Background()); err == nil || !strings.Contains(err.Error(), "unreachable or credentials rejected") { + t.Errorf("want credentials error, got %v", err) + } + }) +} + +func TestK8sBackendDefaults(t *testing.T) { + fake := newFakeKube(t) + b, err := NewKubernetesBackend(KubernetesConfig{KubeconfigPath: fake.kubeconfigPath(t), WorkspaceClaim: "ws"}) + if err != nil { + t.Fatal(err) + } + if got := b.Namespace(); got != defaultK8sNamespace { + t.Errorf("default namespace = %q, want %q", got, defaultK8sNamespace) + } + if b.cfg.NetworkPolicyName != defaultK8sNetworkPolicy { + t.Errorf("default networkpolicy = %q", b.cfg.NetworkPolicyName) + } + if b.StartTimeout() != defaultK8sStartTimeout { + t.Errorf("default start timeout = %v", b.StartTimeout()) + } +} + +func writeKubeconfig(t *testing.T, body string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "kubeconfig") + if err := os.WriteFile(path, []byte(body), 0o600); err != nil { + t.Fatal(err) + } + return path +} + +func TestKubeconfigFailClosed(t *testing.T) { + fake := newFakeKube(t) + caPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: fake.srv.Certificate().Raw}) + caB64 := base64.StdEncoding.EncodeToString(caPEM) + + t.Run("insecure-skip-tls-verify refused", func(t *testing.T) { + path := writeKubeconfig(t, fmt.Sprintf(`current-context: c +contexts: [{name: c, context: {cluster: cl, user: u}}] +clusters: [{name: cl, cluster: {server: %s, insecure-skip-tls-verify: true}}] +users: [{name: u, user: {token: t}}] +`, fake.srv.URL)) + if _, _, err := newKubeconfigClient(path); err == nil || !strings.Contains(err.Error(), "insecure-skip-tls-verify") { + t.Errorf("want insecure refusal, got %v", err) + } + }) + t.Run("exec plugin refused", func(t *testing.T) { + path := writeKubeconfig(t, fmt.Sprintf(`current-context: c +contexts: [{name: c, context: {cluster: cl, user: u}}] +clusters: [{name: cl, cluster: {server: %s, certificate-authority-data: %s}}] +users: [{name: u, user: {exec: {command: aws}}}] +`, fake.srv.URL, caB64)) + if _, _, err := newKubeconfigClient(path); err == nil || !strings.Contains(err.Error(), "exec plugin") { + t.Errorf("want exec-plugin refusal, got %v", err) + } + }) + t.Run("http server refused", func(t *testing.T) { + path := writeKubeconfig(t, `current-context: c +contexts: [{name: c, context: {cluster: cl, user: u}}] +clusters: [{name: cl, cluster: {server: http://127.0.0.1:8080}}] +users: [{name: u, user: {token: t}}] +`) + if _, _, err := newKubeconfigClient(path); err == nil || !strings.Contains(err.Error(), "requires TLS") { + t.Errorf("want TLS refusal, got %v", err) + } + }) + t.Run("no credentials refused", func(t *testing.T) { + path := writeKubeconfig(t, fmt.Sprintf(`current-context: c +contexts: [{name: c, context: {cluster: cl, user: u}}] +clusters: [{name: cl, cluster: {server: %s, certificate-authority-data: %s}}] +users: [{name: u, user: {}}] +`, fake.srv.URL, caB64)) + if _, _, err := newKubeconfigClient(path); err == nil || !strings.Contains(err.Error(), "no supported credentials") { + t.Errorf("want no-credentials refusal, got %v", err) + } + }) + t.Run("context namespace surfaces", func(t *testing.T) { + path := writeKubeconfig(t, fmt.Sprintf(`current-context: c +contexts: [{name: c, context: {cluster: cl, user: u, namespace: my-ns}}] +clusters: [{name: cl, cluster: {server: %s, certificate-authority-data: %s}}] +users: [{name: u, user: {token: t}}] +`, fake.srv.URL, caB64)) + _, ns, err := newKubeconfigClient(path) + if err != nil || ns != "my-ns" { + t.Errorf("namespace = %q, %v; want my-ns", ns, err) + } + }) + t.Run("missing current-context refused", func(t *testing.T) { + path := writeKubeconfig(t, "contexts: []\n") + if _, _, err := newKubeconfigClient(path); err == nil || !strings.Contains(err.Error(), "current-context") { + t.Errorf("want current-context error, got %v", err) + } + }) +} diff --git a/internal/sandbox/pool.go b/internal/sandbox/pool.go index 8ffddd88..78dab9c6 100644 --- a/internal/sandbox/pool.go +++ b/internal/sandbox/pool.go @@ -139,9 +139,15 @@ type PoolConfig struct { BridgeScript []byte // Container holds the per-sandbox container settings (image, mounts, - // caps). Required when Mode == ModeContainer. + // caps). Required when Mode == ModeContainer, and the source of the + // backend-shared knobs (image, workspace path, limits, network posture) + // when Mode == ModeKubernetes. Container ContainerConfig + // KubernetesBackend is the boot-built handle for the kubernetes sandbox + // backend (#989). Required when Mode == ModeKubernetes; nil otherwise. + KubernetesBackend *KubernetesBackend + // EgressProxy, when non-nil, is the host-side allowlist proxy (#211) used by // TakeContainerWithEgress for "allowlisted" network mode. nil means // allowlisted mode is unavailable: such requests FAIL CLOSED (an error) @@ -301,7 +307,6 @@ func (p *Pool) TakeContainerWithOverrides(ctx context.Context, ov ResourceOverri } cfg := p.cfg.Container cfg.BridgeScript = p.cfg.BridgeScript - cfg.StorageOptSupported = p.storageOptSupported(ctx) // Network sealing is enforced HERE rather than upstream so the lockdown // contract is impossible to bypass via a bad caller. cfg.NoNetwork = noNetwork @@ -311,9 +316,9 @@ func (p *Pool) TakeContainerWithOverrides(ctx context.Context, ov ResourceOverri cfg = ov.applyTo(cfg) // See newSandbox below for why we resolve the start timeout here // rather than reading it raw from cfg. - startCtx, cancel := context.WithTimeout(ctx, resolveStartTimeout(cfg)+5*time.Second) + startCtx, cancel := context.WithTimeout(ctx, p.startTimeoutFor(cfg)+5*time.Second) defer cancel() - sb, err := NewContainer(startCtx, cfg) + sb, err := p.newBackendSandbox(startCtx, cfg) if err != nil { return nil, func() {}, err } @@ -321,10 +326,45 @@ func (p *Pool) TakeContainerWithOverrides(ctx context.Context, ov ResourceOverri sb.Close() return nil, func() {}, ErrClosed } - sb.SetPythonCellTimeout(p.cfg.PythonCellTimeout) return sb, sb.Close, nil } +// startTimeoutFor resolves the outer construction budget for the active +// backend: the kubernetes backend's pod start ceiling (schedule + pull) when +// it is selected, else the podman container start timeout. +func (p *Pool) startTimeoutFor(cfg ContainerConfig) time.Duration { + if p.cfg.Mode == ModeKubernetes && p.cfg.KubernetesBackend != nil { + return p.cfg.KubernetesBackend.StartTimeout() + } + return resolveStartTimeout(cfg) +} + +// newBackendSandbox constructs one sandbox from a fully-resolved per-call +// cfg, routing to the active container backend (#989): a Kubernetes pod when +// ModeKubernetes, else a rootless-Podman container. The podman path pays the +// storage-opt probe here; kubernetes has no analogue (its disk cap is the +// pod's ephemeral-storage limit, applied unconditionally in the pod spec). +func (p *Pool) newBackendSandbox(ctx context.Context, cfg ContainerConfig) (*Sandbox, error) { + var ( + sb *Sandbox + err error + ) + if p.cfg.Mode == ModeKubernetes { + if p.cfg.KubernetesBackend == nil { + return nil, errors.New("sandbox: kubernetes backend selected but not constructed (fail-closed)") + } + sb, err = p.cfg.KubernetesBackend.newSandbox(ctx, cfg) + } else { + cfg.StorageOptSupported = p.storageOptSupported(ctx) + sb, err = NewContainer(ctx, cfg) + } + if err != nil { + return nil, err + } + sb.SetPythonCellTimeout(p.cfg.PythonCellTimeout) + return sb, nil +} + // TakeContainerWithEgress cold-starts a fresh container in "allowlisted" network // mode (#211): slirp4netns transport with HTTPS_PROXY pointed at the pool's // EgressProxy, scoped to allowlist for THIS turn via a fresh per-turn token. The @@ -345,6 +385,13 @@ func (p *Pool) TakeContainerWithEgress(ctx context.Context, ov ResourceOverride, if p.cfg.Container.Image == "" { return nil, func() {}, ErrContainerUnavailable } + if p.cfg.Mode == ModeKubernetes { + // The egress proxy binds to the control-plane host's loopback; a pod on + // another node cannot reach it. Refuse rather than grant open egress + // under an "allowlisted" banner. buildSandboxPool refuses the mode at + // boot too; this is the can't-bypass-it backstop. + return nil, func() {}, errors.New("allowlisted network mode is not supported by the kubernetes sandbox backend (fail-closed)") + } if p.cfg.EgressProxy == nil { return nil, func() {}, errors.New("allowlisted network mode requested but no egress proxy is configured (fail-closed)") } @@ -502,6 +549,16 @@ func (p *Pool) EgressDefault() (mode string, allowlist []string) { return p.cfg.DefaultNetworkMode, p.cfg.DefaultEgressAllowlist } +// KubernetesBackend exposes the kubernetes backend handle when that backend +// is active (nil under podman/host). Used by boot-time maintenance (the +// orphan-pod prune in cmd/fleet) and diagnostics. +func (p *Pool) KubernetesBackend() *KubernetesBackend { + if p == nil { + return nil + } + return p.cfg.KubernetesBackend +} + func (p *Pool) Close() { if p == nil { return @@ -766,12 +823,11 @@ func (p *Pool) storageOptSupported(ctx context.Context) bool { func (p *Pool) newSandbox(ctx context.Context) (*Sandbox, error) { switch p.cfg.Mode { - case ModeContainer: + case ModeContainer, ModeKubernetes: cfg := p.cfg.Container cfg.BridgeScript = p.cfg.BridgeScript - cfg.StorageOptSupported = p.storageOptSupported(ctx) - // resolveStartTimeout applies the same default NewContainer would - // apply internally. Without this, the OUTER context timeout is + // startTimeoutFor applies the same default the backend constructor + // would apply internally. Without this, the OUTER context timeout is // `0+5s = 5s` when StartTimeout isn't set explicitly, which // cancels podman before its first-run idmapped-layer chown // finishes — that chown takes ~12s on a fresh sandbox image @@ -780,14 +836,9 @@ func (p *Pool) newSandbox(ctx context.Context) (*Sandbox, error) { // "first message after deploy fails, second works fine" // (because by the time the second message lands, the warm pool // has finished filling against the now-cached chowned layer). - startCtx, cancel := context.WithTimeout(ctx, resolveStartTimeout(cfg)+5*time.Second) + startCtx, cancel := context.WithTimeout(ctx, p.startTimeoutFor(cfg)+5*time.Second) defer cancel() - sb, err := NewContainer(startCtx, cfg) - if err != nil { - return nil, err - } - sb.SetPythonCellTimeout(p.cfg.PythonCellTimeout) - return sb, nil + return p.newBackendSandbox(startCtx, cfg) case ModeHost: // Test-only fixture path. agent.go forbids ModeHost in // production; this branch only fires when sandbox_test.go diff --git a/internal/sandbox/sandbox.go b/internal/sandbox/sandbox.go index fae98fa4..fb8e8a1b 100644 --- a/internal/sandbox/sandbox.go +++ b/internal/sandbox/sandbox.go @@ -88,6 +88,12 @@ const ( // Podman container with --read-only / dropped caps. Network egress // is per-turn (ContainerConfig.NoNetwork) — see container.go. ModeContainer + + // ModeKubernetes runs bash and the python bridge inside an ephemeral + // Kubernetes Pod, exec'd over the apiserver (#989) — the enterprise + // split-control-plane backend. Selected by FLEET_SANDBOX_BACKEND; + // see k8s_backend.go. + ModeKubernetes ) // BashRequest is the per-call input the sandbox sees for a bash @@ -237,8 +243,10 @@ func (s *Sandbox) SetDefaultWorkingDir(dir string) { s.mu.Unlock() } -// impl is the backend interface. Two concrete implementations live in -// host.go and container.go. +// impl is the backend interface. Three concrete implementations live in +// container.go (rootless Podman, the single-box production backend), +// k8s_backend.go (Kubernetes pods, the enterprise split backend, #989), and +// host.go (the test-only fixture behind the fleet_host_executor build tag). type impl interface { runBash(ctx context.Context, req BashRequest) (BashResult, error) runPython(ctx context.Context, req PythonRequest) (PythonResult, error) @@ -281,6 +289,8 @@ func (s *Sandbox) ModeName() string { return "host" case ModeContainer: return "container" + case ModeKubernetes: + return "kubernetes" default: return "unknown" } diff --git a/scripts/check_versions_test.go b/scripts/check_versions_test.go index 56a90312..3de84930 100644 --- a/scripts/check_versions_test.go +++ b/scripts/check_versions_test.go @@ -255,8 +255,9 @@ func goMinor(spec string) (string, bool) { // // - web/go.mod — a no-package boundary module, so nothing compiles against it // and a stale `go` line there is completely silent. -// - docs/EKS-DEPLOYMENT.md — a `FROM golang:` build stage an operator -// copies verbatim. Too old and their image cannot build the module at all. +// - docs/DEPLOYMENT-KUBERNETES.md — a `FROM golang:` build stage an +// operator copies verbatim (the control-plane image). Too old and their +// image cannot build the module at all. // // This is the same blind spot the node major had, and it bit the same way: the // pin sat at 1.26 after 1.27 shipped, with nothing to say so. Dependabot cannot @@ -284,19 +285,22 @@ func TestGoMinorAgreesEverywhere(t *testing.T) { t.Errorf("web/go.mod says go %s but go.mod says %s — bump them together (web/go.mod is major.minor only by design, but the minor still has to agree)", got, want) } - const eksDoc = "docs/EKS-DEPLOYMENT.md" - img := regexp.MustCompile(`FROM golang:(\S+?)(?:\s|$)`).FindAllStringSubmatch(readFile(t, root, eksDoc), -1) + const k8sDoc = "docs/DEPLOYMENT-KUBERNETES.md" + img := regexp.MustCompile(`FROM (?:docker\.io/library/)?golang:(\S+?)(?:\s|$)`).FindAllStringSubmatch(readFile(t, root, k8sDoc), -1) if len(img) == 0 { - t.Logf("%s has no `FROM golang:` stage — nothing to check", eksDoc) + // The stage existing is part of what this test pins: the doc's build + // recipe is the copy operators consume, so silently losing it would + // reopen the drift blind spot. + t.Errorf("%s has no `FROM golang:` stage — the control-plane image recipe should declare one", k8sDoc) } for _, m := range img { got, ok := goMinor(m[1]) if !ok { - t.Errorf("%s has `FROM golang:%s` — cannot read a major.minor from it", eksDoc, m[1]) + t.Errorf("%s has `FROM golang:%s` — cannot read a major.minor from it", k8sDoc, m[1]) continue } if got != want { - t.Errorf("%s builds on `golang:%s` but go.mod says %s — an operator copying that stage gets an image too old to build the module", eksDoc, m[1], want) + t.Errorf("%s builds on `golang:%s` but go.mod says %s — an operator copying that stage gets an image too old to build the module", k8sDoc, m[1], want) } } } From 754745b892f5f58aefba5052465cbfc5d8d00af4 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 00:22:48 +0000 Subject: [PATCH 27/34] CI: lint the workflows themselves, close two green-but-vacuous holes, correct the CodeQL waiver docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four review passes over .github/workflows (3.1k lines) and the docs that describe them. The gating architecture was already strong — SHA-pinned actions with a test enforcing the shape, aggregate gates with a test enforcing `needs` completeness, per-job least privilege — so this is mostly about the seams around it. Nothing here weakens a gate; two changes tighten one that could report green over work that never ran. New gate: workflow + shell lint (actionlint, shellcheck) Nothing checked the 3.1k lines of workflow YAML that decide what every other gate runs, and nothing checked the 6.2k lines of bash that ARE the deploy path (update.sh / bootstrap.sh / doctor.sh). Both now gate in ci.yml and dev-ci.yml, wired into `make lint` via `lint-actions`. Both start CLEAN, measured rather than assumed — actionlint found 5 items and shellcheck 3, all fixed here: - `run: "$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh"` in both lanes passed the path to the shell UNQUOTED. The YAML parser consumes the quotes, so the shell never saw them, despite the comment above the line showing quoting was intended. - SC2153 on `RESULTS` is env-supplied; waived in-line with a reason. - SC2148/SC2034 in three scripts: a `shell=bash` directive for the sourced-only lib, a reason on the documentary WEB_USER anchor, and a throwaway loop variable in fleet-upgrade.sh. Both are mutation-tested in both directions. Green-but-vacuous holes closed - ci.yml installed postgresql-client-18 best-effort, ending in `|| echo`. An unreachable PGDG left client 16 in place and backup_test.go's `t.Skipf("pg_dump major %d != server major %d")` turned the ONLY coverage of `fleet backup`/`fleet restore` off — behind the single required check on main. It now asserts the major. - The docs-only classifier initialised `docs_only=true` and only ever cleared it inside the loop, so an EMPTY diff classified as docs-only and skipped the suite — and `ci-gate` trusts exactly that value when deciding a skip is acceptable. An empty range is the absence of evidence, not evidence of prose. - dev-ci's `go test` lacked `-count=1` (ci.yml passes it on all three invocations). setup-go restores the build cache, which holds test RESULTS, and Go's cache key cannot see a Postgres service container. Honesty in docs Five documents advertised an in-source `// codeql[rule-id]` comment as a waiver route. codeql.yml itself records that all three forms were tried on #1249 and none produced a `suppressions` array — and carried BOTH claims, the stale "honored end-to-end" paragraph directly above the note refuting it, with no `packs:` input in the matrix. The register is the only route that works; the gate's own runtime advice said otherwise to the one person guaranteed to read it, a blocked contributor. ADR-0048's normative Decision stated the threshold as an OR where the jq implements a fallback; read literally it blocks on go/log-injection (error @ 6.1), the exact deadlock the ADR exists to undo. Also: CONTRIBUTING said Go 1.26.x (go.mod says 1.27.0); CODEOWNERS listed 7 of ci-gate's 13 needs; `make ci-web` ran 4 of the web job's 8 steps, dropping both npm audits, the override canary and the explicit typecheck — a clean local run and a red PR. Least privilege `issues: write` sat at WORKFLOW scope in both scheduled scan lanes, so it was live for every step of a long job running `govulncheck@latest` and a podman build pulling ~400 RPMs. Split into its own alarm job that checks out nothing — the shape scan-cron-alarm.yml already uses. Added `persist-credentials: false` to the write-scoped checkouts. Runner economics ci.yml — the expensive lane — had no `concurrency:`, so stacked pushes ran full suites to completion. Cancellation is scoped to `pull_request`; a push to main is the only tree-wide CodeQL verdict and must not be cancelled. `timeout-minutes` now on every job (was 2 of 13 workflows); values sized per job, after a first cut gave identical grype work 20 and 30 minutes. New tests, because this repo asserts invariants rather than remembering them: every workflow declares a top-level `permissions:` block, and the actionlint version/checksum agree across both lanes. Signed-off-by: Claude --- .github/CODEOWNERS | 15 +- .github/workflows/auto-merge-dependabot.yml | 1 + .github/workflows/benchmark.yml | 1 + .github/workflows/build-sandbox-image.yml | 1 + .github/workflows/ci.yml | 193 ++++++++++++++++++-- .github/workflows/codeql.yml | 32 ++-- .github/workflows/dev-ci.yml | 111 ++++++++++- .github/workflows/e2e-canary.yml | 2 + .github/workflows/govulncheck-scheduled.yml | 27 ++- .github/workflows/grype-scheduled.yml | 27 ++- .github/workflows/publish-sandbox-image.yml | 6 + .github/workflows/scan-cron-alarm.yml | 1 + .github/workflows/screenshots.yml | 1 + .github/workflows/semgrep.yml | 11 +- CONTRIBUTING.md | 11 +- Makefile | 45 ++++- docs/CODEQL.md | 18 +- docs/SCANNING.md | 9 +- docs/adr/0048-codeql-severity-gating.md | 23 ++- scripts/check_permissions_test.go | 57 ++++++ scripts/check_versions_test.go | 2 + scripts/doctor.sh | 3 + scripts/fleet-upgrade.sh | 3 +- scripts/lib/node-version.sh | 3 + 24 files changed, 524 insertions(+), 79 deletions(-) create mode 100644 scripts/check_permissions_test.go diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 977dfe50..5ed098d7 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -26,11 +26,16 @@ # - Require a pull request before merging (no direct pushes to `main`). # - Require status checks to pass, with "branches up to date" (strict) ON. The # SINGLE required check is the `CI gate` job in .github/workflows/ci.yml: it -# `needs` every other CI job (Go build / vet / lint / test — which also runs -# vet, the -race lane and govulncheck; Web lint / test / build; Playwright -# mocked; Playwright live; the gitleaks secret scan; and the Grype -# container-image CVE scan) and fails unless each -# one succeeded or was cleanly skipped. Requiring that one aggregate check +# `needs` EVERY other job in that file — the docs-only classifier; the +# gitleaks secret scan; the actionlint workflow lint; the migration DDL +# lint; the Helm chart lint; Go build / vet / lint / test (which also runs +# the -race lane and govulncheck); the ruff Python lint; CodeQL and Semgrep +# (called as reusable workflows); Web lint / typecheck / test / build with +# both npm audits; Playwright mocked; Playwright live; and the Grype +# container-image CVE scan — and fails unless each +# one succeeded or was cleanly skipped. +# Do not maintain this list by hand alone: scripts/check_gate_needs_test.go +# fails `make test` if any job in ci.yml is missing from `ci-gate`'s needs. Requiring that one aggregate check # rather than each job by name is what lets a docs-only PR skip the heavy jobs # without being left blocked on a required check that never reported. # - Block force-pushes (non-fast-forward) and branch deletion. diff --git a/.github/workflows/auto-merge-dependabot.yml b/.github/workflows/auto-merge-dependabot.yml index 00b60e27..776108ca 100644 --- a/.github/workflows/auto-merge-dependabot.yml +++ b/.github/workflows/auto-merge-dependabot.yml @@ -52,6 +52,7 @@ jobs: auto-merge: if: ${{ github.actor == 'dependabot[bot]' }} runs-on: ubuntu-latest + timeout-minutes: 5 permissions: contents: write pull-requests: write diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index 461f407c..97f064f0 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -21,6 +21,7 @@ jobs: benchmarks: name: Subsystem throughput benchmarks runs-on: ubuntu-latest + timeout-minutes: 60 services: postgres: diff --git a/.github/workflows/build-sandbox-image.yml b/.github/workflows/build-sandbox-image.yml index 98ace253..fffdc5d6 100644 --- a/.github/workflows/build-sandbox-image.yml +++ b/.github/workflows/build-sandbox-image.yml @@ -94,6 +94,7 @@ jobs: build: name: Build sandbox (no push) runs-on: ubuntu-latest + timeout-minutes: 45 env: BUNDLE_DIR: ${{ inputs.bundle_dir }} steps: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 13db2c3d..ac9ee351 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,6 +15,27 @@ permissions: # e2e-live, which npm-install and run thousands of third-party packages. contents: read +# Rapid successive pushes to the SAME pull request cancel the previous run. +# `dev-ci.yml` has had this since it was written; this lane — the expensive one, +# carrying mocked Playwright, the live e2e against a real backend + rootless +# sandbox, and the Grype image scan — did not, so a branch pushed three times in +# a minute queued three full suites and the two stale ones still had to finish +# before the interesting one started. +# +# `cancel-in-progress` is deliberately NOT blanket-true: a push to `main` is a +# merge that already happened, and its run is the tree-wide verdict the +# diff-informed PR runs cannot give (see docs/SCANNING.md on why a +# `pull_request` CodeQL run certifies a diff, not a tree). Cancelling that to +# make room for the next merge would throw away the only full-tree result. So +# cancellation is scoped to `pull_request`, where the superseding run covers +# strictly newer code. +concurrency: + # github.ref already differs between a PR (refs/pull/N/merge) and a push to + # main (refs/heads/main); event_name is included so the two can never share a + # group and cancel each other. + group: ci-${{ github.event_name }}-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + jobs: changes: # Classify the PR (or push) as docs-only or not. The heavy jobs below gate on @@ -26,6 +47,7 @@ jobs: # pasted into a markdown file, so it runs on every change. name: Detect docs-only changes runs-on: ubuntu-latest + timeout-minutes: 10 outputs: docs_only: ${{ steps.detect.outputs.docs_only }} steps: @@ -64,6 +86,18 @@ jobs: echo "Changed files in ${range}:" printf '%s\n' "$files" + # FAIL-SAFE on an empty diff. docs_only is initialised true and only + # ever cleared inside the loop below, so an empty $files would classify + # as docs-only and skip the whole suite — and `ci-gate` trusts exactly + # that value when it decides a `skipped` job is acceptable. An empty + # range is not evidence that only prose changed; it is the absence of + # evidence (a force-push can collapse the three-dot range), so run + # everything. + if [ -z "$(printf '%s' "$files" | tr -d '[:space:]')" ]; then + echo "empty diff for ${range}; refusing to classify as docs-only." + echo "docs_only=false" >> "$GITHUB_OUTPUT"; exit 0 + fi + docs_only=true while IFS= read -r f; do [ -z "$f" ] && continue @@ -103,6 +137,7 @@ jobs: gitleaks: name: Secret scan (gitleaks) runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -132,6 +167,95 @@ jobs: # allowlist in .gitleaks.toml is just local-run robustness. run: gitleaks dir . --redact --exit-code 1 + actions: + # Lint the workflow files THEMSELVES. This repo carries ~3.1k lines of + # workflow YAML that decide what every other gate on this list even runs, + # and until now nothing checked them. The overlap with what already runs is + # narrow and worth stating: Semgrep's `p/github-actions` pack meets this on + # exactly ONE axis (actions referenced by a mutable tag) and CodeQL's + # `actions` language on taint into a checkout. Neither parses `${{ }}` + # expressions and neither shellchecks a `run:` block, which is where + # actionlint earns its place — expression syntax and type errors, undefined + # contexts, invalid `needs:` / `runs-on:` / cron, deprecated syntax, plus + # shellcheck over the bash in every `run:`. + # + # The tree was at ZERO findings when this gate went in, so a failure here is + # a regression rather than a backlog to wade through. Getting there took two + # real fixes and one waiver: `run: "$GITHUB_WORKSPACE/..."` in both lanes + # passed the path to the shell UNQUOTED (the YAML quotes are consumed by the + # YAML parser, so the shell never saw them, despite the comment above the + # line showing quoting was intended), and the SC2153 on `RESULTS` is an + # `env:`-supplied name shellcheck cannot see, waived in-line with a reason. + # + # Deliberately NOT gated on the docs-only classifier. A change under + # .github/workflows/ can never be docs-only by that allowlist anyway, but a + # broken workflow is the one failure that can disarm every OTHER gate here, + # so it runs unconditionally rather than depending on a classifier to let it. + name: Workflow + shell lint (actionlint, shellcheck) + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Install actionlint + # Pinned release + checksum, the same contract as the gitleaks step + # above: an upstream re-tag cannot silently change what this gate + # enforces. Checksum verified against the release's own + # actionlint_1.7.7_checksums.txt. + env: + ACTIONLINT_VERSION: '1.7.7' + ACTIONLINT_SHA256: '023070a287cd8cccd71515fedc843f1985bf96c436b7effaecce67290e7e0757' + run: | + set -euo pipefail + tarball="actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz" + curl -sSL -o "$tarball" \ + "https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/${tarball}" + echo "${ACTIONLINT_SHA256} ${tarball}" | sha256sum -c - + tar -xzf "$tarball" actionlint + sudo install -m 0755 actionlint /usr/local/bin/actionlint + actionlint --version + + - name: Run actionlint + # shellcheck ships preinstalled on the ubuntu-latest image, so actionlint + # finds it and the `run:` blocks get checked too. ASSERT that rather than + # assume it: a silently absent shellcheck drops half this job's coverage + # while the job still reports green — the green-but-vacuous outcome this + # repo keeps writing post-mortems about (see docs/SCANNING.md). + run: | + set -euo pipefail + command -v shellcheck >/dev/null || { + echo "::error::shellcheck is missing from the runner image; actionlint would silently skip every run: block." + exit 1 + } + shellcheck --version | head -2 + actionlint -color + + - name: Lint shell scripts (shellcheck) + # The 18 tracked *.sh files are ~6.2k lines, and they are the DEPLOY + # PATH, not helpers: scripts/update.sh, bootstrap.sh and doctor.sh are + # what `fleet update` / `fleet bootstrap` actually run on an operator's + # box. Go has golangci-lint, the web tier has oxlint + tsc, Python has + # ruff — bash had nothing, which is the same gap docs/SCANNING.md used to + # justify adding ruff, one language over. Several scripts already carry + # hand-written `# shellcheck` directives, so it was being run by hand; + # nothing made that reproducible. + # + # -S warning, and the tree is CLEAN at that level: the backlog was three + # findings total (two SC2034, one SC2148), all fixed or annotated with a + # reason in the same change that added this gate. Gating over an unfixed + # backlog is how a gate becomes something people learn to ignore, so the + # level was chosen by measuring, not by taste. The info tier (11 findings, + # mostly style) stays off. + run: | + set -euo pipefail + mapfile -t sh_files < <(git ls-files '*.sh') + printf 'shellchecking %d files\n' "${#sh_files[@]}" + # A vacuous pass is the failure mode this repo keeps writing up: if the + # glob ever matches nothing, say so instead of reporting green. + [ "${#sh_files[@]}" -gt 0 ] || { echo "::error::no shell scripts matched"; exit 1; } + shellcheck -S warning "${sh_files[@]}" + helm: # Lint + render the fleet Helm chart (#989) so a values/template drift # fails here, not at an operator's install. Fast (<15s) and not gated on @@ -139,6 +263,7 @@ jobs: # ships preinstalled on the ubuntu-latest runner image. name: Helm chart lint (#989) runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -168,6 +293,7 @@ jobs: migrations: name: Migration DDL lint (#256) runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -189,6 +315,7 @@ jobs: go: name: Go build / vet / lint / test runs-on: ubuntu-latest + timeout-minutes: 45 needs: changes # Skipped for a docs-only change; the `CI gate` job treats a skip as a pass. if: ${{ needs.changes.outputs.docs_only != 'true' }} @@ -236,20 +363,35 @@ jobs: # The runner ships client 16; the postgres service is server 18, and # pg_dump refuses to dump a newer server ("server version mismatch"). Pull # the matching client from PGDG so the backup/restore round-trip test - # (cmd/fleet-admin) runs for real. Best-effort: if the repo is unreachable - # the step still succeeds and the test skips on the version mismatch (it is - # written to skip, never fail, when client and server majors disagree). + # (cmd/fleet-admin) runs for real. + # + # NOT best-effort, and that is a deliberate change. This step used to end + # in `|| echo "... will skip on version mismatch"`, which meant an + # unreachable PGDG left the runner's client 16 in place, and + # backup_test.go's `t.Skipf("pg_dump major %d != server major %d")` + # (internal/admincli/backup_test.go:155) turned the ONLY coverage of + # `fleet backup` / `fleet restore` off — behind a green `CI gate`, the + # single required check on main. That is the same green-but-vacuous shape + # this repo keeps writing post-mortems about, and the same one `e2e-live` + # already refuses by grepping its own log for `--- SKIP`. A PGDG outage + # is now a red build with an obvious cause rather than a silent hole. run: | - set -x - (sudo install -d /usr/share/postgresql-common/pgdg \ - && sudo curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \ - -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc \ - && echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \ - | sudo tee /etc/apt/sources.list.d/pgdg.list >/dev/null \ - && sudo apt-get update -qq \ - && sudo apt-get install -y -qq postgresql-client-18) \ - || echo "could not install postgresql-client-18; round-trip test will skip on version mismatch" - pg_dump --version || true + set -euxo pipefail + sudo install -d /usr/share/postgresql-common/pgdg + sudo curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \ + -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc + echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \ + | sudo tee /etc/apt/sources.list.d/pgdg.list >/dev/null + sudo apt-get update -qq + sudo apt-get install -y -qq postgresql-client-18 + # Assert the major, don't just print it: the test self-skips on a + # mismatch, so an install that "succeeded" with the wrong major is + # exactly as invisible as one that failed. + got="$(pg_dump --version | grep -oE '[0-9]+' | head -1)" + [ "$got" = "18" ] || { + echo "::error::pg_dump major ${got} != server major 18 — the backup/restore round-trip test would self-skip." + exit 1 + } - name: Create test databases run: | @@ -336,10 +478,12 @@ jobs: # Write the per-package (and per-function) coverage table to the Actions # job summary, visible in the UI without downloading any artifact (#249). run: | - echo "## Go coverage by package" >> "$GITHUB_STEP_SUMMARY" - echo '```' >> "$GITHUB_STEP_SUMMARY" - go tool cover -func=coverage.out >> "$GITHUB_STEP_SUMMARY" - echo '```' >> "$GITHUB_STEP_SUMMARY" + { + echo "## Go coverage by package" + echo '```' + go tool cover -func=coverage.out + echo '```' + } >> "$GITHUB_STEP_SUMMARY" - name: govulncheck (dependency CVEs) # Call-graph-aware scan of the (broad, fast-moving) dependency tree for @@ -379,6 +523,7 @@ jobs: python: name: Python lint (ruff) runs-on: ubuntu-latest + timeout-minutes: 10 needs: changes # Skipped for a docs-only change; the `CI gate` job treats a skip as a pass. if: ${{ needs.changes.outputs.docs_only != 'true' }} @@ -444,6 +589,7 @@ jobs: web: name: Web lint / test / build runs-on: ubuntu-latest + timeout-minutes: 20 needs: changes # Skipped for a docs-only change; the `CI gate` job treats a skip as a pass. if: ${{ needs.changes.outputs.docs_only != 'true' }} @@ -488,7 +634,7 @@ jobs: # registry flake skips with a notice (the audit above is the CVE gate). # Absolute path: this job's default working-directory is web/, which is # exactly how run 32579378165 caught the repo-relative form (exit 127). - run: "$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh" + run: '"$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh"' - name: Install dependencies run: npm ci @@ -514,6 +660,7 @@ jobs: playwright: name: Web e2e (Playwright, mocked) runs-on: ubuntu-latest + timeout-minutes: 30 needs: changes # Skipped for a docs-only change; the `CI gate` job treats a skip as a pass. if: ${{ needs.changes.outputs.docs_only != 'true' }} @@ -576,6 +723,7 @@ jobs: e2e-live: name: Web e2e (Playwright, live — real backend + sandbox) runs-on: ubuntu-latest + timeout-minutes: 60 needs: changes # Skipped for a docs-only change; the `CI gate` job treats a skip as a pass. if: ${{ needs.changes.outputs.docs_only != 'true' }} @@ -760,6 +908,7 @@ jobs: grype-scan: name: Container image vulnerability scan (Grype) runs-on: ubuntu-latest + timeout-minutes: 35 needs: [e2e-live] # runs after e2e-live succeeds (which already builds the image) # When e2e-live is skipped for a docs-only change, this job is skipped too # (a job that `needs` a skipped job is itself skipped); the `CI gate` job @@ -774,6 +923,11 @@ jobs: steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + # This job holds a write scope. Nothing in it pushes or does an + # authenticated fetch, so do not leave the token in .git/config while + # a PR-controlled Containerfile goes through `podman build`. + persist-credentials: false - name: Install Grype # Pin a specific Grype release and verify its checksum, exactly as the @@ -867,8 +1021,9 @@ jobs: # allowed (docs-only), but any failure or cancellation fails the gate. name: CI gate if: ${{ always() }} - needs: [changes, gitleaks, migrations, helm, go, python, codeql, semgrep, web, playwright, e2e-live, grype-scan] + needs: [changes, gitleaks, actions, migrations, helm, go, python, codeql, semgrep, web, playwright, e2e-live, grype-scan] runs-on: ubuntu-latest + timeout-minutes: 5 steps: - name: Require all upstream jobs to have succeeded or been skipped env: diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 1b4f8659..6ea4ebf1 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -142,15 +142,6 @@ jobs: # 55 findings. Do not re-derive a tree-wide claim from a PR run; the # full-tree numbers come from push/schedule runs. See # docs/adr/0048-codeql-severity-gating.md. - # Go additionally runs the standard pack's AlertSuppression query so - # an in-source `// codeql[rule-id]` comment is honored end-to-end: - # the analyze step stamps the SARIF result's `suppressions` array, - # the gate below classifies it into the ACCEPTED tier, and code - # scanning closes the Security-tab alert — the one waiver mechanism - # that keeps CI, the SARIF, and the Security tab telling the same - # story (an accepted-findings.json entry, by contrast, is invisible - # to the Security tab). The pack is resolved from the bundled - # standard library — no network pull, no `packages: read`. # NOTE on in-source `// codeql[rule-id]` suppressions: they do NOT # work with this pipeline, and this was measured, not assumed. Three # forms were tried on PR #1249 (the `packs:` input, `packs:` with @@ -304,7 +295,7 @@ jobs: else ($rows | sort_by(.rule, .file, .line) | .[] | fmt) end); tier("BLOCKING — High band (security-severity >= 7.0), not waived"; .blocking), "", - tier("ACCEPTED — High band, waived in codeql-accepted-findings.json or in-source"; .accepted), + tier("ACCEPTED — High band, waived in codeql-accepted-findings.json"; .accepted), "", tier("ADVISORY — below the High band; triage in the Security tab"; .advisory), "", @@ -343,10 +334,11 @@ jobs: # go/log-injection at 6.1 included, so banding on it would block all # 23 log-injection findings and reproduce the deadlock this replaced. # - a rule with no security-severity falls back to level error/warning. - # - a (rule, file) pair in .github/codeql-accepted-findings.json, or an - # in-source `// codeql[rule-id]` comment, moves a High-band finding - # to ACCEPTED. The register is per-FILE, so the rule stays live - # everywhere else. + # - a (rule, file) pair in .github/codeql-accepted-findings.json moves + # a High-band finding to ACCEPTED. The register is per-FILE, so the + # rule stays live everywhere else. It is the ONLY waiver route: the + # in-source `// codeql[rule-id]` form does not work here (see the + # measured note by the matrix above). # # WHY NOT "any finding" — that was tried in #1246 and it deadlocked the # repo. See docs/adr/0048-codeql-severity-gating.md: the zero it was armed @@ -369,9 +361,11 @@ jobs: jq -r '.blocking[] | " \(.rule) \(.file):\(.line) (security-severity \(.sev))"' "$CLASSIFIED" echo "" echo "Fix it. If it is a false positive the honest options are a code" - echo "change that removes the sink, an in-source // codeql[rule-id]" - echo "comment, or an entry in .github/codeql-accepted-findings.json" - echo "with a written reason. Note that dismissing the alert in the" + echo "change that removes the sink, or an entry in" + echo ".github/codeql-accepted-findings.json with a written reason." + echo "An in-source // codeql[rule-id] comment does NOT work with this" + echo "pipeline — measured on #1249; see the note by the matrix above." + echo "Note that dismissing the alert in the" echo "Security tab will NOT turn this check green: this step reads the" echo "run's own SARIF and never consults the code-scanning API." exit 1 @@ -391,6 +385,7 @@ jobs: if: always() needs: [analyze] runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Fail if any CodeQL analysis did not succeed # RESULTS via env, not interpolated into the run: block. The values are @@ -401,6 +396,9 @@ jobs: env: RESULTS: ${{ join(needs.*.result, ' ') }} run: | + # RESULTS is supplied by the `env:` block above; shellcheck cannot see + # that and reads the all-caps name as a typo for the local `results`. + # shellcheck disable=SC2153 results="$RESULTS" echo "job results: $results" for r in $results; do diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index 20c9a34b..34a623ac 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -53,6 +53,7 @@ jobs: go: name: Go compile / vet / lint / test (fast) runs-on: ubuntu-latest + timeout-minutes: 20 # Postgres service mirroring ci.yml's full-lane Go job (#723), so the # DB-gated suites — internal/store, internal/httpapi, internal/sched/*, @@ -133,11 +134,18 @@ jobs: - name: Test (no -race; DB suites run against the Postgres service, #723) # Same tag as make test / ci.yml — internal/sandbox's host tests only # compile with it. - run: go test -p 1 -tags fleet_host_executor ./... + # -count=1 defeats the build cache, matching ci.yml's three + # invocations. setup-go restores ~/.cache/go-build, which stores test + # RESULTS, and Go's cache key covers the binary, argv, env and files + # opened — it cannot see the Postgres SERVICE CONTAINER these suites talk + # to. Without it a DB-backed suite can print "ok (cached)" against a + # server it never contacted, which is a green check over an unrun test. + run: go test -p 1 -tags fleet_host_executor ./... -count=1 python: name: Python lint (ruff) runs-on: ubuntu-latest + timeout-minutes: 10 # Mirrors ci.yml's `python` job exactly, for the same reason the web lane was # added to this file: a change should not first be checked at the dev->main # promotion. ruff takes about a second, so there is no speed argument for @@ -193,6 +201,7 @@ jobs: web: name: Web lint / test / build (fast) runs-on: ubuntu-latest + timeout-minutes: 20 # The lane dev was missing entirely. Nothing here ran web/ at all, so a # web-only change — every one of the npm Dependabot PRs — reached dev with no # build behind it. Commands mirror ci.yml's `web` job exactly so the fast lane @@ -230,7 +239,7 @@ jobs: # registry flake skips with a notice (the audit above is the CVE gate). # Absolute path: this job's default working-directory is web/, which is # exactly how run 32579378165 caught the repo-relative form (exit 127). - run: "$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh" + run: '"$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh"' - name: Install dependencies run: npm ci @@ -260,6 +269,7 @@ jobs: # renders. helm is preinstalled on the runner image. name: Helm chart lint runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -287,6 +297,7 @@ jobs: migrations: name: Migration DDL lint runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -310,6 +321,7 @@ jobs: gitleaks: name: Secret scan (gitleaks) runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -333,13 +345,103 @@ jobs: - name: Run gitleaks run: gitleaks dir . --redact --exit-code 1 + actions: + # Lint the workflow files THEMSELVES. This repo carries ~3.1k lines of + # workflow YAML that decide what every other gate on this list even runs, + # and until now nothing checked them. The overlap with what already runs is + # narrow and worth stating: Semgrep's `p/github-actions` pack meets this on + # exactly ONE axis (actions referenced by a mutable tag) and CodeQL's + # `actions` language on taint into a checkout. Neither parses `${{ }}` + # expressions and neither shellchecks a `run:` block, which is where + # actionlint earns its place — expression syntax and type errors, undefined + # contexts, invalid `needs:` / `runs-on:` / cron, deprecated syntax, plus + # shellcheck over the bash in every `run:`. + # + # The tree was at ZERO findings when this gate went in, so a failure here is + # a regression rather than a backlog to wade through. Getting there took two + # real fixes and one waiver: `run: "$GITHUB_WORKSPACE/..."` in both lanes + # passed the path to the shell UNQUOTED (the YAML quotes are consumed by the + # YAML parser, so the shell never saw them, despite the comment above the + # line showing quoting was intended), and the SC2153 on `RESULTS` is an + # `env:`-supplied name shellcheck cannot see, waived in-line with a reason. + # + # Deliberately NOT gated on the docs-only classifier. A change under + # .github/workflows/ can never be docs-only by that allowlist anyway, but a + # broken workflow is the one failure that can disarm every OTHER gate here, + # so it runs unconditionally rather than depending on a classifier to let it. + name: Workflow + shell lint (actionlint, shellcheck) + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Install actionlint + # Pinned release + checksum, the same contract as the gitleaks step + # above: an upstream re-tag cannot silently change what this gate + # enforces. Checksum verified against the release's own + # actionlint_1.7.7_checksums.txt. + env: + ACTIONLINT_VERSION: '1.7.7' + ACTIONLINT_SHA256: '023070a287cd8cccd71515fedc843f1985bf96c436b7effaecce67290e7e0757' + run: | + set -euo pipefail + tarball="actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz" + curl -sSL -o "$tarball" \ + "https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/${tarball}" + echo "${ACTIONLINT_SHA256} ${tarball}" | sha256sum -c - + tar -xzf "$tarball" actionlint + sudo install -m 0755 actionlint /usr/local/bin/actionlint + actionlint --version + + - name: Run actionlint + # shellcheck ships preinstalled on the ubuntu-latest image, so actionlint + # finds it and the `run:` blocks get checked too. ASSERT that rather than + # assume it: a silently absent shellcheck drops half this job's coverage + # while the job still reports green — the green-but-vacuous outcome this + # repo keeps writing post-mortems about (see docs/SCANNING.md). + run: | + set -euo pipefail + command -v shellcheck >/dev/null || { + echo "::error::shellcheck is missing from the runner image; actionlint would silently skip every run: block." + exit 1 + } + shellcheck --version | head -2 + actionlint -color + + - name: Lint shell scripts (shellcheck) + # The 18 tracked *.sh files are ~6.2k lines, and they are the DEPLOY + # PATH, not helpers: scripts/update.sh, bootstrap.sh and doctor.sh are + # what `fleet update` / `fleet bootstrap` actually run on an operator's + # box. Go has golangci-lint, the web tier has oxlint + tsc, Python has + # ruff — bash had nothing, which is the same gap docs/SCANNING.md used to + # justify adding ruff, one language over. Several scripts already carry + # hand-written `# shellcheck` directives, so it was being run by hand; + # nothing made that reproducible. + # + # -S warning, and the tree is CLEAN at that level: the backlog was three + # findings total (two SC2034, one SC2148), all fixed or annotated with a + # reason in the same change that added this gate. Gating over an unfixed + # backlog is how a gate becomes something people learn to ignore, so the + # level was chosen by measuring, not by taste. The info tier (11 findings, + # mostly style) stays off. + run: | + set -euo pipefail + mapfile -t sh_files < <(git ls-files '*.sh') + printf 'shellchecking %d files\n' "${#sh_files[@]}" + # A vacuous pass is the failure mode this repo keeps writing up: if the + # glob ever matches nothing, say so instead of reporting green. + [ "${#sh_files[@]}" -gt 0 ] || { echo "::error::no shell scripts matched"; exit 1; } + shellcheck -S warning "${sh_files[@]}" + dev-gate: name: Dev gate # Aggregate check for branch protection: passes only when every fast-lane # job succeeded (mirrors ci.yml's `CI gate`). if: always() - needs: [go, python, codeql, semgrep, web, migrations, gitleaks, helm] + needs: [go, python, codeql, semgrep, web, migrations, gitleaks, helm, actions] runs-on: ubuntu-latest + timeout-minutes: 5 steps: - name: Fail if any fast-lane job failed # RESULTS via env, not interpolated into the run: block. The values are @@ -350,6 +452,9 @@ jobs: env: RESULTS: ${{ join(needs.*.result, ' ') }} run: | + # RESULTS is supplied by the `env:` block above; shellcheck cannot see + # that and reads the all-caps name as a typo for the local `results`. + # shellcheck disable=SC2153 results="$RESULTS" echo "job results: $results" for r in $results; do diff --git a/.github/workflows/e2e-canary.yml b/.github/workflows/e2e-canary.yml index aa5aac1c..24da2d67 100644 --- a/.github/workflows/e2e-canary.yml +++ b/.github/workflows/e2e-canary.yml @@ -31,6 +31,7 @@ jobs: guard: name: Check for OPENROUTER_API_KEY secret runs-on: ubuntu-latest + timeout-minutes: 5 outputs: has_key: ${{ steps.check.outputs.has_key }} steps: @@ -51,6 +52,7 @@ jobs: needs: guard if: ${{ needs.guard.outputs.has_key == 'true' }} runs-on: ubuntu-latest + timeout-minutes: 30 services: postgres: diff --git a/.github/workflows/govulncheck-scheduled.yml b/.github/workflows/govulncheck-scheduled.yml index d29b3f06..618e1e5c 100644 --- a/.github/workflows/govulncheck-scheduled.yml +++ b/.github/workflows/govulncheck-scheduled.yml @@ -36,20 +36,31 @@ on: - cron: '0 8 * * *' workflow_dispatch: # Manual trigger (e.g. to confirm a fix cleared an advisory) +# NOTE: `issues: write` deliberately does NOT live at workflow scope. These +# lanes have one long job that runs third-party code with a full, +# default-branch token — `govulncheck@latest` resolved at run time, and a +# `podman build` pulling ~400 RPMs from Fedora mirrors. A workflow-level scope +# is live for every step of that job, so the scan ran next to a token that +# could open issues and write code-scanning alerts. The alarm needs no source +# tree and no scan output, so it is its own job holding `issues: write` alone — +# the same shape scan-cron-alarm.yml already uses for CodeQL and Semgrep. permissions: contents: read security-events: write # required to upload SARIF to the Security tab - issues: write # the failure-alarm step files an issue when a scheduled run breaks jobs: govulncheck-scheduled: name: Daily Go vulnerability scan runs-on: ubuntu-latest + timeout-minutes: 20 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: + # Nothing here pushes or does an authenticated fetch, so do not leave + # the job's token in .git/config for the third-party code below. + persist-credentials: false ref: main # always scan the tip of main, not a PR branch - name: Set up Go @@ -100,6 +111,19 @@ jobs: sarif_file: 'govulncheck.sarif' category: 'govulncheck-scheduled' + + alarm: + # Split out of the scan job above so `issues: write` is the ONLY scope in + # play here, and it is held by a job that checks out nothing and runs no + # third-party code — it only calls `gh`. + name: File an issue so a red cron cannot rot silently + needs: [govulncheck-scheduled] + if: ${{ failure() && github.event_name == 'schedule' }} + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + issues: write + steps: - name: File an issue so a red cron cannot rot silently # A scheduled failure has no PR attached, so nothing surfaces it — the # exact rot pattern that let the CodeQL toolchain break sit red for @@ -107,7 +131,6 @@ jobs: # re-failures. schedule-only: a red manual dispatch has a human watching. # This step body is duplicated across the four scheduled scan lanes # (codeql, semgrep, govulncheck, grype) — keep them in sync. - if: ${{ failure() && github.event_name == 'schedule' }} env: GH_TOKEN: ${{ github.token }} WORKFLOW_NAME: ${{ github.workflow }} diff --git a/.github/workflows/grype-scheduled.yml b/.github/workflows/grype-scheduled.yml index d48f4d6e..3b08f984 100644 --- a/.github/workflows/grype-scheduled.yml +++ b/.github/workflows/grype-scheduled.yml @@ -19,15 +19,23 @@ on: - cron: '0 9 * * 1' # Every Monday at 09:00 UTC workflow_dispatch: # Allow a manual trigger (e.g. after a base-image bump) +# NOTE: `issues: write` deliberately does NOT live at workflow scope. These +# lanes have one long job that runs third-party code with a full, +# default-branch token — `govulncheck@latest` resolved at run time, and a +# `podman build` pulling ~400 RPMs from Fedora mirrors. A workflow-level scope +# is live for every step of that job, so the scan ran next to a token that +# could open issues and write code-scanning alerts. The alarm needs no source +# tree and no scan output, so it is its own job holding `issues: write` alone — +# the same shape scan-cron-alarm.yml already uses for CodeQL and Semgrep. permissions: contents: read security-events: write # required to upload SARIF to the Security tab - issues: write # the failure-alarm step files an issue when a scheduled run breaks jobs: grype-scheduled: name: Weekly container vulnerability scan runs-on: ubuntu-latest + timeout-minutes: 35 env: SANDBOX_IMAGE_LOCAL: localhost/fleet-sandbox:latest @@ -35,6 +43,9 @@ jobs: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: + # Nothing here pushes or does an authenticated fetch, so do not leave + # the job's token in .git/config for the third-party code below. + persist-credentials: false ref: main # always scan the tip of main, not a PR branch - name: Install Grype @@ -88,6 +99,19 @@ jobs: sarif_file: 'grype-results.sarif' category: 'grype-scheduled' + + alarm: + # Split out of the scan job above so `issues: write` is the ONLY scope in + # play here, and it is held by a job that checks out nothing and runs no + # third-party code — it only calls `gh`. + name: File an issue so a red cron cannot rot silently + needs: [grype-scheduled] + if: ${{ failure() && github.event_name == 'schedule' }} + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + issues: write + steps: - name: File an issue so a red cron cannot rot silently # A scheduled failure has no PR attached, so nothing surfaces it — the # exact rot pattern that let the CodeQL toolchain break sit red for @@ -95,7 +119,6 @@ jobs: # re-failures. schedule-only: a red manual dispatch has a human watching. # This step body is duplicated across the four scheduled scan lanes # (codeql, semgrep, govulncheck, grype) — keep them in sync. - if: ${{ failure() && github.event_name == 'schedule' }} env: GH_TOKEN: ${{ github.token }} WORKFLOW_NAME: ${{ github.workflow }} diff --git a/.github/workflows/publish-sandbox-image.yml b/.github/workflows/publish-sandbox-image.yml index 790221cf..7ee0f477 100644 --- a/.github/workflows/publish-sandbox-image.yml +++ b/.github/workflows/publish-sandbox-image.yml @@ -204,6 +204,7 @@ jobs: publish: name: Build + push sandbox image runs-on: ubuntu-latest + timeout-minutes: 60 outputs: image_ref: ${{ steps.push.outputs.image_ref }} image_digest: ${{ steps.push.outputs.image_digest }} @@ -216,6 +217,11 @@ jobs: steps: - name: Checkout caller repo uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + # This job holds a write scope. Nothing in it pushes or does an + # authenticated fetch, so do not leave the token in .git/config while + # a PR-controlled Containerfile goes through `podman build`. + persist-credentials: false # The build script lives in fleet core; a client-config caller doesn't # have it. Checking it out into a subdir keeps ONE canonical builder diff --git a/.github/workflows/scan-cron-alarm.yml b/.github/workflows/scan-cron-alarm.yml index 07b62a4e..c55c77c3 100644 --- a/.github/workflows/scan-cron-alarm.yml +++ b/.github/workflows/scan-cron-alarm.yml @@ -55,6 +55,7 @@ jobs: github.event.workflow_run.conclusion != 'success' && github.event.workflow_run.conclusion != 'skipped' runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: File or update the alarm issue # Body mirrors the in-job alarm steps in govulncheck-scheduled.yml and diff --git a/.github/workflows/screenshots.yml b/.github/workflows/screenshots.yml index 729d4184..1fa1d3d7 100644 --- a/.github/workflows/screenshots.yml +++ b/.github/workflows/screenshots.yml @@ -48,6 +48,7 @@ jobs: screenshots: name: Generate GUI screenshots runs-on: ubuntu-latest + timeout-minutes: 30 defaults: run: working-directory: web diff --git a/.github/workflows/semgrep.yml b/.github/workflows/semgrep.yml index 50c0a8f5..043618e8 100644 --- a/.github/workflows/semgrep.yml +++ b/.github/workflows/semgrep.yml @@ -121,13 +121,20 @@ jobs: # The JSON is written first and the exit status captured, so the # summary step below still runs and still says WHAT failed — a gate # that fails without printing the finding is a gate nobody can act on. - semgrep scan \ + # + # The `if` is load-bearing. `set -uo pipefail` does NOT clear the `-e` + # the runner supplies (GitHub's default shell for a `run:` block is + # `bash -e {0}`), so a bare `semgrep ...` followed by `status=$?` + # aborts the step ON the semgrep line and never reaches the capture — + # the deferral this block describes never actually happened. Capturing + # through an `if` is exempt from `-e` and makes the comment true. + if semgrep scan \ --config p/github-actions \ --config p/golang \ --config p/javascript \ --config p/python \ --metrics=off --error --json -o semgrep.json --quiet - status=$? + then status=0; else status=$?; fi if [ ! -s semgrep.json ]; then echo "semgrep produced no JSON — treating as a scan failure" >&2 exit 1 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0d16cd11..37100417 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -28,7 +28,7 @@ removed.) ## Prerequisites -- **Go** — the version pinned in `go.mod` (currently 1.26.x). +- **Go** — the version pinned in `go.mod` (currently 1.27.x). - **Node.js** — the major in [`web/.nvmrc`](web/.nvmrc) (currently 24) — and npm, for the `web/` app. - **Podman** (rootless) for the execution sandbox — only needed to run the sandbox-backed tests/e2e locally; most unit tests self-skip when podman is @@ -114,10 +114,11 @@ Every pull request must be green before merge. CI runs: python / javascript-typescript / actions) fails on an unwaived finding in the **High band** — `security-severity >= 7.0`, or level `error`/`warning` for a rule that publishes no security-severity — with lower-severity findings - reported as advisory. A false positive is waived either by an in-source - `// codeql[rule-id]` comment or by an entry in - `.github/codeql-accepted-findings.json` **with a written reason**; both are - reviewable in the diff, and fixing the code is always preferred. See + reported as advisory. A false positive is waived by an entry in + `.github/codeql-accepted-findings.json` **with a written reason** — that is the + only waiver route that works here; an in-source `// codeql[rule-id]` comment + does not (measured on #1249). The register entry is reviewable in the diff, and + fixing the code is always preferred. See [`docs/SCANNING.md`](docs/SCANNING.md), [`docs/CODEQL.md`](docs/CODEQL.md) and [ADR-0048](docs/adr/0048-codeql-severity-gating.md). - **Dependency CVEs** — `govulncheck` for the Go module, and diff --git a/Makefile b/Makefile index 6686d860..e1f2318c 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: all build compile bins fleet-bench install test test-race test-cover lint lint-go lint-python lint-migrations fmt tidy clean help \ +.PHONY: all build compile bins fleet-bench install test test-race test-cover lint lint-go lint-python lint-migrations lint-actions fmt tidy clean help \ govulncheck ci-go ci-web ci-e2e-mocked ci-local # GOTOOLCHAIN=auto — the operator does NOT have to hand-install the pinned Go. @@ -122,7 +122,7 @@ test-cover: go test -coverprofile=coverage.out -covermode=atomic -p 1 -tags fleet_host_executor ./... @go tool cover -func=coverage.out | tail -1 -lint: lint-go lint-python lint-migrations +lint: lint-go lint-python lint-migrations lint-actions lint-go: golangci-lint run @@ -150,6 +150,33 @@ lint-python: lint-migrations: scripts/check-migrations.sh +# lint-actions: actionlint over .github/workflows/*.yml — the ~3.1k lines of +# workflow YAML that decide what every other gate on this list even runs. +# +# It is the one checker here whose subject is CI itself, and it covers ground +# no other lane does: expression syntax and type errors inside ${{ }}, unknown +# contexts, invalid `needs:`/`runs-on:`/cron, deprecated action syntax, and — +# via shellcheck — the bash inside every `run:` block. Semgrep's +# p/github-actions pack overlaps on ONE axis only (mutable action tags); it +# does not parse expressions and does not shellcheck run blocks. +# +# Skips LOUDLY when actionlint is absent, same contract as lint-python: CI +# enforces the gate regardless (ci.yml + dev-ci.yml `actions` job), so a local +# skip is a choice, not a surprise. +lint-actions: + @if command -v actionlint >/dev/null 2>&1; then \ + actionlint; \ + else \ + echo "actionlint not installed — SKIPPING the workflow lint (CI still enforces it)."; \ + echo " install: go install github.com/rhysd/actionlint/cmd/actionlint@v1.7.7"; \ + fi + @if command -v shellcheck >/dev/null 2>&1; then \ + git ls-files '*.sh' | xargs shellcheck -S warning; \ + else \ + echo "shellcheck not installed — SKIPPING the shell lint (CI still enforces it)."; \ + echo " install: dnf install ShellCheck # or: apt-get install shellcheck"; \ + fi + fmt: gofmt -w . @@ -219,9 +246,19 @@ ci-go: compile $(MAKE) test-race $(MAKE) govulncheck -# The Web CI job, verbatim, run from web/: npm ci → lint → vitest → build. +# The Web CI job, verbatim: both npm audits → the override canary → npm ci → +# lint → typecheck → vitest → build. It said "verbatim" while running four of +# those eight, which is the wrong half of a CI==local promise: the two +# `npm audit` CVE gates, scripts/check-npm-overrides.sh, and the explicit +# `npm run typecheck` were all missing, so a contributor could go green locally +# and red on the PR. The typecheck matters most of the four — `next build` +# type-checks too, but it runs LAST, so dropping the explicit gate is what turns +# a one-line type error into a multi-minute discovery. ci-web: - cd web && npm ci && npm run lint && npx vitest run && npm run build + cd web && npm audit --audit-level=low + cd scripts/rampart-service && npm audit --audit-level=low + scripts/check-npm-overrides.sh + cd web && npm ci && npm run lint && npm run typecheck && npx vitest run && npm run build # The mocked Playwright CI job, run from web/. Assumes browsers are installed # (`cd web && npx playwright install --with-deps chromium`); CI installs them in diff --git a/docs/CODEQL.md b/docs/CODEQL.md index a085d066..e298dbc2 100644 --- a/docs/CODEQL.md +++ b/docs/CODEQL.md @@ -508,10 +508,14 @@ Per-**file** is the whole point of preferring it to a `query-filters` exclude. A `exclude: {id: go/request-forgery}` switches a security-severity 9.1 query off for the entire repository; a register entry waives it in `internal/tools/web_fetch.go` and `internal/mcpoauth/discovery.go` and leaves the -query live everywhere else, including elsewhere in those same packages. An -in-source `// codeql[rule-id]` comment is the second waiver route — CodeQL emits -it as a `suppressions` array on the result; the comment must sit on its own line -and covers the line immediately below it. +query live everywhere else, including elsewhere in those same packages. + +The register is also the **only** waiver route. An in-source `// codeql[rule-id]` +comment is the mechanism CodeQL documents, and it does **not** work with this +pipeline — measured on PR #1249: three forms were tried (the `packs:` input, `packs:` with the additive `+` prefix, and an inline `config:` combining security-extended with codeql/go-queries' `AlertSuppression.ql`) and in every case the uploaded SARIF carried no `suppressions` on the annotated result, the gate kept classifying the waiver from the register, +and the Security-tab alert stayed open. The analyze action's interpret step is +not configurable enough to change that. A deliberately-waived alert is closed in +the Security tab by a one-time human dismissal, which persists across analyses. Of the 55 findings run 527 surfaced, **four were reachable and were fixed in code**: an unsanitized `task.Prompt` in the task-create log (its update-path twin @@ -567,8 +571,8 @@ exits 0 and reports green"**, and that is no longer true. It was true of the The step reads the run's **own SARIF** and never consults the code-scanning API, which has one consequence worth stating plainly: **dismissing an alert in the Security tab does not turn this check green.** The honest routes are a code -change, an in-source `// codeql[rule-id]` comment, or a register entry with a -reason. +change or a register entry with a reason — an in-source `// codeql[rule-id]` +comment is not one of them here (see above). The second row remains available and nothing depends on it. fleet is a **public** repository, so code scanning merge protection is free (on private repos it needs @@ -645,7 +649,7 @@ What is fixed is the format: BLOCKING — High band (security-severity >= 7.0), not waived (): none -ACCEPTED — High band, waived in codeql-accepted-findings.json or in-source (): +ACCEPTED — High band, waived in codeql-accepted-findings.json (): [error] sec-sev=9.1 go/request-forgery internal/tools/web_fetch.go: [error] sec-sev=7.5 go/clear-text-logging cmd/fleet/main.go: ... diff --git a/docs/SCANNING.md b/docs/SCANNING.md index 397b3275..371eae23 100644 --- a/docs/SCANNING.md +++ b/docs/SCANNING.md @@ -143,9 +143,10 @@ per-**file**, not per-rule, and that is the whole point of preferring it to a `query-filters` exclude: excluding `go/request-forgery` would switch a security-severity 9.1 query off for the entire repository, whereas a register entry waives it in the two files that were read and leaves the query live -everywhere else. An in-source `// codeql[rule-id]` comment waives too (CodeQL -emits it as a `suppressions` array on the result; the comment must sit on its own -line and covers the line below it). Widening the register is a security decision +everywhere else. The register is the **only** waiver route that works here: an +in-source `// codeql[rule-id]` comment does **not** waive with this pipeline — +measured on PR #1249: three forms were tried (the `packs:` input, `packs:` with the additive `+` prefix, and an inline `config:` combining security-extended with codeql/go-queries' `AlertSuppression.ql`) and in every case the uploaded SARIF carried no `suppressions` on the annotated result. A deliberately-waived Security-tab alert is closed by a one-time +human dismissal there. Widening the register is a security decision that appears in the PR diff, and `scripts/check_codeql_register_test.go` fails `make test` on an entry naming a file that does not exist, a missing reason, or a register that `codeql.yml` has stopped referencing. @@ -302,7 +303,7 @@ move with every commit; what is fixed is the format: ### CodeQL findings — go BLOCKING — High band (security-severity >= 7.0), not waived (): none -ACCEPTED — High band, waived in codeql-accepted-findings.json or in-source (): +ACCEPTED — High band, waived in codeql-accepted-findings.json (): [error] sec-sev=9.1 go/request-forgery internal/tools/web_fetch.go: [error] sec-sev=7.5 go/clear-text-logging cmd/fleet/main.go: ... diff --git a/docs/adr/0048-codeql-severity-gating.md b/docs/adr/0048-codeql-severity-gating.md index ea16bd9e..e29fd0fe 100644 --- a/docs/adr/0048-codeql-severity-gating.md +++ b/docs/adr/0048-codeql-severity-gating.md @@ -88,17 +88,26 @@ block both. ## Decision -**CodeQL blocks on a finding that is (a) at SARIF level `error`/`warning`, or has -security-severity >= 7.0, and (b) is not waived.** Findings below that band are +**CodeQL blocks on a finding that is (a) in the High band and (b) is not +waived.** A finding is in the High band when its rule publishes a +`security-severity >= 7.0`; **only for a rule that publishes no +security-severity** does the band fall back to SARIF level `error`/`warning`. +The fallback is not an OR: level is deliberately *not* consulted for a rule that +does publish a security-severity, because nearly every CodeQL security query is +`@problem.severity error` — `go/log-injection` is `error` at 6.1 — so an OR +would block on all 23 log-injection findings and reproduce the deadlock this +ADR exists to undo. Findings below that band are printed and uploaded to the Security tab as advisory. Waivers come from two -places: +place: 1. `.github/codeql-accepted-findings.json` — a register of accepted `(rule, file)` pairs, each with a mandatory written reason. -2. An in-source `// codeql[rule-id]` comment, which CodeQL emits as a - `suppressions` array on the result. (Both `go` and `javascript` ship an - `AlertSuppression.ql`; the comment must sit on its own line and covers the - line immediately below it.) + +An in-source `// codeql[rule-id]` comment — the mechanism CodeQL documents, and +which an earlier revision of this ADR listed as a second route — does **not** +work with this pipeline. It was measured on PR #1249: three forms were tried (the `packs:` input, `packs:` with the additive `+` prefix, and an inline `config:` combining security-extended with codeql/go-queries' `AlertSuppression.ql`) and in every case the uploaded SARIF carried no `suppressions` on the annotated result. The register is therefore the sole +waiver route for the gate; a deliberately-waived alert is closed in the Security +tab by a one-time human dismissal. The register is **per-file, not per-rule**, and that is the whole point of preferring it to a `query-filters` exclude. A `query-filters: exclude: {id: diff --git a/scripts/check_permissions_test.go b/scripts/check_permissions_test.go new file mode 100644 index 00000000..4374e89a --- /dev/null +++ b/scripts/check_permissions_test.go @@ -0,0 +1,57 @@ +// Copyright (c) 2025 ElcanoTek +// SPDX-License-Identifier: MIT + +package scripts + +import ( + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// Every workflow in this repo declares a top-level `permissions:` block, and +// that is load-bearing rather than tidy: a workflow WITHOUT one inherits the +// repository default, which can be read-write for every scope. The whole +// least-privilege posture documented in docs/SCANNING.md rests on the property +// holding for all of them, and a new workflow added without the block is a +// silent, invisible regression — there is no failing check, just a job quietly +// holding more token than it asked for. +// +// So assert it, in the same spirit as check_action_pins_test.go and +// check_gate_needs_test.go: the invariants this repo cares about are tests, not +// review habits. The assertion is deliberately only that the block EXISTS — its +// contents are a per-workflow judgement call (`{}`, `contents: read`, or a +// scoped set), and pinning those here would fight every legitimate change. +var topLevelPermissionsRe = regexp.MustCompile(`(?m)^permissions:`) + +func TestWorkflowsDeclareTopLevelPermissions(t *testing.T) { + root := repoRoot(t) + dir := filepath.Join(root, ".github", "workflows") + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatalf("read %s: %v", dir, err) + } + + seen := 0 + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".yml") { + continue + } + seen++ + raw, err := os.ReadFile(filepath.Join(dir, e.Name())) + if err != nil { + t.Fatalf("read %s: %v", e.Name(), err) + } + if !topLevelPermissionsRe.Match(raw) { + t.Errorf("%s: no top-level `permissions:` block — the workflow inherits the "+ + "repository default token scopes. Declare one (`permissions: {}` if the "+ + "workflow needs nothing) and grant writes on the job that needs them.", e.Name()) + } + } + if seen == 0 { + t.Fatal("no workflow files found — this test would pass vacuously") + } + t.Logf("checked %d workflow files", seen) +} diff --git a/scripts/check_versions_test.go b/scripts/check_versions_test.go index 3de84930..fa560a26 100644 --- a/scripts/check_versions_test.go +++ b/scripts/check_versions_test.go @@ -173,6 +173,8 @@ func TestDuplicatedToolPinsAgree(t *testing.T) { {"GRYPE_SHA256", ".github/workflows/grype-scheduled.yml", regexp.MustCompile(`GRYPE_SHA256:\s*'([^']+)'`)}, {"GITLEAKS_VERSION", ".github/workflows/dev-ci.yml", regexp.MustCompile(`GITLEAKS_VERSION:\s*'([^']+)'`)}, {"RUFF_VERSION", ".github/workflows/dev-ci.yml", regexp.MustCompile(`RUFF_VERSION:\s*'([^']+)'`)}, + {"ACTIONLINT_VERSION", ".github/workflows/dev-ci.yml", regexp.MustCompile(`ACTIONLINT_VERSION:\s*'([^']+)'`)}, + {"ACTIONLINT_SHA256", ".github/workflows/dev-ci.yml", regexp.MustCompile(`ACTIONLINT_SHA256:\s*'([^']+)'`)}, {"golangci-lint version", ".github/workflows/dev-ci.yml", regexp.MustCompile(`golangci-lint-action@\S+[^\n]*\n\s*with:\s+(?:#[^\n]*\n\s+)*version:\s*(v[\d.]+)`)}, } { a := tc.re.FindStringSubmatch(ci) diff --git a/scripts/doctor.sh b/scripts/doctor.sh index 40814168..982ea4d6 100755 --- a/scripts/doctor.sh +++ b/scripts/doctor.sh @@ -52,6 +52,9 @@ SRC_DIR="${SRC_DIR:-$REPO_ROOT}" # scripts/bootstrap.sh (SERVICE_USER/SERVICE_HOME there). SERVICE_USER="${FLEET_SERVICE_USER:-fleet}" SERVICE_HOME="${FLEET_SERVICE_HOME:-/var/lib/fleet}" +# shellcheck disable=SC2034 # unread here on purpose: this block documents the +# service-account contract that deploy/fleet-web.service and bootstrap.sh must +# match, and dropping the name would remove the anchor the comment above names. WEB_USER="fleet-web" SERVICE_NAME="${FLEET_SERVICE_NAME:-fleet}" INSTALL_DIR="${FLEET_INSTALL_DIR:-/opt/fleet}" diff --git a/scripts/fleet-upgrade.sh b/scripts/fleet-upgrade.sh index e63fbeaf..409f4fd0 100755 --- a/scripts/fleet-upgrade.sh +++ b/scripts/fleet-upgrade.sh @@ -293,8 +293,7 @@ restart_web_tier() { fi # Read the resolved state back rather than trusting the restart's exit code — # a unit can accept the restart and then fail its ExecStart. - local i - for i in 1 2 3 4 5 6 7 8; do + for _ in 1 2 3 4 5 6 7 8; do if [[ "$(systemctl is-active fleet-web 2>/dev/null || true)" == "active" ]]; then WEB_TIER_UP="yes"; ok "fleet-web is active again (systemctl is-active)"; return 0 fi diff --git a/scripts/lib/node-version.sh b/scripts/lib/node-version.sh index bff01822..42cce621 100644 --- a/scripts/lib/node-version.sh +++ b/scripts/lib/node-version.sh @@ -1,3 +1,6 @@ +# shellcheck shell=bash +# ^ no shebang: this file is only ever sourced, never executed. The directive +# tells shellcheck which dialect to check it as (see the bash note below). # scripts/lib/node-version.sh — the ONE implementation of "which node?". # # Sourced by scripts/bootstrap.sh, scripts/doctor.sh and scripts/update.sh. It From 849c5bcb4ef9486e68398c871fb4376e5d0df892 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 00:24:11 +0000 Subject: [PATCH 28/34] Add PR/issue templates, a contributor-facing CI section, and correct the semgrep pinning claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The workflow engineering here was already ahead of most projects its size. Where it was behind them was the community-facing surface: a contributor had no prompt for the obligations CONTRIBUTING.md and AGENTS.md impose, and no account of why CI might be red through no fault of theirs. - .github/PULL_REQUEST_TEMPLATE.md — prose first (what/why, how you verified, scope and deviations), then a short checklist for the things that are currently a reviewer's job to remember: DCO sign-off, a CHANGELOG entry, a docs/.md note, and an ADR in the SAME PR when an invariant moves. - .github/ISSUE_TEMPLATE/ — bug and feature forms plus a config.yml that disables blank issues so the private security-disclosure link is unmissable. SECURITY.md and CODE_OF_CONDUCT.md both say "do not open a public issue for a vulnerability", and the New Issue button was offering a blank box with no such warning — the exact moment a reporter is most likely to get it wrong. The feature form asks up front whether the idea touches an invariant (ADR required) and whether it could ship as a client-config bundle instead of an engine change. - CONTRIBUTING.md gains "If CI is red and you don't recognise the failure". Three lanes depend on live external data and can redden an untouched tree — govulncheck's advisory DB, npm audit, and Semgrep's registry-fetched rules, which cannot be vendored for license reasons. All three are documented for maintainers in docs/SCANNING.md and were documented for contributors nowhere. Also notes that a first PR waits on maintainer approval before CI starts. - semgrep.yml claimed its `pip install semgrep==X` was pinned "like every other tool this repo installs in CI (gitleaks, grype, golangci-lint)". It is not: those are checksum-verified, while pip resolves semgrep's ~40-package dependency closure unverified, inside a job that sits in both merge gates. The comment now states the real guarantee and the mitigating fact (PyPI forbids re-uploading a version, so the exposure is a new malicious release rather than mutation of a pinned one). Signed-off-by: Claude --- .github/ISSUE_TEMPLATE/bug_report.yml | 44 ++++++++++++++++++++++ .github/ISSUE_TEMPLATE/config.yml | 13 +++++++ .github/ISSUE_TEMPLATE/feature_request.yml | 39 +++++++++++++++++++ .github/PULL_REQUEST_TEMPLATE.md | 34 +++++++++++++++++ .github/workflows/semgrep.yml | 18 +++++++-- CONTRIBUTING.md | 24 ++++++++++++ 6 files changed, 169 insertions(+), 3 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.yml create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/feature_request.yml create mode 100644 .github/PULL_REQUEST_TEMPLATE.md diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 00000000..fe89f4b2 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,44 @@ +name: Bug report +description: Something in fleet does not behave as documented. +labels: [bug] +body: + - type: markdown + attributes: + value: | + For a **security vulnerability**, stop and read + [SECURITY.md](https://github.com/ElcanoTek/fleet/blob/main/SECURITY.md) + instead — do not file it here. + - type: textarea + id: what-happened + attributes: + label: What happened + description: What you observed, and what you expected instead. + validations: {required: true} + - type: textarea + id: repro + attributes: + label: Steps to reproduce + description: The smallest sequence that shows the problem. + placeholder: | + 1. fleet serve with ... + 2. ... + 3. Observed: ... + validations: {required: true} + - type: textarea + id: version + attributes: + label: Version and environment + description: >- + Output of `fleet version` (or the commit you built from), your OS, and + whether Podman is running rootless. `fleet doctor` output is ideal. + render: shell + validations: {required: true} + - type: textarea + id: logs + attributes: + label: Relevant logs + description: >- + Redact before pasting. fleet brokers credentials host-side precisely so + they never reach a log, but check anyway. + render: shell + validations: {required: false} diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..f1677b7e --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,13 @@ +# Blank issues are disabled so the security contact link below is unmissable. +# SECURITY.md and CODE_OF_CONDUCT.md both say "do not open a public issue for a +# vulnerability", and until now the New Issue button offered a blank box with no +# such warning — the one moment a reporter is most likely to get it wrong. +blank_issues_enabled: false +contact_links: + - name: Report a security vulnerability (private) + url: https://github.com/ElcanoTek/fleet/blob/main/SECURITY.md + about: Please do NOT open a public issue. SECURITY.md has the private + disclosure process and the response SLA. + - name: Contributing guide + url: https://github.com/ElcanoTek/fleet/blob/main/CONTRIBUTING.md + about: Build, test and lint commands, the CI gates, and DCO sign-off. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 00000000..b87abfea --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,39 @@ +name: Feature request +description: Propose a capability or a change to how fleet behaves. +labels: [enhancement] +body: + - type: textarea + id: problem + attributes: + label: The problem + description: >- + What are you trying to do that fleet makes hard or impossible? Describe + the situation rather than the solution you have in mind. + validations: {required: true} + - type: textarea + id: proposal + attributes: + label: What you would like fleet to do + validations: {required: true} + - type: dropdown + id: invariant + attributes: + label: Does this touch one of the invariants in AGENTS.md? + description: >- + The mandatory sandbox, host-side credentials, the single governed loop, + no secrets in the repo, honest docs, or client content living in an + out-of-repo bundle. If yes, it needs an ADR — say so here and we can + work out the shape before anyone writes code. + options: + - "No — this is additive and does not touch an invariant" + - "Yes — or I am not sure" + validations: {required: true} + - type: textarea + id: bundle + attributes: + label: Could this ship as a client-config bundle instead? + description: >- + fleet is an engine; per-customer MCP servers, personas, protocols and + prompts belong in a bundle rather than in this repo. If you think this + genuinely needs an engine change, say why the bundle cannot express it. + validations: {required: false} diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 00000000..4351e65e --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,34 @@ + + +## What changed, and why + + + +## How you verified it + + + +## Scope and deviations + + + +--- + +- [ ] Commits are signed off (`git commit -s`) — see CONTRIBUTING.md +- [ ] `CHANGELOG.md` updated, if this is a user-visible change +- [ ] A design note (`docs/.md`) added, if this ships a feature +- [ ] An ADR added or superseded in `docs/adr/`, if this adds, weakens or + reverses an invariant — required in the *same* PR +- [ ] The diff is scoped to one change (no unrelated refactors) diff --git a/.github/workflows/semgrep.yml b/.github/workflows/semgrep.yml index 043618e8..4d79e401 100644 --- a/.github/workflows/semgrep.yml +++ b/.github/workflows/semgrep.yml @@ -98,9 +98,21 @@ jobs: # python3, and this repo pins no setup-python version anywhere else, so # adding one would be inventing an unverified action pin for nothing. # - # The semgrep version IS pinned, like every other tool this repo installs - # in CI (gitleaks, grype, golangci-lint), so an upstream release cannot - # change the findings under us without a visible diff. + # The semgrep version IS pinned, so an upstream release cannot change the + # findings under us without a visible diff. + # + # It is NOT the same guarantee gitleaks and grype get, and this comment + # used to claim it was. Those two are downloaded at a pinned version AND + # sha256-verified; `pip install semgrep==X` pins semgrep and resolves its + # ~40-package dependency closure (attrs, click, glom, requests, rich, + # ruamel.yaml, urllib3, …) to whatever is latest that day, unverified — + # inside a job that sits in both merge gates. The mitigating fact, which + # is why this is a caveat and not an incident: PyPI forbids re-uploading + # an existing version, so the exposure is a NEW malicious release of a + # transitive dep, not silent mutation of a pinned one — materially weaker + # than the mutable-GitHub-release-asset case the pinning exercise fixed. + # Closing it properly means a `--require-hashes` requirements file + # regenerated on every semgrep bump; tracked, not done here. env: SEMGREP_VERSION: '1.174.0' run: | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 37100417..593a5e61 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -147,6 +147,30 @@ If golangci-lint flags something, either fix it or add a `//nolint` with a reason (the `nolintlint` linter requires the reason). The lint backlog is at zero — please keep it there. +### If CI is red and you don't recognise the failure + +Three lanes here depend on **live external data**, so they can go red on a diff +that did not cause it — including a one-line documentation change. This is by +design (a new advisory *should* redden an unchanged tree), but it means a red +check is not automatically yours: + +- **`govulncheck`** queries the Go vulnerability database on every run. +- **`npm audit`** runs over both npm trees at `--audit-level=low` and fails on + any severity. +- **Semgrep** fetches its rule packs from the registry. They cannot be pinned by + vendoring — the Semgrep Rules License forbids redistribution — so a + registry-side rule addition can turn CI red with no commit to blame. + +If the failure names a package, advisory or rule you did not touch, say so in the +PR rather than trying to fix it; a maintainer will confirm and handle it. + +Two other things that surprise first-time contributors, neither of them a problem +with your change: a first PR waits for a maintainer to approve the workflow run +before CI starts at all, and the full `main` suite is around a dozen jobs +including a ~1.3 GB sandbox image build, so it is thorough rather than fast. +`make lint && make test && make ci-web` locally will catch nearly everything +first. + ## Branch and pull-request conventions - Branch off the latest `main`. Use a short, descriptive prefix, e.g. From 216120218b97bd502fbf0a0464646735bfe2d6e5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 00:30:46 +0000 Subject: [PATCH 29/34] Remove automatic merging and the unenforced DCO requirement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two owner decisions, applied across every file that referenced them. Auto-merge is gone .github/workflows/auto-merge-dependabot.yml is deleted. Its own header had already argued the case against it without noticing: it explained that `gh pr merge --auto` holds a merge only on REQUIRED checks, named `dev` as a branch whose ruleset requires none, said the `branches:` filter existed so the workflow "can never silently start applying to a branch nobody protected" — and then listed `dev`. Since every Dependabot version update targets `dev`, the mitigation SECURITY.md and docs/SCANNING.md both credited was in fact the delivery mechanism: same-day patch bumps landing unattended while Dev gate was still running. Every dependency bump now waits for a human, whatever the ecosystem or bump level. References updated rather than merely deleted, since several carried reasoning that only made sense under auto-merge: - SECURITY.md: the cooldown rationale no longer rests on "patch bumps are auto-merged", and the github-actions exception now says what actually contains it. - dependabot.yml: the cooldown header and both copies of the semver-major note were justified in terms of the auto-merge path. - CODEOWNERS: the "auto-merge interaction" paragraph described a hazard that no longer exists. - docs/SCANNING.md: the Known-gaps entry credited three auto-merge mitigations. It now records that the workflow's removal closes the compounding risk while the underlying gap — nothing requires Dev gate on dev — remains open and still needs a repo-settings change. Also corrected there: the action-pin census said 13 workflow files / 12 with a `uses:` / 53 references. It is now 12 / 11 / 56, with a note that check_action_pins_test.go rather than the paragraph is what holds the invariant — a hand-maintained count is exactly what goes stale. DCO is gone CONTRIBUTING.md required a Signed-off-by trailer and nothing enforced it. Rather than add a gate, the requirement is dropped: the sign-off section, the PR-template checkbox, and the pointers in AGENTS.md and the issue config. "Commit messages" keeps the useful half — imperative subjects, and why-not-what in the body. CHANGELOG records this alongside the rest of the CI review. Historical CHANGELOG entries that describe auto-merge as it existed are left alone; they are a record of what shipped, not a claim about today. --- .github/CODEOWNERS | 16 ++-- .github/ISSUE_TEMPLATE/config.yml | 2 +- .github/PULL_REQUEST_TEMPLATE.md | 1 - .github/dependabot.yml | 33 ++++---- .github/workflows/auto-merge-dependabot.yml | 75 ------------------ AGENTS.md | 3 +- CHANGELOG.md | 86 +++++++++++++++++++++ CONTRIBUTING.md | 13 +--- SECURITY.md | 18 ++--- docs/SCANNING.md | 29 ++++--- 10 files changed, 137 insertions(+), 139 deletions(-) delete mode 100644 .github/workflows/auto-merge-dependabot.yml diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 5ed098d7..24f5f805 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -42,14 +42,12 @@ # Deliberately NOT enabled — a CI-only gate so the sole maintainer can self-merge # a green PR — are required approvals and Code Owner review (see the note above). # -# Dependabot auto-merge interaction: .github/workflows/auto-merge-dependabot.yml -# enables `gh pr merge --auto` on PATCH-level bumps only, and the required `CI -# gate` check holds that merge until CI is green. Because Code Owner review is -# NOT currently required, such a patch PR auto-merges on green CI with no human -# approval — INCLUDING a github-actions patch bump that touches a CODEOWNERS- -# matched path like .github/workflows/. To force a human gate on those sensitive -# paths, turn on Code Owner review in the ruleset; minor and major bumps already -# wait for a human regardless. +# Dependabot interaction: there is no auto-merge in this repository. Every +# dependency bump — every ecosystem, every bump level — is merged by a human, so +# a github-actions bump touching a CODEOWNERS-matched path like +# .github/workflows/ cannot land unattended. Turning on Code Owner review in the +# ruleset would additionally force review by the owner named below rather than +# any maintainer. # --------------------------------------------------------------------------- # Catch-all: every file has an owner so nothing is silently unowned. Specific @@ -86,7 +84,7 @@ /internal/sched/db/migrations/ @bradflaugher # CI gates and dependency/agent automation — these workflows ARE the merge -# guarantees, including the Dependabot auto-merge path. +# guarantees. /.github/workflows/ @bradflaugher /.github/dependabot.yml @bradflaugher /.github/CODEOWNERS @bradflaugher diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index f1677b7e..37b131ea 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -10,4 +10,4 @@ contact_links: disclosure process and the response SLA. - name: Contributing guide url: https://github.com/ElcanoTek/fleet/blob/main/CONTRIBUTING.md - about: Build, test and lint commands, the CI gates, and DCO sign-off. + about: Build, test and lint commands, the CI gates, and PR conventions. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 4351e65e..c5e9c111 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -26,7 +26,6 @@ sentence, not a wasted one. --- -- [ ] Commits are signed off (`git commit -s`) — see CONTRIBUTING.md - [ ] `CHANGELOG.md` updated, if this is a user-visible change - [ ] A design note (`docs/.md`) added, if this ships a feature - [ ] An ADR added or superseded in `docs/adr/`, if this adds, weakens or diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 85841447..fcfdb755 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -9,12 +9,11 @@ # release. This is a deliberate defense against fast typosquat / account-takeover # attacks, where a compromised version is published and then yanked once the # ecosystem flags it — by the time Dependabot proposes the bump, that window has -# usually closed. It matters most for patch bumps, which the auto-merge workflow -# (`.github/workflows/auto-merge-dependabot.yml`) merges on its own once CI is -# green: the cooldown keeps a minutes-old patch from being proposed (and thus -# auto-merged) before the ecosystem has had a chance to scrutinize it. Cooldown -# applies to version updates only, never to Dependabot security updates, so -# urgent CVE fixes are not delayed. +# usually closed. Every bump here is merged by a human (this repository has no +# auto-merge), so the cooldown is not load-bearing on its own; what it buys is +# that a reviewer is never looking at a release the ecosystem has not yet had a +# chance to scrutinize. Cooldown applies to version updates only, never to +# Dependabot security updates, so urgent CVE fixes are not delayed. # # Schema (key `cooldown`, integer `*-days` sub-keys; supported for gomod and npm; # applies to version updates only): @@ -113,15 +112,12 @@ updates: # CVE fixes are never delayed behind dev. target-branch: dev open-pull-requests-limit: 10 - # Wait before proposing freshly published releases (see header). Patch is the - # tightest gate because auto-merge-dependabot.yml auto-merges patch bumps. + # Wait before proposing freshly published releases (see header). cooldown: default-days: 7 - # 7, not 14: the supply-chain argument in the header is about the - # AUTO-MERGE path, and auto-merge-dependabot.yml merges patch only — - # a major is never auto-merged, so delaying it protects nothing and - # costs up to 14 days of visibility (cooldown + the weekly interval) - # before anyone even learns the major exists. + # 7, not 14 for a major: a long delay protects nothing here — a major is + # reviewed by a human either way — and costs up to 14 days of visibility + # (cooldown + the weekly interval) before anyone even learns it exists. semver-major-days: 7 semver-minor-days: 7 semver-patch-days: 3 @@ -156,11 +152,9 @@ updates: # Same supply-chain cooldown as gomod (see header). cooldown: default-days: 7 - # 7, not 14: the supply-chain argument in the header is about the - # AUTO-MERGE path, and auto-merge-dependabot.yml merges patch only — - # a major is never auto-merged, so delaying it protects nothing and - # costs up to 14 days of visibility (cooldown + the weekly interval) - # before anyone even learns the major exists. + # 7, not 14 for a major: a long delay protects nothing here — a major is + # reviewed by a human either way — and costs up to 14 days of visibility + # (cooldown + the weekly interval) before anyone even learns it exists. semver-major-days: 7 semver-minor-days: 7 semver-patch-days: 3 @@ -180,8 +174,7 @@ updates: # # Know what a green build means here: no CI job exercises this service, and # `@nationaldesignstudio/rampart` is pre-1.0 (^0.1.3), where a MINOR bump is - # allowed to break. Auto-merge cannot touch these (it gates on patch only), - # so they land in human review — which is the right outcome, not a gap. + # allowed to break. Like every other bump here, these land in human review. - package-ecosystem: npm directory: "/scripts/rampart-service" schedule: diff --git a/.github/workflows/auto-merge-dependabot.yml b/.github/workflows/auto-merge-dependabot.yml deleted file mode 100644 index 776108ca..00000000 --- a/.github/workflows/auto-merge-dependabot.yml +++ /dev/null @@ -1,75 +0,0 @@ -# Dependabot already opens dependency-update PRs (see .github/dependabot.yml), -# but every one — even a routine patch — currently waits on a human to merge. -# This workflow lets PATCH-level bumps merge themselves once the full CI gate -# (build / vet / lint / test / -race / govulncheck, web lint+test+build, -# Playwright mocked + live, and the gitleaks secret scan) is green. Minor and -# major bumps are intentionally left for a human, where an API change or a -# transitive surprise is more likely. -# -# TWO LIMITS THAT ARE LOAD-BEARING, both learned the hard way: -# -# 1. "CI is the approval signal, and it is never bypassed" IS ONLY TRUE WHERE -# THE GATE IS A REQUIRED CHECK. `gh pr merge --auto` asks GitHub to hold the -# merge until every REQUIRED check passes — so on a branch whose ruleset -# requires nothing, there is nothing to hold it and the PR merges as soon as -# it is mergeable. The `dev` ruleset currently requires no status checks at -# all (only `deletion` and `non_fast_forward`), and .github/dependabot.yml -# points every version update at `dev`. So the `branches:` filter below is -# not cosmetic: it keeps this workflow from applying to a branch where its -# central assumption does not hold. Getting `Dev gate` into the dev ruleset -# is the real fix and is a repo-settings action; see docs/SCANNING.md -# ("Known gaps"). -# -# 2. A `github-actions` bump IS A REWRITE OF .github/workflows/*. It changes -# what CI executes, on a surface where the cooldown that protects gomod and -# npm is not even available (Dependabot supports `cooldown` for those two -# ecosystems only), so a freshly published action version can be proposed -# the same day. That combination — self-modifying CI, no cooldown, no -# required check on the target branch — is not something to auto-merge, so -# that ecosystem is excluded below and takes a human. -# -# Requires "Allow auto-merge" to be enabled on the repository (Settings → -# General → Pull Requests). This is the pattern documented in GitHub's -# "Automating Dependabot with GitHub Actions" guide. -name: Auto-merge Dependabot patch PRs - -on: - pull_request: - # See limit 1 in the header: this workflow's safety rests on the target - # branch having required checks. Naming the branches explicitly means it can - # never silently start applying to one nobody protected. - branches: [main, dev] - -# What actually confines these scopes is the `if: github.actor == -# 'dependabot[bot]'` guard on the job below — a `permissions:` block is honored -# for whatever run reaches it, regardless of actor. github.actor is not -# spoofable, so the guard holds; the scopes are declared on the JOB rather than -# the workflow so a second job added here later does not inherit write access it -# never asked for. -permissions: {} - -jobs: - auto-merge: - if: ${{ github.actor == 'dependabot[bot]' }} - runs-on: ubuntu-latest - timeout-minutes: 5 - permissions: - contents: write - pull-requests: write - steps: - - name: Fetch Dependabot metadata - id: meta - uses: dependabot/fetch-metadata@25dd0e34f4fe68f24cc83900b1fe3fe149efef98 # v3.1.0 - with: - github-token: ${{ secrets.GITHUB_TOKEN }} - - - name: Enable auto-merge for patch updates - # Only patch bumps auto-merge; minor and major get human review. And - # never github-actions, whatever the bump level — see limit 2 in the - # header: that ecosystem's "dependency" is the CI definition itself. - if: ${{ steps.meta.outputs.update-type == 'version-update:semver-patch' - && steps.meta.outputs.package-ecosystem != 'github_actions' }} - run: gh pr merge --auto --squash "$PR_URL" - env: - PR_URL: ${{ github.event.pull_request.html_url }} - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/AGENTS.md b/AGENTS.md index 70131b55..bf1e0489 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -168,8 +168,7 @@ same PR. to this file — that is how it grew past 300 lines once already; the historical notes now live in [`docs/FEATURE-NOTES.md`](docs/FEATURE-NOTES.md). - One focused branch + PR per change; keep diffs scoped. Don't refactor unrelated - code in a feature PR. See `CONTRIBUTING.md` for branch/PR conventions and DCO - sign-off. + code in a feature PR. See `CONTRIBUTING.md` for branch/PR conventions. ## Where to look diff --git a/CHANGELOG.md b/CHANGELOG.md index 9314ed86..3326edd0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,92 @@ prior versions are listed because none have shipped. ### Added +- **A workflow + shell lint gate (`actionlint`, `shellcheck`):** nothing checked + the ~3.1k lines of workflow YAML that decide what every other gate runs, and + nothing checked the ~6.2k lines of bash that *are* the deploy path + (`update.sh` / `bootstrap.sh` / `doctor.sh`). Both now block in `ci.yml` and + `dev-ci.yml` and run from `make lint` via `lint-actions`, pinned and + checksum-verified like `gitleaks` and `grype`. `actionlint` covers what + Semgrep's `p/github-actions` pack and CodeQL's `actions` language do not — + `${{ }}` expression syntax and types, undefined contexts, invalid + `needs:`/`runs-on:`/cron — plus shellcheck over every `run:` block. + Both start at **zero findings**, measured: the 5 actionlint and 3 shellcheck + items were fixed or annotated with reasons in the same change, so a failure is + a regression rather than a backlog. +- **Two new CI invariant tests**, in the spirit of the existing pin/gate tests: + every workflow must declare a top-level `permissions:` block (without one it + silently inherits the repository default), and the `actionlint` + version/checksum must agree across both lanes. +- **PR and issue templates.** The PR template prompts for what/why, how it was + verified, and scope-and-deviations, plus the obligations that were previously + a reviewer's job to remember (CHANGELOG entry, `docs/.md` note, an + ADR in the *same* PR when an invariant moves). `ISSUE_TEMPLATE/config.yml` + disables blank issues so the private security-disclosure link is unmissable — + `SECURITY.md` says "do not open a public issue for a vulnerability" and the + New Issue button was offering a blank box with no such warning. + +### Changed + +- **Automatic merging removed.** `auto-merge-dependabot.yml` is deleted and every + reference to it across `SECURITY.md`, `dependabot.yml`, `CODEOWNERS` and + `docs/SCANNING.md` is gone. Its own header had argued the case against it: it + explained that `gh pr merge --auto` holds a merge only on *required* checks, + named `dev` as a branch that requires none, and then listed `dev` in its own + `branches:` filter — so the mitigation the docs credited was in fact the + delivery mechanism for same-day patch bumps landing unattended. Every + dependency bump now waits for a human. +- **DCO sign-off is no longer requested.** The requirement was documented in + `CONTRIBUTING.md` and enforced nowhere; rather than add a gate for it, the + requirement was dropped. +- **`ci.yml` gained a `concurrency:` group** (the expensive lane had none, so + stacked pushes ran full suites to completion). Cancellation is scoped to + `pull_request` — a push to `main` is the only tree-wide CodeQL verdict and must + not be cancelled. **`timeout-minutes` is now set on every job** (it was present + in 2 of 13 workflows). +- **`issues: write` no longer sits at workflow scope** in the two scheduled scan + lanes, where it was live for every step of a long job running + `govulncheck@latest` and a podman build pulling ~400 RPMs. It moved to a + dedicated alarm job that checks out nothing — the shape `scan-cron-alarm.yml` + already used. `persist-credentials: false` added to the write-scoped checkouts. +- **`make ci-web` now mirrors the real web job.** It ran 4 of its 8 steps, + dropping both `npm audit` gates, the override canary and the explicit + `npm run typecheck` — a clean local run and a red PR. + +### Fixed + +- **Two green-but-vacuous holes in `ci.yml`.** The `postgresql-client-18` install + was best-effort (`|| echo`), so an unreachable PGDG left client 16 in place and + `backup_test.go`'s major-mismatch `t.Skipf` turned the *only* coverage of + `fleet backup` / `fleet restore` off behind the single required check on + `main`; it now asserts the major. And the docs-only classifier initialised + `docs_only=true` and only ever cleared it inside its loop, so an **empty** diff + classified as docs-only and skipped the suite — which `ci-gate` then waved + through, because an empty diff is the absence of evidence, not evidence that + only prose changed. +- **`dev-ci.yml`'s `go test` could report cached results.** It lacked the + `-count=1` that `ci.yml` passes on all three of its invocations; `setup-go` + restores the build cache, which holds test *results*, and Go's cache key cannot + see a Postgres service container. +- **Semgrep's deferred-failure path was unreachable.** `set -uo pipefail` does not + clear the `-e` the runner supplies, so the step aborted on the `semgrep` line + and `status=$?` never ran. The outcome was still correct; the documented design + was not what executed. +- **The docs advertised a CodeQL waiver route that does not work.** Five places — + `SCANNING.md`, `CODEQL.md`, ADR-0048, `CONTRIBUTING.md` and the gate's own + runtime advice — offered an in-source `// codeql[rule-id]` comment, while + `codeql.yml` recorded that all three forms were tried on #1249 and none + produced a `suppressions` array. `codeql.yml` carried *both* claims, the stale + one directly above the note refuting it. The accepted-findings register is the + only route that works. +- **ADR-0048's normative Decision stated the gate threshold as an OR** where + `codeql-gate.jq` implements a fallback; read literally it blocks on + `go/log-injection` (`error` at security-severity 6.1) — the exact deadlock the + ADR exists to undo. +- Assorted doc drift: `CONTRIBUTING.md` named Go 1.26.x (`go.mod` says 1.27.0); + `CODEOWNERS` enumerated 7 of `ci-gate`'s 13 `needs`; `semgrep.yml` claimed its + `pip install` had the same checksum guarantee as the `gitleaks`/`grype` + downloads, which it does not. + - **Kubernetes as a first-class deployment (#989 / ADR-0049):** the fleet control plane can now run in a cluster with agent sandboxes as **ephemeral pods**, selected by one knob — `FLEET_SANDBOX_BACKEND=podman|kubernetes` diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 593a5e61..33787e58 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -181,18 +181,11 @@ first. - Make sure the full local suite (Go + web + mocked Playwright) is green before you push. -## Commit messages and sign-off +## Commit messages - Write clear, imperative commit subjects ("Add X", not "Added X"). -- Sign off your commits with the Developer Certificate of Origin - () by adding a `Signed-off-by` trailer: - - ```bash - git commit -s -m "Your message" - ``` - - By signing off you certify that you wrote the patch (or otherwise have the - right to submit it) under the project's MIT license. +- Explain *why* in the body when the change is not self-evident. The diff + already says what changed. ## Reporting bugs and proposing features diff --git a/SECURITY.md b/SECURITY.md index a3ee7176..b24db581 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -202,10 +202,7 @@ compromised or fresh-and-unvetted release from reaching `main`: Dependabot waits a few days (3 for patch, 7 for minor, 14 for major) before proposing a freshly published release. This blunts fast typosquat / account-takeover attacks, where a malicious version is published and then yanked - once the ecosystem flags it. It matters most for **patch** bumps, which - `.github/workflows/auto-merge-dependabot.yml` auto-merges once the CI gate is - green: without a cooldown a minutes-old patch could be proposed and auto-merged - before any scrutiny. Cooldown applies to version updates only — Dependabot + once the ecosystem flags it. Cooldown applies to version updates only — Dependabot **security** updates are never delayed, so urgent CVE fixes still flow immediately. @@ -214,18 +211,17 @@ compromised or fresh-and-unvetted release from reaching `main`: only, so the one ecosystem whose "dependency" is *the CI definition itself* — a `github-actions` bump rewrites `.github/workflows/*` and therefore changes what CI executes — cannot be made to wait, and it is configured daily against - `dev`. Because `dev` additionally has no required status checks (see "Static - analysis" above), that combination is not something to auto-merge, so - `auto-merge-dependabot.yml` **excludes `github_actions` at any bump level** and - those PRs take a human. The workflow also carries an explicit - `branches: [main, dev]` filter, so it can never silently begin applying to some - other branch, and declares its write scopes on the job rather than the workflow. + `dev`, which additionally has no required status checks (see "Static analysis" + above). What contains that combination now is simply that **every** Dependabot + PR takes a human: automatic merging was removed from this repository, so no + dependency bump of any ecosystem or bump level reaches a branch without someone + looking at it. The cooldown reduces the window for a fast attack but is **not** a guarantee: a patient attacker who waits out the cooldown, or a compromise the ecosystem never flags, would still slip through. The committed `go.sum` + checksum DB, `govulncheck` and `npm audit` are the stronger, always-on controls; the cooldown -is defense-in-depth on top of the auto-merge path, and it does not cover +is defense-in-depth on top of human review, and it does not cover `github-actions` at all. ## CSRF protection (cookie-authenticated routes) diff --git a/docs/SCANNING.md b/docs/SCANNING.md index 371eae23..970f8d7e 100644 --- a/docs/SCANNING.md +++ b/docs/SCANNING.md @@ -193,9 +193,12 @@ fixing every real finding and adjudicating every false one. `p/github-actions` found one issue class nothing else in this repo checks — actions referenced by a **mutable tag** (`actions/checkout@v7`) instead of an immutable commit SHA. If a tag moves, attacker-controlled code runs with this -repo's `GITHUB_TOKEN`. There are **13** workflow files, **12** of which reference +repo's `GITHUB_TOKEN`. There are **12** workflow files, **11** of which reference an action at all (`scan-cron-alarm.yml` has no `uses:`), and every one of the -**53** third-party action references across them is now pinned: +**56** third-party action references across them is pinned. (The counts move with +every workflow added or removed — they were 13/12/53 when this was written, and +`scripts/check_action_pins_test.go`, not this paragraph, is what actually holds +the invariant.) ```yaml uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -430,14 +433,20 @@ Stated rather than left for rediscovery: 2. A `github-actions` bump **is a rewrite of `.github/workflows/*`**: it changes what CI executes. - So the pre-existing shape was: a same-day patch bump to a third-party action, - auto-merged into a branch with no required checks, rewriting the workflows that - are supposed to check it. Three workflow-side mitigations ship alongside this - document — `auto-merge-dependabot.yml` now **excludes the `github_actions` - ecosystem** whatever the bump level, carries an explicit - `branches: [main, dev]` filter so it can never silently start applying to an - unprotected branch, and declares its write scopes on the job rather than the - workflow. Those narrow the blast radius; they do not make `Dev gate` required. + So the shape to avoid is: a same-day patch bump to a third-party action landing + unattended on a branch with no required checks, rewriting the workflows that are + supposed to check it. + + **What removes it is that this repository no longer merges anything + automatically.** `auto-merge-dependabot.yml` was deleted. Its header had argued + the case against itself — it explained that `gh pr merge --auto` holds a merge + only on *required* checks, named `dev` as a branch with none, and then listed + `dev` in its own `branches:` filter, so the mitigation the previous revision of + this document credited was in fact the delivery mechanism. Every dependency + bump, every ecosystem, every bump level now waits for a human. + + That closes the compounding risk but **not** the underlying gap: a hand-merged + PR into `dev` still merges over a red `Dev gate`, because nothing requires it. **The remaining fix is a repo-settings action and belongs to the owner:** add `Dev gate` to the `dev` ruleset's required status checks. Nothing in a workflow From 60d8abdf3b05d860faba8415aeed714f982cce70 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 00:33:49 +0000 Subject: [PATCH 30/34] Reconcile the CI docs with the workflows, and close two residual exposures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The docs here are unusually thorough, which is what made the drift worth chasing: a reader trusts them. Everything below was verified against the YAML rather than taken from a previous revision of the prose. Docs that overclaimed or understated - ci.yml's Grype step said "Block only when a CRITICAL RPM finding has a packaged fix". scripts/check-grype-policy.sh selects `critical or high`. The comment understated the gate; every prose doc had it right. - docs/TESTING.md said the web job was `npm ci -> npm run lint (ESLint) -> vitest -> build`. `web/package.json` runs oxlint, and the SAME document spends 40 lines above explaining why ESLint was replaced. It also omitted four of the job's eight steps (both npm audits, the override canary, and the explicit typecheck). - docs/SCANNING.md described the sandbox-workflow hardening as "refuses refs/pull/* refs". That deny-list was replaced by an allow-list BECAUSE it was bypassable, and the two holes are worth stating rather than quietly updating: a workflow_call input may carry newlines, so `main\nresolved=refs/pull/1/head` matched no deny pattern and emitted a second GITHUB_OUTPUT assignment (last wins); and "every named ref is collaborator-written" is false of reachable COMMITS, since fork-PR objects live in the base repo's store. A reader re-implementing the documented form would have inherited both. - scan-cron-alarm.yml watches three workflows, not the two documented. Controls that existed but were cited nowhere check_gate_needs_test.go is the strongest anti-rot control in this setup — it fails `make test` when a job is missing from its gate's `needs`, which is the regression that produced the red-but-not-required CodeQL break — and it appeared in no document. docs/SCANNING.md now names it alongside check_action_pins_test.go and the new check_permissions_test.go, spells out ci-gate's full needs list, and covers the `helm` and `migrations` lanes, which were in both gates and in no doc at all. The stack table gains actionlint and shellcheck rows with an explicit note on why they earn a place next to Semgrep's p/github-actions pack: that pack and CodeQL's `actions` language are security rule sets, and neither parses ${{ }} expressions or shellchecks a `run:` block. Also corrected there: gitleaks was described as running on "every branch"; the job exists in ci.yml and dev-ci.yml, so a feature branch with no open PR gets no scan. Residual exposures closed - e2e-canary.yml uploaded `.e2e-run/logs/` on failure. That is the ONE job booting fleet with the real OPENROUTER_API_KEY, the server's stdout is redirected into that directory, and the repo is public. Nothing in the Go tree logs the key today, so this was tail risk, not a known leak — but the Playwright report is what a failure is diagnosed from anyway. ci.yml's e2e-live keeps its logs: fake LLM, throwaway key. - bundle_dir reached the same build-script invocation as fleet_ref with no validation at all, while fleet_ref has a character allow-list, a bare-SHA refusal and a newline fix. No injection (it goes through env: and is quoted), but nothing stopped `../../.fleet-core` from repointing the build at another manifest. Guard added to both reusable workflows and exercised against absolute paths, `..`, spaces and `$(id)`. AGENTS.md's `make lint` and CI-mirrors lines now include the new lanes and the Helm lint. Not changed, and deliberately: setup-go's inputs differ between lanes (`check-latest: true` in some, `cache: true` in others). These are not alternatives — they control different things, and go.mod pins an exact 1.27.0 — so "make them consistent" would be churn without a defect. --- .github/workflows/build-sandbox-image.yml | 25 ++++++ .github/workflows/ci.yml | 8 +- .github/workflows/e2e-canary.yml | 19 ++++- .github/workflows/publish-sandbox-image.yml | 25 ++++++ AGENTS.md | 10 ++- docs/SCANNING.md | 95 +++++++++++++++++---- docs/TESTING.md | 14 ++- 7 files changed, 168 insertions(+), 28 deletions(-) diff --git a/.github/workflows/build-sandbox-image.yml b/.github/workflows/build-sandbox-image.yml index fffdc5d6..a7f0b9ea 100644 --- a/.github/workflows/build-sandbox-image.yml +++ b/.github/workflows/build-sandbox-image.yml @@ -113,6 +113,31 @@ jobs: # actions/untrusted-checkout under security-extended; the same pattern in # publish-sandbox-image.yml was hardened symmetrically even though the # query's privileged/taint split happened to flag neither variant there.) + - name: Validate bundle_dir + # `fleet_ref`, which reaches the SAME build-script invocation, got a + # character allow-list, a bare-SHA refusal and a GITHUB_OUTPUT-newline + # fix. `bundle_dir` got nothing, and the asymmetry is the whole reason + # this exists — there is no shell injection here (it goes through `env:` + # and is quoted at every use), but nothing stopped `../../.fleet-core` + # or an absolute path from silently repointing the build at a different + # manifest. The reusable workflow runs in the CALLER's context with the + # caller's token, so the blast radius is a client repo misbuilding its + # own image; this is defence in depth and consistency, not a live hole. + env: + REQUESTED: ${{ inputs.bundle_dir }} + run: | + set -euo pipefail + case "$REQUESTED" in + ""|.) ;; + /*) + echo "::error::bundle_dir must be relative to the caller repo, not absolute."; exit 1 ;; + *..*) + echo "::error::bundle_dir must not contain '..' — it may only name a path inside the caller repo."; exit 1 ;; + *[!a-zA-Z0-9._/-]*) + echo "::error::bundle_dir contains a character outside [A-Za-z0-9._/-]. Refused before the build."; exit 1 ;; + esac + echo "bundle_dir accepted: '${REQUESTED:-.}'" + - name: Pin fleet_ref to collaborator-controlled refs id: pin env: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ac9ee351..97705e89 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -993,9 +993,11 @@ jobs: --output sarif=grype-results.sarif - name: Enforce actionable Fedora RPM policy - # The generic image follows Fedora latest. Block only when a CRITICAL - # RPM finding has a packaged fix; report upstream language records but - # do not force hand-maintained overlay pins over distro-owned packages. + # The generic image follows Fedora latest. Block when a CRITICAL **or + # HIGH** RPM finding has a packaged fix (scripts/check-grype-policy.sh + # selects both severities — this comment said CRITICAL only, which + # understated the gate); report upstream language records but do not + # force hand-maintained overlay pins over distro-owned packages. run: scripts/check-grype-policy.sh grype-results.json - name: Upload Grype SARIF results to the GitHub Security tab diff --git a/.github/workflows/e2e-canary.yml b/.github/workflows/e2e-canary.yml index 24da2d67..a44c2d38 100644 --- a/.github/workflows/e2e-canary.yml +++ b/.github/workflows/e2e-canary.yml @@ -131,12 +131,25 @@ jobs: run: npx playwright test --project=canary --reporter=list - name: Upload canary report on failure + # Playwright report ONLY — deliberately not `.e2e-run/logs/`. + # + # This is the one job in the repo that boots fleet with the REAL + # OPENROUTER_API_KEY, and scripts/e2e-boot-server.sh redirects the + # server's stdout/stderr into that log directory. The repo is public, so + # an artifact here is world-downloadable. Nothing in the Go tree logs the + # key today (internal/agentcore/openrouter_models.go notes explicitly + # that it never enters that path), so this is residual rather than a + # known leak — it would take an upstream HTTP error string echoing an + # Authorization header. But the report is what a failure is actually + # diagnosed from, and the logs are not worth that tail risk. + # + # ci.yml's e2e-live job still uploads its logs, correctly: it runs + # against the fake LLM (cmd/fake-llm) with a throwaway key, so there is + # nothing there to leak. if: ${{ failure() }} uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: canary-report - path: | - web/playwright-report/ - .e2e-run/logs/ + path: web/playwright-report/ retention-days: 14 if-no-files-found: ignore diff --git a/.github/workflows/publish-sandbox-image.yml b/.github/workflows/publish-sandbox-image.yml index 7ee0f477..7f7a07fc 100644 --- a/.github/workflows/publish-sandbox-image.yml +++ b/.github/workflows/publish-sandbox-image.yml @@ -235,6 +235,31 @@ jobs: # actions/untrusted-checkout under security-extended; the same pattern in # build-sandbox-image.yml was flagged; this file escaped both query # variants despite holding packages: write — the MORE dangerous twin.) + - name: Validate bundle_dir + # `fleet_ref`, which reaches the SAME build-script invocation, got a + # character allow-list, a bare-SHA refusal and a GITHUB_OUTPUT-newline + # fix. `bundle_dir` got nothing, and the asymmetry is the whole reason + # this exists — there is no shell injection here (it goes through `env:` + # and is quoted at every use), but nothing stopped `../../.fleet-core` + # or an absolute path from silently repointing the build at a different + # manifest. The reusable workflow runs in the CALLER's context with the + # caller's token, so the blast radius is a client repo misbuilding its + # own image; this is defence in depth and consistency, not a live hole. + env: + REQUESTED: ${{ inputs.bundle_dir }} + run: | + set -euo pipefail + case "$REQUESTED" in + ""|.) ;; + /*) + echo "::error::bundle_dir must be relative to the caller repo, not absolute."; exit 1 ;; + *..*) + echo "::error::bundle_dir must not contain '..' — it may only name a path inside the caller repo."; exit 1 ;; + *[!a-zA-Z0-9._/-]*) + echo "::error::bundle_dir contains a character outside [A-Za-z0-9._/-]. Refused before the build."; exit 1 ;; + esac + echo "bundle_dir accepted: '${REQUESTED:-.}'" + - name: Pin fleet_ref to collaborator-controlled refs id: pin env: diff --git a/AGENTS.md b/AGENTS.md index bf1e0489..fe28f1b7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,8 @@ make compile # go build ./... (compile-check only; no artifacts) make test # go test -p 1 ./... — run in the FOREGROUND make test-race # go test -race -p 1 ./... (use when touching concurrency) make test-cover # run Go tests with coverage profiling (writes coverage.out) -make lint # golangci-lint + ruff check/format (Python) + migration DDL lint — must pass clean +make lint # golangci-lint + ruff check/format (Python) + migration DDL lint + # + actionlint & shellcheck (workflows + shell) — must pass clean make fmt # gofmt -w . make tidy # go mod tidy ``` @@ -40,9 +41,10 @@ cd web && npx playwright test --project=mocked # mocked e2e CI mirrors all of this — Go build/vet/lint/test (including a `-race` lane) plus a `govulncheck` dependency-CVE scan, a Grype container-image CVE scan (fail on a -fixable CRITICAL/HIGH) of the sandbox image, a Python lint (ruff), web lint (oxlint) / typecheck (TS 7) / test / build, Playwright (mocked -**and** live, against a real backend + sandbox), a migration DDL lint, and a -gitleaks secret scan. **Every job must be green before merge.** Tests are +fixable CRITICAL/HIGH) of the sandbox image, a Python lint (ruff), a workflow + +shell lint (actionlint & shellcheck), web lint (oxlint) / typecheck (TS 7) / test / build, Playwright (mocked +**and** live, against a real backend + sandbox), a Helm chart lint, a migration +DDL lint, and a gitleaks secret scan. **Every job must be green before merge.** Tests are deterministic without a live model: use the fake-LLM seam (`internal/fakellm` via `OPENROUTER_BASE_URL`), never a real key. diff --git a/docs/SCANNING.md b/docs/SCANNING.md index 970f8d7e..2e52d180 100644 --- a/docs/SCANNING.md +++ b/docs/SCANNING.md @@ -12,15 +12,32 @@ security queries only) and [`TESTING.md`](TESTING.md) (the rest of the ladder). | `golangci-lint` (incl. `gosec`) | Go lint + Go SAST patterns | ~30s | **blocks** (`ci-gate`) | job log | | `oxlint` + `tsc` | web tier lint + types | ~5s | **blocks** (`ci-gate`) | job log | | **`ruff`** | **Python lint** | **~1s** | **blocks** (`ci-gate`) | job log | +| **`actionlint`** | **workflow YAML: `${{ }}` expressions, contexts, `needs`/`runs-on`/cron, + shellcheck over every `run:`** | **~2s** | **blocks** (`ci-gate`) | job log | +| **`shellcheck`** | **the 18 tracked `*.sh` files (~6.2k lines) — the deploy path** | **~2s** | **blocks** (`ci-gate`) | job log | | `govulncheck` | Go dependency CVEs (called symbols) | ~30s | **blocks** (`ci-gate`) | job log + Security tab | | `grype` | sandbox image CVEs (fixable **CRITICAL + HIGH**, **RPMs only**) | ~1m | **blocks** (`ci-gate`) | job log + Security tab | -| `gitleaks` | secrets, every branch | ~10s | **blocks** (`ci-gate`) | job log | +| `gitleaks` | secrets (on `main` and `dev` — the two branches with a CI lane) | ~10s | **blocks** (`ci-gate`) | job log | | **`npm audit`** | npm dependency CVEs (web + rampart-service) | ~5s | **blocks** (`ci-gate`) | job log | | CodeQL | **interprocedural taint / `security-extended`** | ~2m | **blocks** on an unwaived High-band finding (`ci-gate`/`Dev gate` via workflow_call) | job log + Security tab | | **Semgrep** | **Go/JS/Python SAST + Actions supply chain** | ~40s | **blocks** on any unsuppressed finding (`ci-gate`/`Dev gate` via workflow_call) | job log + artifact | -Two things were added here (**ruff**, **Semgrep**) and one was narrowed -(**CodeQL**, to security queries only). +Four things were added here (**ruff**, **Semgrep**, then **actionlint** and +**shellcheck**) and one was narrowed (**CodeQL**, to security queries only). + +**Why actionlint and shellcheck, given Semgrep already runs `p/github-actions`.** +The overlap is one axis wide: Semgrep's Actions pack is a *security* rule set — +mutable action tags, template injection, `pull_request_target` misuse — and +CodeQL's `actions` language is likewise security-query-only. Neither parses +`${{ }}` expressions and neither shellchecks a `run:` block. That mattered here +concretely: the census below found `run: "$GITHUB_WORKSPACE/scripts/..."` in both +CI lanes passing its path to the shell **unquoted**, because the YAML parser +consumes the quotes — a bug the comment directly above the line showed the author +believed they had avoided. And bash was the only language in the repo with no +linter at all, while being the language `fleet update` and `fleet bootstrap` +actually execute on an operator's box. Both gates started at **zero findings**; +the 5 actionlint and 3 shellcheck items were fixed or annotated with reasons +before the gate went in, because a gate switched on over a backlog is a gate +people learn to scroll past. **Read "blocks" with one caveat, and it is a big one.** Every lane above reaches its branch's aggregate gate job — but a gate job only *blocks a merge* where it @@ -171,12 +188,30 @@ Getting the extended suite adopted was itself a fix, not a rubber stamp: the one `actions`-language finding was `actions/untrusted-checkout/medium` on `build-sandbox-image.yml`'s `fleet_ref`-fed checkout. Rather than waive it (the `actions` language has no `AlertSuppression.ql`, so there is no in-code waiver -anyway), the workflow now **refuses `refs/pull/*` refs** before checking out — a -fork-PR ref would put fork-controlled code into a workflow that runs the -checked-out build script — and the identical hardening went into +anyway), the workflow now validates `fleet_ref` before checking out — a fork-PR +ref would put fork-controlled code into a workflow that runs the checked-out +build script — and the identical hardening went into `publish-sandbox-image.yml`, the *unflagged* twin that holds `packages: write` and only escaped the (name-heuristic) query because its plumbing was named -differently. Details in [`CODEQL.md`](CODEQL.md). +differently. + +**It is an ALLOW-LIST, and an earlier revision of this document described the +deny-list it replaced.** Worth correcting rather than quietly updating, because +the deny-list (`refs/pull/*|pull/*|-*`) had two holes that a reader +re-implementing "refuse `refs/pull/*`" elsewhere would inherit: + +1. **`GITHUB_OUTPUT` newline injection.** A `workflow_call` string input may + contain newlines, so `fleet_ref: "main\nresolved=refs/pull/1/head"` matched no + deny pattern (it starts `main`), exited 0, and emitted **two** `resolved=` + lines — last-wins handed the attacker the ref. The same primitive forges any + step output. +2. **A bare commit SHA.** "Every ref in this repo is collaborator-written except + `refs/pull/*`" is true of *named refs* and false of reachable *commits*: + GitHub keeps fork-PR commits in the base repo's object store and + `actions/checkout` will fetch a bare SHA happily. + +The shipped form admits only `[A-Za-z0-9._/-]` (so no newline can carry a second +assignment), then refuses `-*`, `*..*`, `*//*`, `refs/pull/*` and a bare hex SHA. Details in [`CODEQL.md`](CODEQL.md). ### Semgrep owns fast multi-language SAST + Actions supply chain (new, blocking) @@ -359,12 +394,37 @@ validated by `tsc`). Every lane in the table reaches the branch's aggregate gate: -- `ci-gate` (the single required status check on `main`) `needs` the lint, test - and build jobs — **and the two scanners**. -- `Dev gate` `needs` the same set on `dev` — but nothing in the `dev` ruleset - requires `Dev gate` to be green, so on that branch it is a red check rather - than a closed gate. That gap is the first item under "Known gaps" and it is - the single most important qualifier on this whole document. +- `ci-gate` (the single required status check on `main`) `needs` **every other + job in `ci.yml`** — the docs-only classifier, gitleaks, the actionlint/shellcheck + workflow+shell lint, the migration DDL lint, the Helm chart lint, Go, ruff, + CodeQL, Semgrep, web, both Playwright lanes and Grype. +- `Dev gate` `needs` the same set that exists on `dev` — but nothing in the `dev` + ruleset requires `Dev gate` to be green, so on that branch it is a red check + rather than a closed gate. That gap is the first item under "Known gaps" and it + is the single most important qualifier on this whole document. + +**That "every other job" is a test, not a habit — and it is the strongest +anti-rot control here, so it should not stay invisible the way it did until +now.** `scripts/check_gate_needs_test.go` parses both workflow files and fails +`make test` if any job is missing from its gate's `needs`. Adding a job and +forgetting to extend `needs` is otherwise a silent one-line regression that +produces a red-but-not-required lane — exactly how the CodeQL Go extraction +break sat unnoticed for weeks. Two sibling tests hold the neighbouring +invariants: `check_action_pins_test.go` (every `uses:` is a 40-hex commit SHA +with an exact version comment) and `check_permissions_test.go` (every workflow +declares a top-level `permissions:` block, so none silently inherits the +repository default). + +Two lanes in `ci-gate`'s list are not in the table above because they are not +scanners: **`helm`** lints and renders the Helm chart so a values/template drift +fails here rather than at an operator's install, and **`migrations`** rejects +dangerous DDL in new or changed migration files. + +One qualifier on "every lane reaches the gate": on a **docs-only** change the +`changes` job skips the heavy lanes, and `ci-gate` passes over those skips. That +is deliberate and narrowly bounded — the classifier is a prose allow-list, and +`ci-gate` refuses a skip whenever the classifier did *not* say docs-only, so a +skip from any other cause fails the gate. The scanners get there because `codeql.yml` and `semgrep.yml` are **reusable workflows** (`on: workflow_call`): `ci.yml` and `dev-ci.yml` each call them as a @@ -483,8 +543,11 @@ Stated rather than left for rediscovery: CodeQL toolchain break sit red for weeks. Deduped by title; re-failures comment on the same issue. Mechanism differs by necessity: govulncheck and grype carry an in-job step, while CodeQL and Semgrep are watched by - `scan-cron-alarm.yml` (a `workflow_run` watcher) — because a CALLED workflow - may not request permissions its caller did not grant, and the check fires at - plan time before any `if:` can skip the job. Learned by breaking it: an + `scan-cron-alarm.yml` (a `workflow_run` watcher, which also covers the + `E2E canary (real model)` lane — three workflows, not two) — because a CALLED + workflow may not request permissions its caller did not grant, and the check + fires at plan time before any `if:` can skip the job. In `govulncheck` and + `grype` the alarm is a separate JOB rather than a step, holding `issues: write` + on its own so the scan itself does not run beside that scope. Learned by breaking it: an `issues: write` alarm job inside the called workflows startup-failed the entire calling Dev CI run. diff --git a/docs/TESTING.md b/docs/TESTING.md index 83336744..aec5565e 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -421,8 +421,18 @@ make ci-go ## Web lint / test / build — CI job `web` -Runs from `web/`. The job is: `npm ci` → `npm run lint` (ESLint) → -`npx vitest run` (unit tests) → `npm run build` (`next build`). +Runs from `web/`. The job is: `npm audit --audit-level=low` (web) → +`npm audit --audit-level=low` (`scripts/rampart-service`) → +`scripts/check-npm-overrides.sh` → `npm ci` → `npm run lint` (**oxlint**, not +ESLint — see the TypeScript 7 section above, which explains why ESLint was +replaced) → `npm run typecheck` (`tsc --noEmit`) → `npx vitest run` (unit tests) +→ `npm run build` (`next build`). + +The two audits and the override canary run **before** `npm ci`, deliberately: +they are lockfile-only, so they cost seconds and fail before the expensive +install. The explicit typecheck is not redundant with `next build` — the build +type-checks too, but it runs last, so without this step a one-line type error +surfaces minutes in. ```sh cd web From df9602b7496c7046f4cb15b541986505eb63c83e Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sun, 23 Aug 2026 00:54:37 +0000 Subject: [PATCH 31/34] Kubernetes: let a sandbox image serve the bundle's doc reads MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The k8s backend drops the supporting-doc bind mounts — a pod has no host filesystem to bind from — and because the fileop path anchor only trusts roots that are actually mounted, that made `view_file protocols/foo.yaml` a REFUSAL ("fileop root is not inside a sandbox bind mount"), not a miss. For a protocol-driven bundle that is most of the product: the system prompt lists protocols and skills by relative path and the agent reads them on demand. A sandbox image can carry those dirs at the same absolute paths the control plane reads them from — that is what makes the workspace symlinks resolve for bash/run_python — but nothing could tell fleet so. Now something can: sandbox.kubernetes.bundle_docs_in_image (FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE) chart: sandbox.kubernetes.bundleDocsInImage With it, those roots keep their read anchors inside a pod and the file tools work as they do under podman. The declaration cannot widen anything: it re-admits READ-ONLY anchors for roots the operator already configured, the read still executes inside the sandbox, and a write or a writable bind beneath one is refused exactly as before (covered by a fake-apiserver test that runs the real fileops.py). fleet cannot inspect an image, so it is trusted the way sandbox.image and runtime_class are — a wrong declaration degrades to a not-found read, the podman missing-dir behavior. Scope of the declaration is deliberately narrow, and boot logs every decision: - Only the BUNDLE's own doc dirs (personas, protocols, system_prompts, skills) are covered. Other entries in the mount list — the uploads root — are control-plane state no image can contain; they stay dropped, with a reason logged per path. - A materialized skills tree is NEVER covered. Inheriting fleet's built-in pack resolves SkillsDir to $FLEET_DATA_DIR/skills-merged/, which sandbox pods do not mount and no image can reproduce. The log names the fix (skills_builtin: false) instead of leaving an operator to wonder why protocols read and skills do not. No configuration yields both the built-in pack and in-sandbox skill files on this backend; docs/SKILLS.md and the guide's honest-scope list now say so. - A non-boolean value refuses to boot, and `fleet validate-config` runs the same parse plus reports which way it resolved. The keep/drop rule is one pure, total function (k8sDocMounts) so the policy is pinned by tests rather than read out of a boot log. Docs: a new "Bundle docs inside a sandbox pod" section in docs/DEPLOYMENT-KUBERNETES.md (mechanism, the derived-image recipe, the four things to be honest about), the config-reference row, the reworked honest-scope bullets, docs/SKILLS.md, config/default/manifest.yaml, CHANGELOG, and ADR-0049's non-goals — which now records WHY fleet does not synthesize these mounts (a ConfigMap projection or a push into the workspace claim would put bundle content on a writable, agent-reachable surface). Verified: go build/vet clean, gofmt clean, full tagged suite green, `helm lint` + template render with the flag on and off. golangci-lint was not run locally — the installed binary predates this repo's Go 1.27 target and refuses to load the config; CI's lane covers it. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01WbgdPXBYGKKBrykZukQV9N Signed-off-by: Brad Flaugher --- CHANGELOG.md | 30 +++++ cmd/fleet/main.go | 7 ++ cmd/fleet/sandbox_backend_resolve_test.go | 91 ++++++++++++++ cmd/fleet/validate_config.go | 18 ++- config/default/manifest.yaml | 8 ++ deploy/helm/fleet/templates/deployment.yaml | 4 + deploy/helm/fleet/values.yaml | 12 ++ docs/DEPLOYMENT-KUBERNETES.md | 70 ++++++++++- docs/SKILLS.md | 12 ++ ...-kubernetes-backend-split-control-plane.md | 17 ++- internal/agent/k8s_doc_mounts_test.go | 76 ++++++++++++ internal/agent/manager.go | 92 ++++++++++++-- internal/clientconfig/builtin_skills.go | 21 ++++ internal/clientconfig/builtin_skills_test.go | 34 ++++++ internal/clientconfig/clientconfig.go | 12 ++ internal/config/config.go | 36 +++--- internal/sandbox/k8s_backend.go | 20 +++ internal/sandbox/k8s_backend_test.go | 115 ++++++++++++++++++ 18 files changed, 643 insertions(+), 32 deletions(-) create mode 100644 cmd/fleet/sandbox_backend_resolve_test.go create mode 100644 internal/agent/k8s_doc_mounts_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 9314ed86..0912d76f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -81,6 +81,36 @@ prior versions are listed because none have shipped. amends ADR-0004: the single-box podman install **stays the default and is unchanged**; only the no-k8s-artifacts enforcement clause is superseded. +- **Kubernetes: bundle docs can serve the file tools again + (`sandbox.kubernetes.bundle_docs_in_image` / + `FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE`).** A sandbox pod mounts only the + workspace claim, so the supporting-doc bind mounts (`protocols/`, + `personas/`, `system_prompts/`, skills) do not apply — and because the + fileop path anchor only trusts roots that are actually mounted, dropping + them made `view_file protocols/foo.yaml` a *refusal* + (`fileop root is not inside a sandbox bind mount`), not a miss. For a + protocol-driven bundle that is most of the product. An operator whose + sandbox image carries the bundle's doc dirs at the **same absolute paths** + the control plane reads them from can now declare it, and those roots keep + their read anchors inside a pod, so the file tools work exactly as they do + under podman. + + The declaration cannot widen anything: it re-admits *read-only* anchors for + roots the operator already configured, the read still executes inside the + sandbox, and fleet cannot inspect an image — so a wrong declaration surfaces + as a not-found read (the podman missing-dir behavior), never as a boundary + change. It covers only the bundle's own doc dirs; other entries in the mount + list (the uploads root) stay dropped, each with a log line. A malformed + value refuses to boot, at boot and in `fleet validate-config`, which also + now reports which way it resolved. + + One case no declaration can fix, now stated plainly in the docs: a bundle + that inherits fleet's built-in skills pack resolves `SkillsDir` to a merged + tree under the control plane's data dir, which no sandbox image can carry — + so in-sandbox skill reads need the bundle's `skills_builtin: false`, and + there is no configuration that yields both the built-in pack and working + in-sandbox skill files. + ### Removed - **`docs/EKS-DEPLOYMENT.md`** — the hand-verified recipe for running the diff --git a/cmd/fleet/main.go b/cmd/fleet/main.go index 78879d1d..c050766f 100644 --- a/cmd/fleet/main.go +++ b/cmd/fleet/main.go @@ -2050,6 +2050,13 @@ func resolveSandboxBackendInto(cfg *config.Config, bundle *clientconfig.Bundle) fill(&cfg.SandboxK8sSeccompProfile, k.SeccompProfile) fill(&cfg.SandboxK8sKubeconfig, k.Kubeconfig) fill(&cfg.SandboxK8sNetworkPolicy, k.NetworkPolicy) + // bundle_docs_in_image is a manifest bool and an env string; canonicalize + // to the env form so the pool build parses one source. Only a manifest + // TRUE needs carrying: false is already the empty-string default, and + // writing "false" here would make an unset env look explicitly disabled. + if strings.TrimSpace(cfg.SandboxK8sBundleDocsInImage) == "" && k.BundleDocsInImage { + cfg.SandboxK8sBundleDocsInImage = "true" + } // The scheduling knobs are structured in the manifest; canonicalize them // into the same string forms the env vars use so the pool build has ONE // source to parse (env wins, like every other field). diff --git a/cmd/fleet/sandbox_backend_resolve_test.go b/cmd/fleet/sandbox_backend_resolve_test.go new file mode 100644 index 00000000..adc5ecdb --- /dev/null +++ b/cmd/fleet/sandbox_backend_resolve_test.go @@ -0,0 +1,91 @@ +package main + +import ( + "os" + "path/filepath" + "testing" + + "github.com/ElcanoTek/fleet/internal/clientconfig" + "github.com/ElcanoTek/fleet/internal/config" + "github.com/ElcanoTek/fleet/internal/sandbox" +) + +// k8sBundle writes a minimal bundle whose sandbox block carries the given +// kubernetes: YAML, so the manifest→config fill can be exercised without a +// cluster. +func k8sBundle(t *testing.T, kubernetesYAML string) *clientconfig.Bundle { + t.Helper() + dir := t.TempDir() + // sandbox.image spares the fixture a Containerfile: the loader insists on + // one only for the build-on-box path. + manifest := "sandbox:\n image: registry.example/fleet-sandbox:test\n backend: kubernetes\n kubernetes:\n" + kubernetesYAML + if err := os.WriteFile(filepath.Join(dir, "manifest.yaml"), []byte(manifest), 0o600); err != nil { + t.Fatalf("write manifest: %v", err) + } + bundle, err := clientconfig.Load(dir) + if err != nil { + t.Fatalf("load bundle: %v", err) + } + return bundle +} + +// bundle_docs_in_image is a manifest bool and an env string. It must follow the +// same env-wins-else-bundle precedence as every other kubernetes setting — +// including the case that motivated the field: an operator turning it OFF from +// the chart for a bundle whose manifest turns it on. +func TestResolveSandboxBackendBundleDocsInImage(t *testing.T) { + tests := []struct { + name string + manifest string + env string + want string + }{ + {name: "manifest on, env unset", manifest: " bundle_docs_in_image: true\n", want: "true"}, + {name: "manifest on, env off wins", manifest: " bundle_docs_in_image: true\n", env: "false", want: "false"}, + {name: "manifest off, env unset stays empty", manifest: " bundle_docs_in_image: false\n", want: ""}, + {name: "manifest silent, env on wins", manifest: " workspace_claim: ws\n", env: "true", want: "true"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + bundle := k8sBundle(t, tc.manifest) + cfg := &config.Config{SandboxK8sBundleDocsInImage: tc.env} + if err := resolveSandboxBackendInto(cfg, bundle); err != nil { + t.Fatalf("resolveSandboxBackendInto: %v", err) + } + if cfg.SandboxK8sBundleDocsInImage != tc.want { + t.Errorf("SandboxK8sBundleDocsInImage = %q, want %q", cfg.SandboxK8sBundleDocsInImage, tc.want) + } + // Whatever the resolved string, it must parse — the pool build + // treats a parse error as a boot failure. + if _, err := sandbox.ParseK8sBundleDocsInImage(cfg.SandboxK8sBundleDocsInImage); err != nil { + t.Errorf("resolved value does not parse: %v", err) + } + }) + } +} + +// Under the podman backend the kubernetes block is not consulted at all: a +// bundle that declares baked-in docs for its cluster deployment must not have +// that leak into a single-box install, where the real bind mounts apply. +func TestResolveSandboxBackendPodmanIgnoresKubernetesBlock(t *testing.T) { + dir := t.TempDir() + manifest := "sandbox:\n image: registry.example/fleet-sandbox:test\n kubernetes:\n bundle_docs_in_image: true\n workspace_claim: ws\n" + if err := os.WriteFile(filepath.Join(dir, "manifest.yaml"), []byte(manifest), 0o600); err != nil { + t.Fatalf("write manifest: %v", err) + } + bundle, err := clientconfig.Load(dir) + if err != nil { + t.Fatalf("load bundle: %v", err) + } + cfg := &config.Config{} + if err := resolveSandboxBackendInto(cfg, bundle); err != nil { + t.Fatalf("resolveSandboxBackendInto: %v", err) + } + if cfg.SandboxBackend != sandbox.BackendPodman { + t.Fatalf("SandboxBackend = %q, want podman (the default)", cfg.SandboxBackend) + } + if cfg.SandboxK8sBundleDocsInImage != "" || cfg.SandboxK8sWorkspaceClaim != "" { + t.Errorf("kubernetes settings leaked into the podman path: docs=%q claim=%q", + cfg.SandboxK8sBundleDocsInImage, cfg.SandboxK8sWorkspaceClaim) + } +} diff --git a/cmd/fleet/validate_config.go b/cmd/fleet/validate_config.go index ba63804f..b2dd41c2 100644 --- a/cmd/fleet/validate_config.go +++ b/cmd/fleet/validate_config.go @@ -844,6 +844,16 @@ func checkKubernetesSandbox(ctx context.Context, res checkResult, cfg *config.Co } tolerations = parsed } + docsInImage := k8s.BundleDocsInImage + if strings.TrimSpace(cfg.SandboxK8sBundleDocsInImage) != "" { + parsed, err := sandbox.ParseK8sBundleDocsInImage(cfg.SandboxK8sBundleDocsInImage) + if err != nil { + res.Status = statusFail + res.Detail = "FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE: " + err.Error() + return res + } + docsInImage = parsed + } fill := func(env, bundleVal string) string { if strings.TrimSpace(env) != "" { return strings.TrimSpace(env) @@ -873,7 +883,13 @@ func checkKubernetesSandbox(ctx context.Context, res checkResult, cfg *config.Co return res } res.Status = statusOK - res.Detail = fmt.Sprintf("kubernetes backend ok; image %q, sandbox namespace %q (image pullability is checked at first pod start)", image, backend.Namespace()) + // bundle-doc reads are the one behavior an operator cannot infer from the + // cluster state this check just proved, so it is reported either way. + docs := "bundle docs NOT in the sandbox image — in-sandbox protocol/skill reads will not resolve" + if docsInImage { + docs = "bundle docs declared present in the sandbox image (unverifiable here — a wrong declaration reads as not-found)" + } + res.Detail = fmt.Sprintf("kubernetes backend ok; image %q, sandbox namespace %q (image pullability is checked at first pod start); %s", image, backend.Namespace(), docs) return res } diff --git a/config/default/manifest.yaml b/config/default/manifest.yaml index 4637ff58..05817f0f 100644 --- a/config/default/manifest.yaml +++ b/config/default/manifest.yaml @@ -46,6 +46,14 @@ sandbox: # network_policy: fleet-sandbox-deny-all # node_selector: {} # pin sandbox pods to a dedicated runner pool # tolerations: [] # [{key,operator,value,effect}] for a tainted pool + # # Declare that the SANDBOX IMAGE carries this bundle's protocols/, + # # personas/, system_prompts/ and skills/ at the same absolute paths the + # # control plane reads them from (a pod mounts only the workspace claim, + # # so that is the only way it can see them). Off = fleet drops those + # # fileop anchors and view_file on `protocols/…` is refused; on = the + # # file tools work again. Unverifiable by fleet — a wrong declaration + # # reads as not-found, never as a wider boundary. + # bundle_docs_in_image: false branding: app_name: "Fleet" diff --git a/deploy/helm/fleet/templates/deployment.yaml b/deploy/helm/fleet/templates/deployment.yaml index b77f3e0b..20a18e50 100644 --- a/deploy/helm/fleet/templates/deployment.yaml +++ b/deploy/helm/fleet/templates/deployment.yaml @@ -95,6 +95,10 @@ spec: - name: FLEET_SANDBOX_K8S_SECCOMP_PROFILE value: {{ . | quote }} {{- end }} + {{- if .Values.sandbox.kubernetes.bundleDocsInImage }} + - name: FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE + value: "true" + {{- end }} {{- with .Values.sandbox.kubernetes.nodeSelector }} - name: FLEET_SANDBOX_K8S_NODE_SELECTOR value: {{ $pairs := list }}{{- range $k, $v := . }}{{- $pairs = append $pairs (printf "%s=%s" $k $v) }}{{- end }}{{ join "," $pairs | quote }} diff --git a/deploy/helm/fleet/values.yaml b/deploy/helm/fleet/values.yaml index a8550ad5..5b2a1035 100644 --- a/deploy/helm/fleet/values.yaml +++ b/deploy/helm/fleet/values.yaml @@ -61,6 +61,18 @@ sandbox: # Node-local seccomp profile applied as a Localhost profile (path # relative to the kubelet seccomp root). Empty = RuntimeDefault. seccompProfile: "" + # Declare that your SANDBOX IMAGE carries the client bundle's doc dirs + # (protocols/, personas/, system_prompts/, skills/) at the SAME absolute + # paths the control plane reads them from — i.e. you built the sandbox + # image with the bundle baked in. A pod mounts only the workspace claim, + # so without this fleet drops those roots and view_file on `protocols/…` + # is refused; with it the file tools work again. fleet cannot verify an + # image's contents: a wrong declaration shows up as a not-found read. + # Bundles that inherit fleet's built-in skills pack cannot serve + # in-sandbox skill reads either way (the merged tree lives on the + # control-plane data PVC) — set skills_builtin: false in the bundle to + # make skills/ bake-able. See docs/DEPLOYMENT-KUBERNETES.md. + bundleDocsInImage: false # Pin sandbox pods to a DEDICATED runner node pool: label the pool and # select it here; taint it and add the matching toleration so nothing # else lands on it. This is fleet's horizontal scaling story — more diff --git a/docs/DEPLOYMENT-KUBERNETES.md b/docs/DEPLOYMENT-KUBERNETES.md index 42a97acd..82cd53f2 100644 --- a/docs/DEPLOYMENT-KUBERNETES.md +++ b/docs/DEPLOYMENT-KUBERNETES.md @@ -234,6 +234,61 @@ spec: persistentVolumeClaim: {claimName: fleet-data} ``` +## Bundle docs inside a sandbox pod + +A bundle's `protocols/`, `personas/`, `system_prompts/` and `skills/` are how a +protocol-driven deployment works at all: the system prompt lists them by +relative path and the agent reads them on demand. Under podman fleet +bind-mounts each read-only at its own absolute path and symlinks them into the +per-conversation workspace, so `protocols/foo.yaml` resolves for `view_file`, +`bash` and `run_python` alike. + +A sandbox pod mounts only the workspace claim. There is no host filesystem to +bind from, so by default fleet drops those roots — and because the fileop path +anchor only trusts roots that are actually mounted, `view_file +protocols/foo.yaml` is *refused* (`fileop root is not inside a sandbox bind +mount`) rather than attempted. The workspace symlinks still point at the +bundle's absolute paths, so `bash`/`run_python` reads fail too, as not-found. + +The fix is the sandbox image. Build it with the bundle's doc dirs baked in at +the **same absolute paths** the control plane uses (`FLEET_CLIENT_CONFIG_DIR`), +then declare it: + +```yaml +sandbox: + kubernetes: + bundleDocsInImage: true # chart values → FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE +``` + +```dockerfile +# derived sandbox image; same paths as the control plane's bundle +FROM REGISTRY/fleet-sandbox:v1 +USER root +COPY protocols/ personas/ system_prompts/ skills/ /opt/fleet/client/... +RUN chown -R 0:0 /opt/fleet/client && chmod -R a-w,a+rX /opt/fleet/client +USER 1000 +``` + +With the declaration, the anchors for those roots stay valid, so the file tools +read them out of the pod's image layer and the symlinked relative paths work +for bash and python. Four things to be honest about: + +- **It is a declaration, not a probe.** fleet cannot inspect an image's + contents. It also cannot widen anything: the flag only re-admits *read-only* + anchors for roots the operator already configured, and the read still runs + inside the sandbox. A wrong declaration surfaces as a not-found read. +- **Only the bundle's own doc dirs are covered.** Other entries in the mount + list (the uploads root) live in control-plane state no image can contain; + they stay dropped, with a log line each. +- **The merged skills tree is never covered** — see the honest-scope list. +- **The baked copy is a snapshot.** Build and roll the control-plane and + sandbox images from the same bundle commit, or the agent reads one release's + protocols while the control plane runs another's. Nothing enforces this. + +Boot logs which roots survived and which were dropped, and why. `kubectl logs +deploy/fleet | grep 'bundle_docs_in_image\|supporting-doc'` is the fastest way +to see what a running deployment decided. + ## Provider notes - **EKS**: EFS (via the EFS CSI driver) is the standard RWX workspace class; @@ -264,6 +319,7 @@ Every knob can come from env (the chart sets these) or the bundle manifest's | `FLEET_SANDBOX_K8S_SECCOMP_PROFILE` | `…seccomp_profile` | node-local Localhost seccomp profile; empty = RuntimeDefault | | `FLEET_SANDBOX_K8S_KUBECONFIG` | `…kubeconfig` | out-of-cluster auth (token / client-cert kubeconfigs only); empty = in-cluster | | `FLEET_SANDBOX_K8S_NETWORK_POLICY` | `…network_policy` | deny-all policy name the preflight requires (default `fleet-sandbox-deny-all`) | +| `FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE` | `…bundle_docs_in_image` | the sandbox image carries the bundle's doc dirs at the same absolute paths — keeps their fileop read anchors valid in a pod ([above](#bundle-docs-inside-a-sandbox-pod)); a non-boolean refuses to boot | | `FLEET_SANDBOX_K8S_NODE_SELECTOR` | `…node_selector` | pin sandbox pods to a dedicated runner pool — env form `"pool=sandboxes,arch=amd64"`, manifest form a map; a malformed value refuses to boot | | `FLEET_SANDBOX_K8S_TOLERATIONS` | `…tolerations` | tolerations for a tainted runner pool — env form a JSON array of `{key,operator,value,effect}`, manifest form a YAML list | @@ -320,10 +376,18 @@ Recorded here so nobody discovers them in production: `FLEET_SANDBOX_K8S_SECCOMP_PROFILE`. Setting the podman `FLEET_SANDBOX_SECCOMP_PROFILE` under this backend refuses to boot rather than being silently ignored. -- **Supporting-doc bind mounts don't apply.** The podman backend bind-mounts +- **Supporting-doc bind mounts don't apply** — the podman backend bind-mounts persona/protocol dirs same-path into containers; a pod only mounts the - workspace claim. In-sandbox reads of those host paths degrade exactly like - the podman missing-dir case. + workspace claim. Bake them into the sandbox image and declare it + (`sandbox.kubernetes.bundle_docs_in_image`) to get the reads back; see + [Bundle docs inside a sandbox pod](#bundle-docs-inside-a-sandbox-pod). + Undeclared, in-sandbox reads of those paths do not resolve at all. +- **A bundle inheriting fleet's built-in skills pack cannot serve in-sandbox + skill reads**, declaration or not: the merged tree is materialized under the + control plane's data dir, so no sandbox image can carry it. `skills_builtin: + false` in the bundle manifest makes `skills/` the bundle's own (bake-able) + dir at the cost of the built-in pack. There is no setting that gives you + both. - **Disk quota is per-pod ephemeral storage**, which caps the writable layer and scratch emptyDirs — a *stronger* cap than podman's per-file ulimit — but the workspace claim is still unbounded by it, same as the bind mount is diff --git a/docs/SKILLS.md b/docs/SKILLS.md index 4c7a2630..20a83afe 100644 --- a/docs/SKILLS.md +++ b/docs/SKILLS.md @@ -151,6 +151,18 @@ supporting-doc read exception. Treat bundle skills as an interactive-chat capability; a scheduled task that needs one should inline the instructions in its prompt. +**On the kubernetes sandbox backend, inheriting the built-in pack costs +in-sandbox skill files.** `SkillsDir` is then the merged tree under the control +plane's data dir, which sandbox pods do not mount and no sandbox image can +carry (its name is derived from the bundle path, and it is rebuilt at boot) — so +`skills//SKILL.md` resolves for neither the file tools nor bash, and the +roster degrades to name + description. `skills_builtin: false` makes `SkillsDir` +the bundle's own `skills/`, which an operator CAN bake into the sandbox image +and declare with `sandbox.kubernetes.bundle_docs_in_image` +([DEPLOYMENT-KUBERNETES.md](DEPLOYMENT-KUBERNETES.md#bundle-docs-inside-a-sandbox-pod)). +There is no setting that gives you both the built-in pack and working +in-sandbox skill files on that backend. + Manifest knobs (mirroring the MCP directory): ```yaml diff --git a/docs/adr/0049-kubernetes-backend-split-control-plane.md b/docs/adr/0049-kubernetes-backend-split-control-plane.md index f1a26942..2c44597b 100644 --- a/docs/adr/0049-kubernetes-backend-split-control-plane.md +++ b/docs/adr/0049-kubernetes-backend-split-control-plane.md @@ -97,9 +97,20 @@ privileged node" packaging track as a stepping stone. (fail-closed) instead of silently granting open egress. Cluster-side egress shaping via NetworkPolicy is the replacement. - Per-pod pids limits (not expressible in a Pod spec), the bundled seccomp - JSON (nodes take a Localhost profile instead), `podman stats` resource - telemetry (#263), and same-path supporting-doc bind mounts — each recorded - as an honest deviation in `docs/DEPLOYMENT-KUBERNETES.md`. + JSON (nodes take a Localhost profile instead), and `podman stats` resource + telemetry (#263) — each recorded as an honest deviation in + `docs/DEPLOYMENT-KUBERNETES.md`. +- Same-path supporting-doc bind mounts: a pod has no host filesystem to bind + from. fleet does not synthesize them (no ConfigMap projection, no + control-plane push into the workspace claim — both would put bundle content + on a writable, agent-reachable surface). Instead the sandbox IMAGE may carry + the bundle's doc dirs at the same absolute paths, and + `sandbox.kubernetes.bundle_docs_in_image` declares that, which keeps those + roots' **read-only** fileop anchors valid inside a pod. A declaration, not a + probe: fleet cannot inspect an image, so it is trusted the way + `sandbox.image` and `runtime_class` are — and it can only re-admit reads of + operator-configured paths, executed inside the sandbox, so a wrong + declaration degrades to not-found rather than widening any boundary. ## Consequences diff --git a/internal/agent/k8s_doc_mounts_test.go b/internal/agent/k8s_doc_mounts_test.go new file mode 100644 index 00000000..4f7ce4ce --- /dev/null +++ b/internal/agent/k8s_doc_mounts_test.go @@ -0,0 +1,76 @@ +package agent + +import ( + "path/filepath" + "testing" +) + +// The kubernetes backend keeps or drops each supporting-doc root's fileop +// anchor by one rule set (k8sDocMounts). These tests pin it, because the +// consequence of getting it wrong is invisible until an agent tries to read a +// protocol: too permissive and the anchor trusts a path no pod has, too strict +// and view_file refuses a file the sandbox image really does carry. +func TestK8sDocMountsDropsEverythingWithoutTheDeclaration(t *testing.T) { + bundle := []string{"/opt/fleet/client/personas", "/opt/fleet/client/protocols", "/opt/fleet/client/system_prompts", "/opt/fleet/client/skills"} + mounts := append(append([]string{}, bundle...), "/var/lib/fleet/uploads") + + kept, dropped := k8sDocMounts(mounts, bundle, false) + if len(kept) != 0 { + t.Errorf("kept = %v; a pod mounts only the workspace claim, so nothing may keep its anchor", kept) + } + if len(dropped) != len(mounts) { + t.Errorf("dropped = %v; want all %d mounts", dropped, len(mounts)) + } +} + +func TestK8sDocMountsKeepsOnlyBundleDocsWithTheDeclaration(t *testing.T) { + bundle := []string{"/opt/fleet/client/personas", "/opt/fleet/client/protocols", "/opt/fleet/client/system_prompts", "/opt/fleet/client/skills"} + uploads := "/var/lib/fleet/uploads" + mounts := append(append([]string{}, bundle...), uploads) + + kept, dropped := k8sDocMounts(mounts, bundle, true) + if len(kept) != len(bundle) { + t.Fatalf("kept = %v; want the %d bundle doc dirs", kept, len(bundle)) + } + for i, want := range bundle { + if kept[i] != want { + t.Errorf("kept[%d] = %q, want %q (order preserved)", i, kept[i], want) + } + } + // The uploads root is control-plane state; no sandbox image contains it, + // and the declaration says nothing about it. + if len(dropped) != 1 || dropped[0] != uploads { + t.Errorf("dropped = %v; want only %q", dropped, uploads) + } +} + +func TestK8sDocMountsNeverKeepsAMaterializedSkillsTree(t *testing.T) { + // The merged built-in + bundle skills tree lives under the control plane's + // data dir with a hash-derived name — a sandbox image cannot carry it, so + // the declaration must not extend to it even though it IS the bundle's + // resolved skills dir. + merged := filepath.Join("/var/lib/fleet", "skills-merged", "f693617985b1") + bundle := []string{"/opt/fleet/client/protocols", merged} + mounts := bundle + + kept, dropped := k8sDocMounts(mounts, bundle, true) + if len(kept) != 1 || kept[0] != "/opt/fleet/client/protocols" { + t.Errorf("kept = %v; want only the bundle-path protocols dir", kept) + } + if len(dropped) != 1 || dropped[0] != merged { + t.Errorf("dropped = %v; want the merged skills tree %q", dropped, merged) + } +} + +func TestK8sDocMountsIgnoresUnlistedAndEmptyPaths(t *testing.T) { + bundle := []string{"/opt/fleet/client/protocols/"} // trailing slash, same dir + mounts := []string{"", "/opt/fleet/client/protocols", "/somewhere/else"} + + kept, dropped := k8sDocMounts(mounts, bundle, true) + if len(kept) != 1 || kept[0] != "/opt/fleet/client/protocols" { + t.Errorf("kept = %v; want the protocols dir matched after path cleaning", kept) + } + if len(dropped) != 1 || dropped[0] != "/somewhere/else" { + t.Errorf("dropped = %v; want only the path that is not a bundle doc dir", dropped) + } +} diff --git a/internal/agent/manager.go b/internal/agent/manager.go index 553d83b2..4faa7e8a 100644 --- a/internal/agent/manager.go +++ b/internal/agent/manager.go @@ -16,6 +16,7 @@ import ( "github.com/ElcanoTek/fleet/internal/admission" "github.com/ElcanoTek/fleet/internal/agentcore" + "github.com/ElcanoTek/fleet/internal/clientconfig" "github.com/ElcanoTek/fleet/internal/config" "github.com/ElcanoTek/fleet/internal/creds" "github.com/ElcanoTek/fleet/internal/mcp" @@ -586,7 +587,8 @@ func buildSandboxPool(cfg *config.Config, personasDir, protocolsDir, systemPromp // below (bridge-file prune, OCI-runtime preflight, egress proxy) is // replaced by the backend's own fail-closed cluster preflight. if cfg.SandboxBackend == sandbox.BackendKubernetes { - return buildKubernetesSandboxPool(cfg, poolCfg, sandboxRuntime) + return buildKubernetesSandboxPool(cfg, poolCfg, sandboxRuntime, + absSupportingDocs(personasDir, protocolsDir, systemPromptsDir, skillsDir)) } // Reclaim bridge-script/seccomp temp files orphaned by a PRIOR crash: only // the graceful close path removes them, so without this sweep every @@ -658,7 +660,12 @@ func buildSandboxPool(cfg *config.Config, personasDir, protocolsDir, systemPromp // silently ignored (a configured-but-inert security knob is the failure mode // ADR-0010's no-degrade rule exists for), builds the backend handle, and runs // the fail-closed cluster preflight before the warm pool spawns its first pod. -func buildKubernetesSandboxPool(cfg *config.Config, poolCfg sandbox.PoolConfig, sandboxRuntime string) (*sandbox.Pool, error) { +// +// bundleDocDirs are the bundle's own supporting-doc roots (personas, +// protocols, system_prompts, skills) — the subset of +// poolCfg.Container.ReadOnlyMounts a sandbox IMAGE could plausibly carry, and +// therefore the only ones bundle_docs_in_image can vouch for. +func buildKubernetesSandboxPool(cfg *config.Config, poolCfg sandbox.PoolConfig, sandboxRuntime string, bundleDocDirs []string) (*sandbox.Pool, error) { if sandboxRuntime != "" { return nil, fmt.Errorf( "FLEET_SANDBOX_RUNTIME=%q is a podman OCI-runtime knob and has no effect under FLEET_SANDBOX_BACKEND=kubernetes; "+ @@ -674,15 +681,42 @@ func buildKubernetesSandboxPool(cfg *config.Config, poolCfg sandbox.PoolConfig, "FLEET_DEFAULT_NETWORK_MODE=allowlisted is not supported under FLEET_SANDBOX_BACKEND=kubernetes: the host-side egress proxy " + "is unreachable from sandbox pods. Use lockdown (sealed by the deny-all NetworkPolicy) or open, and shape egress with cluster NetworkPolicies (fail-closed)") } - // Supporting-doc mounts are same-path HOST bind mounts; a pod has no host - // filesystem to bind them from, and leaving them in the config would make - // the fileop anchor logic trust paths that are not actually mounted. - // Bundles relying on in-sandbox persona/protocol reads degrade exactly - // like the podman missing-dir case (skipped; host-path view_file still - // works via the workspace when the bundle lives under it). - if len(poolCfg.Container.ReadOnlyMounts) > 0 { - log.Printf("sandbox: kubernetes backend — supporting-doc bind mounts do not apply (pods mount only the workspace claim); in-sandbox reads of %d host dir(s) will not resolve", len(poolCfg.Container.ReadOnlyMounts)) - poolCfg.Container.ReadOnlyMounts = nil + // Supporting-doc mounts are same-path HOST bind mounts, and a pod has no + // host filesystem to bind them from — so by default they are dropped, and + // the fileop anchor then refuses those roots rather than trusting paths + // nothing mounted. A sandbox IMAGE can still carry the bundle's doc dirs + // at the same absolute paths (that is how bash/run_python keep resolving + // `protocols/…` through the workspace symlinks); an operator who built + // such an image declares it with bundle_docs_in_image, and the anchors for + // those roots stay valid so the FILE TOOLS work too. + // + // The declaration cannot be probed — fleet does not inspect image + // contents — but it cannot widen anything either: it only re-admits + // read-only anchors for operator-configured bundle paths, and the reads + // still execute inside the sandbox. A wrong declaration surfaces as a + // not-found read, which is the podman missing-dir behavior. + docsInImage, err := sandbox.ParseK8sBundleDocsInImage(cfg.SandboxK8sBundleDocsInImage) + if err != nil { + return nil, fmt.Errorf("FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE / sandbox.kubernetes.bundle_docs_in_image: %w", err) + } + kept, dropped := k8sDocMounts(poolCfg.Container.ReadOnlyMounts, bundleDocDirs, docsInImage) + poolCfg.Container.ReadOnlyMounts = kept + if len(kept) > 0 { + //nolint:gosec // G706: operator-configured bundle paths from the manifest/env, not request input. + log.Printf("sandbox: kubernetes backend — bundle_docs_in_image declared: keeping fileop read anchors for %d bundle doc root(s) %v; the SANDBOX IMAGE must carry them at these exact paths or reads fail not-found", len(kept), kept) + } + for _, d := range dropped { + switch { + case clientconfig.IsMaterializedSkillsDir(d): + //nolint:gosec // G706: as above — a boot-resolved bundle path. + log.Printf("sandbox: kubernetes backend — skills dir %q is the merged built-in+bundle tree under the control plane's data dir, which no sandbox image can carry; in-sandbox skill reads will not resolve. Set skills_builtin: false in the bundle manifest to make skills/ the bundle's own (bake-able) dir", d) + case docsInImage: + //nolint:gosec // G706: as above. + log.Printf("sandbox: kubernetes backend — %q is not a bundle doc dir, so bundle_docs_in_image does not vouch for it; in-sandbox reads there will not resolve", d) + } + } + if !docsInImage && len(dropped) > 0 { + log.Printf("sandbox: kubernetes backend — supporting-doc bind mounts do not apply (pods mount only the workspace claim); in-sandbox reads of %d host dir(s) will not resolve. If your sandbox image carries the bundle's doc dirs at the same paths, set sandbox.kubernetes.bundle_docs_in_image (FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE=true)", len(dropped)) } poolCfg.Container.Runtime = "" @@ -738,6 +772,42 @@ func buildKubernetesSandboxPool(cfg *config.Config, poolCfg sandbox.PoolConfig, return sandbox.NewPool(poolCfg), nil } +// k8sDocMounts splits the supporting-doc mount list into the roots whose +// fileop anchors survive under the kubernetes backend and the roots that are +// dropped, given the operator's bundle_docs_in_image declaration. +// +// Pure and total so the policy is pinned by tests rather than read out of the +// boot log. Three rules, in order: +// +// - Without the declaration, nothing survives: a pod mounts only the +// workspace claim, so every host path is a path the anchor must not trust. +// - With it, only the BUNDLE's own doc dirs survive. Everything else in the +// list (the uploads root) lives in control-plane state a sandbox image +// cannot contain, and the declaration says nothing about it. +// - A materialized (merged built-in + bundle) skills tree never survives: +// it lives under the data dir with a path derived from the bundle path, so +// no image can carry it. See clientconfig.IsMaterializedSkillsDir. +func k8sDocMounts(mounts, bundleDocDirs []string, docsInImage bool) (kept, dropped []string) { + bundle := make(map[string]bool, len(bundleDocDirs)) + for _, d := range bundleDocDirs { + if d != "" { + bundle[filepath.Clean(d)] = true + } + } + for _, m := range mounts { + if m == "" { + continue + } + clean := filepath.Clean(m) + if docsInImage && bundle[clean] && !clientconfig.IsMaterializedSkillsDir(clean) { + kept = append(kept, m) + continue + } + dropped = append(dropped, m) + } + return kept, dropped +} + // absSupportingDocs absolutizes the persona/protocol/skill/system-prompt dirs // (plus the uploads root) and drops empties so they can be passed as // ContainerConfig.ReadOnlyMounts. The container backend bind-mounts each at the diff --git a/internal/clientconfig/builtin_skills.go b/internal/clientconfig/builtin_skills.go index f0d20328..bac26bf1 100644 --- a/internal/clientconfig/builtin_skills.go +++ b/internal/clientconfig/builtin_skills.go @@ -98,6 +98,27 @@ func ensureTrustedDir(path string) error { return verifyExistingDir(path) } +// IsMaterializedSkillsDir reports whether dir is a merged tree this package +// materialized (`/skills-merged/`) rather than a bundle's +// own `skills/`. It exists for one caller: the kubernetes sandbox backend, +// where a supporting-doc dir is only readable in a sandbox if the sandbox +// IMAGE carries it at the same absolute path — and a merged tree lives under +// the control plane's data dir, which no sandbox image can plausibly reproduce +// (its hash is derived from the bundle path, and the tree is rebuilt at boot). +// So a bundle inheriting the built-in pack can never serve in-sandbox skill +// reads on that backend; the caller drops the mount and says so, and the fix +// is the bundle's `skills_builtin: false`. +// +// Shape-based on purpose: the layout is this package's own convention, so the +// check belongs here, next to the code that builds the path. +func IsMaterializedSkillsDir(dir string) bool { + dir = strings.TrimSpace(dir) + if dir == "" { + return false + } + return filepath.Base(filepath.Dir(filepath.Clean(dir))) == mergedSkillsDirName +} + // verifyExistingDir is the ssh-style ownership/mode check: Lstat (so a // symlink is not followed), must be a directory we own, must not be // group- or world-writable. diff --git a/internal/clientconfig/builtin_skills_test.go b/internal/clientconfig/builtin_skills_test.go index f5af259e..41b86e96 100644 --- a/internal/clientconfig/builtin_skills_test.go +++ b/internal/clientconfig/builtin_skills_test.go @@ -247,3 +247,37 @@ func TestMaterializeMergedSkills_DoesNotAdoptUntrustedPath(t *testing.T) { t.Fatalf("adopted %q instead of falling back to the bundle dir %q", got, bundle) } } + +// IsMaterializedSkillsDir tells a merged tree apart from a bundle's own +// skills/ — the distinction the kubernetes sandbox backend needs, because only +// a bundle-path skills dir can be baked into a sandbox image. Pinned against +// the real materialized path so the shape check cannot drift from the builder. +func TestIsMaterializedSkillsDir(t *testing.T) { + data := t.TempDir() + t.Setenv("FLEET_DATA_DIR", data) + bundleSkills := filepath.Join(t.TempDir(), "skills") + merged, err := materializeMergedSkills(bundleSkills, true, nil) + if err != nil { + t.Fatalf("materialize: %v", err) + } + if !IsMaterializedSkillsDir(merged) { + t.Errorf("IsMaterializedSkillsDir(%q) = false; the path materializeMergedSkills built must be recognized", merged) + } + if IsMaterializedSkillsDir(bundleSkills) { + t.Errorf("IsMaterializedSkillsDir(%q) = true; a bundle's own skills/ is bake-able", bundleSkills) + } + // Opting out of the built-in pack returns the bundle dir itself, which is + // exactly the case an operator reaches for on the kubernetes backend. + own, err := materializeMergedSkills(bundleSkills, false, nil) + if err != nil { + t.Fatalf("materialize (builtins off): %v", err) + } + if IsMaterializedSkillsDir(own) { + t.Errorf("with skills_builtin: false the resolved dir %q must not read as materialized", own) + } + for _, in := range []string{"", " ", "/opt/fleet/client/skills", "/var/lib/fleet/skills-merged"} { + if IsMaterializedSkillsDir(in) { + t.Errorf("IsMaterializedSkillsDir(%q) = true; want false", in) + } + } +} diff --git a/internal/clientconfig/clientconfig.go b/internal/clientconfig/clientconfig.go index e61c8929..f0c46d25 100644 --- a/internal/clientconfig/clientconfig.go +++ b/internal/clientconfig/clientconfig.go @@ -459,6 +459,18 @@ type KubernetesSandbox struct { // NetworkPolicy is the deny-all NetworkPolicy name the boot preflight // requires to exist (default "fleet-sandbox-deny-all"). NetworkPolicy string `yaml:"network_policy"` + // BundleDocsInImage declares that the sandbox IMAGE carries this bundle's + // supporting-doc dirs (protocols/, personas/, system_prompts/, skills/) at + // the SAME absolute paths the control plane reads them from — the only way + // a pod can see them, since it mounts just the workspace claim. Set it and + // the fileop path anchors for those roots stay valid inside a pod, so + // view_file works on `protocols/…` again; leave it false (the default) and + // the anchors are dropped, which is what refuses those reads. A + // declaration, not a probe: fleet cannot inspect an image's contents, so a + // wrong declaration surfaces as a not-found read, never as a widened + // boundary (reads only, still read-only, still inside the sandbox). + // FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE overrides it. + BundleDocsInImage bool `yaml:"bundle_docs_in_image"` // NodeSelector pins sandbox pods to labeled nodes (a dedicated runner // pool). FLEET_SANDBOX_K8S_NODE_SELECTOR ("k=v,k=v") overrides it. NodeSelector map[string]string `yaml:"node_selector"` diff --git a/internal/config/config.go b/internal/config/config.go index c79c2271..5a449acc 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -392,6 +392,7 @@ var allowedEnvVars = map[string]bool{ "FLEET_SANDBOX_K8S_SECCOMP_PROFILE": true, "FLEET_SANDBOX_K8S_KUBECONFIG": true, "FLEET_SANDBOX_K8S_NETWORK_POLICY": true, + "FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE": true, "FLEET_SANDBOX_K8S_NODE_SELECTOR": true, "FLEET_SANDBOX_K8S_TOLERATIONS": true, "FLEET_DEFAULT_NETWORK_MODE": true, @@ -1041,8 +1042,14 @@ type Config struct { SandboxK8sSeccompProfile string // FLEET_SANDBOX_K8S_SECCOMP_PROFILE — node-local Localhost profile; empty = RuntimeDefault SandboxK8sKubeconfig string // FLEET_SANDBOX_K8S_KUBECONFIG — out-of-cluster auth; empty = in-cluster SandboxK8sNetworkPolicy string // FLEET_SANDBOX_K8S_NETWORK_POLICY — deny-all policy the preflight requires; default "fleet-sandbox-deny-all" - SandboxK8sNodeSelector string // FLEET_SANDBOX_K8S_NODE_SELECTOR — "key=value,key=value" pinning sandbox pods to a runner pool - SandboxK8sTolerations string // FLEET_SANDBOX_K8S_TOLERATIONS — JSON array of {key,operator,value,effect} for a tainted runner pool + // SandboxK8sBundleDocsInImage declares that the sandbox IMAGE carries the + // bundle's supporting-doc dirs at the same absolute paths the control plane + // reads them from, keeping the fileop anchors for those roots valid inside + // a pod. Raw string, parsed fail-closed where it is consumed (like the + // selector/toleration knobs): empty = false. + SandboxK8sBundleDocsInImage string // FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE + SandboxK8sNodeSelector string // FLEET_SANDBOX_K8S_NODE_SELECTOR — "key=value,key=value" pinning sandbox pods to a runner pool + SandboxK8sTolerations string // FLEET_SANDBOX_K8S_TOLERATIONS — JSON array of {key,operator,value,effect} for a tainted runner pool // PIIRedactionEnabled gates the OPTIONAL PII redaction pass (#450) applied to // tool output before it enters the model context. FLEET_PII_REDACTION_ENABLED, // default false (byte-for-byte unchanged when off). Provider-neutral; the @@ -1557,18 +1564,19 @@ func Load(envFile string) (*Config, error) { SandboxRuntime: getenvFleet("SANDBOX_RUNTIME"), // Sandbox backend (#989). Lower-cased here; validated fail-closed at // boot (resolveSandboxBackend in cmd/fleet) against the bundle value. - SandboxBackend: strings.ToLower(strings.TrimSpace(getenvFleet("SANDBOX_BACKEND"))), - SandboxK8sNamespace: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NAMESPACE")), - SandboxK8sWorkspaceClaim: strings.TrimSpace(getenvFleet("SANDBOX_K8S_WORKSPACE_CLAIM")), - SandboxK8sServiceAccount: strings.TrimSpace(getenvFleet("SANDBOX_K8S_SERVICE_ACCOUNT")), - SandboxK8sImagePullSecret: strings.TrimSpace(getenvFleet("SANDBOX_K8S_IMAGE_PULL_SECRET")), - SandboxK8sRuntimeClass: strings.TrimSpace(getenvFleet("SANDBOX_K8S_RUNTIME_CLASS")), - SandboxK8sSeccompProfile: strings.TrimSpace(getenvFleet("SANDBOX_K8S_SECCOMP_PROFILE")), - SandboxK8sKubeconfig: strings.TrimSpace(getenvFleet("SANDBOX_K8S_KUBECONFIG")), - SandboxK8sNetworkPolicy: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NETWORK_POLICY")), - SandboxK8sNodeSelector: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NODE_SELECTOR")), - SandboxK8sTolerations: strings.TrimSpace(getenvFleet("SANDBOX_K8S_TOLERATIONS")), - DefaultNetworkMode: strings.ToLower(strings.TrimSpace(getenvFleet("DEFAULT_NETWORK_MODE"))), + SandboxBackend: strings.ToLower(strings.TrimSpace(getenvFleet("SANDBOX_BACKEND"))), + SandboxK8sNamespace: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NAMESPACE")), + SandboxK8sWorkspaceClaim: strings.TrimSpace(getenvFleet("SANDBOX_K8S_WORKSPACE_CLAIM")), + SandboxK8sServiceAccount: strings.TrimSpace(getenvFleet("SANDBOX_K8S_SERVICE_ACCOUNT")), + SandboxK8sImagePullSecret: strings.TrimSpace(getenvFleet("SANDBOX_K8S_IMAGE_PULL_SECRET")), + SandboxK8sRuntimeClass: strings.TrimSpace(getenvFleet("SANDBOX_K8S_RUNTIME_CLASS")), + SandboxK8sSeccompProfile: strings.TrimSpace(getenvFleet("SANDBOX_K8S_SECCOMP_PROFILE")), + SandboxK8sKubeconfig: strings.TrimSpace(getenvFleet("SANDBOX_K8S_KUBECONFIG")), + SandboxK8sNetworkPolicy: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NETWORK_POLICY")), + SandboxK8sBundleDocsInImage: strings.TrimSpace(getenvFleet("SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE")), + SandboxK8sNodeSelector: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NODE_SELECTOR")), + SandboxK8sTolerations: strings.TrimSpace(getenvFleet("SANDBOX_K8S_TOLERATIONS")), + DefaultNetworkMode: strings.ToLower(strings.TrimSpace(getenvFleet("DEFAULT_NETWORK_MODE"))), // PII redaction (#450) — optional, default off. PIIRedactionEnabled: lp.getenvFleetBool("PII_REDACTION_ENABLED", false), diff --git a/internal/sandbox/k8s_backend.go b/internal/sandbox/k8s_backend.go index c48633dc..0d841df6 100644 --- a/internal/sandbox/k8s_backend.go +++ b/internal/sandbox/k8s_backend.go @@ -196,6 +196,26 @@ func ParseK8sTolerations(s string) ([]K8sToleration, error) { return out, nil } +// ParseK8sBundleDocsInImage parses the +// FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE form — a boolean declaring that the +// sandbox IMAGE carries the bundle's supporting-doc dirs at the same absolute +// paths the control plane reads them from, so the fileop path anchors for +// those roots stay valid inside a pod (see the ReadOnlyMounts handling in +// internal/agent). Empty input is false (the safe default: fleet assumes +// nothing about a sandbox image's contents). A malformed value is an error — +// a typo'd "ture" must not read as "keep the anchors". +func ParseK8sBundleDocsInImage(s string) (bool, error) { + s = strings.TrimSpace(s) + if s == "" { + return false, nil + } + v, err := strconv.ParseBool(s) + if err != nil { + return false, fmt.Errorf("invalid boolean %q (want true or false)", s) + } + return v, nil +} + // defaultK8sNamespace / defaultK8sNetworkPolicy are the conventions the Helm // chart ships; the backend defaults match so a chart install needs no extra // wiring. diff --git a/internal/sandbox/k8s_backend_test.go b/internal/sandbox/k8s_backend_test.go index c315d4c4..68dfb7f1 100644 --- a/internal/sandbox/k8s_backend_test.go +++ b/internal/sandbox/k8s_backend_test.go @@ -397,6 +397,66 @@ func TestK8sFileOpsThroughRealExecutor(t *testing.T) { } } +// End to end through the fake apiserver + the real fileops.py: when the +// operator declares the bundle's doc dirs are in the sandbox image (the agent +// layer then keeps them in ReadOnlyMounts), a read of one resolves and a write +// beneath it is refused. This is the whole point of +// bundle_docs_in_image — view_file on `protocols/…` working in a pod — and it +// must not come with a write path. +func TestK8sFileOpsReadBundleDocsWhenDeclaredInImage(t *testing.T) { + if !pythonAvailable() { + t.Skip("python3 not available on the test host") + } + // Stands in for the path the sandbox image carries the bundle doc dir at; + // the fake execs on the host, so the same path must exist here. + docs := filepath.Join(t.TempDir(), "client", "protocols") + if err := os.MkdirAll(docs, 0o755); err != nil { + t.Fatal(err) + } + protocol := filepath.Join(docs, "deal-creation.yaml") + if err := os.WriteFile(protocol, []byte("steps: [prepare, confirm, create]\n"), 0o644); err != nil { + t.Fatal(err) + } + + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + cfg := testContainerConfig(t) + cfg.BridgeScript = []byte("# unused\n") + cfg.ReadOnlyMounts = []string{docs} + sb, err := backend.newSandbox(context.Background(), cfg) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + + res, err := sb.RunFileOp(context.Background(), FileOpRequest{Op: FileOpRead, Path: protocol, Root: docs, Limit: 1024}) + if err != nil { + t.Fatalf("read declared bundle doc: %v", err) + } + if !strings.Contains(string(res.Data), "prepare") { + t.Errorf("read back %q", res.Data) + } + // Read-only means read-only: the declaration re-admits reads, never writes. + if _, err := sb.RunFileOp(context.Background(), FileOpRequest{Op: FileOpWrite, Path: protocol, Root: docs, Data: []byte("tampered\n")}); !errors.Is(err, ErrFileOpUnsafePath) { + t.Errorf("write beneath a declared doc root err = %v, want ErrFileOpUnsafePath", err) + } + if err := sb.BindFileOpRoot(context.Background(), docs); !errors.Is(err, ErrFileOpUnsafePath) { + t.Errorf("binding a declared doc root as writable err = %v, want ErrFileOpUnsafePath", err) + } + // Without the declaration the agent layer passes no mounts, and the same + // read is refused by the anchor before any exec. + bare := testContainerConfig(t) + bare.BridgeScript = []byte("# unused\n") + sb2, err := backend.newSandbox(context.Background(), bare) + if err != nil { + t.Fatalf("newSandbox (no mounts): %v", err) + } + defer sb2.Close() + if _, err := sb2.RunFileOp(context.Background(), FileOpRequest{Op: FileOpRead, Path: protocol, Root: docs, Limit: 1024}); !errors.Is(err, ErrFileOpUnsafePath) { + t.Errorf("undeclared doc read err = %v, want ErrFileOpUnsafePath", err) + } +} + func TestK8sPoolRouting(t *testing.T) { fake := newFakeKube(t) fake.bashBehaviors["echo pool"] = func(_ string, stdout, _ io.Writer, _ *websocket.Conn) int { @@ -560,3 +620,58 @@ func TestK8sBridgeUploadVerified(t *testing.T) { t.Errorf("fileops upload missing or wrong (%d bytes)", len(got)) } } + +// TestParseK8sBundleDocsInImage pins the fail-closed boolean: unset is false +// (fleet assumes nothing about a sandbox image's contents), and a typo must +// refuse to boot rather than read as "keep the anchors". +func TestParseK8sBundleDocsInImage(t *testing.T) { + for _, raw := range []string{"", " "} { + v, err := ParseK8sBundleDocsInImage(raw) + if err != nil || v { + t.Errorf("ParseK8sBundleDocsInImage(%q) = %v, %v; want false, nil", raw, v, err) + } + } + for _, raw := range []string{"true", "TRUE", "1", " true "} { + v, err := ParseK8sBundleDocsInImage(raw) + if err != nil || !v { + t.Errorf("ParseK8sBundleDocsInImage(%q) = %v, %v; want true, nil", raw, v, err) + } + } + for _, raw := range []string{"false", "0"} { + v, err := ParseK8sBundleDocsInImage(raw) + if err != nil || v { + t.Errorf("ParseK8sBundleDocsInImage(%q) = %v, %v; want false, nil", raw, v, err) + } + } + for _, raw := range []string{"ture", "yes-please", "on"} { + if _, err := ParseK8sBundleDocsInImage(raw); err == nil { + t.Errorf("ParseK8sBundleDocsInImage(%q) must error (fail closed)", raw) + } + } +} + +// TestFileOpAnchorSupportingDocsByBackend pins the behavior the +// bundle_docs_in_image declaration exists for: with the supporting-doc mounts +// retained, a bundle doc root anchors a READ-ONLY fileop; with them dropped +// (the kubernetes default, where a pod mounts only the workspace claim), the +// same root is refused before the file is looked for. +func TestFileOpAnchorSupportingDocsByBackend(t *testing.T) { + const ws = "/var/lib/fleet/workspace" + docs := []string{"/opt/fleet/client/protocols", "/opt/fleet/client/skills"} + + for _, root := range docs { + anchor, readOnly, err := fileOpAnchorFor(ws, docs, root) + if err != nil || anchor != root || !readOnly { + t.Errorf("mounts retained: fileOpAnchorFor(%q) = %q, ro=%v, %v; want the root itself, read-only, nil", root, anchor, readOnly, err) + } + if _, _, err := fileOpAnchorFor(ws, nil, root); !errors.Is(err, ErrFileOpUnsafePath) { + t.Errorf("mounts dropped: fileOpAnchorFor(%q) err = %v; want ErrFileOpUnsafePath", root, err) + } + } + + // The workspace claim itself is unaffected either way — it is the one + // mount a sandbox pod always has. + if anchor, readOnly, err := fileOpAnchorFor(ws, nil, ws+"/conv-1"); err != nil || anchor != ws || readOnly { + t.Errorf("workspace anchor = %q, ro=%v, %v; want %q, writable, nil", anchor, readOnly, err, ws) + } +} From 35890f00487edef6a9dbd1222803fd2ce06ecc90 Mon Sep 17 00:00:00 2001 From: Brad Flaugher Date: Sun, 23 Aug 2026 00:59:11 +0000 Subject: [PATCH 32/34] Drop three unused nolint:gosec directives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit nolintlint failed the lint lane: the log.Printf calls added with the bundle_docs_in_image knob use constant format strings, so G706 never fires on them and the suppressions were dead. Removed rather than kept "just in case" — an unused directive is exactly what nolintlint exists to catch, and the surrounding comments already say why the interpolated values (operator- configured bundle paths) are not request input. Verified with golangci-lint v2.13.1, the version and config CI runs: 0 issues. Previous local run used an older binary that refused this repo's Go 1.27 target, which is why this landed in CI instead. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01WbgdPXBYGKKBrykZukQV9N Signed-off-by: Brad Flaugher --- internal/agent/manager.go | 3 --- 1 file changed, 3 deletions(-) diff --git a/internal/agent/manager.go b/internal/agent/manager.go index 4faa7e8a..6f64dac5 100644 --- a/internal/agent/manager.go +++ b/internal/agent/manager.go @@ -702,16 +702,13 @@ func buildKubernetesSandboxPool(cfg *config.Config, poolCfg sandbox.PoolConfig, kept, dropped := k8sDocMounts(poolCfg.Container.ReadOnlyMounts, bundleDocDirs, docsInImage) poolCfg.Container.ReadOnlyMounts = kept if len(kept) > 0 { - //nolint:gosec // G706: operator-configured bundle paths from the manifest/env, not request input. log.Printf("sandbox: kubernetes backend — bundle_docs_in_image declared: keeping fileop read anchors for %d bundle doc root(s) %v; the SANDBOX IMAGE must carry them at these exact paths or reads fail not-found", len(kept), kept) } for _, d := range dropped { switch { case clientconfig.IsMaterializedSkillsDir(d): - //nolint:gosec // G706: as above — a boot-resolved bundle path. log.Printf("sandbox: kubernetes backend — skills dir %q is the merged built-in+bundle tree under the control plane's data dir, which no sandbox image can carry; in-sandbox skill reads will not resolve. Set skills_builtin: false in the bundle manifest to make skills/ the bundle's own (bake-able) dir", d) case docsInImage: - //nolint:gosec // G706: as above. log.Printf("sandbox: kubernetes backend — %q is not a bundle doc dir, so bundle_docs_in_image does not vouch for it; in-sandbox reads there will not resolve", d) } } From 82231f7b118b0d4fcb1dd6a0846ba9c9f7fef76b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 01:15:32 +0000 Subject: [PATCH 33/34] Reconcile AGENTS.md with the tree it describes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-0049 made the sandbox backend pluggable (FLEET_SANDBOX_BACKEND= podman|kubernetes) and added an AGENTS.md index row — but nothing else in the file moved, so the headline paragraph and the "sandbox is mandatory" invariant still described the sandbox as *rootless-Podman*, full stop. An agent reading only AGENTS.md would read the kubernetes backend as an invariant violation rather than a supported deployment. Both now state the actual shape: the sandbox is mandatory, the backend is pluggable. While there, the invariant bullet gained the two mechanisms it had left implicit — the `fleet_host_executor` build-tag fence (#159), which is what makes "the host executor cannot ship enabled in a production build" a property of the artifact rather than a runtime flag, and the kubernetes backend's fail-closed boot preflight (no degrade to podman, none to host execution). The rest is drift between the file and the Makefile/workflows: - `make test` / `test-race` / `test-cover` were documented without `-tags fleet_host_executor`, which every one of them passes. A bare `go test ./...` builds a different tree than CI does, so the tag is documented as load-bearing rather than elided. - The web block was a stale hand-copy of the web CI job: it dropped the second npm tree (`scripts/rampart-service`) and the override canary. `make ci-web` runs all eight steps and is now the recommended path. - `make govulncheck`, `ci-go`, `ci-web`, `ci-local` were absent from the target list; `lint-python` and `lint-actions` skip loudly when the tool is missing, which a green local `make lint` does not distinguish. - CI was described as one lane. `ci.yml` fires on `main` only and `dev-ci.yml` is dev's only signal, deferring the -race lane, govulncheck, Grype and both Playwright suites — so the dev→main promotion PR is the first time the full gate ever sees the code. Said plainly, along with `CI gate` being the single required check and auto-merge being gone. - The merge-gate enumeration omitted actionlint/shellcheck, the Helm chart lint and the Playwright suites. - The repository map omitted `deploy/` (systemd units + the Helm chart), the harness binaries under `cmd/`, and the `/settings` + `/admin` web routes. Every claim in the diff was checked against the Makefile, the workflows, `.golangci.yml`, ADR-0049 and the tree; no docs link in the file is dead. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_018eFXcF6BM7bxFVEQH67iuk --- AGENTS.md | 133 ++++++++++++++++++++++++++++++++++++++------------- CHANGELOG.md | 20 ++++++++ 2 files changed, 119 insertions(+), 34 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index fe28f1b7..7e49ef61 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,48 +11,96 @@ this file is the agent-facing distillation, not a replacement for them. ## What fleet is (one paragraph) fleet is a self-hosted, general-purpose agent platform. **One** Go process runs -interactive chat *and* a scheduling engine on one box, driven by **one** unified -agent runtime (`internal/agentcore`). Model-authored local execution — bash, -Python, and file I/O — runs inside a rootless-Podman sandbox; fixed host-side -brokers handle MCP credentials/network and the small control-plane exception -set enumerated in ADR-0036. See -the README "Architecture at a glance" for the full picture. +interactive chat *and* a scheduling engine, driven by **one** unified agent +runtime (`internal/agentcore`). Model-authored local execution — bash, Python, +and file I/O — runs inside a mandatory sandbox; fixed host-side brokers handle +MCP credentials/network and the small control-plane exception set enumerated in +ADR-0036. The sandbox **backend** is pluggable (ADR-0049): +`FLEET_SANDBOX_BACKEND=podman` — the default, rootless Podman co-located with +the process, which is the single-box install — or `kubernetes`, one ephemeral +pod per sandbox exec'd over the apiserver, for the split control-plane +deployment. "On one box" is the default shape, not the only one, so do not write +code (or docs) that assumes podman is the only executor. See the README +"Architecture at a glance" for the full picture. ## Build · test · lint (run before opening any PR) ```sh make build # compile-check ./... AND emit ./fleet + ./fleet-admin -make compile # go build ./... (compile-check only; no artifacts) -make test # go test -p 1 ./... — run in the FOREGROUND -make test-race # go test -race -p 1 ./... (use when touching concurrency) -make test-cover # run Go tests with coverage profiling (writes coverage.out) +make compile # go build ./... (release config — see the build tag below) +make test # go test -p 1 -tags fleet_host_executor ./... — run in the FOREGROUND +make test-race # the same, with -race (use when touching concurrency) +make test-cover # the same, with -coverprofile/-covermode=atomic (writes coverage.out) make lint # golangci-lint + ruff check/format (Python) + migration DDL lint # + actionlint & shellcheck (workflows + shell) — must pass clean +make govulncheck # call-graph-aware CVE scan of the dependency tree make fmt # gofmt -w . make tidy # go mod tidy +make ci-go # the whole Go gate locally: compile, vet, lint, test, -race, govulncheck +make ci-web # the Web CI job verbatim (see below) +make ci-local # ci-go + ci-web — the fast PR gates, locally ``` +**`-tags fleet_host_executor` is load-bearing, not decoration.** The unsandboxed +host executor is fenced behind that build tag (#159) so it is *not* compiled into +a release binary: `make compile` deliberately omits it, and `host_disabled.go` +then stubs `newHostSandbox` out and rejects MockMode at boot. Tests opt in — every +`go test`/`go vet` target above carries the tag, and `.golangci.yml` sets +`build-tags` so the linter agrees — so a bare `go test ./...` builds a *different* +tree than CI does (`host.go` unvetted, untested). Use the Makefile targets. + +`make lint`'s `lint-python` (ruff) and `lint-actions` (actionlint/shellcheck) +**skip loudly when the tool is missing**, printing the install command. So a green +local `make lint` is not proof — read the output, and remember CI enforces both +regardless. + When you touch `web/` (the Next.js app): ```sh -cd web && npm audit --audit-level=low && npm ci && npm run lint && npm run typecheck && npm run test && npm run build +make ci-web # the Web CI job, verbatim — prefer this +cd web && npm ci && npm run lint && npm run typecheck && npm run test && npm run build cd web && npx playwright test --project=mocked # mocked e2e ``` -CI mirrors all of this — Go build/vet/lint/test (including a `-race` lane) plus a -`govulncheck` dependency-CVE scan, a Grype container-image CVE scan (fail on a -fixable CRITICAL/HIGH) of the sandbox image, a Python lint (ruff), a workflow + -shell lint (actionlint & shellcheck), web lint (oxlint) / typecheck (TS 7) / test / build, Playwright (mocked -**and** live, against a real backend + sandbox), a Helm chart lint, a migration -DDL lint, and a gitleaks secret scan. **Every job must be green before merge.** Tests are -deterministic without a live model: use the fake-LLM seam (`internal/fakellm` -via `OPENROUTER_BASE_URL`), never a real key. +There are **two** npm trees — `web/` and `scripts/rampart-service/` — and CI +audits both (`npm audit --audit-level=low`, lockfile-only) plus +`scripts/check-npm-overrides.sh`, the override canary. `make ci-web` runs all +eight steps; the hand-rolled line above skips the audits and the canary, which is +how a clean local run turns into a red PR. + +CI mirrors all of this across **two lanes**, and which one you get depends on the +branch you target: + +- **`CI` (`ci.yml`) — `main` only** (pushes to `main` and PRs targeting it). The + full gate: Go build/vet/lint/test (including a `-race` lane) plus a + `govulncheck` dependency-CVE scan, a Grype container-image CVE scan (fail on a + fixable CRITICAL/HIGH) of the sandbox image, a Python lint (ruff), a workflow + + shell lint (actionlint & shellcheck), web lint (oxlint) / typecheck (TS 7) / + test / build, Playwright (mocked **and** live, against a real backend + + sandbox), a Helm chart lint, a migration DDL lint, and a gitleaks secret scan. + `CI gate` is the **single required status check** on `main`: it `needs` every + other job and always reports, so a docs-only PR (heavy jobs skipped by the + `changes` classifier) still merges, while a code PR cannot go green over a skip. +- **`Dev CI (fast lane)` (`dev-ci.yml`) — `dev` only** (pushes to `dev` and PRs + targeting it). Compile/vet/lint/test against a Postgres service, ruff, the web + lane, the migration DDL lint, gitleaks, actionlint/shellcheck, the Helm lint, + CodeQL and Semgrep. Deliberately deferred to the promotion PR: the `-race` + lane, govulncheck, the Grype image scan, and both Playwright suites. There is + no docs-only classifier here — the fast lane runs on every change. + +`ci.yml` does not fire on `dev` at all, so **the dev→main promotion PR is the +first time the full gate ever sees that code**; expect it to surface things dev +never told you about. **Every job must be green before merge**, and nothing +merges itself — auto-merge was removed, so every PR, dependency bumps included, +waits for a human. Tests are deterministic without a live model: use the fake-LLM +seam (`internal/fakellm` via `OPENROUTER_BASE_URL`), never a real key. CodeQL (security queries, `security-extended`) and Semgrep (Go/JS/Python SAST + -Actions supply chain) also run per PR and are **inside `ci-gate` and `Dev gate`** -— both are reusable workflows that ci.yml/dev-ci.yml call as jobs. `npm audit` -(both npm trees, lockfile-only, any severity) and ruff (`check` **and** -`format --check`) gate the same way. +Actions supply chain) run per PR in **both** lanes: `codeql.yml` and `semgrep.yml` +are reusable workflows that ci.yml/dev-ci.yml call as jobs, so their results roll +up into `CI gate` / `Dev gate` like any other job. `npm audit` (both npm trees, +lockfile-only, any severity) and ruff (`check` **and** `format --check`) gate the +same way. Their thresholds differ, and the difference is load-bearing: @@ -77,10 +125,13 @@ mergeable PR. See [`docs/SCANNING.md`](docs/SCANNING.md) ("Known gaps"). See the README "Repository layout" for the annotated tree. In short: `cmd/` (the one unified `fleet` binary — `fleet serve` runs the server, every other verb is the operator CLI; `fleet-admin` is a transitional deprecation shim that still works for -one release), `internal/` (`agentcore` the one run loop, `sandbox`, +one release; plus the `fleet-bench`, `fake-llm` and `sandbox-probe` harness +binaries), `internal/` (`agentcore` the one run loop, `sandbox`, `mcp`, `creds`, `clientconfig`, `store`, `sched`, `httpapi`, …), `web/` (one -Next.js app: `/chat` + `/orchestrator`), and `config/default/` (the generic -client bundle baked in so fleet runs bare). +Next.js app: `/chat`, `/orchestrator`, `/settings`, `/admin`), `deploy/` (the +systemd units + Caddyfile for the single-box install and the +`deploy/helm/fleet` chart for the Kubernetes one), and `config/default/` (the +generic client bundle baked in so fleet runs bare). ## Non-negotiable invariants — do NOT weaken these @@ -90,11 +141,21 @@ recorded as Architecture Decision Records in [`docs/adr/`](docs/adr/) — a chan that adds, weakens, or reverses an invariant must add or supersede an ADR in the same PR. -- **The sandbox is mandatory.** The agent loop runs in the fleet process, but - every agent tool call's data-plane execution — bash, Python, **and file I/O +- **The sandbox is mandatory** — the *backend* is pluggable, the sandbox is not. + The agent loop runs in the fleet process, but every agent tool call's data-plane + execution — bash, Python, **and file I/O (`view_file`/`write_file`/`edit_file`, via the sandbox FileOp seam, #784)** — - runs inside the rootless-Podman sandbox; there is **no** fast path that skips - it and no host-execution fallback (they fail closed without a sandbox). The + runs inside the sandbox (rootless Podman by default; an ephemeral Kubernetes pod + under `FLEET_SANDBOX_BACKEND=kubernetes`, ADR-0049); there is **no** fast path + that skips it and no host-execution fallback (they fail closed without a + sandbox). The unsandboxed host executor is compiled in **only** behind the + `fleet_host_executor` build tag (#159), which is what makes "it cannot ship + enabled in a production build" a property of the artifact rather than a runtime + flag — do not widen that fence, and do not add a path that reaches `host.go` + from an untagged build. Selecting the kubernetes backend runs a fail-closed boot + preflight (apiserver + credentials, the exact RBAC verbs, the workspace claim, + the sealed-egress NetworkPolicy object) and refuses podman-only knobs rather + than ignoring them: no degrade to podman, none to host execution. The loop holds no privileged local executor of its own: each tool call is handed to the sandbox under host policy. A small set of native tools are host-side **control-plane / broker** operations by design (host network fetch, brokered @@ -143,9 +204,10 @@ same PR. upload only ever produced a missing-token warning. Treat coverage as a quality signal, not a gate: add tests that catch real behavior, not to chase a number. (The merge gates are build/vet/lint, ruff — `check` and - `format --check` — the test suites, the `-race` lane, govulncheck, Grype, - `npm audit` + `scripts/check-npm-overrides.sh`, CodeQL, Semgrep, the migration - linter, and gitleaks.) + `format --check` — actionlint + shellcheck, the test suites, the `-race` lane, + govulncheck, Grype, `npm audit` + `scripts/check-npm-overrides.sh`, CodeQL, + Semgrep, the Helm chart lint, both Playwright suites, the migration + linter, and gitleaks — all rolled up into the one required `CI gate` check.) - **Match the surrounding code:** naming, idioms, and comment density. The `internal/agentcore` package comments explain *why* each governance invariant holds — preserve that level of explanation when you extend it. @@ -170,7 +232,10 @@ same PR. to this file — that is how it grew past 300 lines once already; the historical notes now live in [`docs/FEATURE-NOTES.md`](docs/FEATURE-NOTES.md). - One focused branch + PR per change; keep diffs scoped. Don't refactor unrelated - code in a feature PR. See `CONTRIBUTING.md` for branch/PR conventions. + code in a feature PR. `.github/PULL_REQUEST_TEMPLATE.md` asks for exactly the + three things above — what/why, what you actually ran to verify it, and + scope-and-deviations — so fill it in rather than deleting it. See + `CONTRIBUTING.md` for branch/PR conventions. ## Where to look diff --git a/CHANGELOG.md b/CHANGELOG.md index a6d1f05d..ebef6ed9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -72,6 +72,26 @@ prior versions are listed because none have shipped. ### Fixed +- **`AGENTS.md` reconciled with the tree it describes.** The Kubernetes backend + (ADR-0049) landed an index row and nothing else, so the headline paragraph and + the "sandbox is mandatory" invariant still called the sandbox + *rootless-Podman* — an agent reading only that file would treat + `FLEET_SANDBOX_BACKEND=kubernetes` as an invariant violation rather than a + supported backend. Both now say pluggable-backend/mandatory-sandbox, and the + invariant additionally names the mechanism it had left implicit: the + `fleet_host_executor` build-tag fence (#159) and the kubernetes backend's + fail-closed preflight. Also corrected: the `make test`/`test-race`/`test-cover` + lines omitted `-tags fleet_host_executor` (a bare `go test ./...` builds a + different tree than CI); the web block was a stale copy of the CI job that + dropped the second npm tree (`scripts/rampart-service`) and the override + canary, where `make ci-web` now runs all eight steps; `make govulncheck`, + `ci-go`, `ci-web` and `ci-local` were missing from the target list; CI was + described as one lane when `ci.yml` fires on `main` only and `dev-ci.yml` is + `dev`'s only signal (so the promotion PR is the first full-gate run); the + merge-gate enumeration omitted actionlint/shellcheck, the Helm lint and the + Playwright suites; and the repository map omitted `deploy/`, the harness + binaries under `cmd/`, and the `/settings` + `/admin` web routes. + - **Two green-but-vacuous holes in `ci.yml`.** The `postgresql-client-18` install was best-effort (`|| echo`), so an unreachable PGDG left client 16 in place and `backup_test.go`'s major-mismatch `t.Skipf` turned the *only* coverage of From f60767e3e028f693ce74e8dda43f0cc4df152890 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 01:31:35 +0000 Subject: [PATCH 34/34] Make ci.yml's pg_dump major assertion able to pass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The assertion added in this range failed on its first real outing — got=16 against server 18 — over an install that had plainly succeeded (`Setting up postgresql-client-18 (18.6-1.pgdg24.04+2)`). Installing the package is not what decides the answer: /usr/bin/pg_dump is a symlink to postgresql-common's pg_wrapper, and the wrapper's own header states the rule — it calls the client "with the version, cluster and default database specified in ~/.postgresqlrc or /etc/postgresql-common/user_clusters". That is cluster configuration, not "the newest client installed", so on a runner image carrying a PostgreSQL 16 cluster, adding a client package alongside it changes nothing about the dispatch. Verified rather than reasoned about: this container has the same shape as the runner (/usr/bin/pg_dump -> ../share/postgresql-common/pg_wrapper, the versioned tree at /usr/lib/postgresql/16/bin), and there the wrapper likewise reports 16 while prepending the versioned bin dir resolves pg_dump to it directly. So the versioned bin dir goes first on $GITHUB_PATH. That is also what the test needs: internal/admincli/backup.go execs `pg_dump` off PATH, with no versioned path of its own, so PATH resolution — not package presence — is the thing that decides whether backup_test.go's major-mismatch t.Skipf turns the only coverage of `fleet backup`/`fleet restore` off. Asserted in two places, because $GITHUB_PATH takes effect only from the next step: the install by absolute path in the step that performs it, and PATH resolution in a step of its own. The second one is the assertion that mirrors what the test actually invokes, so the two cannot drift apart again without going red. The CHANGELOG entry claiming the step "now asserts the major" is corrected in the same commit — the assertion was there and could not pass, which is the kind of half-true the honesty-in-docs invariant is about. Verified: actionlint clean over every workflow; go test ./scripts/ passes, including TestPostgresMajorAgreesAcrossCI, which requires every postgres major named anywhere in .github/workflows to agree. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_018eFXcF6BM7bxFVEQH67iuk --- .github/workflows/ci.yml | 35 +++++++++++++++++++++++++++++++++-- CHANGELOG.md | 13 ++++++++++++- 2 files changed, 45 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 97705e89..b7a958e6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -384,12 +384,43 @@ jobs: | sudo tee /etc/apt/sources.list.d/pgdg.list >/dev/null sudo apt-get update -qq sudo apt-get install -y -qq postgresql-client-18 + # Installing the package does NOT change what `pg_dump` resolves to, + # and that is what made this step's first outing fail with `got=16` + # over an install that had plainly succeeded. /usr/bin/pg_dump is a + # symlink to postgresql-common's pg_wrapper, and the wrapper's own + # header states the rule: it calls the client "with the version, + # cluster and default database specified in ~/.postgresqlrc or + # /etc/postgresql-common/user_clusters". That is cluster + # configuration, NOT "the newest client installed" — so on a runner + # image carrying a PostgreSQL 16 cluster, adding a 16-agnostic client + # package changes nothing about the dispatch. Put the versioned bin + # dir first on PATH instead, which is what the Go test needs anyway: + # it execs `pg_dump` off PATH (internal/admincli/backup.go), with no + # versioned path of its own. + echo "/usr/lib/postgresql/18/bin" >> "$GITHUB_PATH" # Assert the major, don't just print it: the test self-skips on a # mismatch, so an install that "succeeded" with the wrong major is - # exactly as invisible as one that failed. + # exactly as invisible as one that failed. Absolute path here because + # $GITHUB_PATH only takes effect in LATER steps — PATH resolution is + # asserted in the next step, where it has. + got="$(/usr/lib/postgresql/18/bin/pg_dump --version | grep -oE '[0-9]+' | head -1)" + [ "$got" = "18" ] || { + echo "::error::installed pg_dump major ${got} != server major 18 — the backup/restore round-trip test would self-skip." + exit 1 + } + + - name: Assert pg_dump on PATH is the server major + # A separate step because $GITHUB_PATH applies from the next step on, + # and PATH resolution is the half that actually broke: the package + # installed correctly and `pg_dump` still meant 16. This asserts the + # exact lookup the round-trip test performs, so the two cannot drift + # again without going red. + run: | + set -euo pipefail + command -v pg_dump got="$(pg_dump --version | grep -oE '[0-9]+' | head -1)" [ "$got" = "18" ] || { - echo "::error::pg_dump major ${got} != server major 18 — the backup/restore round-trip test would self-skip." + echo "::error::pg_dump on PATH is major ${got}, not server major 18 — the backup/restore round-trip test would self-skip." exit 1 } diff --git a/CHANGELOG.md b/CHANGELOG.md index ebef6ed9..4bd3167d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -96,7 +96,18 @@ prior versions are listed because none have shipped. was best-effort (`|| echo`), so an unreachable PGDG left client 16 in place and `backup_test.go`'s major-mismatch `t.Skipf` turned the *only* coverage of `fleet backup` / `fleet restore` off behind the single required check on - `main`; it now asserts the major. And the docs-only classifier initialised + `main`; it now asserts the major — and asserts it twice, because the first + version of that assertion could not pass: installing `postgresql-client-18` + does not change what `pg_dump` resolves to. `/usr/bin/pg_dump` is + postgresql-common's `pg_wrapper`, which dispatches on the version/cluster in + `~/.postgresqlrc` or `/etc/postgresql-common/user_clusters` rather than on the + newest client present, so on a runner carrying a PostgreSQL 16 cluster the + wrapper kept selecting 16 and the step failed with `got=16` over a successful + install. The versioned bin dir now goes first on `$GITHUB_PATH` — which is + what the round-trip test needs anyway, since it execs `pg_dump` off PATH — and + the install is asserted by absolute path in that step, PATH resolution in the + next one (`$GITHUB_PATH` only applies from the following step on). And the + docs-only classifier initialised `docs_only=true` and only ever cleared it inside its loop, so an **empty** diff classified as docs-only and skipped the suite — which `ci-gate` then waved through, because an empty diff is the absence of evidence, not evidence that