diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 977dfe50b..24f5f805f 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -26,25 +26,28 @@ # - Require a pull request before merging (no direct pushes to `main`). # - Require status checks to pass, with "branches up to date" (strict) ON. The # SINGLE required check is the `CI gate` job in .github/workflows/ci.yml: it -# `needs` every other CI job (Go build / vet / lint / test — which also runs -# vet, the -race lane and govulncheck; Web lint / test / build; Playwright -# mocked; Playwright live; the gitleaks secret scan; and the Grype -# container-image CVE scan) and fails unless each -# one succeeded or was cleanly skipped. Requiring that one aggregate check +# `needs` EVERY other job in that file — the docs-only classifier; the +# gitleaks secret scan; the actionlint workflow lint; the migration DDL +# lint; the Helm chart lint; Go build / vet / lint / test (which also runs +# the -race lane and govulncheck); the ruff Python lint; CodeQL and Semgrep +# (called as reusable workflows); Web lint / typecheck / test / build with +# both npm audits; Playwright mocked; Playwright live; and the Grype +# container-image CVE scan — and fails unless each +# one succeeded or was cleanly skipped. +# Do not maintain this list by hand alone: scripts/check_gate_needs_test.go +# fails `make test` if any job in ci.yml is missing from `ci-gate`'s needs. Requiring that one aggregate check # rather than each job by name is what lets a docs-only PR skip the heavy jobs # without being left blocked on a required check that never reported. # - Block force-pushes (non-fast-forward) and branch deletion. # Deliberately NOT enabled — a CI-only gate so the sole maintainer can self-merge # a green PR — are required approvals and Code Owner review (see the note above). # -# Dependabot auto-merge interaction: .github/workflows/auto-merge-dependabot.yml -# enables `gh pr merge --auto` on PATCH-level bumps only, and the required `CI -# gate` check holds that merge until CI is green. Because Code Owner review is -# NOT currently required, such a patch PR auto-merges on green CI with no human -# approval — INCLUDING a github-actions patch bump that touches a CODEOWNERS- -# matched path like .github/workflows/. To force a human gate on those sensitive -# paths, turn on Code Owner review in the ruleset; minor and major bumps already -# wait for a human regardless. +# Dependabot interaction: there is no auto-merge in this repository. Every +# dependency bump — every ecosystem, every bump level — is merged by a human, so +# a github-actions bump touching a CODEOWNERS-matched path like +# .github/workflows/ cannot land unattended. Turning on Code Owner review in the +# ruleset would additionally force review by the owner named below rather than +# any maintainer. # --------------------------------------------------------------------------- # Catch-all: every file has an owner so nothing is silently unowned. Specific @@ -81,7 +84,7 @@ /internal/sched/db/migrations/ @bradflaugher # CI gates and dependency/agent automation — these workflows ARE the merge -# guarantees, including the Dependabot auto-merge path. +# guarantees. /.github/workflows/ @bradflaugher /.github/dependabot.yml @bradflaugher /.github/CODEOWNERS @bradflaugher diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 000000000..fe89f4b2a --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,44 @@ +name: Bug report +description: Something in fleet does not behave as documented. +labels: [bug] +body: + - type: markdown + attributes: + value: | + For a **security vulnerability**, stop and read + [SECURITY.md](https://github.com/ElcanoTek/fleet/blob/main/SECURITY.md) + instead — do not file it here. + - type: textarea + id: what-happened + attributes: + label: What happened + description: What you observed, and what you expected instead. + validations: {required: true} + - type: textarea + id: repro + attributes: + label: Steps to reproduce + description: The smallest sequence that shows the problem. + placeholder: | + 1. fleet serve with ... + 2. ... + 3. Observed: ... + validations: {required: true} + - type: textarea + id: version + attributes: + label: Version and environment + description: >- + Output of `fleet version` (or the commit you built from), your OS, and + whether Podman is running rootless. `fleet doctor` output is ideal. + render: shell + validations: {required: true} + - type: textarea + id: logs + attributes: + label: Relevant logs + description: >- + Redact before pasting. fleet brokers credentials host-side precisely so + they never reach a log, but check anyway. + render: shell + validations: {required: false} diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 000000000..37b131ea0 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,13 @@ +# Blank issues are disabled so the security contact link below is unmissable. +# SECURITY.md and CODE_OF_CONDUCT.md both say "do not open a public issue for a +# vulnerability", and until now the New Issue button offered a blank box with no +# such warning — the one moment a reporter is most likely to get it wrong. +blank_issues_enabled: false +contact_links: + - name: Report a security vulnerability (private) + url: https://github.com/ElcanoTek/fleet/blob/main/SECURITY.md + about: Please do NOT open a public issue. SECURITY.md has the private + disclosure process and the response SLA. + - name: Contributing guide + url: https://github.com/ElcanoTek/fleet/blob/main/CONTRIBUTING.md + about: Build, test and lint commands, the CI gates, and PR conventions. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 000000000..b87abfea2 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,39 @@ +name: Feature request +description: Propose a capability or a change to how fleet behaves. +labels: [enhancement] +body: + - type: textarea + id: problem + attributes: + label: The problem + description: >- + What are you trying to do that fleet makes hard or impossible? Describe + the situation rather than the solution you have in mind. + validations: {required: true} + - type: textarea + id: proposal + attributes: + label: What you would like fleet to do + validations: {required: true} + - type: dropdown + id: invariant + attributes: + label: Does this touch one of the invariants in AGENTS.md? + description: >- + The mandatory sandbox, host-side credentials, the single governed loop, + no secrets in the repo, honest docs, or client content living in an + out-of-repo bundle. If yes, it needs an ADR — say so here and we can + work out the shape before anyone writes code. + options: + - "No — this is additive and does not touch an invariant" + - "Yes — or I am not sure" + validations: {required: true} + - type: textarea + id: bundle + attributes: + label: Could this ship as a client-config bundle instead? + description: >- + fleet is an engine; per-customer MCP servers, personas, protocols and + prompts belong in a bundle rather than in this repo. If you think this + genuinely needs an engine change, say why the bundle cannot express it. + validations: {required: false} diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 000000000..c5e9c1118 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,33 @@ + + +## What changed, and why + + + +## How you verified it + + + +## Scope and deviations + + + +--- + +- [ ] `CHANGELOG.md` updated, if this is a user-visible change +- [ ] A design note (`docs/.md`) added, if this ships a feature +- [ ] An ADR added or superseded in `docs/adr/`, if this adds, weakens or + reverses an invariant — required in the *same* PR +- [ ] The diff is scoped to one change (no unrelated refactors) diff --git a/.github/codeql-accepted-findings.json b/.github/codeql-accepted-findings.json index f2fac66d0..9ed176e96 100644 --- a/.github/codeql-accepted-findings.json +++ b/.github/codeql-accepted-findings.json @@ -85,6 +85,11 @@ "rule": "js/insecure-temporary-file", "file": "web/e2e/test-auth-key.ts", "reason": "Test-only, and the reported defect is fixed as far as it can be without changing the cross-process rendezvous contract: the write is now O_EXCL (flag \"wx\") at mode 0600 with crypto random bytes in the sibling name, so it cannot follow or clobber a pre-planted symlink and does not leave the private half world-readable. The query recognizes only mkdtemp as safe, but the fixed path is a deliberate rendezvous — playwright.config.ts is loaded in the main process AND re-imported in every worker, which must all read the same throwaway keypair. The key is generated per run, protects nothing real, and is never committed." + }, + { + "rule": "go/command-injection", + "file": "internal/sandbox/host.go", + "reason": "hostImpl.runBash executing caller-supplied shell (`bash -c req.Command`) IS the component's contract: it is the unsandboxed TEST/DEV-ONLY executor, compiled solely behind the fleet_host_executor build tag (#159) — a release `go build ./...` contains the fail-closed stub in host_disabled.go and MockMode refuses to boot without the tag, so this sink cannot ship in a production binary (ADR-0002's enforcement). Every production tool call runs through the container or kubernetes backend instead. The same line already carries the equivalent gosec waiver ('shell execution is the purpose of this tool'). The finding surfaced on #1249 because the diff-informed PR run intersected the flow's path (sandbox.go/pool.go edits), not because a new source reached the sink." } ] } diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 85841447d..fcfdb7557 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -9,12 +9,11 @@ # release. This is a deliberate defense against fast typosquat / account-takeover # attacks, where a compromised version is published and then yanked once the # ecosystem flags it — by the time Dependabot proposes the bump, that window has -# usually closed. It matters most for patch bumps, which the auto-merge workflow -# (`.github/workflows/auto-merge-dependabot.yml`) merges on its own once CI is -# green: the cooldown keeps a minutes-old patch from being proposed (and thus -# auto-merged) before the ecosystem has had a chance to scrutinize it. Cooldown -# applies to version updates only, never to Dependabot security updates, so -# urgent CVE fixes are not delayed. +# usually closed. Every bump here is merged by a human (this repository has no +# auto-merge), so the cooldown is not load-bearing on its own; what it buys is +# that a reviewer is never looking at a release the ecosystem has not yet had a +# chance to scrutinize. Cooldown applies to version updates only, never to +# Dependabot security updates, so urgent CVE fixes are not delayed. # # Schema (key `cooldown`, integer `*-days` sub-keys; supported for gomod and npm; # applies to version updates only): @@ -113,15 +112,12 @@ updates: # CVE fixes are never delayed behind dev. target-branch: dev open-pull-requests-limit: 10 - # Wait before proposing freshly published releases (see header). Patch is the - # tightest gate because auto-merge-dependabot.yml auto-merges patch bumps. + # Wait before proposing freshly published releases (see header). cooldown: default-days: 7 - # 7, not 14: the supply-chain argument in the header is about the - # AUTO-MERGE path, and auto-merge-dependabot.yml merges patch only — - # a major is never auto-merged, so delaying it protects nothing and - # costs up to 14 days of visibility (cooldown + the weekly interval) - # before anyone even learns the major exists. + # 7, not 14 for a major: a long delay protects nothing here — a major is + # reviewed by a human either way — and costs up to 14 days of visibility + # (cooldown + the weekly interval) before anyone even learns it exists. semver-major-days: 7 semver-minor-days: 7 semver-patch-days: 3 @@ -156,11 +152,9 @@ updates: # Same supply-chain cooldown as gomod (see header). cooldown: default-days: 7 - # 7, not 14: the supply-chain argument in the header is about the - # AUTO-MERGE path, and auto-merge-dependabot.yml merges patch only — - # a major is never auto-merged, so delaying it protects nothing and - # costs up to 14 days of visibility (cooldown + the weekly interval) - # before anyone even learns the major exists. + # 7, not 14 for a major: a long delay protects nothing here — a major is + # reviewed by a human either way — and costs up to 14 days of visibility + # (cooldown + the weekly interval) before anyone even learns it exists. semver-major-days: 7 semver-minor-days: 7 semver-patch-days: 3 @@ -180,8 +174,7 @@ updates: # # Know what a green build means here: no CI job exercises this service, and # `@nationaldesignstudio/rampart` is pre-1.0 (^0.1.3), where a MINOR bump is - # allowed to break. Auto-merge cannot touch these (it gates on patch only), - # so they land in human review — which is the right outcome, not a gap. + # allowed to break. Like every other bump here, these land in human review. - package-ecosystem: npm directory: "/scripts/rampart-service" schedule: diff --git a/.github/workflows/auto-merge-dependabot.yml b/.github/workflows/auto-merge-dependabot.yml deleted file mode 100644 index 00b60e271..000000000 --- a/.github/workflows/auto-merge-dependabot.yml +++ /dev/null @@ -1,74 +0,0 @@ -# Dependabot already opens dependency-update PRs (see .github/dependabot.yml), -# but every one — even a routine patch — currently waits on a human to merge. -# This workflow lets PATCH-level bumps merge themselves once the full CI gate -# (build / vet / lint / test / -race / govulncheck, web lint+test+build, -# Playwright mocked + live, and the gitleaks secret scan) is green. Minor and -# major bumps are intentionally left for a human, where an API change or a -# transitive surprise is more likely. -# -# TWO LIMITS THAT ARE LOAD-BEARING, both learned the hard way: -# -# 1. "CI is the approval signal, and it is never bypassed" IS ONLY TRUE WHERE -# THE GATE IS A REQUIRED CHECK. `gh pr merge --auto` asks GitHub to hold the -# merge until every REQUIRED check passes — so on a branch whose ruleset -# requires nothing, there is nothing to hold it and the PR merges as soon as -# it is mergeable. The `dev` ruleset currently requires no status checks at -# all (only `deletion` and `non_fast_forward`), and .github/dependabot.yml -# points every version update at `dev`. So the `branches:` filter below is -# not cosmetic: it keeps this workflow from applying to a branch where its -# central assumption does not hold. Getting `Dev gate` into the dev ruleset -# is the real fix and is a repo-settings action; see docs/SCANNING.md -# ("Known gaps"). -# -# 2. A `github-actions` bump IS A REWRITE OF .github/workflows/*. It changes -# what CI executes, on a surface where the cooldown that protects gomod and -# npm is not even available (Dependabot supports `cooldown` for those two -# ecosystems only), so a freshly published action version can be proposed -# the same day. That combination — self-modifying CI, no cooldown, no -# required check on the target branch — is not something to auto-merge, so -# that ecosystem is excluded below and takes a human. -# -# Requires "Allow auto-merge" to be enabled on the repository (Settings → -# General → Pull Requests). This is the pattern documented in GitHub's -# "Automating Dependabot with GitHub Actions" guide. -name: Auto-merge Dependabot patch PRs - -on: - pull_request: - # See limit 1 in the header: this workflow's safety rests on the target - # branch having required checks. Naming the branches explicitly means it can - # never silently start applying to one nobody protected. - branches: [main, dev] - -# What actually confines these scopes is the `if: github.actor == -# 'dependabot[bot]'` guard on the job below — a `permissions:` block is honored -# for whatever run reaches it, regardless of actor. github.actor is not -# spoofable, so the guard holds; the scopes are declared on the JOB rather than -# the workflow so a second job added here later does not inherit write access it -# never asked for. -permissions: {} - -jobs: - auto-merge: - if: ${{ github.actor == 'dependabot[bot]' }} - runs-on: ubuntu-latest - permissions: - contents: write - pull-requests: write - steps: - - name: Fetch Dependabot metadata - id: meta - uses: dependabot/fetch-metadata@25dd0e34f4fe68f24cc83900b1fe3fe149efef98 # v3.1.0 - with: - github-token: ${{ secrets.GITHUB_TOKEN }} - - - name: Enable auto-merge for patch updates - # Only patch bumps auto-merge; minor and major get human review. And - # never github-actions, whatever the bump level — see limit 2 in the - # header: that ecosystem's "dependency" is the CI definition itself. - if: ${{ steps.meta.outputs.update-type == 'version-update:semver-patch' - && steps.meta.outputs.package-ecosystem != 'github_actions' }} - run: gh pr merge --auto --squash "$PR_URL" - env: - PR_URL: ${{ github.event.pull_request.html_url }} - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index 461f407cb..97f064f02 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -21,6 +21,7 @@ jobs: benchmarks: name: Subsystem throughput benchmarks runs-on: ubuntu-latest + timeout-minutes: 60 services: postgres: diff --git a/.github/workflows/build-sandbox-image.yml b/.github/workflows/build-sandbox-image.yml index 98ace2532..a7f0b9ea3 100644 --- a/.github/workflows/build-sandbox-image.yml +++ b/.github/workflows/build-sandbox-image.yml @@ -94,6 +94,7 @@ jobs: build: name: Build sandbox (no push) runs-on: ubuntu-latest + timeout-minutes: 45 env: BUNDLE_DIR: ${{ inputs.bundle_dir }} steps: @@ -112,6 +113,31 @@ jobs: # actions/untrusted-checkout under security-extended; the same pattern in # publish-sandbox-image.yml was hardened symmetrically even though the # query's privileged/taint split happened to flag neither variant there.) + - name: Validate bundle_dir + # `fleet_ref`, which reaches the SAME build-script invocation, got a + # character allow-list, a bare-SHA refusal and a GITHUB_OUTPUT-newline + # fix. `bundle_dir` got nothing, and the asymmetry is the whole reason + # this exists — there is no shell injection here (it goes through `env:` + # and is quoted at every use), but nothing stopped `../../.fleet-core` + # or an absolute path from silently repointing the build at a different + # manifest. The reusable workflow runs in the CALLER's context with the + # caller's token, so the blast radius is a client repo misbuilding its + # own image; this is defence in depth and consistency, not a live hole. + env: + REQUESTED: ${{ inputs.bundle_dir }} + run: | + set -euo pipefail + case "$REQUESTED" in + ""|.) ;; + /*) + echo "::error::bundle_dir must be relative to the caller repo, not absolute."; exit 1 ;; + *..*) + echo "::error::bundle_dir must not contain '..' — it may only name a path inside the caller repo."; exit 1 ;; + *[!a-zA-Z0-9._/-]*) + echo "::error::bundle_dir contains a character outside [A-Za-z0-9._/-]. Refused before the build."; exit 1 ;; + esac + echo "bundle_dir accepted: '${REQUESTED:-.}'" + - name: Pin fleet_ref to collaborator-controlled refs id: pin env: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 612edae39..b7a958e69 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,6 +15,27 @@ permissions: # e2e-live, which npm-install and run thousands of third-party packages. contents: read +# Rapid successive pushes to the SAME pull request cancel the previous run. +# `dev-ci.yml` has had this since it was written; this lane — the expensive one, +# carrying mocked Playwright, the live e2e against a real backend + rootless +# sandbox, and the Grype image scan — did not, so a branch pushed three times in +# a minute queued three full suites and the two stale ones still had to finish +# before the interesting one started. +# +# `cancel-in-progress` is deliberately NOT blanket-true: a push to `main` is a +# merge that already happened, and its run is the tree-wide verdict the +# diff-informed PR runs cannot give (see docs/SCANNING.md on why a +# `pull_request` CodeQL run certifies a diff, not a tree). Cancelling that to +# make room for the next merge would throw away the only full-tree result. So +# cancellation is scoped to `pull_request`, where the superseding run covers +# strictly newer code. +concurrency: + # github.ref already differs between a PR (refs/pull/N/merge) and a push to + # main (refs/heads/main); event_name is included so the two can never share a + # group and cancel each other. + group: ci-${{ github.event_name }}-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + jobs: changes: # Classify the PR (or push) as docs-only or not. The heavy jobs below gate on @@ -26,6 +47,7 @@ jobs: # pasted into a markdown file, so it runs on every change. name: Detect docs-only changes runs-on: ubuntu-latest + timeout-minutes: 10 outputs: docs_only: ${{ steps.detect.outputs.docs_only }} steps: @@ -64,6 +86,18 @@ jobs: echo "Changed files in ${range}:" printf '%s\n' "$files" + # FAIL-SAFE on an empty diff. docs_only is initialised true and only + # ever cleared inside the loop below, so an empty $files would classify + # as docs-only and skip the whole suite — and `ci-gate` trusts exactly + # that value when it decides a `skipped` job is acceptable. An empty + # range is not evidence that only prose changed; it is the absence of + # evidence (a force-push can collapse the three-dot range), so run + # everything. + if [ -z "$(printf '%s' "$files" | tr -d '[:space:]')" ]; then + echo "empty diff for ${range}; refusing to classify as docs-only." + echo "docs_only=false" >> "$GITHUB_OUTPUT"; exit 0 + fi + docs_only=true while IFS= read -r f; do [ -z "$f" ] && continue @@ -103,6 +137,7 @@ jobs: gitleaks: name: Secret scan (gitleaks) runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -132,9 +167,133 @@ jobs: # allowlist in .gitleaks.toml is just local-run robustness. run: gitleaks dir . --redact --exit-code 1 + actions: + # Lint the workflow files THEMSELVES. This repo carries ~3.1k lines of + # workflow YAML that decide what every other gate on this list even runs, + # and until now nothing checked them. The overlap with what already runs is + # narrow and worth stating: Semgrep's `p/github-actions` pack meets this on + # exactly ONE axis (actions referenced by a mutable tag) and CodeQL's + # `actions` language on taint into a checkout. Neither parses `${{ }}` + # expressions and neither shellchecks a `run:` block, which is where + # actionlint earns its place — expression syntax and type errors, undefined + # contexts, invalid `needs:` / `runs-on:` / cron, deprecated syntax, plus + # shellcheck over the bash in every `run:`. + # + # The tree was at ZERO findings when this gate went in, so a failure here is + # a regression rather than a backlog to wade through. Getting there took two + # real fixes and one waiver: `run: "$GITHUB_WORKSPACE/..."` in both lanes + # passed the path to the shell UNQUOTED (the YAML quotes are consumed by the + # YAML parser, so the shell never saw them, despite the comment above the + # line showing quoting was intended), and the SC2153 on `RESULTS` is an + # `env:`-supplied name shellcheck cannot see, waived in-line with a reason. + # + # Deliberately NOT gated on the docs-only classifier. A change under + # .github/workflows/ can never be docs-only by that allowlist anyway, but a + # broken workflow is the one failure that can disarm every OTHER gate here, + # so it runs unconditionally rather than depending on a classifier to let it. + name: Workflow + shell lint (actionlint, shellcheck) + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Install actionlint + # Pinned release + checksum, the same contract as the gitleaks step + # above: an upstream re-tag cannot silently change what this gate + # enforces. Checksum verified against the release's own + # actionlint_1.7.7_checksums.txt. + env: + ACTIONLINT_VERSION: '1.7.7' + ACTIONLINT_SHA256: '023070a287cd8cccd71515fedc843f1985bf96c436b7effaecce67290e7e0757' + run: | + set -euo pipefail + tarball="actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz" + curl -sSL -o "$tarball" \ + "https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/${tarball}" + echo "${ACTIONLINT_SHA256} ${tarball}" | sha256sum -c - + tar -xzf "$tarball" actionlint + sudo install -m 0755 actionlint /usr/local/bin/actionlint + actionlint --version + + - name: Run actionlint + # shellcheck ships preinstalled on the ubuntu-latest image, so actionlint + # finds it and the `run:` blocks get checked too. ASSERT that rather than + # assume it: a silently absent shellcheck drops half this job's coverage + # while the job still reports green — the green-but-vacuous outcome this + # repo keeps writing post-mortems about (see docs/SCANNING.md). + run: | + set -euo pipefail + command -v shellcheck >/dev/null || { + echo "::error::shellcheck is missing from the runner image; actionlint would silently skip every run: block." + exit 1 + } + shellcheck --version | head -2 + actionlint -color + + - name: Lint shell scripts (shellcheck) + # The 18 tracked *.sh files are ~6.2k lines, and they are the DEPLOY + # PATH, not helpers: scripts/update.sh, bootstrap.sh and doctor.sh are + # what `fleet update` / `fleet bootstrap` actually run on an operator's + # box. Go has golangci-lint, the web tier has oxlint + tsc, Python has + # ruff — bash had nothing, which is the same gap docs/SCANNING.md used to + # justify adding ruff, one language over. Several scripts already carry + # hand-written `# shellcheck` directives, so it was being run by hand; + # nothing made that reproducible. + # + # -S warning, and the tree is CLEAN at that level: the backlog was three + # findings total (two SC2034, one SC2148), all fixed or annotated with a + # reason in the same change that added this gate. Gating over an unfixed + # backlog is how a gate becomes something people learn to ignore, so the + # level was chosen by measuring, not by taste. The info tier (11 findings, + # mostly style) stays off. + run: | + set -euo pipefail + mapfile -t sh_files < <(git ls-files '*.sh') + printf 'shellchecking %d files\n' "${#sh_files[@]}" + # A vacuous pass is the failure mode this repo keeps writing up: if the + # glob ever matches nothing, say so instead of reporting green. + [ "${#sh_files[@]}" -gt 0 ] || { echo "::error::no shell scripts matched"; exit 1; } + shellcheck -S warning "${sh_files[@]}" + + helm: + # Lint + render the fleet Helm chart (#989) so a values/template drift + # fails here, not at an operator's install. Fast (<15s) and not gated on + # the docs-only classifier — chart files are product, not prose. helm + # ships preinstalled on the ubuntu-latest runner image. + name: Helm chart lint (#989) + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Lint and render the fleet chart + run: | + set -euo pipefail + helm version --short + helm lint deploy/helm/fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci + # Minimal render (the defaults every install starts from) … + helm template ci deploy/helm/fleet --namespace fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci >/dev/null + # … and the everything-on render so every optional template compiles. + helm template ci deploy/helm/fleet --namespace fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci \ + --set postgres.enabled=true \ + --set web.enabled=true --set web.image=registry.invalid/fleet-web:ci \ + --set ingress.enabled=true --set ingress.host=fleet.example.com \ + --set networkPolicies.openEgress.create=true \ + --set 'networkPolicies.openEgress.blockedCIDRs={10.0.0.0/8}' \ + --set sandbox.kubernetes.runtimeClass=kata >/dev/null + migrations: name: Migration DDL lint (#256) runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -156,6 +315,7 @@ jobs: go: name: Go build / vet / lint / test runs-on: ubuntu-latest + timeout-minutes: 45 needs: changes # Skipped for a docs-only change; the `CI gate` job treats a skip as a pass. if: ${{ needs.changes.outputs.docs_only != 'true' }} @@ -203,20 +363,66 @@ jobs: # The runner ships client 16; the postgres service is server 18, and # pg_dump refuses to dump a newer server ("server version mismatch"). Pull # the matching client from PGDG so the backup/restore round-trip test - # (cmd/fleet-admin) runs for real. Best-effort: if the repo is unreachable - # the step still succeeds and the test skips on the version mismatch (it is - # written to skip, never fail, when client and server majors disagree). + # (cmd/fleet-admin) runs for real. + # + # NOT best-effort, and that is a deliberate change. This step used to end + # in `|| echo "... will skip on version mismatch"`, which meant an + # unreachable PGDG left the runner's client 16 in place, and + # backup_test.go's `t.Skipf("pg_dump major %d != server major %d")` + # (internal/admincli/backup_test.go:155) turned the ONLY coverage of + # `fleet backup` / `fleet restore` off — behind a green `CI gate`, the + # single required check on main. That is the same green-but-vacuous shape + # this repo keeps writing post-mortems about, and the same one `e2e-live` + # already refuses by grepping its own log for `--- SKIP`. A PGDG outage + # is now a red build with an obvious cause rather than a silent hole. + run: | + set -euxo pipefail + sudo install -d /usr/share/postgresql-common/pgdg + sudo curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \ + -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc + echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \ + | sudo tee /etc/apt/sources.list.d/pgdg.list >/dev/null + sudo apt-get update -qq + sudo apt-get install -y -qq postgresql-client-18 + # Installing the package does NOT change what `pg_dump` resolves to, + # and that is what made this step's first outing fail with `got=16` + # over an install that had plainly succeeded. /usr/bin/pg_dump is a + # symlink to postgresql-common's pg_wrapper, and the wrapper's own + # header states the rule: it calls the client "with the version, + # cluster and default database specified in ~/.postgresqlrc or + # /etc/postgresql-common/user_clusters". That is cluster + # configuration, NOT "the newest client installed" — so on a runner + # image carrying a PostgreSQL 16 cluster, adding a 16-agnostic client + # package changes nothing about the dispatch. Put the versioned bin + # dir first on PATH instead, which is what the Go test needs anyway: + # it execs `pg_dump` off PATH (internal/admincli/backup.go), with no + # versioned path of its own. + echo "/usr/lib/postgresql/18/bin" >> "$GITHUB_PATH" + # Assert the major, don't just print it: the test self-skips on a + # mismatch, so an install that "succeeded" with the wrong major is + # exactly as invisible as one that failed. Absolute path here because + # $GITHUB_PATH only takes effect in LATER steps — PATH resolution is + # asserted in the next step, where it has. + got="$(/usr/lib/postgresql/18/bin/pg_dump --version | grep -oE '[0-9]+' | head -1)" + [ "$got" = "18" ] || { + echo "::error::installed pg_dump major ${got} != server major 18 — the backup/restore round-trip test would self-skip." + exit 1 + } + + - name: Assert pg_dump on PATH is the server major + # A separate step because $GITHUB_PATH applies from the next step on, + # and PATH resolution is the half that actually broke: the package + # installed correctly and `pg_dump` still meant 16. This asserts the + # exact lookup the round-trip test performs, so the two cannot drift + # again without going red. run: | - set -x - (sudo install -d /usr/share/postgresql-common/pgdg \ - && sudo curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \ - -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc \ - && echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \ - | sudo tee /etc/apt/sources.list.d/pgdg.list >/dev/null \ - && sudo apt-get update -qq \ - && sudo apt-get install -y -qq postgresql-client-18) \ - || echo "could not install postgresql-client-18; round-trip test will skip on version mismatch" - pg_dump --version || true + set -euo pipefail + command -v pg_dump + got="$(pg_dump --version | grep -oE '[0-9]+' | head -1)" + [ "$got" = "18" ] || { + echo "::error::pg_dump on PATH is major ${got}, not server major 18 — the backup/restore round-trip test would self-skip." + exit 1 + } - name: Create test databases run: | @@ -303,10 +509,12 @@ jobs: # Write the per-package (and per-function) coverage table to the Actions # job summary, visible in the UI without downloading any artifact (#249). run: | - echo "## Go coverage by package" >> "$GITHUB_STEP_SUMMARY" - echo '```' >> "$GITHUB_STEP_SUMMARY" - go tool cover -func=coverage.out >> "$GITHUB_STEP_SUMMARY" - echo '```' >> "$GITHUB_STEP_SUMMARY" + { + echo "## Go coverage by package" + echo '```' + go tool cover -func=coverage.out + echo '```' + } >> "$GITHUB_STEP_SUMMARY" - name: govulncheck (dependency CVEs) # Call-graph-aware scan of the (broad, fast-moving) dependency tree for @@ -346,6 +554,7 @@ jobs: python: name: Python lint (ruff) runs-on: ubuntu-latest + timeout-minutes: 10 needs: changes # Skipped for a docs-only change; the `CI gate` job treats a skip as a pass. if: ${{ needs.changes.outputs.docs_only != 'true' }} @@ -411,6 +620,7 @@ jobs: web: name: Web lint / test / build runs-on: ubuntu-latest + timeout-minutes: 20 needs: changes # Skipped for a docs-only change; the `CI gate` job treats a skip as a pass. if: ${{ needs.changes.outputs.docs_only != 'true' }} @@ -455,7 +665,7 @@ jobs: # registry flake skips with a notice (the audit above is the CVE gate). # Absolute path: this job's default working-directory is web/, which is # exactly how run 32579378165 caught the repo-relative form (exit 127). - run: "$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh" + run: '"$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh"' - name: Install dependencies run: npm ci @@ -481,6 +691,7 @@ jobs: playwright: name: Web e2e (Playwright, mocked) runs-on: ubuntu-latest + timeout-minutes: 30 needs: changes # Skipped for a docs-only change; the `CI gate` job treats a skip as a pass. if: ${{ needs.changes.outputs.docs_only != 'true' }} @@ -543,6 +754,7 @@ jobs: e2e-live: name: Web e2e (Playwright, live — real backend + sandbox) runs-on: ubuntu-latest + timeout-minutes: 60 needs: changes # Skipped for a docs-only change; the `CI gate` job treats a skip as a pass. if: ${{ needs.changes.outputs.docs_only != 'true' }} @@ -727,6 +939,7 @@ jobs: grype-scan: name: Container image vulnerability scan (Grype) runs-on: ubuntu-latest + timeout-minutes: 35 needs: [e2e-live] # runs after e2e-live succeeds (which already builds the image) # When e2e-live is skipped for a docs-only change, this job is skipped too # (a job that `needs` a skipped job is itself skipped); the `CI gate` job @@ -741,6 +954,11 @@ jobs: steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + # This job holds a write scope. Nothing in it pushes or does an + # authenticated fetch, so do not leave the token in .git/config while + # a PR-controlled Containerfile goes through `podman build`. + persist-credentials: false - name: Install Grype # Pin a specific Grype release and verify its checksum, exactly as the @@ -806,9 +1024,11 @@ jobs: --output sarif=grype-results.sarif - name: Enforce actionable Fedora RPM policy - # The generic image follows Fedora latest. Block only when a CRITICAL - # RPM finding has a packaged fix; report upstream language records but - # do not force hand-maintained overlay pins over distro-owned packages. + # The generic image follows Fedora latest. Block when a CRITICAL **or + # HIGH** RPM finding has a packaged fix (scripts/check-grype-policy.sh + # selects both severities — this comment said CRITICAL only, which + # understated the gate); report upstream language records but do not + # force hand-maintained overlay pins over distro-owned packages. run: scripts/check-grype-policy.sh grype-results.json - name: Upload Grype SARIF results to the GitHub Security tab @@ -834,8 +1054,9 @@ jobs: # allowed (docs-only), but any failure or cancellation fails the gate. name: CI gate if: ${{ always() }} - needs: [changes, gitleaks, migrations, go, python, codeql, semgrep, web, playwright, e2e-live, grype-scan] + needs: [changes, gitleaks, actions, migrations, helm, go, python, codeql, semgrep, web, playwright, e2e-live, grype-scan] runs-on: ubuntu-latest + timeout-minutes: 5 steps: - name: Require all upstream jobs to have succeeded or been skipped env: diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 60c692327..6ea4ebf11 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -142,6 +142,19 @@ jobs: # 55 findings. Do not re-derive a tree-wide claim from a PR run; the # full-tree numbers come from push/schedule runs. See # docs/adr/0048-codeql-severity-gating.md. + # NOTE on in-source `// codeql[rule-id]` suppressions: they do NOT + # work with this pipeline, and this was measured, not assumed. Three + # forms were tried on PR #1249 (the `packs:` input, `packs:` with + # the documented "+" additive prefix, and an inline `config:` block + # combining security-extended with codeql/go-queries' + # AlertSuppression.ql) — in every case the uploaded SARIF carried no + # `suppressions` on the annotated result, the gate below kept + # classifying the waiver from the register, and the Security-tab + # alert stayed open. The analyze action's interpret step is not + # configurable enough to change that, so the accepted-findings + # register stays the ONE waiver mechanism for the CI gate, and a + # deliberately-waived alert is closed in the Security tab by a + # one-time human dismissal there (which persists across analyses). - language: go build-mode: autobuild queries: security-extended @@ -282,7 +295,7 @@ jobs: else ($rows | sort_by(.rule, .file, .line) | .[] | fmt) end); tier("BLOCKING — High band (security-severity >= 7.0), not waived"; .blocking), "", - tier("ACCEPTED — High band, waived in codeql-accepted-findings.json or in-source"; .accepted), + tier("ACCEPTED — High band, waived in codeql-accepted-findings.json"; .accepted), "", tier("ADVISORY — below the High band; triage in the Security tab"; .advisory), "", @@ -321,10 +334,11 @@ jobs: # go/log-injection at 6.1 included, so banding on it would block all # 23 log-injection findings and reproduce the deadlock this replaced. # - a rule with no security-severity falls back to level error/warning. - # - a (rule, file) pair in .github/codeql-accepted-findings.json, or an - # in-source `// codeql[rule-id]` comment, moves a High-band finding - # to ACCEPTED. The register is per-FILE, so the rule stays live - # everywhere else. + # - a (rule, file) pair in .github/codeql-accepted-findings.json moves + # a High-band finding to ACCEPTED. The register is per-FILE, so the + # rule stays live everywhere else. It is the ONLY waiver route: the + # in-source `// codeql[rule-id]` form does not work here (see the + # measured note by the matrix above). # # WHY NOT "any finding" — that was tried in #1246 and it deadlocked the # repo. See docs/adr/0048-codeql-severity-gating.md: the zero it was armed @@ -347,9 +361,11 @@ jobs: jq -r '.blocking[] | " \(.rule) \(.file):\(.line) (security-severity \(.sev))"' "$CLASSIFIED" echo "" echo "Fix it. If it is a false positive the honest options are a code" - echo "change that removes the sink, an in-source // codeql[rule-id]" - echo "comment, or an entry in .github/codeql-accepted-findings.json" - echo "with a written reason. Note that dismissing the alert in the" + echo "change that removes the sink, or an entry in" + echo ".github/codeql-accepted-findings.json with a written reason." + echo "An in-source // codeql[rule-id] comment does NOT work with this" + echo "pipeline — measured on #1249; see the note by the matrix above." + echo "Note that dismissing the alert in the" echo "Security tab will NOT turn this check green: this step reads the" echo "run's own SARIF and never consults the code-scanning API." exit 1 @@ -369,6 +385,7 @@ jobs: if: always() needs: [analyze] runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Fail if any CodeQL analysis did not succeed # RESULTS via env, not interpolated into the run: block. The values are @@ -379,6 +396,9 @@ jobs: env: RESULTS: ${{ join(needs.*.result, ' ') }} run: | + # RESULTS is supplied by the `env:` block above; shellcheck cannot see + # that and reads the all-caps name as a typo for the local `results`. + # shellcheck disable=SC2153 results="$RESULTS" echo "job results: $results" for r in $results; do diff --git a/.github/workflows/dev-ci.yml b/.github/workflows/dev-ci.yml index 027cdcbfc..34a623acf 100644 --- a/.github/workflows/dev-ci.yml +++ b/.github/workflows/dev-ci.yml @@ -53,6 +53,7 @@ jobs: go: name: Go compile / vet / lint / test (fast) runs-on: ubuntu-latest + timeout-minutes: 20 # Postgres service mirroring ci.yml's full-lane Go job (#723), so the # DB-gated suites — internal/store, internal/httpapi, internal/sched/*, @@ -133,11 +134,18 @@ jobs: - name: Test (no -race; DB suites run against the Postgres service, #723) # Same tag as make test / ci.yml — internal/sandbox's host tests only # compile with it. - run: go test -p 1 -tags fleet_host_executor ./... + # -count=1 defeats the build cache, matching ci.yml's three + # invocations. setup-go restores ~/.cache/go-build, which stores test + # RESULTS, and Go's cache key covers the binary, argv, env and files + # opened — it cannot see the Postgres SERVICE CONTAINER these suites talk + # to. Without it a DB-backed suite can print "ok (cached)" against a + # server it never contacted, which is a green check over an unrun test. + run: go test -p 1 -tags fleet_host_executor ./... -count=1 python: name: Python lint (ruff) runs-on: ubuntu-latest + timeout-minutes: 10 # Mirrors ci.yml's `python` job exactly, for the same reason the web lane was # added to this file: a change should not first be checked at the dev->main # promotion. ruff takes about a second, so there is no speed argument for @@ -193,6 +201,7 @@ jobs: web: name: Web lint / test / build (fast) runs-on: ubuntu-latest + timeout-minutes: 20 # The lane dev was missing entirely. Nothing here ran web/ at all, so a # web-only change — every one of the npm Dependabot PRs — reached dev with no # build behind it. Commands mirror ci.yml's `web` job exactly so the fast lane @@ -230,7 +239,7 @@ jobs: # registry flake skips with a notice (the audit above is the CVE gate). # Absolute path: this job's default working-directory is web/, which is # exactly how run 32579378165 caught the repo-relative form (exit 127). - run: "$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh" + run: '"$GITHUB_WORKSPACE/scripts/check-npm-overrides.sh"' - name: Install dependencies run: npm ci @@ -255,9 +264,40 @@ jobs: # dependency bump breaking the app. run: npm run build + helm: + # Same chart lint as ci.yml (#989): lint + minimal and everything-on + # renders. helm is preinstalled on the runner image. + name: Helm chart lint + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Lint and render the fleet chart + run: | + set -euo pipefail + helm version --short + helm lint deploy/helm/fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci + helm template ci deploy/helm/fleet --namespace fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci >/dev/null + helm template ci deploy/helm/fleet --namespace fleet \ + --set image.repository=registry.invalid/fleet --set image.tag=ci \ + --set sandbox.image=registry.invalid/fleet-sandbox:ci \ + --set postgres.enabled=true \ + --set web.enabled=true --set web.image=registry.invalid/fleet-web:ci \ + --set ingress.enabled=true --set ingress.host=fleet.example.com \ + --set networkPolicies.openEgress.create=true \ + --set 'networkPolicies.openEgress.blockedCIDRs={10.0.0.0/8}' \ + --set sandbox.kubernetes.runtimeClass=kata >/dev/null + migrations: name: Migration DDL lint runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -281,6 +321,7 @@ jobs: gitleaks: name: Secret scan (gitleaks) runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -304,13 +345,103 @@ jobs: - name: Run gitleaks run: gitleaks dir . --redact --exit-code 1 + actions: + # Lint the workflow files THEMSELVES. This repo carries ~3.1k lines of + # workflow YAML that decide what every other gate on this list even runs, + # and until now nothing checked them. The overlap with what already runs is + # narrow and worth stating: Semgrep's `p/github-actions` pack meets this on + # exactly ONE axis (actions referenced by a mutable tag) and CodeQL's + # `actions` language on taint into a checkout. Neither parses `${{ }}` + # expressions and neither shellchecks a `run:` block, which is where + # actionlint earns its place — expression syntax and type errors, undefined + # contexts, invalid `needs:` / `runs-on:` / cron, deprecated syntax, plus + # shellcheck over the bash in every `run:`. + # + # The tree was at ZERO findings when this gate went in, so a failure here is + # a regression rather than a backlog to wade through. Getting there took two + # real fixes and one waiver: `run: "$GITHUB_WORKSPACE/..."` in both lanes + # passed the path to the shell UNQUOTED (the YAML quotes are consumed by the + # YAML parser, so the shell never saw them, despite the comment above the + # line showing quoting was intended), and the SC2153 on `RESULTS` is an + # `env:`-supplied name shellcheck cannot see, waived in-line with a reason. + # + # Deliberately NOT gated on the docs-only classifier. A change under + # .github/workflows/ can never be docs-only by that allowlist anyway, but a + # broken workflow is the one failure that can disarm every OTHER gate here, + # so it runs unconditionally rather than depending on a classifier to let it. + name: Workflow + shell lint (actionlint, shellcheck) + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Install actionlint + # Pinned release + checksum, the same contract as the gitleaks step + # above: an upstream re-tag cannot silently change what this gate + # enforces. Checksum verified against the release's own + # actionlint_1.7.7_checksums.txt. + env: + ACTIONLINT_VERSION: '1.7.7' + ACTIONLINT_SHA256: '023070a287cd8cccd71515fedc843f1985bf96c436b7effaecce67290e7e0757' + run: | + set -euo pipefail + tarball="actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz" + curl -sSL -o "$tarball" \ + "https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/${tarball}" + echo "${ACTIONLINT_SHA256} ${tarball}" | sha256sum -c - + tar -xzf "$tarball" actionlint + sudo install -m 0755 actionlint /usr/local/bin/actionlint + actionlint --version + + - name: Run actionlint + # shellcheck ships preinstalled on the ubuntu-latest image, so actionlint + # finds it and the `run:` blocks get checked too. ASSERT that rather than + # assume it: a silently absent shellcheck drops half this job's coverage + # while the job still reports green — the green-but-vacuous outcome this + # repo keeps writing post-mortems about (see docs/SCANNING.md). + run: | + set -euo pipefail + command -v shellcheck >/dev/null || { + echo "::error::shellcheck is missing from the runner image; actionlint would silently skip every run: block." + exit 1 + } + shellcheck --version | head -2 + actionlint -color + + - name: Lint shell scripts (shellcheck) + # The 18 tracked *.sh files are ~6.2k lines, and they are the DEPLOY + # PATH, not helpers: scripts/update.sh, bootstrap.sh and doctor.sh are + # what `fleet update` / `fleet bootstrap` actually run on an operator's + # box. Go has golangci-lint, the web tier has oxlint + tsc, Python has + # ruff — bash had nothing, which is the same gap docs/SCANNING.md used to + # justify adding ruff, one language over. Several scripts already carry + # hand-written `# shellcheck` directives, so it was being run by hand; + # nothing made that reproducible. + # + # -S warning, and the tree is CLEAN at that level: the backlog was three + # findings total (two SC2034, one SC2148), all fixed or annotated with a + # reason in the same change that added this gate. Gating over an unfixed + # backlog is how a gate becomes something people learn to ignore, so the + # level was chosen by measuring, not by taste. The info tier (11 findings, + # mostly style) stays off. + run: | + set -euo pipefail + mapfile -t sh_files < <(git ls-files '*.sh') + printf 'shellchecking %d files\n' "${#sh_files[@]}" + # A vacuous pass is the failure mode this repo keeps writing up: if the + # glob ever matches nothing, say so instead of reporting green. + [ "${#sh_files[@]}" -gt 0 ] || { echo "::error::no shell scripts matched"; exit 1; } + shellcheck -S warning "${sh_files[@]}" + dev-gate: name: Dev gate # Aggregate check for branch protection: passes only when every fast-lane # job succeeded (mirrors ci.yml's `CI gate`). if: always() - needs: [go, python, codeql, semgrep, web, migrations, gitleaks] + needs: [go, python, codeql, semgrep, web, migrations, gitleaks, helm, actions] runs-on: ubuntu-latest + timeout-minutes: 5 steps: - name: Fail if any fast-lane job failed # RESULTS via env, not interpolated into the run: block. The values are @@ -321,6 +452,9 @@ jobs: env: RESULTS: ${{ join(needs.*.result, ' ') }} run: | + # RESULTS is supplied by the `env:` block above; shellcheck cannot see + # that and reads the all-caps name as a typo for the local `results`. + # shellcheck disable=SC2153 results="$RESULTS" echo "job results: $results" for r in $results; do diff --git a/.github/workflows/e2e-canary.yml b/.github/workflows/e2e-canary.yml index aa5aac1cb..a44c2d38a 100644 --- a/.github/workflows/e2e-canary.yml +++ b/.github/workflows/e2e-canary.yml @@ -31,6 +31,7 @@ jobs: guard: name: Check for OPENROUTER_API_KEY secret runs-on: ubuntu-latest + timeout-minutes: 5 outputs: has_key: ${{ steps.check.outputs.has_key }} steps: @@ -51,6 +52,7 @@ jobs: needs: guard if: ${{ needs.guard.outputs.has_key == 'true' }} runs-on: ubuntu-latest + timeout-minutes: 30 services: postgres: @@ -129,12 +131,25 @@ jobs: run: npx playwright test --project=canary --reporter=list - name: Upload canary report on failure + # Playwright report ONLY — deliberately not `.e2e-run/logs/`. + # + # This is the one job in the repo that boots fleet with the REAL + # OPENROUTER_API_KEY, and scripts/e2e-boot-server.sh redirects the + # server's stdout/stderr into that log directory. The repo is public, so + # an artifact here is world-downloadable. Nothing in the Go tree logs the + # key today (internal/agentcore/openrouter_models.go notes explicitly + # that it never enters that path), so this is residual rather than a + # known leak — it would take an upstream HTTP error string echoing an + # Authorization header. But the report is what a failure is actually + # diagnosed from, and the logs are not worth that tail risk. + # + # ci.yml's e2e-live job still uploads its logs, correctly: it runs + # against the fake LLM (cmd/fake-llm) with a throwaway key, so there is + # nothing there to leak. if: ${{ failure() }} uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: canary-report - path: | - web/playwright-report/ - .e2e-run/logs/ + path: web/playwright-report/ retention-days: 14 if-no-files-found: ignore diff --git a/.github/workflows/govulncheck-scheduled.yml b/.github/workflows/govulncheck-scheduled.yml index d29b3f06f..618e1e5c7 100644 --- a/.github/workflows/govulncheck-scheduled.yml +++ b/.github/workflows/govulncheck-scheduled.yml @@ -36,20 +36,31 @@ on: - cron: '0 8 * * *' workflow_dispatch: # Manual trigger (e.g. to confirm a fix cleared an advisory) +# NOTE: `issues: write` deliberately does NOT live at workflow scope. These +# lanes have one long job that runs third-party code with a full, +# default-branch token — `govulncheck@latest` resolved at run time, and a +# `podman build` pulling ~400 RPMs from Fedora mirrors. A workflow-level scope +# is live for every step of that job, so the scan ran next to a token that +# could open issues and write code-scanning alerts. The alarm needs no source +# tree and no scan output, so it is its own job holding `issues: write` alone — +# the same shape scan-cron-alarm.yml already uses for CodeQL and Semgrep. permissions: contents: read security-events: write # required to upload SARIF to the Security tab - issues: write # the failure-alarm step files an issue when a scheduled run breaks jobs: govulncheck-scheduled: name: Daily Go vulnerability scan runs-on: ubuntu-latest + timeout-minutes: 20 steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: + # Nothing here pushes or does an authenticated fetch, so do not leave + # the job's token in .git/config for the third-party code below. + persist-credentials: false ref: main # always scan the tip of main, not a PR branch - name: Set up Go @@ -100,6 +111,19 @@ jobs: sarif_file: 'govulncheck.sarif' category: 'govulncheck-scheduled' + + alarm: + # Split out of the scan job above so `issues: write` is the ONLY scope in + # play here, and it is held by a job that checks out nothing and runs no + # third-party code — it only calls `gh`. + name: File an issue so a red cron cannot rot silently + needs: [govulncheck-scheduled] + if: ${{ failure() && github.event_name == 'schedule' }} + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + issues: write + steps: - name: File an issue so a red cron cannot rot silently # A scheduled failure has no PR attached, so nothing surfaces it — the # exact rot pattern that let the CodeQL toolchain break sit red for @@ -107,7 +131,6 @@ jobs: # re-failures. schedule-only: a red manual dispatch has a human watching. # This step body is duplicated across the four scheduled scan lanes # (codeql, semgrep, govulncheck, grype) — keep them in sync. - if: ${{ failure() && github.event_name == 'schedule' }} env: GH_TOKEN: ${{ github.token }} WORKFLOW_NAME: ${{ github.workflow }} diff --git a/.github/workflows/grype-scheduled.yml b/.github/workflows/grype-scheduled.yml index d48f4d6e1..3b08f984a 100644 --- a/.github/workflows/grype-scheduled.yml +++ b/.github/workflows/grype-scheduled.yml @@ -19,15 +19,23 @@ on: - cron: '0 9 * * 1' # Every Monday at 09:00 UTC workflow_dispatch: # Allow a manual trigger (e.g. after a base-image bump) +# NOTE: `issues: write` deliberately does NOT live at workflow scope. These +# lanes have one long job that runs third-party code with a full, +# default-branch token — `govulncheck@latest` resolved at run time, and a +# `podman build` pulling ~400 RPMs from Fedora mirrors. A workflow-level scope +# is live for every step of that job, so the scan ran next to a token that +# could open issues and write code-scanning alerts. The alarm needs no source +# tree and no scan output, so it is its own job holding `issues: write` alone — +# the same shape scan-cron-alarm.yml already uses for CodeQL and Semgrep. permissions: contents: read security-events: write # required to upload SARIF to the Security tab - issues: write # the failure-alarm step files an issue when a scheduled run breaks jobs: grype-scheduled: name: Weekly container vulnerability scan runs-on: ubuntu-latest + timeout-minutes: 35 env: SANDBOX_IMAGE_LOCAL: localhost/fleet-sandbox:latest @@ -35,6 +43,9 @@ jobs: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: + # Nothing here pushes or does an authenticated fetch, so do not leave + # the job's token in .git/config for the third-party code below. + persist-credentials: false ref: main # always scan the tip of main, not a PR branch - name: Install Grype @@ -88,6 +99,19 @@ jobs: sarif_file: 'grype-results.sarif' category: 'grype-scheduled' + + alarm: + # Split out of the scan job above so `issues: write` is the ONLY scope in + # play here, and it is held by a job that checks out nothing and runs no + # third-party code — it only calls `gh`. + name: File an issue so a red cron cannot rot silently + needs: [grype-scheduled] + if: ${{ failure() && github.event_name == 'schedule' }} + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + issues: write + steps: - name: File an issue so a red cron cannot rot silently # A scheduled failure has no PR attached, so nothing surfaces it — the # exact rot pattern that let the CodeQL toolchain break sit red for @@ -95,7 +119,6 @@ jobs: # re-failures. schedule-only: a red manual dispatch has a human watching. # This step body is duplicated across the four scheduled scan lanes # (codeql, semgrep, govulncheck, grype) — keep them in sync. - if: ${{ failure() && github.event_name == 'schedule' }} env: GH_TOKEN: ${{ github.token }} WORKFLOW_NAME: ${{ github.workflow }} diff --git a/.github/workflows/publish-sandbox-image.yml b/.github/workflows/publish-sandbox-image.yml index 1335e2875..7f7a07fcd 100644 --- a/.github/workflows/publish-sandbox-image.yml +++ b/.github/workflows/publish-sandbox-image.yml @@ -98,7 +98,7 @@ name: Publish sandbox image (reusable) # # NO image_name NEEDED. It is derived from the bundle itself — see "WHERE THE # IMAGE NAME COMES FROM" below. Pass it only to publish somewhere other than -# GHCR under the caller's own owner (e.g. an ECR ref, per docs/EKS-DEPLOYMENT.md). +# GHCR under the caller's own owner (e.g. an ECR ref, per docs/DEPLOYMENT-KUBERNETES.md). # # WHERE THE IMAGE NAME COMES FROM # @@ -204,6 +204,7 @@ jobs: publish: name: Build + push sandbox image runs-on: ubuntu-latest + timeout-minutes: 60 outputs: image_ref: ${{ steps.push.outputs.image_ref }} image_digest: ${{ steps.push.outputs.image_digest }} @@ -216,6 +217,11 @@ jobs: steps: - name: Checkout caller repo uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + # This job holds a write scope. Nothing in it pushes or does an + # authenticated fetch, so do not leave the token in .git/config while + # a PR-controlled Containerfile goes through `podman build`. + persist-credentials: false # The build script lives in fleet core; a client-config caller doesn't # have it. Checking it out into a subdir keeps ONE canonical builder @@ -229,6 +235,31 @@ jobs: # actions/untrusted-checkout under security-extended; the same pattern in # build-sandbox-image.yml was flagged; this file escaped both query # variants despite holding packages: write — the MORE dangerous twin.) + - name: Validate bundle_dir + # `fleet_ref`, which reaches the SAME build-script invocation, got a + # character allow-list, a bare-SHA refusal and a GITHUB_OUTPUT-newline + # fix. `bundle_dir` got nothing, and the asymmetry is the whole reason + # this exists — there is no shell injection here (it goes through `env:` + # and is quoted at every use), but nothing stopped `../../.fleet-core` + # or an absolute path from silently repointing the build at a different + # manifest. The reusable workflow runs in the CALLER's context with the + # caller's token, so the blast radius is a client repo misbuilding its + # own image; this is defence in depth and consistency, not a live hole. + env: + REQUESTED: ${{ inputs.bundle_dir }} + run: | + set -euo pipefail + case "$REQUESTED" in + ""|.) ;; + /*) + echo "::error::bundle_dir must be relative to the caller repo, not absolute."; exit 1 ;; + *..*) + echo "::error::bundle_dir must not contain '..' — it may only name a path inside the caller repo."; exit 1 ;; + *[!a-zA-Z0-9._/-]*) + echo "::error::bundle_dir contains a character outside [A-Za-z0-9._/-]. Refused before the build."; exit 1 ;; + esac + echo "bundle_dir accepted: '${REQUESTED:-.}'" + - name: Pin fleet_ref to collaborator-controlled refs id: pin env: diff --git a/.github/workflows/scan-cron-alarm.yml b/.github/workflows/scan-cron-alarm.yml index 07b62a4ec..c55c77c37 100644 --- a/.github/workflows/scan-cron-alarm.yml +++ b/.github/workflows/scan-cron-alarm.yml @@ -55,6 +55,7 @@ jobs: github.event.workflow_run.conclusion != 'success' && github.event.workflow_run.conclusion != 'skipped' runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: File or update the alarm issue # Body mirrors the in-job alarm steps in govulncheck-scheduled.yml and diff --git a/.github/workflows/screenshots.yml b/.github/workflows/screenshots.yml index 729d4184d..1fa1d3d73 100644 --- a/.github/workflows/screenshots.yml +++ b/.github/workflows/screenshots.yml @@ -48,6 +48,7 @@ jobs: screenshots: name: Generate GUI screenshots runs-on: ubuntu-latest + timeout-minutes: 30 defaults: run: working-directory: web diff --git a/.github/workflows/semgrep.yml b/.github/workflows/semgrep.yml index 50c0a8f5f..4d79e4014 100644 --- a/.github/workflows/semgrep.yml +++ b/.github/workflows/semgrep.yml @@ -98,9 +98,21 @@ jobs: # python3, and this repo pins no setup-python version anywhere else, so # adding one would be inventing an unverified action pin for nothing. # - # The semgrep version IS pinned, like every other tool this repo installs - # in CI (gitleaks, grype, golangci-lint), so an upstream release cannot - # change the findings under us without a visible diff. + # The semgrep version IS pinned, so an upstream release cannot change the + # findings under us without a visible diff. + # + # It is NOT the same guarantee gitleaks and grype get, and this comment + # used to claim it was. Those two are downloaded at a pinned version AND + # sha256-verified; `pip install semgrep==X` pins semgrep and resolves its + # ~40-package dependency closure (attrs, click, glom, requests, rich, + # ruamel.yaml, urllib3, …) to whatever is latest that day, unverified — + # inside a job that sits in both merge gates. The mitigating fact, which + # is why this is a caveat and not an incident: PyPI forbids re-uploading + # an existing version, so the exposure is a NEW malicious release of a + # transitive dep, not silent mutation of a pinned one — materially weaker + # than the mutable-GitHub-release-asset case the pinning exercise fixed. + # Closing it properly means a `--require-hashes` requirements file + # regenerated on every semgrep bump; tracked, not done here. env: SEMGREP_VERSION: '1.174.0' run: | @@ -121,13 +133,20 @@ jobs: # The JSON is written first and the exit status captured, so the # summary step below still runs and still says WHAT failed — a gate # that fails without printing the finding is a gate nobody can act on. - semgrep scan \ + # + # The `if` is load-bearing. `set -uo pipefail` does NOT clear the `-e` + # the runner supplies (GitHub's default shell for a `run:` block is + # `bash -e {0}`), so a bare `semgrep ...` followed by `status=$?` + # aborts the step ON the semgrep line and never reaches the capture — + # the deferral this block describes never actually happened. Capturing + # through an `if` is exempt from `-e` and makes the comment true. + if semgrep scan \ --config p/github-actions \ --config p/golang \ --config p/javascript \ --config p/python \ --metrics=off --error --json -o semgrep.json --quiet - status=$? + then status=0; else status=$?; fi if [ ! -s semgrep.json ]; then echo "semgrep produced no JSON — treating as a scan failure" >&2 exit 1 diff --git a/AGENTS.md b/AGENTS.md index aaa527824..7e49ef614 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,46 +11,96 @@ this file is the agent-facing distillation, not a replacement for them. ## What fleet is (one paragraph) fleet is a self-hosted, general-purpose agent platform. **One** Go process runs -interactive chat *and* a scheduling engine on one box, driven by **one** unified -agent runtime (`internal/agentcore`). Model-authored local execution — bash, -Python, and file I/O — runs inside a rootless-Podman sandbox; fixed host-side -brokers handle MCP credentials/network and the small control-plane exception -set enumerated in ADR-0036. See -the README "Architecture at a glance" for the full picture. +interactive chat *and* a scheduling engine, driven by **one** unified agent +runtime (`internal/agentcore`). Model-authored local execution — bash, Python, +and file I/O — runs inside a mandatory sandbox; fixed host-side brokers handle +MCP credentials/network and the small control-plane exception set enumerated in +ADR-0036. The sandbox **backend** is pluggable (ADR-0049): +`FLEET_SANDBOX_BACKEND=podman` — the default, rootless Podman co-located with +the process, which is the single-box install — or `kubernetes`, one ephemeral +pod per sandbox exec'd over the apiserver, for the split control-plane +deployment. "On one box" is the default shape, not the only one, so do not write +code (or docs) that assumes podman is the only executor. See the README +"Architecture at a glance" for the full picture. ## Build · test · lint (run before opening any PR) ```sh make build # compile-check ./... AND emit ./fleet + ./fleet-admin -make compile # go build ./... (compile-check only; no artifacts) -make test # go test -p 1 ./... — run in the FOREGROUND -make test-race # go test -race -p 1 ./... (use when touching concurrency) -make test-cover # run Go tests with coverage profiling (writes coverage.out) -make lint # golangci-lint + ruff check/format (Python) + migration DDL lint — must pass clean +make compile # go build ./... (release config — see the build tag below) +make test # go test -p 1 -tags fleet_host_executor ./... — run in the FOREGROUND +make test-race # the same, with -race (use when touching concurrency) +make test-cover # the same, with -coverprofile/-covermode=atomic (writes coverage.out) +make lint # golangci-lint + ruff check/format (Python) + migration DDL lint + # + actionlint & shellcheck (workflows + shell) — must pass clean +make govulncheck # call-graph-aware CVE scan of the dependency tree make fmt # gofmt -w . make tidy # go mod tidy +make ci-go # the whole Go gate locally: compile, vet, lint, test, -race, govulncheck +make ci-web # the Web CI job verbatim (see below) +make ci-local # ci-go + ci-web — the fast PR gates, locally ``` +**`-tags fleet_host_executor` is load-bearing, not decoration.** The unsandboxed +host executor is fenced behind that build tag (#159) so it is *not* compiled into +a release binary: `make compile` deliberately omits it, and `host_disabled.go` +then stubs `newHostSandbox` out and rejects MockMode at boot. Tests opt in — every +`go test`/`go vet` target above carries the tag, and `.golangci.yml` sets +`build-tags` so the linter agrees — so a bare `go test ./...` builds a *different* +tree than CI does (`host.go` unvetted, untested). Use the Makefile targets. + +`make lint`'s `lint-python` (ruff) and `lint-actions` (actionlint/shellcheck) +**skip loudly when the tool is missing**, printing the install command. So a green +local `make lint` is not proof — read the output, and remember CI enforces both +regardless. + When you touch `web/` (the Next.js app): ```sh -cd web && npm audit --audit-level=low && npm ci && npm run lint && npm run typecheck && npm run test && npm run build +make ci-web # the Web CI job, verbatim — prefer this +cd web && npm ci && npm run lint && npm run typecheck && npm run test && npm run build cd web && npx playwright test --project=mocked # mocked e2e ``` -CI mirrors all of this — Go build/vet/lint/test (including a `-race` lane) plus a -`govulncheck` dependency-CVE scan, a Grype container-image CVE scan (fail on a -fixable CRITICAL/HIGH) of the sandbox image, a Python lint (ruff), web lint (oxlint) / typecheck (TS 7) / test / build, Playwright (mocked -**and** live, against a real backend + sandbox), a migration DDL lint, and a -gitleaks secret scan. **Every job must be green before merge.** Tests are -deterministic without a live model: use the fake-LLM seam (`internal/fakellm` -via `OPENROUTER_BASE_URL`), never a real key. +There are **two** npm trees — `web/` and `scripts/rampart-service/` — and CI +audits both (`npm audit --audit-level=low`, lockfile-only) plus +`scripts/check-npm-overrides.sh`, the override canary. `make ci-web` runs all +eight steps; the hand-rolled line above skips the audits and the canary, which is +how a clean local run turns into a red PR. + +CI mirrors all of this across **two lanes**, and which one you get depends on the +branch you target: + +- **`CI` (`ci.yml`) — `main` only** (pushes to `main` and PRs targeting it). The + full gate: Go build/vet/lint/test (including a `-race` lane) plus a + `govulncheck` dependency-CVE scan, a Grype container-image CVE scan (fail on a + fixable CRITICAL/HIGH) of the sandbox image, a Python lint (ruff), a workflow + + shell lint (actionlint & shellcheck), web lint (oxlint) / typecheck (TS 7) / + test / build, Playwright (mocked **and** live, against a real backend + + sandbox), a Helm chart lint, a migration DDL lint, and a gitleaks secret scan. + `CI gate` is the **single required status check** on `main`: it `needs` every + other job and always reports, so a docs-only PR (heavy jobs skipped by the + `changes` classifier) still merges, while a code PR cannot go green over a skip. +- **`Dev CI (fast lane)` (`dev-ci.yml`) — `dev` only** (pushes to `dev` and PRs + targeting it). Compile/vet/lint/test against a Postgres service, ruff, the web + lane, the migration DDL lint, gitleaks, actionlint/shellcheck, the Helm lint, + CodeQL and Semgrep. Deliberately deferred to the promotion PR: the `-race` + lane, govulncheck, the Grype image scan, and both Playwright suites. There is + no docs-only classifier here — the fast lane runs on every change. + +`ci.yml` does not fire on `dev` at all, so **the dev→main promotion PR is the +first time the full gate ever sees that code**; expect it to surface things dev +never told you about. **Every job must be green before merge**, and nothing +merges itself — auto-merge was removed, so every PR, dependency bumps included, +waits for a human. Tests are deterministic without a live model: use the fake-LLM +seam (`internal/fakellm` via `OPENROUTER_BASE_URL`), never a real key. CodeQL (security queries, `security-extended`) and Semgrep (Go/JS/Python SAST + -Actions supply chain) also run per PR and are **inside `ci-gate` and `Dev gate`** -— both are reusable workflows that ci.yml/dev-ci.yml call as jobs. `npm audit` -(both npm trees, lockfile-only, any severity) and ruff (`check` **and** -`format --check`) gate the same way. +Actions supply chain) run per PR in **both** lanes: `codeql.yml` and `semgrep.yml` +are reusable workflows that ci.yml/dev-ci.yml call as jobs, so their results roll +up into `CI gate` / `Dev gate` like any other job. `npm audit` (both npm trees, +lockfile-only, any severity) and ruff (`check` **and** `format --check`) gate the +same way. Their thresholds differ, and the difference is load-bearing: @@ -75,10 +125,13 @@ mergeable PR. See [`docs/SCANNING.md`](docs/SCANNING.md) ("Known gaps"). See the README "Repository layout" for the annotated tree. In short: `cmd/` (the one unified `fleet` binary — `fleet serve` runs the server, every other verb is the operator CLI; `fleet-admin` is a transitional deprecation shim that still works for -one release), `internal/` (`agentcore` the one run loop, `sandbox`, +one release; plus the `fleet-bench`, `fake-llm` and `sandbox-probe` harness +binaries), `internal/` (`agentcore` the one run loop, `sandbox`, `mcp`, `creds`, `clientconfig`, `store`, `sched`, `httpapi`, …), `web/` (one -Next.js app: `/chat` + `/orchestrator`), and `config/default/` (the generic -client bundle baked in so fleet runs bare). +Next.js app: `/chat`, `/orchestrator`, `/settings`, `/admin`), `deploy/` (the +systemd units + Caddyfile for the single-box install and the +`deploy/helm/fleet` chart for the Kubernetes one), and `config/default/` (the +generic client bundle baked in so fleet runs bare). ## Non-negotiable invariants — do NOT weaken these @@ -88,11 +141,21 @@ recorded as Architecture Decision Records in [`docs/adr/`](docs/adr/) — a chan that adds, weakens, or reverses an invariant must add or supersede an ADR in the same PR. -- **The sandbox is mandatory.** The agent loop runs in the fleet process, but - every agent tool call's data-plane execution — bash, Python, **and file I/O +- **The sandbox is mandatory** — the *backend* is pluggable, the sandbox is not. + The agent loop runs in the fleet process, but every agent tool call's data-plane + execution — bash, Python, **and file I/O (`view_file`/`write_file`/`edit_file`, via the sandbox FileOp seam, #784)** — - runs inside the rootless-Podman sandbox; there is **no** fast path that skips - it and no host-execution fallback (they fail closed without a sandbox). The + runs inside the sandbox (rootless Podman by default; an ephemeral Kubernetes pod + under `FLEET_SANDBOX_BACKEND=kubernetes`, ADR-0049); there is **no** fast path + that skips it and no host-execution fallback (they fail closed without a + sandbox). The unsandboxed host executor is compiled in **only** behind the + `fleet_host_executor` build tag (#159), which is what makes "it cannot ship + enabled in a production build" a property of the artifact rather than a runtime + flag — do not widen that fence, and do not add a path that reaches `host.go` + from an untagged build. Selecting the kubernetes backend runs a fail-closed boot + preflight (apiserver + credentials, the exact RBAC verbs, the workspace claim, + the sealed-egress NetworkPolicy object) and refuses podman-only knobs rather + than ignoring them: no degrade to podman, none to host execution. The loop holds no privileged local executor of its own: each tool call is handed to the sandbox under host policy. A small set of native tools are host-side **control-plane / broker** operations by design (host network fetch, brokered @@ -141,9 +204,10 @@ same PR. upload only ever produced a missing-token warning. Treat coverage as a quality signal, not a gate: add tests that catch real behavior, not to chase a number. (The merge gates are build/vet/lint, ruff — `check` and - `format --check` — the test suites, the `-race` lane, govulncheck, Grype, - `npm audit` + `scripts/check-npm-overrides.sh`, CodeQL, Semgrep, the migration - linter, and gitleaks.) + `format --check` — actionlint + shellcheck, the test suites, the `-race` lane, + govulncheck, Grype, `npm audit` + `scripts/check-npm-overrides.sh`, CodeQL, + Semgrep, the Helm chart lint, both Playwright suites, the migration + linter, and gitleaks — all rolled up into the one required `CI gate` check.) - **Match the surrounding code:** naming, idioms, and comment density. The `internal/agentcore` package comments explain *why* each governance invariant holds — preserve that level of explanation when you extend it. @@ -168,8 +232,10 @@ same PR. to this file — that is how it grew past 300 lines once already; the historical notes now live in [`docs/FEATURE-NOTES.md`](docs/FEATURE-NOTES.md). - One focused branch + PR per change; keep diffs scoped. Don't refactor unrelated - code in a feature PR. See `CONTRIBUTING.md` for branch/PR conventions and DCO - sign-off. + code in a feature PR. `.github/PULL_REQUEST_TEMPLATE.md` asks for exactly the + three things above — what/why, what you actually ran to verify it, and + scope-and-deviations — so fill it in rather than deleting it. See + `CONTRIBUTING.md` for branch/PR conventions. ## Where to look @@ -237,6 +303,12 @@ same PR. - **Installing the backup/maintenance timers on an existing box** (`fleet timers install`, the `fleet update` offer + `--no-timers` opt-out, the non-systemd/Kubernetes posture): [`docs/TIMERS.md`](docs/TIMERS.md) +- **Kubernetes as a first-class deployment** (the `deploy/helm/fleet` chart, + the pluggable sandbox backend — `FLEET_SANDBOX_BACKEND=podman|kubernetes`, + sandboxes as ephemeral pods, the fail-closed cluster preflight, and the + honest deviations from the podman backend): + [`docs/DEPLOYMENT-KUBERNETES.md`](docs/DEPLOYMENT-KUBERNETES.md) + + [ADR-0049](docs/adr/0049-kubernetes-backend-split-control-plane.md) - **Load testing & benchmarks** (`fleet-bench` HTTP chat load via the fake-LLM seam + subsystem throughput benchmarks): [`docs/LOAD-TESTING.md`](docs/LOAD-TESTING.md) - **Prompt-cache prefix-stability contract** (what must stay byte-stable in the diff --git a/CHANGELOG.md b/CHANGELOG.md index 707592105..4bd3167da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,228 @@ prior versions are listed because none have shipped. ## [Unreleased] +### Added + +- **A workflow + shell lint gate (`actionlint`, `shellcheck`):** nothing checked + the ~3.1k lines of workflow YAML that decide what every other gate runs, and + nothing checked the ~6.2k lines of bash that *are* the deploy path + (`update.sh` / `bootstrap.sh` / `doctor.sh`). Both now block in `ci.yml` and + `dev-ci.yml` and run from `make lint` via `lint-actions`, pinned and + checksum-verified like `gitleaks` and `grype`. `actionlint` covers what + Semgrep's `p/github-actions` pack and CodeQL's `actions` language do not — + `${{ }}` expression syntax and types, undefined contexts, invalid + `needs:`/`runs-on:`/cron — plus shellcheck over every `run:` block. + Both start at **zero findings**, measured: the 5 actionlint and 3 shellcheck + items were fixed or annotated with reasons in the same change, so a failure is + a regression rather than a backlog. +- **Two new CI invariant tests**, in the spirit of the existing pin/gate tests: + every workflow must declare a top-level `permissions:` block (without one it + silently inherits the repository default), and the `actionlint` + version/checksum must agree across both lanes. +- **PR and issue templates.** The PR template prompts for what/why, how it was + verified, and scope-and-deviations, plus the obligations that were previously + a reviewer's job to remember (CHANGELOG entry, `docs/.md` note, an + ADR in the *same* PR when an invariant moves). `ISSUE_TEMPLATE/config.yml` + disables blank issues so the private security-disclosure link is unmissable — + `SECURITY.md` says "do not open a public issue for a vulnerability" and the + New Issue button was offering a blank box with no such warning. + +### Changed + +- **Automatic merging removed.** `auto-merge-dependabot.yml` is deleted and every + reference to it across `SECURITY.md`, `dependabot.yml`, `CODEOWNERS` and + `docs/SCANNING.md` is gone. Its own header had argued the case against it: it + explained that `gh pr merge --auto` holds a merge only on *required* checks, + named `dev` as a branch that requires none, and then listed `dev` in its own + `branches:` filter — so the mitigation the docs credited was in fact the + delivery mechanism for same-day patch bumps landing unattended. Every + dependency bump now waits for a human. +- **DCO sign-off is no longer requested.** The requirement was documented in + `CONTRIBUTING.md` and enforced nowhere; rather than add a gate for it, the + requirement was dropped. +- **`ci.yml` gained a `concurrency:` group** (the expensive lane had none, so + stacked pushes ran full suites to completion). Cancellation is scoped to + `pull_request` — a push to `main` is the only tree-wide CodeQL verdict and must + not be cancelled. **`timeout-minutes` is now set on every job** (it was present + in 2 of 13 workflows). +- **`issues: write` no longer sits at workflow scope** in the two scheduled scan + lanes, where it was live for every step of a long job running + `govulncheck@latest` and a podman build pulling ~400 RPMs. It moved to a + dedicated alarm job that checks out nothing — the shape `scan-cron-alarm.yml` + already used. `persist-credentials: false` added to the write-scoped checkouts. +- **`make ci-web` now mirrors the real web job.** It ran 4 of its 8 steps, + dropping both `npm audit` gates, the override canary and the explicit + `npm run typecheck` — a clean local run and a red PR. + +### Fixed + +- **`AGENTS.md` reconciled with the tree it describes.** The Kubernetes backend + (ADR-0049) landed an index row and nothing else, so the headline paragraph and + the "sandbox is mandatory" invariant still called the sandbox + *rootless-Podman* — an agent reading only that file would treat + `FLEET_SANDBOX_BACKEND=kubernetes` as an invariant violation rather than a + supported backend. Both now say pluggable-backend/mandatory-sandbox, and the + invariant additionally names the mechanism it had left implicit: the + `fleet_host_executor` build-tag fence (#159) and the kubernetes backend's + fail-closed preflight. Also corrected: the `make test`/`test-race`/`test-cover` + lines omitted `-tags fleet_host_executor` (a bare `go test ./...` builds a + different tree than CI); the web block was a stale copy of the CI job that + dropped the second npm tree (`scripts/rampart-service`) and the override + canary, where `make ci-web` now runs all eight steps; `make govulncheck`, + `ci-go`, `ci-web` and `ci-local` were missing from the target list; CI was + described as one lane when `ci.yml` fires on `main` only and `dev-ci.yml` is + `dev`'s only signal (so the promotion PR is the first full-gate run); the + merge-gate enumeration omitted actionlint/shellcheck, the Helm lint and the + Playwright suites; and the repository map omitted `deploy/`, the harness + binaries under `cmd/`, and the `/settings` + `/admin` web routes. + +- **Two green-but-vacuous holes in `ci.yml`.** The `postgresql-client-18` install + was best-effort (`|| echo`), so an unreachable PGDG left client 16 in place and + `backup_test.go`'s major-mismatch `t.Skipf` turned the *only* coverage of + `fleet backup` / `fleet restore` off behind the single required check on + `main`; it now asserts the major — and asserts it twice, because the first + version of that assertion could not pass: installing `postgresql-client-18` + does not change what `pg_dump` resolves to. `/usr/bin/pg_dump` is + postgresql-common's `pg_wrapper`, which dispatches on the version/cluster in + `~/.postgresqlrc` or `/etc/postgresql-common/user_clusters` rather than on the + newest client present, so on a runner carrying a PostgreSQL 16 cluster the + wrapper kept selecting 16 and the step failed with `got=16` over a successful + install. The versioned bin dir now goes first on `$GITHUB_PATH` — which is + what the round-trip test needs anyway, since it execs `pg_dump` off PATH — and + the install is asserted by absolute path in that step, PATH resolution in the + next one (`$GITHUB_PATH` only applies from the following step on). And the + docs-only classifier initialised + `docs_only=true` and only ever cleared it inside its loop, so an **empty** diff + classified as docs-only and skipped the suite — which `ci-gate` then waved + through, because an empty diff is the absence of evidence, not evidence that + only prose changed. +- **`dev-ci.yml`'s `go test` could report cached results.** It lacked the + `-count=1` that `ci.yml` passes on all three of its invocations; `setup-go` + restores the build cache, which holds test *results*, and Go's cache key cannot + see a Postgres service container. +- **Semgrep's deferred-failure path was unreachable.** `set -uo pipefail` does not + clear the `-e` the runner supplies, so the step aborted on the `semgrep` line + and `status=$?` never ran. The outcome was still correct; the documented design + was not what executed. +- **The docs advertised a CodeQL waiver route that does not work.** Five places — + `SCANNING.md`, `CODEQL.md`, ADR-0048, `CONTRIBUTING.md` and the gate's own + runtime advice — offered an in-source `// codeql[rule-id]` comment, while + `codeql.yml` recorded that all three forms were tried on #1249 and none + produced a `suppressions` array. `codeql.yml` carried *both* claims, the stale + one directly above the note refuting it. The accepted-findings register is the + only route that works. +- **ADR-0048's normative Decision stated the gate threshold as an OR** where + `codeql-gate.jq` implements a fallback; read literally it blocks on + `go/log-injection` (`error` at security-severity 6.1) — the exact deadlock the + ADR exists to undo. +- Assorted doc drift: `CONTRIBUTING.md` named Go 1.26.x (`go.mod` says 1.27.0); + `CODEOWNERS` enumerated 7 of `ci-gate`'s 13 `needs`; `semgrep.yml` claimed its + `pip install` had the same checksum guarantee as the `gitleaks`/`grype` + downloads, which it does not. + +- **Kubernetes as a first-class deployment (#989 / ADR-0049):** the fleet + control plane can now run in a cluster with agent sandboxes as **ephemeral + pods**, selected by one knob — `FLEET_SANDBOX_BACKEND=podman|kubernetes` + (env overrides the bundle manifest's `sandbox.backend`, mirroring + `sandbox.runtime`'s precedence; an unrecognized value refuses to boot). + What landed, in one pass per the issue's plan: + + - **A third sandbox backend** in `internal/sandbox` (`k8sImpl`) behind the + same internal interface as the podman and host executors: one pod per + sandbox (`sleep infinity` + exec over the apiserver's WebSocket channel + protocol), bash as one-shot execs, the python bridge as a held exec + session, and file ops running the same embedded `fileops.py`. Pods are + read-only-rootfs, non-root (uid 1000), all capabilities dropped, seccomp + RuntimeDefault (or an operator-installed Localhost profile), + `automountServiceAccountToken=false` — and the workspace is a shared + ReadWriteMany PVC mounted at the **same absolute path** as the control + plane, preserving the same-path invariant. The #796 poison-and-retire + containment carries over: a cancelled/timed-out call deletes the pod with + zero grace and retires the sandbox. A boot-time orphan-pod sweep mirrors + the podman container prune. No client-go: a minimal hand-rolled REST + + WebSocket-exec client (gorilla/websocket was already in the tree — zero + new modules), with kubeconfig support deliberately narrow (token / + client-cert; exec plugins and `insecure-skip-tls-verify` refused). + - **Fail-closed preflight** when the backend is selected, at boot and in + `fleet validate-config`: apiserver reachability + credentials, the exact + RBAC verbs (pods create/get/list/delete, pods/exec create), the workspace + claim, the sealed-egress NetworkPolicy object, and the RuntimeClass when + configured. Podman-only knobs are refused rather than silently ignored + (`FLEET_SANDBOX_RUNTIME` → use `FLEET_SANDBOX_K8S_RUNTIME_CLASS`; + `FLEET_SANDBOX_SECCOMP_PROFILE` → `FLEET_SANDBOX_K8S_SECCOMP_PROFILE`; + `FLEET_DEFAULT_NETWORK_MODE=allowlisted` is unsupported — the host egress + proxy is unreachable from pods). + - **Dedicated runner pools**: sandbox pods can be pinned to their own node + pool with `FLEET_SANDBOX_K8S_NODE_SELECTOR` ("k=v,k=v") and + `FLEET_SANDBOX_K8S_TOLERATIONS` (a JSON array), or the manifest's + structured `sandbox.kubernetes.node_selector` / `.tolerations` — fleet's + horizontal scaling story made concrete (more runner capacity = a bigger + pool, never more fleet replicas). Malformed values refuse to boot. Sandbox + pods also pin `imagePullPolicy: IfNotPresent` explicitly (the API's + `Always`-for-`:latest` default breaks side-loaded kind images and re-pulls + a mutable tag mid-run). + - **A Helm chart** (`deploy/helm/fleet`): single-replica control-plane + Deployment (strategy Recreate, deliberately no replica knob — the + scheduler is single-owner), the runner RBAC Role/Binding, workspace/data + PVCs, the `fleet-sandbox-deny-all` NetworkPolicy (selecting pods labeled + `fleet.elcanotek.com/egress=none`), optional egress shaping for open + pods, optional evaluation Postgres, optional web tier + Ingress. Linted + and template-rendered in CI (new `helm` job inside both gates). + - **Docs**: `docs/DEPLOYMENT-KUBERNETES.md` — the one Kubernetes reference: + 15-minute kind path, the two image builds (the control-plane + Containerfile's `FROM golang:` stage is now pinned to go.mod by + `scripts/check_versions_test.go`), production checklist, provider notes + (EKS/GKE/AKS), day-2 operations (the CronJob equivalents of the systemd + timers), troubleshooting, and an explicit honest-deviations list + (NetworkPolicy enforcement belongs to the CNI, no per-pod pids limit, no + #263 resource telemetry, no bundled-seccomp/supporting-doc mounts). Plus + updates to `DEPLOYMENT.md` and `SANDBOX-RUNTIMES.md` + (`FLEET_SANDBOX_BACKEND` documented next to `FLEET_SANDBOX_RUNTIME`). + [ADR-0049](docs/adr/0049-kubernetes-backend-split-control-plane.md) + amends ADR-0004: the single-box podman install **stays the default and is + unchanged**; only the no-k8s-artifacts enforcement clause is superseded. + +- **Kubernetes: bundle docs can serve the file tools again + (`sandbox.kubernetes.bundle_docs_in_image` / + `FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE`).** A sandbox pod mounts only the + workspace claim, so the supporting-doc bind mounts (`protocols/`, + `personas/`, `system_prompts/`, skills) do not apply — and because the + fileop path anchor only trusts roots that are actually mounted, dropping + them made `view_file protocols/foo.yaml` a *refusal* + (`fileop root is not inside a sandbox bind mount`), not a miss. For a + protocol-driven bundle that is most of the product. An operator whose + sandbox image carries the bundle's doc dirs at the **same absolute paths** + the control plane reads them from can now declare it, and those roots keep + their read anchors inside a pod, so the file tools work exactly as they do + under podman. + + The declaration cannot widen anything: it re-admits *read-only* anchors for + roots the operator already configured, the read still executes inside the + sandbox, and fleet cannot inspect an image — so a wrong declaration surfaces + as a not-found read (the podman missing-dir behavior), never as a boundary + change. It covers only the bundle's own doc dirs; other entries in the mount + list (the uploads root) stay dropped, each with a log line. A malformed + value refuses to boot, at boot and in `fleet validate-config`, which also + now reports which way it resolved. + + One case no declaration can fix, now stated plainly in the docs: a bundle + that inherits fleet's built-in skills pack resolves `SkillsDir` to a merged + tree under the control plane's data dir, which no sandbox image can carry — + so in-sandbox skill reads need the bundle's `skills_builtin: false`, and + there is no configuration that yields both the built-in pack and working + in-sandbox skill files. + +### Removed + +- **`docs/EKS-DEPLOYMENT.md`** — the hand-verified recipe for running the + whole single-box model (rootless Podman included) inside one privileged pod + on one large EKS node. Retired in favor of the first-class path above + rather than kept as a parallel track: an unmaintained privileged-pod recipe + beside a supported unprivileged one would imply support it never had (it + was explicitly "hand-verified, not CI-exercised"). Its durable content — + EFS/RWX storage, ECR/IRSA, NetworkPolicy-enforcement caveats, the backup + CronJob, day-2 mappings — was folded into `docs/DEPLOYMENT-KUBERNETES.md`. + ### Fixed - **Three own-rows authorization holes on the task surface.** The read path for diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0d16cd114..33787e588 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -28,7 +28,7 @@ removed.) ## Prerequisites -- **Go** — the version pinned in `go.mod` (currently 1.26.x). +- **Go** — the version pinned in `go.mod` (currently 1.27.x). - **Node.js** — the major in [`web/.nvmrc`](web/.nvmrc) (currently 24) — and npm, for the `web/` app. - **Podman** (rootless) for the execution sandbox — only needed to run the sandbox-backed tests/e2e locally; most unit tests self-skip when podman is @@ -114,10 +114,11 @@ Every pull request must be green before merge. CI runs: python / javascript-typescript / actions) fails on an unwaived finding in the **High band** — `security-severity >= 7.0`, or level `error`/`warning` for a rule that publishes no security-severity — with lower-severity findings - reported as advisory. A false positive is waived either by an in-source - `// codeql[rule-id]` comment or by an entry in - `.github/codeql-accepted-findings.json` **with a written reason**; both are - reviewable in the diff, and fixing the code is always preferred. See + reported as advisory. A false positive is waived by an entry in + `.github/codeql-accepted-findings.json` **with a written reason** — that is the + only waiver route that works here; an in-source `// codeql[rule-id]` comment + does not (measured on #1249). The register entry is reviewable in the diff, and + fixing the code is always preferred. See [`docs/SCANNING.md`](docs/SCANNING.md), [`docs/CODEQL.md`](docs/CODEQL.md) and [ADR-0048](docs/adr/0048-codeql-severity-gating.md). - **Dependency CVEs** — `govulncheck` for the Go module, and @@ -146,6 +147,30 @@ If golangci-lint flags something, either fix it or add a `//nolint` with a reason (the `nolintlint` linter requires the reason). The lint backlog is at zero — please keep it there. +### If CI is red and you don't recognise the failure + +Three lanes here depend on **live external data**, so they can go red on a diff +that did not cause it — including a one-line documentation change. This is by +design (a new advisory *should* redden an unchanged tree), but it means a red +check is not automatically yours: + +- **`govulncheck`** queries the Go vulnerability database on every run. +- **`npm audit`** runs over both npm trees at `--audit-level=low` and fails on + any severity. +- **Semgrep** fetches its rule packs from the registry. They cannot be pinned by + vendoring — the Semgrep Rules License forbids redistribution — so a + registry-side rule addition can turn CI red with no commit to blame. + +If the failure names a package, advisory or rule you did not touch, say so in the +PR rather than trying to fix it; a maintainer will confirm and handle it. + +Two other things that surprise first-time contributors, neither of them a problem +with your change: a first PR waits for a maintainer to approve the workflow run +before CI starts at all, and the full `main` suite is around a dozen jobs +including a ~1.3 GB sandbox image build, so it is thorough rather than fast. +`make lint && make test && make ci-web` locally will catch nearly everything +first. + ## Branch and pull-request conventions - Branch off the latest `main`. Use a short, descriptive prefix, e.g. @@ -156,18 +181,11 @@ zero — please keep it there. - Make sure the full local suite (Go + web + mocked Playwright) is green before you push. -## Commit messages and sign-off +## Commit messages - Write clear, imperative commit subjects ("Add X", not "Added X"). -- Sign off your commits with the Developer Certificate of Origin - () by adding a `Signed-off-by` trailer: - - ```bash - git commit -s -m "Your message" - ``` - - By signing off you certify that you wrote the patch (or otherwise have the - right to submit it) under the project's MIT license. +- Explain *why* in the body when the change is not self-evident. The diff + already says what changed. ## Reporting bugs and proposing features diff --git a/Makefile b/Makefile index 6686d860f..e1f2318c2 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: all build compile bins fleet-bench install test test-race test-cover lint lint-go lint-python lint-migrations fmt tidy clean help \ +.PHONY: all build compile bins fleet-bench install test test-race test-cover lint lint-go lint-python lint-migrations lint-actions fmt tidy clean help \ govulncheck ci-go ci-web ci-e2e-mocked ci-local # GOTOOLCHAIN=auto — the operator does NOT have to hand-install the pinned Go. @@ -122,7 +122,7 @@ test-cover: go test -coverprofile=coverage.out -covermode=atomic -p 1 -tags fleet_host_executor ./... @go tool cover -func=coverage.out | tail -1 -lint: lint-go lint-python lint-migrations +lint: lint-go lint-python lint-migrations lint-actions lint-go: golangci-lint run @@ -150,6 +150,33 @@ lint-python: lint-migrations: scripts/check-migrations.sh +# lint-actions: actionlint over .github/workflows/*.yml — the ~3.1k lines of +# workflow YAML that decide what every other gate on this list even runs. +# +# It is the one checker here whose subject is CI itself, and it covers ground +# no other lane does: expression syntax and type errors inside ${{ }}, unknown +# contexts, invalid `needs:`/`runs-on:`/cron, deprecated action syntax, and — +# via shellcheck — the bash inside every `run:` block. Semgrep's +# p/github-actions pack overlaps on ONE axis only (mutable action tags); it +# does not parse expressions and does not shellcheck run blocks. +# +# Skips LOUDLY when actionlint is absent, same contract as lint-python: CI +# enforces the gate regardless (ci.yml + dev-ci.yml `actions` job), so a local +# skip is a choice, not a surprise. +lint-actions: + @if command -v actionlint >/dev/null 2>&1; then \ + actionlint; \ + else \ + echo "actionlint not installed — SKIPPING the workflow lint (CI still enforces it)."; \ + echo " install: go install github.com/rhysd/actionlint/cmd/actionlint@v1.7.7"; \ + fi + @if command -v shellcheck >/dev/null 2>&1; then \ + git ls-files '*.sh' | xargs shellcheck -S warning; \ + else \ + echo "shellcheck not installed — SKIPPING the shell lint (CI still enforces it)."; \ + echo " install: dnf install ShellCheck # or: apt-get install shellcheck"; \ + fi + fmt: gofmt -w . @@ -219,9 +246,19 @@ ci-go: compile $(MAKE) test-race $(MAKE) govulncheck -# The Web CI job, verbatim, run from web/: npm ci → lint → vitest → build. +# The Web CI job, verbatim: both npm audits → the override canary → npm ci → +# lint → typecheck → vitest → build. It said "verbatim" while running four of +# those eight, which is the wrong half of a CI==local promise: the two +# `npm audit` CVE gates, scripts/check-npm-overrides.sh, and the explicit +# `npm run typecheck` were all missing, so a contributor could go green locally +# and red on the PR. The typecheck matters most of the four — `next build` +# type-checks too, but it runs LAST, so dropping the explicit gate is what turns +# a one-line type error into a multi-minute discovery. ci-web: - cd web && npm ci && npm run lint && npx vitest run && npm run build + cd web && npm audit --audit-level=low + cd scripts/rampart-service && npm audit --audit-level=low + scripts/check-npm-overrides.sh + cd web && npm ci && npm run lint && npm run typecheck && npx vitest run && npm run build # The mocked Playwright CI job, run from web/. Assumes browsers are installed # (`cd web && npx playwright install --with-deps chromium`); CI installs them in diff --git a/README.md b/README.md index 9c66d7bad..d550cb817 100644 --- a/README.md +++ b/README.md @@ -324,6 +324,14 @@ sudo bash /opt/fleet/src/scripts/bootstrap.sh --postgres=local --enable-service **→ Full deployment guide** — host sizing, the one-command web + Caddy/TLS stack, the env file, and every option: **[`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md)**. +**Kubernetes shop?** fleet also ships a first-class cluster path +([ADR-0049](docs/adr/0049-kubernetes-backend-split-control-plane.md)): a Helm +chart (`deploy/helm/fleet`) for the single-replica control plane, with agent +sandboxes running as **ephemeral pods** via +`FLEET_SANDBOX_BACKEND=kubernetes` — same loop, same security model, one +backend switch. See +**[`docs/DEPLOYMENT-KUBERNETES.md`](docs/DEPLOYMENT-KUBERNETES.md)**. + ## Operating fleet The operator lifecycle is **bootstrap → update → status**, one box. The server @@ -354,6 +362,7 @@ Deep references live in [`docs/`](docs/) so this README stays an orientation, no | Doc | What it covers | |---|---| | [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | Full deployment guide — host sizing, the one-command web + Caddy/TLS stack, options | +| [`docs/DEPLOYMENT-KUBERNETES.md`](docs/DEPLOYMENT-KUBERNETES.md) | Kubernetes as a first-class path — the Helm chart, the `kubernetes` sandbox backend (agent sandboxes as ephemeral pods), kind walkthrough + production checklist | | [`docs/OPERATORS.md`](docs/OPERATORS.md) | Operator runbook — the env file, the client-config checkout, every lifecycle verb | | [`docs/AGENT-RUNTIME.md`](docs/AGENT-RUNTIME.md) | Agent runtime mechanics — per-turn sandbox, ceilings, compaction, verifier, artifacts | | [`docs/SANDBOX-RUNTIMES.md`](docs/SANDBOX-RUNTIMES.md) | Sandbox OCI runtimes — `runc` / Kata / libkrun isolation tiers | diff --git a/SECURITY.md b/SECURITY.md index a3ee7176a..b24db581f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -202,10 +202,7 @@ compromised or fresh-and-unvetted release from reaching `main`: Dependabot waits a few days (3 for patch, 7 for minor, 14 for major) before proposing a freshly published release. This blunts fast typosquat / account-takeover attacks, where a malicious version is published and then yanked - once the ecosystem flags it. It matters most for **patch** bumps, which - `.github/workflows/auto-merge-dependabot.yml` auto-merges once the CI gate is - green: without a cooldown a minutes-old patch could be proposed and auto-merged - before any scrutiny. Cooldown applies to version updates only — Dependabot + once the ecosystem flags it. Cooldown applies to version updates only — Dependabot **security** updates are never delayed, so urgent CVE fixes still flow immediately. @@ -214,18 +211,17 @@ compromised or fresh-and-unvetted release from reaching `main`: only, so the one ecosystem whose "dependency" is *the CI definition itself* — a `github-actions` bump rewrites `.github/workflows/*` and therefore changes what CI executes — cannot be made to wait, and it is configured daily against - `dev`. Because `dev` additionally has no required status checks (see "Static - analysis" above), that combination is not something to auto-merge, so - `auto-merge-dependabot.yml` **excludes `github_actions` at any bump level** and - those PRs take a human. The workflow also carries an explicit - `branches: [main, dev]` filter, so it can never silently begin applying to some - other branch, and declares its write scopes on the job rather than the workflow. + `dev`, which additionally has no required status checks (see "Static analysis" + above). What contains that combination now is simply that **every** Dependabot + PR takes a human: automatic merging was removed from this repository, so no + dependency bump of any ecosystem or bump level reaches a branch without someone + looking at it. The cooldown reduces the window for a fast attack but is **not** a guarantee: a patient attacker who waits out the cooldown, or a compromise the ecosystem never flags, would still slip through. The committed `go.sum` + checksum DB, `govulncheck` and `npm audit` are the stronger, always-on controls; the cooldown -is defense-in-depth on top of the auto-merge path, and it does not cover +is defense-in-depth on top of human review, and it does not cover `github-actions` at all. ## CSRF protection (cookie-authenticated routes) diff --git a/cmd/fleet/main.go b/cmd/fleet/main.go index 94751cf39..c050766f4 100644 --- a/cmd/fleet/main.go +++ b/cmd/fleet/main.go @@ -321,21 +321,11 @@ func run() error { // MCP catalog. Empty in the generic bundle. cfg.HTTPTools = bundle.HTTPToolConfigs() - // The sandbox image is a per-client bundle artifact: resolve it from the - // bundle manifest (sandbox.image when set — the opt-in prebuilt/registry - // path — else sandbox.tag, the build-on-box default). An explicit - // FLEET_SANDBOX_IMAGE / CHAT_SANDBOX_IMAGE in the process env still wins - // (config.Load already populated cfg.SandboxImage from it). fleet does NOT - // build the image here — bootstrap / scripts/build-sandbox-image.sh does; - // this only feeds the resolved ref to the consuming sandbox pool. - if strings.TrimSpace(cfg.SandboxImage) == "" { - if ref := bundle.Sandbox().ResolvedImageRef(); ref != "" { - cfg.SandboxImage = ref - log.Printf("sandbox: image resolved from bundle = %s", ref) - } + // Resolve the sandbox image, OCI runtime, and backend from env + bundle + // (env wins). Fail-closed on an unrecognized backend value (#989). + if err := resolveSandboxInto(cfg, bundle); err != nil { + return err } - - resolveSandboxRuntimeInto(cfg, bundle) // Sandbox egress allowlist (#211): the bundle manifest supplies the default // allowed domains for allowlisted network mode (operator-authored deployment // config, like the runtime + image). FLEET_DEFAULT_NETWORK_MODE selects the @@ -858,13 +848,7 @@ func run() error { // skips containers carrying this process's own instance label in every // state — otherwise a warm container caught in "created" state would be // force-removed by its own process. See sandbox.PruneOrphanedContainers. - pruneCtx, pruneCancel := context.WithTimeout(context.Background(), 30*time.Second) - if n, err := sandbox.PruneOrphanedContainers(pruneCtx, "podman"); err != nil { - log.Printf("startup: prune orphaned sandbox containers: %v", err) - } else if n > 0 { - log.Printf("startup: pruned %d orphaned sandbox container(s) from a prior run", n) - } - pruneCancel() + pruneOrphanedSandboxes(mgr.SandboxPool()) // Share the process shutdown grace with the pool so its in-flight-task drain // uses the same budget as the chat-turn drain (#278). A non-positive grace @@ -1988,6 +1972,128 @@ func resolveSandboxRuntimeInto(cfg *config.Config, bundle *clientconfig.Bundle) cfg.SandboxRuntime = resolved } +// pruneOrphanedSandboxes reclaims sandboxes orphaned by a PRIOR crash, +// routed by the active backend: pods under the kubernetes backend (#989), +// podman containers otherwise. Best-effort — log and continue. Extracted from +// run() to keep it within the cyclomatic budget; see the call site's comment +// for why this must run only AFTER the manager (and thus the pool) exists. +func pruneOrphanedSandboxes(pool *sandbox.Pool) { + pruneCtx, pruneCancel := context.WithTimeout(context.Background(), 30*time.Second) + defer pruneCancel() + if kb := pool.KubernetesBackend(); kb != nil { + // Kubernetes backend (#989): orphans are pods, not podman containers. + if n, err := kb.PruneOrphanedPods(pruneCtx); err != nil { + log.Printf("startup: prune orphaned sandbox pods: %v", err) + } else if n > 0 { + //nolint:gosec // G706: n is an integer count of deleted pods — no attacker-controllable text reaches the log. + log.Printf("startup: pruned %d orphaned sandbox pod(s) from a prior run", n) + } + return + } + if n, err := sandbox.PruneOrphanedContainers(pruneCtx, "podman"); err != nil { + log.Printf("startup: prune orphaned sandbox containers: %v", err) + } else if n > 0 { + log.Printf("startup: pruned %d orphaned sandbox container(s) from a prior run", n) + } +} + +// resolveSandboxInto resolves the sandbox image, OCI runtime, and backend +// from env + bundle, in that order. Extracted from run() to keep it within +// the cyclomatic budget. +// +// The image is a per-client bundle artifact: the bundle manifest's +// sandbox.image (the opt-in prebuilt/registry path — else sandbox.tag, the +// build-on-box default) fills it when no explicit FLEET_SANDBOX_IMAGE / +// CHAT_SANDBOX_IMAGE env is set (config.Load already populated +// cfg.SandboxImage from those). fleet does NOT build the image here — +// bootstrap / scripts/build-sandbox-image.sh does; this only feeds the +// resolved ref to the consuming sandbox pool. +func resolveSandboxInto(cfg *config.Config, bundle *clientconfig.Bundle) error { + if strings.TrimSpace(cfg.SandboxImage) == "" { + if ref := bundle.Sandbox().ResolvedImageRef(); ref != "" { + cfg.SandboxImage = ref + log.Printf("sandbox: image resolved from bundle = %s", ref) + } + } + resolveSandboxRuntimeInto(cfg, bundle) + return resolveSandboxBackendInto(cfg, bundle) +} + +// resolveSandboxBackendInto resolves the sandbox backend (#989) into +// cfg.SandboxBackend with the SAME env-wins-else-bundle precedence as the +// image and runtime (sandbox.ResolveBackend is the one shared resolver, so +// boot and validate-config cannot drift), then fills each kubernetes setting +// from the bundle's sandbox.kubernetes block when the corresponding +// FLEET_SANDBOX_K8S_* env var is empty. An unrecognized backend value is a +// boot error, never a silent fallback to podman. +func resolveSandboxBackendInto(cfg *config.Config, bundle *clientconfig.Bundle) error { + sb := bundle.Sandbox() + resolved, err := sandbox.ResolveBackend(cfg.SandboxBackend, sb.Backend) + if err != nil { + return err + } + cfg.SandboxBackend = resolved + if resolved != sandbox.BackendKubernetes { + return nil + } + fill := func(dst *string, bundleVal string) { + if strings.TrimSpace(*dst) == "" { + *dst = strings.TrimSpace(bundleVal) + } + } + k := sb.Kubernetes + fill(&cfg.SandboxK8sNamespace, k.Namespace) + fill(&cfg.SandboxK8sWorkspaceClaim, k.WorkspaceClaim) + fill(&cfg.SandboxK8sServiceAccount, k.ServiceAccount) + fill(&cfg.SandboxK8sImagePullSecret, k.ImagePullSecret) + fill(&cfg.SandboxK8sRuntimeClass, k.RuntimeClass) + fill(&cfg.SandboxK8sSeccompProfile, k.SeccompProfile) + fill(&cfg.SandboxK8sKubeconfig, k.Kubeconfig) + fill(&cfg.SandboxK8sNetworkPolicy, k.NetworkPolicy) + // bundle_docs_in_image is a manifest bool and an env string; canonicalize + // to the env form so the pool build parses one source. Only a manifest + // TRUE needs carrying: false is already the empty-string default, and + // writing "false" here would make an unset env look explicitly disabled. + if strings.TrimSpace(cfg.SandboxK8sBundleDocsInImage) == "" && k.BundleDocsInImage { + cfg.SandboxK8sBundleDocsInImage = "true" + } + // The scheduling knobs are structured in the manifest; canonicalize them + // into the same string forms the env vars use so the pool build has ONE + // source to parse (env wins, like every other field). + if strings.TrimSpace(cfg.SandboxK8sNodeSelector) == "" && len(k.NodeSelector) > 0 { + keys := make([]string, 0, len(k.NodeSelector)) + for key := range k.NodeSelector { + keys = append(keys, key) + } + sort.Strings(keys) + pairs := make([]string, 0, len(keys)) + for _, key := range keys { + pairs = append(pairs, key+"="+k.NodeSelector[key]) + } + cfg.SandboxK8sNodeSelector = strings.Join(pairs, ",") + } + if strings.TrimSpace(cfg.SandboxK8sTolerations) == "" && len(k.Tolerations) > 0 { + raw, err := json.Marshal(k.Tolerations) + if err != nil { + return fmt.Errorf("encode sandbox.kubernetes.tolerations: %w", err) + } + cfg.SandboxK8sTolerations = string(raw) + } + //nolint:gosec // G706: backend + namespace are operator config (FLEET_SANDBOX_* / bundle manifest), quoted with %q — not request input. + log.Printf("sandbox: backend resolved to %q (control plane and runners split — pods in namespace %q)", + resolved, defaultString(cfg.SandboxK8sNamespace, "fleet-sandboxes")) + return nil +} + +// defaultString returns v, or fallback when v is empty — a log-formatting +// helper for resolveSandboxBackendInto. +func defaultString(v, fallback string) string { + if strings.TrimSpace(v) == "" { + return fallback + } + return v +} + // registerRuntimeMetrics wires the pull-at-scrape gauges (#176): in-flight turn // counts (interactive/scheduled), warm sandbox depth, host disk headroom, and // the Go runtime's goroutine/heap counters. Extracted from run() to keep it diff --git a/cmd/fleet/sandbox_backend_resolve_test.go b/cmd/fleet/sandbox_backend_resolve_test.go new file mode 100644 index 000000000..adc5ecdbd --- /dev/null +++ b/cmd/fleet/sandbox_backend_resolve_test.go @@ -0,0 +1,91 @@ +package main + +import ( + "os" + "path/filepath" + "testing" + + "github.com/ElcanoTek/fleet/internal/clientconfig" + "github.com/ElcanoTek/fleet/internal/config" + "github.com/ElcanoTek/fleet/internal/sandbox" +) + +// k8sBundle writes a minimal bundle whose sandbox block carries the given +// kubernetes: YAML, so the manifest→config fill can be exercised without a +// cluster. +func k8sBundle(t *testing.T, kubernetesYAML string) *clientconfig.Bundle { + t.Helper() + dir := t.TempDir() + // sandbox.image spares the fixture a Containerfile: the loader insists on + // one only for the build-on-box path. + manifest := "sandbox:\n image: registry.example/fleet-sandbox:test\n backend: kubernetes\n kubernetes:\n" + kubernetesYAML + if err := os.WriteFile(filepath.Join(dir, "manifest.yaml"), []byte(manifest), 0o600); err != nil { + t.Fatalf("write manifest: %v", err) + } + bundle, err := clientconfig.Load(dir) + if err != nil { + t.Fatalf("load bundle: %v", err) + } + return bundle +} + +// bundle_docs_in_image is a manifest bool and an env string. It must follow the +// same env-wins-else-bundle precedence as every other kubernetes setting — +// including the case that motivated the field: an operator turning it OFF from +// the chart for a bundle whose manifest turns it on. +func TestResolveSandboxBackendBundleDocsInImage(t *testing.T) { + tests := []struct { + name string + manifest string + env string + want string + }{ + {name: "manifest on, env unset", manifest: " bundle_docs_in_image: true\n", want: "true"}, + {name: "manifest on, env off wins", manifest: " bundle_docs_in_image: true\n", env: "false", want: "false"}, + {name: "manifest off, env unset stays empty", manifest: " bundle_docs_in_image: false\n", want: ""}, + {name: "manifest silent, env on wins", manifest: " workspace_claim: ws\n", env: "true", want: "true"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + bundle := k8sBundle(t, tc.manifest) + cfg := &config.Config{SandboxK8sBundleDocsInImage: tc.env} + if err := resolveSandboxBackendInto(cfg, bundle); err != nil { + t.Fatalf("resolveSandboxBackendInto: %v", err) + } + if cfg.SandboxK8sBundleDocsInImage != tc.want { + t.Errorf("SandboxK8sBundleDocsInImage = %q, want %q", cfg.SandboxK8sBundleDocsInImage, tc.want) + } + // Whatever the resolved string, it must parse — the pool build + // treats a parse error as a boot failure. + if _, err := sandbox.ParseK8sBundleDocsInImage(cfg.SandboxK8sBundleDocsInImage); err != nil { + t.Errorf("resolved value does not parse: %v", err) + } + }) + } +} + +// Under the podman backend the kubernetes block is not consulted at all: a +// bundle that declares baked-in docs for its cluster deployment must not have +// that leak into a single-box install, where the real bind mounts apply. +func TestResolveSandboxBackendPodmanIgnoresKubernetesBlock(t *testing.T) { + dir := t.TempDir() + manifest := "sandbox:\n image: registry.example/fleet-sandbox:test\n kubernetes:\n bundle_docs_in_image: true\n workspace_claim: ws\n" + if err := os.WriteFile(filepath.Join(dir, "manifest.yaml"), []byte(manifest), 0o600); err != nil { + t.Fatalf("write manifest: %v", err) + } + bundle, err := clientconfig.Load(dir) + if err != nil { + t.Fatalf("load bundle: %v", err) + } + cfg := &config.Config{} + if err := resolveSandboxBackendInto(cfg, bundle); err != nil { + t.Fatalf("resolveSandboxBackendInto: %v", err) + } + if cfg.SandboxBackend != sandbox.BackendPodman { + t.Fatalf("SandboxBackend = %q, want podman (the default)", cfg.SandboxBackend) + } + if cfg.SandboxK8sBundleDocsInImage != "" || cfg.SandboxK8sWorkspaceClaim != "" { + t.Errorf("kubernetes settings leaked into the podman path: docs=%q claim=%q", + cfg.SandboxK8sBundleDocsInImage, cfg.SandboxK8sWorkspaceClaim) + } +} diff --git a/cmd/fleet/validate_config.go b/cmd/fleet/validate_config.go index f3c545bad..b2dd41c2c 100644 --- a/cmd/fleet/validate_config.go +++ b/cmd/fleet/validate_config.go @@ -700,6 +700,20 @@ func checkSandbox(ctx context.Context, cfg *config.Config, bundle *clientconfig. return res } + // Kubernetes backend (#989): none of the podman checks apply — validate + // the backend selection and run the same fail-closed cluster preflight the + // boot path runs (apiserver reachability, RBAC, workspace claim, the + // sealed-egress NetworkPolicy, the RuntimeClass when one is configured). + backend, err := resolveValidateSandboxBackend(cfg, bundle) + if err != nil { + res.Status = statusFail + res.Detail = err.Error() + return res + } + if backend == sandbox.BackendKubernetes { + return checkKubernetesSandbox(ctx, res, cfg, bundle) + } + const podmanBin = "podman" if _, err := exec.LookPath(podmanBin); err != nil { res.Status = statusFail @@ -766,6 +780,119 @@ func checkSandbox(ctx context.Context, cfg *config.Config, bundle *clientconfig. return res } +// resolveValidateSandboxBackend resolves the sandbox backend the same way the +// boot path does (sandbox.ResolveBackend: env FLEET_SANDBOX_BACKEND wins, else +// the bundle manifest's sandbox.backend, else podman; unrecognized = error). +func resolveValidateSandboxBackend(cfg *config.Config, bundle *clientconfig.Bundle) (string, error) { + envBackend := "" + if cfg != nil { + envBackend = cfg.SandboxBackend + } + bundleBackend := "" + if bundle != nil { + bundleBackend = bundle.Sandbox().Backend + } + return sandbox.ResolveBackend(envBackend, bundleBackend) +} + +// checkKubernetesSandbox validates the kubernetes sandbox backend: the image +// ref resolves, the podman-only knobs are unset, and the boot preflight's +// cluster checks pass. Image PRESENCE is not checked — pulls happen on the +// sandbox nodes' kubelets, which this process cannot see; a bad ref fails +// fast at the first pod start instead. +func checkKubernetesSandbox(ctx context.Context, res checkResult, cfg *config.Config, bundle *clientconfig.Bundle) checkResult { + if rt := resolveSandboxRuntime(cfg, bundle); rt != "" { + res.Status = statusFail + res.Detail = fmt.Sprintf("FLEET_SANDBOX_RUNTIME=%q has no effect under the kubernetes backend — use FLEET_SANDBOX_K8S_RUNTIME_CLASS", rt) + return res + } + if cfg.DefaultNetworkMode == sandbox.NetworkModeAllowlisted { + res.Status = statusFail + res.Detail = "FLEET_DEFAULT_NETWORK_MODE=allowlisted is not supported under the kubernetes backend (the host egress proxy is unreachable from pods) — use lockdown or open" + return res + } + image := resolveSandboxImage(cfg, bundle) + if image == "" { + res.Status = statusFail + res.Detail = "no sandbox image resolved (set FLEET_SANDBOX_IMAGE or the bundle manifest's sandbox.image — kubernetes nodes cannot consume a build-on-box tag)" + return res + } + // Same env-wins-else-bundle resolution and fail-closed parse as the boot + // path: an env value is parsed from its string form; with no env value + // the bundle's structured knobs apply directly. + k8s := bundle.Sandbox().Kubernetes + nodeSelector := k8s.NodeSelector + if strings.TrimSpace(cfg.SandboxK8sNodeSelector) != "" { + parsed, err := sandbox.ParseK8sNodeSelector(cfg.SandboxK8sNodeSelector) + if err != nil { + res.Status = statusFail + res.Detail = "FLEET_SANDBOX_K8S_NODE_SELECTOR: " + err.Error() + return res + } + nodeSelector = parsed + } + var tolerations []sandbox.K8sToleration + for _, tol := range k8s.Tolerations { + tolerations = append(tolerations, sandbox.K8sToleration(tol)) + } + if strings.TrimSpace(cfg.SandboxK8sTolerations) != "" { + parsed, err := sandbox.ParseK8sTolerations(cfg.SandboxK8sTolerations) + if err != nil { + res.Status = statusFail + res.Detail = "FLEET_SANDBOX_K8S_TOLERATIONS: " + err.Error() + return res + } + tolerations = parsed + } + docsInImage := k8s.BundleDocsInImage + if strings.TrimSpace(cfg.SandboxK8sBundleDocsInImage) != "" { + parsed, err := sandbox.ParseK8sBundleDocsInImage(cfg.SandboxK8sBundleDocsInImage) + if err != nil { + res.Status = statusFail + res.Detail = "FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE: " + err.Error() + return res + } + docsInImage = parsed + } + fill := func(env, bundleVal string) string { + if strings.TrimSpace(env) != "" { + return strings.TrimSpace(env) + } + return strings.TrimSpace(bundleVal) + } + backend, err := sandbox.NewKubernetesBackend(sandbox.KubernetesConfig{ + Namespace: fill(cfg.SandboxK8sNamespace, k8s.Namespace), + WorkspaceClaim: fill(cfg.SandboxK8sWorkspaceClaim, k8s.WorkspaceClaim), + ServiceAccount: fill(cfg.SandboxK8sServiceAccount, k8s.ServiceAccount), + ImagePullSecret: fill(cfg.SandboxK8sImagePullSecret, k8s.ImagePullSecret), + RuntimeClassName: fill(cfg.SandboxK8sRuntimeClass, k8s.RuntimeClass), + SeccompLocalhostProfile: fill(cfg.SandboxK8sSeccompProfile, k8s.SeccompProfile), + KubeconfigPath: fill(cfg.SandboxK8sKubeconfig, k8s.Kubeconfig), + NetworkPolicyName: fill(cfg.SandboxK8sNetworkPolicy, k8s.NetworkPolicy), + NodeSelector: nodeSelector, + Tolerations: tolerations, + }) + if err != nil { + res.Status = statusFail + res.Detail = err.Error() + return res + } + if err := backend.Preflight(ctx); err != nil { + res.Status = statusFail + res.Detail = err.Error() + return res + } + res.Status = statusOK + // bundle-doc reads are the one behavior an operator cannot infer from the + // cluster state this check just proved, so it is reported either way. + docs := "bundle docs NOT in the sandbox image — in-sandbox protocol/skill reads will not resolve" + if docsInImage { + docs = "bundle docs declared present in the sandbox image (unverifiable here — a wrong declaration reads as not-found)" + } + res.Detail = fmt.Sprintf("kubernetes backend ok; image %q, sandbox namespace %q (image pullability is checked at first pod start); %s", image, backend.Namespace(), docs) + return res +} + // sandboxIsContainerBacked reports whether this binary will run agent tool calls // in a container (the only sandbox path that needs podman). True for a release // build (host executor not compiled in). When the host executor IS compiled in, diff --git a/config/default/manifest.yaml b/config/default/manifest.yaml index 79e275e56..05817f0f3 100644 --- a/config/default/manifest.yaml +++ b/config/default/manifest.yaml @@ -28,6 +28,32 @@ sandbox: # fail-closed boot preflight. # See docs/SANDBOX-RUNTIMES.md. FLEET_SANDBOX_RUNTIME overrides this. runtime: "${FLEET_SANDBOX_RUNTIME:-}" + # WHERE sandboxes run (#989 / ADR-0049): "podman" (default — co-located + # rootless containers) or "kubernetes" (each sandbox an ephemeral pod; the + # split control-plane/runner enterprise path — see + # docs/DEPLOYMENT-KUBERNETES.md). FLEET_SANDBOX_BACKEND overrides this, and + # each sandbox.kubernetes.* field below is overridden by the matching + # FLEET_SANDBOX_K8S_* env var. Uncomment to opt a bundle in: + # backend: kubernetes + # kubernetes: + # namespace: "" # default: the control plane's own namespace + # workspace_claim: fleet-workspace # required: shared RWX PVC + # service_account: fleet-sandbox + # image_pull_secret: "" + # runtime_class: "" # e.g. kata (preflighted fail-closed) + # seccomp_profile: "" # node-local Localhost profile; empty = RuntimeDefault + # kubeconfig: "" # out-of-cluster auth; empty = in-cluster + # network_policy: fleet-sandbox-deny-all + # node_selector: {} # pin sandbox pods to a dedicated runner pool + # tolerations: [] # [{key,operator,value,effect}] for a tainted pool + # # Declare that the SANDBOX IMAGE carries this bundle's protocols/, + # # personas/, system_prompts/ and skills/ at the same absolute paths the + # # control plane reads them from (a pod mounts only the workspace claim, + # # so that is the only way it can see them). Off = fleet drops those + # # fileop anchors and view_file on `protocols/…` is refused; on = the + # # file tools work again. Unverifiable by fleet — a wrong declaration + # # reads as not-found, never as a wider boundary. + # bundle_docs_in_image: false branding: app_name: "Fleet" diff --git a/deploy/helm/fleet/Chart.yaml b/deploy/helm/fleet/Chart.yaml new file mode 100644 index 000000000..305bbfc4f --- /dev/null +++ b/deploy/helm/fleet/Chart.yaml @@ -0,0 +1,17 @@ +apiVersion: v2 +name: fleet +description: >- + fleet control plane on Kubernetes with pluggable sandbox runners (issue + #989 / ADR-0049): a single-replica control-plane Deployment plus RBAC, + workspace storage, and NetworkPolicies for the kubernetes sandbox backend + (FLEET_SANDBOX_BACKEND=kubernetes). The single-box podman install remains + the default fleet deployment; this chart is the enterprise split path. +type: application +# Chart version tracks chart changes; appVersion is informational — operators +# build and pin their own fleet images (no public fleet image is published). +version: 0.1.0 +appVersion: "unreleased" +kubeVersion: ">=1.29.0-0" +home: https://github.com/ElcanoTek/fleet +sources: + - https://github.com/ElcanoTek/fleet diff --git a/deploy/helm/fleet/README.md b/deploy/helm/fleet/README.md new file mode 100644 index 000000000..b9b8654e6 --- /dev/null +++ b/deploy/helm/fleet/README.md @@ -0,0 +1,47 @@ +# fleet Helm chart + +> The enterprise deployment path (issue #989 / [ADR-0049](../../../docs/adr/0049-kubernetes-backend-split-control-plane.md)): +> the fleet control plane as a single-replica Deployment, with agent sandboxes +> running as ephemeral pods via `FLEET_SANDBOX_BACKEND=kubernetes`. The +> single-box podman install ([`docs/DEPLOYMENT.md`](../../../docs/DEPLOYMENT.md)) +> remains the default fleet deployment; use this chart when Kubernetes is your +> platform standard. + +Full walkthrough — a 15-minute kind path and the production checklist — lives +in [`docs/DEPLOYMENT-KUBERNETES.md`](../../../docs/DEPLOYMENT-KUBERNETES.md). + +## What it installs + +| Piece | Object | Notes | +| --- | --- | --- | +| Control plane | Deployment (1 replica, Recreate) | chat :8080 + orchestrator :8000; no replica knob — the scheduler is single-owner | +| Sandbox RBAC | Role/RoleBinding | exactly the verbs the boot preflight checks: pods create/get/list/delete, pods/exec create, PVC + NetworkPolicy get | +| Workspace | RWX PVC | mounted at the SAME path in the control plane and every sandbox pod | +| Sealed egress | NetworkPolicy `fleet-sandbox-deny-all` | selects pods labeled `fleet.elcanotek.com/egress=none`; the preflight requires it to exist | +| Postgres | optional StatefulSet | evaluation only — production points at a managed database | +| Web / Ingress | optional | you build the web image from `web/` | + +## Minimum install + +```sh +helm install fleet deploy/helm/fleet \ + --namespace fleet --create-namespace \ + --set image.repository=REGISTRY/fleet --set image.tag=v1 \ + --set sandbox.image=REGISTRY/fleet-sandbox:v1 \ + --set postgres.enabled=true \ + --set config.existingSecret=fleet-secrets # OPENROUTER_API_KEY etc. +``` + +You build both images yourself — fleet publishes none. See the deployment +guide for the two Containerfiles. + +## Honest scope + +- **NetworkPolicy enforcement is the CNI's job.** The chart ships the deny-all + object and fleet's preflight verifies it exists; on a CNI without + NetworkPolicy support it seals nothing. Verify enforcement (the guide's + checklist shows how). +- **One control-plane replica.** No HPA, no `replicas`. Scale work with more + sandbox capacity, the control plane with bigger requests. +- The chart is linted in CI but not exercised against a live cluster there; + the kind walkthrough in the guide is the verified path. diff --git a/deploy/helm/fleet/templates/NOTES.txt b/deploy/helm/fleet/templates/NOTES.txt new file mode 100644 index 000000000..2997000da --- /dev/null +++ b/deploy/helm/fleet/templates/NOTES.txt @@ -0,0 +1,22 @@ +fleet {{ .Chart.Version }} installed as {{ .Release.Name }} in {{ .Release.Namespace }}. + +Control plane : one replica (by design — single-owner scheduler; do not scale it) +Sandbox pods : namespace {{ include "fleet.sandboxNamespace" . }}, image {{ .Values.sandbox.image }} +Workspace : PVC {{ .Values.workspace.claimName }} mounted at {{ .Values.workspaceRoot }} (control plane + every sandbox pod) + +Check boot (the kubernetes sandbox preflight runs before serving; a missing +RBAC grant, workspace claim, or NetworkPolicy aborts start with the reason): + + kubectl -n {{ .Release.Namespace }} logs deploy/{{ include "fleet.fullname" . }} | grep -E 'sandbox|preflight' + +Reach the chat API: + + kubectl -n {{ .Release.Namespace }} port-forward svc/{{ include "fleet.fullname" . }} 8080:8080 + curl -s localhost:8080/healthz + +Verify a turn executes in a sandbox pod: + + kubectl -n {{ include "fleet.sandboxNamespace" . }} get pods -l app.kubernetes.io/name=fleet-sandbox -w + +Full guide (15-minute kind path + production checklist): + docs/DEPLOYMENT-KUBERNETES.md in the fleet repository. diff --git a/deploy/helm/fleet/templates/_helpers.tpl b/deploy/helm/fleet/templates/_helpers.tpl new file mode 100644 index 000000000..0d0ef2425 --- /dev/null +++ b/deploy/helm/fleet/templates/_helpers.tpl @@ -0,0 +1,42 @@ +{{/* Chart name + fullname, standard helpers. */}} +{{- define "fleet.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{- define "fleet.fullname" -}} +{{- if .Values.fullnameOverride -}} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- printf "%s" .Release.Name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +{{- end -}} + +{{- define "fleet.labels" -}} +app.kubernetes.io/name: {{ include "fleet.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version }} +{{- end -}} + +{{- define "fleet.selectorLabels" -}} +app.kubernetes.io/name: {{ include "fleet.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "fleet.serviceAccountName" -}} +{{- if .Values.serviceAccount.create -}} +{{- default (include "fleet.fullname" .) .Values.serviceAccount.name -}} +{{- else -}} +{{- required "serviceAccount.name is required when serviceAccount.create=false" .Values.serviceAccount.name -}} +{{- end -}} +{{- end -}} + +{{/* Namespace sandbox pods run in: explicit value or the release namespace. */}} +{{- define "fleet.sandboxNamespace" -}} +{{- default .Release.Namespace .Values.sandbox.kubernetes.namespace -}} +{{- end -}} + +{{/* In-cluster Postgres endpoints (postgres.enabled=true only). */}} +{{- define "fleet.postgresHost" -}} +{{- printf "%s-postgres" (include "fleet.fullname" .) -}} +{{- end -}} diff --git a/deploy/helm/fleet/templates/deployment.yaml b/deploy/helm/fleet/templates/deployment.yaml new file mode 100644 index 000000000..20a18e507 --- /dev/null +++ b/deploy/helm/fleet/templates/deployment.yaml @@ -0,0 +1,178 @@ +# The fleet control plane. ONE replica, forever: scheduler leases and the +# worker semaphore are single-owner (ADR-0004 / ADR-0049) — two replicas +# against one database pair is a correctness bug. There is deliberately no +# replicas knob; horizontal scale of work = more sandbox pods. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "fleet.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + replicas: 1 + strategy: + # Recreate, not RollingUpdate: a rolling update would run two control + # planes side by side for the overlap window — the exact multi-writer + # state the single-owner invariant forbids. + type: Recreate + selector: + matchLabels: + {{- include "fleet.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "fleet.selectorLabels" . | nindent 8 }} + {{- with .Values.podLabels }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + serviceAccountName: {{ include "fleet.serviceAccountName" . }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: fleet + image: "{{ required "image.repository is required (you build the fleet image yourself — see docs/DEPLOYMENT-KUBERNETES.md)" .Values.image.repository }}:{{ required "image.tag is required" .Values.image.tag }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + args: ["serve"] + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + ports: + - name: chat + containerPort: 8080 + - name: orchestrator + containerPort: 8000 + env: + # Listeners bind all pod interfaces; the Service/Ingress is the + # exposure boundary (the single-box default is loopback + Caddy). + - name: FLEET_SERVER_ADDR + value: "0.0.0.0:8080" + - name: FLEET_ORCHESTRATOR_ADDR + value: "0.0.0.0:8000" + - name: FLEET_DATA_DIR + value: {{ .Values.dataDir | quote }} + - name: FLEET_WORKSPACE_ROOT + value: {{ .Values.workspaceRoot | quote }} + # ── sandbox backend (#989) ── + - name: FLEET_SANDBOX_BACKEND + value: {{ .Values.sandbox.backend | quote }} + - name: FLEET_SANDBOX_IMAGE + value: {{ required "sandbox.image is required (a registry ref sandbox nodes can pull)" .Values.sandbox.image | quote }} + - name: FLEET_SANDBOX_K8S_NAMESPACE + value: {{ include "fleet.sandboxNamespace" . | quote }} + - name: FLEET_SANDBOX_K8S_WORKSPACE_CLAIM + value: {{ .Values.workspace.claimName | quote }} + - name: FLEET_SANDBOX_K8S_NETWORK_POLICY + value: {{ .Values.networkPolicies.denyAll.name | quote }} + {{- if .Values.sandbox.kubernetes.podServiceAccount.name }} + - name: FLEET_SANDBOX_K8S_SERVICE_ACCOUNT + value: {{ .Values.sandbox.kubernetes.podServiceAccount.name | quote }} + {{- end }} + {{- with .Values.sandbox.kubernetes.imagePullSecret }} + - name: FLEET_SANDBOX_K8S_IMAGE_PULL_SECRET + value: {{ . | quote }} + {{- end }} + {{- with .Values.sandbox.kubernetes.runtimeClass }} + - name: FLEET_SANDBOX_K8S_RUNTIME_CLASS + value: {{ . | quote }} + {{- end }} + {{- with .Values.sandbox.kubernetes.seccompProfile }} + - name: FLEET_SANDBOX_K8S_SECCOMP_PROFILE + value: {{ . | quote }} + {{- end }} + {{- if .Values.sandbox.kubernetes.bundleDocsInImage }} + - name: FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE + value: "true" + {{- end }} + {{- with .Values.sandbox.kubernetes.nodeSelector }} + - name: FLEET_SANDBOX_K8S_NODE_SELECTOR + value: {{ $pairs := list }}{{- range $k, $v := . }}{{- $pairs = append $pairs (printf "%s=%s" $k $v) }}{{- end }}{{ join "," $pairs | quote }} + {{- end }} + {{- with .Values.sandbox.kubernetes.tolerations }} + - name: FLEET_SANDBOX_K8S_TOLERATIONS + value: {{ toJson . | quote }} + {{- end }} + - name: FLEET_SANDBOX_MEMORY + value: {{ .Values.sandbox.memory | quote }} + - name: FLEET_SANDBOX_CPUS + value: {{ .Values.sandbox.cpus | quote }} + - name: FLEET_SANDBOX_DISK_GB + value: {{ .Values.sandbox.diskGB | quote }} + {{- if gt (int .Values.sandbox.warmSize) 0 }} + - name: FLEET_SANDBOX_WARM_SIZE + value: {{ .Values.sandbox.warmSize | quote }} + {{- end }} + {{- if .Values.postgres.enabled }} + - name: FLEET_CHAT_DATABASE_URL + valueFrom: + secretKeyRef: + name: {{ include "fleet.fullname" . }}-postgres + key: chat-url + - name: FLEET_SCHED_DATABASE_URL + valueFrom: + secretKeyRef: + name: {{ include "fleet.fullname" . }}-postgres + key: sched-url + {{- end }} + {{- range $k, $v := .Values.config.env }} + - name: {{ $k }} + value: {{ $v | quote }} + {{- end }} + {{- with .Values.config.existingSecret }} + envFrom: + - secretRef: + name: {{ . }} + {{- end }} + volumeMounts: + - name: workspace + mountPath: {{ .Values.workspaceRoot }} + - name: data + mountPath: {{ .Values.dataDir }} + readinessProbe: + httpGet: + path: /healthz + port: chat + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: + path: /healthz + port: chat + initialDelaySeconds: 30 + periodSeconds: 30 + resources: + {{- toYaml .Values.resources | nindent 12 }} + volumes: + - name: workspace + persistentVolumeClaim: + claimName: {{ .Values.workspace.claimName }} + - name: data + persistentVolumeClaim: + claimName: {{ .Values.data.claimName }} + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} diff --git a/deploy/helm/fleet/templates/ingress.yaml b/deploy/helm/fleet/templates/ingress.yaml new file mode 100644 index 000000000..5b539f3df --- /dev/null +++ b/deploy/helm/fleet/templates/ingress.yaml @@ -0,0 +1,38 @@ +{{- if .Values.ingress.enabled }} +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: {{ include "fleet.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} + {{- with .Values.ingress.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + {{- with .Values.ingress.className }} + ingressClassName: {{ . }} + {{- end }} + {{- with .Values.ingress.tls }} + tls: + {{- toYaml . | nindent 4 }} + {{- end }} + rules: + - host: {{ required "ingress.host is required when ingress.enabled" .Values.ingress.host }} + http: + paths: + - path: / + pathType: Prefix + backend: + service: + {{- if .Values.web.enabled }} + name: {{ include "fleet.fullname" . }}-web + port: + name: http + {{- else }} + name: {{ include "fleet.fullname" . }} + port: + name: chat + {{- end }} +{{- end }} diff --git a/deploy/helm/fleet/templates/networkpolicy.yaml b/deploy/helm/fleet/templates/networkpolicy.yaml new file mode 100644 index 000000000..8f79eac08 --- /dev/null +++ b/deploy/helm/fleet/templates/networkpolicy.yaml @@ -0,0 +1,57 @@ +{{- if .Values.networkPolicies.denyAll.create }} +# The sealed-egress policy for sandbox pods labeled +# fleet.elcanotek.com/egress=none (lockdown turns, sealed scheduled runs). +# fleet's boot preflight REQUIRES this object to exist before it will start +# with the kubernetes backend. HONEST LIMIT: a NetworkPolicy object is +# enforced by the cluster CNI — on a CNI without NetworkPolicy support this +# object exists but seals nothing. Verify your CNI enforces it (the +# production checklist in docs/DEPLOYMENT-KUBERNETES.md shows how). +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ .Values.networkPolicies.denyAll.name }} + namespace: {{ include "fleet.sandboxNamespace" . }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: fleet-sandbox + fleet.elcanotek.com/egress: "none" + policyTypes: ["Ingress", "Egress"] + # No ingress/egress rules: everything denied, both directions. +{{- end }} +{{- if .Values.networkPolicies.openEgress.create }} +--- +# Egress shaping for OPEN sandbox pods: allow DNS + everything EXCEPT the +# blocked CIDRs (your cluster/node ranges), so `pip install` works but the +# sandbox cannot reach in-cluster Services. Ingress stays denied. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "fleet.fullname" . }}-sandbox-open-egress + namespace: {{ include "fleet.sandboxNamespace" . }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: fleet-sandbox + fleet.elcanotek.com/egress: "open" + policyTypes: ["Ingress", "Egress"] + egress: + - to: + - ipBlock: + cidr: 0.0.0.0/0 + {{- with .Values.networkPolicies.openEgress.blockedCIDRs }} + except: + {{- toYaml . | nindent 14 }} + {{- end }} + # DNS (kube-dns lives inside the cluster CIDRs excluded above). + - to: [] + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 +{{- end }} diff --git a/deploy/helm/fleet/templates/postgres.yaml b/deploy/helm/fleet/templates/postgres.yaml new file mode 100644 index 000000000..078711457 --- /dev/null +++ b/deploy/helm/fleet/templates/postgres.yaml @@ -0,0 +1,119 @@ +{{- if .Values.postgres.enabled }} +# Optional in-cluster PostgreSQL for evaluation and small installs: one +# replica, one PVC, both fleet databases (chat + sched) in one server. +# Production should use a managed database and config.existingSecret URLs. +{{- /* The password never comes from values: reuse the existing Secret's + (operator-pre-created or a prior install's) or generate a random one. */ -}} +{{- $password := "" }} +{{- $existing := lookup "v1" "Secret" .Release.Namespace (printf "%s-postgres" (include "fleet.fullname" .)) }} +{{- if $existing }} +{{- $password = index $existing.data "password" | b64dec }} +{{- else }} +{{- $password = randAlphaNum 32 }} +{{- end }} +apiVersion: v1 +kind: Secret +metadata: + name: {{ include "fleet.fullname" . }}-postgres + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +type: Opaque +stringData: + password: {{ $password | quote }} + chat-url: "postgres://fleet:{{ $password }}@{{ include "fleet.postgresHost" . }}:5432/chat?sslmode=disable" + sched-url: "postgres://fleet:{{ $password }}@{{ include "fleet.postgresHost" . }}:5432/sched?sslmode=disable" +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "fleet.fullname" . }}-postgres-init + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +data: + init.sql: | + CREATE DATABASE chat; + CREATE DATABASE sched; +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: {{ include "fleet.fullname" . }}-postgres + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + serviceName: {{ include "fleet.postgresHost" . }} + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: {{ include "fleet.name" . }}-postgres + app.kubernetes.io/instance: {{ .Release.Name }} + template: + metadata: + labels: + app.kubernetes.io/name: {{ include "fleet.name" . }}-postgres + app.kubernetes.io/instance: {{ .Release.Name }} + spec: + securityContext: + fsGroup: 999 + seccompProfile: + type: RuntimeDefault + containers: + - name: postgres + image: {{ .Values.postgres.image }} + env: + - name: POSTGRES_USER + value: fleet + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: {{ include "fleet.fullname" . }}-postgres + key: password + - name: PGDATA + value: /var/lib/postgresql/data/pgdata + ports: + - name: postgres + containerPort: 5432 + volumeMounts: + - name: pgdata + mountPath: /var/lib/postgresql/data + - name: init + mountPath: /docker-entrypoint-initdb.d + readinessProbe: + exec: + command: ["pg_isready", "-U", "fleet"] + periodSeconds: 5 + volumes: + - name: init + configMap: + name: {{ include "fleet.fullname" . }}-postgres-init + volumeClaimTemplates: + - metadata: + name: pgdata + spec: + accessModes: ["ReadWriteOnce"] + {{- with .Values.postgres.storageClassName }} + storageClassName: {{ . }} + {{- end }} + resources: + requests: + storage: {{ .Values.postgres.size }} +--- +apiVersion: v1 +kind: Service +metadata: + name: {{ include "fleet.postgresHost" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + clusterIP: None + selector: + app.kubernetes.io/name: {{ include "fleet.name" . }}-postgres + app.kubernetes.io/instance: {{ .Release.Name }} + ports: + - name: postgres + port: 5432 +{{- end }} diff --git a/deploy/helm/fleet/templates/rbac.yaml b/deploy/helm/fleet/templates/rbac.yaml new file mode 100644 index 000000000..e9d018dc8 --- /dev/null +++ b/deploy/helm/fleet/templates/rbac.yaml @@ -0,0 +1,70 @@ +# RBAC for the kubernetes sandbox backend: exactly the verbs the backend's +# boot preflight checks (internal/sandbox/k8s_preflight.go) — pod lifecycle + +# exec in the sandbox namespace, plus read access to the two objects the +# preflight verifies (the workspace claim and the deny-all NetworkPolicy). +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "fleet.fullname" . }}-runner + namespace: {{ include "fleet.sandboxNamespace" . }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["create", "get", "list", "delete"] + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] + - apiGroups: [""] + resources: ["persistentvolumeclaims"] + verbs: ["get"] + - apiGroups: ["networking.k8s.io"] + resources: ["networkpolicies"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "fleet.fullname" . }}-runner + namespace: {{ include "fleet.sandboxNamespace" . }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "fleet.fullname" . }}-runner +subjects: + - kind: ServiceAccount + name: {{ include "fleet.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- if .Values.sandbox.kubernetes.runtimeClass }} +--- +# RuntimeClass objects are cluster-scoped; the preflight GETs the configured +# one to fail closed when it does not exist (ADR-0010 posture). +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "fleet.fullname" . }}-runtimeclass-reader + labels: + {{- include "fleet.labels" . | nindent 4 }} +rules: + - apiGroups: ["node.k8s.io"] + resources: ["runtimeclasses"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "fleet.fullname" . }}-runtimeclass-reader + labels: + {{- include "fleet.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "fleet.fullname" . }}-runtimeclass-reader +subjects: + - kind: ServiceAccount + name: {{ include "fleet.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/deploy/helm/fleet/templates/service.yaml b/deploy/helm/fleet/templates/service.yaml new file mode 100644 index 000000000..7f4b87d17 --- /dev/null +++ b/deploy/helm/fleet/templates/service.yaml @@ -0,0 +1,18 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ include "fleet.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + type: {{ .Values.service.type }} + selector: + {{- include "fleet.selectorLabels" . | nindent 4 }} + ports: + - name: chat + port: 8080 + targetPort: chat + - name: orchestrator + port: 8000 + targetPort: orchestrator diff --git a/deploy/helm/fleet/templates/serviceaccount.yaml b/deploy/helm/fleet/templates/serviceaccount.yaml new file mode 100644 index 000000000..a0ded8ebc --- /dev/null +++ b/deploy/helm/fleet/templates/serviceaccount.yaml @@ -0,0 +1,23 @@ +{{- if .Values.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "fleet.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +{{- end }} +{{- if .Values.sandbox.kubernetes.podServiceAccount.create }} +--- +# Identity stamped on sandbox pods so admission policies can key on it. +# No RBAC is bound to it and no token is ever mounted +# (automountServiceAccountToken=false on every sandbox pod). +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.sandbox.kubernetes.podServiceAccount.name }} + namespace: {{ include "fleet.sandboxNamespace" . }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +automountServiceAccountToken: false +{{- end }} diff --git a/deploy/helm/fleet/templates/storage.yaml b/deploy/helm/fleet/templates/storage.yaml new file mode 100644 index 000000000..8c055d2a1 --- /dev/null +++ b/deploy/helm/fleet/templates/storage.yaml @@ -0,0 +1,50 @@ +{{- if .Values.workspace.create }} +{{- if ne (include "fleet.sandboxNamespace" .) .Release.Namespace }} +{{- fail "workspace.create=true only supports sandbox pods in the release namespace: a PVC cannot be mounted across namespaces. For a separate sandbox namespace, set workspace.create=false and provision a claim with the same name in EACH namespace, both bound to the same ReadWriteMany export (static NFS/EFS PVs)." }} +{{- end }} +# The shared workspace claim: mounted by the control plane AND every sandbox +# pod at {{ .Values.workspaceRoot }}. ReadWriteMany in production (EFS, NFS, +# CephFS); ReadWriteOnce only works on a single-node cluster (kind). +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: {{ .Values.workspace.claimName }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} + annotations: + # Keep the workspace when the release is deleted/pruned — it holds user + # data (conversation files, task outputs). + helm.sh/resource-policy: keep +spec: + accessModes: + {{- toYaml .Values.workspace.accessModes | nindent 4 }} + {{- with .Values.workspace.storageClassName }} + storageClassName: {{ . }} + {{- end }} + resources: + requests: + storage: {{ .Values.workspace.size }} +{{- end }} +{{- if .Values.data.create }} +--- +# The fleet data dir (journals, logs, caches). Control plane only. +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: {{ .Values.data.claimName }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} + annotations: + helm.sh/resource-policy: keep +spec: + accessModes: + {{- toYaml .Values.data.accessModes | nindent 4 }} + {{- with .Values.data.storageClassName }} + storageClassName: {{ . }} + {{- end }} + resources: + requests: + storage: {{ .Values.data.size }} +{{- end }} diff --git a/deploy/helm/fleet/templates/web.yaml b/deploy/helm/fleet/templates/web.yaml new file mode 100644 index 000000000..7d9389900 --- /dev/null +++ b/deploy/helm/fleet/templates/web.yaml @@ -0,0 +1,67 @@ +{{- if .Values.web.enabled }} +# Optional web tier (the Next.js app you build from web/). It talks to the +# control-plane Service; expose it via the Ingress below or your own. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "fleet.fullname" . }}-web + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: {{ include "fleet.name" . }}-web + app.kubernetes.io/instance: {{ .Release.Name }} + template: + metadata: + labels: + app.kubernetes.io/name: {{ include "fleet.name" . }}-web + app.kubernetes.io/instance: {{ .Release.Name }} + spec: + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + containers: + - name: web + image: {{ required "web.image is required when web.enabled" .Values.web.image }} + ports: + - name: http + containerPort: {{ .Values.web.port }} + env: + - name: CHAT_SERVER_URL + value: "http://{{ include "fleet.fullname" . }}:8080" + - name: ORCHESTRATOR_URL + value: "http://{{ include "fleet.fullname" . }}:8000" + {{- range $k, $v := .Values.web.env }} + - name: {{ $k }} + value: {{ $v | quote }} + {{- end }} + {{- with .Values.config.existingSecret }} + envFrom: + - secretRef: + name: {{ . }} + {{- end }} +--- +apiVersion: v1 +kind: Service +metadata: + name: {{ include "fleet.fullname" . }}-web + namespace: {{ .Release.Namespace }} + labels: + {{- include "fleet.labels" . | nindent 4 }} +spec: + selector: + app.kubernetes.io/name: {{ include "fleet.name" . }}-web + app.kubernetes.io/instance: {{ .Release.Name }} + ports: + - name: http + port: {{ .Values.web.port }} + targetPort: http +{{- end }} diff --git a/deploy/helm/fleet/values.yaml b/deploy/helm/fleet/values.yaml new file mode 100644 index 000000000..5b2a1035e --- /dev/null +++ b/deploy/helm/fleet/values.yaml @@ -0,0 +1,203 @@ +# Default values for the fleet chart (issue #989 / ADR-0049). +# +# READ FIRST — the two invariants this chart encodes: +# +# 1. ONE control-plane replica, forever. fleet's scheduler leases and worker +# semaphore are single-owner; two replicas against one database pair is a +# correctness bug, not a capacity increase. The Deployment pins +# replicas: 1 with strategy Recreate and exposes no replica knob. +# Horizontal scale of WORK = more sandbox pods (bigger node pool), never +# more fleet processes. +# 2. Sandboxes never hold credentials. Sandbox pods are created by the +# control plane with automountServiceAccountToken=false, no env, no +# secret mounts — only the shared workspace claim. MCP credentials stay +# in the control-plane process (ADR-0003). +# +# You build the images yourself — fleet does not publish images: +# control plane: a container image with the `fleet` binary + the client +# bundle (see docs/DEPLOYMENT-KUBERNETES.md §images) +# sandbox: scripts/build-sandbox-image.sh, pushed to your registry + +image: + # REQUIRED: your fleet control-plane image (binary + bundle baked in or + # mounted). Example: 123456789.dkr.ecr.us-east-1.amazonaws.com/fleet:v42 + repository: "" + tag: "" + pullPolicy: IfNotPresent + +imagePullSecrets: [] +# - name: regcred + +serviceAccount: + # Control-plane ServiceAccount (holds the RBAC grant for sandbox pods). + create: true + name: "" + +sandbox: + # REQUIRED: the sandbox image sandbox pods run + # (FLEET_SANDBOX_IMAGE). Must live in a registry the sandbox nodes can + # pull from — a localhost/ build-on-box tag cannot work here. + image: "" + # Backend is what this chart exists for; podman-in-a-pod is NOT a supported + # chart topology (the podman backend's supported home is a VM — + # docs/DEPLOYMENT.md). + backend: kubernetes + kubernetes: + # Namespace for sandbox pods. Empty = the release namespace (the default, + # because the shared workspace PVC cannot be mounted across namespaces — + # use a separate namespace only with storage that supports two PVs on one + # export, e.g. static NFS/EFS PVs). + namespace: "" + # ServiceAccount stamped on sandbox pods (identity for admission + # policies; no token is ever mounted). Created when create=true. + podServiceAccount: + create: true + name: fleet-sandbox + # Pull secret for the sandbox image, when the registry is private. + imagePullSecret: "" + # RuntimeClass for hypervisor-isolated sandboxes (e.g. kata). Preflighted + # fail-closed at boot when set. + runtimeClass: "" + # Node-local seccomp profile applied as a Localhost profile (path + # relative to the kubelet seccomp root). Empty = RuntimeDefault. + seccompProfile: "" + # Declare that your SANDBOX IMAGE carries the client bundle's doc dirs + # (protocols/, personas/, system_prompts/, skills/) at the SAME absolute + # paths the control plane reads them from — i.e. you built the sandbox + # image with the bundle baked in. A pod mounts only the workspace claim, + # so without this fleet drops those roots and view_file on `protocols/…` + # is refused; with it the file tools work again. fleet cannot verify an + # image's contents: a wrong declaration shows up as a not-found read. + # Bundles that inherit fleet's built-in skills pack cannot serve + # in-sandbox skill reads either way (the merged tree lives on the + # control-plane data PVC) — set skills_builtin: false in the bundle to + # make skills/ bake-able. See docs/DEPLOYMENT-KUBERNETES.md. + bundleDocsInImage: false + # Pin sandbox pods to a DEDICATED runner node pool: label the pool and + # select it here; taint it and add the matching toleration so nothing + # else lands on it. This is fleet's horizontal scaling story — more + # runner capacity is a bigger pool, never more fleet replicas. + nodeSelector: {} + # fleet.elcanotek.com/pool: sandboxes + tolerations: [] + # - key: fleet.elcanotek.com/sandbox + # operator: Exists + # effect: NoSchedule + # Per-sandbox ceilings, podman-flag format (converted by fleet): + memory: "512m" + cpus: "1.0" + diskGB: 5 + # Warm pool depth (FLEET_SANDBOX_WARM_SIZE; 0 = derive from + # FLEET_MAX_CONCURRENT_AGENTS). + warmSize: 0 + +networkPolicies: + # Ship the deny-all egress policy for sandbox pods labeled + # fleet.elcanotek.com/egress=none. The boot preflight REQUIRES this object + # to exist; disable only if you manage an equivalent policy yourself (and + # then set its name below so the preflight finds it). + denyAll: + create: true + name: fleet-sandbox-deny-all + # Optional egress shaping for OPEN sandbox pods (egress=open): allow DNS + + # anything except the listed CIDRs (typically your cluster + node CIDRs), + # so an open sandbox can reach PyPI but not your Services. Off by default + # because the CIDRs are cluster-specific. + openEgress: + create: false + # CIDRs an open sandbox must NOT reach (cluster Pod/Service/node ranges). + blockedCIDRs: [] + # - 10.0.0.0/8 + # - 172.16.0.0/12 + # - 192.168.0.0/16 + +workspace: + # The shared workspace claim: mounted by the control plane AND every + # sandbox pod at the same absolute path (workspaceRoot below). + create: true + claimName: fleet-workspace + size: 20Gi + # MUST be a ReadWriteMany-capable class in production (EFS, NFS, CephFS). + # On a single-node cluster (kind), a ReadWriteOnce class works because + # every pod lands on the one node — fine for evaluation, not production. + storageClassName: "" + accessModes: + - ReadWriteMany + +data: + # The fleet data dir (FLEET_DATA_DIR): journals, logs, caches. Control + # plane only — sandbox pods never mount it. + create: true + claimName: fleet-data + size: 10Gi + storageClassName: "" + accessModes: + - ReadWriteOnce + +# Absolute path the workspace claim is mounted at in BOTH the control plane +# and every sandbox pod. Same-path mounting is what keeps absolute workspace +# paths meaningful across the control plane, MCP brokers, and sandboxes. +workspaceRoot: /var/lib/fleet/workspace +dataDir: /var/lib/fleet + +config: + # Name of an EXISTING Secret carrying the sensitive env (recommended): + # OPENROUTER_API_KEY, FLEET_CHAT_DATABASE_URL, FLEET_SCHED_DATABASE_URL, + # FLEET_SERVER_TOKEN, and any SMTP/webhook secrets. + # When postgres.enabled=true the DB URLs are wired automatically and may be + # omitted from the secret. + existingSecret: "" + # Extra non-secret env for the control plane, verbatim. + env: {} + # FLEET_MAX_CONCURRENT_AGENTS: "8" + # FLEET_DEFAULT_NETWORK_MODE: "lockdown" + # FLEET_TIMEZONE: "UTC" + +postgres: + # Optional in-cluster PostgreSQL (single replica, one PVC) for evaluation + # and small installs. Production should point config at a managed database + # (RDS, Cloud SQL) via the existingSecret DB URLs instead. + # + # The database password is NOT a values entry: the chart auto-generates a + # random one into the -postgres Secret on first install and reuses + # it on upgrades (helm lookup). To pick your own, pre-create that Secret + # with a `password` key before installing. Rotating it is your job — this + # is the evaluation path. + enabled: false + image: docker.io/library/postgres:17 + size: 10Gi + storageClassName: "" + +web: + # Optional Next.js web tier (you build web/ into an image yourself). + enabled: false + image: "" + port: 3000 + env: {} + +service: + # ClusterIP service exposing chat (8080) and orchestrator (8000). + type: ClusterIP + +ingress: + enabled: false + className: "" + annotations: {} + host: "" + tls: [] + +resources: + # Control-plane resources. Size for your FLEET_MAX_CONCURRENT_AGENTS — the + # sandboxes themselves run as separate pods with their own limits. + requests: + cpu: "1" + memory: 2Gi + limits: + memory: 4Gi + +nodeSelector: {} +tolerations: [] +affinity: {} + +podAnnotations: {} +podLabels: {} diff --git a/docs/CODEQL.md b/docs/CODEQL.md index a085d0667..e298dbc23 100644 --- a/docs/CODEQL.md +++ b/docs/CODEQL.md @@ -508,10 +508,14 @@ Per-**file** is the whole point of preferring it to a `query-filters` exclude. A `exclude: {id: go/request-forgery}` switches a security-severity 9.1 query off for the entire repository; a register entry waives it in `internal/tools/web_fetch.go` and `internal/mcpoauth/discovery.go` and leaves the -query live everywhere else, including elsewhere in those same packages. An -in-source `// codeql[rule-id]` comment is the second waiver route — CodeQL emits -it as a `suppressions` array on the result; the comment must sit on its own line -and covers the line immediately below it. +query live everywhere else, including elsewhere in those same packages. + +The register is also the **only** waiver route. An in-source `// codeql[rule-id]` +comment is the mechanism CodeQL documents, and it does **not** work with this +pipeline — measured on PR #1249: three forms were tried (the `packs:` input, `packs:` with the additive `+` prefix, and an inline `config:` combining security-extended with codeql/go-queries' `AlertSuppression.ql`) and in every case the uploaded SARIF carried no `suppressions` on the annotated result, the gate kept classifying the waiver from the register, +and the Security-tab alert stayed open. The analyze action's interpret step is +not configurable enough to change that. A deliberately-waived alert is closed in +the Security tab by a one-time human dismissal, which persists across analyses. Of the 55 findings run 527 surfaced, **four were reachable and were fixed in code**: an unsanitized `task.Prompt` in the task-create log (its update-path twin @@ -567,8 +571,8 @@ exits 0 and reports green"**, and that is no longer true. It was true of the The step reads the run's **own SARIF** and never consults the code-scanning API, which has one consequence worth stating plainly: **dismissing an alert in the Security tab does not turn this check green.** The honest routes are a code -change, an in-source `// codeql[rule-id]` comment, or a register entry with a -reason. +change or a register entry with a reason — an in-source `// codeql[rule-id]` +comment is not one of them here (see above). The second row remains available and nothing depends on it. fleet is a **public** repository, so code scanning merge protection is free (on private repos it needs @@ -645,7 +649,7 @@ What is fixed is the format: BLOCKING — High band (security-severity >= 7.0), not waived (): none -ACCEPTED — High band, waived in codeql-accepted-findings.json or in-source (): +ACCEPTED — High band, waived in codeql-accepted-findings.json (): [error] sec-sev=9.1 go/request-forgery internal/tools/web_fetch.go: [error] sec-sev=7.5 go/clear-text-logging cmd/fleet/main.go: ... diff --git a/docs/DEPLOYMENT-KUBERNETES.md b/docs/DEPLOYMENT-KUBERNETES.md new file mode 100644 index 000000000..82cd53f2a --- /dev/null +++ b/docs/DEPLOYMENT-KUBERNETES.md @@ -0,0 +1,400 @@ +# Deploying fleet on Kubernetes + +> The first-class Kubernetes path (issue #989 / +> [ADR-0049](adr/0049-kubernetes-backend-split-control-plane.md)): the fleet +> control plane as a single-replica Deployment, with agent sandboxes running as +> **ephemeral pods** via the pluggable sandbox backend +> (`FLEET_SANDBOX_BACKEND=kubernetes`). The single-box podman install +> ([`DEPLOYMENT.md`](DEPLOYMENT.md)) remains the default and an equally +> supported path; come here when Kubernetes is your platform standard. + +## The model + +Same agent loop, same security model, one backend switch: + +| Piece | Where it runs | +| --- | --- | +| fleet control plane (chat + orchestrator + MCP broker) | one Deployment replica — **never more**; the scheduler leases and worker semaphore are single-owner | +| Agent sandboxes (bash, run_python, file ops) | **ephemeral pods**, one per turn / sealed run / persistent-REPL conversation, created and exec'd by the control plane over the apiserver | +| MCP credentials | the control-plane process, always (ADR-0003) — sandbox pods carry no env, no secrets, no service-account token | +| Workspace | one **ReadWriteMany** PVC mounted at the *same absolute path* in the control plane and every sandbox pod | + +``` + browser ──TLS──▶ Ingress ──▶ web (optional) ──▶ fleet Service + │ chat :8080 + │ orchestrator :8000 + ┌────────────────────────────────────────────────────┴──────────┐ + │ fleet control plane pod (1 replica, Recreate) │ + │ agent loop · scheduler · MCP broker (credentials stay here) │ + └───────┬──────────────────────────────┬────────────────────────┘ + │ pods/exec (WebSocket) │ Postgres (managed, or the + ▼ ▼ chart's eval StatefulSet) + fleet-sandbox- pods (ephemeral) chat + sched databases + read-only rootfs · non-root · no caps + no ServiceAccount token · egress by label + │ + └── workspace PVC (RWX) — same path as the control plane +``` + +The backend is selected by `FLEET_SANDBOX_BACKEND` (`podman`, the default, or +`kubernetes`), overriding the bundle manifest's `sandbox.backend` — exactly the +precedence `FLEET_SANDBOX_RUNTIME` / `sandbox.runtime` uses +([SANDBOX-RUNTIMES.md](SANDBOX-RUNTIMES.md)). An unrecognized value refuses to +boot; there is no silent fallback. + +**Fail-closed preflight.** With `kubernetes` selected, fleet refuses to start +unless, at boot: the apiserver is reachable with valid credentials; RBAC grants +`create/get/list/delete pods` and `create pods/exec` in the sandbox namespace; +the workspace claim exists; the sealed-egress NetworkPolicy object exists; and +the RuntimeClass exists when one is configured. `fleet validate-config` runs +the same checks. + +## Build the two images + +fleet publishes no images — you build both and push them to a registry your +nodes can pull from (ECR, GAR, ACR, GHCR, …). A `localhost/` build-on-box tag +cannot work outside a single-node kind cluster. + +**Sandbox image** — the bundle artifact the sandboxes run, unchanged from the +single-box install: + +```sh +scripts/build-sandbox-image.sh +podman tag localhost/fleet-sandbox:latest REGISTRY/fleet-sandbox:v1 +podman push REGISTRY/fleet-sandbox:v1 +``` + +(Or let CI publish it — see `.github/workflows/publish-sandbox-image.yml`.) + +**Control-plane image** — the `fleet` binary plus your client bundle. A +reproducible multi-stage Containerfile (build it from the repo root; the +builder stage's Go minor is pinned to `go.mod` by +`scripts/check_versions_test.go`, so a stale copy of this stage fails CI): + +```dockerfile +# ── build stage ── +FROM docker.io/library/golang:1.27 AS build +WORKDIR /src +COPY go.mod go.sum ./ +RUN go mod download +COPY . . +RUN CGO_ENABLED=0 go build -ldflags "-X github.com/ElcanoTek/fleet/internal/version.version=$(cat VERSION)" -o /out/fleet ./cmd/fleet + +# ── runtime stage ── +FROM registry.fedoraproject.org/fedora-minimal:latest +RUN microdnf install -y git ca-certificates tzdata && microdnf clean all +COPY --from=build /out/fleet /usr/local/bin/fleet +# Bake the client bundle in (the generic one here; substitute your own). +COPY config/default /opt/fleet/client +ENV FLEET_CLIENT_CONFIG_DIR=/opt/fleet/client +RUN mkdir -p /var/lib/fleet && chown 1000:1000 /var/lib/fleet +USER 1000 +ENTRYPOINT ["/usr/local/bin/fleet"] +``` + +An out-of-repo client bundle can be baked into your image the same way, or +mounted from a ConfigMap/volume at `FLEET_CLIENT_CONFIG_DIR` — either satisfies +the engine/bundle split (ADR-0006). + +**Web image** (optional) — build `web/` with its own `next build` stage and set +`web.image` in the chart; the chart wires `CHAT_SERVER_URL` / +`ORCHESTRATOR_URL` at the fleet Service automatically. + +## 15-minute path (kind) + +Prereqs: `kind`, `kubectl`, `helm`, `podman` or `docker` to build images, and +an OpenRouter API key. + +```sh +# 1. A cluster. +kind create cluster --name fleet + +# 2. Build both images and load them into kind. Save the control-plane +# Containerfile from "Build the two images" above as Containerfile.fleet. +scripts/build-sandbox-image.sh +podman save localhost/fleet-sandbox:latest -o /tmp/sandbox.tar +kind load image-archive /tmp/sandbox.tar --name fleet +podman build -t localhost/fleet:dev -f Containerfile.fleet . +podman save localhost/fleet:dev -o /tmp/fleet.tar +kind load image-archive /tmp/fleet.tar --name fleet + +# 3. Secrets (the API key; DB URLs come from the chart's eval Postgres). +kubectl create namespace fleet +kubectl -n fleet create secret generic fleet-secrets \ + --from-literal=OPENROUTER_API_KEY=sk-or-... + +# 4. Install. kind is single-node, so the default (ReadWriteOnce) storage +# class works for the shared workspace — every pod lands on the one node. +helm install fleet deploy/helm/fleet --namespace fleet \ + --set image.repository=localhost/fleet --set image.tag=dev \ + --set image.pullPolicy=Never \ + --set sandbox.image=localhost/fleet-sandbox:latest \ + --set 'workspace.accessModes={ReadWriteOnce}' \ + --set postgres.enabled=true \ + --set config.existingSecret=fleet-secrets + +# 5. Watch boot — the sandbox preflight logs its verdict before serving. +kubectl -n fleet logs deploy/fleet -f | grep -E 'sandbox|preflight' + +# 6. Talk to it, and watch a sandbox pod appear for the turn. +kubectl -n fleet port-forward svc/fleet 8080:8080 & +kubectl -n fleet get pods -l app.kubernetes.io/name=fleet-sandbox -w +``` + +A chat turn that runs bash or python creates a `fleet-sandbox-` pod, +execs into it, and deletes it when the turn ends. Cancelling a turn deletes +the pod immediately (zero grace) — the same poison-and-retire containment the +podman backend guarantees (#796). + +## Production checklist + +1. **Storage: the workspace claim must be ReadWriteMany** (EFS, NFS, CephFS, + Azure Files). A ReadWriteOnce class only works when every pod shares one + node (kind). Verify: + `kubectl -n fleet get pvc fleet-workspace -o jsonpath='{.spec.accessModes}'`. +2. **Database: managed Postgres.** Put `FLEET_CHAT_DATABASE_URL` and + `FLEET_SCHED_DATABASE_URL` in your `config.existingSecret` and leave + `postgres.enabled=false`. The chart's Postgres is an evaluation + convenience: one replica, one PVC, no backups. +3. **NetworkPolicy enforcement is your CNI's job.** fleet verifies the + deny-all policy *object* exists; only a CNI that implements NetworkPolicy + (Calico, Cilium, the EKS VPC CNI's network-policy agent, GKE Dataplane V2, + Azure CNI with policy) makes it real. Verify from a sealed sandbox: + ```sh + kubectl -n fleet run seal-test --restart=Never --rm -it \ + --labels=app.kubernetes.io/name=fleet-sandbox,fleet.elcanotek.com/egress=none \ + --image=busybox -- wget -T 5 -q -O- https://example.com && echo "NOT SEALED" + ``` + A CNI that enforces the policy times that request out. +4. **Shape open-sandbox egress.** Non-lockdown sandboxes are labeled + `egress=open` and unrestricted by default (they need PyPI etc.). Set + `networkPolicies.openEgress.create=true` with your cluster/node CIDRs in + `blockedCIDRs` so an open sandbox can reach the internet but not your + Services. +5. **Registry, not build-on-box.** Both images in a registry the nodes pull + from; set `sandbox.kubernetes.imagePullSecret` for private registries (on + EKS, node-role ECR access covers sandbox pulls without a secret). +6. **Hypervisor isolation** (optional): install Kata Containers on the sandbox + nodes, create a `kata` RuntimeClass, set + `sandbox.kubernetes.runtimeClass=kata`. Preflighted fail-closed, mirroring + `FLEET_SANDBOX_RUNTIME` (ADR-0010). Note `FLEET_SANDBOX_RUNTIME` itself is + a podman knob and is **refused** under this backend. +7. **One replica.** Do not add an HPA or `replicas: 2` for the control plane. + Scale work by raising `FLEET_MAX_CONCURRENT_AGENTS` and giving the sandbox + namespace more node capacity; scale the control plane vertically + (`resources` in values). Size for peak: the control plane runs the agent + loop + brokers; the sandboxes' cost lives in their own pods, so warm-pool + pods hold their requests while parked — size `FLEET_SANDBOX_WARM_SIZE` + accordingly. +8. **Give runners their own node pool.** Label (and usually taint) a dedicated + pool, then set `sandbox.kubernetes.nodeSelector` + `.tolerations` in the + chart — sandbox pods pin there and autoscale the pool, while the control + plane stays on your general nodes. This is the horizontal scaling story: + more runner capacity is a bigger pool, never a second fleet. +9. **Run `fleet validate-config`** (`kubectl -n fleet exec deploy/fleet -- + fleet validate-config`) after any config change: it runs the same + fail-closed preflight boot does, plus everything else the verb checks. + +## Day-2 operations + +The systemd timers and host scripts (`bootstrap.sh`, `fleet update`, +`scripts/doctor.sh`, `fleet timers install`) are single-box tooling and do not +apply here ([TIMERS.md](TIMERS.md)). Their cluster equivalents: + +| Single-box | Kubernetes | +| --- | --- | +| `fleet update` | build + push a new control-plane image, `helm upgrade` (strategy Recreate = a brief restart; in-flight turns drain per `FLEET_SHUTDOWN_GRACE_SECONDS`) | +| `fleet-backup.timer` | a CronJob running `fleet backup --db=all --prune` — or skip it entirely by using managed-database backups (RDS/Cloud SQL snapshots), the recommended posture | +| `fleet-maintenance.timer` | a CronJob running `fleet cleanup` daily | +| journald | `kubectl logs` / your log stack; set `FLEET_LOG_FILE` only if you also mount somewhere rotatable | +| Grafana node dashboards | scrape the control plane's `/metrics` (orchestrator port). NOTE it is **admin-API-key gated** (`X-API-Key`) — cost/token data must not be public — and stock Prometheus cannot send custom headers, so use a scraper that can (Grafana Alloy, vmagent) or a small header-injecting sidecar. Sandbox pods are ordinary pods your cluster metrics already see | + +Minimal backup CronJob (only needed when you run the eval Postgres or want +`fleet backup`'s application-level dumps next to managed snapshots): + +```yaml +apiVersion: batch/v1 +kind: CronJob +metadata: {name: fleet-backup, namespace: fleet} +spec: + schedule: "0 2 * * *" + jobTemplate: + spec: + template: + spec: + restartPolicy: Never + containers: + - name: backup + image: REGISTRY/fleet:v1 # the control-plane image + args: ["backup", "--db=all", "--prune"] + envFrom: [{secretRef: {name: fleet-secrets}}] + volumeMounts: [{name: data, mountPath: /var/lib/fleet}] + volumes: + - name: data + persistentVolumeClaim: {claimName: fleet-data} +``` + +## Bundle docs inside a sandbox pod + +A bundle's `protocols/`, `personas/`, `system_prompts/` and `skills/` are how a +protocol-driven deployment works at all: the system prompt lists them by +relative path and the agent reads them on demand. Under podman fleet +bind-mounts each read-only at its own absolute path and symlinks them into the +per-conversation workspace, so `protocols/foo.yaml` resolves for `view_file`, +`bash` and `run_python` alike. + +A sandbox pod mounts only the workspace claim. There is no host filesystem to +bind from, so by default fleet drops those roots — and because the fileop path +anchor only trusts roots that are actually mounted, `view_file +protocols/foo.yaml` is *refused* (`fileop root is not inside a sandbox bind +mount`) rather than attempted. The workspace symlinks still point at the +bundle's absolute paths, so `bash`/`run_python` reads fail too, as not-found. + +The fix is the sandbox image. Build it with the bundle's doc dirs baked in at +the **same absolute paths** the control plane uses (`FLEET_CLIENT_CONFIG_DIR`), +then declare it: + +```yaml +sandbox: + kubernetes: + bundleDocsInImage: true # chart values → FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE +``` + +```dockerfile +# derived sandbox image; same paths as the control plane's bundle +FROM REGISTRY/fleet-sandbox:v1 +USER root +COPY protocols/ personas/ system_prompts/ skills/ /opt/fleet/client/... +RUN chown -R 0:0 /opt/fleet/client && chmod -R a-w,a+rX /opt/fleet/client +USER 1000 +``` + +With the declaration, the anchors for those roots stay valid, so the file tools +read them out of the pod's image layer and the symlinked relative paths work +for bash and python. Four things to be honest about: + +- **It is a declaration, not a probe.** fleet cannot inspect an image's + contents. It also cannot widen anything: the flag only re-admits *read-only* + anchors for roots the operator already configured, and the read still runs + inside the sandbox. A wrong declaration surfaces as a not-found read. +- **Only the bundle's own doc dirs are covered.** Other entries in the mount + list (the uploads root) live in control-plane state no image can contain; + they stay dropped, with a log line each. +- **The merged skills tree is never covered** — see the honest-scope list. +- **The baked copy is a snapshot.** Build and roll the control-plane and + sandbox images from the same bundle commit, or the agent reads one release's + protocols while the control plane runs another's. Nothing enforces this. + +Boot logs which roots survived and which were dropped, and why. `kubectl logs +deploy/fleet | grep 'bundle_docs_in_image\|supporting-doc'` is the fastest way +to see what a running deployment decided. + +## Provider notes + +- **EKS**: EFS (via the EFS CSI driver) is the standard RWX workspace class; + ECR for both images (node-role pull, no secret needed); enable the VPC CNI + network-policy agent or run Calico/Cilium so the deny-all policy is + enforced; ALB via the AWS Load Balancer Controller for `ingress`. RDS for + Postgres. Kata needs a bare-metal (`*.metal`) node group for `/dev/kvm`. +- **GKE**: Filestore CSI for RWX; Dataplane V2 enforces NetworkPolicy natively; + Artifact Registry with Workload Identity; Cloud SQL. +- **AKS**: Azure Files (NFS) for RWX; enable Azure Network Policy or Cilium; + ACR with the kubelet identity; Azure Database for PostgreSQL. +- **Bare metal / on-prem**: any NFS/CephFS class for RWX; Calico or Cilium for + policy; your own registry. + +## Configuration reference + +Every knob can come from env (the chart sets these) or the bundle manifest's +`sandbox:` block (env wins, field by field): + +| Env | Manifest | Meaning | +| --- | --- | --- | +| `FLEET_SANDBOX_BACKEND` | `sandbox.backend` | `podman` (default) or `kubernetes` | +| `FLEET_SANDBOX_K8S_NAMESPACE` | `sandbox.kubernetes.namespace` | sandbox pod namespace (default: the control plane's own, else `fleet-sandboxes`) | +| `FLEET_SANDBOX_K8S_WORKSPACE_CLAIM` | `…workspace_claim` | **required** — the shared RWX PVC name | +| `FLEET_SANDBOX_K8S_SERVICE_ACCOUNT` | `…service_account` | identity stamped on sandbox pods (no token is mounted) | +| `FLEET_SANDBOX_K8S_IMAGE_PULL_SECRET` | `…image_pull_secret` | pull secret for the sandbox image | +| `FLEET_SANDBOX_K8S_RUNTIME_CLASS` | `…runtime_class` | hypervisor isolation (kata); preflighted | +| `FLEET_SANDBOX_K8S_SECCOMP_PROFILE` | `…seccomp_profile` | node-local Localhost seccomp profile; empty = RuntimeDefault | +| `FLEET_SANDBOX_K8S_KUBECONFIG` | `…kubeconfig` | out-of-cluster auth (token / client-cert kubeconfigs only); empty = in-cluster | +| `FLEET_SANDBOX_K8S_NETWORK_POLICY` | `…network_policy` | deny-all policy name the preflight requires (default `fleet-sandbox-deny-all`) | +| `FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE` | `…bundle_docs_in_image` | the sandbox image carries the bundle's doc dirs at the same absolute paths — keeps their fileop read anchors valid in a pod ([above](#bundle-docs-inside-a-sandbox-pod)); a non-boolean refuses to boot | +| `FLEET_SANDBOX_K8S_NODE_SELECTOR` | `…node_selector` | pin sandbox pods to a dedicated runner pool — env form `"pool=sandboxes,arch=amd64"`, manifest form a map; a malformed value refuses to boot | +| `FLEET_SANDBOX_K8S_TOLERATIONS` | `…tolerations` | tolerations for a tainted runner pool — env form a JSON array of `{key,operator,value,effect}`, manifest form a YAML list | + +The shared sandbox knobs apply to both backends: `FLEET_SANDBOX_IMAGE`, +`FLEET_SANDBOX_MEMORY` / `_CPUS` (converted to pod resource limits), +`FLEET_SANDBOX_DISK_GB` (the pod's ephemeral-storage limit), +`FLEET_SANDBOX_WARM_SIZE` / `_WARM_TTL` (the warm pool holds pre-started +pods), and the python REPL knobs. + +## Troubleshooting + +- **Boot fails with "kubernetes sandbox preflight"** — the message names the + exact missing piece (RBAC verb, claim, NetworkPolicy, RuntimeClass). The + chart's `fleet-runner` Role carries every needed verb; if you wrote your own + RBAC, diff it against `deploy/helm/fleet/templates/rbac.yaml`. +- **First turn fails with `ErrImagePull` / `ImagePullBackOff`** — the sandbox + image ref isn't pullable *from the nodes* (fleet fails the pod fast with the + kubelet's reason instead of burning the start timeout). Check the ref and + `sandbox.kubernetes.imagePullSecret`. +- **`sandbox pod … not ready before start timeout`** — usually scheduling + (no node fits the sandbox requests) or a slow first pull; `kubectl describe + pod fleet-sandbox-…` shows which. +- **Workspace files owned by the wrong uid** — the claim's storage class must + honor `fsGroup` (1000) or be provisioned world-writable at the root; both + the control plane and sandbox pods run uid/gid 1000. +- **A sealed turn can still reach the network** — your CNI is not enforcing + NetworkPolicy (checklist item 3). The policy *object* existing is not + enforcement. +- **Turn cancelled but you want proof nothing survived** — cancellation + deletes the pod with zero grace; `kubectl get pods -l + app.kubernetes.io/name=fleet-sandbox` should not show the pod after the + cancel completes. A pod that lingers past a crash is reclaimed by the + boot-time orphan sweep on the next control-plane start. + +## Honest scope — what the kubernetes backend does differently + +Recorded here so nobody discovers them in production: + +- **Egress sealing is delegated.** Podman's `--network=none` is a kernel + namespace with no interface; the k8s equivalent is a label + (`fleet.elcanotek.com/egress=none`) matched by a deny-all NetworkPolicy. + fleet verifies the object exists — it cannot verify the CNI enforces it. +- **`FLEET_DEFAULT_NETWORK_MODE=allowlisted` is refused** at boot: the + host-side egress proxy (ADR-0012) is unreachable from pods. Use `lockdown` + or `open` + NetworkPolicy shaping. +- **No per-pod pids limit.** `FLEET_SANDBOX_PIDS` has no Pod-spec equivalent; + runaway process counts are bounded by pod memory/CPU limits and node + `podPidsLimit` if you configure the kubelet. +- **No per-sandbox resource telemetry (#263).** `podman stats` has no + in-process counterpart here; task resource summaries are absent. Use your + cluster's metrics stack on the `fleet-sandbox` pods. +- **The bundled seccomp profile does not apply.** Pods run `RuntimeDefault`, + or a profile you install on the nodes yourself via + `FLEET_SANDBOX_K8S_SECCOMP_PROFILE`. Setting the podman + `FLEET_SANDBOX_SECCOMP_PROFILE` under this backend refuses to boot rather + than being silently ignored. +- **Supporting-doc bind mounts don't apply** — the podman backend bind-mounts + persona/protocol dirs same-path into containers; a pod only mounts the + workspace claim. Bake them into the sandbox image and declare it + (`sandbox.kubernetes.bundle_docs_in_image`) to get the reads back; see + [Bundle docs inside a sandbox pod](#bundle-docs-inside-a-sandbox-pod). + Undeclared, in-sandbox reads of those paths do not resolve at all. +- **A bundle inheriting fleet's built-in skills pack cannot serve in-sandbox + skill reads**, declaration or not: the merged tree is materialized under the + control plane's data dir, so no sandbox image can carry it. `skills_builtin: + false` in the bundle manifest makes `skills/` the bundle's own (bake-able) + dir at the cost of the built-in pack. There is no setting that gives you + both. +- **Disk quota is per-pod ephemeral storage**, which caps the writable layer + and scratch emptyDirs — a *stronger* cap than podman's per-file ulimit — but + the workspace claim is still unbounded by it, same as the bind mount is + under podman: many files still add up. +- **Warm-pool pods hold cluster resources while parked.** Requests equal + limits; size `FLEET_SANDBOX_WARM_SIZE` accordingly. +- **kind e2e is a documented walkthrough, not a CI job.** CI lints and + template-renders the chart (`helm` job) and unit-tests the backend against a + fake apiserver (including exec streaming and the poison path); it does not + stand up a cluster. diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 177c20b37..28e49102b 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -15,13 +15,17 @@ proxies, server-side over loopback, to the two Go backends the single process boots (chat on `127.0.0.1:8080`, orchestrator on `127.0.0.1:8000`). Caddy fronts the web app with TLS; the backends stay loopback-only. -> **Your platform standard is Kubernetes?** fleet's shipped target is this -> single-VM/systemd model ([ADR-0004](adr/0004-single-box-vm-native-deployment.md)), -> and no chart or manifest lives in the tree. For an operator recipe that keeps -> the one-process/one-node model intact inside EKS — one pod on one big node, -> Podman running *inside* it, nothing split across worker nodes — see -> [`docs/EKS-DEPLOYMENT.md`](EKS-DEPLOYMENT.md). It is hand-verified, not -> CI-exercised. +> **Your platform standard is Kubernetes?** fleet's default install remains +> this single-VM/systemd model +> ([ADR-0004](adr/0004-single-box-vm-native-deployment.md)), but Kubernetes is +> now a first-class path +> ([ADR-0049](adr/0049-kubernetes-backend-split-control-plane.md)): a Helm +> chart (`deploy/helm/fleet`) runs the control plane as a single-replica +> Deployment, and `FLEET_SANDBOX_BACKEND=kubernetes` runs every agent sandbox +> as an ephemeral pod — no Podman on the node, no privileged pod. See +> [`docs/DEPLOYMENT-KUBERNETES.md`](DEPLOYMENT-KUBERNETES.md): a 15-minute +> kind path, the production checklist, provider notes (EKS/GKE/AKS), and +> day-2 operations. > **Single-host by design.** Scheduled-task crash recovery uses single-owner > database leases and the worker-pool concurrency cap is a per-process semaphore — diff --git a/docs/EKS-DEPLOYMENT.md b/docs/EKS-DEPLOYMENT.md deleted file mode 100644 index 5b2de9405..000000000 --- a/docs/EKS-DEPLOYMENT.md +++ /dev/null @@ -1,1334 +0,0 @@ -# Deploying fleet on Amazon EKS (one pod, one big node) - -> Operator recipe for organizations whose platform standard is Kubernetes. It -> keeps fleet's single-process, single-node model intact — one pod on one large -> node, scaled vertically — and does **not** try to spread the Podman sandboxes -> across worker nodes. For the supported single-host install see -> [`docs/DEPLOYMENT.md`](DEPLOYMENT.md). - -## Read this first (scope, honesty, and what is not shipped) - -- **fleet's shipped deployment target is a single VM under systemd** - ([ADR-0004](adr/0004-single-box-vm-native-deployment.md)). That ADR stands. - There is no Helm chart, no operator, and no k8s manifest in this repo, and - **CI does not exercise this path** — the CI matrix builds and tests the - systemd/single-host model. Everything below is a hand-verified recipe you own - and must validate on your own cluster. -- **No fleet container image ships either.** `deploy/` contains systemd units, not - images. You build two images yourself (§3): the fleet runtime image (Go binary - **+ Podman inside it**) and the Next.js web image. The sandbox image stays what - it already is — a per-client *bundle* artifact. -- **One pod. One replica. Forever.** Scheduled-task crash recovery uses - single-owner database leases and the concurrency cap is a per-process - semaphore. Two fleet pods against one pair of databases is a **correctness - bug**, not a capacity increase. No `Deployment` with rolling updates, no HPA, - no `replicas: 2`. -- **The sandbox stays local to the process.** Every agent tool call's data plane - runs in a rootless-Podman container that `agentcore` starts and `podman exec`s - into on the same host as the run loop - ([ADR-0002](adr/0002-mandatory-rootless-podman-sandbox.md)); the remote worker - registry was deliberately removed - ([ADR-0011](adr/0011-remove-worker-node-registry.md)). There is no seam that - dispatches a sandbox to another node, so "put the runners on their own node - group" is not a configuration — it would be a rewrite. This guide runs Podman - **inside the fleet pod**, which is why the pod needs the privileges in §2. -- **What you gain** by doing this at all: your existing ECR/IRSA/ALB/Secrets - Manager/observability plumbing, one node group to patch, and node-failure - rescheduling. **What you give up** versus the systemd path: `bootstrap.sh`, - `fleet update`, and `scripts/doctor.sh` all assume a systemd host — you replace - them with image rebuilds and `kubectl exec` (§9). - -## The objections a Kubernetes-native reviewer will raise - -Answer these before the design review, not during it. Each links to the section -that implements it. - -| "This isn't Kubernetes-native because…" | Answer | -|---|---| -| "…there's no Helm chart / it's not GitOps" | Package the §7 manifests as Kustomize or a thin Helm chart and sync with Argo CD or Flux — [§7 GitOps](#packaging-these-manifests-for-gitops). Two Argo-specific gotchas are called out there. | -| "…a privileged pod will never pass admission" | It won't under `restricted`/`baseline` Pod Security Standards. You need a labelled namespace and a scoped policy exception — [§7 admission control](#namespace-admission-control-and-identity). If your org forbids privileged pods outright, [§2](#if-your-policy-forbids-privileged-pods) is the unprivileged variant and its costs. | -| "…one replica isn't highly available" | Correct, and it cannot be: single-owner task leases + a per-process semaphore. HA here means fast, *graceful* recovery, not zero downtime — [§6 availability](#az-pinning-node-loss-and-what-ha-means-here) states the RTO/RPO plainly. | -| "…we can't autoscale it" | Scale vertically: raise `FLEET_MAX_CONCURRENT_AGENTS` and the pod resources together ([§6](#resource-requests-count-the-sandboxes)). HPA and VPA are both actively harmful here ([§8](#cluster-integration-gotchas)). | -| "…the workloads are invisible to the cluster" | True and worth naming: sandboxes are Podman containers inside the pod, so they never appear in `kubectl get pods` or cAdvisor. Where to see them instead: [§8](#cluster-integration-gotchas). | -| "…it pins itself to one AZ" | It does — `ReadWriteOnce` EBS. Make the node group single-AZ deliberately rather than discovering it during an incident ([§6](#az-pinning-node-loss-and-what-ha-means-here)). | -| "…NetworkPolicy can't govern what the agent runs" | It can. Sandbox egress traverses the pod's network namespace via the rootless network helper, so pod-level NetworkPolicy applies to agent-executed code too ([§7](#networkpolicy), [§8](#cluster-integration-gotchas)). | -| "…secrets are in a `Secret`" | Swap in External Secrets Operator or the Secrets Store CSI driver ([§7](#secrets)). fleet's own guarantee is stronger than either: MCP credentials are brokered host-side and never enter a sandbox. | -| "…nothing here is CI-tested" | Also true. Run [§10](#10-verification-checklist) as an acceptance gate in your own pipeline; that is the substitute. | - -## 1. Topology - -Everything that was a process on the single box becomes a container in **one -pod**, so the loopback wiring the code expects still holds (containers in a pod -share a network namespace, so `127.0.0.1:8080` from the web container reaches the -chat listener): - -``` - ┌──────────────── EKS node (dedicated, one big instance) ───────────────┐ - Internet ─TLS─▶ ALB ──┼─▶ Service :3000 ─▶ pod │ - (ACM cert) │ ┌───────────────────────────────────────────────────────────┐ │ - │ │ container: web Next.js, 0.0.0.0:3000 │ │ - │ │ │ server-side proxy over loopback │ │ - │ │ ├─▶ 127.0.0.1:8080 chat ┐ │ │ - │ │ └─▶ 127.0.0.1:8000 orchestr. ┘ container: fleet │ │ - │ │ (one process: │ │ - │ │ chat + orchestrator │ │ - │ │ + scheduler + pool) │ │ - │ │ │ podman (rootless, │ │ - │ │ │ in-container) │ │ - │ │ ├─▶ sandbox ctr 1 │ │ - │ │ ├─▶ sandbox ctr 2 │ │ - │ │ └─▶ … up to │ │ - │ │ FLEET_MAX_CONCURRENT_AGENTS│ │ - │ └───────────────────────────────────────────────────────────┘ │ - └──────────────────────────────────────────────────────────────────────┘ - │ - └─▶ RDS PostgreSQL (two databases: chat + sched) -``` - -The Go listeners stay **loopback-only**. The orchestrator in particular is -impersonation-load-bearing and must remain on `127.0.0.1` — do not bind it to the -pod IP (see §8 for how metrics scraping works without breaking that). - -## 2. The hard part: rootless Podman inside a pod - -The sandbox is mandatory and fails closed, so the pod must be able to run -rootless Podman. Concretely fleet shells out to -`podman run --userns=keep-id:uid=1000,gid=1000 --read-only --cap-drop=ALL ---security-opt=no-new-privileges --security-opt seccomp=… --memory=… --cpus=… ---pids-limit=… …` and then `podman exec`s each tool call into it. Network posture -is per-turn: normal turns pass **no** `--network` flag (podman's rootless default -— pasta on ≥ 5.0, slirp4netns before it), lockdown and scheduled runs get -`--network=none`, and the allowlisted-egress posture explicitly requests -`--network=slirp4netns:allow_host_loopback=true`. That needs, inside the fleet -container: - -| Requirement | Why | How | -|---|---|---| -| `/etc/subuid` + `/etc/subgid` ranges for the container's user | `--userns=keep-id` maps uids into the range; without it Podman fails with a `newuidmap` mapping error | baked into the image (§3) | -| `newuidmap`/`newgidmap` with their file capabilities intact | performs the uid/gid mapping | `shadow-utils` in the image **and** `allowPrivilegeEscalation: true` (file caps are neutralized by `NoNewPrivileges` — the same reason `deploy/fleet.service` sets `NoNewPrivileges=no`) | -| a writable, **persistent** graph root (`$HOME/.local/share/containers`) | holds the ~1.5 GB sandbox image + per-container writable layers | the PVC mounted at `/var/lib/fleet` (§5) | -| an overlay-capable storage driver | `vfs` copies the whole ~1.5 GB image per container start — fatal for a per-turn warm pool | native `overlay` (privileged) or `fuse-overlayfs` + `/dev/fuse` | -| `/dev/net/tun` | the rootless network helper: **pasta** on Podman ≥ 5.0 (podman's own default, used by normal turns), or **slirp4netns**, which the allowlisted-egress posture specifically requires | privileged, or a device plugin | -| a **writable cgroup subtree** | otherwise Podman silently ignores `--memory`/`--cpus`, so the per-sandbox caps and per-task `sandbox_limits` **do not bind** | privileged (rw `/sys/fs/cgroup`); the analogue of `Delegate=yes` in the systemd unit | -| cgroup **v2** on the node | project-quota/limit behavior above | Amazon Linux 2023 nodes default to cgroup v2 | - -### Recommendation: run the fleet container privileged on a dedicated node - -```yaml -securityContext: - privileged: true - allowPrivilegeEscalation: true - runAsUser: 1000 # the image's fleet user, NOT root - runAsGroup: 1000 -``` - -**The container must run as uid 1000, not root — even privileged.** Podman -running as real root is *rootful*, and rootful Podman **ignores** -`--userns=keep-id`. The sandbox's uid 1000 then no longer maps to the process -that owns the workspace directory, so the agent can neither `chdir` into its -per-conversation workspace nor write files there — the failure the -`keep-id`/same-path invariant tests in `internal/sandbox` exist to catch. Keep -`runAsUser: 1000` and give the image a fixed uid-1000 user with subuid ranges -(§3b). - -This is the configuration that reliably satisfies all six rows above. It is also -the honest trade: **a privileged container is not a security boundary**, so the -security model becomes "the *node* is the blast radius, and the pod owns it." -That is materially weaker than the systemd deployment, where the fleet process is -an unprivileged system user. Mitigate deliberately: - -- **Dedicate the node group to fleet** — taint it and schedule nothing else there - (§6). Never co-schedule other tenants' workloads. -- **Block IMDS from pods** on that node group - (`--metadata-options http-put-response-hop-limit=1`) so agent-executed code - cannot mint the node role's credentials. Give fleet its own IRSA role with only - what it needs (ECR pull; Secrets Manager read if you use it). -- **Give the node role the minimum**, and keep the cluster's own secrets out of - the namespace. -- **NetworkPolicy** on the namespace: ingress only from the ALB target group, - egress only to RDS, your model provider, and the MCP endpoints you intend. -- The **inner** hardening is unchanged and still does the real work per turn: - read-only rootfs, `--cap-drop=ALL`, no-new-privileges, the default-deny seccomp - profile, `--network=none` for lockdown/scheduled runs, per-container - memory/CPU/pid caps, and the credential broker keeping MCP secrets host-side - (they never enter a sandbox — [ADR-0003](adr/0003-host-side-mcp-credential-brokering.md)). - -### If your policy forbids privileged pods - -The unprivileged variant needs, at minimum, `SYS_ADMIN` plus device access to -`/dev/fuse` and `/dev/net/tun` (a device plugin such as smarter-device-manager, -because containerd's default device cgroup denies both), `fuse-overlayfs` as the -driver, and a `RuntimeDefault` seccomp profile that permits `unshare`/`clone` -with `CLONE_NEWUSER`. Expect to fight the cgroup-delegation row above — and -**verify that `--memory` actually binds** (§10) rather than assuming it, because -if it doesn't, a `pandas` job takes the whole pod down instead of one sandbox. - -`kata`/`libkrun` microVM runtimes ([`docs/SANDBOX-RUNTIMES.md`](SANDBOX-RUNTIMES.md)) -need read-write `/dev/kvm` inside the pod. On EC2 that means a `.metal` instance -(nested KVM is not exposed on normal instance types), plus device access. Fleet's -boot preflight is fail-closed, so a missing `/dev/kvm` aborts startup rather than -silently downgrading to a shared kernel. Leave `sandbox.runtime` at the default -unless you have committed to metal nodes. - -## 3. Build the images - -### 3a. Sandbox image → ECR - -Unchanged from the single-host path: the Containerfile is a **bundle** artifact -(`/sandbox/Containerfile`), and fleet **never builds it at startup**. -Build and push it in CI — the repo already ships the reusable workflow -`.github/workflows/publish-sandbox-image.yml` (`workflow_call`) for exactly this, -which builds with `scripts/build-sandbox-image.sh` and pushes an immutable -`{git-sha}` tag. It exposes the pushed `image_ref` and `image_digest` as workflow -outputs, so a deploy job can consume the exact digest this section wants without -re-deriving it. Point it at ECR instead of GHCR, or mirror the GHCR tag into ECR. - -**Pull credentials depend on the package's visibility**, which in this org -follows the publishing repo (measured 2026-08-20): the images from the public -`fleet` and `example-config` repos are anonymously pullable, so a cluster needs -no `imagePullSecret` for them; the client-bundle images come from private repos -and do. GitHub's docs describe a private-by-default that these packages did not -follow, so verify a new package's visibility rather than assuming — an image you -expect to pull anonymously failing with a 403 at pod start is the symptom. - -The workflow publishes but does **not** pin: adoption is the explicit step -below. (It used to open a PR pinning `sandbox.image` in the client repo; that -step never once succeeded and was removed on 2026-08-20 — see the reusable -workflow's header.) - -Then set `sandbox.image` in the bundle's `manifest.yaml` to the immutable ref, or -override it per deployment with `FLEET_SANDBOX_IMAGE` -(`.dkr.ecr..amazonaws.com/fleet-sandbox@sha256:…`). Pin by digest — -`:latest` in a rebuilt-nightly registry means a turn's execution environment can -change under you. - -### 3b. fleet runtime image - -Podman lives in this image. A Fedora base keeps you on the same `crun`/ -`slirp4netns`/`fuse-overlayfs` versions the project develops against: - -```dockerfile -# Containerfile.fleet — build with: podman build -f Containerfile.fleet -t /fleet: . -FROM golang:1.27 AS build -WORKDIR /src -COPY go.mod go.sum ./ -RUN go mod download -COPY . . -RUN make build # → ./fleet and ./fleet-admin - -FROM fedora:44 -# podman + the rootless stack fleet actually invokes; curl for the exec probes (§7). -# Install BOTH rootless network helpers: passt/pasta is podman >= 5.0's default -# (normal turns), and slirp4netns is required by the allowlisted-egress posture — -# where a missing binary now aborts boot with a fail-closed preflight rather than -# erroring on every turn. awscli2 is for the ECR-login init container (§7); curl -# is for the exec probes. -RUN dnf install -y --setopt=install_weak_deps=False \ - podman crun conmon passt slirp4netns fuse-overlayfs containers-common \ - shadow-utils catatonit iptables-nft git curl ca-certificates awscli2 \ - && dnf clean all -# Fixed unprivileged user, uid 1000 — matches --userns=keep-id:uid=1000 and the -# sandbox image's USER, so bind-mounted workspace files line up from both sides. -RUN useradd --uid 1000 --home-dir /var/lib/fleet --shell /sbin/nologin fleet \ - && echo 'fleet:100000:65536' > /etc/subuid \ - && echo 'fleet:100000:65536' > /etc/subgid -# Same rootless-Podman settings scripts/bootstrap.sh writes for the service user: -# cgroupfs avoids needing a systemd user D-Bus session; the file events logger -# avoids journald permissions. fleet also passes --cgroup-manager=cgroupfs itself. -RUN install -d -o fleet -g fleet -m 0755 /etc/containers \ - && printf '[engine]\ncgroup_manager = "cgroupfs"\nevents_logger = "file"\n' \ - > /etc/containers/containers.conf -COPY --from=build /src/fleet /usr/local/bin/fleet -# The client config bundle. Bake it in for an immutable deploy (recommended) or -# mount it from a PVC/initContainer git clone; either way it must be WRITABLE by -# uid 1000, because the sandbox bind-mounts bundle dirs with SELinux relabeling. -COPY --chown=fleet:fleet config/default /opt/fleet/client -USER fleet -ENV HOME=/var/lib/fleet \ - XDG_RUNTIME_DIR=/var/lib/fleet/run \ - FLEET_CLIENT_CONFIG_DIR=/opt/fleet/client -WORKDIR /var/lib/fleet -# `fleet serve` is the explicit server verb (bare `fleet` also serves). -ENTRYPOINT ["/usr/local/bin/fleet", "serve"] -``` - -Notes that matter: - -- **`XDG_RUNTIME_DIR` must be writable and on the PVC-or-emptyDir**, not on the - read-only image layer — it holds per-container runtime state. -- **`FLEET_ENV_FILE` is a real choice, not a leftover.** With config injected as - pod env (the manifests below), leave it unset — but know the consequence: - config hot-reload re-reads the **env file** and honors boot's - process-env-over-file precedence, so anything pinned in the pod's environment - is **fixed until the pod restarts**, and `SIGUSR2` / - `POST /admin/reload-config` will report it under `skipped`. That is the right - trade for immutable-config deployments. If you want the reload path - ([`docs/CONFIG-RELOAD.md`](CONFIG-RELOAD.md)) to work, mount the credential - file from a Secret instead (e.g. `/etc/fleet/fleet.env`), point - `FLEET_ENV_FILE` at it, and do **not** also inject those keys as env vars. -- **Leave `FLEET_LOG_FILE` unset** so the process log goes to stdout/stderr for - your normal cluster log pipeline. -- Keep the bundle **writable by uid 1000** — the sandbox mounts `protocols/`, - `personas/`, `skills/`, and `system_prompts/` with `:Z`, which needs to write - the `security.selinux` xattr. This is the same reason `deploy/fleet.service` - lists `/opt/fleet/client` in `ReadWritePaths`. - -### 3c. web image - -```dockerfile -FROM node:24 AS build -WORKDIR /app -COPY web/package*.json ./ -RUN npm ci -COPY web/ . -RUN npm run build -FROM node:24-slim -WORKDIR /app -COPY --from=build /app ./ -ENV NODE_ENV=production PORT=3000 -USER node -CMD ["npm", "run", "start"] -``` - -## 4. PostgreSQL - -Use RDS (or Aurora PostgreSQL). fleet needs **two databases** in the same -cluster — chat and sched are deliberately separate -([ADR-0005](adr/0005-separate-chat-and-sched-databases.md)) — and each service -**self-migrates on first start**, so create empty databases and roles only: - -```sql -CREATE ROLE chat LOGIN PASSWORD '…'; CREATE DATABASE chat OWNER chat; -CREATE ROLE sched LOGIN PASSWORD '…'; CREATE DATABASE sched OWNER sched; -``` - -``` -FLEET_CHAT_DATABASE_URL=postgres://chat:…@:5432/chat?sslmode=require -FLEET_SCHED_DATABASE_URL=postgres://sched:…@:5432/sched?sslmode=require -``` - -Use `sslmode=require` or stricter (`verify-full` with the RDS CA bundle mounted). -Both pools are **critical readiness checks** — if either is down, `/readyz` -returns 503, which is exactly the signal you want the ALB to see. Tune -`FLEET_CHAT_DB_MAX_CONNS` / `FLEET_SCHED_DB_MAX_CONNS` against the instance -class's connection limit. `fleet migrate status` (via `kubectl exec`) reports -applied vs pending migrations; see [`docs/MIGRATIONS.md`](MIGRATIONS.md). - -Running Postgres in-cluster works but buys you a second stateful single-writer -workload to babysit; managed is the better trade here, and it lowers the pod's -base footprint. - -## 5. Storage - -One `ReadWriteOnce` EBS volume mounted at `/var/lib/fleet` carries **everything -stateful in the pod**: the rootless Podman graph root, the per-conversation -workspaces, the data dir (attachments/uploads, audit), and `XDG_RUNTIME_DIR`. - -```yaml -apiVersion: storage.k8s.io/v1 -kind: StorageClass -metadata: - name: fleet-gp3-xfs -provisioner: ebs.csi.aws.com -parameters: - type: gp3 - iops: "6000" - throughput: "500" - fsType: xfs # xfs + prjquota is what makes --storage-opt size work -mountOptions: - - prjquota -allowVolumeExpansion: true -volumeBindingMode: WaitForFirstConsumer -``` - -**Why `xfs` + `prjquota`:** the two disk caps are **layered, not either/or**. A -per-file `--ulimit fsize` cap is applied on every container regardless of -filesystem, and it is what bounds writes to the workspace bind mount. On top of -that, fleet adds `--storage-opt size=…` — a hard **total** cap on the writable -layer — but Podman only accepts that on a quota-capable driver (overlay+xfs with -pquota, btrfs, zfs — **not** overlay+ext4, and not vfs). fleet probes this once at -boot; where it isn't supported, the total-size cap is simply omitted and an agent -can still fill the writable layer with many individually-legal files. The omission -is logged at startup. Making the probe succeed is the point of this StorageClass. - -Sizing: the sandbox image (~1.5 GB) + one writable layer per concurrent sandbox -(`FLEET_SANDBOX_DISK_GB`, default 5 GiB each) + persistent workspaces + uploads -(`FLEET_UPLOAD_MAX_BYTES`, default 1 GiB per file). Start from the disk column of -the sizing table in [`docs/DEPLOYMENT.md`](DEPLOYMENT.md#choosing-a-host-sizing) -and add your workspace retention. `allowVolumeExpansion` matters — a full volume -is an outage. - -*Optional:* on an instance with local NVMe you can put the graph root on the -instance store (a `hostPath` plus a `storage.conf` `graphroot`) and keep only -workspaces/data on the PVC. Images and warm-container layers are reconstructible, -so ephemeral is fine for them, and you get much faster container starts. Verify -the quota probe still passes on that filesystem. - -## 6. Node group and scheduling - -One dedicated managed node group, one instance, nothing else on it: - -``` -eksctl create nodegroup --cluster --name fleet \ - --node-type m7i.12xlarge --nodes 1 --nodes-min 1 --nodes-max 1 \ - --node-zones \ - --node-taints dedicated=fleet:NoSchedule \ - --node-labels workload=fleet \ - --node-volume-size 200 --node-volume-type gp3 \ - --metadata-options httpPutResponseHopLimit=1 -``` - -- **AMI: Amazon Linux 2023** (cgroup v2 by default, ordinary writable - containerd). Bottlerocket and other minimal/immutable AMIs are **untested for - nested Podman** here — if you must, prove out §10's checks first. If your nodes - enforce SELinux, confirm the `:z`/`:Z` relabels the sandbox performs actually - succeed; AL2023's permissive default is what this recipe assumes. -- **Instance family:** memory-per-vCPU is the binding constraint (~1.5–3 GB of - RAM per concurrent agent for `pandas`/`matplotlib` work), so `r7i`/`r7a` beats - `c7i`. Size from the arithmetic in the next subsection, not from the vCPU - count: the 32-agent worked example used below and in the appendix needs - ~36 vCPU / 72 GiB once the base and web tier are counted, so **`m7i.12xlarge` - (48 vCPU / 192 GiB)** fits it with headroom while a 32-vCPU instance is already - short. Step up to `r7i.12xlarge` (48/384) if you intend to raise - `FLEET_SANDBOX_MEMORY` to 4–8 GiB for heavy `pandas`/`matplotlib` work, and to - `r7i.24xlarge` (96/768) for `FLEET_MAX_CONCURRENT_AGENTS=64` at those per-agent - sizes. Don't buy the memory before you've raised the per-sandbox cap that would - use it — the default is 512 MiB. -- **`maxPods`:** the sandboxes are Podman containers *inside* the pod, so they - don't consume pod IPs or count against `maxPods`. Only fleet's own pod does. -- **Karpenter:** annotate the pod `karpenter.sh/do-not-disrupt: "true"`. Node - consolidation on a single stateful pod means unplanned restarts. -- **Cluster Autoscaler / HPA:** neither applies. Do not attach an HPA. -- **PodDisruptionBudget:** don't set a blocking one (`minAvailable: 1` on a - single-replica workload blocks node drains indefinitely). A single-pod - deployment means node replacement is a **planned downtime window** — the honest - consequence of the single-writer design, same as rebooting the single box. - -### AZ pinning, node loss, and what "HA" means here - -Decide this deliberately — it is the question your reviewer will press hardest on. - -- **The pod is pinned to one Availability Zone.** A `ReadWriteOnce` EBS volume - exists in exactly one AZ, and `WaitForFirstConsumer` binds it where the pod - first scheduled. If your node group spans AZs, a replacement node in a - different AZ **cannot** mount the volume and the pod stays `Pending` with a - volume-node-affinity conflict. Make the node group **single-AZ on purpose** so - a replacement node always lands where the volume is. (EFS as an alternative - gets you cross-AZ at the cost of a network filesystem under the Podman graph - root and per-conversation workspaces — don't.) -- **Node loss does not self-heal quickly.** When a node goes `NotReady`, a - StatefulSet pod is *not* recreated until the old one is confirmed gone — - Kubernetes will not risk two writers, which is the same invariant fleet needs. - Recovery is: the node object is deleted (or you `kubectl delete pod --force`), - the EBS volume detaches, and the new pod attaches it on a fresh node in the - same AZ. Budget minutes, and prefer letting the node group replace the instance - over force-deleting by hand. -- **What HA actually means for this workload:** RTO is one pod restart plus - volume reattach; RPO for conversations, tasks, and run history is your RDS - backup window (in-flight turns are lost, and the graceful drain is what keeps - that number near zero for planned restarts). There is no zero-downtime rolling - upgrade, on EKS or on the single box — that is a property of the single-writer - design, not of this recipe. -- **PodDisruptionBudget:** leave it unset, or `maxUnavailable: 1`. A - `minAvailable: 1` PDB on a one-replica workload blocks every node drain - indefinitely and will page someone at 3am during a routine AMI upgrade. - -### Resource requests: count the sandboxes - -The sandbox containers' cgroups nest **under the pod's cgroup**, so their memory -and CPU count against the pod's limits. Size the pod, not just the process: - -``` -pod limit ≈ base (2 vCPU / 6 GB: Go process + Next app) - + FLEET_MAX_CONCURRENT_AGENTS × (FLEET_SANDBOX_CPUS, FLEET_SANDBOX_MEMORY) - + headroom -``` - -With `FLEET_MAX_CONCURRENT_AGENTS=32` and per-sandbox caps of `2g`/`1.0` CPU: -≈ 34 vCPU and ≈ 70 GB. Set **`requests == limits`** (Guaranteed QoS) and leave -real headroom: if the pod cgroup hits its memory limit, the kernel OOM killer -picks the biggest process in the cgroup — which can be the fleet process itself, -turning one runaway sandbox into a full restart. Raise the per-sandbox ceilings -(`FLEET_SANDBOX_MEMORY`, `FLEET_SANDBOX_CPUS`, `FLEET_SANDBOX_PIDS`, and the -operator maxima `FLEET_SANDBOX_{MEMORY_MAX_MB,CPUS_MAX,PIDS_MAX}`) and the pod -limits **together** — the defaults are 512 MiB / 1.0 CPU / 128 pids per sandbox, -and heavy analysis workloads get OOM-killed against that default long before your -node runs out of RAM. - -## 7. Manifests - -### Namespace, admission control, and identity - -**A privileged pod is rejected outright under the `baseline` or `restricted` Pod -Security Standards.** This is the single most likely reason a first deploy fails -in a governed cluster, and it fails at admission with no pod to debug. Label the -namespace so PSA permits it, and keep the exception scoped to this one namespace: - -```yaml -apiVersion: v1 -kind: Namespace -metadata: - name: fleet - labels: - # Required for the privileged fleet container (§2). Scope the exception to - # THIS namespace; do not relax the cluster-wide default. - pod-security.kubernetes.io/enforce: privileged - pod-security.kubernetes.io/enforce-version: latest - # Keep the warnings visible so you can see exactly which controls you gave up. - pod-security.kubernetes.io/audit: baseline - pod-security.kubernetes.io/warn: baseline - # AWS Load Balancer Controller: hold the pod un-Ready until the ALB target is - # registered, so a restart doesn't briefly 5xx (§7 ingress). - elbv2.k8s.aws/pod-readiness-gate-inject: enabled -``` - -If you run **Kyverno** or **Gatekeeper** as well, PSA labels are not enough — -those policies evaluate independently. Add a narrowly-scoped exception (namespace -`fleet`, the `fleet` StatefulSet, the specific rules: privileged, -`allowPrivilegeEscalation`, host devices) rather than a blanket exemption, and -write the §2 mitigations into the exception's justification field so the next -auditor finds the reasoning instead of just the hole. - -Identity: **fleet needs no Kubernetes API access at all** — nothing in the -process talks to the API server. Its ServiceAccount exists only to carry an AWS -role for ECR pulls (and Secrets Manager, if you use it), so it gets **no Role or -RoleBinding**, which is a useful thing to be able to say in review: - -```yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: fleet - namespace: fleet - annotations: - # IRSA. EKS Pod Identity (eks-pod-identity-agent + a PodIdentityAssociation) - # is the newer equivalent and avoids the OIDC-trust-policy boilerplate; use - # whichever your platform standardizes on. Neither needs IMDS, which is why - # the §6 hop-limit-1 hardening is safe. - eks.amazonaws.com/role-arn: arn:aws:iam:::role/fleet -# No RBAC Role/RoleBinding: fleet makes zero Kubernetes API calls. -automountServiceAccountToken: false -``` - -The attached IAM policy needs only `ecr:GetAuthorizationToken`, -`ecr:BatchGetImage`, `ecr:GetDownloadUrlForLayer`, and -`ecr:BatchCheckLayerAvailability` on the two repositories — plus -`secretsmanager:GetSecretValue` on your specific secret ARNs if you use External -Secrets with this role. - -### Secrets - -The literal `Secret` below is the minimum. For a GitOps repo, replace it with an -`ExternalSecret` (External Secrets Operator) or a `SecretProviderClass` (Secrets -Store CSI driver) pointing at Secrets Manager or Parameter Store — the pod spec -is unchanged either way, since both project a normal `Secret`. Note that rotating -these takes effect on **pod restart** unless you use the env-file mount described -in §3b. - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: fleet-env - namespace: fleet -stringData: - OPENROUTER_API_KEY: "…" - FLEET_CHAT_DATABASE_URL: "postgres://chat:…@rds:5432/chat?sslmode=require" - FLEET_SCHED_DATABASE_URL: "postgres://sched:…@rds:5432/sched?sslmode=require" - FLEET_SERVER_TOKEN: "…" # web container's CHAT_SERVER_TOKEN must match - ADMIN_API_KEY: "…" # orchestrator admin key - APP_SESSION_SECRET: "…" # signs the web session cookie - # plus the MCP connector credentials the bundle's manifest names -``` - -The workload — a `StatefulSet`, because it gives at-most-one-pod semantics with -an `RWO` volume (a `Deployment`'s rolling update would briefly run two fleet -processes against one database pair, which the single-owner leases forbid): - -```yaml -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: fleet - namespace: fleet -spec: - replicas: 1 # never raise this - serviceName: fleet - podManagementPolicy: OrderedReady - updateStrategy: - type: RollingUpdate # for a single replica: terminate, then create - selector: - matchLabels: { app: fleet } - template: - metadata: - labels: { app: fleet } - annotations: - karpenter.sh/do-not-disrupt: "true" - spec: - serviceAccountName: fleet - # A freshly provisioned EBS volume is root-owned; without fsGroup the - # uid-1000 process cannot create the Podman graph root, the workspace, or - # XDG_RUNTIME_DIR, and the pod crash-loops on startup. OnRootMismatch keeps - # restarts fast once the volume is large (no full recursive rechown). - securityContext: - fsGroup: 1000 - fsGroupChangePolicy: OnRootMismatch - nodeSelector: { workload: fleet } - tolerations: - - key: dedicated - value: fleet - effect: NoSchedule - # Must exceed FLEET_SHUTDOWN_GRACE_SECONDS: on SIGTERM fleet stops - # admitting work, flips /healthz + /readyz to 503 so the ALB drains it, - # then drains in-flight chat turns AND scheduled tasks before exiting. - terminationGracePeriodSeconds: 90 - - initContainers: - # Pre-pull the sandbox image into the rootless store on the PVC. fleet - # never builds it and this keeps the first turn off a 1.5 GB download - # (and keeps a registry outage from surfacing as a failed turn). - - name: pull-sandbox - image: .dkr.ecr..amazonaws.com/fleet: - command: ["/bin/sh", "-c"] - args: - - | - set -e - aws ecr get-login-password --region "$AWS_REGION" \ - | podman login --username AWS --password-stdin "${ECR_REGISTRY}" - podman pull "$FLEET_SANDBOX_IMAGE" - env: - - { name: AWS_REGION, value: "" } - - { name: ECR_REGISTRY, value: ".dkr.ecr..amazonaws.com" } - - { name: FLEET_SANDBOX_IMAGE, value: ".dkr.ecr..amazonaws.com/fleet-sandbox@sha256:…" } - - { name: HOME, value: /var/lib/fleet } - - { name: XDG_RUNTIME_DIR, value: /var/lib/fleet/run } - securityContext: - privileged: true # same reasons as the main container (§2) - runAsUser: 1000 - volumeMounts: - - { name: state, mountPath: /var/lib/fleet } - - containers: - - name: fleet - image: .dkr.ecr..amazonaws.com/fleet: - envFrom: - - secretRef: { name: fleet-env } - env: - # Listeners stay loopback — the web container reaches them through - # the shared pod network namespace. The orchestrator MUST stay on - # 127.0.0.1 (it is impersonation-load-bearing). - - { name: FLEET_SERVER_ADDR, value: "127.0.0.1:8080" } - - { name: FLEET_ORCHESTRATOR_ADDR, value: "127.0.0.1:8000" } - - { name: FLEET_CLIENT_CONFIG_DIR, value: "/opt/fleet/client" } - # Absolute, not CWD-relative: don't depend on WORKDIR surviving an - # image refactor. - - { name: FLEET_DATA_DIR, value: "/var/lib/fleet/data" } - - { name: FLEET_WORKSPACE_ROOT, value: "/var/lib/fleet/workspace" } - - { name: HOME, value: "/var/lib/fleet" } - - { name: XDG_RUNTIME_DIR, value: "/var/lib/fleet/run" } - - { name: FLEET_SANDBOX_IMAGE, value: ".dkr.ecr..amazonaws.com/fleet-sandbox@sha256:…" } - # Sizing knobs — keep in lockstep with the pod resources below (§6). - - { name: FLEET_MAX_CONCURRENT_AGENTS, value: "32" } - - { name: FLEET_SANDBOX_MEMORY, value: "2g" } - - { name: FLEET_SANDBOX_CPUS, value: "1.0" } - - { name: FLEET_SANDBOX_WARM_SIZE, value: "4" } - - { name: FLEET_SHUTDOWN_GRACE_SECONDS, value: "60" } - - { name: FLEET_TIMEZONE, value: "UTC" } - # Trust the ALB's X-Forwarded-For only from in-pod/in-VPC hops. - - { name: FLEET_TRUSTED_PROXIES, value: "127.0.0.1,::1" } - securityContext: - privileged: true - allowPrivilegeEscalation: true # newuidmap/newgidmap file caps - runAsUser: 1000 - runAsGroup: 1000 - resources: - requests: { cpu: "34", memory: "70Gi" } - limits: { cpu: "34", memory: "70Gi" } - # Probes are exec, not httpGet: kubelet dials the POD IP, which cannot - # reach a 127.0.0.1-only listener. curl is in the image for this. - startupProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/readyz"] } - periodSeconds: 10 - failureThreshold: 30 # DB self-migration + warm-pool fill - livenessProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/livez"] } - periodSeconds: 30 - failureThreshold: 4 - readinessProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/readyz"] } - periodSeconds: 10 - volumeMounts: - - { name: state, mountPath: /var/lib/fleet } - - - name: web - image: .dkr.ecr..amazonaws.com/fleet-web: - ports: - - { name: http, containerPort: 3000 } - env: - - { name: CHAT_SERVER_URL, value: "http://127.0.0.1:8080" } - - { name: ORCHESTRATOR_SERVER_URL, value: "http://127.0.0.1:8000" } - - { name: CHAT_SERVER_TOKEN, valueFrom: { secretKeyRef: { name: fleet-env, key: FLEET_SERVER_TOKEN } } } - - { name: ORCHESTRATOR_SERVER_TOKEN, valueFrom: { secretKeyRef: { name: fleet-env, key: ADMIN_API_KEY } } } - - { name: APP_SESSION_SECRET, valueFrom: { secretKeyRef: { name: fleet-env, key: APP_SESSION_SECRET } } } - securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: false # Next writes .next/cache at runtime - capabilities: { drop: ["ALL"] } - resources: - requests: { cpu: "500m", memory: "1Gi" } - limits: { cpu: "2", memory: "2Gi" } - readinessProbe: - httpGet: { path: /, port: 3000 } - periodSeconds: 10 - # SIGTERM reaches every container at once, so without this the public - # tier can die while fleet is still draining a turn — the browser sees a - # dropped stream instead of a finished answer. Sleep past the ALB's - # deregistration delay, then let Next exit. - lifecycle: - preStop: - exec: { command: ["sleep", "20"] } - - volumeClaimTemplates: - - metadata: { name: state } - spec: - accessModes: ["ReadWriteOnce"] - storageClassName: fleet-gp3-xfs - resources: { requests: { storage: 400Gi } } -``` - -Service + ALB ingress (TLS terminates at the ALB with an ACM cert, so no Caddy -container is needed — the Next app remains the only public entrypoint): - -```yaml -apiVersion: v1 -kind: Service -metadata: { name: fleet, namespace: fleet } -spec: - selector: { app: fleet } - ports: [{ name: http, port: 3000, targetPort: 3000 }] ---- -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: fleet - namespace: fleet - annotations: - alb.ingress.kubernetes.io/scheme: internet-facing - alb.ingress.kubernetes.io/target-type: ip - alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' - alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:… - alb.ingress.kubernetes.io/ssl-redirect: "443" - # SSE: agent turns stream for minutes. The default 60s idle timeout cuts them off. - alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=1800 - alb.ingress.kubernetes.io/healthcheck-path: / -spec: - ingressClassName: alb - rules: - - host: fleet.example.com - http: - paths: - - path: / - pathType: Prefix - backend: { service: { name: fleet, port: { number: 3000 } } } -``` - -**The ALB idle timeout is the one setting people get wrong.** Chat responses are -SSE streams that can run for many minutes; the ALB's 60-second default idle -timeout will sever them mid-turn. Raise it (1800s above) — the equivalent of -`flush_interval -1` + `read_timeout 30m` in `deploy/Caddyfile`. - -Set `FLEET_PUBLIC_URL` / `FLEET_PUBLIC_BASE_URL` to the public origin so -notification links and share URLs resolve. Login works exactly as on the single -box (email + password, optional magic-link, optional OIDC SSO — all in the Next -layer); see [`docs/DEPLOYMENT.md`](DEPLOYMENT.md) for the login model. - -### NetworkPolicy - -Worth stating explicitly because it answers a real objection: **agent-executed -code is covered by pod-level NetworkPolicy.** Sandbox containers have no pod IP of -their own — their egress is NAT'd through the fleet pod's network namespace by the -rootless network helper (pasta on Podman ≥ 5.0, slirp4netns before it) — so a -policy on this pod governs what the model's `bash` and `run_python` can reach. This composes with, and does not replace, fleet's own -egress controls: `--network=none` for lockdown and scheduled runs is the hard -seal, and the allowlisted-egress proxy mode is -[ADR-0012](adr/0012-sandbox-egress-allowlist.md) / -[ADR-0031](adr/0031-chat-sandbox-egress.md). - -Requires a policy-enforcing CNI — the **VPC CNI enforces NetworkPolicy** only -with `enableNetworkPolicy: true` (EKS 1.25+); otherwise use Calico or Cilium. - -```yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: { name: fleet, namespace: fleet } -spec: - podSelector: { matchLabels: { app: fleet } } - policyTypes: ["Ingress", "Egress"] - ingress: - - from: [{ ipBlock: { cidr: } }] # ALB target-type: ip - ports: [{ port: 3000, protocol: TCP }] - egress: - - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }, - podSelector: { matchLabels: { k8s-app: kube-dns } } }] - ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }] - - to: [{ ipBlock: { cidr: } }] - ports: [{ port: 5432, protocol: TCP }] - # Model provider, ECR, and the MCP endpoints you intend. Narrow this as far as - # your provider's addressing allows; it is the boundary on agent egress. - - to: [{ ipBlock: { cidr: 0.0.0.0/0, except: [, 169.254.169.254/32] } }] - ports: [{ port: 443, protocol: TCP }] -``` - -Excluding `169.254.169.254/32` is belt-and-braces alongside the §6 IMDS hop -limit: two independent controls stopping agent code from reaching instance -credentials. - -### Metrics scrape sidecar - -`/metrics` is on the orchestrator's loopback listener and is admin-key gated -(§8). Rather than binding that listener to the pod IP — which would break the -impersonation boundary — add a tiny proxy that exposes only `GET /metrics` on the -pod IP and injects the key, then point a `ServiceMonitor`/`PodMonitor` (Prometheus -Operator) or your scrape config at port 9090: - -```yaml - - name: metrics-proxy - image: nginx:1.30-alpine - ports: [{ name: metrics, containerPort: 9090 }] - # nginx.conf (mount from a ConfigMap): - # server { listen 9090; - # location = /metrics { - # proxy_pass http://127.0.0.1:8000/metrics; - # proxy_set_header Authorization "Bearer "; - # } - # location / { return 404; } } - # Render the key in via envsubst on an nginx.conf.template at startup — - # don't bake it into the ConfigMap. - securityContext: - allowPrivilegeEscalation: false - capabilities: { drop: ["ALL"] } - resources: - requests: { cpu: "50m", memory: "64Mi" } - limits: { cpu: "200m", memory: "128Mi" } -``` - -The orchestrator authenticates admin reads with `Authorization: Bearer -`. Keep the proxy's `location /` a 404 so the sidecar cannot become -a general-purpose hole into the orchestrator, and keep it off the Service that -backs the Ingress. - -### Packaging these manifests for GitOps - -Nothing above needs templating to be managed declaratively. A Kustomize base with -per-environment overlays is the smaller-footprint option; a thin Helm chart is -fine if charts are your standard: - -``` -deploy/k8s/ - base/ namespace.yaml serviceaccount.yaml statefulset.yaml service.yaml - ingress.yaml networkpolicy.yaml storageclass.yaml kustomization.yaml - overlays/prod/ kustomization.yaml (images, replicas:1, resources, host, ARNs) -``` - -Two things bite in **Argo CD** specifically: - -1. **`volumeClaimTemplates` are immutable.** Any change to them makes the - StatefulSet un-patchable, and Argo reports a permanently `OutOfSync` app. To - resize, patch the **PVC** directly (`allowVolumeExpansion: true`, §5) and leave - the template alone; for a template change, delete the StatefulSet with - `--cascade=orphan` and re-apply. -2. **Set `Replace=false` and avoid auto-prune on the PVC.** An automated sync that - prunes the volume claim destroys workspaces, uploads, and the audit dir. Add - `argocd.argoproj.io/sync-options: Prune=false` on the PVC, or exclude - PersistentVolumeClaims from the app's prune scope. - -Also keep **automated sync from being an upgrade mechanism you didn't intend**: -this workload restarts (with downtime, §6) on every pod-spec change, so pin -digests in the overlay and let a human promote them. - -## 8. Observability and cluster integration - -- **Logs** go to stdout/stderr → your CloudWatch/Fluent Bit pipeline. Leave - `FLEET_LOG_FILE` unset; the file sink exists for hosts without a log collector - and would only duplicate lines onto the PVC. -- **Metrics:** `/metrics` (Prometheus text format) is served by the - **orchestrator** on `127.0.0.1:8000` and is **admin-API-key gated** — cost and - token data must not be public. Because that listener must stay loopback, give - the pod a tiny reverse-proxy sidecar that listens on the pod IP, forwards only - `GET /metrics` to `127.0.0.1:8000`, and injects the admin key; point your - scraper at the sidecar. Do not "solve" this by binding the orchestrator to the - pod IP. -- **Useful series** for this deployment: `fleet_sandbox_memory_usage_bytes` / - `fleet_sandbox_memory_limit_bytes` (right-size the per-sandbox caps), - `fleet_sandbox_pids_peak`, and the sandbox pool gauge. Per-run peaks come from - read-only `podman stats` sampling — observability only, never affecting - isolation ([`docs/DEPLOYMENT.md`](DEPLOYMENT.md)). -- **Tracing:** `FLEET_OTEL_ENDPOINT` + `FLEET_OTEL_SAMPLE_RATIO` if you run a - collector. - -### Cluster integration gotchas - -Each of these is something a Kubernetes-native environment does by default that -either breaks this pod or silently misleads you about it. - -- **The sandboxes are invisible to the Kubernetes API.** They are Podman - containers inside the pod: no entry in `kubectl get pods`, no cAdvisor - container metrics, no kubelet events, no k8s audit records for them. Where to - look instead: `kubectl exec … -- podman ps`, the `fleet_sandbox_*` metrics, the - per-task resource telemetry, and the per-run logs / audit dir. Say this out - loud in review — a platform team that expects pod-level visibility into agent - workloads will otherwise assume it exists. -- **NodeLocal DNSCache breaks DNS inside sandboxes.** If the node's - `/etc/resolv.conf` points at a link-local or loopback address (`169.254.20.10`, - `127.0.0.1`), that address means something different inside the sandbox's - network namespace under either helper, and name resolution fails for every - outbound HTTP tool — while the fleet process itself resolves fine, so it looks - like a model problem, not a DNS problem. Pin explicit resolvers for Podman in - the image's `containers.conf`: - - ```ini - [containers] - dns_servers = ["172.20.0.10"] # your cluster's kube-dns Service IP, or a VPC resolver - ``` - -- **`ResourceQuota` / `LimitRange` in the namespace will reject the pod.** A - 34-vCPU/70-GiB request trips inherited defaults, and a `LimitRange` with a - low `max` silently caps it. Give the namespace its own quota sized to the node, - or none. -- **VPA in `Auto` mode is destructive here** — it restarts the pod to resize it. - If you run VPA cluster-wide, exclude this workload or set `updateMode: "Off"` - and use its recommendations to hand-tune §6. -- **`automountServiceAccountToken: false`** is safe and recommended: fleet makes - no API calls, and the pod runs code the model wrote. IRSA/Pod Identity project - their own token separately and keep working. -- **Runtime security tooling** (Falco, GuardDuty Runtime Monitoring, Aqua/Sysdig) - will see nested container creation, user-namespace clones, and `newuidmap` from - a privileged pod, and will alert on all of it. Baseline those signatures for - this namespace *before* go-live — otherwise fleet's normal operation reads as an - ongoing container-escape attempt, and the noise trains everyone to ignore the - detector. -- **Node AMI upgrades are planned outages** (§6), so exclude this node group from - any automatic AMI-refresh schedule and drain it deliberately. - -## 9. Day-2 operations (what replaces bootstrap/update/doctor) - -| Single-host | On EKS | -|---|---| -| `scripts/bootstrap.sh` | build images (§3) + `kubectl apply` | -| `fleet update` | build a new image tag, `kubectl set image` / re-apply, pod restarts | -| `fleet restart` | `kubectl rollout restart statefulset/fleet` | -| `scripts/doctor.sh` (systemd-specific) | `kubectl exec … -- fleet validate-config` plus §10 — and the in-process Doctor panel still works, see below | -| `fleet admin add ` | `kubectl exec -it sts/fleet -c fleet -- fleet admin add ` | -| `fleet mcp account set …` | same, via `kubectl exec` | -| journald | `kubectl logs sts/fleet -c fleet` | - -`fleet validate-config` is the portable check — it verifies the bundle, podman -reachability, the sandbox image's presence, and the runtime preflight. - -**Settings → Admin → Doctor works here too**, and degrades honestly: its -container-portable checks (chat and sched databases, model API key, -subuid/subgid ranges, rootless podman, sandbox image) all run normally, while the -systemd-dependent ones (sibling unit state, the scheduled-backup timer) report -`skip` with "systemctl not on PATH (no systemd)" rather than inventing advisories -about units that were never meant to exist here. Note the consequence, though: a -`skip` on scheduled backups is *not* reassurance — it is the gap you closed by -hand above. - -**Config and bundle changes.** With the bundle baked into the image, a bundle -change is an image rebuild + pod restart. If you instead mount the bundle from a -PVC or clone it in an init container, MCP server definitions can be reloaded live -with `fleet mcp reload` / SIGHUP / the admin endpoint -([`docs/MCP-RELOAD.md`](MCP-RELOAD.md)). Reloadable env ceilings need the env-file -setup described in §3b; otherwise change them by editing the manifest and -restarting. - -**Backups — read this one carefully.** Two things are stateful: the databases and -the PVC (EBS snapshots via the CSI `VolumeSnapshot` API — workspaces, uploads, -audit). The Podman image store on the PVC is reconstructible; don't optimize -backups for it. - -The trap: fleet now ships `deploy/fleet-backup.service` + `fleet-backup.timer`, -which `bootstrap.sh --enable-service` installs and enables **by default**, and -`fleet doctor` reports on. **None of that exists here** — those are systemd units, -`bootstrap.sh` never runs on this deployment, and nothing in the pod will tell you -backups aren't happening. That gap is precisely the failure the timer was added to -fix (#966: a box reporting "38 ok, 0 advisories" while holding no backups at all, -for five days, with live client data). So pick one deliberately and write it down: - -- **RDS automated backups + snapshots** (simplest, and what this guide assumes) — - covers exactly the loss of a host or volume that a same-host `pg_dump` does not. -- **A Kubernetes `CronJob`** running `fleet backup` or `pg_dump` on a schedule, if - you want the logical dump the timer would have produced (recoverable from a bad - migration or an accidental delete). Give it its own ServiceAccount and write to - S3, not to the PVC — a dump beside the data it protects is not a backup. - -Either way, note that neither captures attachment/upload files, which live on the -PVC — those need the `VolumeSnapshot` schedule. See -[`docs/BACKUP_RESTORE.md`](BACKUP_RESTORE.md) for what a dump does and does not -cover. - -**Upgrades and node patching** are downtime windows. Sequence: cordon nothing, -just `kubectl delete pod` / `rollout restart` and let the drain budget run — -fleet flips `/readyz` to 503, the ALB stops sending traffic, in-flight turns and -scheduled tasks drain within `FLEET_SHUTDOWN_GRACE_SECONDS`, and the new pod -re-attaches the same PVC. - -## 10. Verification checklist - -Run these in the fleet container (`kubectl exec -it sts/fleet -c fleet -- bash`) -before you call the deployment done. Each maps to a row in §2 that fails -*silently* if you skip it. - -```sh -# 1. Rootless podman works at all, with the expected driver. -podman info --format '{{.Host.CgroupsVersion}} {{.Store.GraphDriverName}} {{.Host.Security.Rootless}}' -# want: v2 true (NOT vfs) - -# 2. The uid mapping fleet actually uses. -podman run --rm --userns=keep-id:uid=1000,gid=1000 "$FLEET_SANDBOX_IMAGE" id -# want: uid=1000 gid=1000 - -# 3. Memory limits BIND. If this prints "max", --memory is being ignored and -# every per-sandbox and per-task cap is fiction. -podman run --rm --memory=64m "$FLEET_SANDBOX_IMAGE" cat /sys/fs/cgroup/memory.max -# want: 67108864 - -# 4. All three network postures (each needs /dev/net/tun for its helper). -# a) normal turns — podman's rootless default (pasta on >= 5.0): -podman run --rm "$FLEET_SANDBOX_IMAGE" \ - python3 -c 'import socket;socket.create_connection(("1.1.1.1",443),5);print("default egress ok")' -# b) allowlisted-egress posture — needs the slirp4netns binary specifically. -# A missing binary now aborts BOOT with a fail-closed preflight, so check it -# here if you plan to enable that mode: -podman run --rm --network=slirp4netns:allow_host_loopback=true \ - "$FLEET_SANDBOX_IMAGE" python3 -c 'import socket;socket.create_connection(("1.1.1.1",443),5);print("slirp egress ok")' -# c) lockdown / scheduled runs — the hard seal: -podman run --rm --network=none "$FLEET_SANDBOX_IMAGE" true && echo "sealed mode ok" - -# 5. The TOTAL writable-layer cap (the per-file ulimit applies either way, §5). -podman run --rm --storage-opt size=1g "$FLEET_SANDBOX_IMAGE" true \ - && echo "total-size cap available" || echo "per-file cap only — total layer size unbounded" - -# 6. DNS inside a sandbox — the NodeLocal DNSCache trap (§8). Resolution can -# fail here while the fleet process itself resolves fine. -podman run --rm --network=slirp4netns:allow_host_loopback=true \ - "$FLEET_SANDBOX_IMAGE" python3 -c 'import socket;print(socket.gethostbyname("api.openai.com"))' - -# 7. The volume is actually writable by uid 1000 (fsGroup, §7). -touch /var/lib/fleet/.write-probe && rm /var/lib/fleet/.write-probe && echo "volume writable" - -# 8. fleet's own preflight: bundle, podman, image, runtime. -fleet validate-config - -# 9. Health + drain semantics. -curl -fsS http://127.0.0.1:8080/readyz; curl -fsS http://127.0.0.1:8080/livez -``` - -From outside the pod, confirm the cluster-side wiring: - -```sh -# Admission actually permits the pod (fails at admission, with no pod to debug). -kubectl -n fleet get statefulset fleet -o jsonpath='{.status.readyReplicas}' -kubectl -n fleet describe statefulset fleet | grep -iA3 'FailedCreate\|forbidden' - -# The volume landed in the AZ the node group lives in (§6). -kubectl -n fleet get pvc state-fleet-0 -o jsonpath='{.spec.volumeName}' \ - | xargs -I{} kubectl get pv {} -o jsonpath='{.spec.nodeAffinity}' - -# SSE survives the ALB: this must stream for minutes, not cut off at 60s. -curl -N https://fleet.example.com/… # any streaming chat turn - -# Graceful drain: /readyz flips to 503 and in-flight work finishes, no SIGKILL. -kubectl -n fleet delete pod fleet-0 --wait=true -``` - -Then, from the UI, run one interactive turn that executes `run_python` and one -scheduled task, and confirm in `kubectl logs` that no line reports the -`--storage-opt` fallback or a warm-pool cold-start failure. - -## Appendix: the complete manifest set - -The sections above explain each piece; this is all of it assembled in apply -order, so nothing gets missed in transcription. It is the same content — if the -two ever disagree, the numbered sections are the explanation and this is the -transcription. - -**Fill these in first.** Every placeholder appears in angle brackets: - -| Placeholder | Where it comes from | -|---|---| -| ``, `` | your AWS account ID and region | -| `` | the image tags you built in §3 | -| `` | `sha256:…` of the sandbox image pushed in §3a — pin by digest | -| `` | the IRSA/Pod Identity role from §7 | -| `` | the ACM certificate for your hostname | -| ``, `` | the RDS endpoint and its subnet range (§4) | -| `` | the cluster VPC range, for the ALB ingress rule (§7) | -| `` | `kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'` | -| `` | the public hostname, e.g. `fleet.example.com` | -| secret values | §7; prefer External Secrets over literals | - -Sizing below is the worked 32-concurrent-agent example (`m7i.12xlarge`): raise -`FLEET_MAX_CONCURRENT_AGENTS`, the per-sandbox caps, the pod resources, and the -instance type **together** — see [§6](#resource-requests-count-the-sandboxes). - -```yaml -# 1 ── Namespace. The PSA labels are what let the privileged pod be admitted (§7). -apiVersion: v1 -kind: Namespace -metadata: - name: fleet - labels: - pod-security.kubernetes.io/enforce: privileged - pod-security.kubernetes.io/enforce-version: latest - pod-security.kubernetes.io/audit: baseline - pod-security.kubernetes.io/warn: baseline - elbv2.k8s.aws/pod-readiness-gate-inject: enabled ---- -# 2 ── StorageClass. xfs + prjquota is what makes the sandbox disk quota a HARD -# cap on top of the per-file ulimit that applies regardless (§5). -apiVersion: storage.k8s.io/v1 -kind: StorageClass -metadata: - name: fleet-gp3-xfs -provisioner: ebs.csi.aws.com -parameters: - type: gp3 - iops: "6000" - throughput: "500" - fsType: xfs -mountOptions: ["prjquota"] -allowVolumeExpansion: true -volumeBindingMode: WaitForFirstConsumer ---- -# 3 ── Identity. No Role/RoleBinding: fleet makes zero Kubernetes API calls (§7). -apiVersion: v1 -kind: ServiceAccount -metadata: - name: fleet - namespace: fleet - annotations: - eks.amazonaws.com/role-arn: -automountServiceAccountToken: false ---- -# 4 ── Secrets. Replace with an ExternalSecret / SecretProviderClass in a GitOps -# repo — the pod spec below is identical either way (§7). -apiVersion: v1 -kind: Secret -metadata: - name: fleet-env - namespace: fleet -stringData: - OPENROUTER_API_KEY: "" - FLEET_CHAT_DATABASE_URL: "postgres://chat:@:5432/chat?sslmode=require" - FLEET_SCHED_DATABASE_URL: "postgres://sched:@:5432/sched?sslmode=require" - FLEET_SERVER_TOKEN: "" - ADMIN_API_KEY: "" - APP_SESSION_SECRET: "" - # plus every MCP connector credential the bundle's manifest.yaml names ---- -# 5 ── The workload. -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: fleet - namespace: fleet -spec: - replicas: 1 # NEVER raise: single-owner leases + per-process semaphore - serviceName: fleet - podManagementPolicy: OrderedReady - updateStrategy: { type: RollingUpdate } - selector: - matchLabels: { app: fleet } - template: - metadata: - labels: { app: fleet } - annotations: - karpenter.sh/do-not-disrupt: "true" - spec: - serviceAccountName: fleet - # Without fsGroup, uid 1000 cannot write the fresh EBS volume and the pod - # crash-loops before it ever starts podman (§7). - securityContext: - fsGroup: 1000 - fsGroupChangePolicy: OnRootMismatch - nodeSelector: { workload: fleet } - tolerations: - - { key: dedicated, value: fleet, effect: NoSchedule } - terminationGracePeriodSeconds: 90 # > FLEET_SHUTDOWN_GRACE_SECONDS - - initContainers: - - name: pull-sandbox - image: .dkr.ecr..amazonaws.com/fleet: - command: ["/bin/sh", "-c"] - args: - - | - set -e - aws ecr get-login-password --region "$AWS_REGION" \ - | podman login --username AWS --password-stdin "$ECR_REGISTRY" - podman pull "$FLEET_SANDBOX_IMAGE" - env: - - { name: AWS_REGION, value: "" } - - { name: ECR_REGISTRY, value: ".dkr.ecr..amazonaws.com" } - - { name: FLEET_SANDBOX_IMAGE, value: ".dkr.ecr..amazonaws.com/fleet-sandbox@" } - - { name: HOME, value: "/var/lib/fleet" } - - { name: XDG_RUNTIME_DIR, value: "/var/lib/fleet/run" } - securityContext: { privileged: true, runAsUser: 1000 } - volumeMounts: - - { name: state, mountPath: /var/lib/fleet } - - containers: - - name: fleet - image: .dkr.ecr..amazonaws.com/fleet: - envFrom: - - secretRef: { name: fleet-env } - env: - - { name: FLEET_SERVER_ADDR, value: "127.0.0.1:8080" } - - { name: FLEET_ORCHESTRATOR_ADDR, value: "127.0.0.1:8000" } # must stay loopback - - { name: FLEET_CLIENT_CONFIG_DIR, value: "/opt/fleet/client" } - - { name: FLEET_DATA_DIR, value: "/var/lib/fleet/data" } - - { name: FLEET_WORKSPACE_ROOT, value: "/var/lib/fleet/workspace" } - - { name: HOME, value: "/var/lib/fleet" } - - { name: XDG_RUNTIME_DIR, value: "/var/lib/fleet/run" } - - { name: FLEET_SANDBOX_IMAGE, value: ".dkr.ecr..amazonaws.com/fleet-sandbox@" } - - { name: FLEET_PUBLIC_URL, value: "https://" } - - { name: FLEET_MAX_CONCURRENT_AGENTS, value: "32" } - - { name: FLEET_SANDBOX_MEMORY, value: "2g" } - - { name: FLEET_SANDBOX_CPUS, value: "1.0" } - - { name: FLEET_SANDBOX_WARM_SIZE, value: "4" } - - { name: FLEET_SHUTDOWN_GRACE_SECONDS, value: "60" } - - { name: FLEET_TIMEZONE, value: "UTC" } - - { name: FLEET_TRUSTED_PROXIES, value: "127.0.0.1,::1" } - securityContext: - privileged: true # see §2 for what this buys and costs - allowPrivilegeEscalation: true # newuidmap/newgidmap file caps - runAsUser: 1000 # NOT root — rootful podman ignores keep-id - runAsGroup: 1000 - resources: - requests: { cpu: "34", memory: "70Gi" } # base + 32 × per-sandbox cap - limits: { cpu: "34", memory: "70Gi" } - # exec, not httpGet: kubelet dials the pod IP and cannot reach loopback. - startupProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/readyz"] } - periodSeconds: 10 - failureThreshold: 30 - livenessProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/livez"] } - periodSeconds: 30 - failureThreshold: 4 - readinessProbe: - exec: { command: ["curl", "-fsS", "http://127.0.0.1:8080/readyz"] } - periodSeconds: 10 - volumeMounts: - - { name: state, mountPath: /var/lib/fleet } - - - name: web - image: .dkr.ecr..amazonaws.com/fleet-web: - ports: - - { name: http, containerPort: 3000 } - env: - - { name: CHAT_SERVER_URL, value: "http://127.0.0.1:8080" } - - { name: ORCHESTRATOR_SERVER_URL, value: "http://127.0.0.1:8000" } - - { name: CHAT_SERVER_TOKEN, valueFrom: { secretKeyRef: { name: fleet-env, key: FLEET_SERVER_TOKEN } } } - - { name: ORCHESTRATOR_SERVER_TOKEN, valueFrom: { secretKeyRef: { name: fleet-env, key: ADMIN_API_KEY } } } - - { name: APP_SESSION_SECRET, valueFrom: { secretKeyRef: { name: fleet-env, key: APP_SESSION_SECRET } } } - securityContext: - allowPrivilegeEscalation: false - capabilities: { drop: ["ALL"] } - resources: - requests: { cpu: "500m", memory: "1Gi" } - limits: { cpu: "2", memory: "2Gi" } - readinessProbe: - httpGet: { path: /, port: 3000 } - periodSeconds: 10 - lifecycle: - preStop: - exec: { command: ["sleep", "20"] } # outlive ALB deregistration - - volumeClaimTemplates: # immutable — see the Argo notes in §7 - - metadata: { name: state } - spec: - accessModes: ["ReadWriteOnce"] - storageClassName: fleet-gp3-xfs - resources: { requests: { storage: 400Gi } } ---- -# 6 ── Service (the web tier is the only exposed port). -apiVersion: v1 -kind: Service -metadata: { name: fleet, namespace: fleet } -spec: - selector: { app: fleet } - ports: [{ name: http, port: 3000, targetPort: 3000 }] ---- -# 7 ── Ingress. The idle timeout is what keeps SSE turns from being severed. -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: fleet - namespace: fleet - annotations: - alb.ingress.kubernetes.io/scheme: internet-facing - alb.ingress.kubernetes.io/target-type: ip - alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' - alb.ingress.kubernetes.io/certificate-arn: - alb.ingress.kubernetes.io/ssl-redirect: "443" - alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=1800 - alb.ingress.kubernetes.io/healthcheck-path: / -spec: - ingressClassName: alb - rules: - - host: - http: - paths: - - path: / - pathType: Prefix - backend: { service: { name: fleet, port: { number: 3000 } } } ---- -# 8 ── NetworkPolicy. This governs agent-executed code too: sandbox egress NATs -# through the pod's netns (§7). Needs a policy-enforcing CNI. -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: { name: fleet, namespace: fleet } -spec: - podSelector: { matchLabels: { app: fleet } } - policyTypes: ["Ingress", "Egress"] - ingress: - - from: [{ ipBlock: { cidr: } }] - ports: [{ port: 3000, protocol: TCP }] - egress: - - to: - - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } - podSelector: { matchLabels: { k8s-app: kube-dns } } - ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }] - - to: [{ ipBlock: { cidr: } }] - ports: [{ port: 5432, protocol: TCP }] - - to: - - ipBlock: - cidr: 0.0.0.0/0 - except: [, 169.254.169.254/32] - ports: [{ port: 443, protocol: TCP }] -``` - -Then, in order: create the node group (§6), apply the above, `kubectl exec` in and -run the §10 checklist, add an admin (`fleet admin add `), and log in. - -Not included above, deliberately: the metrics scrape sidecar and its ConfigMap -(optional, §7 — add it once Prometheus is wired up), and the `containers.conf` -`dns_servers` pin, which belongs in the **image** rather than the manifest (§8). - -## What this deployment does not change - -- One governed run loop (`agentcore.Run`) — policy, cost/token ceilings, audit - ([ADR-0001](adr/0001-one-governed-run-loop.md)). -- The mandatory sandbox for every tool call's data plane, with no host-execution - fallback ([ADR-0002](adr/0002-mandatory-rootless-podman-sandbox.md), - [ADR-0036](adr/0036-sandboxed-file-tools-and-host-io-exceptions.md)). -- Host-side MCP credential brokering — secrets never enter a sandbox - ([ADR-0003](adr/0003-host-side-mcp-credential-brokering.md)). -- Client content stays in an out-of-repo bundle - ([ADR-0006](adr/0006-external-client-config-bundle.md)). - -What it *does* change is the outer boundary: on the single box the fleet process -is an unprivileged system user, and here it is a privileged container on a -dedicated node. Treat the node as the trust boundary and size the isolation you -add around it (§2) accordingly. diff --git a/docs/SANDBOX-RUNTIMES.md b/docs/SANDBOX-RUNTIMES.md index f25c1b5a2..e5558be9f 100644 --- a/docs/SANDBOX-RUNTIMES.md +++ b/docs/SANDBOX-RUNTIMES.md @@ -21,6 +21,17 @@ through to `podman run --runtime=`; fleet adds a fail-closed boot preflight and, for Kata, a guest-memory adjustment. See [ADR-0010](adr/0010-microvm-sandbox-runtimes.md) for the design rationale. +> **Sibling knob — `FLEET_SANDBOX_BACKEND`** (manifest `sandbox.backend`, +> same env-wins precedence): while the *runtime* picks the isolation posture +> of a podman sandbox, the *backend* picks WHERE sandboxes run at all — +> `podman` (this page's co-located default) or `kubernetes` (each sandbox an +> ephemeral pod; the split enterprise deployment, +> [ADR-0049](adr/0049-kubernetes-backend-split-control-plane.md)). Everything +> below applies to the podman backend; under the kubernetes backend +> `FLEET_SANDBOX_RUNTIME` is refused (use a cluster RuntimeClass via +> `FLEET_SANDBOX_K8S_RUNTIME_CLASS` instead) — see +> [DEPLOYMENT-KUBERNETES.md](DEPLOYMENT-KUBERNETES.md). + ## The three tiers | | **runc / crun** (default) | **Kata Containers** | **libkrun** | diff --git a/docs/SCANNING.md b/docs/SCANNING.md index 397b3275e..2e52d180f 100644 --- a/docs/SCANNING.md +++ b/docs/SCANNING.md @@ -12,15 +12,32 @@ security queries only) and [`TESTING.md`](TESTING.md) (the rest of the ladder). | `golangci-lint` (incl. `gosec`) | Go lint + Go SAST patterns | ~30s | **blocks** (`ci-gate`) | job log | | `oxlint` + `tsc` | web tier lint + types | ~5s | **blocks** (`ci-gate`) | job log | | **`ruff`** | **Python lint** | **~1s** | **blocks** (`ci-gate`) | job log | +| **`actionlint`** | **workflow YAML: `${{ }}` expressions, contexts, `needs`/`runs-on`/cron, + shellcheck over every `run:`** | **~2s** | **blocks** (`ci-gate`) | job log | +| **`shellcheck`** | **the 18 tracked `*.sh` files (~6.2k lines) — the deploy path** | **~2s** | **blocks** (`ci-gate`) | job log | | `govulncheck` | Go dependency CVEs (called symbols) | ~30s | **blocks** (`ci-gate`) | job log + Security tab | | `grype` | sandbox image CVEs (fixable **CRITICAL + HIGH**, **RPMs only**) | ~1m | **blocks** (`ci-gate`) | job log + Security tab | -| `gitleaks` | secrets, every branch | ~10s | **blocks** (`ci-gate`) | job log | +| `gitleaks` | secrets (on `main` and `dev` — the two branches with a CI lane) | ~10s | **blocks** (`ci-gate`) | job log | | **`npm audit`** | npm dependency CVEs (web + rampart-service) | ~5s | **blocks** (`ci-gate`) | job log | | CodeQL | **interprocedural taint / `security-extended`** | ~2m | **blocks** on an unwaived High-band finding (`ci-gate`/`Dev gate` via workflow_call) | job log + Security tab | | **Semgrep** | **Go/JS/Python SAST + Actions supply chain** | ~40s | **blocks** on any unsuppressed finding (`ci-gate`/`Dev gate` via workflow_call) | job log + artifact | -Two things were added here (**ruff**, **Semgrep**) and one was narrowed -(**CodeQL**, to security queries only). +Four things were added here (**ruff**, **Semgrep**, then **actionlint** and +**shellcheck**) and one was narrowed (**CodeQL**, to security queries only). + +**Why actionlint and shellcheck, given Semgrep already runs `p/github-actions`.** +The overlap is one axis wide: Semgrep's Actions pack is a *security* rule set — +mutable action tags, template injection, `pull_request_target` misuse — and +CodeQL's `actions` language is likewise security-query-only. Neither parses +`${{ }}` expressions and neither shellchecks a `run:` block. That mattered here +concretely: the census below found `run: "$GITHUB_WORKSPACE/scripts/..."` in both +CI lanes passing its path to the shell **unquoted**, because the YAML parser +consumes the quotes — a bug the comment directly above the line showed the author +believed they had avoided. And bash was the only language in the repo with no +linter at all, while being the language `fleet update` and `fleet bootstrap` +actually execute on an operator's box. Both gates started at **zero findings**; +the 5 actionlint and 3 shellcheck items were fixed or annotated with reasons +before the gate went in, because a gate switched on over a backlog is a gate +people learn to scroll past. **Read "blocks" with one caveat, and it is a big one.** Every lane above reaches its branch's aggregate gate job — but a gate job only *blocks a merge* where it @@ -143,9 +160,10 @@ per-**file**, not per-rule, and that is the whole point of preferring it to a `query-filters` exclude: excluding `go/request-forgery` would switch a security-severity 9.1 query off for the entire repository, whereas a register entry waives it in the two files that were read and leaves the query live -everywhere else. An in-source `// codeql[rule-id]` comment waives too (CodeQL -emits it as a `suppressions` array on the result; the comment must sit on its own -line and covers the line below it). Widening the register is a security decision +everywhere else. The register is the **only** waiver route that works here: an +in-source `// codeql[rule-id]` comment does **not** waive with this pipeline — +measured on PR #1249: three forms were tried (the `packs:` input, `packs:` with the additive `+` prefix, and an inline `config:` combining security-extended with codeql/go-queries' `AlertSuppression.ql`) and in every case the uploaded SARIF carried no `suppressions` on the annotated result. A deliberately-waived Security-tab alert is closed by a one-time +human dismissal there. Widening the register is a security decision that appears in the PR diff, and `scripts/check_codeql_register_test.go` fails `make test` on an entry naming a file that does not exist, a missing reason, or a register that `codeql.yml` has stopped referencing. @@ -170,12 +188,30 @@ Getting the extended suite adopted was itself a fix, not a rubber stamp: the one `actions`-language finding was `actions/untrusted-checkout/medium` on `build-sandbox-image.yml`'s `fleet_ref`-fed checkout. Rather than waive it (the `actions` language has no `AlertSuppression.ql`, so there is no in-code waiver -anyway), the workflow now **refuses `refs/pull/*` refs** before checking out — a -fork-PR ref would put fork-controlled code into a workflow that runs the -checked-out build script — and the identical hardening went into +anyway), the workflow now validates `fleet_ref` before checking out — a fork-PR +ref would put fork-controlled code into a workflow that runs the checked-out +build script — and the identical hardening went into `publish-sandbox-image.yml`, the *unflagged* twin that holds `packages: write` and only escaped the (name-heuristic) query because its plumbing was named -differently. Details in [`CODEQL.md`](CODEQL.md). +differently. + +**It is an ALLOW-LIST, and an earlier revision of this document described the +deny-list it replaced.** Worth correcting rather than quietly updating, because +the deny-list (`refs/pull/*|pull/*|-*`) had two holes that a reader +re-implementing "refuse `refs/pull/*`" elsewhere would inherit: + +1. **`GITHUB_OUTPUT` newline injection.** A `workflow_call` string input may + contain newlines, so `fleet_ref: "main\nresolved=refs/pull/1/head"` matched no + deny pattern (it starts `main`), exited 0, and emitted **two** `resolved=` + lines — last-wins handed the attacker the ref. The same primitive forges any + step output. +2. **A bare commit SHA.** "Every ref in this repo is collaborator-written except + `refs/pull/*`" is true of *named refs* and false of reachable *commits*: + GitHub keeps fork-PR commits in the base repo's object store and + `actions/checkout` will fetch a bare SHA happily. + +The shipped form admits only `[A-Za-z0-9._/-]` (so no newline can carry a second +assignment), then refuses `-*`, `*..*`, `*//*`, `refs/pull/*` and a bare hex SHA. Details in [`CODEQL.md`](CODEQL.md). ### Semgrep owns fast multi-language SAST + Actions supply chain (new, blocking) @@ -192,9 +228,12 @@ fixing every real finding and adjudicating every false one. `p/github-actions` found one issue class nothing else in this repo checks — actions referenced by a **mutable tag** (`actions/checkout@v7`) instead of an immutable commit SHA. If a tag moves, attacker-controlled code runs with this -repo's `GITHUB_TOKEN`. There are **13** workflow files, **12** of which reference +repo's `GITHUB_TOKEN`. There are **12** workflow files, **11** of which reference an action at all (`scan-cron-alarm.yml` has no `uses:`), and every one of the -**53** third-party action references across them is now pinned: +**56** third-party action references across them is pinned. (The counts move with +every workflow added or removed — they were 13/12/53 when this was written, and +`scripts/check_action_pins_test.go`, not this paragraph, is what actually holds +the invariant.) ```yaml uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -302,7 +341,7 @@ move with every commit; what is fixed is the format: ### CodeQL findings — go BLOCKING — High band (security-severity >= 7.0), not waived (): none -ACCEPTED — High band, waived in codeql-accepted-findings.json or in-source (): +ACCEPTED — High band, waived in codeql-accepted-findings.json (): [error] sec-sev=9.1 go/request-forgery internal/tools/web_fetch.go: [error] sec-sev=7.5 go/clear-text-logging cmd/fleet/main.go: ... @@ -355,12 +394,37 @@ validated by `tsc`). Every lane in the table reaches the branch's aggregate gate: -- `ci-gate` (the single required status check on `main`) `needs` the lint, test - and build jobs — **and the two scanners**. -- `Dev gate` `needs` the same set on `dev` — but nothing in the `dev` ruleset - requires `Dev gate` to be green, so on that branch it is a red check rather - than a closed gate. That gap is the first item under "Known gaps" and it is - the single most important qualifier on this whole document. +- `ci-gate` (the single required status check on `main`) `needs` **every other + job in `ci.yml`** — the docs-only classifier, gitleaks, the actionlint/shellcheck + workflow+shell lint, the migration DDL lint, the Helm chart lint, Go, ruff, + CodeQL, Semgrep, web, both Playwright lanes and Grype. +- `Dev gate` `needs` the same set that exists on `dev` — but nothing in the `dev` + ruleset requires `Dev gate` to be green, so on that branch it is a red check + rather than a closed gate. That gap is the first item under "Known gaps" and it + is the single most important qualifier on this whole document. + +**That "every other job" is a test, not a habit — and it is the strongest +anti-rot control here, so it should not stay invisible the way it did until +now.** `scripts/check_gate_needs_test.go` parses both workflow files and fails +`make test` if any job is missing from its gate's `needs`. Adding a job and +forgetting to extend `needs` is otherwise a silent one-line regression that +produces a red-but-not-required lane — exactly how the CodeQL Go extraction +break sat unnoticed for weeks. Two sibling tests hold the neighbouring +invariants: `check_action_pins_test.go` (every `uses:` is a 40-hex commit SHA +with an exact version comment) and `check_permissions_test.go` (every workflow +declares a top-level `permissions:` block, so none silently inherits the +repository default). + +Two lanes in `ci-gate`'s list are not in the table above because they are not +scanners: **`helm`** lints and renders the Helm chart so a values/template drift +fails here rather than at an operator's install, and **`migrations`** rejects +dangerous DDL in new or changed migration files. + +One qualifier on "every lane reaches the gate": on a **docs-only** change the +`changes` job skips the heavy lanes, and `ci-gate` passes over those skips. That +is deliberate and narrowly bounded — the classifier is a prose allow-list, and +`ci-gate` refuses a skip whenever the classifier did *not* say docs-only, so a +skip from any other cause fails the gate. The scanners get there because `codeql.yml` and `semgrep.yml` are **reusable workflows** (`on: workflow_call`): `ci.yml` and `dev-ci.yml` each call them as a @@ -429,14 +493,20 @@ Stated rather than left for rediscovery: 2. A `github-actions` bump **is a rewrite of `.github/workflows/*`**: it changes what CI executes. - So the pre-existing shape was: a same-day patch bump to a third-party action, - auto-merged into a branch with no required checks, rewriting the workflows that - are supposed to check it. Three workflow-side mitigations ship alongside this - document — `auto-merge-dependabot.yml` now **excludes the `github_actions` - ecosystem** whatever the bump level, carries an explicit - `branches: [main, dev]` filter so it can never silently start applying to an - unprotected branch, and declares its write scopes on the job rather than the - workflow. Those narrow the blast radius; they do not make `Dev gate` required. + So the shape to avoid is: a same-day patch bump to a third-party action landing + unattended on a branch with no required checks, rewriting the workflows that are + supposed to check it. + + **What removes it is that this repository no longer merges anything + automatically.** `auto-merge-dependabot.yml` was deleted. Its header had argued + the case against itself — it explained that `gh pr merge --auto` holds a merge + only on *required* checks, named `dev` as a branch with none, and then listed + `dev` in its own `branches:` filter, so the mitigation the previous revision of + this document credited was in fact the delivery mechanism. Every dependency + bump, every ecosystem, every bump level now waits for a human. + + That closes the compounding risk but **not** the underlying gap: a hand-merged + PR into `dev` still merges over a red `Dev gate`, because nothing requires it. **The remaining fix is a repo-settings action and belongs to the owner:** add `Dev gate` to the `dev` ruleset's required status checks. Nothing in a workflow @@ -473,8 +543,11 @@ Stated rather than left for rediscovery: CodeQL toolchain break sit red for weeks. Deduped by title; re-failures comment on the same issue. Mechanism differs by necessity: govulncheck and grype carry an in-job step, while CodeQL and Semgrep are watched by - `scan-cron-alarm.yml` (a `workflow_run` watcher) — because a CALLED workflow - may not request permissions its caller did not grant, and the check fires at - plan time before any `if:` can skip the job. Learned by breaking it: an + `scan-cron-alarm.yml` (a `workflow_run` watcher, which also covers the + `E2E canary (real model)` lane — three workflows, not two) — because a CALLED + workflow may not request permissions its caller did not grant, and the check + fires at plan time before any `if:` can skip the job. In `govulncheck` and + `grype` the alarm is a separate JOB rather than a step, holding `issues: write` + on its own so the scan itself does not run beside that scope. Learned by breaking it: an `issues: write` alarm job inside the called workflows startup-failed the entire calling Dev CI run. diff --git a/docs/SKILLS.md b/docs/SKILLS.md index 4c7a26302..20a83afe5 100644 --- a/docs/SKILLS.md +++ b/docs/SKILLS.md @@ -151,6 +151,18 @@ supporting-doc read exception. Treat bundle skills as an interactive-chat capability; a scheduled task that needs one should inline the instructions in its prompt. +**On the kubernetes sandbox backend, inheriting the built-in pack costs +in-sandbox skill files.** `SkillsDir` is then the merged tree under the control +plane's data dir, which sandbox pods do not mount and no sandbox image can +carry (its name is derived from the bundle path, and it is rebuilt at boot) — so +`skills//SKILL.md` resolves for neither the file tools nor bash, and the +roster degrades to name + description. `skills_builtin: false` makes `SkillsDir` +the bundle's own `skills/`, which an operator CAN bake into the sandbox image +and declare with `sandbox.kubernetes.bundle_docs_in_image` +([DEPLOYMENT-KUBERNETES.md](DEPLOYMENT-KUBERNETES.md#bundle-docs-inside-a-sandbox-pod)). +There is no setting that gives you both the built-in pack and working +in-sandbox skill files on that backend. + Manifest knobs (mirroring the MCP directory): ```yaml diff --git a/docs/TESTING.md b/docs/TESTING.md index 83336744d..aec5565ec 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -421,8 +421,18 @@ make ci-go ## Web lint / test / build — CI job `web` -Runs from `web/`. The job is: `npm ci` → `npm run lint` (ESLint) → -`npx vitest run` (unit tests) → `npm run build` (`next build`). +Runs from `web/`. The job is: `npm audit --audit-level=low` (web) → +`npm audit --audit-level=low` (`scripts/rampart-service`) → +`scripts/check-npm-overrides.sh` → `npm ci` → `npm run lint` (**oxlint**, not +ESLint — see the TypeScript 7 section above, which explains why ESLint was +replaced) → `npm run typecheck` (`tsc --noEmit`) → `npx vitest run` (unit tests) +→ `npm run build` (`next build`). + +The two audits and the override canary run **before** `npm ci`, deliberately: +they are lockfile-only, so they cost seconds and fail before the expensive +install. The explicit typecheck is not redundant with `next build` — the build +type-checks too, but it runs last, so without this step a one-line type error +surfaces minutes in. ```sh cd web diff --git a/docs/TIMERS.md b/docs/TIMERS.md index 854f3ff05..e494dccac 100644 --- a/docs/TIMERS.md +++ b/docs/TIMERS.md @@ -66,7 +66,9 @@ the command does not pretend: it explains that the equivalent jobs belong to the platform's scheduler — daily `fleet backup --db=all --prune` and daily `fleet cleanup` (cron, a Kubernetes CronJob) — and exits non-zero. `fleet update`'s offer and doctor's advisories are likewise skipped entirely where -there is no systemd. +there is no systemd. For the first-class Kubernetes deployment, the CronJob +equivalents are part of the production checklist in +[`DEPLOYMENT-KUBERNETES.md`](DEPLOYMENT-KUBERNETES.md). ## Honest scope / deliberately not done diff --git a/docs/adr/0004-single-box-vm-native-deployment.md b/docs/adr/0004-single-box-vm-native-deployment.md index e4e5e5623..2cac65408 100644 --- a/docs/adr/0004-single-box-vm-native-deployment.md +++ b/docs/adr/0004-single-box-vm-native-deployment.md @@ -1,6 +1,10 @@ # ADR-0004: Single-box, VM-native deployment (no Kubernetes) -- **Status:** Accepted +- **Status:** Accepted; amended by [ADR-0049](0049-kubernetes-backend-split-control-plane.md) + (the single-box default install stands; the "no k8s manifest, Helm chart, or + operator in the tree" enforcement clause and the cluster-work-is-out-of-scope + consequence are superseded — `deploy/helm/fleet` and the kubernetes sandbox + backend are the sanctioned enterprise path) - **Date:** 2026-06-28 (documents a decision that predates this record) - **Deciders:** fleet maintainers diff --git a/docs/adr/0048-codeql-severity-gating.md b/docs/adr/0048-codeql-severity-gating.md index ea16bd9e5..e29fd0fe5 100644 --- a/docs/adr/0048-codeql-severity-gating.md +++ b/docs/adr/0048-codeql-severity-gating.md @@ -88,17 +88,26 @@ block both. ## Decision -**CodeQL blocks on a finding that is (a) at SARIF level `error`/`warning`, or has -security-severity >= 7.0, and (b) is not waived.** Findings below that band are +**CodeQL blocks on a finding that is (a) in the High band and (b) is not +waived.** A finding is in the High band when its rule publishes a +`security-severity >= 7.0`; **only for a rule that publishes no +security-severity** does the band fall back to SARIF level `error`/`warning`. +The fallback is not an OR: level is deliberately *not* consulted for a rule that +does publish a security-severity, because nearly every CodeQL security query is +`@problem.severity error` — `go/log-injection` is `error` at 6.1 — so an OR +would block on all 23 log-injection findings and reproduce the deadlock this +ADR exists to undo. Findings below that band are printed and uploaded to the Security tab as advisory. Waivers come from two -places: +place: 1. `.github/codeql-accepted-findings.json` — a register of accepted `(rule, file)` pairs, each with a mandatory written reason. -2. An in-source `// codeql[rule-id]` comment, which CodeQL emits as a - `suppressions` array on the result. (Both `go` and `javascript` ship an - `AlertSuppression.ql`; the comment must sit on its own line and covers the - line immediately below it.) + +An in-source `// codeql[rule-id]` comment — the mechanism CodeQL documents, and +which an earlier revision of this ADR listed as a second route — does **not** +work with this pipeline. It was measured on PR #1249: three forms were tried (the `packs:` input, `packs:` with the additive `+` prefix, and an inline `config:` combining security-extended with codeql/go-queries' `AlertSuppression.ql`) and in every case the uploaded SARIF carried no `suppressions` on the annotated result. The register is therefore the sole +waiver route for the gate; a deliberately-waived alert is closed in the Security +tab by a one-time human dismissal. The register is **per-file, not per-rule**, and that is the whole point of preferring it to a `query-filters` exclude. A `query-filters: exclude: {id: diff --git a/docs/adr/0049-kubernetes-backend-split-control-plane.md b/docs/adr/0049-kubernetes-backend-split-control-plane.md new file mode 100644 index 000000000..2c44597b1 --- /dev/null +++ b/docs/adr/0049-kubernetes-backend-split-control-plane.md @@ -0,0 +1,142 @@ +# ADR-0049: Kubernetes as a first-class deployment — split control plane, pluggable sandbox backend + +- **Status:** Accepted +- **Date:** 2026-08-22 +- **Deciders:** fleet maintainers +- **Amends:** [ADR-0004](0004-single-box-vm-native-deployment.md) (supersedes + its "no k8s manifest, Helm chart, or operator in the tree" enforcement + clause and its cluster-work-is-out-of-scope consequence; the single-box + default install it decides **stands**) + +## Context + +ADR-0004 made fleet VM-native on one box: systemd, Caddy, rootless Podman +co-located with the process. That remains the right default install for +individuals and small teams. But Kubernetes-native organizations were left +with no supported path (issue #989): `deploy/` shipped only systemd units, the +sandbox was **always** co-located with the fleet process, and the only k8s +document was a hand-verified EKS recipe (`docs/EKS-DEPLOYMENT.md`, since +removed) that ran rootless Podman inside a privileged pod — an operator +workaround, not a product. + +The owner decision on #989: ship the **enterprise path in one pass** — the +fleet control plane separate from execution runners, with a pluggable sandbox +backend — and do **not** build a co-located "fleet pod + Podman on a +privileged node" packaging track as a stepping stone. + +## Decision + +1. **The sandbox backend is pluggable, selected by one knob.** The internal + per-sandbox interface (`internal/sandbox`'s `impl`) gains a third + implementation: alongside the rootless-Podman backend (`containerImpl`) and + the test-only host executor, `k8sImpl` runs each sandbox as an **ephemeral + Kubernetes Pod** exec'd over the apiserver. `FLEET_SANDBOX_BACKEND` + (overriding the bundle manifest's `sandbox.backend`) selects + `podman` (default) or `kubernetes`, mirroring `sandbox.runtime`'s + precedence exactly (ADR-0010). An unrecognized value refuses to boot. +2. **The kubernetes backend fails closed at boot.** Selecting it triggers a + preflight — apiserver reachable, RBAC verbs present (pods CRUD + + `pods/exec`), the shared workspace claim exists, the sealed-egress + NetworkPolicy object exists, the RuntimeClass exists when configured — and + any failure aborts boot. There is no fallback to podman or host execution + (the ADR-0010 no-degrade posture, applied to backends). +3. **The workspace is a shared ReadWriteMany claim, mounted same-path.** The + control plane and every sandbox pod mount the same PVC at the same absolute + path, preserving the invariant (ADR-0036 territory) that an absolute + workspace path means the same thing to the process, host-side brokers, and + sandboxed bash/python. +4. **Sealing is expressed as labels + a required NetworkPolicy.** Sandbox pods + carry `fleet.elcanotek.com/egress=none|open`; the Helm chart ships a + deny-all policy selecting `none`. fleet verifies the policy **object** + exists; enforcement is the CNI's, and the docs say so plainly rather than + implying a seal fleet cannot provide (the podman `--network=none` namespace + seal has no per-pod apiserver equivalent). +5. **Enterprise packaging is one Helm chart** (`deploy/helm/fleet`): + single-replica control-plane Deployment (strategy Recreate, no replica + knob), the runner RBAC, workspace storage, the NetworkPolicies, optional + in-cluster Postgres / web / Ingress. No operator, no CRDs in v1. +6. **The API client is hand-rolled, not client-go.** The backend needs five + verbs plus WebSocket exec streaming; client-go would add dozens of modules + to a tree gated by govulncheck and image CVE scans. `internal/sandbox` + speaks plain REST via net/http and `v4.channel.k8s.io` exec framing via + gorilla/websocket (already a dependency). If the backend ever needs + watches/informers or exotic auth, revisit client-go rather than growing the + hand-rolled client. Kubeconfig support is deliberately narrow — token, + token-file, client-cert; exec plugins and `insecure-skip-tls-verify` are + refused. + +## What does not change + +- **The single-box podman install stays the default** and its story is + untouched: bootstrap, systemd units, Caddy, `fleet timers`. ADR-0004's + decision section stands for that install. +- **The sandbox is still mandatory** (ADR-0002): the kubernetes backend is a + different *where*, not a weaker *whether*. Pods run read-only-rootfs, + non-root, all capabilities dropped, seccomp RuntimeDefault (or an + operator-installed Localhost profile), `automountServiceAccountToken=false`. +- **Credentials stay in the control plane** (ADR-0003): sandbox pods get no + env, no secrets, no service-account token — only the workspace mount. The + MCP broker never moves. +- **One governed loop** (ADR-0001): the backend swap is entirely below + `agentcore`; no second governance path exists. +- **Single-owner control plane:** one fleet replica, ever. Horizontal scale of + *work* is more sandbox pods / bigger node pools, not more fleet processes. +- **Poison-and-retire (#796)** carries over: a cancelled or timed-out call + deletes the whole pod with zero grace — destroying its PID namespace and + every straggler — and retires the sandbox. + +## Explicit non-goals (v1) + +- Co-located "fleet + Podman in a privileged pod" as the supported enterprise + story. The EKS recipe that documented it is **removed** rather than kept as + a parallel path — an unmaintained privileged-pod recipe beside a first-class + unprivileged one would imply support it does not have. +- Multi-replica / active-active fleet; a Kubernetes operator or CRDs. +- The **allowlisted** egress mode under the kubernetes backend: the host-side + egress proxy is unreachable from pods, so the mode is refused at boot + (fail-closed) instead of silently granting open egress. Cluster-side egress + shaping via NetworkPolicy is the replacement. +- Per-pod pids limits (not expressible in a Pod spec), the bundled seccomp + JSON (nodes take a Localhost profile instead), and `podman stats` resource + telemetry (#263) — each recorded as an honest deviation in + `docs/DEPLOYMENT-KUBERNETES.md`. +- Same-path supporting-doc bind mounts: a pod has no host filesystem to bind + from. fleet does not synthesize them (no ConfigMap projection, no + control-plane push into the workspace claim — both would put bundle content + on a writable, agent-reachable surface). Instead the sandbox IMAGE may carry + the bundle's doc dirs at the same absolute paths, and + `sandbox.kubernetes.bundle_docs_in_image` declares that, which keeps those + roots' **read-only** fileop anchors valid inside a pod. A declaration, not a + probe: fleet cannot inspect an image, so it is trusted the way + `sandbox.image` and `runtime_class` are — and it can only re-admit reads of + operator-configured paths, executed inside the sandbox, so a wrong + declaration degrades to not-found rather than widening any boundary. + +## Consequences + +- Kubernetes-native organizations get a supported, preflighted, CI-linted + path: `helm install` + two images they build. The EKS privileged-pod recipe + is retired in its favor. +- `deploy/` now contains cluster artifacts, so ADR-0004's enforcement clause + ("no k8s manifest, Helm chart, or operator in the tree") is superseded; its + index row and status note point here. +- A second execution substrate must be kept honest: the backend seam + (`sandbox.Backend`-shaped `impl`) is now a contract two production backends + implement, and behavior-affecting changes must land in both or say why not. +- The chart is linted and template-rendered in CI (`helm` job in ci.yml / + dev-ci.yml) but not exercised against a live cluster there; the kind + walkthrough in `docs/DEPLOYMENT-KUBERNETES.md` is the verified end-to-end + path. + +## Alternatives considered + +- **client-go.** Rejected for dependency weight against a five-verb surface; + recorded above as the explicit revisit trigger. +- **Chart-only first, backend later.** Rejected by the issue itself: a chart + that still requires privileged Podman-in-pod would enshrine the workaround. +- **A Kubernetes operator/CRD.** Unnecessary for v1 — Helm + RBAC covers + install; fleet's own scheduler owns runtime orchestration. +- **Running sandbox pods in a dedicated namespace by default.** The RBAC story + is marginally nicer, but a PVC cannot be mounted across namespaces, so the + default topology shares the release namespace; a split namespace remains + possible with static same-export PVs and is documented. diff --git a/docs/adr/README.md b/docs/adr/README.md index 18bec27c1..1e597386c 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -29,7 +29,7 @@ reviewable, and citable. Each record names the file or test that enforces it. | [0001](0001-one-governed-run-loop.md) | One governed agent run loop | Accepted | | [0002](0002-mandatory-rootless-podman-sandbox.md) | Mandatory rootless-Podman sandbox; host executor never ships | Accepted | | [0003](0003-host-side-mcp-credential-brokering.md) | Host-side MCP credential brokering | Accepted | -| [0004](0004-single-box-vm-native-deployment.md) | Single-box, VM-native deployment (no Kubernetes) | Accepted | +| [0004](0004-single-box-vm-native-deployment.md) | Single-box, VM-native deployment (no Kubernetes) | Accepted; amended by ADR-0049 | | [0005](0005-separate-chat-and-sched-databases.md) | Separate Postgres databases for chat and sched | Accepted | | [0006](0006-external-client-config-bundle.md) | Client content lives in an external config bundle | Accepted | | [0007](0007-governed-sub-agents.md) | Governed sub-agents spawn only through the one run loop | Accepted | @@ -57,3 +57,4 @@ reviewable, and citable. Each record names the file or test that enforces it. | [0045](0045-remove-node-name-scopes.md) | Remove node-name scopes; a principal's authority is its permission set | Accepted | | [0046](0046-remove-per-key-spending-caps.md) | Remove per-API-key spending caps; rolling budgets are the one spend gate | Accepted | | [0047](0047-self-serve-team-membership.md) | Self-serve team membership — create/leave is yours, joining is granted | Accepted | +| [0049](0049-kubernetes-backend-split-control-plane.md) | Kubernetes as a first-class deployment — split control plane, pluggable sandbox backend | Accepted | diff --git a/go.mod b/go.mod index 30af55f3a..cbb5a9902 100644 --- a/go.mod +++ b/go.mod @@ -15,6 +15,7 @@ require ( github.com/goccy/go-yaml v1.19.2 github.com/golang-migrate/migrate/v4 v4.19.1 github.com/google/uuid v1.6.0 + github.com/gorilla/websocket v1.5.3 github.com/itchyny/gojq v0.12.19 github.com/jackc/pgx/v5 v5.10.0 github.com/robfig/cron/v3 v3.0.1 @@ -84,7 +85,6 @@ require ( github.com/googleapis/enterprise-certificate-proxy v0.3.20 // indirect github.com/googleapis/gax-go/v2 v2.23.0 // indirect github.com/gorilla/css v1.0.1 // indirect - github.com/gorilla/websocket v1.5.3 // indirect github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect github.com/invopop/jsonschema v0.14.0 // indirect github.com/itchyny/timefmt-go v0.1.8 // indirect diff --git a/internal/agent/k8s_doc_mounts_test.go b/internal/agent/k8s_doc_mounts_test.go new file mode 100644 index 000000000..4f7ce4ce0 --- /dev/null +++ b/internal/agent/k8s_doc_mounts_test.go @@ -0,0 +1,76 @@ +package agent + +import ( + "path/filepath" + "testing" +) + +// The kubernetes backend keeps or drops each supporting-doc root's fileop +// anchor by one rule set (k8sDocMounts). These tests pin it, because the +// consequence of getting it wrong is invisible until an agent tries to read a +// protocol: too permissive and the anchor trusts a path no pod has, too strict +// and view_file refuses a file the sandbox image really does carry. +func TestK8sDocMountsDropsEverythingWithoutTheDeclaration(t *testing.T) { + bundle := []string{"/opt/fleet/client/personas", "/opt/fleet/client/protocols", "/opt/fleet/client/system_prompts", "/opt/fleet/client/skills"} + mounts := append(append([]string{}, bundle...), "/var/lib/fleet/uploads") + + kept, dropped := k8sDocMounts(mounts, bundle, false) + if len(kept) != 0 { + t.Errorf("kept = %v; a pod mounts only the workspace claim, so nothing may keep its anchor", kept) + } + if len(dropped) != len(mounts) { + t.Errorf("dropped = %v; want all %d mounts", dropped, len(mounts)) + } +} + +func TestK8sDocMountsKeepsOnlyBundleDocsWithTheDeclaration(t *testing.T) { + bundle := []string{"/opt/fleet/client/personas", "/opt/fleet/client/protocols", "/opt/fleet/client/system_prompts", "/opt/fleet/client/skills"} + uploads := "/var/lib/fleet/uploads" + mounts := append(append([]string{}, bundle...), uploads) + + kept, dropped := k8sDocMounts(mounts, bundle, true) + if len(kept) != len(bundle) { + t.Fatalf("kept = %v; want the %d bundle doc dirs", kept, len(bundle)) + } + for i, want := range bundle { + if kept[i] != want { + t.Errorf("kept[%d] = %q, want %q (order preserved)", i, kept[i], want) + } + } + // The uploads root is control-plane state; no sandbox image contains it, + // and the declaration says nothing about it. + if len(dropped) != 1 || dropped[0] != uploads { + t.Errorf("dropped = %v; want only %q", dropped, uploads) + } +} + +func TestK8sDocMountsNeverKeepsAMaterializedSkillsTree(t *testing.T) { + // The merged built-in + bundle skills tree lives under the control plane's + // data dir with a hash-derived name — a sandbox image cannot carry it, so + // the declaration must not extend to it even though it IS the bundle's + // resolved skills dir. + merged := filepath.Join("/var/lib/fleet", "skills-merged", "f693617985b1") + bundle := []string{"/opt/fleet/client/protocols", merged} + mounts := bundle + + kept, dropped := k8sDocMounts(mounts, bundle, true) + if len(kept) != 1 || kept[0] != "/opt/fleet/client/protocols" { + t.Errorf("kept = %v; want only the bundle-path protocols dir", kept) + } + if len(dropped) != 1 || dropped[0] != merged { + t.Errorf("dropped = %v; want the merged skills tree %q", dropped, merged) + } +} + +func TestK8sDocMountsIgnoresUnlistedAndEmptyPaths(t *testing.T) { + bundle := []string{"/opt/fleet/client/protocols/"} // trailing slash, same dir + mounts := []string{"", "/opt/fleet/client/protocols", "/somewhere/else"} + + kept, dropped := k8sDocMounts(mounts, bundle, true) + if len(kept) != 1 || kept[0] != "/opt/fleet/client/protocols" { + t.Errorf("kept = %v; want the protocols dir matched after path cleaning", kept) + } + if len(dropped) != 1 || dropped[0] != "/somewhere/else" { + t.Errorf("dropped = %v; want only the path that is not a bundle doc dir", dropped) + } +} diff --git a/internal/agent/manager.go b/internal/agent/manager.go index 278656463..6f64dac5b 100644 --- a/internal/agent/manager.go +++ b/internal/agent/manager.go @@ -16,6 +16,7 @@ import ( "github.com/ElcanoTek/fleet/internal/admission" "github.com/ElcanoTek/fleet/internal/agentcore" + "github.com/ElcanoTek/fleet/internal/clientconfig" "github.com/ElcanoTek/fleet/internal/config" "github.com/ElcanoTek/fleet/internal/creds" "github.com/ElcanoTek/fleet/internal/mcp" @@ -581,6 +582,14 @@ func buildSandboxPool(cfg *config.Config, personasDir, protocolsDir, systemPromp BridgeDir: filepath.Join(filepath.Dir(workspaceRoot), "data", "sandbox-bridge"), ReadOnlyMounts: absSupportingDocs(personasDir, protocolsDir, systemPromptsDir, skillsDir, uploadsRoot), } + // Kubernetes backend (#989): sandboxes are ephemeral pods in a cluster + // instead of co-located podman containers. All podman-specific boot work + // below (bridge-file prune, OCI-runtime preflight, egress proxy) is + // replaced by the backend's own fail-closed cluster preflight. + if cfg.SandboxBackend == sandbox.BackendKubernetes { + return buildKubernetesSandboxPool(cfg, poolCfg, sandboxRuntime, + absSupportingDocs(personasDir, protocolsDir, systemPromptsDir, skillsDir)) + } // Reclaim bridge-script/seccomp temp files orphaned by a PRIOR crash: only // the graceful close path removes them, so without this sweep every // non-graceful exit leaks them into BridgeDir permanently. Age-bounded and @@ -646,6 +655,156 @@ func buildSandboxPool(cfg *config.Config, personasDir, protocolsDir, systemPromp return sandbox.NewPool(poolCfg), nil } +// buildKubernetesSandboxPool finishes pool construction for the kubernetes +// backend (#989): it refuses podman-only knobs that would otherwise be +// silently ignored (a configured-but-inert security knob is the failure mode +// ADR-0010's no-degrade rule exists for), builds the backend handle, and runs +// the fail-closed cluster preflight before the warm pool spawns its first pod. +// +// bundleDocDirs are the bundle's own supporting-doc roots (personas, +// protocols, system_prompts, skills) — the subset of +// poolCfg.Container.ReadOnlyMounts a sandbox IMAGE could plausibly carry, and +// therefore the only ones bundle_docs_in_image can vouch for. +func buildKubernetesSandboxPool(cfg *config.Config, poolCfg sandbox.PoolConfig, sandboxRuntime string, bundleDocDirs []string) (*sandbox.Pool, error) { + if sandboxRuntime != "" { + return nil, fmt.Errorf( + "FLEET_SANDBOX_RUNTIME=%q is a podman OCI-runtime knob and has no effect under FLEET_SANDBOX_BACKEND=kubernetes; "+ + "select hypervisor isolation with FLEET_SANDBOX_K8S_RUNTIME_CLASS (a cluster RuntimeClass, e.g. kata) instead (fail-closed)", sandboxRuntime) + } + if v := strings.TrimSpace(os.Getenv("FLEET_SANDBOX_SECCOMP_PROFILE")); v != "" { + return nil, fmt.Errorf( + "FLEET_SANDBOX_SECCOMP_PROFILE=%q is a podman knob and has no effect under FLEET_SANDBOX_BACKEND=kubernetes; "+ + "install the profile on the sandbox nodes and set FLEET_SANDBOX_K8S_SECCOMP_PROFILE (a kubelet-relative Localhost profile) instead (fail-closed)", v) + } + if cfg.DefaultNetworkMode == sandbox.NetworkModeAllowlisted { + return nil, fmt.Errorf( + "FLEET_DEFAULT_NETWORK_MODE=allowlisted is not supported under FLEET_SANDBOX_BACKEND=kubernetes: the host-side egress proxy " + + "is unreachable from sandbox pods. Use lockdown (sealed by the deny-all NetworkPolicy) or open, and shape egress with cluster NetworkPolicies (fail-closed)") + } + // Supporting-doc mounts are same-path HOST bind mounts, and a pod has no + // host filesystem to bind them from — so by default they are dropped, and + // the fileop anchor then refuses those roots rather than trusting paths + // nothing mounted. A sandbox IMAGE can still carry the bundle's doc dirs + // at the same absolute paths (that is how bash/run_python keep resolving + // `protocols/…` through the workspace symlinks); an operator who built + // such an image declares it with bundle_docs_in_image, and the anchors for + // those roots stay valid so the FILE TOOLS work too. + // + // The declaration cannot be probed — fleet does not inspect image + // contents — but it cannot widen anything either: it only re-admits + // read-only anchors for operator-configured bundle paths, and the reads + // still execute inside the sandbox. A wrong declaration surfaces as a + // not-found read, which is the podman missing-dir behavior. + docsInImage, err := sandbox.ParseK8sBundleDocsInImage(cfg.SandboxK8sBundleDocsInImage) + if err != nil { + return nil, fmt.Errorf("FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE / sandbox.kubernetes.bundle_docs_in_image: %w", err) + } + kept, dropped := k8sDocMounts(poolCfg.Container.ReadOnlyMounts, bundleDocDirs, docsInImage) + poolCfg.Container.ReadOnlyMounts = kept + if len(kept) > 0 { + log.Printf("sandbox: kubernetes backend — bundle_docs_in_image declared: keeping fileop read anchors for %d bundle doc root(s) %v; the SANDBOX IMAGE must carry them at these exact paths or reads fail not-found", len(kept), kept) + } + for _, d := range dropped { + switch { + case clientconfig.IsMaterializedSkillsDir(d): + log.Printf("sandbox: kubernetes backend — skills dir %q is the merged built-in+bundle tree under the control plane's data dir, which no sandbox image can carry; in-sandbox skill reads will not resolve. Set skills_builtin: false in the bundle manifest to make skills/ the bundle's own (bake-able) dir", d) + case docsInImage: + log.Printf("sandbox: kubernetes backend — %q is not a bundle doc dir, so bundle_docs_in_image does not vouch for it; in-sandbox reads there will not resolve", d) + } + } + if !docsInImage && len(dropped) > 0 { + log.Printf("sandbox: kubernetes backend — supporting-doc bind mounts do not apply (pods mount only the workspace claim); in-sandbox reads of %d host dir(s) will not resolve. If your sandbox image carries the bundle's doc dirs at the same paths, set sandbox.kubernetes.bundle_docs_in_image (FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE=true)", len(dropped)) + } + poolCfg.Container.Runtime = "" + + // Scheduling knobs fail closed on a malformed value: a typo'd selector + // must not silently schedule sandboxes onto the wrong (untainted, + // unlabeled) nodes. + nodeSelector, err := sandbox.ParseK8sNodeSelector(cfg.SandboxK8sNodeSelector) + if err != nil { + return nil, fmt.Errorf("FLEET_SANDBOX_K8S_NODE_SELECTOR / sandbox.kubernetes.node_selector: %w", err) + } + tolerations, err := sandbox.ParseK8sTolerations(cfg.SandboxK8sTolerations) + if err != nil { + return nil, fmt.Errorf("FLEET_SANDBOX_K8S_TOLERATIONS / sandbox.kubernetes.tolerations: %w", err) + } + backend, err := sandbox.NewKubernetesBackend(sandbox.KubernetesConfig{ + Namespace: cfg.SandboxK8sNamespace, + WorkspaceClaim: cfg.SandboxK8sWorkspaceClaim, + ServiceAccount: cfg.SandboxK8sServiceAccount, + ImagePullSecret: cfg.SandboxK8sImagePullSecret, + RuntimeClassName: cfg.SandboxK8sRuntimeClass, + SeccompLocalhostProfile: cfg.SandboxK8sSeccompProfile, + KubeconfigPath: cfg.SandboxK8sKubeconfig, + NetworkPolicyName: cfg.SandboxK8sNetworkPolicy, + NodeSelector: nodeSelector, + Tolerations: tolerations, + }) + if err != nil { + return nil, err + } + // Fail closed BEFORE the warm pool spawns its first pod: a cluster that + // cannot run sandboxes (unreachable apiserver, missing RBAC, absent + // workspace claim or sealed-egress policy) must abort boot, never + // silently fall back to podman or host execution. + if err := backend.Preflight(context.Background()); err != nil { + return nil, fmt.Errorf("kubernetes sandbox preflight failed (fail-closed): %w", err) + } + poolCfg.Mode = sandbox.ModeKubernetes + poolCfg.KubernetesBackend = backend + + poolCfg.DefaultNetworkMode = cfg.DefaultNetworkMode + poolCfg.DefaultEgressAllowlist = nil + log.Printf("sandbox: kubernetes backend — image=%s, pool=%d, workspace=%s, namespace=%s, runtime_class=%s", + poolCfg.Container.Image, poolCfg.Size, poolCfg.Container.WorkspaceHostDir, backend.Namespace(), defaultIfEmpty(cfg.SandboxK8sRuntimeClass, "cluster default")) + if poolCfg.PersistentREPL { + log.Printf("sandbox: run_python REPL mode=persistent — one kernel per conversation survives across turns (idle TTL %s, max %d sessions)", + poolCfg.PersistentIdleTTL, cfg.PythonREPLMaxSessions) + } else { + log.Printf("sandbox: run_python REPL mode=per-turn — kernel is fresh each turn (the default)") + } + if cfg.DefaultNetworkMode == sandbox.NetworkModeLockdown { + log.Printf("sandbox: network mode=lockdown — every sandbox pod is labeled %s=none for the deny-all NetworkPolicy (enforcement is the cluster CNI's job — see docs/DEPLOYMENT-KUBERNETES.md)", "fleet.elcanotek.com/egress") + } + return sandbox.NewPool(poolCfg), nil +} + +// k8sDocMounts splits the supporting-doc mount list into the roots whose +// fileop anchors survive under the kubernetes backend and the roots that are +// dropped, given the operator's bundle_docs_in_image declaration. +// +// Pure and total so the policy is pinned by tests rather than read out of the +// boot log. Three rules, in order: +// +// - Without the declaration, nothing survives: a pod mounts only the +// workspace claim, so every host path is a path the anchor must not trust. +// - With it, only the BUNDLE's own doc dirs survive. Everything else in the +// list (the uploads root) lives in control-plane state a sandbox image +// cannot contain, and the declaration says nothing about it. +// - A materialized (merged built-in + bundle) skills tree never survives: +// it lives under the data dir with a path derived from the bundle path, so +// no image can carry it. See clientconfig.IsMaterializedSkillsDir. +func k8sDocMounts(mounts, bundleDocDirs []string, docsInImage bool) (kept, dropped []string) { + bundle := make(map[string]bool, len(bundleDocDirs)) + for _, d := range bundleDocDirs { + if d != "" { + bundle[filepath.Clean(d)] = true + } + } + for _, m := range mounts { + if m == "" { + continue + } + clean := filepath.Clean(m) + if docsInImage && bundle[clean] && !clientconfig.IsMaterializedSkillsDir(clean) { + kept = append(kept, m) + continue + } + dropped = append(dropped, m) + } + return kept, dropped +} + // absSupportingDocs absolutizes the persona/protocol/skill/system-prompt dirs // (plus the uploads root) and drops empties so they can be passed as // ContainerConfig.ReadOnlyMounts. The container backend bind-mounts each at the diff --git a/internal/clientconfig/builtin_skills.go b/internal/clientconfig/builtin_skills.go index f0d20328b..bac26bf12 100644 --- a/internal/clientconfig/builtin_skills.go +++ b/internal/clientconfig/builtin_skills.go @@ -98,6 +98,27 @@ func ensureTrustedDir(path string) error { return verifyExistingDir(path) } +// IsMaterializedSkillsDir reports whether dir is a merged tree this package +// materialized (`/skills-merged/`) rather than a bundle's +// own `skills/`. It exists for one caller: the kubernetes sandbox backend, +// where a supporting-doc dir is only readable in a sandbox if the sandbox +// IMAGE carries it at the same absolute path — and a merged tree lives under +// the control plane's data dir, which no sandbox image can plausibly reproduce +// (its hash is derived from the bundle path, and the tree is rebuilt at boot). +// So a bundle inheriting the built-in pack can never serve in-sandbox skill +// reads on that backend; the caller drops the mount and says so, and the fix +// is the bundle's `skills_builtin: false`. +// +// Shape-based on purpose: the layout is this package's own convention, so the +// check belongs here, next to the code that builds the path. +func IsMaterializedSkillsDir(dir string) bool { + dir = strings.TrimSpace(dir) + if dir == "" { + return false + } + return filepath.Base(filepath.Dir(filepath.Clean(dir))) == mergedSkillsDirName +} + // verifyExistingDir is the ssh-style ownership/mode check: Lstat (so a // symlink is not followed), must be a directory we own, must not be // group- or world-writable. diff --git a/internal/clientconfig/builtin_skills_test.go b/internal/clientconfig/builtin_skills_test.go index f5af259e5..41b86e964 100644 --- a/internal/clientconfig/builtin_skills_test.go +++ b/internal/clientconfig/builtin_skills_test.go @@ -247,3 +247,37 @@ func TestMaterializeMergedSkills_DoesNotAdoptUntrustedPath(t *testing.T) { t.Fatalf("adopted %q instead of falling back to the bundle dir %q", got, bundle) } } + +// IsMaterializedSkillsDir tells a merged tree apart from a bundle's own +// skills/ — the distinction the kubernetes sandbox backend needs, because only +// a bundle-path skills dir can be baked into a sandbox image. Pinned against +// the real materialized path so the shape check cannot drift from the builder. +func TestIsMaterializedSkillsDir(t *testing.T) { + data := t.TempDir() + t.Setenv("FLEET_DATA_DIR", data) + bundleSkills := filepath.Join(t.TempDir(), "skills") + merged, err := materializeMergedSkills(bundleSkills, true, nil) + if err != nil { + t.Fatalf("materialize: %v", err) + } + if !IsMaterializedSkillsDir(merged) { + t.Errorf("IsMaterializedSkillsDir(%q) = false; the path materializeMergedSkills built must be recognized", merged) + } + if IsMaterializedSkillsDir(bundleSkills) { + t.Errorf("IsMaterializedSkillsDir(%q) = true; a bundle's own skills/ is bake-able", bundleSkills) + } + // Opting out of the built-in pack returns the bundle dir itself, which is + // exactly the case an operator reaches for on the kubernetes backend. + own, err := materializeMergedSkills(bundleSkills, false, nil) + if err != nil { + t.Fatalf("materialize (builtins off): %v", err) + } + if IsMaterializedSkillsDir(own) { + t.Errorf("with skills_builtin: false the resolved dir %q must not read as materialized", own) + } + for _, in := range []string{"", " ", "/opt/fleet/client/skills", "/var/lib/fleet/skills-merged"} { + if IsMaterializedSkillsDir(in) { + t.Errorf("IsMaterializedSkillsDir(%q) = true; want false", in) + } + } +} diff --git a/internal/clientconfig/clientconfig.go b/internal/clientconfig/clientconfig.go index 32652e65c..f0c46d255 100644 --- a/internal/clientconfig/clientconfig.go +++ b/internal/clientconfig/clientconfig.go @@ -416,6 +416,76 @@ type Sandbox struct { // manifest sandbox.network_allowlist. Empty in allowlisted mode = deny all // egress (best-effort — see ADR-0012). NetworkAllowlist []string + + // Backend selects WHERE sandboxes run (#989 / ADR-0049): "" or "podman" + // for the co-located rootless-Podman backend (the single-box default), or + // "kubernetes" for ephemeral pods in a cluster (the split + // control-plane/runner enterprise path). Stored VERBATIM; the consuming + // layer (cmd/fleet) validates fail-closed and an explicit + // FLEET_SANDBOX_BACKEND env var wins, mirroring sandbox.runtime. + Backend string + + // Kubernetes carries the kubernetes-backend settings (manifest + // sandbox.kubernetes). Meaningful only when the resolved backend is + // "kubernetes"; each FLEET_SANDBOX_K8S_* env var overrides its field. + Kubernetes KubernetesSandbox +} + +// KubernetesSandbox is the resolved sandbox.kubernetes block: where sandbox +// pods run and what they mount. All fields are trusted operator config, same +// authority tier as sandbox.image / sandbox.runtime. +type KubernetesSandbox struct { + // Namespace for sandbox pods (default applied at consume time: + // "fleet-sandboxes" — kept separate from the control plane's namespace so + // RBAC and the deny-all NetworkPolicy stay narrowly scoped). + Namespace string `yaml:"namespace"` + // WorkspaceClaim is the ReadWriteMany PVC (in Namespace) holding the + // workspace root, mounted into every sandbox pod at the same absolute + // path the control plane mounts it. Required for the kubernetes backend. + WorkspaceClaim string `yaml:"workspace_claim"` + // ServiceAccount stamped on sandbox pods (identity only — the token is + // never mounted). + ServiceAccount string `yaml:"service_account"` + // ImagePullSecret for private sandbox-image registries. + ImagePullSecret string `yaml:"image_pull_secret"` + // RuntimeClass selects hypervisor isolation (e.g. kata) — the kubernetes + // counterpart of sandbox.runtime, preflighted fail-closed (ADR-0010). + RuntimeClass string `yaml:"runtime_class"` + // SeccompProfile is a node-local Localhost seccomp profile path (relative + // to the kubelet seccomp root); empty = RuntimeDefault. + SeccompProfile string `yaml:"seccomp_profile"` + // Kubeconfig selects out-of-cluster auth; empty = in-cluster. + Kubeconfig string `yaml:"kubeconfig"` + // NetworkPolicy is the deny-all NetworkPolicy name the boot preflight + // requires to exist (default "fleet-sandbox-deny-all"). + NetworkPolicy string `yaml:"network_policy"` + // BundleDocsInImage declares that the sandbox IMAGE carries this bundle's + // supporting-doc dirs (protocols/, personas/, system_prompts/, skills/) at + // the SAME absolute paths the control plane reads them from — the only way + // a pod can see them, since it mounts just the workspace claim. Set it and + // the fileop path anchors for those roots stay valid inside a pod, so + // view_file works on `protocols/…` again; leave it false (the default) and + // the anchors are dropped, which is what refuses those reads. A + // declaration, not a probe: fleet cannot inspect an image's contents, so a + // wrong declaration surfaces as a not-found read, never as a widened + // boundary (reads only, still read-only, still inside the sandbox). + // FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE overrides it. + BundleDocsInImage bool `yaml:"bundle_docs_in_image"` + // NodeSelector pins sandbox pods to labeled nodes (a dedicated runner + // pool). FLEET_SANDBOX_K8S_NODE_SELECTOR ("k=v,k=v") overrides it. + NodeSelector map[string]string `yaml:"node_selector"` + // Tolerations let sandbox pods schedule onto a tainted runner pool. + // FLEET_SANDBOX_K8S_TOLERATIONS (a JSON array) overrides it. + Tolerations []KubernetesToleration `yaml:"tolerations"` +} + +// KubernetesToleration is the manifest shape of one sandbox-pod toleration +// (the four core/v1 fields fleet forwards). +type KubernetesToleration struct { + Key string `yaml:"key" json:"key,omitempty"` + Operator string `yaml:"operator" json:"operator,omitempty"` + Value string `yaml:"value" json:"value,omitempty"` + Effect string `yaml:"effect" json:"effect,omitempty"` } // ResolvedImageRef returns the image reference the fleet process should consume: @@ -429,11 +499,13 @@ func (s Sandbox) ResolvedImageRef() string { // sandboxManifest is the on-disk YAML shape of the manifest's sandbox: block. type sandboxManifest struct { - Containerfile string `yaml:"containerfile"` - Tag string `yaml:"tag"` - Image string `yaml:"image"` - Runtime string `yaml:"runtime"` - NetworkAllowlist []string `yaml:"network_allowlist"` + Containerfile string `yaml:"containerfile"` + Tag string `yaml:"tag"` + Image string `yaml:"image"` + Runtime string `yaml:"runtime"` + NetworkAllowlist []string `yaml:"network_allowlist"` + Backend string `yaml:"backend"` + Kubernetes *KubernetesSandbox `yaml:"kubernetes"` } // Branding carries the white-label strings surfaced in the web UI + login. @@ -1345,12 +1417,18 @@ func resolveSandbox(sm *sandboxManifest, bundleDir string) Sandbox { allowlist = append(allowlist, d) } } + var k8s KubernetesSandbox + if raw.Kubernetes != nil { + k8s = *raw.Kubernetes + } return Sandbox{ ContainerfileAbsPath: filepath.Join(bundleDir, cf), Tag: tag, Image: strings.TrimSpace(raw.Image), Runtime: strings.TrimSpace(raw.Runtime), NetworkAllowlist: allowlist, + Backend: strings.ToLower(strings.TrimSpace(raw.Backend)), + Kubernetes: k8s, } } diff --git a/internal/config/config.go b/internal/config/config.go index 9669ddba0..5a449acc2 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -383,6 +383,18 @@ var allowedEnvVars = map[string]bool{ "CHAT_WORKSPACE_ROOT": true, "FLEET_SANDBOX_IMAGE": true, "FLEET_SANDBOX_RUNTIME": true, + "FLEET_SANDBOX_BACKEND": true, + "FLEET_SANDBOX_K8S_NAMESPACE": true, + "FLEET_SANDBOX_K8S_WORKSPACE_CLAIM": true, + "FLEET_SANDBOX_K8S_SERVICE_ACCOUNT": true, + "FLEET_SANDBOX_K8S_IMAGE_PULL_SECRET": true, + "FLEET_SANDBOX_K8S_RUNTIME_CLASS": true, + "FLEET_SANDBOX_K8S_SECCOMP_PROFILE": true, + "FLEET_SANDBOX_K8S_KUBECONFIG": true, + "FLEET_SANDBOX_K8S_NETWORK_POLICY": true, + "FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE": true, + "FLEET_SANDBOX_K8S_NODE_SELECTOR": true, + "FLEET_SANDBOX_K8S_TOLERATIONS": true, "FLEET_DEFAULT_NETWORK_MODE": true, "FLEET_PII_REDACTION_ENABLED": true, "FLEET_PII_REDACTION_MODE": true, @@ -1011,6 +1023,33 @@ type Config struct { // ── sandbox ── SandboxImage string SandboxRuntime string + // SandboxBackend selects WHERE sandboxes run (#989): "" / "podman" — the + // co-located rootless-Podman backend (the single-box default) — or + // "kubernetes" — ephemeral pods in a cluster, for the split + // control-plane/runner enterprise deployment. FLEET_SANDBOX_BACKEND + // overrides the bundle manifest's sandbox.backend (same precedence as + // sandbox.runtime). Anything else refuses to boot (fail-closed, #1119 + // posture: an unrecognized value must never silently mean "podman"). + SandboxBackend string + // SandboxK8s* configure the kubernetes backend; ignored (and refused if + // set, to catch dead config) under the podman backend. Env values override + // the bundle manifest's sandbox.kubernetes block field-by-field. + SandboxK8sNamespace string // FLEET_SANDBOX_K8S_NAMESPACE, default "fleet-sandboxes" + SandboxK8sWorkspaceClaim string // FLEET_SANDBOX_K8S_WORKSPACE_CLAIM — required RWX PVC name + SandboxK8sServiceAccount string // FLEET_SANDBOX_K8S_SERVICE_ACCOUNT — pod identity (no token is ever mounted) + SandboxK8sImagePullSecret string // FLEET_SANDBOX_K8S_IMAGE_PULL_SECRET + SandboxK8sRuntimeClass string // FLEET_SANDBOX_K8S_RUNTIME_CLASS — hypervisor isolation (kata), preflighted + SandboxK8sSeccompProfile string // FLEET_SANDBOX_K8S_SECCOMP_PROFILE — node-local Localhost profile; empty = RuntimeDefault + SandboxK8sKubeconfig string // FLEET_SANDBOX_K8S_KUBECONFIG — out-of-cluster auth; empty = in-cluster + SandboxK8sNetworkPolicy string // FLEET_SANDBOX_K8S_NETWORK_POLICY — deny-all policy the preflight requires; default "fleet-sandbox-deny-all" + // SandboxK8sBundleDocsInImage declares that the sandbox IMAGE carries the + // bundle's supporting-doc dirs at the same absolute paths the control plane + // reads them from, keeping the fileop anchors for those roots valid inside + // a pod. Raw string, parsed fail-closed where it is consumed (like the + // selector/toleration knobs): empty = false. + SandboxK8sBundleDocsInImage string // FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE + SandboxK8sNodeSelector string // FLEET_SANDBOX_K8S_NODE_SELECTOR — "key=value,key=value" pinning sandbox pods to a runner pool + SandboxK8sTolerations string // FLEET_SANDBOX_K8S_TOLERATIONS — JSON array of {key,operator,value,effect} for a tainted runner pool // PIIRedactionEnabled gates the OPTIONAL PII redaction pass (#450) applied to // tool output before it enters the model context. FLEET_PII_REDACTION_ENABLED, // default false (byte-for-byte unchanged when off). Provider-neutral; the @@ -1521,9 +1560,23 @@ func Load(envFile string) (*Config, error) { AdminEmails: splitEmails(os.Getenv("ADMIN_EMAILS")), // ── sandbox ── - SandboxImage: getenvFleet("SANDBOX_IMAGE"), - SandboxRuntime: getenvFleet("SANDBOX_RUNTIME"), - DefaultNetworkMode: strings.ToLower(strings.TrimSpace(getenvFleet("DEFAULT_NETWORK_MODE"))), + SandboxImage: getenvFleet("SANDBOX_IMAGE"), + SandboxRuntime: getenvFleet("SANDBOX_RUNTIME"), + // Sandbox backend (#989). Lower-cased here; validated fail-closed at + // boot (resolveSandboxBackend in cmd/fleet) against the bundle value. + SandboxBackend: strings.ToLower(strings.TrimSpace(getenvFleet("SANDBOX_BACKEND"))), + SandboxK8sNamespace: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NAMESPACE")), + SandboxK8sWorkspaceClaim: strings.TrimSpace(getenvFleet("SANDBOX_K8S_WORKSPACE_CLAIM")), + SandboxK8sServiceAccount: strings.TrimSpace(getenvFleet("SANDBOX_K8S_SERVICE_ACCOUNT")), + SandboxK8sImagePullSecret: strings.TrimSpace(getenvFleet("SANDBOX_K8S_IMAGE_PULL_SECRET")), + SandboxK8sRuntimeClass: strings.TrimSpace(getenvFleet("SANDBOX_K8S_RUNTIME_CLASS")), + SandboxK8sSeccompProfile: strings.TrimSpace(getenvFleet("SANDBOX_K8S_SECCOMP_PROFILE")), + SandboxK8sKubeconfig: strings.TrimSpace(getenvFleet("SANDBOX_K8S_KUBECONFIG")), + SandboxK8sNetworkPolicy: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NETWORK_POLICY")), + SandboxK8sBundleDocsInImage: strings.TrimSpace(getenvFleet("SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE")), + SandboxK8sNodeSelector: strings.TrimSpace(getenvFleet("SANDBOX_K8S_NODE_SELECTOR")), + SandboxK8sTolerations: strings.TrimSpace(getenvFleet("SANDBOX_K8S_TOLERATIONS")), + DefaultNetworkMode: strings.ToLower(strings.TrimSpace(getenvFleet("DEFAULT_NETWORK_MODE"))), // PII redaction (#450) — optional, default off. PIIRedactionEnabled: lp.getenvFleetBool("PII_REDACTION_ENABLED", false), diff --git a/internal/sandbox/container.go b/internal/sandbox/container.go index 8eaa850a1..679f27298 100644 --- a/internal/sandbox/container.go +++ b/internal/sandbox/container.go @@ -1011,13 +1011,20 @@ func (c *containerImpl) executeFileOp(ctx context.Context, req FileOpRequest, an // it using directory descriptors. A model-controlled Root can therefore never // turn the shared workspace mount into authority over a sibling conversation. func (c *containerImpl) fileOpAnchor(root string) (anchor string, readOnly bool, err error) { + return fileOpAnchorFor(c.cfg.WorkspaceHostDir, c.cfg.ReadOnlyMounts, root) +} + +// fileOpAnchorFor is the backend-shared anchor resolution (see fileOpAnchor's +// doc): the kubernetes backend applies the identical policy over its own +// mount set, so anchor semantics cannot drift between backends. +func fileOpAnchorFor(workspaceDir string, readOnlyMounts []string, root string) (anchor string, readOnly bool, err error) { type mount struct { path string readOnly bool } - candidates := make([]mount, 0, len(c.cfg.ReadOnlyMounts)+1) - candidates = append(candidates, mount{path: c.cfg.WorkspaceHostDir}) - for _, path := range c.cfg.ReadOnlyMounts { + candidates := make([]mount, 0, len(readOnlyMounts)+1) + candidates = append(candidates, mount{path: workspaceDir}) + for _, path := range readOnlyMounts { candidates = append(candidates, mount{path: path, readOnly: true}) } best := "" diff --git a/internal/sandbox/host.go b/internal/sandbox/host.go index 707a29811..fcf9d86af 100644 --- a/internal/sandbox/host.go +++ b/internal/sandbox/host.go @@ -81,6 +81,15 @@ func (h *hostImpl) runBash(ctx context.Context, req BashRequest) (BashResult, er cmdCtx, cancel := context.WithTimeout(ctx, timeout) defer cancel() + // Running the caller's shell IS this component's contract: it is the + // unsandboxed TEST/DEV-ONLY executor behind the fleet_host_executor build + // tag (#159) — a release build gets the fail-closed stub in + // host_disabled.go, so this sink cannot ship (ADR-0002 enforcement). The + // CodeQL go/command-injection finding here is waived in + // .github/codeql-accepted-findings.json for the same reason (the CodeQL + // workflow deliberately compiles this file into the database so it is + // scanned rather than being a coverage hole; in-source codeql[] + // suppressions do not work with that pipeline — see codeql.yml). //nolint:gosec // shell execution is the purpose of this tool cmd := exec.CommandContext(cmdCtx, "bash", "-c", req.Command) if req.WorkingDir != "" { diff --git a/internal/sandbox/k8s_backend.go b/internal/sandbox/k8s_backend.go new file mode 100644 index 000000000..0d841df60 --- /dev/null +++ b/internal/sandbox/k8s_backend.go @@ -0,0 +1,1072 @@ +// Copyright (c) 2026 ElcanoTek +// SPDX-License-Identifier: MIT + +package sandbox + +// k8s_backend.go is the Kubernetes sandbox backend (#989): the same per-turn +// execution boundary as the rootless-Podman backend, delivered as an +// ephemeral Pod per sandbox instead of a local container. One Sandbox = one +// Pod running `sleep infinity`; bash is a one-shot exec, the python bridge is +// a held exec session, and file operations run the same embedded fileops.py — +// all over the apiserver's exec subresource, so the fleet control plane never +// shares a kernel with model-authored execution. +// +// What carries over from the podman backend unchanged: the workspace is +// mounted at the SAME absolute path as the control plane sees it (a shared +// RWX PersistentVolumeClaim replaces the bind mount), the rootfs is +// read-only with tmpfs-equivalent emptyDirs for scratch, all capabilities are +// dropped, and a cancelled/timed-out call poisons the sandbox and destroys +// the whole PID namespace — here by deleting the Pod with zero grace (#796). +// +// What is honestly different (documented in docs/DEPLOYMENT-KUBERNETES.md and +// ADR-0049): egress sealing is delegated to a NetworkPolicy the chart ships +// (verified to exist at boot, but ENFORCED by the cluster CNI, not by fleet); +// the per-pod pids limit is not expressible in a Pod spec; the "allowlisted" +// egress mode is unsupported (fail-closed at boot); resource telemetry (#263) +// is not collected; and seccomp is RuntimeDefault or an operator-installed +// Localhost profile rather than the bundled JSON. +// +// MCP credentials keep their ADR-0003 posture automatically: the broker runs +// in the control-plane process, and nothing in a sandbox Pod's spec, env, or +// mounts carries a credential — automountServiceAccountToken is explicitly +// false so a sandbox cannot even talk to the apiserver that created it. + +import ( + "bufio" + "bytes" + "context" + "crypto/rand" + "encoding/base64" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "log" + "os" + "strconv" + "strings" + "sync" + "sync/atomic" + "time" + + "github.com/ElcanoTek/fleet/internal/safe" +) + +// Sandbox backend names (#989): where every sandbox runs. The knob mirrors +// sandbox.runtime's precedence (FLEET_SANDBOX_BACKEND env wins, else the +// bundle manifest's sandbox.backend, else podman). +const ( + // BackendPodman is the co-located rootless-Podman backend — the + // single-box default and the only backend before #989. + BackendPodman = "podman" + // BackendKubernetes runs each sandbox as an ephemeral pod in a cluster: + // the split control-plane/runner enterprise deployment. + BackendKubernetes = "kubernetes" +) + +// ResolveBackend applies the sandbox-backend precedence in ONE place so every +// entrypoint (fleet boot, `fleet validate-config`) resolves identically: an +// explicit env value (FLEET_SANDBOX_BACKEND) wins, else the bundle manifest's +// sandbox.backend, else podman. An unrecognized value is an ERROR, never a +// silent fallback to podman — the #1119 posture: a typo'd security-relevant +// knob must refuse to boot rather than quietly mean something else. +func ResolveBackend(envBackend, bundleBackend string) (string, error) { + raw := strings.ToLower(strings.TrimSpace(envBackend)) + if raw == "" { + raw = strings.ToLower(strings.TrimSpace(bundleBackend)) + } + switch raw { + case "", BackendPodman: + return BackendPodman, nil + case BackendKubernetes: + return BackendKubernetes, nil + default: + return "", fmt.Errorf("unrecognized sandbox backend %q (FLEET_SANDBOX_BACKEND / manifest sandbox.backend): want %q or %q — refusing to guess (fail-closed)", raw, BackendPodman, BackendKubernetes) + } +} + +// KubernetesConfig configures the kubernetes sandbox backend. It is trusted +// operator config, same authority tier as ContainerConfig / sandbox.runtime. +type KubernetesConfig struct { + // Namespace is where sandbox Pods are created. Defaults to + // "fleet-sandboxes"; in-cluster deployments may point it at any namespace + // the RBAC grant covers. Keeping it SEPARATE from the control plane's own + // namespace is what lets the RBAC grant be pod-scoped and narrow. + Namespace string + + // WorkspaceClaim is the name of the ReadWriteMany PersistentVolumeClaim + // (in Namespace) that holds the workspace root. It is mounted into every + // sandbox Pod at ContainerConfig.WorkspaceHostDir — the same absolute path + // the control plane mounts it at — preserving the same-path invariant that + // keeps MCP-returned paths usable inside bash/run_python. Required. + WorkspaceClaim string + + // ServiceAccount, when set, is stamped as the Pod's serviceAccountName. + // The token is never mounted either way (automountServiceAccountToken is + // forced false); this exists so admission policies can key on identity. + ServiceAccount string + + // ImagePullSecret, when set, is attached for pulling the sandbox image + // from a private registry. + ImagePullSecret string + + // RuntimeClassName, when set, selects a hypervisor-isolated runtime class + // (e.g. kata) for sandbox Pods — the k8s counterpart of sandbox.runtime. + // Preflighted to exist, fail-closed, mirroring ADR-0010. + RuntimeClassName string + + // SeccompLocalhostProfile, when set, is a node-local profile path + // (relative to the kubelet's seccomp root) applied as a Localhost seccomp + // profile. Empty means RuntimeDefault. + SeccompLocalhostProfile string + + // KubeconfigPath selects out-of-cluster auth. Empty means in-cluster + // (the standard service-account mount). + KubeconfigPath string + + // NetworkPolicyName is the deny-all NetworkPolicy the boot preflight + // requires to exist in Namespace — the object that seals egress for Pods + // labeled fleet.elcanotek.com/egress=none. Defaults to + // "fleet-sandbox-deny-all". The preflight verifies the OBJECT exists; + // enforcement is the CNI's job and the docs say so plainly. + NetworkPolicyName string + + // NodeSelector pins sandbox pods to labeled nodes — the standard way to + // give runners a DEDICATED node pool, which is the issue's scaling story + // (more runner capacity = a bigger pool, never more fleet replicas). + NodeSelector map[string]string + + // Tolerations let sandbox pods schedule onto a tainted runner pool, the + // usual companion to NodeSelector for a pool nothing else may land on. + Tolerations []K8sToleration + + // StartTimeout caps pod schedule+pull+start. Zero defaults to 2 minutes + // (image pulls make the podman default of 30s unrealistic). + StartTimeout time.Duration +} + +// K8sToleration mirrors the four core/v1 Toleration fields sandbox pods need +// (tolerationSeconds is a drain concern that does not apply to pods fleet +// deletes itself). +type K8sToleration struct { + Key string `json:"key,omitempty"` + Operator string `json:"operator,omitempty"` + Value string `json:"value,omitempty"` + Effect string `json:"effect,omitempty"` +} + +// ParseK8sNodeSelector parses the FLEET_SANDBOX_K8S_NODE_SELECTOR form — +// comma-separated key=value pairs ("pool=fleet-sandboxes,arch=amd64") — into +// the map the pod spec takes. Empty input is a nil map; a malformed pair is +// an error so a typo'd selector refuses to boot instead of silently +// scheduling sandboxes onto the wrong nodes. +func ParseK8sNodeSelector(s string) (map[string]string, error) { + s = strings.TrimSpace(s) + if s == "" { + return nil, nil + } + out := make(map[string]string) + for _, pair := range strings.Split(s, ",") { + k, v, ok := strings.Cut(strings.TrimSpace(pair), "=") + k, v = strings.TrimSpace(k), strings.TrimSpace(v) + if !ok || k == "" || v == "" { + return nil, fmt.Errorf("invalid node selector pair %q (want key=value, comma-separated)", pair) + } + out[k] = v + } + return out, nil +} + +// ParseK8sTolerations parses the FLEET_SANDBOX_K8S_TOLERATIONS form — a JSON +// array of {key, operator, value, effect} objects. Empty input is nil; +// malformed JSON or an unknown field is an error (fail-closed, strict +// decoding, matching the additive-first schema posture). +func ParseK8sTolerations(s string) ([]K8sToleration, error) { + s = strings.TrimSpace(s) + if s == "" { + return nil, nil + } + dec := json.NewDecoder(strings.NewReader(s)) + dec.DisallowUnknownFields() + var out []K8sToleration + if err := dec.Decode(&out); err != nil { + return nil, fmt.Errorf("invalid tolerations JSON (want an array of {key,operator,value,effect}): %w", err) + } + return out, nil +} + +// ParseK8sBundleDocsInImage parses the +// FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE form — a boolean declaring that the +// sandbox IMAGE carries the bundle's supporting-doc dirs at the same absolute +// paths the control plane reads them from, so the fileop path anchors for +// those roots stay valid inside a pod (see the ReadOnlyMounts handling in +// internal/agent). Empty input is false (the safe default: fleet assumes +// nothing about a sandbox image's contents). A malformed value is an error — +// a typo'd "ture" must not read as "keep the anchors". +func ParseK8sBundleDocsInImage(s string) (bool, error) { + s = strings.TrimSpace(s) + if s == "" { + return false, nil + } + v, err := strconv.ParseBool(s) + if err != nil { + return false, fmt.Errorf("invalid boolean %q (want true or false)", s) + } + return v, nil +} + +// defaultK8sNamespace / defaultK8sNetworkPolicy are the conventions the Helm +// chart ships; the backend defaults match so a chart install needs no extra +// wiring. +const ( + defaultK8sNamespace = "fleet-sandboxes" + defaultK8sNetworkPolicy = "fleet-sandbox-deny-all" + defaultK8sStartTimeout = 2 * time.Minute +) + +// sandboxContainerName is the single container in every sandbox Pod. +const sandboxContainerName = "sandbox" + +// k8sBridgeDir is the writable emptyDir where the bridge + fileops scripts +// are uploaded at pod start (the k8s counterpart of the /opt/bridge bind +// mount — there is no host filesystem to bind from). +const ( + k8sBridgeDir = "/opt/fleet-bridge" + k8sBridgePath = k8sBridgeDir + "/bridge.py" + k8sFileOpsPath = k8sBridgeDir + "/fileops.py" + k8sPodNamePrefix = "fleet-sandbox-" +) + +// Pod labels. app.kubernetes.io/* follow the k8s recommended-label +// convention; the fleet.elcanotek.com/* pair carries the ownership identity +// the orphan sweep keys on (the k8s counterpart of the podman +// fleet.instance label) and the egress posture the chart's NetworkPolicies +// select on. +const ( + k8sLabelName = "app.kubernetes.io/name" + k8sLabelManagedBy = "app.kubernetes.io/managed-by" + k8sLabelInstance = "fleet.elcanotek.com/instance" + k8sLabelEgress = "fleet.elcanotek.com/egress" + k8sLabelNameValue = "fleet-sandbox" + k8sLabelManagedVal = "fleet" +) + +// k8sInstanceLabel is thisInstanceLabel ("@") re-encoded to the +// label-safe form "p-t" ('@' is not a legal label character). +var k8sInstanceLabel = func() string { + pid, start, _ := instanceLabelOwner(thisInstanceLabel) + return fmt.Sprintf("p%d-t%d", pid, start) +}() + +// parseK8sInstanceLabel inverts k8sInstanceLabel's encoding. ok is false for +// anything unparseable — callers treat that as "ownership unknown". +func parseK8sInstanceLabel(label string) (pid int, startedAt int64, ok bool) { + rest, found := strings.CutPrefix(label, "p") + if !found { + return 0, 0, false + } + pidStr, startStr, found := strings.Cut(rest, "-t") + if !found { + return 0, 0, false + } + pid, err := strconv.Atoi(pidStr) + if err != nil || pid <= 0 { + return 0, 0, false + } + startedAt, err = strconv.ParseInt(startStr, 10, 64) + if err != nil || startedAt <= 0 { + return pid, 0, true + } + return pid, startedAt, true +} + +// KubernetesBackend is the boot-built handle for the kubernetes sandbox +// backend: one API client plus the resolved config, shared by the pool, the +// preflight, and the orphan sweep. Construct with NewKubernetesBackend. +type KubernetesBackend struct { + cfg KubernetesConfig + client *k8sClient +} + +// NewKubernetesBackend resolves credentials (in-cluster unless a kubeconfig +// is configured) and defaults, returning the backend handle. It performs no +// network I/O — Preflight does the fail-closed cluster checks. +func NewKubernetesBackend(cfg KubernetesConfig) (*KubernetesBackend, error) { + var ( + client *k8sClient + kubeconfigNS string + err error + ) + if cfg.KubeconfigPath != "" { + client, kubeconfigNS, err = newKubeconfigClient(cfg.KubeconfigPath) + } else { + client, err = newInClusterClient() + } + if err != nil { + return nil, fmt.Errorf("kubernetes sandbox backend: %w", err) + } + if cfg.Namespace == "" { + // Precedence: explicit config, else the kubeconfig context's + // namespace, else (in-cluster) the control plane's own namespace — + // the Helm chart's default topology, because a PersistentVolumeClaim + // cannot be mounted across namespaces and the workspace claim is + // shared with the control plane — else the shipped default name. + cfg.Namespace = kubeconfigNS + if cfg.Namespace == "" && cfg.KubeconfigPath == "" { + cfg.Namespace = inClusterNamespace() + } + if cfg.Namespace == "" { + cfg.Namespace = defaultK8sNamespace + } + } + if cfg.NetworkPolicyName == "" { + cfg.NetworkPolicyName = defaultK8sNetworkPolicy + } + if cfg.StartTimeout <= 0 { + cfg.StartTimeout = defaultK8sStartTimeout + } + return &KubernetesBackend{cfg: cfg, client: client}, nil +} + +// Namespace reports the resolved sandbox namespace (for logs and doctor +// output). +func (b *KubernetesBackend) Namespace() string { return b.cfg.Namespace } + +// StartTimeout reports the resolved pod start ceiling; the pool derives its +// outer construction contexts from it (mirroring resolveStartTimeout). +func (b *KubernetesBackend) StartTimeout() time.Duration { return b.cfg.StartTimeout } + +// newSandbox starts one sandbox Pod and returns the wrapping handle. cfg +// carries the backend-shared knobs (image, workspace path, limits, network +// posture); the pool routes here from the same take paths that call +// NewContainer for podman. +func (b *KubernetesBackend) newSandbox(ctx context.Context, cfg ContainerConfig) (*Sandbox, error) { + if cfg.Image == "" { + return nil, fmt.Errorf("sandbox: ContainerConfig.Image required") + } + if cfg.WorkspaceHostDir == "" { + return nil, fmt.Errorf("sandbox: ContainerConfig.WorkspaceHostDir required") + } + if cfg.BridgeScript == nil { + return nil, fmt.Errorf("sandbox: ContainerConfig.BridgeScript required") + } + if cfg.ProxyURL != "" { + // The allowlisted egress proxy binds to the control-plane host's + // loopback; a pod on another node cannot reach it, and pretending + // otherwise would grant open egress under an "allowlisted" banner. + return nil, errors.New("sandbox: allowlisted egress mode is not supported by the kubernetes backend (fail-closed)") + } + cfg = applyContainerDefaults(cfg) + k := &k8sImpl{backend: b, cfg: cfg} + if err := k.start(ctx); err != nil { + k.close() + return nil, err + } + return &Sandbox{mode: ModeKubernetes, impl: k}, nil +} + +// k8sImpl is the kubernetes impl: one Pod, exec'd into for every operation. +// The struct mirrors containerImpl's locking discipline — podMu guards the +// pod name (cleared by close, snapshotted by cross-goroutine readers), mu +// guards the bridge session and is held for a whole run_python cell. +type k8sImpl struct { + backend *KubernetesBackend + cfg ContainerConfig + + podMu sync.Mutex + podName string + + mu sync.Mutex + bridge *k8sExecSession + bridgeStdout *bufio.Reader + bridgeStderr *syncBuffer + bridgeStarted bool + + execPoisoned atomic.Bool +} + +// generatePodName mirrors generateContainerName with the pod prefix. +func generatePodName() string { + var buf [8]byte + _, _ = rand.Read(buf[:]) + return k8sPodNamePrefix + hex.EncodeToString(buf[:]) +} + +// k8sQuantityFromPodmanMemory converts a podman --memory value ("512m", +// "2g", bare bytes) into a Kubernetes resource quantity (plain bytes — +// unambiguous and accepted everywhere a quantity is). +func k8sQuantityFromPodmanMemory(limit string) (string, error) { + b, err := parseMemoryToBytes(limit) + if err != nil { + return "", err + } + return strconv.FormatInt(b, 10), nil +} + +// k8sQuantityFromPodmanCPU converts a podman --cpus value ("1.0", "2.50") +// into a Kubernetes CPU quantity in millicores. +func k8sQuantityFromPodmanCPU(limit string) (string, error) { + f, err := strconv.ParseFloat(strings.TrimSpace(limit), 64) + if err != nil || f <= 0 { + return "", fmt.Errorf("invalid cpu limit %q", limit) + } + return strconv.FormatInt(int64(f*1000), 10) + "m", nil +} + +// buildSandboxPod is the pure pod-spec builder — the k8s counterpart of the +// `podman run` argument list in containerImpl.start, kept side-effect-free so +// the hardening posture is pinned by unit tests the way podman_args_test.go +// pins the flag list. +func buildSandboxPod(cfg ContainerConfig, kcfg KubernetesConfig, name string) (*k8sPod, error) { + memory, err := k8sQuantityFromPodmanMemory(cfg.MemoryLimit) + if err != nil { + return nil, fmt.Errorf("sandbox pod memory limit: %w", err) + } + cpu, err := k8sQuantityFromPodmanCPU(cfg.CPULimit) + if err != nil { + return nil, fmt.Errorf("sandbox pod cpu limit: %w", err) + } + + egress := "open" + if cfg.NoNetwork { + // Sealed posture: selected by the deny-all NetworkPolicy the preflight + // verified. The label is the contract; enforcement is the CNI's. + egress = "none" + } + + boolPtr := func(v bool) *bool { return &v } + int64Ptr := func(v int64) *int64 { return &v } + + limits := map[string]string{"memory": memory, "cpu": cpu} + requests := map[string]string{"memory": memory, "cpu": cpu} + if cfg.DiskLimitGB > 0 { + // ephemeral-storage caps the pod's writable layer AND its emptyDirs — + // a strictly stronger surface than podman's per-file ulimit + layer + // quota. The workspace PVC is still outside it, exactly like the bind + // mount is outside podman's quota; the docs carry the same honest + // limit statement. + limits["ephemeral-storage"] = fmt.Sprintf("%dGi", cfg.DiskLimitGB) + } + + seccomp := &k8sSeccompProfile{Type: "RuntimeDefault"} + if kcfg.SeccompLocalhostProfile != "" { + profile := kcfg.SeccompLocalhostProfile + seccomp = &k8sSeccompProfile{Type: "Localhost", LocalhostProfile: &profile} + } + + // emptyDir scratch mounts mirror the podman --tmpfs set (sizes included) + // so a --read-only-rootfs image behaves identically in both backends, + // plus the bridge dir the scripts are uploaded into. + mounts := []k8sVolumeMount{ + {Name: "workspace", MountPath: cfg.WorkspaceHostDir}, + {Name: "bridge", MountPath: k8sBridgeDir}, + {Name: "tmp", MountPath: "/tmp"}, + {Name: "ipython", MountPath: "/home/sandbox/.ipython"}, + {Name: "cache", MountPath: "/home/sandbox/.cache"}, + {Name: "config", MountPath: "/home/sandbox/.config"}, + } + volumes := []k8sVolume{ + {Name: "workspace", PersistentVolumeClaim: &k8sPVCVolSource{ClaimName: kcfg.WorkspaceClaim}}, + {Name: "bridge", EmptyDir: &k8sEmptyDir{SizeLimit: "8Mi"}}, + {Name: "tmp", EmptyDir: &k8sEmptyDir{SizeLimit: "128Mi"}}, + {Name: "ipython", EmptyDir: &k8sEmptyDir{SizeLimit: "32Mi"}}, + {Name: "cache", EmptyDir: &k8sEmptyDir{SizeLimit: "32Mi"}}, + {Name: "config", EmptyDir: &k8sEmptyDir{SizeLimit: "8Mi"}}, + } + + spec := k8sPodSpec{ + RestartPolicy: "Never", + // The sandbox must not be able to reach the apiserver that made it: + // no token, no service links, ever. + AutomountServiceAccountToken: boolPtr(false), + EnableServiceLinks: boolPtr(false), + TerminationGracePeriodSeconds: int64Ptr(5), + ServiceAccountName: kcfg.ServiceAccount, + SecurityContext: &k8sPodSecurityCtx{ + RunAsNonRoot: boolPtr(true), + // uid/gid 1000 matches the image's USER sandbox and the podman + // keep-id mapping, so workspace files are owned consistently across + // backends. fsGroup makes the PVC group-writable for that gid. + RunAsUser: int64Ptr(1000), + RunAsGroup: int64Ptr(1000), + FSGroup: int64Ptr(1000), + SeccompProfile: seccomp, + }, + Containers: []k8sContainer{{ + Name: sandboxContainerName, + Image: cfg.Image, + // Explicit IfNotPresent: the API default for a :latest tag is + // Always, which breaks side-loaded images (kind) and re-pulls a + // mutable tag mid-fleet-run — sandbox image freshness is a deploy + // concern, not a per-pod one. + ImagePullPolicy: "IfNotPresent", + // PID 1: a do-nothing process to keep the pod alive; every real + // operation execs into it — same shape as the podman backend. + Command: []string{"sleep", "infinity"}, + WorkingDir: cfg.WorkspaceHostDir, + SecurityContext: &k8sContainerSecCtx{ + AllowPrivilegeEscalation: boolPtr(false), + ReadOnlyRootFilesystem: boolPtr(true), + Capabilities: &k8sCapabilities{Drop: []string{"ALL"}}, + }, + Resources: &k8sResources{Limits: limits, Requests: requests}, + VolumeMounts: mounts, + }}, + Volumes: volumes, + } + if kcfg.RuntimeClassName != "" { + rc := kcfg.RuntimeClassName + spec.RuntimeClassName = &rc + } + if kcfg.ImagePullSecret != "" { + spec.ImagePullSecrets = []k8sLocalObjRef{{Name: kcfg.ImagePullSecret}} + } + // Dedicated runner pool: selector + taints, when configured. + if len(kcfg.NodeSelector) > 0 { + spec.NodeSelector = kcfg.NodeSelector + } + for _, tol := range kcfg.Tolerations { + spec.Tolerations = append(spec.Tolerations, k8sToleration(tol)) + } + + return &k8sPod{ + Metadata: k8sObjectMeta{ + Name: name, + Namespace: kcfg.Namespace, + Labels: map[string]string{ + k8sLabelName: k8sLabelNameValue, + k8sLabelManagedBy: k8sLabelManagedVal, + k8sLabelInstance: k8sInstanceLabel, + k8sLabelEgress: egress, + }, + }, + Spec: spec, + }, nil +} + +// k8sPodPollInterval is how often start() re-reads the pod while waiting for +// Running. +const k8sPodPollInterval = 500 * time.Millisecond + +func (k *k8sImpl) start(ctx context.Context) error { + name := generatePodName() + k.podMu.Lock() + k.podName = name + k.podMu.Unlock() + + pod, err := buildSandboxPod(k.cfg, k.backend.cfg, name) + if err != nil { + return err + } + startCtx, cancel := context.WithTimeout(ctx, k.backend.cfg.StartTimeout) + defer cancel() + if err := k.backend.client.createPod(startCtx, k.backend.cfg.Namespace, pod); err != nil { + return fmt.Errorf("create sandbox pod: %w", err) + } + if err := k.waitForRunning(startCtx, name); err != nil { + return err + } + // Upload the bridge + fileops scripts into the pod's writable bridge + // emptyDir. There is no host filesystem to bind-mount them from; the + // upload verifies byte counts so a truncated transfer fails loudly here + // rather than as an opaque bridge error mid-turn. + if err := k.uploadFile(startCtx, k8sBridgePath, k.cfg.BridgeScript); err != nil { + return fmt.Errorf("upload bridge script: %w", err) + } + if err := k.uploadFile(startCtx, k8sFileOpsPath, fileOpsScript); err != nil { + return fmt.Errorf("upload fileops script: %w", err) + } + return nil +} + +func (k *k8sImpl) waitForRunning(ctx context.Context, name string) error { + ticker := time.NewTicker(k8sPodPollInterval) + defer ticker.Stop() + var lastState string + for { + pod, err := k.backend.client.getPod(ctx, k.backend.cfg.Namespace, name) + if err == nil { + switch pod.Status.Phase { + case "Running": + for _, cs := range pod.Status.ContainerStatuses { + if cs.Name == sandboxContainerName && cs.Ready { + return nil + } + } + case "Failed", "Succeeded": + // Status text is cluster-derived — sanitized before it enters + // an error that upstream code logs (go/log-injection). + return fmt.Errorf("sandbox pod %s entered terminal phase %s before becoming ready: %s", name, sanitizeClusterText(pod.Status.Phase), sanitizeClusterText(pod.Status.Message)) + } + for _, cs := range pod.Status.ContainerStatuses { + if cs.State.Waiting != nil { + lastState = sanitizeClusterText(cs.State.Waiting.Reason) + // Pull failures never self-heal within a start timeout — + // fail fast with the reason instead of burning the window. + if lastState == "ErrImagePull" || lastState == "ImagePullBackOff" || lastState == "InvalidImageName" { + return fmt.Errorf("sandbox pod %s cannot pull image %s (%s): %s", name, k.cfg.Image, lastState, sanitizeClusterText(cs.State.Waiting.Message)) + } + } + } + } + select { + case <-ctx.Done(): + if lastState != "" { + return fmt.Errorf("sandbox pod %s not ready before start timeout (last container state: %s): %w", name, lastState, ctx.Err()) + } + return fmt.Errorf("sandbox pod %s not ready before start timeout: %w", name, ctx.Err()) + case <-ticker.C: + } + } +} + +// uploadFile writes data to path inside the pod via a one-shot exec. The v4 +// exec protocol cannot half-close stdin, so the reader is bounded with +// `head -c `; the write is verified by byte count. +func (k *k8sImpl) uploadFile(ctx context.Context, path string, data []byte) error { + podName := k.currentPodName() + if podName == "" { + return ErrClosed + } + // path is one of the two fixed k8sBridgeDir constants — never + // model-supplied — so embedding it in the shell line is safe. + script := fmt.Sprintf("head -c %d > %s && wc -c < %s", len(data), path, path) + var stdout, stderr bytes.Buffer + code, err := k.backend.client.runOneShotExec(ctx, k.backend.cfg.Namespace, podName, sandboxContainerName, + []string{"/bin/sh", "-c", script}, data, &stdout, &stderr) + if err != nil { + return fmt.Errorf("upload %s: %w", path, err) + } + if code != 0 { + // stderr is pod output — sanitized like all cluster-derived text. + return fmt.Errorf("upload %s: exit %d (%.200s)", path, code, sanitizeClusterText(stderr.String())) + } + if got := sanitizeClusterText(strings.TrimSpace(stdout.String())); got != strconv.Itoa(len(data)) { + return fmt.Errorf("upload %s: wrote %s of %d bytes", path, got, len(data)) + } + return nil +} + +// currentPodName snapshots the pod name under podMu — same discipline as +// containerImpl.currentContainerID; "" means already torn down. +func (k *k8sImpl) currentPodName() string { + k.podMu.Lock() + defer k.podMu.Unlock() + return k.podName +} + +func (k *k8sImpl) poisoned() bool { return k.execPoisoned.Load() } + +// deletePodNow removes the pod immediately on a fresh context — the #796 +// containment: destroying the pod destroys its PID namespace and every +// straggler in it. Reports whether the deletion (or prior disappearance) was +// confirmed. Mirrors killContainerNow, including taking the name as a +// parameter so it never touches k.mu. +func (k *k8sImpl) deletePodNow(podName string) bool { + if podName == "" { + return true // already torn down by close() + } + delCtx, cancel := context.WithTimeout(context.Background(), execReapTimeout) + defer cancel() + if err := k.backend.client.deletePod(delCtx, k.backend.cfg.Namespace, podName); err != nil { + if isK8sNotFound(err) { + return true + } + log.Printf("sandbox: cancelled-exec pod delete unconfirmed (%s): %v", podName, err) + return false + } + return true +} + +func (k *k8sImpl) runBash(ctx context.Context, req BashRequest) (BashResult, error) { + podName := k.currentPodName() + if podName == "" { + return BashResult{}, fmt.Errorf("run bash: %w", ErrClosed) + } + timeout := req.Timeout + if timeout <= 0 { + timeout = 5 * time.Minute + } + cmdCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + + // The exec API has no --workdir; a positional-parameter wrapper applies + // the cwd without any quoting of the user command or the directory. + command := []string{"bash", "-c", req.Command} + if req.WorkingDir != "" { + command = []string{"/bin/sh", "-c", `cd -- "$1" || exit 126; shift; exec bash -c "$1"`, "fleet-bash", req.WorkingDir, req.Command} + } + + stdoutBuf := &cappedBuffer{cap: BashOutputCaptureCap} + stderrBuf := &cappedBuffer{cap: BashOutputCaptureCap} + code, execErr := k.backend.client.runOneShotExec(cmdCtx, k.backend.cfg.Namespace, podName, sandboxContainerName, + command, nil, stdoutBuf, stderrBuf) + + res := BashResult{ + ExitCode: code, + Stdout: stdoutBuf.buf.Bytes(), + Stderr: stderrBuf.buf.Bytes(), + StdoutDiscarded: stdoutBuf.discarded, + StderrDiscarded: stderrBuf.discarded, + } + if cmdCtx.Err() != nil { + // Cancellation/timeout only tore down the exec CONNECTION; the shell + // and its descendants keep running in the pod (#796). Delete the pod + // synchronously — destroying the PID namespace is the one guaranteed + // containment — and poison the sandbox so it is retired, exactly like + // the podman backend. + res.TimedOut = errors.Is(cmdCtx.Err(), context.DeadlineExceeded) + res.Cancelled = !res.TimedOut + k.execPoisoned.Store(true) + res.CleanupConfirmed = k.deletePodNow(podName) + res.SandboxRetired = true + return res, nil + } + if execErr != nil { + return res, fmt.Errorf("pod exec bash: %w", execErr) + } + return res, nil +} + +func (k *k8sImpl) runFileOp(ctx context.Context, req FileOpRequest) (FileOpResult, error) { + anchor, readOnly, err := fileOpAnchorFor(k.cfg.WorkspaceHostDir, k.cfg.ReadOnlyMounts, req.Root) + if err != nil { + return FileOpResult{}, err + } + if readOnly && req.Op != FileOpRead { + return FileOpResult{}, fmt.Errorf("fileop write requested beneath a read-only mount: %w", ErrFileOpUnsafePath) + } + if !readOnly && !req.rootBound { + return FileOpResult{}, fmt.Errorf("writable fileop root was not bound before the turn: %w", ErrFileOpUnsafePath) + } + return k.executeFileOp(ctx, req, anchor) +} + +func (k *k8sImpl) bindFileOpRoot(ctx context.Context, root string) (FileOpRootIdentity, error) { + anchor, readOnly, err := fileOpAnchorFor(k.cfg.WorkspaceHostDir, k.cfg.ReadOnlyMounts, root) + if err != nil { + return FileOpRootIdentity{}, err + } + if readOnly { + return FileOpRootIdentity{}, fmt.Errorf("cannot bind a read-only mount as the writable fileop root: %w", ErrFileOpUnsafePath) + } + res, err := k.executeFileOp(ctx, FileOpRequest{Op: fileOpBindRoot, Path: root, Root: root}, anchor) + if err != nil { + return FileOpRootIdentity{}, err + } + return res.rootIdentity, nil +} + +func (k *k8sImpl) executeFileOp(ctx context.Context, req FileOpRequest, anchor string) (FileOpResult, error) { + podName := k.currentPodName() + if podName == "" { + return FileOpResult{}, fmt.Errorf("fileop %s: %w", req.Op, ErrClosed) + } + cmdCtx, cancel := context.WithTimeout(ctx, fileOpTimeout) + defer cancel() + + reqJSON, err := encodeFileOpWire(req, anchor) + if err != nil { + return FileOpResult{}, err + } + + // fileops.py reads stdin to EOF; v4 exec cannot half-close stdin, so the + // read is bounded with `head -c ` — EOF arrives when head exits. + script := fmt.Sprintf("head -c %d | python3 %s", len(reqJSON), k8sFileOpsPath) + var stdout, stderr bytes.Buffer + code, execErr := k.backend.client.runOneShotExec(cmdCtx, k.backend.cfg.Namespace, podName, sandboxContainerName, + []string{"/bin/sh", "-c", script}, reqJSON, &stdout, &stderr) + if execErr != nil || cmdCtx.Err() != nil { + if cmdCtx.Err() != nil { + // Same containment as bash: the helper may still be alive inside a + // persistent pod and could complete a rename after the turn stopped. + k.execPoisoned.Store(true) + _ = k.deletePodNow(podName) + return FileOpResult{}, fmt.Errorf("fileop %s interrupted (%w); sandbox retired: %w", req.Op, cmdCtx.Err(), ErrPoisoned) + } + return FileOpResult{}, fmt.Errorf("fileop %s exec: %w (%.200s)", req.Op, execErr, sanitizeClusterText(stderr.String())) + } + if code != 0 { + return FileOpResult{}, fmt.Errorf("fileop %s: helper exit %d (%.200s)", req.Op, code, sanitizeClusterText(stderr.String())) + } + return decodeFileOpResponse(stdout.Bytes()) +} + +// encodeFileOpWire builds the JSON request fileops.py reads — shared shape +// with the podman/host backends (their inline wire-building predates this +// helper; the k8s backend uses it so the three cannot drift further). +func encodeFileOpWire(req FileOpRequest, anchor string) ([]byte, error) { + wire := map[string]any{ + "op": string(req.Op), + "path": req.Path, + "root": req.Root, + "anchor": anchor, + } + switch req.Op { + case FileOpRead: + wire["offset"] = req.Offset + wire["limit"] = req.Limit + case FileOpWrite: + wire["data_b64"] = base64.StdEncoding.EncodeToString(req.Data) + case FileOpEdit: + wire["old_b64"] = base64.StdEncoding.EncodeToString([]byte(req.OldText)) + wire["new_b64"] = base64.StdEncoding.EncodeToString([]byte(req.NewText)) + wire["replace_all"] = req.ReplaceAll + if req.ExpectedSHA256 != "" { + wire["expected_sha256"] = req.ExpectedSHA256 + } + case fileOpBindRoot: + // Root + anchor are the complete request. + default: + return nil, fmt.Errorf("unknown fileop %q", req.Op) + } + if req.testPause > 0 { + wire["test_pause_ms"] = req.testPause.Milliseconds() + wire["test_ready_name"] = req.testReadyName + } + if req.rootBound { + wire["expected_dev"] = req.expectedDev + wire["expected_ino"] = req.expectedIno + } + out, err := json.Marshal(wire) + if err != nil { + return nil, fmt.Errorf("marshal fileop: %w", err) + } + return out, nil +} + +func (k *k8sImpl) runPython(ctx context.Context, req PythonRequest) (PythonResult, error) { + timeout := req.Timeout + if timeout <= 0 { + timeout = 5 * time.Minute + } + if err := k.ensureBridge(); err != nil { + return PythonResult{}, fmt.Errorf("start python bridge in pod: %w", err) + } + + wireReq := bridgeRequest{ + Code: req.Code, + ReturnVars: req.ReturnVars, + TimeoutSeconds: int(timeout.Seconds()), + WorkspaceDir: req.WorkspaceDir, + ResetKernel: req.ResetKernel, + } + reqBytes, err := json.Marshal(wireReq) + if err != nil { + return PythonResult{}, fmt.Errorf("marshal bridge request: %w", err) + } + + k.mu.Lock() + defer k.mu.Unlock() + + // Re-validate under the lock (mirrors containerImpl): a concurrent close() + // nils the bridge fields between ensureBridge and this re-acquire. + if k.bridge == nil || k.bridgeStdout == nil { + return PythonResult{}, fmt.Errorf("send bridge request: %w", ErrClosed) + } + if err := k.bridge.writeStdin(append(reqBytes, '\n')); err != nil { + k.terminateBridgeLocked() + return PythonResult{}, fmt.Errorf("send bridge request: %w%s", err, k.bridgeStderrSuffix()) + } + + type readResult struct { + data []byte + discarded int64 + err error + } + ch := make(chan readResult, 1) + // Snapshot the reader before launching the goroutine — the cancel/timeout + // arms nil the field via terminateBridgeLocked (#583's lesson, upheld). + stdout := k.bridgeStdout + go func() { + defer safe.Recover("sandbox.k8s.bridge_read", func(any) { + ch <- readResult{err: fmt.Errorf("bridge reader panicked")} + }) + data, discarded, err := readCappedLine(stdout, bridgeResponseCaptureCap) + ch <- readResult{data: data, discarded: discarded, err: err} + }() + timer := time.NewTimer(timeout) + defer timer.Stop() + select { + case <-ctx.Done(): + // The cell keeps executing in the pod until its PID namespace goes + // away — delete the pod and poison, exactly the podman #796 handling. + k.execPoisoned.Store(true) + _ = k.deletePodNow(k.currentPodName()) + k.terminateBridgeLocked() + return PythonResult{}, fmt.Errorf("python execution cancelled (%w); sandbox retired: %w", ctx.Err(), ErrPoisoned) + case <-timer.C: + k.execPoisoned.Store(true) + _ = k.deletePodNow(k.currentPodName()) + k.terminateBridgeLocked() + return PythonResult{}, fmt.Errorf("python execution timed out after %v; sandbox retired: %w", timeout, ErrPoisoned) + case r := <-ch: + if r.err != nil { + // Session dead (pod-side bridge exited, connection dropped). The pod + // itself is intact — reset the bridge so the next call boots fresh. + k.terminateBridgeLocked() + return PythonResult{}, fmt.Errorf("bridge closed unexpectedly: %w%s", r.err, k.bridgeStderrSuffix()) + } + if r.discarded > 0 { + return PythonResult{}, fmt.Errorf("bridge response exceeded %d bytes (%d bytes discarded) and was dropped — return large results by writing them to a workspace file instead", bridgeResponseCaptureCap, r.discarded) + } + return parseBridgeResponse(r.data) + } +} + +// ensureBridge starts the bridge exec session on first use, mirroring +// containerImpl.ensureBridge (including the post-start settle delay outside +// the lock). +func (k *k8sImpl) ensureBridge() error { + started, err := k.startBridgeIfNeeded() + if err != nil || !started { + return err + } + time.Sleep(100 * time.Millisecond) + return nil +} + +func (k *k8sImpl) startBridgeIfNeeded() (started bool, err error) { + k.mu.Lock() + defer k.mu.Unlock() + + if k.bridgeStarted && k.bridge != nil { + select { + case <-k.bridge.done: + // Session ended behind our back (pod-side exit) — fall through and + // start a fresh one. + default: + return false, nil + } + } + + podName := k.currentPodName() + if podName == "" { + return false, fmt.Errorf("start bridge: %w", ErrClosed) + } + + // The bridge reads JSON-per-line from stdin and writes JSON-per-line to + // stdout; the demux loop feeds a pipe the reader side wraps in bufio. + pr, pw := io.Pipe() + stderrBuf := &syncBuffer{} + // The bridge intentionally outlives any single request ctx and is torn + // down in close() / terminateBridgeLocked — dial under a handshake-bounded + // background context, mirroring the podman backend's noctx bridge exec. + session, err := k.backend.client.execPod(context.Background(), k.backend.cfg.Namespace, podName, sandboxContainerName, + []string{"python3", k8sBridgePath}, true, pw, io.MultiWriter(os.Stderr, stderrBuf)) + if err != nil { + _ = pw.Close() + _ = pr.Close() + return false, fmt.Errorf("start bridge exec: %w", err) + } + // Unblock any reader waiting on the pipe once the session ends — the + // demux loop does not own the pipe writer. + go func() { + <-session.done + _ = pw.CloseWithError(io.EOF) + }() + k.bridge = session + k.bridgeStdout = bufio.NewReader(pr) + k.bridgeStderr = stderrBuf + k.bridgeStarted = true + return true, nil +} + +// terminateBridgeLocked tears the bridge session down and clears the state so +// the next ensureBridge starts fresh. Caller must hold k.mu. Closing the +// websocket ends the server-side exec streams; an orphaned kernel inside a +// still-live pod is reaped by the next bridge start (reap_stale_kernels in +// python_bridge.py), the same recovery story as the podman backend. +func (k *k8sImpl) terminateBridgeLocked() { + if k.bridge != nil { + k.bridge.close() + } + k.bridge = nil + k.bridgeStdout = nil + k.bridgeStarted = false +} + +func (k *k8sImpl) bridgeStderrSuffix() string { + if k.bridgeStderr == nil { + return "" + } + // Pod output — sanitized like all cluster-derived text before it joins an + // error string that upstream code logs. + stderr := sanitizeClusterText(strings.TrimSpace(k.bridgeStderr.Snapshot())) + if stderr == "" { + return "" + } + const maxLen = 1024 + if len(stderr) > maxLen { + stderr = stderr[len(stderr)-maxLen:] + } + return " (bridge stderr: " + stderr + ")" +} + +// resourceUsage reports no telemetry: the k8s backend has no `podman stats` +// counterpart wired up (kubelet metrics need a different collection path). +// Recorded as an honest deviation in docs/DEPLOYMENT-KUBERNETES.md. +func (k *k8sImpl) resourceUsage() (ResourceUsageSummary, bool) { + return ResourceUsageSummary{}, false +} + +func (k *k8sImpl) close() { + k.mu.Lock() + k.terminateBridgeLocked() + k.bridgeStderr = nil + k.mu.Unlock() + + k.podMu.Lock() + podName := k.podName + k.podName = "" + k.podMu.Unlock() + + if podName != "" { + delCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + if err := k.backend.client.deletePod(delCtx, k.backend.cfg.Namespace, podName); err != nil && !isK8sNotFound(err) { + log.Printf("sandbox: close-time pod delete unconfirmed (%s): %v — the pod may linger until the boot-time orphan prune", podName, err) + } + } +} + +// PruneOrphanedPods removes leftover sandbox pods from a prior fleet process +// (a crash never runs close(), so in-flight + warm pods outlive it). The +// kubernetes counterpart of PruneOrphanedContainers, with the same ownership +// discipline: pods carrying THIS process's instance label are never touched +// (the warm pool is already filling when the sweep runs); other instances' +// pods are removed only when their labeled owner pid is provably not running +// here anymore — and since a restarted control plane is a fresh process (in +// k8s, usually a fresh container), a prior incarnation's pods always qualify. +func (b *KubernetesBackend) PruneOrphanedPods(ctx context.Context) (int, error) { + selector := k8sLabelName + "=" + k8sLabelNameValue + "," + k8sLabelManagedBy + "=" + k8sLabelManagedVal + list, err := b.client.listPods(ctx, b.cfg.Namespace, selector) + if err != nil { + return 0, fmt.Errorf("list orphaned sandbox pods: %w", err) + } + removed := 0 + for _, pod := range list.Items { + label := pod.Metadata.Labels[k8sLabelInstance] + if label == k8sInstanceLabel { + continue + } + pid, startedAt, ok := parseK8sInstanceLabel(label) + if ok && labeledOwnerStillRunning(pid, startedAt) { + // A live process in THIS pid namespace owns it (a sibling fleet + // process sharing the namespace) — leave it alone. Leaking a pod is + // recoverable; deleting a live sibling's sandbox mid-turn is not. + continue + } + if err := b.client.deletePod(ctx, b.cfg.Namespace, pod.Metadata.Name); err != nil { + if isK8sNotFound(err) { + continue + } + // The name came back from the API list — sanitized like all + // cluster-derived text before it enters a logged error. + return removed, fmt.Errorf("remove orphaned sandbox pod %s: %w", sanitizeClusterText(pod.Metadata.Name), err) + } + removed++ + } + return removed, nil +} diff --git a/internal/sandbox/k8s_backend_test.go b/internal/sandbox/k8s_backend_test.go new file mode 100644 index 000000000..68dfb7f15 --- /dev/null +++ b/internal/sandbox/k8s_backend_test.go @@ -0,0 +1,677 @@ +package sandbox + +// Tests for the kubernetes sandbox backend (#989): pod lifecycle, exec +// transport, the #796 poison-and-retire containment, fileops through the real +// executor, the pool routing, and the orphan sweep — all against the fake +// apiserver in k8s_fake_test.go. + +import ( + "context" + "errors" + "io" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/gorilla/websocket" +) + +func testContainerConfig(t *testing.T) ContainerConfig { + t.Helper() + return ContainerConfig{ + Image: "registry.example/fleet-sandbox:test", + WorkspaceHostDir: t.TempDir(), + BridgeScript: []byte("# fake bridge\n"), + } +} + +func TestK8sResolveBackend(t *testing.T) { + cases := []struct { + env, bundle, want string + wantErr bool + }{ + {"", "", BackendPodman, false}, + {"podman", "kubernetes", BackendPodman, false}, + {"", "kubernetes", BackendKubernetes, false}, + {"KUBERNETES", "", BackendKubernetes, false}, + {" kubernetes ", "", BackendKubernetes, false}, + {"", "docker", "", true}, + {"k8s", "", "", true}, + } + for _, tc := range cases { + got, err := ResolveBackend(tc.env, tc.bundle) + if tc.wantErr { + if err == nil { + t.Errorf("ResolveBackend(%q,%q): want error, got %q", tc.env, tc.bundle, got) + } + continue + } + if err != nil || got != tc.want { + t.Errorf("ResolveBackend(%q,%q) = %q, %v; want %q", tc.env, tc.bundle, got, err, tc.want) + } + } +} + +func TestK8sPodSpecHardening(t *testing.T) { + cfg := testContainerConfig(t) + cfg.NoNetwork = true + kcfg := KubernetesConfig{ + Namespace: "fleet-sandboxes", + WorkspaceClaim: "fleet-workspace", + ServiceAccount: "fleet-sandbox", + ImagePullSecret: "regcred", + } + pod, err := buildSandboxPod(applyContainerDefaults(cfg), kcfg, "fleet-sandbox-abc") + if err != nil { + t.Fatalf("buildSandboxPod: %v", err) + } + + // These pins are the k8s counterpart of podman_args_test.go: a change that + // weakens the pod hardening must show up as a failing assertion, not slip + // through a spec refactor. + spec := pod.Spec + if spec.AutomountServiceAccountToken == nil || *spec.AutomountServiceAccountToken { + t.Error("automountServiceAccountToken must be explicitly false — a sandbox must never hold apiserver credentials") + } + if spec.SecurityContext == nil || spec.SecurityContext.RunAsNonRoot == nil || !*spec.SecurityContext.RunAsNonRoot { + t.Error("runAsNonRoot must be true") + } + if spec.SecurityContext.SeccompProfile == nil || spec.SecurityContext.SeccompProfile.Type != "RuntimeDefault" { + t.Error("seccompProfile must default to RuntimeDefault") + } + c := spec.Containers[0] + if c.SecurityContext == nil || c.SecurityContext.ReadOnlyRootFilesystem == nil || !*c.SecurityContext.ReadOnlyRootFilesystem { + t.Error("readOnlyRootFilesystem must be true") + } + if c.SecurityContext.AllowPrivilegeEscalation == nil || *c.SecurityContext.AllowPrivilegeEscalation { + t.Error("allowPrivilegeEscalation must be false") + } + if c.SecurityContext.Capabilities == nil || len(c.SecurityContext.Capabilities.Drop) != 1 || c.SecurityContext.Capabilities.Drop[0] != "ALL" { + t.Error("capabilities must drop ALL") + } + if c.WorkingDir != cfg.WorkspaceHostDir { + t.Errorf("workingDir = %q, want the workspace root %q", c.WorkingDir, cfg.WorkspaceHostDir) + } + // Same-path workspace mount — the invariant that keeps MCP paths valid. + foundWorkspace := false + for _, m := range c.VolumeMounts { + if m.Name == "workspace" { + foundWorkspace = true + if m.MountPath != cfg.WorkspaceHostDir { + t.Errorf("workspace mounted at %q, want same-path %q", m.MountPath, cfg.WorkspaceHostDir) + } + } + } + if !foundWorkspace { + t.Error("workspace volume mount missing") + } + // Limit conversions: 512m podman → bytes; 1.0 cpus → 1000m; disk 5 → 5Gi. + if got := c.Resources.Limits["memory"]; got != "536870912" { + t.Errorf("memory limit = %q, want 536870912 (512 MiB in bytes)", got) + } + if got := c.Resources.Limits["cpu"]; got != "1000m" { + t.Errorf("cpu limit = %q, want 1000m", got) + } + if got := c.Resources.Limits["ephemeral-storage"]; got != "5Gi" { + t.Errorf("ephemeral-storage limit = %q, want 5Gi", got) + } + // Sealed posture label + ownership labels. + if got := pod.Metadata.Labels[k8sLabelEgress]; got != "none" { + t.Errorf("egress label = %q, want none for NoNetwork", got) + } + if got := pod.Metadata.Labels[k8sLabelInstance]; got != k8sInstanceLabel { + t.Errorf("instance label = %q, want %q", got, k8sInstanceLabel) + } + if len(spec.ImagePullSecrets) != 1 || spec.ImagePullSecrets[0].Name != "regcred" { + t.Error("imagePullSecrets not carried") + } + if spec.ServiceAccountName != "fleet-sandbox" { + t.Error("serviceAccountName not carried") + } +} + +func TestK8sPodSpecVariants(t *testing.T) { + cfg := applyContainerDefaults(testContainerConfig(t)) + kcfg := KubernetesConfig{Namespace: "ns", WorkspaceClaim: "ws", RuntimeClassName: "kata", SeccompLocalhostProfile: "profiles/fleet.json"} + pod, err := buildSandboxPod(cfg, kcfg, "fleet-sandbox-x") + if err != nil { + t.Fatalf("buildSandboxPod: %v", err) + } + if got := pod.Metadata.Labels[k8sLabelEgress]; got != "open" { + t.Errorf("egress label = %q, want open without NoNetwork", got) + } + if pod.Spec.RuntimeClassName == nil || *pod.Spec.RuntimeClassName != "kata" { + t.Error("runtimeClassName not carried") + } + sp := pod.Spec.SecurityContext.SeccompProfile + if sp.Type != "Localhost" || sp.LocalhostProfile == nil || *sp.LocalhostProfile != "profiles/fleet.json" { + t.Errorf("seccomp profile = %+v, want Localhost profiles/fleet.json", sp) + } + + // A negative disk limit disables the ephemeral-storage cap. + cfg.DiskLimitGB = -1 + pod, err = buildSandboxPod(cfg, kcfg, "fleet-sandbox-y") + if err != nil { + t.Fatalf("buildSandboxPod: %v", err) + } + if _, ok := pod.Spec.Containers[0].Resources.Limits["ephemeral-storage"]; ok { + t.Error("negative DiskLimitGB must not emit an ephemeral-storage limit") + } +} + +func TestK8sSchedulingKnobs(t *testing.T) { + // Parse helpers fail closed on malformed input. + if _, err := ParseK8sNodeSelector("pool"); err == nil { + t.Error("bare key without =value must error") + } + if _, err := ParseK8sNodeSelector("=v"); err == nil { + t.Error("empty key must error") + } + sel, err := ParseK8sNodeSelector(" pool=sandboxes, arch=amd64 ") + if err != nil || sel["pool"] != "sandboxes" || sel["arch"] != "amd64" { + t.Errorf("ParseK8sNodeSelector = %v, %v", sel, err) + } + if _, err := ParseK8sTolerations(`[{"unknown":"field"}]`); err == nil { + t.Error("unknown toleration field must error (strict decoding)") + } + tols, err := ParseK8sTolerations(`[{"key":"fleet.elcanotek.com/sandbox","operator":"Exists","effect":"NoSchedule"}]`) + if err != nil || len(tols) != 1 || tols[0].Key != "fleet.elcanotek.com/sandbox" { + t.Errorf("ParseK8sTolerations = %+v, %v", tols, err) + } + + // They reach the pod spec, and the pull policy is explicit (the API + // default for a :latest tag is Always, which breaks side-loaded images). + cfg := applyContainerDefaults(testContainerConfig(t)) + pod, err := buildSandboxPod(cfg, KubernetesConfig{ + Namespace: "ns", WorkspaceClaim: "ws", + NodeSelector: sel, Tolerations: tols, + }, "fleet-sandbox-sched") + if err != nil { + t.Fatalf("buildSandboxPod: %v", err) + } + if pod.Spec.NodeSelector["pool"] != "sandboxes" { + t.Errorf("nodeSelector not carried: %v", pod.Spec.NodeSelector) + } + if len(pod.Spec.Tolerations) != 1 || pod.Spec.Tolerations[0].Effect != "NoSchedule" { + t.Errorf("tolerations not carried: %+v", pod.Spec.Tolerations) + } + if got := pod.Spec.Containers[0].ImagePullPolicy; got != "IfNotPresent" { + t.Errorf("imagePullPolicy = %q, want explicit IfNotPresent", got) + } +} + +func TestK8sSanitizeClusterText(t *testing.T) { + // Cluster/pod-derived text is newline-stripped before it can enter a + // logged error — a pod printing "\nFAKE LOG LINE" to stderr must not be + // able to forge journal entries (go/log-injection). + got := sanitizeClusterText("line one\r\nFAKE LOG LINE\ntail") + if strings.ContainsAny(got, "\r\n") { + t.Errorf("sanitizeClusterText left line breaks in %q", got) + } + if got != "line one FAKE LOG LINE tail" { + t.Errorf("sanitizeClusterText = %q", got) + } +} + +func TestK8sQuantityConversions(t *testing.T) { + if _, err := k8sQuantityFromPodmanMemory("512x"); err == nil { + t.Error("bad memory suffix must error") + } + if got, _ := k8sQuantityFromPodmanMemory("2g"); got != "2147483648" { + t.Errorf("2g = %q", got) + } + if got, _ := k8sQuantityFromPodmanCPU("2.50"); got != "2500m" { + t.Errorf("2.50 cpus = %q", got) + } + if _, err := k8sQuantityFromPodmanCPU("zero"); err == nil { + t.Error("bad cpu must error") + } +} + +func TestK8sInstanceLabelRoundTrip(t *testing.T) { + pid, start, ok := parseK8sInstanceLabel(k8sInstanceLabel) + if !ok || pid != os.Getpid() || start <= 0 { + t.Fatalf("parseK8sInstanceLabel(%q) = %d, %d, %v", k8sInstanceLabel, pid, start, ok) + } + if _, _, ok := parseK8sInstanceLabel("garbage"); ok { + t.Error("garbage label must not parse") + } +} + +func TestK8sBashExec(t *testing.T) { + fake := newFakeKube(t) + fake.bashBehaviors["echo hi"] = func(_ string, stdout, _ io.Writer, _ *websocket.Conn) int { + _, _ = stdout.Write([]byte("hi\n")) + return 0 + } + fake.bashBehaviors["exit 3"] = func(_ string, _, stderr io.Writer, _ *websocket.Conn) int { + _, _ = stderr.Write([]byte("boom")) + return 3 + } + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + sb, err := backend.newSandbox(context.Background(), testContainerConfig(t)) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + if got := sb.ModeName(); got != "kubernetes" { + t.Errorf("ModeName = %q", got) + } + + res, err := sb.RunBash(context.Background(), BashRequest{Command: "echo hi"}) + if err != nil { + t.Fatalf("RunBash: %v", err) + } + if res.ExitCode != 0 || string(res.Stdout) != "hi\n" { + t.Errorf("RunBash = exit %d stdout %q", res.ExitCode, res.Stdout) + } + + res, err = sb.RunBash(context.Background(), BashRequest{Command: "exit 3", WorkingDir: "/some/dir"}) + if err != nil { + t.Fatalf("RunBash exit 3: %v", err) + } + if res.ExitCode != 3 || string(res.Stderr) != "boom" { + t.Errorf("RunBash exit3 = exit %d stderr %q", res.ExitCode, res.Stderr) + } + fake.mu.Lock() + workdir := fake.lastBashWorkdir + fake.mu.Unlock() + if workdir != "/some/dir" { + t.Errorf("workdir wrapper carried %q, want /some/dir", workdir) + } +} + +func TestK8sBashCancelPoisonsAndDeletesPod(t *testing.T) { + fake := newFakeKube(t) + started := make(chan struct{}, 1) + fake.bashBehaviors["block"] = func(_ string, _, _ io.Writer, conn *websocket.Conn) int { + started <- struct{}{} + // Simulate a process that never exits: hold until the client closes. + for { + if _, _, err := conn.ReadMessage(); err != nil { + return 0 + } + } + } + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + sb, err := backend.newSandbox(context.Background(), testContainerConfig(t)) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + + ctx, cancel := context.WithCancel(context.Background()) + go func() { + <-started + cancel() + }() + res, err := sb.RunBash(ctx, BashRequest{Command: "block", Timeout: time.Minute}) + if err != nil { + t.Fatalf("RunBash: %v", err) + } + if !res.Cancelled || res.TimedOut { + t.Errorf("want Cancelled, got %+v", res) + } + if !res.SandboxRetired || !res.CleanupConfirmed { + t.Errorf("want SandboxRetired+CleanupConfirmed, got %+v", res) + } + if !sb.Poisoned() { + t.Error("sandbox must be poisoned after a cancelled bash call") + } + fake.mu.Lock() + deleted := len(fake.deleted) + fake.mu.Unlock() + if deleted == 0 { + t.Error("cancelled bash must delete the pod (the #796 containment)") + } + if _, err := sb.RunBash(context.Background(), BashRequest{Command: "echo hi"}); !errors.Is(err, ErrPoisoned) { + t.Errorf("post-poison RunBash err = %v, want ErrPoisoned", err) + } +} + +func TestK8sRunPythonBridge(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + sb, err := backend.newSandbox(context.Background(), testContainerConfig(t)) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + + res, err := sb.RunPython(context.Background(), PythonRequest{Code: "1+1"}) + if err != nil { + t.Fatalf("RunPython: %v", err) + } + if res.Status != "ok" || res.Result != "ran: 1+1" { + t.Errorf("RunPython = %+v", res) + } + // Second call reuses the session. + res, err = sb.RunPython(context.Background(), PythonRequest{Code: "2+2"}) + if err != nil { + t.Fatalf("RunPython second: %v", err) + } + if res.Result != "ran: 2+2" { + t.Errorf("RunPython second = %+v", res) + } +} + +func TestK8sFileOpsThroughRealExecutor(t *testing.T) { + if !pythonAvailable() { + t.Skip("python3 not available on the test host") + } + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + cfg := testContainerConfig(t) + cfg.BridgeScript = []byte("# unused\n") + sb, err := backend.newSandbox(context.Background(), cfg) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + + // The fake runs the UPLOADED fileops.py on the host, so the workspace dir + // is a real host directory here. + root := filepath.Join(cfg.WorkspaceHostDir, "conv1") + if err := os.MkdirAll(root, 0o755); err != nil { + t.Fatal(err) + } + if err := sb.BindFileOpRoot(context.Background(), root); err != nil { + t.Fatalf("BindFileOpRoot: %v", err) + } + target := filepath.Join(root, "hello.txt") + if _, err := sb.RunFileOp(context.Background(), FileOpRequest{Op: FileOpWrite, Path: target, Root: root, Data: []byte("hello k8s\n")}); err != nil { + t.Fatalf("write: %v", err) + } + res, err := sb.RunFileOp(context.Background(), FileOpRequest{Op: FileOpRead, Path: target, Root: root, Limit: 1024}) + if err != nil { + t.Fatalf("read: %v", err) + } + if string(res.Data) != "hello k8s\n" { + t.Errorf("read back %q", res.Data) + } + // A root outside every mount is refused before any exec happens. + if _, err := sb.RunFileOp(context.Background(), FileOpRequest{Op: FileOpRead, Path: "/etc/passwd", Root: "/etc"}); !errors.Is(err, ErrFileOpUnsafePath) { + t.Errorf("outside-mount fileop err = %v, want ErrFileOpUnsafePath", err) + } +} + +// End to end through the fake apiserver + the real fileops.py: when the +// operator declares the bundle's doc dirs are in the sandbox image (the agent +// layer then keeps them in ReadOnlyMounts), a read of one resolves and a write +// beneath it is refused. This is the whole point of +// bundle_docs_in_image — view_file on `protocols/…` working in a pod — and it +// must not come with a write path. +func TestK8sFileOpsReadBundleDocsWhenDeclaredInImage(t *testing.T) { + if !pythonAvailable() { + t.Skip("python3 not available on the test host") + } + // Stands in for the path the sandbox image carries the bundle doc dir at; + // the fake execs on the host, so the same path must exist here. + docs := filepath.Join(t.TempDir(), "client", "protocols") + if err := os.MkdirAll(docs, 0o755); err != nil { + t.Fatal(err) + } + protocol := filepath.Join(docs, "deal-creation.yaml") + if err := os.WriteFile(protocol, []byte("steps: [prepare, confirm, create]\n"), 0o644); err != nil { + t.Fatal(err) + } + + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + cfg := testContainerConfig(t) + cfg.BridgeScript = []byte("# unused\n") + cfg.ReadOnlyMounts = []string{docs} + sb, err := backend.newSandbox(context.Background(), cfg) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + + res, err := sb.RunFileOp(context.Background(), FileOpRequest{Op: FileOpRead, Path: protocol, Root: docs, Limit: 1024}) + if err != nil { + t.Fatalf("read declared bundle doc: %v", err) + } + if !strings.Contains(string(res.Data), "prepare") { + t.Errorf("read back %q", res.Data) + } + // Read-only means read-only: the declaration re-admits reads, never writes. + if _, err := sb.RunFileOp(context.Background(), FileOpRequest{Op: FileOpWrite, Path: protocol, Root: docs, Data: []byte("tampered\n")}); !errors.Is(err, ErrFileOpUnsafePath) { + t.Errorf("write beneath a declared doc root err = %v, want ErrFileOpUnsafePath", err) + } + if err := sb.BindFileOpRoot(context.Background(), docs); !errors.Is(err, ErrFileOpUnsafePath) { + t.Errorf("binding a declared doc root as writable err = %v, want ErrFileOpUnsafePath", err) + } + // Without the declaration the agent layer passes no mounts, and the same + // read is refused by the anchor before any exec. + bare := testContainerConfig(t) + bare.BridgeScript = []byte("# unused\n") + sb2, err := backend.newSandbox(context.Background(), bare) + if err != nil { + t.Fatalf("newSandbox (no mounts): %v", err) + } + defer sb2.Close() + if _, err := sb2.RunFileOp(context.Background(), FileOpRequest{Op: FileOpRead, Path: protocol, Root: docs, Limit: 1024}); !errors.Is(err, ErrFileOpUnsafePath) { + t.Errorf("undeclared doc read err = %v, want ErrFileOpUnsafePath", err) + } +} + +func TestK8sPoolRouting(t *testing.T) { + fake := newFakeKube(t) + fake.bashBehaviors["echo pool"] = func(_ string, stdout, _ io.Writer, _ *websocket.Conn) int { + _, _ = stdout.Write([]byte("pool\n")) + return 0 + } + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + pool := NewPool(PoolConfig{ + Mode: ModeKubernetes, + KubernetesBackend: backend, + BridgeScript: []byte("# bridge\n"), + Container: testContainerConfig(t), + }) + defer pool.Close() + + sb, cleanup, err := pool.Take(context.Background()) + if err != nil { + t.Fatalf("Take: %v", err) + } + res, err := sb.RunBash(context.Background(), BashRequest{Command: "echo pool"}) + if err != nil || res.ExitCode != 0 { + t.Fatalf("RunBash via pool: %v, %+v", err, res) + } + cleanup() + + // Lockdown take: sealed pods come from the same backend. + sb, cleanup, err = pool.TakeContainer(context.Background()) + if err != nil { + t.Fatalf("TakeContainer: %v", err) + } + _ = sb + cleanup() + + // Allowlisted egress is refused, fail-closed. + if _, _, err := pool.TakeContainerWithEgress(context.Background(), ResourceOverride{}, []string{"example.com"}); err == nil { + t.Error("TakeContainerWithEgress must fail closed under the kubernetes backend") + } +} + +func TestK8sResourceOverridesReachPodSpec(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + pool := NewPool(PoolConfig{ + Mode: ModeKubernetes, + KubernetesBackend: backend, + BridgeScript: []byte("# bridge\n"), + Container: testContainerConfig(t), + }) + defer pool.Close() + + sb, cleanup, err := pool.TakeContainerWithOverrides(context.Background(), ResourceOverride{MemoryLimit: "1024m", CPULimit: "2.00"}, true) + if err != nil { + t.Fatalf("TakeContainerWithOverrides: %v", err) + } + defer cleanup() + _ = sb + + fake.mu.Lock() + defer fake.mu.Unlock() + if len(fake.pods) != 1 { + t.Fatalf("want 1 pod, have %d", len(fake.pods)) + } + for _, pod := range fake.pods { + limits := pod.Spec.Containers[0].Resources.Limits + if limits["memory"] != "1073741824" { + t.Errorf("override memory = %q, want 1073741824", limits["memory"]) + } + if limits["cpu"] != "2000m" { + t.Errorf("override cpu = %q, want 2000m", limits["cpu"]) + } + if pod.Metadata.Labels[k8sLabelEgress] != "none" { + t.Errorf("sealed take must label egress=none, got %q", pod.Metadata.Labels[k8sLabelEgress]) + } + } +} + +func TestK8sCloseDeletesPod(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + sb, err := backend.newSandbox(context.Background(), testContainerConfig(t)) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + sb.Close() + fake.mu.Lock() + defer fake.mu.Unlock() + if len(fake.pods) != 0 || len(fake.deleted) != 1 { + t.Errorf("Close must delete the pod: %d live, %d deleted", len(fake.pods), len(fake.deleted)) + } +} + +func TestK8sPruneOrphanedPods(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + + addPod := func(name, instance string) { + fake.mu.Lock() + fake.pods[name] = &k8sPod{Metadata: k8sObjectMeta{ + Name: name, + Labels: map[string]string{ + k8sLabelName: k8sLabelNameValue, + k8sLabelManagedBy: k8sLabelManagedVal, + k8sLabelInstance: instance, + }, + }} + fake.mu.Unlock() + } + addPod("fleet-sandbox-own", k8sInstanceLabel) // this process — never touched + addPod("fleet-sandbox-dead", "p999999999-t12345") // dead owner — pruned + // Unlabeled/unparseable ownership: fails "still running" (unparseable pid), + // so it is pruned — in k8s a pod without a live owner in THIS process tree + // is a leftover by construction (single-replica invariant). + addPod("fleet-sandbox-mystery", "not-a-label") + + n, err := backend.PruneOrphanedPods(context.Background()) + if err != nil { + t.Fatalf("PruneOrphanedPods: %v", err) + } + if n != 2 { + t.Errorf("pruned %d, want 2", n) + } + fake.mu.Lock() + defer fake.mu.Unlock() + if _, ok := fake.pods["fleet-sandbox-own"]; !ok { + t.Error("own pod must never be pruned") + } + if _, ok := fake.pods["fleet-sandbox-dead"]; ok { + t.Error("dead-owner pod must be pruned") + } +} + +func TestK8sProxyURLRefused(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + cfg := testContainerConfig(t) + cfg.ProxyURL = "http://127.0.0.1:9999" + if _, err := backend.newSandbox(context.Background(), cfg); err == nil { + t.Error("a ProxyURL (allowlisted egress) must be refused by the kubernetes backend") + } +} + +func TestK8sBridgeUploadVerified(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes"}) + cfg := testContainerConfig(t) + sb, err := backend.newSandbox(context.Background(), cfg) + if err != nil { + t.Fatalf("newSandbox: %v", err) + } + defer sb.Close() + fake.mu.Lock() + defer fake.mu.Unlock() + var podName string + for name := range fake.pods { + podName = name + } + if got := fake.files[podName+":"+k8sBridgePath]; string(got) != string(cfg.BridgeScript) { + t.Errorf("bridge upload = %q, want %q", got, cfg.BridgeScript) + } + if got := fake.files[podName+":"+k8sFileOpsPath]; len(got) == 0 || !strings.Contains(string(got), "fileops.py") { + t.Errorf("fileops upload missing or wrong (%d bytes)", len(got)) + } +} + +// TestParseK8sBundleDocsInImage pins the fail-closed boolean: unset is false +// (fleet assumes nothing about a sandbox image's contents), and a typo must +// refuse to boot rather than read as "keep the anchors". +func TestParseK8sBundleDocsInImage(t *testing.T) { + for _, raw := range []string{"", " "} { + v, err := ParseK8sBundleDocsInImage(raw) + if err != nil || v { + t.Errorf("ParseK8sBundleDocsInImage(%q) = %v, %v; want false, nil", raw, v, err) + } + } + for _, raw := range []string{"true", "TRUE", "1", " true "} { + v, err := ParseK8sBundleDocsInImage(raw) + if err != nil || !v { + t.Errorf("ParseK8sBundleDocsInImage(%q) = %v, %v; want true, nil", raw, v, err) + } + } + for _, raw := range []string{"false", "0"} { + v, err := ParseK8sBundleDocsInImage(raw) + if err != nil || v { + t.Errorf("ParseK8sBundleDocsInImage(%q) = %v, %v; want false, nil", raw, v, err) + } + } + for _, raw := range []string{"ture", "yes-please", "on"} { + if _, err := ParseK8sBundleDocsInImage(raw); err == nil { + t.Errorf("ParseK8sBundleDocsInImage(%q) must error (fail closed)", raw) + } + } +} + +// TestFileOpAnchorSupportingDocsByBackend pins the behavior the +// bundle_docs_in_image declaration exists for: with the supporting-doc mounts +// retained, a bundle doc root anchors a READ-ONLY fileop; with them dropped +// (the kubernetes default, where a pod mounts only the workspace claim), the +// same root is refused before the file is looked for. +func TestFileOpAnchorSupportingDocsByBackend(t *testing.T) { + const ws = "/var/lib/fleet/workspace" + docs := []string{"/opt/fleet/client/protocols", "/opt/fleet/client/skills"} + + for _, root := range docs { + anchor, readOnly, err := fileOpAnchorFor(ws, docs, root) + if err != nil || anchor != root || !readOnly { + t.Errorf("mounts retained: fileOpAnchorFor(%q) = %q, ro=%v, %v; want the root itself, read-only, nil", root, anchor, readOnly, err) + } + if _, _, err := fileOpAnchorFor(ws, nil, root); !errors.Is(err, ErrFileOpUnsafePath) { + t.Errorf("mounts dropped: fileOpAnchorFor(%q) err = %v; want ErrFileOpUnsafePath", root, err) + } + } + + // The workspace claim itself is unaffected either way — it is the one + // mount a sandbox pod always has. + if anchor, readOnly, err := fileOpAnchorFor(ws, nil, ws+"/conv-1"); err != nil || anchor != ws || readOnly { + t.Errorf("workspace anchor = %q, ro=%v, %v; want %q, writable, nil", anchor, readOnly, err, ws) + } +} diff --git a/internal/sandbox/k8s_client.go b/internal/sandbox/k8s_client.go new file mode 100644 index 000000000..ed0d76d0f --- /dev/null +++ b/internal/sandbox/k8s_client.go @@ -0,0 +1,493 @@ +// Copyright (c) 2026 ElcanoTek +// SPDX-License-Identifier: MIT + +package sandbox + +// k8s_client.go is the minimal Kubernetes API client the kubernetes sandbox +// backend (#989) uses to create, inspect, delete, and exec into sandbox Pods. +// +// It is deliberately NOT client-go. The backend needs exactly five verbs — +// create/get/delete/list on pods, plus the exec subresource — and client-go +// would add several dozen modules to a dependency tree that is gated by +// govulncheck and a container CVE scan. Everything here is plain net/http +// against the well-versioned core/v1 REST surface, plus gorilla/websocket +// (already in the tree) for exec streaming. The trade is accepted and +// recorded in the ADR: if the backend ever needs watches, informers, or +// exotic auth, revisit client-go rather than growing this file into one. +// +// Auth is loaded from the standard in-cluster mount +// (/var/run/secrets/kubernetes.io/serviceaccount) when no kubeconfig is +// configured, else from a kubeconfig file supporting token and client-cert +// credentials. exec-plugin and auth-provider kubeconfigs are refused with an +// actionable error — the control plane runs unattended, so credentials that +// shell out to an interactive helper cannot work anyway. Fail closed, never +// guess. + +import ( + "bytes" + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "net/http" + "net/url" + "os" + "path" + "strings" + "sync" + "time" +) + +// inClusterTokenFile / inClusterCAFile / inClusterNamespaceFile are the +// standard projected service-account mount every Pod gets. The token file is +// re-read per request (see bearerToken) because bound tokens rotate — a +// long-lived fleet process holding the boot-time token would start getting +// 401s about an hour in. +const ( + inClusterTokenFile = "/var/run/secrets/kubernetes.io/serviceaccount/token" //nolint:gosec // well-known mount path, not a credential + inClusterCAFile = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" + inClusterNamespaceFile = "/var/run/secrets/kubernetes.io/serviceaccount/namespace" +) + +// k8sRequestTimeout bounds any single non-streaming API request. Generous for +// a loaded apiserver; short enough that a dead one fails a turn promptly +// instead of hanging it. +const k8sRequestTimeout = 30 * time.Second + +// k8sClient is the minimal REST client. Safe for concurrent use. +type k8sClient struct { + baseURL *url.URL + httpc *http.Client + // tlsConfig is retained for the websocket dialer (exec), which cannot + // share http.Transport. + tlsConfig *tls.Config + + // Exactly one of the following is set. staticToken is a kubeconfig token; + // tokenFile is re-read per request so rotated bound tokens keep working. + // Client-cert auth lives inside tlsConfig and needs neither. + staticToken string + tokenFile string + + tokenMu sync.Mutex + cachedToken string + tokenRead time.Time +} + +// tokenRefreshInterval is how long a token-file read is trusted before the +// file is consulted again. Kubernetes rotates bound tokens well before their +// ~1h expiry, so a 1-minute cache never serves a stale token while keeping +// the common path free of file I/O. +const tokenRefreshInterval = time.Minute + +// bearerToken returns the Authorization bearer value for a request, or "" +// when client-cert auth is in use. +func (c *k8sClient) bearerToken() (string, error) { + if c.staticToken != "" { + return c.staticToken, nil + } + if c.tokenFile == "" { + return "", nil + } + c.tokenMu.Lock() + defer c.tokenMu.Unlock() + if c.cachedToken != "" && time.Since(c.tokenRead) < tokenRefreshInterval { + return c.cachedToken, nil + } + raw, err := os.ReadFile(c.tokenFile) + if err != nil { + return "", fmt.Errorf("read service-account token: %w", err) + } + c.cachedToken = strings.TrimSpace(string(raw)) + c.tokenRead = time.Now() + return c.cachedToken, nil +} + +// newInClusterClient builds a client from the standard service-account mount. +func newInClusterClient() (*k8sClient, error) { + host, port := os.Getenv("KUBERNETES_SERVICE_HOST"), os.Getenv("KUBERNETES_SERVICE_PORT") + if host == "" || port == "" { + return nil, fmt.Errorf("not running in a cluster (KUBERNETES_SERVICE_HOST/PORT unset) and no kubeconfig configured") + } + caPEM, err := os.ReadFile(inClusterCAFile) + if err != nil { + return nil, fmt.Errorf("read in-cluster CA: %w", err) + } + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(caPEM) { + return nil, fmt.Errorf("in-cluster CA at %s contains no usable certificates", inClusterCAFile) + } + if _, err := os.Stat(inClusterTokenFile); err != nil { + return nil, fmt.Errorf("in-cluster service-account token: %w", err) + } + base, err := url.Parse("https://" + net.JoinHostPort(host, port)) + if err != nil { + return nil, fmt.Errorf("parse in-cluster apiserver address: %w", err) + } + tlsCfg := &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12} + return &k8sClient{ + baseURL: base, + tlsConfig: tlsCfg, + tokenFile: inClusterTokenFile, + httpc: &http.Client{ + Transport: &http.Transport{TLSClientConfig: tlsCfg}, + }, + }, nil +} + +// inClusterNamespace reads the namespace the control plane itself runs in, +// used only as a default when the sandbox namespace is not configured. +func inClusterNamespace() string { + raw, err := os.ReadFile(inClusterNamespaceFile) + if err != nil { + return "" + } + return strings.TrimSpace(string(raw)) +} + +// sanitizeClusterText strips newlines from text that originated in the +// cluster API or a pod (status messages, exec status, stderr snippets) before +// it is embedded in an error or log line. Everything remote-derived leaves +// this package through error strings that end up in log.Printf sites all over +// the codebase, and a forged newline in that text is a log-injection vector +// (CodeQL go/log-injection): without this, a pod that prints a crafted line +// to stderr could fabricate whole log entries in the operator's journal. +func sanitizeClusterText(s string) string { + s = strings.ReplaceAll(s, "\r", " ") + s = strings.ReplaceAll(s, "\n", " ") + return s +} + +// k8sStatusError is a non-2xx API response, carrying enough of the +// metav1.Status body to be actionable in logs and boot errors. +type k8sStatusError struct { + Code int + Reason string + Message string +} + +func (e *k8sStatusError) Error() string { + if e.Message != "" { + return fmt.Sprintf("kubernetes API error %d (%s): %s", e.Code, e.Reason, e.Message) + } + return fmt.Sprintf("kubernetes API error %d (%s)", e.Code, e.Reason) +} + +// isK8sNotFound reports whether err is a 404 from the API — the pod-already- +// gone case teardown paths treat as success, mirroring containerAlreadyGone. +func isK8sNotFound(err error) bool { + var se *k8sStatusError + return errors.As(err, &se) && se.Code == http.StatusNotFound +} + +// do performs one JSON API request. body may be nil. A non-2xx response is +// returned as *k8sStatusError with the server's status message decoded. +func (c *k8sClient) do(ctx context.Context, method, apiPath string, query url.Values, body []byte) ([]byte, error) { + reqCtx, cancel := context.WithTimeout(ctx, k8sRequestTimeout) + defer cancel() + + u := *c.baseURL + u.Path = path.Join(u.Path, apiPath) + if query != nil { + u.RawQuery = query.Encode() + } + var rdr io.Reader + if body != nil { + rdr = bytes.NewReader(body) + } + req, err := http.NewRequestWithContext(reqCtx, method, u.String(), rdr) + if err != nil { + return nil, fmt.Errorf("build %s %s: %w", method, apiPath, err) + } + if body != nil { + req.Header.Set("Content-Type", "application/json") + } + req.Header.Set("Accept", "application/json") + token, err := c.bearerToken() + if err != nil { + return nil, err + } + if token != "" { + req.Header.Set("Authorization", "Bearer "+token) + } + resp, err := c.httpc.Do(req) + if err != nil { + return nil, fmt.Errorf("%s %s: %w", method, apiPath, err) + } + defer func() { _ = resp.Body.Close() }() + // Responses are bounded reads: pod objects are a few KB; even a large list + // stays far under this. Guards against a misbehaving endpoint, not real use. + data, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20)) + if err != nil { + return nil, fmt.Errorf("%s %s: read response: %w", method, apiPath, err) + } + if resp.StatusCode < 200 || resp.StatusCode > 299 { + var status struct { + Reason string `json:"reason"` + Message string `json:"message"` + } + _ = json.Unmarshal(data, &status) + // Sanitized at construction so every path that logs this error — + // boot preflights, pool fill, the prune sweep — is covered at once. + return nil, &k8sStatusError{ + Code: resp.StatusCode, + Reason: sanitizeClusterText(status.Reason), + Message: sanitizeClusterText(status.Message), + } + } + return data, nil +} + +// ── typed pod surface (the narrow slice of core/v1 the backend touches) ── + +type k8sPod struct { + Metadata k8sObjectMeta `json:"metadata"` + Spec k8sPodSpec `json:"spec"` + Status k8sPodStatus `json:"status,omitempty"` + + // APIVersion/Kind are emitted on create; ignored on read. + APIVersion string `json:"apiVersion,omitempty"` + Kind string `json:"kind,omitempty"` +} + +type k8sObjectMeta struct { + Name string `json:"name"` + Namespace string `json:"namespace,omitempty"` + Labels map[string]string `json:"labels,omitempty"` +} + +type k8sPodSpec struct { + RestartPolicy string `json:"restartPolicy,omitempty"` + AutomountServiceAccountToken *bool `json:"automountServiceAccountToken,omitempty"` + EnableServiceLinks *bool `json:"enableServiceLinks,omitempty"` + TerminationGracePeriodSeconds *int64 `json:"terminationGracePeriodSeconds,omitempty"` + ServiceAccountName string `json:"serviceAccountName,omitempty"` + RuntimeClassName *string `json:"runtimeClassName,omitempty"` + ImagePullSecrets []k8sLocalObjRef `json:"imagePullSecrets,omitempty"` + SecurityContext *k8sPodSecurityCtx `json:"securityContext,omitempty"` + Containers []k8sContainer `json:"containers"` + Volumes []k8sVolume `json:"volumes,omitempty"` + NodeSelector map[string]string `json:"nodeSelector,omitempty"` + Tolerations []k8sToleration `json:"tolerations,omitempty"` +} + +type k8sLocalObjRef struct { + Name string `json:"name"` +} + +type k8sToleration struct { + Key string `json:"key,omitempty"` + Operator string `json:"operator,omitempty"` + Value string `json:"value,omitempty"` + Effect string `json:"effect,omitempty"` +} + +type k8sPodSecurityCtx struct { + RunAsNonRoot *bool `json:"runAsNonRoot,omitempty"` + RunAsUser *int64 `json:"runAsUser,omitempty"` + RunAsGroup *int64 `json:"runAsGroup,omitempty"` + FSGroup *int64 `json:"fsGroup,omitempty"` + SeccompProfile *k8sSeccompProfile `json:"seccompProfile,omitempty"` +} + +type k8sSeccompProfile struct { + Type string `json:"type"` + LocalhostProfile *string `json:"localhostProfile,omitempty"` +} + +type k8sContainer struct { + Name string `json:"name"` + Image string `json:"image"` + ImagePullPolicy string `json:"imagePullPolicy,omitempty"` + Command []string `json:"command,omitempty"` + WorkingDir string `json:"workingDir,omitempty"` + SecurityContext *k8sContainerSecCtx `json:"securityContext,omitempty"` + Resources *k8sResources `json:"resources,omitempty"` + VolumeMounts []k8sVolumeMount `json:"volumeMounts,omitempty"` +} + +type k8sContainerSecCtx struct { + AllowPrivilegeEscalation *bool `json:"allowPrivilegeEscalation,omitempty"` + ReadOnlyRootFilesystem *bool `json:"readOnlyRootFilesystem,omitempty"` + Capabilities *k8sCapabilities `json:"capabilities,omitempty"` +} + +type k8sCapabilities struct { + Drop []string `json:"drop,omitempty"` +} + +type k8sResources struct { + Limits map[string]string `json:"limits,omitempty"` + Requests map[string]string `json:"requests,omitempty"` +} + +type k8sVolumeMount struct { + Name string `json:"name"` + MountPath string `json:"mountPath"` + ReadOnly bool `json:"readOnly,omitempty"` + SubPath string `json:"subPath,omitempty"` +} + +type k8sVolume struct { + Name string `json:"name"` + EmptyDir *k8sEmptyDir `json:"emptyDir,omitempty"` + PersistentVolumeClaim *k8sPVCVolSource `json:"persistentVolumeClaim,omitempty"` +} + +type k8sEmptyDir struct { + SizeLimit string `json:"sizeLimit,omitempty"` + Medium string `json:"medium,omitempty"` +} + +type k8sPVCVolSource struct { + ClaimName string `json:"claimName"` + ReadOnly bool `json:"readOnly,omitempty"` +} + +type k8sPodStatus struct { + Phase string `json:"phase,omitempty"` + Reason string `json:"reason,omitempty"` + Message string `json:"message,omitempty"` + ContainerStatuses []k8sContainerStatus `json:"containerStatuses,omitempty"` +} + +type k8sContainerStatus struct { + Name string `json:"name"` + Ready bool `json:"ready"` + State struct { + Waiting *struct { + Reason string `json:"reason,omitempty"` + Message string `json:"message,omitempty"` + } `json:"waiting,omitempty"` + } `json:"state,omitempty"` +} + +func (c *k8sClient) createPod(ctx context.Context, namespace string, pod *k8sPod) error { + pod.APIVersion, pod.Kind = "v1", "Pod" + body, err := json.Marshal(pod) + if err != nil { + return fmt.Errorf("marshal pod: %w", err) + } + _, err = c.do(ctx, http.MethodPost, "/api/v1/namespaces/"+namespace+"/pods", nil, body) + return err +} + +func (c *k8sClient) getPod(ctx context.Context, namespace, name string) (*k8sPod, error) { + data, err := c.do(ctx, http.MethodGet, "/api/v1/namespaces/"+namespace+"/pods/"+name, nil, nil) + if err != nil { + return nil, err + } + var pod k8sPod + if err := json.Unmarshal(data, &pod); err != nil { + return nil, fmt.Errorf("decode pod: %w", err) + } + return &pod, nil +} + +// deletePod removes a pod immediately (gracePeriodSeconds=0). The sandbox +// image's PID 1 is `sleep`, which has no state worth a graceful drain, and +// the #796 poison path NEEDS the hard kill: a straggler must not get grace +// time to finish a side effect. +func (c *k8sClient) deletePod(ctx context.Context, namespace, name string) error { + body := []byte(`{"apiVersion":"v1","kind":"DeleteOptions","gracePeriodSeconds":0,"propagationPolicy":"Background"}`) + _, err := c.do(ctx, http.MethodDelete, "/api/v1/namespaces/"+namespace+"/pods/"+name, nil, body) + return err +} + +type k8sPodList struct { + Items []k8sPod `json:"items"` +} + +func (c *k8sClient) listPods(ctx context.Context, namespace, labelSelector string) (*k8sPodList, error) { + q := url.Values{} + if labelSelector != "" { + q.Set("labelSelector", labelSelector) + } + data, err := c.do(ctx, http.MethodGet, "/api/v1/namespaces/"+namespace+"/pods", q, nil) + if err != nil { + return nil, err + } + var list k8sPodList + if err := json.Unmarshal(data, &list); err != nil { + return nil, fmt.Errorf("decode pod list: %w", err) + } + return &list, nil +} + +// getNetworkPolicy fetches one networking.k8s.io/v1 NetworkPolicy, used only +// by the boot preflight to verify the sealed-egress policy object exists. +func (c *k8sClient) getNetworkPolicy(ctx context.Context, namespace, name string) error { + _, err := c.do(ctx, http.MethodGet, "/apis/networking.k8s.io/v1/namespaces/"+namespace+"/networkpolicies/"+name, nil, nil) + return err +} + +// getPVC fetches one PersistentVolumeClaim, used only by the boot preflight to +// verify the shared workspace claim exists before the first pod references it. +func (c *k8sClient) getPVC(ctx context.Context, namespace, name string) error { + _, err := c.do(ctx, http.MethodGet, "/api/v1/namespaces/"+namespace+"/persistentvolumeclaims/"+name, nil, nil) + return err +} + +// getRuntimeClass fetches one node.k8s.io/v1 RuntimeClass (cluster-scoped), +// used only by the boot preflight when a runtime class is configured. +func (c *k8sClient) getRuntimeClass(ctx context.Context, name string) error { + _, err := c.do(ctx, http.MethodGet, "/apis/node.k8s.io/v1/runtimeclasses/"+name, nil, nil) + return err +} + +// selfSubjectAccessReview asks the apiserver whether the client's identity can +// perform verb on resource (optionally subresource) in namespace. Used by the +// boot preflight so a missing RBAC grant fails at start with a precise message +// instead of at the first turn. +func (c *k8sClient) selfSubjectAccessReview(ctx context.Context, namespace, verb, resource, subresource string) (bool, error) { + review := map[string]any{ + "apiVersion": "authorization.k8s.io/v1", + "kind": "SelfSubjectAccessReview", + "spec": map[string]any{ + "resourceAttributes": map[string]any{ + "namespace": namespace, + "verb": verb, + "resource": resource, + "subresource": subresource, + "group": "", + }, + }, + } + body, err := json.Marshal(review) + if err != nil { + return false, fmt.Errorf("marshal access review: %w", err) + } + data, err := c.do(ctx, http.MethodPost, "/apis/authorization.k8s.io/v1/selfsubjectaccessreviews", nil, body) + if err != nil { + return false, err + } + var resp struct { + Status struct { + Allowed bool `json:"allowed"` + Reason string `json:"reason,omitempty"` + } `json:"status"` + } + if err := json.Unmarshal(data, &resp); err != nil { + return false, fmt.Errorf("decode access review: %w", err) + } + return resp.Status.Allowed, nil +} + +// serverVersion fetches /version — the cheapest authenticated "is the +// apiserver reachable and are my credentials valid" probe the preflight runs. +func (c *k8sClient) serverVersion(ctx context.Context) (string, error) { + data, err := c.do(ctx, http.MethodGet, "/version", nil, nil) + if err != nil { + return "", err + } + var v struct { + GitVersion string `json:"gitVersion"` + } + if err := json.Unmarshal(data, &v); err != nil { + return "", fmt.Errorf("decode /version: %w", err) + } + return sanitizeClusterText(v.GitVersion), nil +} diff --git a/internal/sandbox/k8s_exec.go b/internal/sandbox/k8s_exec.go new file mode 100644 index 000000000..1a0a72e28 --- /dev/null +++ b/internal/sandbox/k8s_exec.go @@ -0,0 +1,283 @@ +// Copyright (c) 2026 ElcanoTek +// SPDX-License-Identifier: MIT + +package sandbox + +// k8s_exec.go streams pod exec sessions for the kubernetes sandbox backend +// over the apiserver's WebSocket channel protocol (v4.channel.k8s.io): each +// binary frame's first byte names a channel — 0 stdin (client→server), +// 1 stdout, 2 stderr, 3 error/status (server→client) — and the server closes +// the connection when the exec'd process exits, after publishing a +// metav1.Status on channel 3 carrying the exit code. +// +// v4 is the oldest protocol every supported apiserver speaks; it cannot +// half-close stdin, so an exec whose process reads stdin TO EOF must bound +// the read itself — the backend wraps such commands in `head -c ` (see +// k8s_backend.go) rather than depending on the newer v5 close channel. + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "path" + "strconv" + "sync" + "time" + + "github.com/gorilla/websocket" +) + +const ( + k8sExecProtocolV4 = "v4.channel.k8s.io" + + k8sChannelStdin = 0 + k8sChannelStdout = 1 + k8sChannelStderr = 2 + k8sChannelError = 3 + + // k8sExecHandshakeTimeout bounds the WebSocket dial+upgrade. + k8sExecHandshakeTimeout = 15 * time.Second + + // k8sStdinChunk bounds a single stdin frame. Large writes (a bridge + // request embedding a big code cell, a fileop write payload) are split so + // no single frame approaches server message-size limits. + k8sStdinChunk = 512 * 1024 +) + +// k8sExecSession is one live exec connection. Writes go to the process's +// stdin; the background read loop demultiplexes stdout/stderr/status frames +// into the sinks given at dial time. done is closed when the read loop ends; +// result() then reports the exec's outcome. +type k8sExecSession struct { + conn *websocket.Conn + + writeMu sync.Mutex + + done chan struct{} + + mu sync.Mutex + exitCode int + execErr error +} + +// execPod dials the exec subresource for the named pod and starts the +// background demux loop. stdout/stderr sinks must be goroutine-safe or owned +// solely by the loop. withStdin controls whether the server keeps a stdin +// channel open (a stdin-less exec gives the process an immediately-EOF stdin, +// matching the podman backend's unset cmd.Stdin). +func (c *k8sClient) execPod(ctx context.Context, namespace, pod, container string, command []string, withStdin bool, stdout, stderr io.Writer) (*k8sExecSession, error) { + q := url.Values{} + q.Set("container", container) + q.Set("stdout", "true") + q.Set("stderr", "true") + q.Set("tty", "false") + q.Set("stdin", strconv.FormatBool(withStdin)) + for _, arg := range command { + q.Add("command", arg) + } + u := *c.baseURL + switch u.Scheme { + case "https": + u.Scheme = "wss" + case "http": + u.Scheme = "ws" + } + u.Path = path.Join(u.Path, "/api/v1/namespaces/"+namespace+"/pods/"+pod+"/exec") + u.RawQuery = q.Encode() + + header := http.Header{} + token, err := c.bearerToken() + if err != nil { + return nil, err + } + if token != "" { + header.Set("Authorization", "Bearer "+token) + } + dialer := &websocket.Dialer{ + TLSClientConfig: c.tlsConfig, + Subprotocols: []string{k8sExecProtocolV4}, + HandshakeTimeout: k8sExecHandshakeTimeout, + } + conn, resp, err := dialer.DialContext(ctx, u.String(), header) + if err != nil { + if resp != nil { + // The upgrade response body carries the apiserver's status message + // (RBAC denial, container not found) — surface it, bounded and + // newline-sanitized (cluster-derived text ends up in logs). + body, _ := io.ReadAll(io.LimitReader(resp.Body, 2048)) + _ = resp.Body.Close() + return nil, fmt.Errorf("pod exec dial: %w (HTTP %d: %.500s)", err, resp.StatusCode, sanitizeClusterText(string(body))) + } + return nil, fmt.Errorf("pod exec dial: %w", err) + } + + s := &k8sExecSession{conn: conn, done: make(chan struct{}), exitCode: -1} + go s.readLoop(stdout, stderr) + return s, nil +} + +// readLoop demultiplexes server frames until the connection ends, then +// records the outcome. A clean close after a Success status yields exit 0; a +// NonZeroExitCode status yields the process's code; a connection that ends +// without any status is an error (the process outcome is unknown). +func (s *k8sExecSession) readLoop(stdout, stderr io.Writer) { + defer close(s.done) + var status []byte + sawStatus := false + for { + msgType, data, err := s.conn.ReadMessage() + if err != nil { + code, execErr := parseExecStatus(status, sawStatus, err) + s.mu.Lock() + s.exitCode, s.execErr = code, execErr + s.mu.Unlock() + return + } + if msgType != websocket.BinaryMessage && msgType != websocket.TextMessage { + continue + } + if len(data) == 0 { + continue + } + payload := data[1:] + switch data[0] { + case k8sChannelStdout: + if stdout != nil && len(payload) > 0 { + _, _ = stdout.Write(payload) + } + case k8sChannelStderr: + if stderr != nil && len(payload) > 0 { + _, _ = stderr.Write(payload) + } + case k8sChannelError: + sawStatus = true + status = append(status, payload...) + } + } +} + +// parseExecStatus turns the channel-3 metav1.Status (if any) plus the read +// error that ended the loop into (exitCode, err). Only a normal-closure / +// EOF-family end with a parsed status is a trustworthy outcome. +func parseExecStatus(status []byte, sawStatus bool, readErr error) (int, error) { + if !sawStatus { + if websocket.IsCloseError(readErr, websocket.CloseNormalClosure) { + // Some proxies drop the status frame on a zero-exit process; treat a + // clean close without status as success — the failure directions + // (non-zero exit, kill, RBAC) all DO produce a status or an abnormal + // close, so this cannot mask them. + return 0, nil + } + // %s of the sanitized text, not %w: a close error's reason text is + // server-supplied (remote), and nothing upstream matches on the + // wrapped type — losing the chain costs nothing here. + return -1, fmt.Errorf("pod exec ended without a status frame: %s", sanitizeClusterText(readErr.Error())) + } + var st struct { + Status string `json:"status"` + Reason string `json:"reason"` + Message string `json:"message"` + Details struct { + Causes []struct { + Reason string `json:"reason"` + Message string `json:"message"` + } `json:"causes"` + } `json:"details"` + } + if err := json.Unmarshal(status, &st); err != nil { + return -1, fmt.Errorf("parse pod exec status: %w (raw: %.200s)", err, sanitizeClusterText(string(status))) + } + // Every message below is cluster-derived text that ends up in logged + // errors — sanitized like everything else that leaves this package. + switch { + case st.Status == "Success": + return 0, nil + case st.Reason == "NonZeroExitCode": + for _, cause := range st.Details.Causes { + if cause.Reason == "ExitCode" { + code, err := strconv.Atoi(cause.Message) + if err != nil { + return -1, fmt.Errorf("parse pod exec exit code %q: %w", sanitizeClusterText(cause.Message), err) + } + return code, nil + } + } + return -1, fmt.Errorf("pod exec reported NonZeroExitCode without an ExitCode cause: %s", sanitizeClusterText(st.Message)) + default: + // A failure that is not an exit code: the exec itself failed (command + // not found in a way the shell couldn't report, container gone, …). + return -1, fmt.Errorf("pod exec failed: %s", sanitizeClusterText(st.Message)) + } +} + +// writeStdin sends bytes to the process's stdin, chunked. Goroutine-safe. +func (s *k8sExecSession) writeStdin(p []byte) error { + s.writeMu.Lock() + defer s.writeMu.Unlock() + for len(p) > 0 { + n := len(p) + if n > k8sStdinChunk { + n = k8sStdinChunk + } + // No manual size arithmetic (`1+n` trips CodeQL's + // allocation-size-overflow check, exactly like podmanArgs' old + // `len(rest)+1`); append sizes the backing array itself. + frame := append([]byte{k8sChannelStdin}, p[:n]...) + if err := s.conn.WriteMessage(websocket.BinaryMessage, frame); err != nil { + return fmt.Errorf("write pod exec stdin: %w", err) + } + p = p[n:] + } + return nil +} + +// wait blocks until the exec ends or ctx is done. On ctx expiry the +// connection is torn down (which unblocks the read loop) and ctx's error is +// returned — the PROCESS inside the pod may still be running; the caller owns +// the #796 containment (delete the pod, poison the sandbox). +func (s *k8sExecSession) wait(ctx context.Context) (int, error) { + select { + case <-ctx.Done(): + _ = s.conn.Close() + <-s.done + return -1, ctx.Err() + case <-s.done: + s.mu.Lock() + defer s.mu.Unlock() + return s.exitCode, s.execErr + } +} + +// close tears the connection down and waits for the read loop to exit. +func (s *k8sExecSession) close() { + _ = s.conn.Close() + <-s.done +} + +// runOneShotExec execs command in the pod, optionally feeding stdin, and +// waits for it to finish. ctx bounds the whole call. The command must +// consume a bounded stdin (v4 cannot signal stdin EOF): callers wrap +// stdin-to-EOF readers in `head -c `. +func (c *k8sClient) runOneShotExec(ctx context.Context, namespace, pod, container string, command []string, stdin []byte, stdout, stderr io.Writer) (int, error) { + session, err := c.execPod(ctx, namespace, pod, container, command, len(stdin) > 0, stdout, stderr) + if err != nil { + return -1, err + } + defer session.close() + if len(stdin) > 0 { + if err := session.writeStdin(stdin); err != nil { + // The write can fail because the process already exited (its outcome + // frame may still be in flight) — fall through to wait, which reports + // the authoritative result; surface the write error only if the exec + // outcome is itself unusable. + if code, werr := session.wait(ctx); werr == nil { + return code, nil + } + return -1, err + } + } + return session.wait(ctx) +} diff --git a/internal/sandbox/k8s_fake_test.go b/internal/sandbox/k8s_fake_test.go new file mode 100644 index 000000000..0cde74f7a --- /dev/null +++ b/internal/sandbox/k8s_fake_test.go @@ -0,0 +1,481 @@ +package sandbox + +// k8s_fake_test.go is the fake Kubernetes apiserver the kubernetes-backend +// tests run against: enough of the core/v1 REST surface (pods CRUD, access +// reviews, the preflight objects) plus a v4.channel.k8s.io WebSocket exec +// endpoint whose "processes" are Go handlers. File uploads store bytes; the +// fileops exec pipes them through a REAL host python3 running the uploaded +// script, so the k8s transport is tested against the genuine executor. + +import ( + "bytes" + "encoding/base64" + "encoding/json" + "encoding/pem" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "regexp" + "strconv" + "strings" + "sync" + "testing" + "time" + + "github.com/gorilla/websocket" +) + +const fakeKubeToken = "test-token" + +type fakeKube struct { + t *testing.T + srv *httptest.Server + + mu sync.Mutex + pods map[string]*k8sPod // name → object (status stamped Running/ready) + files map[string][]byte // ":" → uploaded bytes + deleted []string + + // Failure injection for preflight tests. + denied map[string]bool // " [/]" → deny + noPVC bool + noNetpol bool + noRuntimeClass bool + + // bashBehaviors maps a bash command string to its fake process. The + // handler receives the parsed workdir ("" when the call carried none). + bashBehaviors map[string]func(workdir string, stdout, stderr io.Writer, conn *websocket.Conn) int + + // lastBashWorkdir records the workdir the most recent bash exec carried. + lastBashWorkdir string +} + +func newFakeKube(t *testing.T) *fakeKube { + t.Helper() + f := &fakeKube{ + t: t, + pods: make(map[string]*k8sPod), + files: make(map[string][]byte), + denied: make(map[string]bool), + bashBehaviors: make(map[string]func(string, io.Writer, io.Writer, *websocket.Conn) int), + } + f.srv = httptest.NewTLSServer(http.HandlerFunc(f.handle)) + t.Cleanup(f.srv.Close) + return f +} + +// kubeconfigPath writes a kubeconfig pointing at the fake server (token auth, +// CA pinned to the httptest certificate) and returns its path. +func (f *fakeKube) kubeconfigPath(t *testing.T) string { + t.Helper() + caPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: f.srv.Certificate().Raw}) + dir := t.TempDir() + path := filepath.Join(dir, "kubeconfig") + content := fmt.Sprintf(`apiVersion: v1 +kind: Config +current-context: fake +contexts: + - name: fake + context: + cluster: fake + user: fake +clusters: + - name: fake + cluster: + server: %s + certificate-authority-data: %s +users: + - name: fake + user: + token: %s +`, f.srv.URL, base64.StdEncoding.EncodeToString(caPEM), fakeKubeToken) + if err := os.WriteFile(path, []byte(content), 0o600); err != nil { + t.Fatalf("write kubeconfig: %v", err) + } + return path +} + +// backend builds a KubernetesBackend against the fake server. +func (f *fakeKube) backend(t *testing.T, cfg KubernetesConfig) *KubernetesBackend { + t.Helper() + cfg.KubeconfigPath = f.kubeconfigPath(t) + if cfg.WorkspaceClaim == "" { + cfg.WorkspaceClaim = "fleet-workspace" + } + b, err := NewKubernetesBackend(cfg) + if err != nil { + t.Fatalf("NewKubernetesBackend: %v", err) + } + return b +} + +func (f *fakeKube) authorized(r *http.Request) bool { + return r.Header.Get("Authorization") == "Bearer "+fakeKubeToken +} + +var ( + podPathRe = regexp.MustCompile(`^/api/v1/namespaces/([^/]+)/pods/([^/]+)$`) + podExecRe = regexp.MustCompile(`^/api/v1/namespaces/([^/]+)/pods/([^/]+)/exec$`) + podListRe = regexp.MustCompile(`^/api/v1/namespaces/([^/]+)/pods$`) + pvcPathRe = regexp.MustCompile(`^/api/v1/namespaces/([^/]+)/persistentvolumeclaims/([^/]+)$`) + netpolRe = regexp.MustCompile(`^/apis/networking\.k8s\.io/v1/namespaces/([^/]+)/networkpolicies/([^/]+)$`) + runtimeClsRe = regexp.MustCompile(`^/apis/node\.k8s\.io/v1/runtimeclasses/([^/]+)$`) +) + +func writeK8sStatus(w http.ResponseWriter, code int, reason, msg string) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(code) + _ = json.NewEncoder(w).Encode(map[string]any{ + "kind": "Status", "apiVersion": "v1", "reason": reason, "message": msg, "code": code, + }) +} + +func (f *fakeKube) handle(w http.ResponseWriter, r *http.Request) { + if !f.authorized(r) { + writeK8sStatus(w, http.StatusUnauthorized, "Unauthorized", "bad token") + return + } + path := r.URL.Path + switch { + case path == "/version": + _, _ = w.Write([]byte(`{"gitVersion":"v1.31.0-fake"}`)) + case path == "/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": + f.handleAccessReview(w, r) + case podExecRe.MatchString(path): + f.handleExec(w, r) + case podPathRe.MatchString(path): + f.handlePod(w, r) + case podListRe.MatchString(path): + if r.Method == http.MethodPost { + f.handleCreatePod(w, r) + return + } + f.handleListPods(w, r) + case pvcPathRe.MatchString(path): + if f.noPVC { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "pvc not found") + return + } + _, _ = w.Write([]byte(`{"kind":"PersistentVolumeClaim"}`)) + case netpolRe.MatchString(path): + if f.noNetpol { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "networkpolicy not found") + return + } + _, _ = w.Write([]byte(`{"kind":"NetworkPolicy"}`)) + case runtimeClsRe.MatchString(path): + if f.noRuntimeClass { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "runtimeclass not found") + return + } + _, _ = w.Write([]byte(`{"kind":"RuntimeClass"}`)) + default: + writeK8sStatus(w, http.StatusNotFound, "NotFound", "no fake route for "+path) + } +} + +func (f *fakeKube) handleAccessReview(w http.ResponseWriter, r *http.Request) { + var review struct { + Spec struct { + ResourceAttributes struct { + Verb string `json:"verb"` + Resource string `json:"resource"` + Subresource string `json:"subresource"` + } `json:"resourceAttributes"` + } `json:"spec"` + } + _ = json.NewDecoder(r.Body).Decode(&review) + key := review.Spec.ResourceAttributes.Verb + " " + review.Spec.ResourceAttributes.Resource + if review.Spec.ResourceAttributes.Subresource != "" { + key += "/" + review.Spec.ResourceAttributes.Subresource + } + f.mu.Lock() + allowed := !f.denied[key] + f.mu.Unlock() + _ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"allowed": allowed}}) +} + +func (f *fakeKube) handleCreatePod(w http.ResponseWriter, r *http.Request) { + var pod k8sPod + if err := json.NewDecoder(r.Body).Decode(&pod); err != nil { + writeK8sStatus(w, http.StatusBadRequest, "BadRequest", err.Error()) + return + } + pod.Status = k8sPodStatus{ + Phase: "Running", + ContainerStatuses: []k8sContainerStatus{ + {Name: sandboxContainerName, Ready: true}, + }, + } + f.mu.Lock() + f.pods[pod.Metadata.Name] = &pod + f.mu.Unlock() + w.WriteHeader(http.StatusCreated) + _ = json.NewEncoder(w).Encode(&pod) +} + +func (f *fakeKube) handlePod(w http.ResponseWriter, r *http.Request) { + m := podPathRe.FindStringSubmatch(r.URL.Path) + name := m[2] + f.mu.Lock() + pod, ok := f.pods[name] + f.mu.Unlock() + switch r.Method { + case http.MethodGet: + if !ok { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "pod not found") + return + } + _ = json.NewEncoder(w).Encode(pod) + case http.MethodDelete: + if !ok { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "pod not found") + return + } + f.mu.Lock() + delete(f.pods, name) + f.deleted = append(f.deleted, name) + f.mu.Unlock() + _ = json.NewEncoder(w).Encode(map[string]any{"kind": "Status", "status": "Success"}) + default: + writeK8sStatus(w, http.StatusMethodNotAllowed, "MethodNotAllowed", r.Method) + } +} + +func (f *fakeKube) handleListPods(w http.ResponseWriter, r *http.Request) { + selector := r.URL.Query().Get("labelSelector") + var want [][2]string + if selector != "" { + for _, kv := range strings.Split(selector, ",") { + k, v, _ := strings.Cut(kv, "=") + want = append(want, [2]string{k, v}) + } + } + list := k8sPodList{Items: []k8sPod{}} + f.mu.Lock() + for _, pod := range f.pods { + match := true + for _, kv := range want { + if pod.Metadata.Labels[kv[0]] != kv[1] { + match = false + break + } + } + if match { + list.Items = append(list.Items, *pod) + } + } + f.mu.Unlock() + _ = json.NewEncoder(w).Encode(&list) +} + +// ── exec ── + +var ( + uploadRe = regexp.MustCompile(`^head -c (\d+) > (\S+) && wc -c < (\S+)$`) + fileOpsRe = regexp.MustCompile(`^head -c (\d+) \| python3 (\S+)$`) +) + +var execUpgrader = websocket.Upgrader{Subprotocols: []string{k8sExecProtocolV4}} + +// execConn wraps the server side of one exec connection. +type execConn struct { + conn *websocket.Conn + writeMu sync.Mutex +} + +func (e *execConn) send(channel byte, data []byte) { + e.writeMu.Lock() + defer e.writeMu.Unlock() + _ = e.conn.WriteMessage(websocket.BinaryMessage, append([]byte{channel}, data...)) +} + +func (e *execConn) finish(exitCode int) { + var status []byte + if exitCode == 0 { + status = []byte(`{"metadata":{},"status":"Success"}`) + } else { + status = []byte(fmt.Sprintf(`{"metadata":{},"status":"Failure","reason":"NonZeroExitCode","details":{"causes":[{"reason":"ExitCode","message":"%d"}]}}`, exitCode)) + } + e.send(k8sChannelError, status) + e.writeMu.Lock() + _ = e.conn.WriteControl(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""), timeNowPlusSecond()) + e.writeMu.Unlock() + _ = e.conn.Close() +} + +// readStdin reads stdin frames until n bytes have arrived (or the conn ends). +func (e *execConn) readStdin(n int) []byte { + var buf bytes.Buffer + for buf.Len() < n { + _, data, err := e.conn.ReadMessage() + if err != nil { + break + } + if len(data) > 0 && data[0] == k8sChannelStdin { + buf.Write(data[1:]) + } + } + return buf.Bytes() +} + +func (f *fakeKube) handleExec(w http.ResponseWriter, r *http.Request) { + m := podExecRe.FindStringSubmatch(r.URL.Path) + podName := m[2] + f.mu.Lock() + _, podExists := f.pods[podName] + f.mu.Unlock() + if !podExists { + writeK8sStatus(w, http.StatusNotFound, "NotFound", "pod not found") + return + } + command := r.URL.Query()["command"] + conn, err := execUpgrader.Upgrade(w, r, nil) + if err != nil { + return + } + e := &execConn{conn: conn} + f.runFakeProcess(podName, command, e) +} + +//nolint:gocognit // test fixture dispatch over the fake process kinds +func (f *fakeKube) runFakeProcess(podName string, command []string, e *execConn) { + // Bridge session: read JSON lines, respond with canned bridge responses. + if len(command) == 2 && command[0] == "python3" && command[1] == k8sBridgePath { + f.runFakeBridge(podName, e) + return + } + + // Shell one-shots. + if len(command) >= 3 && (command[0] == "/bin/sh" || command[0] == "sh") && command[1] == "-c" { + script := command[2] + if um := uploadRe.FindStringSubmatch(script); um != nil { + n, _ := strconv.Atoi(um[1]) + data := e.readStdin(n) + f.mu.Lock() + f.files[podName+":"+um[2]] = data + f.mu.Unlock() + e.send(k8sChannelStdout, []byte(strconv.Itoa(len(data))+"\n")) + e.finish(0) + return + } + if fm := fileOpsRe.FindStringSubmatch(script); fm != nil { + n, _ := strconv.Atoi(fm[1]) + req := e.readStdin(n) + f.mu.Lock() + script := f.files[podName+":"+fm[2]] + f.mu.Unlock() + if script == nil { + e.send(k8sChannelStderr, []byte("fileops script not uploaded")) + e.finish(1) + return + } + // Run the REAL uploaded fileops.py on the test host so the k8s + // transport is exercised against the genuine executor. + cmd := exec.Command("python3", "-c", string(script)) + cmd.Stdin = bytes.NewReader(req) + var stdout, stderr bytes.Buffer + cmd.Stdout, cmd.Stderr = &stdout, &stderr + code := 0 + if err := cmd.Run(); err != nil { + var ee *exec.ExitError + if errors.As(err, &ee) { + code = ee.ExitCode() + } else { + e.send(k8sChannelStderr, []byte(err.Error())) + e.finish(127) + return + } + } + e.send(k8sChannelStdout, stdout.Bytes()) + if stderr.Len() > 0 { + e.send(k8sChannelStderr, stderr.Bytes()) + } + e.finish(code) + return + } + // Workdir-wrapped bash: ["/bin/sh","-c",script,"fleet-bash",dir,cmd] + if len(command) == 6 && command[3] == "fleet-bash" { + f.dispatchBash(command[5], command[4], e) + return + } + } + if len(command) == 3 && command[0] == "bash" && command[1] == "-c" { + f.dispatchBash(command[2], "", e) + return + } + e.send(k8sChannelStderr, []byte(fmt.Sprintf("fake apiserver: unhandled command %q", command))) + e.finish(127) +} + +func (f *fakeKube) dispatchBash(cmd, workdir string, e *execConn) { + f.mu.Lock() + f.lastBashWorkdir = workdir + behavior := f.bashBehaviors[cmd] + f.mu.Unlock() + if behavior == nil { + e.send(k8sChannelStderr, []byte("fake apiserver: no behavior for bash command "+cmd)) + e.finish(127) + return + } + stdout := &channelWriter{e: e, channel: k8sChannelStdout} + stderr := &channelWriter{e: e, channel: k8sChannelStderr} + e.finish(behavior(workdir, stdout, stderr, e.conn)) +} + +// runFakeBridge speaks the bridge line protocol: each request line gets a +// canned success response echoing the code back in `result`. +func (f *fakeKube) runFakeBridge(_ string, e *execConn) { + var pending bytes.Buffer + for { + _, data, err := e.conn.ReadMessage() + if err != nil { + _ = e.conn.Close() + return + } + if len(data) == 0 || data[0] != k8sChannelStdin { + continue + } + pending.Write(data[1:]) + for { + line, rest, found := bytes.Cut(pending.Bytes(), []byte("\n")) + if !found { + break + } + var req bridgeRequest + if err := json.Unmarshal(line, &req); err != nil { + e.send(k8sChannelStderr, []byte("bad bridge request: "+err.Error())) + pending = *bytes.NewBuffer(append([]byte(nil), rest...)) + continue + } + resp, _ := json.Marshal(bridgeResponse{Status: "ok", Result: "ran: " + req.Code}) + e.send(k8sChannelStdout, append(resp, '\n')) + pending = *bytes.NewBuffer(append([]byte(nil), rest...)) + } + } +} + +type channelWriter struct { + e *execConn + channel byte +} + +func (c *channelWriter) Write(p []byte) (int, error) { + c.e.send(c.channel, p) + return len(p), nil +} + +// pythonAvailable reports whether the test host has python3 (the fileops +// integration tests need it; CI always does — it runs the host-executor +// suite). +func pythonAvailable() bool { + _, err := exec.LookPath("python3") + return err == nil +} + +func timeNowPlusSecond() time.Time { return time.Now().Add(time.Second) } diff --git a/internal/sandbox/k8s_kubeconfig.go b/internal/sandbox/k8s_kubeconfig.go new file mode 100644 index 000000000..fcfbe14c4 --- /dev/null +++ b/internal/sandbox/k8s_kubeconfig.go @@ -0,0 +1,227 @@ +// Copyright (c) 2026 ElcanoTek +// SPDX-License-Identifier: MIT + +package sandbox + +// k8s_kubeconfig.go loads client credentials for the kubernetes sandbox +// backend from a kubeconfig file — the out-of-cluster path used when the +// fleet control plane runs OUTSIDE the cluster that hosts its sandbox pods +// (a dev box pointed at kind, or a single-box install delegating runners to +// a cluster). In-cluster service-account auth (the production Helm path) +// lives in newInClusterClient (k8s_client.go). +// +// Deliberately minimal: current-context resolution, token / token-file / +// client-certificate credentials, CA bundles inline or by path. exec +// plugins and auth-providers are REFUSED with an actionable error rather +// than half-supported — the fleet process runs unattended, and shelling out +// to an interactive credential helper on every token expiry is exactly the +// kind of silent-degradation surface the fail-closed posture forbids. + +import ( + "crypto/tls" + "crypto/x509" + "encoding/base64" + "fmt" + "net/http" + "net/url" + "os" + "path/filepath" + + "github.com/goccy/go-yaml" +) + +// kubeconfigFile mirrors the subset of the kubeconfig v1 schema the loader +// reads. Unknown fields are ignored by the YAML decoder, EXCEPT the exec / +// auth-provider blocks, which are decoded precisely so they can be refused. +type kubeconfigFile struct { + CurrentContext string `yaml:"current-context"` + Contexts []struct { + Name string `yaml:"name"` + Context struct { + Cluster string `yaml:"cluster"` + User string `yaml:"user"` + Namespace string `yaml:"namespace"` + } `yaml:"context"` + } `yaml:"contexts"` + Clusters []struct { + Name string `yaml:"name"` + Cluster struct { + Server string `yaml:"server"` + CertificateAuthority string `yaml:"certificate-authority"` + CertificateAuthorityData string `yaml:"certificate-authority-data"` + InsecureSkipTLSVerify bool `yaml:"insecure-skip-tls-verify"` + } `yaml:"cluster"` + } `yaml:"clusters"` + Users []struct { + Name string `yaml:"name"` + User struct { + ClientCertificate string `yaml:"client-certificate"` + ClientCertificateData string `yaml:"client-certificate-data"` + ClientKey string `yaml:"client-key"` + ClientKeyData string `yaml:"client-key-data"` + Token string `yaml:"token"` + TokenFile string `yaml:"tokenFile"` + Exec map[string]any `yaml:"exec"` + AuthProvider map[string]any `yaml:"auth-provider"` + } `yaml:"user"` + } `yaml:"users"` +} + +// newKubeconfigClient builds a k8sClient from the kubeconfig at path, +// following its current-context. The returned namespace is the context's +// default namespace ("" when the context sets none). +func newKubeconfigClient(path string) (*k8sClient, string, error) { + raw, err := os.ReadFile(path) //nolint:gosec // operator-configured kubeconfig path + if err != nil { + return nil, "", fmt.Errorf("read kubeconfig: %w", err) + } + var kc kubeconfigFile + if err := yaml.Unmarshal(raw, &kc); err != nil { + return nil, "", fmt.Errorf("parse kubeconfig %s: %w", path, err) + } + if kc.CurrentContext == "" { + return nil, "", fmt.Errorf("kubeconfig %s has no current-context", path) + } + var clusterName, userName, namespace string + for _, c := range kc.Contexts { + if c.Name == kc.CurrentContext { + clusterName, userName, namespace = c.Context.Cluster, c.Context.User, c.Context.Namespace + break + } + } + if clusterName == "" { + return nil, "", fmt.Errorf("kubeconfig %s: current-context %q not found", path, kc.CurrentContext) + } + + // Relative CA / cert / key paths in a kubeconfig are relative to the FILE, + // not the process cwd — kubectl's convention, kept so the same file works. + baseDir := filepath.Dir(path) + resolve := func(p string) string { + if p == "" || filepath.IsAbs(p) { + return p + } + return filepath.Join(baseDir, p) + } + + tlsCfg := &tls.Config{MinVersion: tls.VersionTLS12} + server, err := applyKubeconfigCluster(&kc, clusterName, path, resolve, tlsCfg) + if err != nil { + return nil, "", err + } + + client := &k8sClient{tlsConfig: tlsCfg} + if err := applyKubeconfigUser(&kc, userName, path, resolve, tlsCfg, client); err != nil { + return nil, "", err + } + + base, err := url.Parse(server) + if err != nil { + return nil, "", fmt.Errorf("kubeconfig %s: parse server URL %q: %w", path, server, err) + } + if base.Scheme != "https" { + return nil, "", fmt.Errorf("kubeconfig %s: server %q is not https — the sandbox control channel requires TLS (fail-closed)", path, server) + } + client.baseURL = base + client.httpc = &http.Client{Transport: &http.Transport{TLSClientConfig: tlsCfg}} + return client, namespace, nil +} + +// applyKubeconfigCluster resolves the named cluster's server URL and installs +// its CA bundle into tlsCfg. insecure-skip-tls-verify is refused rather than +// honored: a sandbox control channel that skips server verification can be +// MITM'd into running tool calls on an attacker's cluster. +func applyKubeconfigCluster(kc *kubeconfigFile, clusterName, path string, resolve func(string) string, tlsCfg *tls.Config) (server string, err error) { + for _, c := range kc.Clusters { + if c.Name != clusterName { + continue + } + server = c.Cluster.Server + if c.Cluster.InsecureSkipTLSVerify { + return "", fmt.Errorf("kubeconfig %s: cluster %q sets insecure-skip-tls-verify, which the sandbox backend refuses (fail-closed) — use a CA bundle", path, clusterName) + } + caPEM := []byte(nil) + switch { + case c.Cluster.CertificateAuthorityData != "": + caPEM, err = base64.StdEncoding.DecodeString(c.Cluster.CertificateAuthorityData) + if err != nil { + return "", fmt.Errorf("kubeconfig %s: decode certificate-authority-data: %w", path, err) + } + case c.Cluster.CertificateAuthority != "": + caPEM, err = os.ReadFile(resolve(c.Cluster.CertificateAuthority)) + if err != nil { + return "", fmt.Errorf("kubeconfig %s: read certificate-authority: %w", path, err) + } + } + if caPEM != nil { + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(caPEM) { + return "", fmt.Errorf("kubeconfig %s: cluster %q CA bundle contains no usable certificates", path, clusterName) + } + tlsCfg.RootCAs = pool + } + break + } + if server == "" { + return "", fmt.Errorf("kubeconfig %s: cluster %q not found or has no server", path, clusterName) + } + return server, nil +} + +// applyKubeconfigUser resolves the named user's credentials into the client +// (token / token-file) or tlsCfg (client certificate). exec plugins and +// auth-providers are refused — the fleet process runs unattended. +func applyKubeconfigUser(kc *kubeconfigFile, userName, path string, resolve func(string) string, tlsCfg *tls.Config, client *k8sClient) error { + for _, u := range kc.Users { + if u.Name != userName { + continue + } + if u.User.Exec != nil || u.User.AuthProvider != nil { + return fmt.Errorf("kubeconfig %s: user %q uses an exec plugin / auth-provider, which the sandbox backend does not support — "+ + "create a ServiceAccount token or client certificate for fleet instead", path, userName) + } + certPEM, err := kubeconfigPEM(u.User.ClientCertificateData, resolve(u.User.ClientCertificate)) + if err != nil { + return fmt.Errorf("kubeconfig %s: client-certificate: %w", path, err) + } + keyPEM, err := kubeconfigPEM(u.User.ClientKeyData, resolve(u.User.ClientKey)) + if err != nil { + return fmt.Errorf("kubeconfig %s: client-key: %w", path, err) + } + switch { + case certPEM != nil && keyPEM != nil: + cert, cerr := tls.X509KeyPair(certPEM, keyPEM) + if cerr != nil { + return fmt.Errorf("kubeconfig %s: load client certificate: %w", path, cerr) + } + tlsCfg.Certificates = []tls.Certificate{cert} + case u.User.Token != "": + client.staticToken = u.User.Token + case u.User.TokenFile != "": + client.tokenFile = resolve(u.User.TokenFile) + default: + return fmt.Errorf("kubeconfig %s: user %q has no supported credentials (token, tokenFile, or client certificate)", path, userName) + } + return nil + } + return fmt.Errorf("kubeconfig %s: user %q not found", path, userName) +} + +// kubeconfigPEM loads a PEM blob from inline base64 data (preferred) or a +// file path; both empty returns (nil, nil). +func kubeconfigPEM(inlineB64, filePath string) ([]byte, error) { + if inlineB64 != "" { + data, err := base64.StdEncoding.DecodeString(inlineB64) + if err != nil { + return nil, fmt.Errorf("decode inline data: %w", err) + } + return data, nil + } + if filePath == "" { + return nil, nil + } + data, err := os.ReadFile(filePath) //nolint:gosec // path from an operator-configured kubeconfig, not request input + if err != nil { + return nil, err + } + return data, nil +} diff --git a/internal/sandbox/k8s_preflight.go b/internal/sandbox/k8s_preflight.go new file mode 100644 index 000000000..498dbda60 --- /dev/null +++ b/internal/sandbox/k8s_preflight.go @@ -0,0 +1,99 @@ +// Copyright (c) 2026 ElcanoTek +// SPDX-License-Identifier: MIT + +package sandbox + +// Boot-time preflight for the kubernetes sandbox backend (#989 / ADR-0049). +// +// When FLEET_SANDBOX_BACKEND=kubernetes is selected, a cluster that cannot +// actually run sandbox pods must abort boot — never silently fall back to +// podman or host execution (the same no-degrade posture as PreflightRuntime +// and PreflightAllowlistedNetwork). The checks run in failure-likelihood +// order so the first error an operator sees is the most actionable one: +// +// 1. apiserver reachable + credentials valid (GET /version) +// 2. RBAC: create/get/list/delete pods and create pods/exec in the sandbox +// namespace (SelfSubjectAccessReview — precise "which verb is missing") +// 3. the shared workspace PVC exists +// 4. the sealed-egress NetworkPolicy object exists (the chart ships it; +// the OBJECT check cannot prove the CNI enforces it, and the docs say so) +// 5. the RuntimeClass exists, when one is configured +// +// Deliberately NOT checked: image pullability (imagePullSecrets are resolved +// by the kubelet per node — the only faithful probe is running a pod, which +// the first turn does, failing fast on ErrImagePull in waitForRunning) and +// PVC access mode (RWX vs RWO is advisory in the API; a wrong mode surfaces +// as a scheduling error the docs' checklist covers). + +import ( + "context" + "fmt" + "log" + "time" +) + +// k8sPreflightTimeout bounds the whole preflight sequence. Generous for a +// cold connection; short enough that a dead apiserver fails boot promptly. +const k8sPreflightTimeout = 30 * time.Second + +// k8sRBACChecks are the (verb, resource, subresource) grants the backend +// needs. Listed as data so the error message names exactly what is missing. +var k8sRBACChecks = []struct { + verb, resource, subresource string +}{ + {"create", "pods", ""}, + {"get", "pods", ""}, + {"list", "pods", ""}, + {"delete", "pods", ""}, + {"create", "pods", "exec"}, +} + +// Preflight verifies, fail-closed, that the cluster can deliver what the +// kubernetes sandbox backend promises. Called from the single production +// pool-construction path (agent.buildSandboxPool) and from +// `fleet validate-config` — mirroring PreflightRuntime's contract. Callers +// must treat any error as fatal to boot; there is no degraded mode. +func (b *KubernetesBackend) Preflight(ctx context.Context) error { + if b.cfg.WorkspaceClaim == "" { + return fmt.Errorf("kubernetes sandbox preflight: no workspace PVC configured — set FLEET_SANDBOX_K8S_WORKSPACE_CLAIM (or the bundle manifest's sandbox.kubernetes.workspace_claim) to the ReadWriteMany claim shared with the control plane") + } + preCtx, cancel := context.WithTimeout(ctx, k8sPreflightTimeout) + defer cancel() + + version, err := b.client.serverVersion(preCtx) + if err != nil { + return fmt.Errorf("kubernetes sandbox preflight: apiserver unreachable or credentials rejected: %w", err) + } + + for _, check := range k8sRBACChecks { + allowed, err := b.client.selfSubjectAccessReview(preCtx, b.cfg.Namespace, check.verb, check.resource, check.subresource) + if err != nil { + return fmt.Errorf("kubernetes sandbox preflight: access review for %s %s/%s failed: %w", check.verb, check.resource, check.subresource, err) + } + if !allowed { + target := check.resource + if check.subresource != "" { + target += "/" + check.subresource + } + return fmt.Errorf("kubernetes sandbox preflight: the fleet service account may not %s %s in namespace %q — grant the fleet-runner Role from the Helm chart (deploy/helm/fleet) or an equivalent RoleBinding", check.verb, target, b.cfg.Namespace) + } + } + + if err := b.client.getPVC(preCtx, b.cfg.Namespace, b.cfg.WorkspaceClaim); err != nil { + return fmt.Errorf("kubernetes sandbox preflight: workspace PVC %q not readable in namespace %q (it must be a ReadWriteMany claim mounted by the control plane at the same path): %w", b.cfg.WorkspaceClaim, b.cfg.Namespace, err) + } + + if err := b.client.getNetworkPolicy(preCtx, b.cfg.Namespace, b.cfg.NetworkPolicyName); err != nil { + return fmt.Errorf("kubernetes sandbox preflight: sealed-egress NetworkPolicy %q not found in namespace %q — the deny-all policy for pods labeled %s=none must exist before sealed sandboxes can be trusted (the Helm chart ships it): %w", + b.cfg.NetworkPolicyName, b.cfg.Namespace, k8sLabelEgress, err) + } + + if b.cfg.RuntimeClassName != "" { + if err := b.client.getRuntimeClass(preCtx, b.cfg.RuntimeClassName); err != nil { + return fmt.Errorf("kubernetes sandbox preflight: RuntimeClass %q not found — a hypervisor runtime that cannot be verified must abort boot, never degrade to the default runtime (ADR-0010 posture): %w", b.cfg.RuntimeClassName, err) + } + } + + log.Printf("sandbox: kubernetes backend preflight OK — apiserver %s, sandbox namespace %q", version, b.cfg.Namespace) + return nil +} diff --git a/internal/sandbox/k8s_preflight_test.go b/internal/sandbox/k8s_preflight_test.go new file mode 100644 index 000000000..0c7cd9ba0 --- /dev/null +++ b/internal/sandbox/k8s_preflight_test.go @@ -0,0 +1,174 @@ +package sandbox + +// Preflight + kubeconfig fail-closed tests for the kubernetes backend (#989). + +import ( + "context" + "encoding/base64" + "encoding/pem" + "fmt" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestK8sPreflightHappyPath(t *testing.T) { + fake := newFakeKube(t) + backend := fake.backend(t, KubernetesConfig{Namespace: "fleet-sandboxes", RuntimeClassName: "kata"}) + if err := backend.Preflight(context.Background()); err != nil { + t.Fatalf("Preflight: %v", err) + } +} + +func TestK8sPreflightFailClosed(t *testing.T) { + t.Run("missing workspace claim", func(t *testing.T) { + fake := newFakeKube(t) + b, err := NewKubernetesBackend(KubernetesConfig{KubeconfigPath: fake.kubeconfigPath(t)}) + if err != nil { + t.Fatal(err) + } + if err := b.Preflight(context.Background()); err == nil || !strings.Contains(err.Error(), "workspace PVC") { + t.Errorf("want workspace-claim error, got %v", err) + } + }) + t.Run("rbac denial names the verb", func(t *testing.T) { + fake := newFakeKube(t) + fake.denied["create pods/exec"] = true + backend := fake.backend(t, KubernetesConfig{}) + err := backend.Preflight(context.Background()) + if err == nil || !strings.Contains(err.Error(), "create pods/exec") { + t.Errorf("want create pods/exec denial, got %v", err) + } + }) + t.Run("missing pvc", func(t *testing.T) { + fake := newFakeKube(t) + fake.noPVC = true + backend := fake.backend(t, KubernetesConfig{}) + if err := backend.Preflight(context.Background()); err == nil || !strings.Contains(err.Error(), "workspace PVC") { + t.Errorf("want PVC error, got %v", err) + } + }) + t.Run("missing networkpolicy", func(t *testing.T) { + fake := newFakeKube(t) + fake.noNetpol = true + backend := fake.backend(t, KubernetesConfig{}) + if err := backend.Preflight(context.Background()); err == nil || !strings.Contains(err.Error(), "NetworkPolicy") { + t.Errorf("want NetworkPolicy error, got %v", err) + } + }) + t.Run("missing runtimeclass", func(t *testing.T) { + fake := newFakeKube(t) + fake.noRuntimeClass = true + backend := fake.backend(t, KubernetesConfig{RuntimeClassName: "kata"}) + if err := backend.Preflight(context.Background()); err == nil || !strings.Contains(err.Error(), "RuntimeClass") { + t.Errorf("want RuntimeClass error, got %v", err) + } + }) + t.Run("bad credentials", func(t *testing.T) { + fake := newFakeKube(t) + path := fake.kubeconfigPath(t) + raw, _ := os.ReadFile(path) + bad := strings.ReplaceAll(string(raw), fakeKubeToken, "wrong-token") + if err := os.WriteFile(path, []byte(bad), 0o600); err != nil { + t.Fatal(err) + } + b, err := NewKubernetesBackend(KubernetesConfig{KubeconfigPath: path, WorkspaceClaim: "ws"}) + if err != nil { + t.Fatal(err) + } + if err := b.Preflight(context.Background()); err == nil || !strings.Contains(err.Error(), "unreachable or credentials rejected") { + t.Errorf("want credentials error, got %v", err) + } + }) +} + +func TestK8sBackendDefaults(t *testing.T) { + fake := newFakeKube(t) + b, err := NewKubernetesBackend(KubernetesConfig{KubeconfigPath: fake.kubeconfigPath(t), WorkspaceClaim: "ws"}) + if err != nil { + t.Fatal(err) + } + if got := b.Namespace(); got != defaultK8sNamespace { + t.Errorf("default namespace = %q, want %q", got, defaultK8sNamespace) + } + if b.cfg.NetworkPolicyName != defaultK8sNetworkPolicy { + t.Errorf("default networkpolicy = %q", b.cfg.NetworkPolicyName) + } + if b.StartTimeout() != defaultK8sStartTimeout { + t.Errorf("default start timeout = %v", b.StartTimeout()) + } +} + +func writeKubeconfig(t *testing.T, body string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "kubeconfig") + if err := os.WriteFile(path, []byte(body), 0o600); err != nil { + t.Fatal(err) + } + return path +} + +func TestKubeconfigFailClosed(t *testing.T) { + fake := newFakeKube(t) + caPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: fake.srv.Certificate().Raw}) + caB64 := base64.StdEncoding.EncodeToString(caPEM) + + t.Run("insecure-skip-tls-verify refused", func(t *testing.T) { + path := writeKubeconfig(t, fmt.Sprintf(`current-context: c +contexts: [{name: c, context: {cluster: cl, user: u}}] +clusters: [{name: cl, cluster: {server: %s, insecure-skip-tls-verify: true}}] +users: [{name: u, user: {token: t}}] +`, fake.srv.URL)) + if _, _, err := newKubeconfigClient(path); err == nil || !strings.Contains(err.Error(), "insecure-skip-tls-verify") { + t.Errorf("want insecure refusal, got %v", err) + } + }) + t.Run("exec plugin refused", func(t *testing.T) { + path := writeKubeconfig(t, fmt.Sprintf(`current-context: c +contexts: [{name: c, context: {cluster: cl, user: u}}] +clusters: [{name: cl, cluster: {server: %s, certificate-authority-data: %s}}] +users: [{name: u, user: {exec: {command: aws}}}] +`, fake.srv.URL, caB64)) + if _, _, err := newKubeconfigClient(path); err == nil || !strings.Contains(err.Error(), "exec plugin") { + t.Errorf("want exec-plugin refusal, got %v", err) + } + }) + t.Run("http server refused", func(t *testing.T) { + path := writeKubeconfig(t, `current-context: c +contexts: [{name: c, context: {cluster: cl, user: u}}] +clusters: [{name: cl, cluster: {server: http://127.0.0.1:8080}}] +users: [{name: u, user: {token: t}}] +`) + if _, _, err := newKubeconfigClient(path); err == nil || !strings.Contains(err.Error(), "requires TLS") { + t.Errorf("want TLS refusal, got %v", err) + } + }) + t.Run("no credentials refused", func(t *testing.T) { + path := writeKubeconfig(t, fmt.Sprintf(`current-context: c +contexts: [{name: c, context: {cluster: cl, user: u}}] +clusters: [{name: cl, cluster: {server: %s, certificate-authority-data: %s}}] +users: [{name: u, user: {}}] +`, fake.srv.URL, caB64)) + if _, _, err := newKubeconfigClient(path); err == nil || !strings.Contains(err.Error(), "no supported credentials") { + t.Errorf("want no-credentials refusal, got %v", err) + } + }) + t.Run("context namespace surfaces", func(t *testing.T) { + path := writeKubeconfig(t, fmt.Sprintf(`current-context: c +contexts: [{name: c, context: {cluster: cl, user: u, namespace: my-ns}}] +clusters: [{name: cl, cluster: {server: %s, certificate-authority-data: %s}}] +users: [{name: u, user: {token: t}}] +`, fake.srv.URL, caB64)) + _, ns, err := newKubeconfigClient(path) + if err != nil || ns != "my-ns" { + t.Errorf("namespace = %q, %v; want my-ns", ns, err) + } + }) + t.Run("missing current-context refused", func(t *testing.T) { + path := writeKubeconfig(t, "contexts: []\n") + if _, _, err := newKubeconfigClient(path); err == nil || !strings.Contains(err.Error(), "current-context") { + t.Errorf("want current-context error, got %v", err) + } + }) +} diff --git a/internal/sandbox/pool.go b/internal/sandbox/pool.go index 8ffddd88e..78dab9c6c 100644 --- a/internal/sandbox/pool.go +++ b/internal/sandbox/pool.go @@ -139,9 +139,15 @@ type PoolConfig struct { BridgeScript []byte // Container holds the per-sandbox container settings (image, mounts, - // caps). Required when Mode == ModeContainer. + // caps). Required when Mode == ModeContainer, and the source of the + // backend-shared knobs (image, workspace path, limits, network posture) + // when Mode == ModeKubernetes. Container ContainerConfig + // KubernetesBackend is the boot-built handle for the kubernetes sandbox + // backend (#989). Required when Mode == ModeKubernetes; nil otherwise. + KubernetesBackend *KubernetesBackend + // EgressProxy, when non-nil, is the host-side allowlist proxy (#211) used by // TakeContainerWithEgress for "allowlisted" network mode. nil means // allowlisted mode is unavailable: such requests FAIL CLOSED (an error) @@ -301,7 +307,6 @@ func (p *Pool) TakeContainerWithOverrides(ctx context.Context, ov ResourceOverri } cfg := p.cfg.Container cfg.BridgeScript = p.cfg.BridgeScript - cfg.StorageOptSupported = p.storageOptSupported(ctx) // Network sealing is enforced HERE rather than upstream so the lockdown // contract is impossible to bypass via a bad caller. cfg.NoNetwork = noNetwork @@ -311,9 +316,9 @@ func (p *Pool) TakeContainerWithOverrides(ctx context.Context, ov ResourceOverri cfg = ov.applyTo(cfg) // See newSandbox below for why we resolve the start timeout here // rather than reading it raw from cfg. - startCtx, cancel := context.WithTimeout(ctx, resolveStartTimeout(cfg)+5*time.Second) + startCtx, cancel := context.WithTimeout(ctx, p.startTimeoutFor(cfg)+5*time.Second) defer cancel() - sb, err := NewContainer(startCtx, cfg) + sb, err := p.newBackendSandbox(startCtx, cfg) if err != nil { return nil, func() {}, err } @@ -321,10 +326,45 @@ func (p *Pool) TakeContainerWithOverrides(ctx context.Context, ov ResourceOverri sb.Close() return nil, func() {}, ErrClosed } - sb.SetPythonCellTimeout(p.cfg.PythonCellTimeout) return sb, sb.Close, nil } +// startTimeoutFor resolves the outer construction budget for the active +// backend: the kubernetes backend's pod start ceiling (schedule + pull) when +// it is selected, else the podman container start timeout. +func (p *Pool) startTimeoutFor(cfg ContainerConfig) time.Duration { + if p.cfg.Mode == ModeKubernetes && p.cfg.KubernetesBackend != nil { + return p.cfg.KubernetesBackend.StartTimeout() + } + return resolveStartTimeout(cfg) +} + +// newBackendSandbox constructs one sandbox from a fully-resolved per-call +// cfg, routing to the active container backend (#989): a Kubernetes pod when +// ModeKubernetes, else a rootless-Podman container. The podman path pays the +// storage-opt probe here; kubernetes has no analogue (its disk cap is the +// pod's ephemeral-storage limit, applied unconditionally in the pod spec). +func (p *Pool) newBackendSandbox(ctx context.Context, cfg ContainerConfig) (*Sandbox, error) { + var ( + sb *Sandbox + err error + ) + if p.cfg.Mode == ModeKubernetes { + if p.cfg.KubernetesBackend == nil { + return nil, errors.New("sandbox: kubernetes backend selected but not constructed (fail-closed)") + } + sb, err = p.cfg.KubernetesBackend.newSandbox(ctx, cfg) + } else { + cfg.StorageOptSupported = p.storageOptSupported(ctx) + sb, err = NewContainer(ctx, cfg) + } + if err != nil { + return nil, err + } + sb.SetPythonCellTimeout(p.cfg.PythonCellTimeout) + return sb, nil +} + // TakeContainerWithEgress cold-starts a fresh container in "allowlisted" network // mode (#211): slirp4netns transport with HTTPS_PROXY pointed at the pool's // EgressProxy, scoped to allowlist for THIS turn via a fresh per-turn token. The @@ -345,6 +385,13 @@ func (p *Pool) TakeContainerWithEgress(ctx context.Context, ov ResourceOverride, if p.cfg.Container.Image == "" { return nil, func() {}, ErrContainerUnavailable } + if p.cfg.Mode == ModeKubernetes { + // The egress proxy binds to the control-plane host's loopback; a pod on + // another node cannot reach it. Refuse rather than grant open egress + // under an "allowlisted" banner. buildSandboxPool refuses the mode at + // boot too; this is the can't-bypass-it backstop. + return nil, func() {}, errors.New("allowlisted network mode is not supported by the kubernetes sandbox backend (fail-closed)") + } if p.cfg.EgressProxy == nil { return nil, func() {}, errors.New("allowlisted network mode requested but no egress proxy is configured (fail-closed)") } @@ -502,6 +549,16 @@ func (p *Pool) EgressDefault() (mode string, allowlist []string) { return p.cfg.DefaultNetworkMode, p.cfg.DefaultEgressAllowlist } +// KubernetesBackend exposes the kubernetes backend handle when that backend +// is active (nil under podman/host). Used by boot-time maintenance (the +// orphan-pod prune in cmd/fleet) and diagnostics. +func (p *Pool) KubernetesBackend() *KubernetesBackend { + if p == nil { + return nil + } + return p.cfg.KubernetesBackend +} + func (p *Pool) Close() { if p == nil { return @@ -766,12 +823,11 @@ func (p *Pool) storageOptSupported(ctx context.Context) bool { func (p *Pool) newSandbox(ctx context.Context) (*Sandbox, error) { switch p.cfg.Mode { - case ModeContainer: + case ModeContainer, ModeKubernetes: cfg := p.cfg.Container cfg.BridgeScript = p.cfg.BridgeScript - cfg.StorageOptSupported = p.storageOptSupported(ctx) - // resolveStartTimeout applies the same default NewContainer would - // apply internally. Without this, the OUTER context timeout is + // startTimeoutFor applies the same default the backend constructor + // would apply internally. Without this, the OUTER context timeout is // `0+5s = 5s` when StartTimeout isn't set explicitly, which // cancels podman before its first-run idmapped-layer chown // finishes — that chown takes ~12s on a fresh sandbox image @@ -780,14 +836,9 @@ func (p *Pool) newSandbox(ctx context.Context) (*Sandbox, error) { // "first message after deploy fails, second works fine" // (because by the time the second message lands, the warm pool // has finished filling against the now-cached chowned layer). - startCtx, cancel := context.WithTimeout(ctx, resolveStartTimeout(cfg)+5*time.Second) + startCtx, cancel := context.WithTimeout(ctx, p.startTimeoutFor(cfg)+5*time.Second) defer cancel() - sb, err := NewContainer(startCtx, cfg) - if err != nil { - return nil, err - } - sb.SetPythonCellTimeout(p.cfg.PythonCellTimeout) - return sb, nil + return p.newBackendSandbox(startCtx, cfg) case ModeHost: // Test-only fixture path. agent.go forbids ModeHost in // production; this branch only fires when sandbox_test.go diff --git a/internal/sandbox/sandbox.go b/internal/sandbox/sandbox.go index fae98fa44..fb8e8a1b0 100644 --- a/internal/sandbox/sandbox.go +++ b/internal/sandbox/sandbox.go @@ -88,6 +88,12 @@ const ( // Podman container with --read-only / dropped caps. Network egress // is per-turn (ContainerConfig.NoNetwork) — see container.go. ModeContainer + + // ModeKubernetes runs bash and the python bridge inside an ephemeral + // Kubernetes Pod, exec'd over the apiserver (#989) — the enterprise + // split-control-plane backend. Selected by FLEET_SANDBOX_BACKEND; + // see k8s_backend.go. + ModeKubernetes ) // BashRequest is the per-call input the sandbox sees for a bash @@ -237,8 +243,10 @@ func (s *Sandbox) SetDefaultWorkingDir(dir string) { s.mu.Unlock() } -// impl is the backend interface. Two concrete implementations live in -// host.go and container.go. +// impl is the backend interface. Three concrete implementations live in +// container.go (rootless Podman, the single-box production backend), +// k8s_backend.go (Kubernetes pods, the enterprise split backend, #989), and +// host.go (the test-only fixture behind the fleet_host_executor build tag). type impl interface { runBash(ctx context.Context, req BashRequest) (BashResult, error) runPython(ctx context.Context, req PythonRequest) (PythonResult, error) @@ -281,6 +289,8 @@ func (s *Sandbox) ModeName() string { return "host" case ModeContainer: return "container" + case ModeKubernetes: + return "kubernetes" default: return "unknown" } diff --git a/scripts/check_permissions_test.go b/scripts/check_permissions_test.go new file mode 100644 index 000000000..4374e89a4 --- /dev/null +++ b/scripts/check_permissions_test.go @@ -0,0 +1,57 @@ +// Copyright (c) 2025 ElcanoTek +// SPDX-License-Identifier: MIT + +package scripts + +import ( + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// Every workflow in this repo declares a top-level `permissions:` block, and +// that is load-bearing rather than tidy: a workflow WITHOUT one inherits the +// repository default, which can be read-write for every scope. The whole +// least-privilege posture documented in docs/SCANNING.md rests on the property +// holding for all of them, and a new workflow added without the block is a +// silent, invisible regression — there is no failing check, just a job quietly +// holding more token than it asked for. +// +// So assert it, in the same spirit as check_action_pins_test.go and +// check_gate_needs_test.go: the invariants this repo cares about are tests, not +// review habits. The assertion is deliberately only that the block EXISTS — its +// contents are a per-workflow judgement call (`{}`, `contents: read`, or a +// scoped set), and pinning those here would fight every legitimate change. +var topLevelPermissionsRe = regexp.MustCompile(`(?m)^permissions:`) + +func TestWorkflowsDeclareTopLevelPermissions(t *testing.T) { + root := repoRoot(t) + dir := filepath.Join(root, ".github", "workflows") + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatalf("read %s: %v", dir, err) + } + + seen := 0 + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".yml") { + continue + } + seen++ + raw, err := os.ReadFile(filepath.Join(dir, e.Name())) + if err != nil { + t.Fatalf("read %s: %v", e.Name(), err) + } + if !topLevelPermissionsRe.Match(raw) { + t.Errorf("%s: no top-level `permissions:` block — the workflow inherits the "+ + "repository default token scopes. Declare one (`permissions: {}` if the "+ + "workflow needs nothing) and grant writes on the job that needs them.", e.Name()) + } + } + if seen == 0 { + t.Fatal("no workflow files found — this test would pass vacuously") + } + t.Logf("checked %d workflow files", seen) +} diff --git a/scripts/check_versions_test.go b/scripts/check_versions_test.go index 56a903125..fa560a261 100644 --- a/scripts/check_versions_test.go +++ b/scripts/check_versions_test.go @@ -173,6 +173,8 @@ func TestDuplicatedToolPinsAgree(t *testing.T) { {"GRYPE_SHA256", ".github/workflows/grype-scheduled.yml", regexp.MustCompile(`GRYPE_SHA256:\s*'([^']+)'`)}, {"GITLEAKS_VERSION", ".github/workflows/dev-ci.yml", regexp.MustCompile(`GITLEAKS_VERSION:\s*'([^']+)'`)}, {"RUFF_VERSION", ".github/workflows/dev-ci.yml", regexp.MustCompile(`RUFF_VERSION:\s*'([^']+)'`)}, + {"ACTIONLINT_VERSION", ".github/workflows/dev-ci.yml", regexp.MustCompile(`ACTIONLINT_VERSION:\s*'([^']+)'`)}, + {"ACTIONLINT_SHA256", ".github/workflows/dev-ci.yml", regexp.MustCompile(`ACTIONLINT_SHA256:\s*'([^']+)'`)}, {"golangci-lint version", ".github/workflows/dev-ci.yml", regexp.MustCompile(`golangci-lint-action@\S+[^\n]*\n\s*with:\s+(?:#[^\n]*\n\s+)*version:\s*(v[\d.]+)`)}, } { a := tc.re.FindStringSubmatch(ci) @@ -255,8 +257,9 @@ func goMinor(spec string) (string, bool) { // // - web/go.mod — a no-package boundary module, so nothing compiles against it // and a stale `go` line there is completely silent. -// - docs/EKS-DEPLOYMENT.md — a `FROM golang:` build stage an operator -// copies verbatim. Too old and their image cannot build the module at all. +// - docs/DEPLOYMENT-KUBERNETES.md — a `FROM golang:` build stage an +// operator copies verbatim (the control-plane image). Too old and their +// image cannot build the module at all. // // This is the same blind spot the node major had, and it bit the same way: the // pin sat at 1.26 after 1.27 shipped, with nothing to say so. Dependabot cannot @@ -284,19 +287,22 @@ func TestGoMinorAgreesEverywhere(t *testing.T) { t.Errorf("web/go.mod says go %s but go.mod says %s — bump them together (web/go.mod is major.minor only by design, but the minor still has to agree)", got, want) } - const eksDoc = "docs/EKS-DEPLOYMENT.md" - img := regexp.MustCompile(`FROM golang:(\S+?)(?:\s|$)`).FindAllStringSubmatch(readFile(t, root, eksDoc), -1) + const k8sDoc = "docs/DEPLOYMENT-KUBERNETES.md" + img := regexp.MustCompile(`FROM (?:docker\.io/library/)?golang:(\S+?)(?:\s|$)`).FindAllStringSubmatch(readFile(t, root, k8sDoc), -1) if len(img) == 0 { - t.Logf("%s has no `FROM golang:` stage — nothing to check", eksDoc) + // The stage existing is part of what this test pins: the doc's build + // recipe is the copy operators consume, so silently losing it would + // reopen the drift blind spot. + t.Errorf("%s has no `FROM golang:` stage — the control-plane image recipe should declare one", k8sDoc) } for _, m := range img { got, ok := goMinor(m[1]) if !ok { - t.Errorf("%s has `FROM golang:%s` — cannot read a major.minor from it", eksDoc, m[1]) + t.Errorf("%s has `FROM golang:%s` — cannot read a major.minor from it", k8sDoc, m[1]) continue } if got != want { - t.Errorf("%s builds on `golang:%s` but go.mod says %s — an operator copying that stage gets an image too old to build the module", eksDoc, m[1], want) + t.Errorf("%s builds on `golang:%s` but go.mod says %s — an operator copying that stage gets an image too old to build the module", k8sDoc, m[1], want) } } } diff --git a/scripts/doctor.sh b/scripts/doctor.sh index 408141683..982ea4d6e 100755 --- a/scripts/doctor.sh +++ b/scripts/doctor.sh @@ -52,6 +52,9 @@ SRC_DIR="${SRC_DIR:-$REPO_ROOT}" # scripts/bootstrap.sh (SERVICE_USER/SERVICE_HOME there). SERVICE_USER="${FLEET_SERVICE_USER:-fleet}" SERVICE_HOME="${FLEET_SERVICE_HOME:-/var/lib/fleet}" +# shellcheck disable=SC2034 # unread here on purpose: this block documents the +# service-account contract that deploy/fleet-web.service and bootstrap.sh must +# match, and dropping the name would remove the anchor the comment above names. WEB_USER="fleet-web" SERVICE_NAME="${FLEET_SERVICE_NAME:-fleet}" INSTALL_DIR="${FLEET_INSTALL_DIR:-/opt/fleet}" diff --git a/scripts/fleet-upgrade.sh b/scripts/fleet-upgrade.sh index e63fbeaf8..409f4fd05 100755 --- a/scripts/fleet-upgrade.sh +++ b/scripts/fleet-upgrade.sh @@ -293,8 +293,7 @@ restart_web_tier() { fi # Read the resolved state back rather than trusting the restart's exit code — # a unit can accept the restart and then fail its ExecStart. - local i - for i in 1 2 3 4 5 6 7 8; do + for _ in 1 2 3 4 5 6 7 8; do if [[ "$(systemctl is-active fleet-web 2>/dev/null || true)" == "active" ]]; then WEB_TIER_UP="yes"; ok "fleet-web is active again (systemctl is-active)"; return 0 fi diff --git a/scripts/lib/node-version.sh b/scripts/lib/node-version.sh index bff018227..42cce6212 100644 --- a/scripts/lib/node-version.sh +++ b/scripts/lib/node-version.sh @@ -1,3 +1,6 @@ +# shellcheck shell=bash +# ^ no shebang: this file is only ever sourced, never executed. The directive +# tells shellcheck which dialect to check it as (see the bash note below). # scripts/lib/node-version.sh — the ONE implementation of "which node?". # # Sourced by scripts/bootstrap.sh, scripts/doctor.sh and scripts/update.sh. It