Skip to content

docs: fold in the fleet#1264 round-1 kind rehearsal findings - #2

Merged
bradflaugher merged 6 commits into
ElcanoTek:mainfrom
KristianYe:docs/kind-round1-findings
Sep 2, 2026
Merged

bradflaugher merged 6 commits into
ElcanoTek:mainfrom
KristianYe:docs/kind-round1-findings

Conversation

@KristianYe

Copy link
Copy Markdown

What changed, and why

Round 1 of ElcanoTek/fleet#1264 validated this bundle's kind walkthrough end-to-end for the first time (kind v0.32.0 / K8s v1.36.1, rootless podman, WSL2) — treating docs/LOCAL-CLUSTER-KIND.md itself as the thing under test. The engine bugs it surfaced went to fleet (ElcanoTek/fleet#1285 exec transport, #1287 readyz, #1288 warmSize:0, #1289 validate-config key probe, issue #1290 file-tool doc reads); this PR fixes everything the rehearsal proved wrong or stale in the bundle's own docs and comments. No behavioral change to the overlays except values-example.yaml leaving warmSize unset (see below).

  • The seal on kind is no longer categorically fake. kindnet enforces NetworkPolicy on recent kind (kindnet ≥ v1.3.0) — verified live on this rehearsal: a sealed-label pod was blocked while a control pod got through, on stock kindnet. Caveat 1, the DEGRADED #3 and network-mode comments in values-kind.yaml, and the "Making the seal real" section (now "Verifying the seal on kind") now tell the version-dependent truth: run the seal test, flip to lockdown once it proves enforcement; the Calico path stays for older kinds. The overlay still ships open so it never claims a seal an old kind won't deliver.
  • The seal test itself had a false-negative bug: it fetched https://example.com with busybox wget, which has no TLS — the fetch fails for TLS reasons even on an unsealed cluster and fakes a sealed result. The test now uses http://, with a comment saying why, plus an explicit control run (same command without the sandbox labels must succeed).
  • openssl joins the prerequisites, and step 4 gains a verify-the-secret snippet. Both traps bit during the rehearsal: a failed $(openssl rand -hex 32) substitution still creates the secret — with an empty CHAT_SERVER_TOKEN — and a junk OPENROUTER_API_KEY sailed through fleet validate-config (its model_api check probed a public endpoint; fixed in fleet#1289) and only failed on the first real completion.
  • Proof (c)'s view_file diagnosis was wrong. It attributed a failure to the bundle_docs_in_image declaration or the image; the actual failure — on every backend — is fleet's task-run path policy (Scheduled runs and fleet task run cannot read bundle docs with the file tools (view_file protocols/… fails on every backend) fleet#1290: workspace root never registered, no supporting-doc exception for scheduled runs). The doc now says so and gives the interim checks: the boot-log anchor line, and an absolute-path read inside a sandbox pod. The original two-outcome diagnosis is kept for after #1290 lands.
  • Caveat 4 ("no warm pool") was false on fleet before #1288: warmSize: 0 silently derived a two-pod Guaranteed-QoS pool. The caveat and the DEGRADED #1 comment now state the fleet-version dependency; values-example.yaml leaves warmSize unset (derive on both old and new fleet) instead of 0, which #1288 makes mean "no warm pool" — the wrong default for a production-shaped example.
  • The Makefile's "chart ships no values schema" comment is stale — fleet's chart ships values.schema.json (a typo guard with additionalProperties:false on fixed-shape objects). Reworded to what the schema does and does not catch (free-form maps like config.env stay open, which is why reading the rendered env block remains the check).
  • Rough-edges table: kubectl cp needs tar, which the control-plane image doesn't ship (stream via kubectl exec -i … 'cat > file' instead), and the kind node container does not survive a host reboot (podman start larkspur-control-plane brings everything back). Both hit during the rehearsal. The "seal test passes traffic" row now names old kindnet and the https/TLS false-negative.

Verification

make helm-lint and make helm-template (FLEET = current fleet dev) pass; cross-generation render checks: values-kind.yaml renders on the pre-#1288 chart (env omitted — the documented degradation) and on the #1288 chart (emits FLEET_SANDBOX_WARM_SIZE: "0"); values-example.yaml renders on both with no WARM_SIZE env (derives on both). gitleaks clean. The seal-test command, secret-verification snippet, kubectl exec streaming workaround, and podman start recovery were all executed verbatim on the rehearsal cluster.

Rendered env block from make helm-template FLEET=<fleet dev> TAG=dev:

          env:
            # Listeners bind all pod interfaces; the Service/Ingress is the
            # exposure boundary (the single-box default is loopback + Caddy).
            - name: FLEET_SERVER_ADDR
              value: "0.0.0.0:8080"
            - name: FLEET_ORCHESTRATOR_ADDR
              value: "0.0.0.0:8000"
            - name: FLEET_DATA_DIR
              value: "/var/lib/fleet"
            # fleet defaults EMAIL_ATTACHMENT_DIR to ./data/attachments —
            # CWD-relative, and nothing derives it from FLEET_DATA_DIR. The
            # image has no WORKDIR, so cwd is /, and boot fails creating
            # /data/attachments/uploads as uid 1000 BEFORE the sandbox
            # preflight ever runs. deploy/fleet.service solves the same trap by
            # anchoring the CWD; a pod spec says it directly.
            - name: EMAIL_ATTACHMENT_DIR
              value: "/var/lib/fleet/attachments"
            # Cluster identity for the sandbox-pod orphan sweep. The UID names
            # this incarnation and the release name marks the owner, so a
            # restarted control plane can tell its predecessor's leaked pods
            # from a co-tenant release's live ones. Without these fleet falls
            # back to a pid heuristic that is meaningless across containers.
            - name: FLEET_POD_UID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid
            - name: FLEET_OWNER_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['app.kubernetes.io/instance']
            - name: FLEET_WORKSPACE_ROOT
              value: "/var/lib/fleet/workspace"
            # ── sandbox backend (#989) ──
            - name: FLEET_SANDBOX_BACKEND
              value: "kubernetes"
            - name: FLEET_SANDBOX_IMAGE
              value: "localhost/larkspur-sandbox:dev"
            - name: FLEET_SANDBOX_K8S_NAMESPACE
              value: "larkspur"
            - name: FLEET_SANDBOX_K8S_WORKSPACE_CLAIM
              value: "fleet-workspace"
            - name: FLEET_SANDBOX_K8S_NETWORK_POLICY
              value: "fleet-sandbox-deny-all"
            # Only stamped when the SA will actually exist. create=false without
            # external=true used to still set this, and every pod create then
            # failed with `serviceaccount "fleet-sandbox" not found` — after a
            # green preflight, which never checks the SA.
            - name: FLEET_SANDBOX_K8S_SERVICE_ACCOUNT
              value: "larkspur-sandbox"
            - name: FLEET_SANDBOX_K8S_IMAGE_PULL_SECRET
              value: "regcred"
            - name: FLEET_SANDBOX_K8S_BUNDLE_DOCS_IN_IMAGE
              value: "true"
            - name: FLEET_SANDBOX_MEMORY
              value: "2g"
            - name: FLEET_SANDBOX_CPUS
              value: "2.0"
            - name: FLEET_SANDBOX_DISK_GB
              value: "10"
            - name: FLEET_DEFAULT_NETWORK_MODE
              value: "lockdown"
            - name: FLEET_MAX_CONCURRENT_AGENTS
              value: "8"
            - name: FLEET_ORCHESTRATOR_BOOTSTRAP_ADMINS
              value: "you@example.com"
            - name: FLEET_TIMEZONE
              value: "UTC"
            - name: PERSONA_DEFAULT
              value: "assistant"
          envFrom:

Refs ElcanoTek/fleet#1264 (round-1 findings), fleet#1285/#1287/#1288/#1289/#1290.

🤖 Generated with Claude Code

The kind walkthrough was validated end-to-end for the first time
(ElcanoTek/fleet#1264, round 1: kind v0.32.0 / K8s v1.36.1, rootless
podman on WSL2). Everything the rehearsal proved wrong or stale in the
docs, fixed here; engine bugs it surfaced went to fleet as
ElcanoTek/fleet#1285/#1287/#1288/#1289 and issue #1290.

- The seal on kind is no longer categorically fake: kindnet enforces
  NetworkPolicy on recent kind (kindnet >= v1.3.0; verified live —
  sealed-label pod blocked, control pod through). Caveat 1, the
  DEGRADED ElcanoTek#3 / network-mode comments in values-kind.yaml, and the
  "Making the seal real" section (now "Verifying the seal") tell the
  version-dependent truth and give the seal test as the check, with the
  Calico path kept for old kinds.
- The seal test itself used https against busybox wget, which has no
  TLS — it fails for TLS reasons even unsealed and fakes a pass. Test
  with http.
- openssl joins the prerequisites, and step 4 gains a verify-the-secret
  snippet: a failed $(openssl ...) substitution still creates the
  secret with an empty token, and a junk OPENROUTER_API_KEY passed
  fleet validate-config outright before ElcanoTek/fleet#1289. Both
  bit during the rehearsal.
- Proof (c)'s view_file diagnosis pointed at the declaration/image;
  the actual failure on every backend today is fleet's task-run path
  policy (ElcanoTek/fleet#1290). The doc now says so and gives the
  interim checks (boot-log anchor line, absolute-path read in a
  sandbox pod).
- Caveat 4 ("no warm pool") was false on fleet before
  ElcanoTek/fleet#1288 — warmSize: 0 silently derived a two-pod pool.
  Noted in the caveat and beside the value; values-example.yaml now
  leaves warmSize unset (derive) instead of 0, which #1288 makes mean
  "no warm pool".
- The Makefile's "chart ships no values schema" comment predates
  fleet's values.schema.json; reworded to what the schema does and
  does not catch.
- Rough-edges table: kubectl cp needs tar (the control-plane image has
  none — stream via exec instead), and the kind node container does
  not survive a host reboot (podman start brings it back).

Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>
@KristianYe

Copy link
Copy Markdown
Author

Heads-up for review: commit daa7be8 reverses a claim the first commit made. The full re-validation run (fresh cluster, all fleet#1264 fix PRs applied) refuted "recent kindnet enforces NetworkPolicy": on kind v0.32.0's bundled kindnetd (v20260528), a sealed-label busybox pod fetched http://example.com with the deny-all object in place and its podSelector matching. The earlier "sealed" reading was the exact https://-on-busybox measurement trap this PR documents — which the now-mandatory unlabeled control run exposes.

The corrected posture was then verified end-to-end on the same rehearsal: cluster recreated with disableDefaultCNI: true + Calico v3.28.2 per the doc's own steps, seal test blocked with the control passing, FLEET_DEFAULT_NETWORK_MODE=lockdown applied, and a real model turn ran python3 -c "print(6*7)" → 42 inside a sealed sandbox pod (cold start, pool=0, zero parked pods, validate-config exit 0, /readyz ready with apiserver v1.36.1).

Kristian Yendrek and others added 2 commits August 26, 2026 14:03
…lean

Found on the fleet#1264 re-validation run: `fleet validate-config` exits
1 on a pristine install because release_tracker's enabled_env gate
(DEPLOY_API_TOKEN) is unset and the server was not marked optional —
the engine rightly treats an unmet gate on a NON-optional server as
blocking ("that server would silently fail to start"). But staying dark
without the token is this server's documented DESIGN ("a fresh checkout
runs clean with no secrets"), so it must be optional: the same check
now reports a non-blocking "optional connectors disabled" warning, and
validate-config exits 0 on a fresh checkout (verified in-cluster).

Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>
The re-validation run (fresh cluster, all fleet#1264 fixes applied)
refuted the previous commit's claim that recent kindnet enforces
NetworkPolicy: on kindnetd v20260528 (kind v0.32.0 / K8s v1.36.1), a
sealed-label busybox pod fetched http://example.com with the deny-all
object in place and its podSelector matching. The earlier "sealed"
reading came from the exact measurement trap this branch documents —
an https:// test URL failing for TLS reasons on busybox and mimicking
a block. (The standalone aojea/kindnet >= v1.3.0 enforces; kind's
embedded kindnetd is a different, minimal artifact.)

Caveat 1, the "Verifying the seal" section, the rough-edges row and
the values-kind comments now state the tested truth and demand both
halves of the measurement: sealed run AND unlabeled control, http not
https. The Calico path is the way to a real seal on kind — executed
verbatim on the rehearsal cluster: sealed pod blocked, control passed,
lockdown applied, and a model turn ran python inside a sealed sandbox.

Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>
@KristianYe
KristianYe force-pushed the docs/kind-round1-findings branch from daa7be8 to 4362939 Compare August 26, 2026 11:03
Kristian Yendrek and others added 3 commits August 26, 2026 15:38
… 8d seal test

Hand-ports the two fixes the kind walkthrough already received, found
again when the production page was executed verbatim on a real
RBAC-enforcing cluster (k3s, fleet#1264 round 2):

- 8c promised the smoke task reads protocols/ with view_file "no
  fallback"; under `fleet task run` that read fails on every backend
  (ElcanoTek/fleet#1290, the harness path-policy gap). The serve path
  is fine — a real chat turn read the doc with view_file, no fallback,
  on the round-2 cluster — so the check now drives a chat turn, and
  the original diagnosis table applies there.
- 8d's seal test fetched https with busybox wget, which has no TLS: it
  fails on an UNSEALED cluster too and fakes a pass. http, plus the
  mandatory unlabeled control run. Observed data points recorded:
  kind's bundled kindnetd does not enforce, k3s's embedded policy
  controller does.

Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>
… is a number to Helm

TAG defaults to the short commit hash, and roughly one hash in
twenty-seven is all digits. Helm's --set parses that as a number, and
fleet's values schema (correctly) rejects a numeric image.tag — hit
live in fleet#1264 round 2, where the bundle commit under test was
4362939 and every helm target failed schema validation until the flag
became --set-string.

Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>
The IPv6-first /models fetch warning (cosmetic; falls back to the
cached catalog, completions ride IPv4) and the one-shot harness
double-reserving warm capacity beside the server's pool on small nodes
(FLEET_SANDBOX_WARM_SIZE=0 for fleet task run). Both hit live on the
round-2 k3s cluster; the honest-scope checklist box in fleet#1264 says
anything hit that is not listed goes on the list.

Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Kristian Yendrek <yendrek.kristian@gmail.com>
@bradflaugher
bradflaugher merged commit 5dd9f55 into ElcanoTek:main Sep 2, 2026
bradflaugher pushed a commit that referenced this pull request Sep 2, 2026
#2 was written before ElcanoTek/fleet#1296 landed, so it told readers the
view_file leg of the smoke task fails under `fleet task run` on every backend
and to prove the docs side via a chat turn. That is fixed; the two-outcome
diagnosis applies to the smoke task as written again, with the old behaviour
kept as a dated aside for anyone pinning an older fleet.

The quoted boot-log line now names 3 bundle doc roots, not 4: skills are
staged into the workspace claim (fleet ADR-0055) rather than declared.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011GTaS29PKrUqckaMDizjm7
bradflaugher added a commit that referenced this pull request Sep 2, 2026
…-ioonpn

docs: the fleet#1290 caveat from #2 is fixed upstream; anchor count is 3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants