docs(fleet): spec one app across every cluster - #60
Merged
Conversation
…hat lies The Show HN thread asked twice for one app across many clusters. The docs already promised it at 04-multi-cluster.md:20 and :334, and promotion.go:144 builds its per-environment map with byEnv[in.Env] = in, so a prod environment holding two contexts renders one cell and drops the other without saying so. Specs a fleet screen, one row per cluster, and a split cell that refuses to collapse a disagreement. Records the TUI decline and the missing Argo CD comparison from the same thread. Leaves --serve under Beyond v1.
Backend first, since the matrix cannot render data nobody gathered: export the shared comparators, teach the engine to split, stop the service dropping contexts, then the fleet package, gatherer, and transport. The design screen gates the React work, per the rule against inventing UI. Three spec corrections found while reading the code. The drop starts at service.go:1006, not promotion.go:144. No cluster UID exists, so dedupe keys off the API server URL. Screen numbering is taken through 6, so fleet is Screen 7.
internal/fleet must order tags the same way the promotion matrix does. Two comparators disagreeing about which version is newer would be the worst defect the fleet view can carry, so it reuses these rather than growing a second implementation. No behavior change. Closes #61
…reement byEnv[in.Env] = in kept the last context and dropped the rest, so a prod environment holding three clusters rendered one confident version nobody verified. The cell now collapses only when every cluster agrees on tag and digest, and renders a split naming the disagreement otherwise. A split never becomes upstream for the next column, for the same reason an unreadable column never did. Closes #62
…and coverage math corrected Four review findings on 9a5394b: - row() counted absent and denied cells as drift, since resolve() routes them through the same "State != StateSame" check the old code used to continue past. Only cells actually evaluated should move drift; a split cell is a real disagreement and still counts. - resolve()'s collapse guard required len(present) == readable, which a namespace collapse (e.g. "shop" and "shop-prod" stripping to one identity in one environment) can never satisfy once Contexts is empty and clusters defaults to 1. That rendered a severe split citing a bogus cluster count. Guard on len(present) > clusters instead, and decide those cells on consensus alone without citing the count. - split()'s priority checked partial coverage before a same-tag/different- digest mismatch, hiding the worse fact (a mutable tag) behind the lesser one (a cluster not caught up yet). Digest mismatch now outranks coverage. - Cell.Namespace was dropped from absent/denied cells; restored where the instance is known. Removed the now-dead orElse helper.
Three defects the plan taught. Two test cases supplied 2 of prod's 3 contexts, so the partial-coverage branch preempted the behavior each test named. row() counted absent and denied cells as drift, which broke the shipped single-context matrix the task was told to protect. And split() reported partial coverage ahead of a mutable tag, hiding the worse fact. Also ignores .superpowers/, the per-plan execution scratch.
… count An unreadable context could still be collapsed into agreement when the environment also had more present instances than declared clusters, because that branch returned before the Unreadable guard ran. The unreadable rule wins in that case too: clusters nobody read might disagree. Also fixes two smaller defects in the same code: the digest-mismatch note cited the declared cluster count instead of the count that actually reported the tag, and the group namespace still dropped when the first instance in the group had an empty one.
service.go skipped a second context bound to the same environment before it was ever connected, on the assumption two contexts in one environment are identical. A team running prod in three regions disproves that, and the matrix rendered one region's version as prod's. Every context is now read, and the ones that do not answer are named on the environment. Closes #63
Task 3 turned up that clusters.Fetch never errors on an RBAC refusal. Every list reads err == nil and records what it could not read in Snapshot.Partial. The plan mapped 'app not in snapshot' straight to absent, so a developer without list permission on Deployments would have seen 'not deployed here' for a cluster nobody was allowed to look at. That is the exact conflation the five states exist to prevent, and service.go:196 already draws the line correctly for the apps list.
The spec said an all-denied environment renders StateDenied 'identical to today'. It does not. Service.Promotion builds denied placeholders with an empty app name, the filter strips them before the engine sees them, and resolve() returns StateAbsent with 'not deployed here' for an environment nobody could read. Tracked as #75 and treated as a ship blocker for the feature, since populating Env.Unreadable and then ignoring it is worse than never having gathered it.
PromotionView.Unreachable was appending an environment's name whenever any one of its bound contexts failed, even when its other contexts read fine. That renders a split cell full of real version data next to a banner claiming the same column is unknown. Unreachable now names an environment only when every context bound to it failed, matching its doc comment and what the split cell/Env.Unreadable already communicate for a partial failure. Also covers dedupeStrings directly (previously never exercised), records that an environment's risk comes from its first-seen context, and corrects the fakeCluster.denied doc comment: it takes the same Connect-failure branch as unreachable today, so the distinction is cosmetic until a fixture returns apierrors.NewForbidden from a list call instead.
resolve()'s empty-present path checked only Denied:true instances, which service.go strips before Build ever sees them (they carry no App name). An environment nobody could read fell through to StateAbsent, "not deployed here", the exact conflation the package doc warns against. resolve() now consults Env.Unreadable when no instances are present: a fully unread environment renders StateDenied naming how many clusters did not answer; a partially unread one renders StateDenied naming what was read and what was not, since the readable clusters agreeing on absence still cannot rule out a cluster nobody read. StateSplit was considered and rejected for the partial case: nothing was observed to disagree, so claiming a split would invent a conflict, and it would wrongly count as drift. Only an environment where every context answered may still say "not deployed here". Spec corrected alongside the fix. Closes #75
Round-1 review on task-3b (#75): add a test asserting Row.Drift and Summary.Split for the unread case the commit's reasoning rests on; strengthen the "not absent" test to pin StateDenied exactly so a future StateSplit refactor can't slip through green. Widen resolve()'s clusters so a comparison and its printed message never disagree ("2 of 1 clusters did not answer" when Unreadable outgrows Contexts). Shorten the partial- read note to match every other terse split() note.
One app, one row per cluster. Newest tag across the clusters that answered, and every cluster behind it. Five states stay distinct so a cluster behind a VPN is never read as a cluster without the app, and two contexts aimed at one cluster merge so the headline count stays honest. Closes #64
CompareTags returns zero for a pair it refuses to order, and zero is not greater than zero, so an incumbent newest that nothing can be ordered against was never displaced. Placements [sha-x, v1.0.0, v2.0.0] answered "newest sha-x, nothing behind" while one cluster sat two minors back, and the same three in another order answered correctly. The headline moved with the caller's slice order. newest is now taken over orderable tags only, via a new promotion.Orderable that reuses the exact predicate CompareTags applies, so the two cannot drift. A build id is labelled in its own row and never counted behind. A present cluster with no tag is its own fact. A digest-pinned image is ordinary GitOps and TagOf returns empty for it; the empty tag then matched the empty newest and every row rendered as up to date, or shipped the literal note "runs , which cannot be ordered against v2.14.0". It now says the version is unknown and names the digest pin as the cause, and a tag that does not exist is never called mutable. Digests are grouped per tag rather than only under the newest one, so an older tag resolving to two digests is the defect it is. An incoming DigestPending is believed instead of overwritten, and View.DigestUnverified counts the present rows whose digest never arrived, so a headline reporting agreement can say how much of it rests on unread digests. A cluster nobody reached now sorts above a cluster whose older version you can see, matching the promotion matrix: neither agreement nor a defect can be claimed from clusters we never saw. An unrecognized state reads as pending and says so rather than rendering blank among the healthy rows. Merging builds a fresh alias slice instead of appending through the caller's backing array, and mergeByCluster now records the upstream invariant that ClusterID must come from the kubeconfig, not from a successful connection. Tests: newest independent of placement order, no newest when nothing is orderable, both empty-tag paths, a mutable tag on a non-newest tag, DigestPending respected and counted, alias contents across three merge shapes, every note string including reason passthrough and fallback, unreadable sorting above behind, and the unrecognized state.
Walks every kubeconfig context, connects lazily, and matches by the same identity the promotion matrix uses. A cluster that will not answer becomes an unreachable row naming the reason, never an omitted one, because a missing row reads as "not deployed there". Deduping keys off the API server URL, which config already normalizes, read before any connection attempt so an unreachable duplicate context still merges with its reachable twin. Replaces the cosmetic denied flag on the shared test fixture, which took the identical branch as unreachable, with refusedKinds: a fake clientset that connects but refuses specific List calls, the only way to exercise a genuine RBAC-denied row rather than an absent one. Closes #65
Every cluster that simply did not run the app reported "denied". The snapshot was taken at the background tier, which appends "Pod" to Snapshot.Partial unconditionally as a deliberate skip, so the len(Partial) > 0 test was constant-true and StateAbsent was dead code. Two changes, either of which would have hidden the other's absence: - Fetch at clusters.TierActive, the way cmd/kubeside's gather already does for this kind of sweep. That drops the phantom Pod entry and brings the digest, which lives only in pod status, so a tag resolving to two digests is findable at all. - Test denial only on kinds that could carry the app. Partial unions a refusal with a skip and only the first implies denial; a refused Pod or ReplicaSet list hides health, never existence. The Scope half of that test could never fire and was inverted besides. A namespace-scoped fallback means the cluster refused to enumerate its namespaces and only one was read, so an app living elsewhere was never looked for. scopeCovers asks whether the namespace we did read would have carried the identity we asked for, using the rule findApp matches with. A denied row now names a verb, a resource and a scope, which is the shape of a Role rule, and never names a kind nobody refused. The sweep gives every context its own goroutine and its own copy of the timeout, because the flag means per-cluster connect and fetch. One budget spent serially let a few slow clusters report every cluster behind them as unreachable, inventing the fact this screen exists to keep honest. Rows are written by index, so the order is the kubeconfig's and not the scheduler's. findApp no longer drops a second match in silence: it prefers the namespace asked for, falls back to name order, and says what it skipped. fleet.Build surfaces a present row's Reason so that choice is visible.
Step 4 tested len(snap.Partial) > 0, which is constant true: fetch.go appends Pod to Partial at any non-active tier as a deliberate skip, not a refusal, and Fleet reads every context at background tier. So absent was dead code and every app not deployed somewhere reported as denied. Records that, the unreachable Scope clause, and the shared serial timeout, with a pointer to what shipped instead.
GET /api/fleet?app=&namespace=. A request without an app is a 400 rather than an empty screen, because the view is about one app by construction and an empty fleet screen would read as "this app runs nowhere" — the conflation this feature exists to prevent. Namespace is optional: Service.Fleet already accepts "" and matches whichever namespace each cluster holds the app in, so the transport does not add a requirement the service does not have. Widens the API interface with Fleet(app, namespace string) fleet.View and updates both test fakes (stubAPI, streamStub) that implement it. Closes #66
Ported from screens/fleet.html in the design project. Five row states render distinctly, an app found nowhere names the clusters it asked rather than showing an empty window, and a tag resolving to two digests raises a banner above everything else on the page. Closes #68
environments.Classify matches by keyword token and returns the name it was
handed, so prod-us-east, production, live, staging-eks, uat and sandbox-3 are
all classified while spelling no tier a screen could compare against. The fleet
screen tested the name against the literal set {qa, stg, prod}, painted
everything else violet, and printed the word "unclassified" over a cluster the
promotion matrix and the shell both render red. Environment colour is a safety
layer in docs/04-multi-cluster.md, so that was a false statement about a safety
property, not a missing colour.
Placement now carries EnvColor and EnvRisk beside Env, populated in the one
literal every state path returns, so an unreachable and a denied row keep their
colour too. The screen reads envToken off the row and prints the resolved name
verbatim, the way every other screen does. The rule stays in one language: a
tokenizer reimplemented in TypeScript would disagree with the binary the first
time someone edits tierRules.
Also on the screen: a row whose namespace differs from the one asked for now
says so, since identity matching strips the environment token and the
substitution was invisible; the mutable-tag banner derives its edge from the
rows actually in conflict rather than inheriting the shell's; the digest note
stops reporting zero as a problem; the severity card stops enumerating a ladder
it cannot keep in step with; and an empty kubeconfig gets a card naming the
cause instead of a header-only table.
An environment whose clusters disagree shows the disagreement and offers the screen that explains it, rather than a version the matrix never verified. Closes #69
The command appears only with an app in hand, because the view is about one app and a palette entry that opens an empty one is a dead end. Closes #70
…line One cluster unreachable, one denied, and the screen still fills, with the two rendering differently: conflating them is the failure the five states exist to prevent. FleetScreen.tsx carried logic vitest could never reach, since the include pattern skips .tsx and there is no jsdom in this package. The not-present label map, the ready-ratio split, and the verdict badge precedence now live in fleet.ts beside it, unit tested, with rendered output unchanged. A Playwright spec exercises every reachable row state in one view and carries the fix for the layout bug that state made visible: the Env column was sized for prod/stg/qa, and a resolved name like prod-us-east overflowed it and pushed Verdict out of place. Closes #71
04-multi-cluster promised per-context expansion inside the promotion view; what shipped is a split cell plus a dedicated screen. It also claimed dedupe by API server URL and cluster UID; only the URL ships, because no cluster UID exists here and reading one needs a permission a namespace-scoped developer would be refused. Closes #72
It was the first objection raised on Show HN and the comparison never mentioned it. Sync state against git is a delivery question; the app across its environments is a different one. Closes #73
Asked on Show HN. 05-architecture already rejected a terminal UI and the reasoning was not reachable from the docs a user reads. Closes #74
05-architecture said contextID derives from a cluster UID and that a rename preserves stored history. Neither is true. Nothing in internal/ reads a cluster UID, and kubeside stores no history to preserve, which is the whole bet recorded in 04-multi-cluster. #72 corrected the same claim in 04-multi-cluster and this was the one remaining spot asserting a mechanism that exists nowhere in the code.
The product spec said seven views ship and then listed an eighth, while exec was counted nowhere. Nine ship. It also said fleet reuses promotion's five states. Promotion has eight, and none of them are present, unreachable, or pending. Only absent and denied are spelled the same, and they differ: promotion has no unreachable, so a cluster that never answered reads denied there and unreachable here. What fleet reuses is CompareTags, Identity, and Orderable. The guide claimed no other screen wakes every context. Promotion does too. Fleet's real difference is that it connects concurrently and reads pod status everywhere, which is where the digest lives.
byEnv split() treated every unreadable-adjacent cell the same way: severe, counted as drift, counted in Summary.Split. But when the clusters that actually answered fully agree with each other, there is nothing to flag about them, only an unrelated cluster that never spoke. That is the same epistemic state issue #75 already routes to StateDenied when the app is absent everywhere readable. Deciding by whether the app happened to be found, rather than by what was actually observed, was the bug. resolve() now intercepts full agreement plus an unread cluster before split() is ever called, and renders it StateDenied: not severe, excluded from drift, excluded from Summary.Split, and still never upstream, since the silent cluster might disagree with the ones that spoke. split() itself no longer lets an unread cluster override a witnessed disagreement. It used to check Unreadable first and report only "N of M readable", silently dropping a real mutable-tag or tag-mismatch finding whenever any cluster also happened to be unread. Now the disagreement drives the note, with the unread count appended rather than substituted. Closes #76
TestPromotionMarksAnUnreachableContextOnItsEnvironment fed Service.Promotion a single present cluster plus one unreachable context in the same environment. Under #76, a lone cluster with nothing to disagree with is no longer treated as a disagreement: it now renders StateDenied, matching every other "readable clusters agree, one didn't answer" case in the matrix. The old assertion (StateSplit) encoded the behavior #76 exists to fix.
Connect released mu before dialing, so two concurrent callers for the same context could both see a non-live state and both dial. The second result silently overwrote session set by the first. Harmless today (Session.Close is a no-op), but the outcome depended on which dial finished last, and it wasted a redundant connection attempt every time it happened. dialMu now wraps the whole check-dial-update sequence per conn. A second caller blocks until the first finishes, then re-checks state under mu and returns that outcome instead of starting its own dial. Different contexts have independent conns and independent dialMu locks, so this changes nothing about one cluster never blocking another. Confirmed the race directly: reverting the fix and running the new test under -count=10 failed 4 times with 2-12 duplicate dials for one context. With the fix, 10/10 clean under -race. Closes #78
Two related conflations in Service.Promotion, both landing on the same principle #75 already established: an environment we could not verify must never render "not deployed here". #77: clusters.Fetch never errors on an RBAC refusal. A forbidden Deployment list lands in Snapshot.Partial with a nil error, and Promotion discarded Partial entirely, so a refused kind looked exactly like an app that does not exist. Now a refused app-carrying kind marks the context unreadable, alongside whatever the read did find, so resolve() can tell "not deployed here" apart from "might be here, we were refused" the same way it already does for a context that never answered. Deliberately coarser than Service.Fleet's per-app precision: Promotion compares every app in bulk, and flagging every namespace-scoped RBAC context (the common case) would bury the signal. #82: Connect's error was captured but never acted on directly — the code proceeded to ClientFor regardless, which returns any existing session whenever one exists, live or not. Connect's error now short-circuits before ClientFor is ever consulted, matching the pattern Service.Fleet already used. Tracing the state machine: the precondition this guards against (state not Live, session still set) was primarily reachable through the double-dial race #78 just closed, where a losing dial's failure could overwrite a winning dial's Live state without touching its session. The check stays as defense in depth regardless. orElseString is now dead code with this rewrite and is removed. Closes #77 Closes #82
…hecked #80: fleetView.rows was fixture order (qa1, prod-us-east, prod-eu-west, stg1, dr-frankfurt), not fleet.severity()'s order. The frontend does not sort — Go does — so the committed baseline rendered whatever order the fixture happened to list, directly under a note card titled "Sorted by what needs attention". Reordered to match severity (unreachable, behind, denied, then the two ties broken alphabetically) and pinned it with a row-order assertion, not only a picture. #81: promotionView had no cell with state "split", so PromotionScreen.tsx's split branch, the only new promotion UI on this branch, was exercised by nothing. It renders a full backend sentence into a version-sized column. Added a split cell (api-gateway, a 3-context prod with a mutable tag) and assertions that the note renders verbatim, in the error hue, and that clicking it opens the fleet screen for that app. Both screenshot baselines regenerated and verified with a clean second run. Closes #80 Closes #81
…eachable
vite.config.ts's vitest include matched .test.ts only, never .test.tsx, and
there was no jsdom or testing-library anywhere in the package. So nothing a
.tsx file decided for itself was ever exercised by a unit test. health.test.ts
tested envToken directly, but never rendered FleetScreen, so the component's
own wiring of envToken shipped the prod-renders-violet bug undetected: the
first version of the fleet screen matched environment names against the
literal set {qa, stg, prod} instead of using the colour/risk the backend
already resolved, and no test in the tree could see it.
Adds jsdom, @testing-library/react, and @testing-library/jest-dom. A .test.tsx
file opts into jsdom per-file with a @vitest-environment pragma rather than a
global environmentMatchGlobs, which vitest 3.2 deprecated in favour of
test.projects — per-file is the right grain anyway, since most tests never
need a DOM.
FleetScreen.test.tsx renders the screen with an environment named
"production" and asserts it resolves to the prod token and prints its name
verbatim. This is the exact class of bug the harness exists to catch, proven
by rendering it, not by asserting the fix exists.
Documents three tiers of test in vite.config.ts: pure logic (.test.ts),
component decision logic (.test.tsx, jsdom), rendering fidelity (Playwright).
Prefer extracting logic out of a component into a plain function next to it,
as fleet.ts already does, so it stays in the fast tier.
npm audit found one high-severity transient advisory (nanoid) pulled in by
these new dev dependencies; npm audit fix resolved it to zero vulnerabilities
before this commit.
Closes #83
PR #60's visual CI check has been failing since the fleet screen shipped: only a macOS baseline was ever committed for fleet.png, and CI runs on Linux, so it had nothing to diff against and reported a failure with no underlying regression. promotion.png's Linux baseline also needed regenerating after the split-cell fixture (#81) and the row reorder (#80) changed the pixels. Generated inside mcr.microsoft.com/playwright:v1.61.0-noble, the exact image ci.yml's visual job runs, matching Node 22 and the same npm ci install. Verified stable with a second run, no --update-snapshots, 27/27 clean.
GoReleaser generates release notes from commit history at tag time, but nothing tracked what a release actually contains between tags. This is the same information in plain language, kept as the branch goes rather than reconstructed from commit subjects later. Unreleased section covers this branch: the fleet screen and the promotion-matrix correctness fixes underneath it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design spec for the work the Show HN thread asked for, plus a defect the thread uncovered.
What the thread said
item 49139573, 25 points, 8 comments.
Two people asked the same thing independently. maxweisel runs one app across several clusters and wants to know whether all of them are on the latest version. I confirmed the same pain: clusters per team, plus uat, plus prod, and the recurring need to check one service everywhere. A third commenter read the README tagline and assumed it already shipped.
The defect
docs/04-multi-cluster.md:20says one environment maps to one or more contexts and the promotion view "aggregates them with per-context detail on expansion." Line 334 repeats it for an environment with 40 contexts.internal/promotion/promotion.go:144builds its per-environment map withbyEnv[in.Env] = in. Last write wins. Aprodenvironment holdingprod-us-eastandprod-eu-westrenders one cell showing whichever context arrived last and discards the other silently.So today the promotion view answers "is the fix in prod" with a version it never verified. That is worse than the gap I described in my reply on the thread.
What this specs
docs/01-problem.md, since it was the first objection raised.docs/05-architecture.md:20, which already rejects a terminal UI with reasoning.--servein-cluster mode stays under Beyond v1. The thread raised demand for it. Demand does not shrink the OIDC and Helm work.Sequencing note
The Claude Design project has no fleet screen. CLAUDE.md forbids inventing UI, so the first issue authors
screens/fleet.htmlthere and human review of it gates the React work.No code in this PR. Spec only, twelve issues scoped, ten under a new
feature:fleetlabel.