Add UI load-test harness: synthetic fake cluster with live pod-count control

[hisco]

What

Lets developers drive Radar's UI at tens of thousands of pods with a live-controllable pod count, without a real cluster, real workloads, a kubelet, or etcd. Radar connects to an in-process fake Kubernetes client and runs its real informer → topology → SSE → React render pipeline, treating the synthetic objects exactly as it would a real cluster.

This is the leanest realization of the industry pattern behind KWOK ("Kubernetes WithOut Kubelet") — fake objects, no containers — but with no separate control-plane process: the objects live only as Go values in the test server's memory. The generated objects are standard Kubernetes objects, so the same generator feeds a KWOK cluster as-is if real-wire fidelity is ever wanted.

How

• internal/loadtest — a generator producing a coherent Deployment → ReplicaSet → Pod population with a matching Service/ConfigMap/Secret per app (the pod template references the ConfigMap and Secret, so topology shows the "Configures" edges), spread across fake nodes and namespaces. Names are deterministic; the object set is a pure function of the pod count.
• cmd/testserver — new -pods / -nodes / -namespaces / -pods-per-app flags seed the population at construction, so it arrives through the informers' initial LIST (safe at any count). With -pods 0 the existing nginx fixture is unchanged (Playwright e2e unaffected). A small admin listener (default <port>+1) exposes POST /loadtest/scale {"pods":N} and GET /loadtest/status.
• Live scaling mutates the fake client through its watch path, whose RaceFreeFakeWatcher has a 100-event buffer that panics on overflow. Mutations are therefore batched below that bound and paced against the informer's observed count between batches — for every watched kind, not just pods — advancing progress per drained batch so a mid-scale failure stays consistent. scale returns once the Pod informer has converged.
• make loadtest target + docs/loadtest.md.

Verification

• 50,000 pods: informer store complete (Pod:50000, plus 250 each of Deployment/ReplicaSet/Service/ConfigMap/Secret), ~800 MB RSS, /api/resources/pods returns all 50k in ~0.2s.
• Live scale up/down (50k→200→8000→0) all converge with zero watch panics; in-browser the Pods count updates live over SSE with no reload.
• Unit tests cover the topology math and a scale round-trip through a live informer (-race clean).

A cross-model review (cursor/gpt-5.5-high) flagged that non-pod app mutations were initially unbatched (panic risk), the boundary reconcile over-patched, progress state could go stale on partial failure, and the admin bind was silent on failure — all fixed in this branch.

Known limits (documented in docs/loadtest.md)

• Exercises everything at and above the informer store (informer memory, topology CPU, SSE fan-out, virtualized render); deliberately not the real network transport / client-go decode / apiserver.
• The test server doesn't initialize API discovery, so sidebar count badges for grouped kinds (Deployments, Jobs, …) render as "–" though the data is present (/api/resource-counts returns real numbers). Wiring discovery is the same step CRD kinds (HTTPRoutes, etc.) will need — the natural next extension.
• Above ~25k pods in scope the Pods table shows Radar's built-in "too many to show" guard — a real responsiveness limit this harness is well suited to exercise.

https://claude.ai/code/session_01LFEw9hKzrvDUfVMf7p38SJ

---

Note

Low Risk
Changes are confined to dev/test tooling (testserver, loadtest, Makefile, docs) and do not alter production cluster connectivity or auth paths.

Overview
Adds a UI load-testing path that runs Radar against a large, topology-realistic fake cluster (no real kubelet/etcd), with optional live pod count control over HTTP.

A new internal/loadtest package seeds and mutates deterministic Deployment → ReplicaSet → Pod graphs plus per-app Services, ConfigMaps, and Secrets across fake nodes/namespaces. ScaleTo batches creates/deletes and waits on informer counts so the fake client-go watch buffer (100 events) does not panic; partial-failure handling avoids orphan app skeletons.

cmd/testserver gains -pods / -nodes / -namespaces / -pods-per-app (load-test mode) while -pods 0 keeps the existing nginx Playwright fixture. When load-test mode is on, an admin listener on <port>+1 exposes GET /loadtest/status and POST /loadtest/scale, pacing against Radar’s resource cache via informerCount.

make loadtest (default 50k pods) and docs/loadtest.md document usage, live scaling, and known limits (e.g. no API discovery for grouped-kind sidebar badges). Unit tests cover topology math, informer round-trips, and failure/retry/orphan cleanup scenarios.

^{Reviewed by Cursor Bugbot for commit 67abbc9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[hisco]

Both Cursor Bugbot findings addressed:

• Orphan apps after failed scale-down (High) — fixed in db62f56: appsHave now comes from an independently-tracked materialized app count, not from pod progress.
• Stale app count after create failure (Medium) — fixed in 158aede: both the pod and app high-water marks now advance per object actually written to the client (app count before each app's objects are written), so a mid-batch Create failure or context-cancel can't leave unrecorded skeletons that a later scale-down would orphan.

Each is covered by a regression test (partial-progress scale-down; injected mid-scale-up Deployment create failure asserting a subsequent scale-down leaves no skeleton of any kind in the client). -race clean.

[cursor]

Incomplete apps skipped on retry

Medium Severity

createApps bumps currentApps before the app's objects are fully created. ScaleTo then treats that index as done and starts the next create at appsHave, so a retry of the same or higher target never finishes a partially written app. Pods for that app can still be created against missing Deploy/RS/Service objects, leaving a broken synthetic topology after a cancelled or failed scale-up.

Additional Locations (1)

• internal/loadtest/generator.go#L365-L373

 

^{Reviewed by Cursor Bugbot for commit 158aede. Configure here.}

[hisco]

Third Bugbot finding addressed:

• Incomplete apps skipped on retry (Medium) — fixed in 1909bfb. A partially-written app needs opposite handling on the two paths (cleanup must delete it; a create retry must finish it), which a single high-water mark can't provide. Now tracked with two marks: appsCompleted (all five objects written — where a create resumes, so the partial app is re-completed via idempotent creates) and appsMaterialized (furthest app any object was written for — the cleanup delete bound). Regression test fails a scale-up mid-app and asserts the retry finishes it; the prior orphan/stale tests still pass. -race clean.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1909bfb. Configure here.}

[hisco]

Fourth Bugbot finding addressed:

• Stale replicas after failed scale-up (Medium) — fixed in 67abbc9. reconcileBoundaryApps derived the old boundary from appsFor(from) (pod-derived); when pods lag the app count after a partial failure that targets the wrong app. Now reconciles against the app count captured before this scale's mutations. Regression test diverges the pod counter below the app count and asserts the former boundary Deployment's replicas are corrected on the next scale-up.