Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
5a226b6
docs: design large-cluster performance qualification
hellices Aug 5, 2026
a681186
test: add versioned scale workload profiles
hellices Aug 5, 2026
0a5285c
docs: clarify schema v1 initial state
hellices Aug 5, 2026
72c9805
test: generate deterministic scale replay traffic
hellices Aug 5, 2026
7964c53
test: expose optional Kubernetes read telemetry
hellices Aug 5, 2026
e0d7659
fix: reuse watch snapshot summaries
hellices Aug 5, 2026
808e008
test: report scale benchmark latency and resource use
hellices Aug 5, 2026
152808f
test: harden scale benchmark metric publishing
hellices Aug 5, 2026
9cf6565
fix: preserve tracemalloc across overlapping samplers
hellices Aug 5, 2026
fbf6a3a
fix: rollback process sampler startup failures
hellices Aug 5, 2026
10816f5
test: replay scale traffic through the real Textual app
hellices Aug 5, 2026
199fa23
test: fix input-timing and oracle-pin defects in replay harness
hellices Aug 5, 2026
bb35316
test: add reproducible scale benchmark CLI
hellices Aug 5, 2026
e1f7f71
test: harden benchmark CLI failures
hellices Aug 5, 2026
1ffbec5
test: fix absolute-offset delay bug in _ReplaySource; add time_scale=…
hellices Aug 5, 2026
38569b4
test: add digest-correctness assertion to time_scale=1 regression test
hellices Aug 5, 2026
e2acda1
test: add time_scale=1 HTTP 410 reconnect regression; correct event-r…
hellices Aug 5, 2026
a8bfb8c
fix(task-7-rereview): restore test sensitivity and eliminate timing r…
hellices Aug 5, 2026
ec26bad
fix(task-7-follow-up): replace sleep_callback with virtual-time seam
hellices Aug 5, 2026
358cf6d
test(task-7-review): pin replay timing across reconnects
hellices Aug 5, 2026
9d79c61
feat: add live seed manifest generator
hellices Aug 5, 2026
8852683
test(issue-186-task-8.2): add guarded real-AKS application-path replay
hellices Aug 5, 2026
f5b995c
fix(issue-186-task-8.2-review): isolate harness reads from app-path A…
hellices Aug 5, 2026
b6e1512
fix(issue-186-whole-branch-review): make the live qualification harne…
hellices Aug 6, 2026
a0eeebc
fix(issue-186-review): honor mutation Retry-After hints
hellices Aug 6, 2026
d04d8a5
fix(issue-186-review): desynchronize hinted retries
hellices Aug 6, 2026
86346dd
fix(issue-186-review): report terminal replay aborts
hellices Aug 6, 2026
ce4e4fa
fix(issue-186-review): complete PR #202 qualification review wave
hellices Aug 6, 2026
1534beb
fix(issue-186-review): bound az aks show call in live identity gate
hellices Aug 6, 2026
35fa7a5
fix: use supported az aks show args
hellices Aug 6, 2026
39c07d6
fix(k8s): refresh expiring exec credentials across configuration copies
hellices Aug 6, 2026
48103f9
test(ui): wait for the node-shell audit outcome, not the notification
hellices Aug 6, 2026
357ab4f
fix(perf-cli): fail live runs on failed UI scenarios and aliased arti…
hellices Aug 6, 2026
2c0d33c
fix(perf-live): make UI-at-scale scenarios prove what they claim
hellices Aug 6, 2026
4807a7c
fix(perf-live): verify the rendered table and stop miscounting reconn…
hellices Aug 6, 2026
4e38fb8
fix(perf): stop double-marking bursts and leaking tracing on sampler …
hellices Aug 6, 2026
ade1fd0
docs: publish the measured 1,000-pod scale envelope
hellices Aug 6, 2026
c298a9f
chore: retrigger CI after Actions outage
hellices Aug 6, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,10 @@ Full key reference: [docs/keybindings.md](docs/keybindings.md).
- **[Air-gapped operation](docs/airgap.md)** — internal LLM/Helm/OLM/image
endpoints, corporate CA trust (`network.ca_bundle`, Helm `--ca-file`),
responsibility boundaries, and a readiness checklist.
- **[Performance and scale](docs/performance.md)** — the measured envelope
(1,000 pods at 24 watch events/second for 31 minutes against a real
cluster), the budgets that pass and the two that miss, and the known
limits — including which size makes interaction feel sluggish.
- **[Threat model](docs/threat-model.md)** — exactly what crosses the
embedded-provider boundary, what is redacted, the MCP and plugin trust
boundaries, and the residual risks that are not mitigated. See
Expand Down

Large diffs are not rendered by default.

114 changes: 114 additions & 0 deletions docs/performance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Performance and scale envelope

What korvid has actually been measured doing, on what hardware, against which
cluster. Every number here comes from a recorded run whose manifest pins the
cluster, the workload profile, and the korvid commit. Nothing here is
extrapolated: a scale that is not listed has not been measured.

Reproduce any of it with the benchmark command described in
[the qualification design](dev/specs/2026-08-06-large-cluster-performance-qualification-design.md).

## Supported envelope

| | Measured |
|---|---|
| Objects in one view | 1,000 pods across 20 namespaces |
| Sustained churn | 24 watch events/second for 31 minutes |
| Session length | 31 minutes unattended, no reconnect, no relist |
| Correctness | 43,200/43,200 events applied, final digest matches, 0 dropped updates |

Replay-only profiles reach 10,000 and 50,000 objects. Those establish that the
store and diff paths do not fall over at that size; they are **not** a claim
that the live budgets below hold there, because they do not exercise the API
server, the watch decoder, or a real terminal.

## Live 1,000-pod result

Run `i186-20260806-195353` against a dedicated AKS cluster (Kubernetes 1.35.6,
5 × `Standard_D4s_v5`), driven from macOS arm64 / Python 3.12 / 10 cores.
Measured before and after the render-path work described below.

| Metric | Budget | Baseline | Optimized | |
|---|---:|---:|---:|---|
| Dropped updates | 0 | 0 | 0 | pass |
| Final digest mismatch | 0 | 0 | 0 | pass |
| LIST to 1,000-row table | ≤ 2 s | 1.12 s | 1.19 s | pass |
| Process start to interactive | ≤ 10 s | 2.75 s | 2.88 s | pass |
| Peak RSS | ≤ 512 MiB | 276 MiB | 271 MiB | pass |
| Post-warm-up RSS slope | ≤ 1 MiB/min | 1.06 MiB/min | 0.53 MiB/min | fixed |
| Event-to-render p95 | ≤ 250 ms @ 20 ev/s | 527 ms @ 24 ev/s | 299 ms @ 24 ev/s | **miss** |
| Cursor-input p95 | ≤ 100 ms | 2,311 ms | 2,447 ms | **miss** |

Supporting numbers: event-to-render p50 266 → 156 ms, p99 659 → 356 ms, max
1,628 → 714 ms; max backlog depth 42 → 39; achieved churn 23.18 → 23.996 ev/s
against a 24.0 target.

Render passes went *up* 51% (3,640 → 5,493) while latency went *down* 43%.
Before the change each pass was expensive enough that the update coalescer was
swallowing work to keep up; afterwards each pass is cheap enough to run more
often, so events reach the screen sooner and the backlog stays shallower. That
is also why achieved churn only reaches its 24.0 ev/s target after the change —
the event driver was previously being back-pressured by the renderer.

### What made it faster

A 31-minute CPU profile of the baseline run (1,877 s of samples, 2.5 B calls)
showed the cost was per-row-per-frame work on 1,000 rows, in three places:

- the in-place table diff read every surviving row back out of the `DataTable`
to decide whether it had changed — 7.35 M `get_row` calls and 102 M cell
comparisons per run, answering a question the writer already knew;
- `format_age` re-parsed the same ~1,000 timestamp strings every frame, pulling
`dateutil` into the hot loop for 96 s combined;
- `phase_style` recomputed a small closed set of styles 7.3 M times.

All three are now memoised. The one thing deliberately *not* cached is the
`Text` object returned for a phase cell: `DataTable` takes ownership of it and
mutates it, so a shared instance would corrupt unrelated rows.

## Known limits

**Cursor input is the binding constraint, not rendering.** Input
acknowledgement p95 sits above 2 s at 1,000 objects under full churn, more than
20× its budget, and the render-path work above did not move it. The client is
CPU-saturated (peak ~99.7%) at this size, so the render win buys headroom
rather than removing the ceiling. Treat 1,000 objects under sustained churn as
the point where interaction becomes visibly sluggish.

**Event-to-render p95 misses its budget, but the budget and the measurement do
not line up.** The budget is written at 20 events/s; the live profile runs at
24. The optimized 299 ms is a miss at the higher rate and has not been
re-measured at 20.

**UI-at-scale interaction timings are not yet trustworthy.** Filter, split-pane
and multi-log key sequences took seconds, not milliseconds, in both runs — but
both runs predate the harness fix that makes those scenarios wait for the
target UI state instead of for the keystroke to return. The recorded values are
upper bounds taken while the app was CPU-saturated, not clean measurements, and
they need re-running on the fixed harness before they mean anything.

**Burst drain is unmeasured live.** The 3-second post-burst drain budget is
exercised in replay only; the live profile contains no burst.

**Memory is not a limit at this size.** End-of-run allocation snapshots show no
unbounded growth attributable to korvid — the retained set is dominated by
transient watch-decode buffers and rich text fragments. The 0.53 MiB/min slope
over 31 minutes against a 271 MiB peak is drift, not a leak.

## Long sessions against AKS

Exec-plugin credentials (`kubelogin`, and any other
`client.authentication.k8s.io` provider) expire mid-session. korvid refreshes
them just before expiry and propagates the new token to every client, including
the websocket clients used for exec, logs, and port-forward. Without this a
watch established at connect time dies with HTTP 401 partway through — first
observed at ~22 minutes into a 30-minute run. If you see a session drop with
401 after a long idle period, that is a bug worth reporting, not expected
behavior.

## Raw artifacts

Each run emits a summary, a metrics JSON, a `cProfile` dump, a `tracemalloc`
snapshot, and the seed manifest that reproduces the exact workload. Those are
kept out of the product source history; issue #186 carries the run summaries
and links the artifacts.
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ dev = [
"types-PyYAML",
"azure-identity>=1.25.3",
"types-regex>=2026.7.19.20260720",
"psutil>=6.1",
]

[tool.ruff]
Expand Down
Loading
Loading