perf: add guarded large-cluster qualification harness

[hellices]

Summary

• add deterministic 1k/10k/50k replay profiles and a benchmark CLI that exercises the real Textual application path
• add guarded AKS live qualification for 1,000 owned Pods, including convergence, input/render latency, memory, API-traffic, and digest correctness evidence
• instrument Kubernetes reads and harden watch/retry behavior, including Retry-After preservation, bounded deterministic jitter, and desynchronized retries
• document the qualification protocol, safety gates, artifact policy, and evidence thresholds

Why

Issue #186 requires evidence-driven large-cluster qualification before changing production hot paths. This harness establishes reproducible offline baselines and a fail-closed live workflow so optimizations are made only when measured AKS evidence demonstrates a budget violation or bottleneck.

Safety and review notes

• live mutations are metadata-only and restricted to resources carrying the run ownership labels and expected UID
• spec/status mutation, deletion, and node drain are prohibited
• JSON Patch ownership tests close the validation-to-mutation race
• raw live artifacts are not committed; the live run requires exactly four named artifacts for external retention
• HTTP 429 is the only retried mutation failure, with finite bounded retry behavior

Validation

• make check
  • ruff: passed
  • mypy: passed (295 source files)
  • pytest: 3486 passed, 21 skipped
  • tach: passed
• targeted retry and live-performance regressions passed before the full gate
• independent re-review of the final retry desynchronization change: PASS

Closes #186

[Copilot]

Pull request overview

Adds deterministic and guarded live large-cluster qualification infrastructure using the real application path.

Changes:

• Adds 1k/10k/50k workload profiles and replay/live benchmark CLIs.
• Adds metrics, Kubernetes read telemetry, guarded churn, and retry handling.
• Adds extensive tests and qualification documentation.

Reviewed changes

Copilot reviewed 27 out of 29 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
pyproject.toml	Adds psutil development dependency.
uv.lock	Locks psutil packages.
src/korvid/k8s/client.py	Instruments Kubernetes reads.
src/korvid/k8s/errors.py	Preserves retry delay metadata.
src/korvid/k8s/telemetry.py	Defines read telemetry events.
tests/k8s/test_client.py	Tests telemetry behavior.
tests/ui/waits.py	Adds operational timeout exception.
tests/ui/test_waits.py	Tests timeout semantics.
tests/performance/__init__.py	Defines performance test package.
tests/performance/cli.py	Implements benchmark CLI.
tests/performance/live.py	Implements guarded AKS replay.
tests/performance/manifests.py	Generates live seed manifests.
tests/performance/metrics.py	Collects and renders measurements.
tests/performance/profile.py	Loads and validates profiles.
tests/performance/replay.py	Drives deterministic real-app replay.
tests/performance/workload.py	Generates deterministic workloads.
tests/performance/test_cli.py	Tests CLI behavior.
tests/performance/test_live.py	Tests live safety and replay.
tests/performance/test_manifests.py	Tests manifest generation.
tests/performance/test_metrics.py	Tests metric calculations.
tests/performance/test_profile.py	Tests profile validation.
tests/performance/test_replay.py	Tests application-path replay.
tests/performance/test_workload.py	Tests workload determinism.
tests/performance/profiles/aks-1k.json	Defines AKS comparison profile.
tests/performance/profiles/aks-live-1k.json	Defines 30-minute live profile.
tests/performance/profiles/burst-50k.json	Defines 50k burst profile.
tests/performance/profiles/smoke-1k.json	Defines CI smoke profile.
tests/performance/profiles/steady-10k.json	Defines 10k steady profile.
docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md	Documents qualification protocol.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 27 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (4)

tests/performance/profile.py:109

• Profiles can declare multiple failures at the same at_event, but run_replay converts them to a dictionary keyed by event number, silently retaining only the last one. Reject duplicate event positions so a profile cannot claim failure injections that were never exercised.

    return tuple(sorted(result, key=lambda failure: failure.at_event))

tests/performance/live.py:1205

• The live path never calls BenchmarkRecorder.mark_burst_end (the only production call is in the offline replay source), so every live report records an empty drain list and max_post_burst_drain_seconds=None. That makes the published ≤3s live burst-drain budget impossible to evaluate. Schedule burst-end marks on the live clock while churn is running, including the case where the backlog is already empty.

        events = scheduled_events(profile)
        state.progress.requested_events = len(events)
        churn_task = asyncio.create_task(
            drive_live_churn(
                events,

docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md:85

• This documented replay-live command is guaranteed to fail validation because the CLI requires all four live artifacts, while the example supplies only --json and --out. Include run-labelled --cpu-profile and --allocation-snapshot destinations so operators can execute the published protocol.

uv run python -m tests.performance.cli replay-live \
  --profile tests/performance/profiles/aks-live-1k.json \
  --context aks-korvid-contract-test \
  --expected-cluster-id <ARM resource id> --run-id <run> \
  --json live.json --out live.md

tests/performance/cli.py:272

• An unwritable allocation-snapshot destination raises OSError here, but both callers invoke this helper from a finally block after their surrounding except OSError handler. The error therefore escapes with a traceback (and can replace the original replay failure) instead of returning the documented operational exit code 1. Move snapshot error handling into the protected path while preserving any active replay exception.

    snapshot = tracemalloc.take_snapshot()
    stats = snapshot.statistics("lineno")[:100]
    Path(path).write_text("\n".join(str(stat) for stat in stats))
    tracemalloc.stop()

[Copilot]

Pull request overview

Copilot reviewed 31 out of 33 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

tests/performance/live.py:1047

• The live driver never calls BenchmarkRecorder.mark_burst_end (the only call is in the offline replay source), so every live report leaves post_burst_drain_seconds empty and max_post_burst_drain_seconds unset. That makes the published ≤3-second live burst-drain budget impossible to evaluate. Mark each profile burst boundary on the same clock used by live render recording and ensure an already-empty backlog records a zero drain.

            for event in events:
                elapsed = now() - start
                delay = event.offset_seconds * options.time_scale - elapsed
                if delay > 0:
                    await sleep(delay)

tests/performance/profile.py:109

• Duplicate at_event values are accepted here, but run_replay later builds {f.at_event: f}, silently discarding all but one declared failure. The profile hash/report can therefore claim multiple injections that were never exercised. Reject duplicate failure event positions during profile loading.

    return tuple(sorted(result, key=lambda failure: failure.at_event))

[Copilot]

Pull request overview

Copilot reviewed 30 out of 33 changed files in this pull request and generated 1 comment.

Suppressed comments (7)

tests/performance/live.py:1544

• The production composition root wires MetricsPoller(kube.list_pod_metrics) (src/korvid/__main__.py:1060), but this measured app omits metrics. Consequently the 30-minute live run does not exercise metrics polling or measure responsiveness/API load while metrics updates are active, despite the qualification requirements. Wire the real poller through the telemetry-enabled client.

                app = MeasuredKorvidApp(
                    config=KorvidConfig(
                        namespace=ALL_NAMESPACES,
                        # Key 1 scopes to a real seeded namespace; key 0 returns
                        # to all namespaces. Without a favorite the toggle is a
                        # no-op (the configured namespace already is
                        # ALL_NAMESPACES) and the scenario proves nothing.
                        favorite_namespaces=(manifests.namespace_name(run_id, 0),),
                    ),
                    store=store,
                    watch_manager=watch_manager,
                    recorder=recorder,

docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md:96

• The generator does not assign UIDs or resource versions: initial_pods leaves PodSummary.uid at its empty default and the event model has no resource-version field. Remove those claims or add them to the versioned workload and hashes; otherwise the reproducibility documentation overstates what is deterministic.

The deterministic generator emits stable names, namespaces, UIDs, resource
versions, and event order from the profile seed. Repeating a profile with the
same seed must produce the same object and event hashes.

tests/performance/metrics.py:198

• reconnects counts every WATCH open after the first on a path, including deliberate stop/start cycles. The live namespace scenario necessarily opens /api/v1/pods three times (test_live.py:1052-1064), so a healthy run is reported as having two reconnects. Emit/record reconnects explicitly from retry recovery instead of inferring them from aggregated opens.

        reconnects = sum(max(counts.get("watch_open", 0) - 1, 0) for counts in paths.values())

tests/performance/metrics.py:456

• When the backlog is already empty at the burst boundary, there is nothing to drain and this sample should be 0.0. Keeping the marker pending makes the next unrelated steady-state render produce a positive drain time (or leaves the sample missing if no later render occurs), corrupting the burst-drain budget.

        self._burst_end_pending.append(at)

tests/performance/replay.py:596

• The report's “final” digest is computed only from ResourceStore, not from the rendered table. A stale/missed ResourceTable cell—especially in the newly cached in-place diff path—can therefore pass with matching digests and zero drops. Validate the table's rendered state independently before reporting digest correctness.

    # Compute final digest from the store (actual state).
    final_digest = summary_digest(cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES)))

tests/performance/cli.py:272

• An unwritable --allocation-snapshot destination raises OSError here from a finally block, outside the command's error handler. After a long run this produces a traceback and can mask the original replay failure, unlike the other artifact writes that return exit 1. Handle snapshot write failures without overriding an in-flight exception.

    Path(path).write_text("\n".join(str(stat) for stat in stats))
    tracemalloc.stop()

docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md:85

• This documented live command is rejected because replay-live now requires all four artifacts and requires each filename to include the run ID. Add --cpu-profile and --allocation-snapshot, and use run-labelled names for every output so the published protocol is executable.

This issue also appears on line 94 of the same file.

uv run python -m tests.performance.cli replay-live \
  --profile tests/performance/profiles/aks-live-1k.json \
  --context aks-korvid-contract-test \
  --expected-cluster-id <ARM resource id> --run-id <run> \
  --json live.json --out live.md

[Copilot]

Pull request overview

Copilot reviewed 30 out of 33 changed files in this pull request and generated no new comments.

Suppressed comments (5)

tests/performance/replay.py:596

• The reported final_digest is computed from ResourceStore, not from the rendered ResourceTable. A stale or skipped table update therefore still produces digest parity and a successful replay, despite the report claiming store/table correctness and despite this PR changing the table diff cache. Validate the rendered rows against the expected summaries while the app/table is still available, as the live path now does.

    # Compute final digest from the store (actual state).
    final_digest = summary_digest(cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES)))

tests/performance/cli.py:378

• The live command can still exit successfully after violating most of the documented hard budgets. This shared check only enforces dropped updates and digest parity (with UI failures handled separately), but it never rejects missing/over-budget startup, LIST-to-table, event/input p95, burst drain, RSS slope/peak, or unexpected GET metrics from the hard-budget table. Add a live-profile budget evaluator that treats missing required measurements and threshold violations as failures.

    if replay.dropped_updates > 0 or replay.expected_digest != replay.final_digest:
        return 1

tests/performance/cli.py:490

• replay-live requires --cpu-profile, so every successful qualification takes this branch and runs the entire measured window under deterministic cProfile. That instrumentation materially changes CPU and event/input latency, making the resulting values unsuitable for the user-facing hard budgets the run is supposed to qualify. Collect the profile in a separate companion run (same SHA/profile) or use low-overhead sampling so the budget run itself remains unprofiled.

        if args.cpu_profile:
            replay = _run_live_with_cpu_profile(
                profile,
                options,
                context=args.context,
                expected_cluster_id=args.expected_cluster_id,
                run_id=args.run_id,
                cpu_profile_path=args.cpu_profile,

tests/performance/replay.py:361

• next_burst resets to zero on every watch generation. After a 410/429 reconnect resumes later in the schedule, the first event re-marks every burst that already ended, producing duplicate and time-shifted post-burst drain samples. Persist the next burst index across source invocations (or initialize it from the resumed event offset).

        burst_ends = burst_end_offsets(self._profile)
        next_burst = 0

tests/performance/metrics.py:384

• If the sampler task exits with a psutil/tracemalloc error, awaiting it raises before managed tracing is released. Callers then also skip their subsequent watch-manager cleanup, so a sampling failure can leak both tracing state and benchmark tasks. Release the owned tracing state in a finally block.

        task.cancel()
        with suppress(asyncio.CancelledError):
            await task
        if self._uses_managed_tracing:
            self._release_tracemalloc()

[hellices]

Round 3 addressed in 00beec4. No blocking comments this round; three of the suppressed findings were real correctness bugs, two of them in the burst-drain machinery earlier rounds touched.

Fixed

• _ReplaySource reset its burst cursor on every watch generation, so a 410 mid-schedule made the first resumed event re-mark every burst that had already closed. Reproduced concretely: single-burst profile + 410 at event 50 produced two drain samples for one burst, the second a spurious 0.0. The cursor now lives on the source next to _next_event_index. RED test: test_replay_does_not_re_mark_bursts_after_a_watch_reconnect.
• ProcessSampler.stop() released managed tracemalloc after awaiting the sampler task, so a sampling error raised past the release and past the caller's watch-manager teardown. Now released from a finally. RED test: test_sampler_stop_releases_tracing_even_when_the_task_failed.
• The offline replay had the same store-vs-store digest hole the live path closed in 28dec94. check_rendered_rows moved to replay.py (avoiding an import cycle, since live.py already imports from it) and now runs on the offline path too, inside the app block where the table still exists.

Not taken, with reasons

• Live hard-budget evaluator (cli.py:378). Correct that the command only enforces dropped updates, digest parity, and UI-scenario outcomes, not the full budget table. That is a new feature rather than a defect in existing behaviour, and it needs the measurement caveat below settled first — a budget gate is meaningless while the numbers it gates are profiler-inflated. Tracked on Qualify korvid at large-cluster scale with reproducible performance and API-load benchmarks #186.
• Every qualification runs under cProfile (cli.py:490). This is correct and it is the most important open item; I raised the same caveat myself when publishing the results on Qualify korvid at large-cluster scale with reproducible performance and API-load benchmarks #186. With ~2.1B calls the profiler dominates absolute wall time, so the published p95/p99 are only valid as before/after comparisons under identical instrumentation, not as user-facing latency. The fix is a protocol change (an unprofiled budget run plus a profiled companion run at the same SHA and profile), not a patch to this PR. Tracked on Qualify korvid at large-cluster scale with reproducible performance and API-load benchmarks #186 together with the budget evaluator, since the two must land together to mean anything.
• Wiring the real MetricsPoller — already tracked on Qualify korvid at large-cluster scale with reproducible performance and API-load benchmarks #186 for the same reason: it adds load inside the measured window and would silently change what the published numbers mean.

Per the review-loop policy this is the first round with no blocking findings.

[hellices]

Rebased onto main (4e38fb8). #209 landed the same in-place-diff optimisation I had here, so I took the upstream implementation and dropped mine.

Specifically: my perf(ui) commit added ResourceTable._rendered_cells so the in-place diff stopped calling DataTable.get_row() once per surviving row. #209 landed _emitted on main, which does the same thing and adds a row memo with an identity fast-path and pruning on top — a strict superset. main also already carries an equivalent guard test (test_unchanged_rows_are_not_read_back_from_the_datatable), so my three tests were redundant too.

Resolution: took main's resource_table.py and test_table_diff_update.py wholesale, kept only the node-shell audit-race fix that commit also carried, and reworded it accordingly (48103f9).

Note for the issue #186 numbers: the live before/after I published measured _rendered_cells, and _emitted is the same optimisation, so the direction and the eliminated-call counts still hold — but the credit belongs to #209, and the memo it adds should improve on those figures again. I have not re-measured on the cluster after the rebase.

Full gate green after the rebase: 4305 passed, ruff/mypy/tach clean.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[hellices]

Status: ready to merge, waiting on GitHub Actions.

Everything on the branch is done — rebased onto main (ade1fd0), all 20 review threads resolved, working tree clean.

CI on the current head has passed everything that could get a runner: pre-commit, test (3.11), test (3.12), test (3.13), ty-experimental, security, dependency-review, analyze, CodeQL — all green. Only windows-test is outstanding, and it failed for infrastructure reasons, not code: The job was not acquired by Runner of type hosted even after multiple attempts. Earlier rounds in the same window failed with Failed to resolve action download info / Service Unavailable / A task was canceled, all at the Set up job step.

GitHub has had Actions in major_outage throughout (githubstatus.com). Re-running windows-test has left it queued for ~40 minutes without picking up a runner.

Local full gate on this exact commit: 4305 passed, 21 skipped, ruff / mypy / tach clean.

Not merging until windows-test actually goes green on a real runner — the outage is a reason to wait, not a reason to skip a required check. Needs a re-run of the failed job once Actions recovers.