From 5a226b6fbec1142776bce92c76c47174d6e3d775 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 00:50:35 +0900 Subject: [PATCH 01/38] docs: design large-cluster performance qualification Define deterministic replay, guarded 1,000-Pod AKS load, evidence-gated optimization, cleanup, and publication requirements for #186. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...luster-performance-qualification-design.md | 326 ++++++++++++++++++ 1 file changed, 326 insertions(+) create mode 100644 docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md diff --git a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md new file mode 100644 index 00000000..f65522ea --- /dev/null +++ b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md @@ -0,0 +1,326 @@ +# Large-Cluster Performance Qualification Design + +**Issue:** #186 + +**Status:** Approved for planning + +## Goal + +Establish a reproducible large-cluster benchmark for korvid, validate it against +real load in the dedicated test AKS cluster, and optimize only bottlenecks +demonstrated by the resulting traces. + +The work is complete when korvid has a published 1,000-Pod support envelope, +repeatable replay coverage at larger scales, and before/after evidence for every +runtime optimization made under this issue. + +## Decision + +Use a hybrid, measurement-first qualification program: + +1. deterministic replay exercises the real store, watch, app, and table paths at + 1,000, 10,000, and 50,000 objects; +2. a guarded live run creates 1,000 real Pods on the existing + `aks-korvid-contract-test` cluster; +3. profiling identifies the dominant bottleneck; +4. one bottleneck is optimized at a time with a correctness regression test and + a deterministic performance comparison; +5. the same live workload is rerun before the support envelope is published. + +This is preferred over a live-only benchmark, which is too variable and costly +for regression testing, and over replay-only testing, which cannot validate API +server, network, watch-recovery, or scheduling behavior. + +## Scope and sequencing + +#186 remains the only selected product issue until its qualification cycle is +finished. The work is split into reviewable deliverables rather than one large +change: + +1. **Benchmark foundation:** deterministic profiles, metrics, reports, and + guarded AKS workload lifecycle. +2. **Baseline qualification:** deterministic results plus the first live + 1,000-Pod run and profile. +3. **Evidence-gated optimization:** a focused TDD change for each material + bottleneck, beginning with the largest measured contributor. +4. **Requalification:** identical replay and AKS runs, final budgets, support + envelope, and documentation. + +The benchmark foundation may be merged independently. Runtime changes that +overlap PR #197, especially changes in `ui/app.py`, begin only after PR #197 is +merged and the issue worktree is synchronized with the resulting `main`. + +## Benchmark architecture + +### Workload profiles + +Profiles are versioned JSON documents with a schema version, seed, object count, +namespace count, initial-state distribution, churn rate, burst pattern, failure +injections, and duration. A resolved run manifest records the profile plus the +korvid SHA, Python, Textual, OS, CPU, memory, Kubernetes, cluster, and node-pool +versions. + +The initial profiles are: + +| Profile | Objects | Purpose | Gate | +|---|---:|---|---| +| `smoke-1k` | 1,000 | Fast deterministic correctness and report smoke | Normal CI, no wall-clock assertion | +| `steady-10k` | 10,000 | Store/render scaling and sustained churn | Manual or scheduled | +| `burst-50k` | 50,000 | Upper-envelope and burst-backlog measurement | Manual or scheduled | +| `live-aks-1k` | 1,000 Pods | Real API, network, LIST/WATCH, and UI qualification | Protected manual run | + +The deterministic generator emits stable names, namespaces, UIDs, resource +versions, and event order from the profile seed. Repeating a profile with the +same seed must produce the same object and event hashes. + +### Runtime path + +Replay and live runs exercise the same application path: + +```text +LIST/WATCH or replay source + -> WatchManager + -> ResourceStore + -> ResourcesUpdated coalescing + -> KorvidApp render + -> ResourceTable +``` + +The benchmark does not replace these components with a simplified table model. +A headless Textual pilot drives the real app and records cursor-response latency +while updates are active. + +Instrumentation is observational. Production defaults remain unchanged, and no +benchmark dependency is required at normal runtime. Cross-platform RSS and CPU +sampling uses `psutil` as a development dependency; benchmark reports also +include Python allocation samples so process growth can be separated from +Python heap growth. + +### Metrics + +Every run emits JSON and Markdown containing: + +- process start to interactive app; +- LIST completion to first fully populated table; +- event receipt to reflected table state, p50/p95/p99 and maximum; +- cursor-input acknowledgement, p50/p95/p99 and maximum; +- update backlog depth and time to drain after each burst; +- process CPU, RSS, peak RSS, and post-warm-up RSS slope; +- Python allocation growth by source location; +- logical LIST/WATCH/GET counts, decoded response bytes, watch events, + reconnects, `410 Gone` relists, throttles, and authorization failures; +- rendered, coalesced, and dropped update counts; +- final store/table digest, which must match the expected workload digest. + +Samples use monotonic clocks. Normal unit tests assert metric semantics and +operation counts, never machine-dependent wall-clock thresholds. + +### API-load accounting + +The benchmark records logical operations at the `KubeClient` boundary and event +payload bytes after decoding. It distinguishes initial LIST, long-lived WATCH, +documented reconnect/re-LIST, user-requested GET, and unexpected GET. + +A passive resource view must use one LIST followed by one WATCH for each active +`(kind, scope)` pair. Object count must not increase GET count. Reconnects and +re-LISTs are reported separately rather than hidden inside aggregate request +counts. + +## Live AKS qualification + +### Fixed target and capacity + +The live target is only: + +- resource group `rg-korvid-contract-test`; +- cluster `aks-korvid-contract-test`; +- tags `purpose=korvid-contract-testing` and + `production-use=prohibited`; +- Kubernetes context `aks-korvid-contract-test`. + +The cluster currently has a stopped `Standard_D2s_v5` system node and a +zero-node `Standard_D2s_v5` workload pool. Korea Central has 100 available +DSv5-family vCPUs and 116 regional vCPUs. + +Create or reuse a user pool named `perftest` with: + +- `Standard_D4s_v5`; +- five nodes during the run and zero while idle; +- `maxPods=250`; +- label `korvid.dev/pool=perftest`; +- taint `korvid.dev/performance=true:NoSchedule`. + +Five nodes provide room for 1,000 test Pods plus required DaemonSets without +running at the 250-Pod-per-node ceiling. They also provide 20 vCPUs and 80 GiB +of memory while remaining well inside the verified quota. + +### Workload + +The workload generator creates 20 labelled namespaces named +`korvid-performance--00` through `korvid-performance--19`, with 50 +Pods in each. Pods use `registry.k8s.io/pause:3.10`, select the `perftest` pool, +tolerate only the performance taint, and request 5 millicores and 16 MiB. Every +namespace and Pod has: + +- `app.kubernetes.io/managed-by=korvid-performance`; +- a unique `korvid.dev/performance-run` value; +- the workload profile and seed. + +The live sequence is: + +1. start the stopped cluster and scale `perftest` to five; +2. run the janitor for stale, labelled performance namespaces; +3. create all 1,000 Pods and require exactly 1,000 Running, Ready Pods before + measuring; +4. capture cold startup and initial LIST/render metrics; +5. run 30 minutes of metadata-only watch churn at 20 events per second; +6. inject three 30-second bursts at 100 events per second during that window; +7. drive filter, sort, namespace switch, split pane, describe, and cursor input; +8. collect final digests, profiles, API counts, and cluster diagnostics; +9. delete only the run-labelled namespaces and verify they are gone; +10. scale `perftest` to zero and stop the cluster in an independent cleanup + path that runs even after benchmark failure or timeout. + +Metadata-only updates create real watch traffic without restarting containers +or changing the workload's resource demand. The generator rate and observed API +throttling are both recorded; requested rate is never reported as achieved rate. + +### Guardrails + +Every mutating command fails closed unless the resource group, cluster name, +test-only tags, context, pool labels, namespace prefix, and run labels all +match. Cleanup lists the exact owned objects before deletion and refuses +unlabelled resources. + +The system pool is never targeted. Existing contract-test namespaces are not +shared with the performance workload. The stop-cluster action is independent of +the benchmark job so a timeout cannot leave compute running. + +## Failure profiles + +Failure behavior is deterministic first and live only when safe: + +- `410 Gone` forces re-LIST and validates that deleted objects do not remain; +- throttling validates bounded backoff and request accounting; +- partial RBAC denial validates one reported denial without namespace fan-out; +- unavailable metrics validates responsive resource navigation without the + metrics poller; +- slow API responses validate bounded input latency and visible backlog; +- slow log streams validate that resource watch/render progress is independent + of log consumption. + +The first live qualification covers normal LIST/WATCH plus real throttling if it +occurs naturally. Synthetic failure injection stays in replay until the normal +live path is stable, avoiding unnecessary API-server disruption. + +## Evidence-gated optimization + +The baseline is profiled before runtime code changes. A candidate is material +when it either: + +- misses a hard budget; +- contributes at least 25% of sampled CPU time during the affected phase; +- causes unbounded growth with object count, churn rate, or duration; or +- creates unexpected per-object API requests. + +The largest material contributor is addressed first. Each optimization must: + +1. add a correctness regression test that fails on the original behavior; +2. add or extend a deterministic benchmark comparison; +3. preserve cursor, viewport, filtering, sorting, split-pane, and recovery + behavior; +4. improve its target metric by at least 20% over the median of three + deterministic runs; +5. introduce no greater than 10% regression in another published performance + metric; +6. pass the full repository gate; +7. improve or preserve the same target in the repeated live run. + +Likely hot paths such as full-store sorting, full-row table diffing, render +message cadence, and hierarchy refresh are hypotheses only. They are not +changed unless the profile identifies them. + +If the first baseline meets every hard budget and no material contributor is +found, no speculative optimization is made. The measured support envelope is +still published. + +## Initial hard budgets + +These budgets define usable behavior for the live 1,000-Pod profile: + +| Metric | Budget | +|---|---:| +| Dropped updates | 0 | +| Final store/table digest mismatch | 0 | +| Unexpected per-object GETs for passive view | 0 | +| LIST completion to 1,000-row table | <= 2 seconds | +| Process start to interactive 1,000-row table | <= 10 seconds | +| Event-to-render p95 at 20 events/s | <= 250 ms | +| Cursor-input p95 during steady churn | <= 100 ms | +| Backlog drain after a 100 events/s burst | <= 3 seconds | +| Post-warm-up RSS slope over 30 minutes | <= 1 MiB/minute | +| Peak RSS at 1,000 Pods | <= 512 MiB | + +The final document records both observed values and budgets. A budget may be +changed only with measured rationale in review; baseline updates never silently +overwrite historical results. The 10,000- and 50,000-object replay profiles +define measured upper envelopes, not claims that the same live budgets apply. + +## Results and documentation + +Main contains: + +- workload schemas and profiles; +- benchmark command and tests; +- reproducibility methodology; +- compact baseline and optimized summaries; +- the supported scale envelope and known limits. + +Large raw JSON samples, process traces, allocation snapshots, and cluster +diagnostics are committed to a separate `benchmark-results` branch. Issue #186 +links the immutable artifact commit and summarizes the run. Raw artifacts are +not added to the product source history. + +Reports never imply validation beyond the exact profile, cluster, and commit +recorded in the run manifest. + +## Test strategy + +- Schema tests reject unknown versions, invalid rates, impossible namespace + distributions, and unsafe live targets. +- Generator tests prove deterministic hashes and exact event counts. +- Metric tests use a fake monotonic clock and fixed process samples. +- Replay tests exercise 1,000 objects without timing assertions and verify + final digest, zero drops, bounded API operations, and report shape. +- Textual tests drive cursor input during controlled churn and verify + acknowledgement accounting. +- AKS lifecycle tests prove guard failures and label-scoped cleanup with fakes. +- Protected live execution verifies the 1,000-Pod workload and records timing + rather than asserting cloud wall-clock behavior inside the normal PR suite. + +## Acceptance criteria + +- The same seed produces identical object and event hashes. +- Replay runs at 1,000, 10,000, and 50,000 objects emit complete JSON and + Markdown reports. +- The protected AKS run has 1,000 simultaneously visible Pods on five + performance nodes and completes the 30-minute churn profile. +- Passive viewing shows bounded LIST/WATCH behavior with no per-object GET + fan-out. +- Reports include latency distributions, CPU/RSS, memory slope, backlog, + reconnects, throttles, dropped updates, and final digests. +- Every material bottleneck is either optimized with before/after evidence or + documented with a reason it is not safe or valuable to change. +- The final live rerun satisfies the hard budgets or publishes the missed budget + as an explicit unsupported limit. +- All benchmark namespaces are removed, `perftest` is at zero nodes, and the + test cluster is stopped after each live run. + +## Non-goals + +- Maintaining a permanently running large AKS cluster. +- Claiming production-scale support beyond the tested profiles. +- Adding wall-clock assertions to normal unit tests. +- Optimizing suspected hot paths before profiling. +- Running destructive failure injection against the AKS control plane. +- Starting another product issue before #186 reaches its documented outcome. From a681186bfc2748041da36197b9ff77dfc171326f Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 01:01:42 +0900 Subject: [PATCH 02/38] test: add versioned scale workload profiles Define strict deterministic profile inputs for issue #186. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- pyproject.toml | 1 + tests/performance/__init__.py | 0 tests/performance/profile.py | 155 ++++++++++++++++++++++++++++++ tests/performance/test_profile.py | 55 +++++++++++ uv.lock | 30 ++++++ 5 files changed, 241 insertions(+) create mode 100644 tests/performance/__init__.py create mode 100644 tests/performance/profile.py create mode 100644 tests/performance/test_profile.py diff --git a/pyproject.toml b/pyproject.toml index cc424512..e3636a5d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -64,6 +64,7 @@ dev = [ "types-PyYAML", "azure-identity>=1.25.3", "types-regex>=2026.7.19.20260720", + "psutil>=6.1", ] [tool.ruff] diff --git a/tests/performance/__init__.py b/tests/performance/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/performance/profile.py b/tests/performance/profile.py new file mode 100644 index 00000000..520beb44 --- /dev/null +++ b/tests/performance/profile.py @@ -0,0 +1,155 @@ +from __future__ import annotations + +import json +from dataclasses import dataclass +from itertools import pairwise +from pathlib import Path +from typing import Any, Literal, cast + +FailureKind = Literal["gone", "throttled", "forbidden", "slow"] +_FAILURE_KINDS = frozenset({"gone", "throttled", "forbidden", "slow"}) +_PROFILE_KEYS = frozenset( + { + "schema_version", + "id", + "seed", + "object_count", + "namespace_count", + "steady_events_per_second", + "duration_seconds", + "bursts", + "failures", + } +) +_BURST_KEYS = frozenset({"start_second", "duration_seconds", "events_per_second"}) +_FAILURE_KEYS = frozenset({"kind", "at_event"}) + + +@dataclass(frozen=True) +class Burst: + start_second: int + duration_seconds: int + events_per_second: int + + +@dataclass(frozen=True) +class FailureInjection: + kind: FailureKind + at_event: int + + +@dataclass(frozen=True) +class WorkloadProfile: + schema_version: int + id: str + seed: int + object_count: int + namespace_count: int + steady_events_per_second: int + duration_seconds: int + bursts: tuple[Burst, ...] + failures: tuple[FailureInjection, ...] + + +def _mapping(value: Any, label: str) -> dict[str, Any]: + if not isinstance(value, dict): + raise ValueError(f"{label} must be a mapping") + return value + + +def _unknown(raw: dict[str, Any], allowed: frozenset[str], label: str) -> None: + unknown = sorted(set(raw) - allowed) + if unknown: + raise ValueError(f"{label} has unknown keys: {', '.join(unknown)}") + + +def _int(raw: dict[str, Any], key: str, *, positive: bool = False) -> int: + value = raw.get(key) + if isinstance(value, bool) or not isinstance(value, int): + raise ValueError(f"{key} must be an integer") + if positive and value < 1: + raise ValueError(f"{key} must be a positive integer") + return value + + +def _bursts(raw: Any, duration: int) -> tuple[Burst, ...]: + if not isinstance(raw, list): + raise ValueError("bursts must be a list") + result: list[Burst] = [] + for index, value in enumerate(raw, 1): + item = _mapping(value, f"burst {index}") + _unknown(item, _BURST_KEYS, f"burst {index}") + burst = Burst( + start_second=_int(item, "start_second"), + duration_seconds=_int(item, "duration_seconds", positive=True), + events_per_second=_int(item, "events_per_second", positive=True), + ) + if burst.start_second < 0 or burst.start_second + burst.duration_seconds > duration: + raise ValueError(f"burst {index} falls outside duration_seconds") + result.append(burst) + ordered = sorted(result, key=lambda burst: burst.start_second) + for previous, current in pairwise(ordered): + if previous.start_second + previous.duration_seconds > current.start_second: + raise ValueError("bursts must not overlap") + return tuple(ordered) + + +def _failures(raw: Any) -> tuple[FailureInjection, ...]: + if not isinstance(raw, list): + raise ValueError("failures must be a list") + result: list[FailureInjection] = [] + for index, value in enumerate(raw, 1): + item = _mapping(value, f"failure {index}") + _unknown(item, _FAILURE_KEYS, f"failure {index}") + kind = item.get("kind") + if kind not in _FAILURE_KINDS: + raise ValueError(f"failure {index} kind must be one of {sorted(_FAILURE_KINDS)}") + result.append( + FailureInjection( + kind=cast(FailureKind, kind), + at_event=_int(item, "at_event", positive=True), + ) + ) + return tuple(sorted(result, key=lambda failure: failure.at_event)) + + +def load_profile(path: Path) -> WorkloadProfile: + raw = _mapping(json.loads(path.read_text()), path.name) + _unknown(raw, _PROFILE_KEYS, path.name) + schema_version = _int(raw, "schema_version", positive=True) + if schema_version != 1: + raise ValueError("schema_version must be 1") + profile_id = raw.get("id") + if not isinstance(profile_id, str) or not profile_id.strip(): + raise ValueError("id must be a non-empty string") + object_count = _int(raw, "object_count", positive=True) + namespace_count = _int(raw, "namespace_count", positive=True) + if object_count % namespace_count: + raise ValueError("object_count must be divisible by namespace_count") + duration = _int(raw, "duration_seconds", positive=True) + steady = _int(raw, "steady_events_per_second") + if steady < 0: + raise ValueError("steady_events_per_second must be non-negative") + profile = WorkloadProfile( + schema_version=schema_version, + id=profile_id, + seed=_int(raw, "seed"), + object_count=object_count, + namespace_count=namespace_count, + steady_events_per_second=steady, + duration_seconds=duration, + bursts=_bursts(raw.get("bursts"), duration), + failures=_failures(raw.get("failures")), + ) + total = planned_event_count(profile) + if any(failure.at_event > total for failure in profile.failures): + raise ValueError("failure at_event exceeds planned event count") + return profile + + +def planned_event_count(profile: WorkloadProfile) -> int: + total = profile.duration_seconds * profile.steady_events_per_second + for burst in profile.bursts: + total -= round(burst.duration_seconds * profile.steady_events_per_second) + total += round(burst.duration_seconds * burst.events_per_second) + return total diff --git a/tests/performance/test_profile.py b/tests/performance/test_profile.py new file mode 100644 index 00000000..b24fc3d8 --- /dev/null +++ b/tests/performance/test_profile.py @@ -0,0 +1,55 @@ +import json +from pathlib import Path + +import pytest + +from tests.performance.profile import load_profile, planned_event_count + + +def _write(tmp_path: Path, **overrides: object) -> Path: + payload: dict[str, object] = { + "schema_version": 1, + "id": "smoke", + "seed": 186, + "object_count": 1000, + "namespace_count": 20, + "steady_events_per_second": 20, + "duration_seconds": 5, + "bursts": [{"start_second": 2, "duration_seconds": 1, "events_per_second": 100}], + "failures": [{"kind": "gone", "at_event": 75}], + } + payload.update(overrides) + path = tmp_path / "profile.json" + path.write_text(json.dumps(payload)) + return path + + +def test_load_profile_is_strict_and_typed(tmp_path: Path) -> None: + profile = load_profile(_write(tmp_path)) + assert profile.id == "smoke" + assert profile.object_count == 1000 + assert profile.bursts[0].events_per_second == 100 + assert profile.failures[0].kind == "gone" + assert planned_event_count(profile) == 180 + + +def test_profile_rejects_unknown_keys(tmp_path: Path) -> None: + with pytest.raises(ValueError, match=r"unknown keys.*extra"): + load_profile(_write(tmp_path, extra=True)) + + +@pytest.mark.parametrize( + ("field", "value", "message"), + [ + ("schema_version", 2, "schema_version must be 1"), + ("object_count", 0, "object_count must be a positive integer"), + ("namespace_count", 21, "object_count must be divisible by namespace_count"), + ("steady_events_per_second", -1, "steady_events_per_second"), + ("duration_seconds", 0, "duration_seconds must be a positive integer"), + ], +) +def test_profile_rejects_invalid_values( + tmp_path: Path, field: str, value: object, message: str +) -> None: + with pytest.raises(ValueError, match=message): + load_profile(_write(tmp_path, **{field: value})) diff --git a/uv.lock b/uv.lock index b9e9e0b6..e2c63642 100644 --- a/uv.lock +++ b/uv.lock @@ -1017,6 +1017,7 @@ dev = [ { name = "deptry" }, { name = "mypy" }, { name = "pre-commit" }, + { name = "psutil" }, { name = "pytest" }, { name = "pytest-asyncio" }, { name = "pytest-cov" }, @@ -1052,6 +1053,7 @@ dev = [ { name = "deptry", specifier = ">=0.20" }, { name = "mypy", specifier = ">=1.14" }, { name = "pre-commit", specifier = ">=4" }, + { name = "psutil", specifier = ">=6.1" }, { name = "pytest", specifier = ">=8" }, { name = "pytest-asyncio", specifier = ">=0.24" }, { name = "pytest-cov", specifier = ">=5" }, @@ -1636,6 +1638,34 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3a/ed/1cdcab6ba3d6ab7feca11fc14f0eeea80755bb53ef4e892079f31b10a25f/propcache-0.5.2-py3-none-any.whl", hash = "sha256:be1ddfcbb376e3de5d2e2db1d58d6d67463e6b4f9f040c000de8e300295465fe", size = 14036, upload-time = "2026-05-08T21:02:10.673Z" }, ] +[[package]] +name = "psutil" +version = "7.2.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372", size = 493740, upload-time = "2026-01-28T18:14:54.428Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/08/510cbdb69c25a96f4ae523f733cdc963ae654904e8db864c07585ef99875/psutil-7.2.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:2edccc433cbfa046b980b0df0171cd25bcaeb3a68fe9022db0979e7aa74a826b", size = 130595, upload-time = "2026-01-28T18:14:57.293Z" }, + { url = "https://files.pythonhosted.org/packages/d6/f5/97baea3fe7a5a9af7436301f85490905379b1c6f2dd51fe3ecf24b4c5fbf/psutil-7.2.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e78c8603dcd9a04c7364f1a3e670cea95d51ee865e4efb3556a3a63adef958ea", size = 131082, upload-time = "2026-01-28T18:14:59.732Z" }, + { url = "https://files.pythonhosted.org/packages/37/d6/246513fbf9fa174af531f28412297dd05241d97a75911ac8febefa1a53c6/psutil-7.2.2-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1a571f2330c966c62aeda00dd24620425d4b0cc86881c89861fbc04549e5dc63", size = 181476, upload-time = "2026-01-28T18:15:01.884Z" }, + { url = "https://files.pythonhosted.org/packages/b8/b5/9182c9af3836cca61696dabe4fd1304e17bc56cb62f17439e1154f225dd3/psutil-7.2.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:917e891983ca3c1887b4ef36447b1e0873e70c933afc831c6b6da078ba474312", size = 184062, upload-time = "2026-01-28T18:15:04.436Z" }, + { url = "https://files.pythonhosted.org/packages/16/ba/0756dca669f5a9300d0cbcbfae9a4c30e446dfc7440ffe43ded5724bfd93/psutil-7.2.2-cp313-cp313t-win_amd64.whl", hash = "sha256:ab486563df44c17f5173621c7b198955bd6b613fb87c71c161f827d3fb149a9b", size = 139893, upload-time = "2026-01-28T18:15:06.378Z" }, + { url = "https://files.pythonhosted.org/packages/1c/61/8fa0e26f33623b49949346de05ec1ddaad02ed8ba64af45f40a147dbfa97/psutil-7.2.2-cp313-cp313t-win_arm64.whl", hash = "sha256:ae0aefdd8796a7737eccea863f80f81e468a1e4cf14d926bd9b6f5f2d5f90ca9", size = 135589, upload-time = "2026-01-28T18:15:08.03Z" }, + { url = "https://files.pythonhosted.org/packages/81/69/ef179ab5ca24f32acc1dac0c247fd6a13b501fd5534dbae0e05a1c48b66d/psutil-7.2.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:eed63d3b4d62449571547b60578c5b2c4bcccc5387148db46e0c2313dad0ee00", size = 130664, upload-time = "2026-01-28T18:15:09.469Z" }, + { url = "https://files.pythonhosted.org/packages/7b/64/665248b557a236d3fa9efc378d60d95ef56dd0a490c2cd37dafc7660d4a9/psutil-7.2.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7b6d09433a10592ce39b13d7be5a54fbac1d1228ed29abc880fb23df7cb694c9", size = 131087, upload-time = "2026-01-28T18:15:11.724Z" }, + { url = "https://files.pythonhosted.org/packages/d5/2e/e6782744700d6759ebce3043dcfa661fb61e2fb752b91cdeae9af12c2178/psutil-7.2.2-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fa4ecf83bcdf6e6c8f4449aff98eefb5d0604bf88cb883d7da3d8d2d909546a", size = 182383, upload-time = "2026-01-28T18:15:13.445Z" }, + { url = "https://files.pythonhosted.org/packages/57/49/0a41cefd10cb7505cdc04dab3eacf24c0c2cb158a998b8c7b1d27ee2c1f5/psutil-7.2.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e452c464a02e7dc7822a05d25db4cde564444a67e58539a00f929c51eddda0cf", size = 185210, upload-time = "2026-01-28T18:15:16.002Z" }, + { url = "https://files.pythonhosted.org/packages/dd/2c/ff9bfb544f283ba5f83ba725a3c5fec6d6b10b8f27ac1dc641c473dc390d/psutil-7.2.2-cp314-cp314t-win_amd64.whl", hash = "sha256:c7663d4e37f13e884d13994247449e9f8f574bc4655d509c3b95e9ec9e2b9dc1", size = 141228, upload-time = "2026-01-28T18:15:18.385Z" }, + { url = "https://files.pythonhosted.org/packages/f2/fc/f8d9c31db14fcec13748d373e668bc3bed94d9077dbc17fb0eebc073233c/psutil-7.2.2-cp314-cp314t-win_arm64.whl", hash = "sha256:11fe5a4f613759764e79c65cf11ebdf26e33d6dd34336f8a337aa2996d71c841", size = 136284, upload-time = "2026-01-28T18:15:19.912Z" }, + { url = "https://files.pythonhosted.org/packages/e7/36/5ee6e05c9bd427237b11b3937ad82bb8ad2752d72c6969314590dd0c2f6e/psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486", size = 129090, upload-time = "2026-01-28T18:15:22.168Z" }, + { url = "https://files.pythonhosted.org/packages/80/c4/f5af4c1ca8c1eeb2e92ccca14ce8effdeec651d5ab6053c589b074eda6e1/psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979", size = 129859, upload-time = "2026-01-28T18:15:23.795Z" }, + { url = "https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9", size = 155560, upload-time = "2026-01-28T18:15:25.976Z" }, + { url = "https://files.pythonhosted.org/packages/63/65/37648c0c158dc222aba51c089eb3bdfa238e621674dc42d48706e639204f/psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e", size = 156997, upload-time = "2026-01-28T18:15:27.794Z" }, + { url = "https://files.pythonhosted.org/packages/8e/13/125093eadae863ce03c6ffdbae9929430d116a246ef69866dad94da3bfbc/psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8", size = 148972, upload-time = "2026-01-28T18:15:29.342Z" }, + { url = "https://files.pythonhosted.org/packages/04/78/0acd37ca84ce3ddffaa92ef0f571e073faa6d8ff1f0559ab1272188ea2be/psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc", size = 148266, upload-time = "2026-01-28T18:15:31.597Z" }, + { url = "https://files.pythonhosted.org/packages/b4/90/e2159492b5426be0c1fef7acba807a03511f97c5f86b3caeda6ad92351a7/psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988", size = 137737, upload-time = "2026-01-28T18:15:33.849Z" }, + { url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617, upload-time = "2026-01-28T18:15:36.514Z" }, +] + [[package]] name = "pycparser" version = "3.0" From 0a5285cb9daa137de8e1670f200e97cd03a1ecc1 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 01:13:55 +0900 Subject: [PATCH 03/38] docs: clarify schema v1 initial state Schema v1 fixes the initial state to all Running/Ready Pods; future distributions need a new field.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...large-cluster-performance-qualification-design.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md index f65522ea..d1b6d2f0 100644 --- a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md +++ b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md @@ -55,10 +55,12 @@ merged and the issue worktree is synchronized with the resulting `main`. ### Workload profiles Profiles are versioned JSON documents with a schema version, seed, object count, -namespace count, initial-state distribution, churn rate, burst pattern, failure -injections, and duration. A resolved run manifest records the profile plus the -korvid SHA, Python, Textual, OS, CPU, memory, Kubernetes, cluster, and node-pool -versions. +namespace count, churn rate, burst pattern, failure injections, and duration. +Schema v1, as reviewed in `tests/performance/profile.py`, fixes the initial +state to all Pods already Running and Ready for the chosen object count; varied +initial-state distributions require a future schema version with an explicit +field. A resolved run manifest records the profile plus the korvid SHA, Python, +Textual, OS, CPU, memory, Kubernetes, cluster, and node-pool versions. The initial profiles are: @@ -171,7 +173,7 @@ The live sequence is: 1. start the stopped cluster and scale `perftest` to five; 2. run the janitor for stale, labelled performance namespaces; 3. create all 1,000 Pods and require exactly 1,000 Running, Ready Pods before - measuring; + measuring, matching schema v1's fixed initial state; 4. capture cold startup and initial LIST/render metrics; 5. run 30 minutes of metadata-only watch churn at 20 events per second; 6. inject three 30-second bursts at 100 events per second during that window; From 72c9805e021dcf4bb14992a936ca2d5465a61328 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 01:18:35 +0900 Subject: [PATCH 04/38] test: generate deterministic scale replay traffic Make object and event streams reproducible from the issue #186 profile seed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/test_workload.py | 49 +++++++++++++++ tests/performance/workload.py | 98 ++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 tests/performance/test_workload.py create mode 100644 tests/performance/workload.py diff --git a/tests/performance/test_workload.py b/tests/performance/test_workload.py new file mode 100644 index 00000000..d22789a4 --- /dev/null +++ b/tests/performance/test_workload.py @@ -0,0 +1,49 @@ +from dataclasses import replace +from itertools import pairwise + +from tests.performance.profile import WorkloadProfile +from tests.performance.workload import ( + apply_events, + event_digest, + initial_pods, + scheduled_events, + summary_digest, +) + + +def _profile(seed: int = 186) -> WorkloadProfile: + return WorkloadProfile( + schema_version=1, + id="test", + seed=seed, + object_count=100, + namespace_count=10, + steady_events_per_second=10, + duration_seconds=2, + bursts=(), + failures=(), + ) + + +def test_same_seed_produces_identical_hashes() -> None: + first = _profile() + second = replace(first) + assert summary_digest(initial_pods(first)) == summary_digest(initial_pods(second)) + assert event_digest(scheduled_events(first)) == event_digest(scheduled_events(second)) + + +def test_different_seed_changes_event_hash_not_initial_hash() -> None: + first = _profile(186) + second = _profile(187) + assert summary_digest(initial_pods(first)) == summary_digest(initial_pods(second)) + assert event_digest(scheduled_events(first)) != event_digest(scheduled_events(second)) + + +def test_events_are_stably_scheduled_and_change_final_digest() -> None: + profile = _profile() + initial = initial_pods(profile) + events = scheduled_events(profile) + assert len(events) == 20 + assert [event.sequence for event in events] == list(range(1, 21)) + assert all(left.offset_seconds <= right.offset_seconds for left, right in pairwise(events)) + assert summary_digest(apply_events(initial, events)) != summary_digest(initial) diff --git a/tests/performance/workload.py b/tests/performance/workload.py new file mode 100644 index 00000000..8df08306 --- /dev/null +++ b/tests/performance/workload.py @@ -0,0 +1,98 @@ +from __future__ import annotations + +import hashlib +import json +import random +from collections.abc import Iterable +from dataclasses import asdict, dataclass, replace + +from korvid.k8s.models import PodSummary +from tests.performance.profile import WorkloadProfile, planned_event_count + + +@dataclass(frozen=True) +class ScheduledEvent: + sequence: int + offset_seconds: float + event_type: str + summary: PodSummary + + +def initial_pods(profile: WorkloadProfile) -> tuple[PodSummary, ...]: + return tuple( + PodSummary( + name=f"pod-{index:06d}", + namespace=f"bench-{index % profile.namespace_count:04d}", + phase="Running", + ready="1/1", + restarts=0, + node=f"node-{index % 5:02d}", + ) + for index in range(profile.object_count) + ) + + +def _rate_at(profile: WorkloadProfile, second: float) -> int: + for burst in profile.bursts: + if burst.start_second <= second < burst.start_second + burst.duration_seconds: + return burst.events_per_second + return profile.steady_events_per_second + + +def scheduled_events(profile: WorkloadProfile) -> tuple[ScheduledEvent, ...]: + rng = random.Random(profile.seed) + current = list(initial_pods(profile)) + result: list[ScheduledEvent] = [] + sequence = 0 + for second in range(profile.duration_seconds): + rate = _rate_at(profile, float(second)) + if rate <= 0: + continue + for tick in range(rate): + sequence += 1 + index = rng.randrange(profile.object_count) + old = current[index] + updated = replace( + old, + phase="Pending" if old.phase == "Running" else "Running", + ready="0/1" if old.ready == "1/1" else "1/1", + restarts=old.restarts + 1, + ) + current[index] = updated + result.append( + ScheduledEvent( + sequence=sequence, + offset_seconds=second + tick / rate, + event_type="MODIFIED", + summary=updated, + ) + ) + assert len(result) == planned_event_count(profile) + return tuple(result) + + +def _hash(payload: object) -> str: + encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str).encode() + return hashlib.sha256(encoded).hexdigest() + + +def summary_digest(summaries: Iterable[PodSummary]) -> str: + ordered = sorted(summaries, key=lambda pod: (pod.namespace, pod.name)) + return _hash([asdict(pod) for pod in ordered]) + + +def event_digest(events: Iterable[ScheduledEvent]) -> str: + return _hash([asdict(event) for event in events]) + + +def apply_events( + initial: Iterable[PodSummary], events: Iterable[ScheduledEvent] +) -> tuple[PodSummary, ...]: + current = {f"{pod.namespace}/{pod.name}": pod for pod in initial} + for event in events: + key = f"{event.summary.namespace}/{event.summary.name}" + if event.event_type == "DELETED": + current.pop(key, None) + else: + current[key] = event.summary + return tuple(sorted(current.values(), key=lambda pod: (pod.namespace, pod.name))) From 7964c53a5aba7a328fe59a6ccdfb066d7c367091 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 01:27:57 +0900 Subject: [PATCH 05/38] test: expose optional Kubernetes read telemetry Measure logical API load for issue #186 without changing the unobserved runtime path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/korvid/k8s/client.py | 193 +++++++++++++++++++++----- src/korvid/k8s/telemetry.py | 19 +++ tests/k8s/test_client.py | 262 ++++++++++++++++++++++++++++++++++++ 3 files changed, 442 insertions(+), 32 deletions(-) create mode 100644 src/korvid/k8s/telemetry.py diff --git a/src/korvid/k8s/client.py b/src/korvid/k8s/client.py index 085bf38d..b4fc9461 100644 --- a/src/korvid/k8s/client.py +++ b/src/korvid/k8s/client.py @@ -40,6 +40,7 @@ from korvid.k8s.metrics import PodMetrics, parse_pod_metrics_list from korvid.k8s.models import GenericSummary, PodSummary, summary_for from korvid.k8s.reads import ReadOps +from korvid.k8s.telemetry import ReadOperation, ReadTelemetry, ReadTelemetryEvent from korvid.k8s.writes import WriteOps logger = logging.getLogger(__name__) @@ -149,7 +150,10 @@ class KubeClient(ReadOps, WriteOps): """Thin wrapper over kubernetes_asyncio; returns typed summaries.""" def __init__( - self, custom_columns: Mapping[str, tuple[CustomColumn, ...]] | None = None + self, + custom_columns: Mapping[str, tuple[CustomColumn, ...]] | None = None, + *, + read_telemetry: ReadTelemetry | None = None, ) -> None: self._api: k8s_client.ApiClient | None = None self._core_v1: k8s_client.CoreV1Api | None = None @@ -166,6 +170,50 @@ def __init__( self._pod_resize_supported: bool | None = None #: cloud provider detection result; None until the first lookup. self._provider_info: ProviderInfo | None = None + self._read_telemetry = read_telemetry + + @staticmethod + def _namespaces_path() -> str: + return "/api/v1/namespaces" + + @staticmethod + def _pods_path(namespace: str | None) -> str: + if namespace is None: + return "/api/v1/pods" + return f"/api/v1/namespaces/{_path_segment(namespace)}/pods" + + def _observe_read( + self, + operation: ReadOperation, + path: str, + *, + payload: object | None = None, + object_count: int = 0, + status: int | None = None, + ) -> None: + if self._read_telemetry is None: + return + decoded_bytes = 0 + if payload is not None: + decoded_bytes = len( + json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str).encode() + ) + self._read_telemetry( + ReadTelemetryEvent( + operation=operation, + path=path, + decoded_bytes=decoded_bytes, + object_count=object_count, + status=status, + ) + ) + + def _observe_read_error( + self, + path: str, + exc: ApiStatusError | k8s_client.exceptions.ApiException, + ) -> None: + self._observe_read("error", path, status=int(getattr(exc, "status", 0) or 0)) def _pod_summary(self, manifest: dict[str, Any]) -> PodSummary: """PodSummary + configured custom column values (issue #45).""" @@ -260,11 +308,18 @@ async def switch_context(self, context: str | None) -> None: async def list_namespaces(self) -> list[str]: if self._core_v1 is None: raise RuntimeError("connect() first") + path = self._namespaces_path() try: resp = await self._core_v1.list_namespace(_preload_content=False) data = await _to_dict(resp) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise except k8s_client.exceptions.ApiException as exc: + self._observe_read_error(path, exc) raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc + items = data.get("items", []) + self._observe_read("list", path, payload=data, object_count=len(items)) return [item["metadata"]["name"] for item in data.get("items", [])] async def detect_cloud_provider(self) -> ProviderInfo: @@ -343,11 +398,18 @@ async def _session() -> AsyncIterator[Any]: async def list_pods(self, namespace: str) -> list[PodSummary]: if self._core_v1 is None: raise RuntimeError("connect() first") + path = self._pods_path(namespace) try: resp = await self._core_v1.list_namespaced_pod(namespace, _preload_content=False) data = await _to_dict(resp) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise except k8s_client.exceptions.ApiException as exc: + self._observe_read_error(path, exc) raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc + items = data.get("items", []) + self._observe_read("list", path, payload=data, object_count=len(items)) return [self._pod_summary(item) for item in data.get("items", [])] async def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSummary]]: @@ -363,6 +425,7 @@ async def _watch_pods_namespaced(self, namespace: str) -> AsyncIterator[tuple[st """Per-namespace pod watch via CoreV1Api (LIST then stream).""" if self._core_v1 is None: raise RuntimeError("connect() first") + path = self._pods_path(namespace) # LIST first: yield pre-existing pods as ADDED and anchor the watch at # the snapshot's resourceVersion so no events are missed between LIST @@ -372,11 +435,17 @@ async def _watch_pods_namespaced(self, namespace: str) -> AsyncIterator[tuple[st try: resp = await self._core_v1.list_namespaced_pod(namespace, _preload_content=False) data = await _to_dict(resp) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise except k8s_client.exceptions.ApiException as exc: + self._observe_read_error(path, exc) raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc + items = data.get("items", []) + self._observe_read("list", path, payload=data, object_count=len(items)) resource_version: str | None = (data.get("metadata") or {}).get("resourceVersion") - for item in data.get("items", []): + for item in items: yield ("ADDED", self._pod_summary(item)) watch_kwargs: dict[str, Any] = {} @@ -384,16 +453,23 @@ async def _watch_pods_namespaced(self, namespace: str) -> AsyncIterator[tuple[st watch_kwargs["resource_version"] = resource_version w = k8s_watch.Watch() + self._observe_read("watch_open", path) try: async with w.stream( self._core_v1.list_namespaced_pod, namespace, **watch_kwargs ) as stream: async for event in stream: + raw_object = event["raw_object"] + self._observe_read("watch_event", path, payload=raw_object, object_count=1) yield ( str(event["type"]), - self._pod_summary(event["raw_object"]), + self._pod_summary(raw_object), ) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise except k8s_client.exceptions.ApiException as exc: + self._observe_read_error(path, exc) raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc async def _watch_pods_cluster(self) -> AsyncIterator[tuple[str, PodSummary]]: @@ -401,11 +477,17 @@ async def _watch_pods_cluster(self) -> AsyncIterator[tuple[str, PodSummary]]: if self._api is None: raise RuntimeError("connect() first") - path = "/api/v1/pods" - data = await self._request_json(path) + path = self._pods_path(None) + try: + data = await self._request_json(path) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise + items = data.get("items", []) + self._observe_read("list", path, payload=data, object_count=len(items)) resource_version: str | None = (data.get("metadata") or {}).get("resourceVersion") - for item in data.get("items", []): + for item in items: yield ("ADDED", self._pod_summary(item)) watch_kwargs: dict[str, Any] = {} @@ -414,14 +496,21 @@ async def _watch_pods_cluster(self) -> AsyncIterator[tuple[str, PodSummary]]: watch_func = self._make_raw_watch_callable(path) w = k8s_watch.Watch() + self._observe_read("watch_open", path) try: async with w.stream(watch_func, **watch_kwargs) as stream: async for event in stream: + raw_object = event["raw_object"] + self._observe_read("watch_event", path, payload=raw_object, object_count=1) yield ( str(event["type"]), - self._pod_summary(event["raw_object"]), + self._pod_summary(raw_object), ) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise except k8s_client.exceptions.ApiException as exc: + self._observe_read_error(path, exc) raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc def _list_path(self, meta: ResourceMeta, namespace: str | None) -> str: @@ -431,6 +520,39 @@ def _list_path(self, meta: ResourceMeta, namespace: str | None) -> str: return f"{meta.api_base}/namespaces/{_path_segment(namespace)}/{meta.plural}" return f"{meta.api_base}/{meta.plural}" + async def _initial_object_snapshot( + self, meta: ResourceMeta, namespace: str | None + ) -> tuple[str, dict[str, Any], dict[str, GenericSummary]]: + list_path = self._list_path(meta, namespace) + try: + data = await self._request_json(list_path) + except ApiStatusError as exc: + self._observe_read_error(list_path, exc) + raise + + items = data.get("items", []) + self._observe_read("list", list_path, payload=data, object_count=len(items)) + known: dict[str, GenericSummary] = {} + for item in items: + summary = self._object_summary(meta, item) + known[f"{summary.namespace}/{summary.name}"] = summary + return list_path, data, known + + def _watch_objects_requires_poll_fallback( + self, + list_path: str, + exc: ApiStatusError | k8s_client.exceptions.ApiException, + ) -> bool: + status = int(getattr(exc, "status", 0) or 0) + self._observe_read_error(list_path, exc) + if status == 405: + return True + raise ApiStatusError( + status, + str(getattr(exc, "reason", "") or ""), + str(getattr(exc, "body", "") or ""), + ) from exc + async def watch_objects( self, meta: ResourceMeta, namespace: str | None ) -> AsyncIterator[tuple[str, GenericSummary]]: @@ -450,14 +572,10 @@ async def watch_objects( raise RuntimeError("connect() first") # LIST phase -------------------------------------------------------- - list_path = self._list_path(meta, namespace) - data = await self._request_json(list_path) - + list_path, data, known = await self._initial_object_snapshot(meta, namespace) resource_version: str | None = (data.get("metadata") or {}).get("resourceVersion") - known: dict[str, GenericSummary] = {} for item in data.get("items", []): summary = self._object_summary(meta, item) - known[f"{summary.namespace}/{summary.name}"] = summary yield ("ADDED", summary) if not meta.watchable: @@ -473,33 +591,28 @@ async def watch_objects( watch_func = self._make_raw_watch_callable(list_path) w = k8s_watch.Watch() + self._observe_read("watch_open", list_path) try: async with w.stream(watch_func, **watch_kwargs) as stream: async for event in stream: + raw_object = event["raw_object"] + self._observe_read("watch_event", list_path, payload=raw_object, object_count=1) yield ( str(event["type"]), - self._object_summary(meta, event["raw_object"]), + self._object_summary(meta, raw_object), ) except (k8s_client.exceptions.ApiException, ApiStatusError) as exc: # The raw-watch adapter surfaces HTTP errors as ApiStatusError # (via _raise_for_status); the kubernetes client's own paths # raise ApiException - both carry .status/.reason, and the 405 # fallback must catch both. - status = int(getattr(exc, "status", 0) or 0) - if status != 405: - # Preserve .body: same-status disambiguation (PDB denial vs - # APF throttling) depends on it; ApiException carries one too. - raise ApiStatusError( - status, - str(getattr(exc, "reason", "") or ""), - str(getattr(exc, "body", "") or ""), - ) from exc - # Discovery advertised watch but the server refuses it: as - # deterministic as it gets - poll instead of letting the - # manager burn retries clearing and re-seeding the store. - logger.info("%s rejects watch (405); falling back to LIST polling", meta.plural) - async for event in self._poll_objects(meta, list_path, known): - yield event + if self._watch_objects_requires_poll_fallback(list_path, exc): + # Discovery advertised watch but the server refuses it: as + # deterministic as it gets - poll instead of letting the + # manager burn retries clearing and re-seeding the store. + logger.info("%s rejects watch (405); falling back to LIST polling", meta.plural) + async for event in self._poll_objects(meta, list_path, known): + yield event async def _poll_objects( self, meta: ResourceMeta, list_path: str, known: dict[str, GenericSummary] @@ -513,9 +626,15 @@ async def _poll_objects( """ while True: await asyncio.sleep(LIST_POLL_INTERVAL) - data = await self._request_json(list_path) + try: + data = await self._request_json(list_path) + except ApiStatusError as exc: + self._observe_read_error(list_path, exc) + raise + items = data.get("items", []) + self._observe_read("list", list_path, payload=data, object_count=len(items)) current: dict[str, GenericSummary] = {} - for item in data.get("items", []): + for item in items: summary = self._object_summary(meta, item) current[f"{summary.namespace}/{summary.name}"] = summary yield ("ADDED", summary) @@ -532,14 +651,24 @@ async def list_objects(self, meta: ResourceMeta, namespace: str | None) -> list[ """ if self._api is None: raise RuntimeError("connect() first") - data = await self._request_json(self._list_path(meta, namespace)) + path = self._list_path(meta, namespace) + try: + data = await self._request_json(path) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise + items = data.get("items", []) + self._observe_read("list", path, payload=data, object_count=len(items)) return [self._object_summary(meta, item) for item in data.get("items", [])] async def get_object( self, meta: ResourceMeta, namespace: str | None, name: str ) -> dict[str, Any]: """Fetch the raw manifest for a single object. ApiException → ApiStatusError.""" - return await self._request_json(self._object_path(meta, namespace, name)) + path = self._object_path(meta, namespace, name) + result = await self._request_json(path) + self._observe_read("get", path, payload=result, object_count=1) + return result # Helm release browsing (issue #28) ---------------------------------- # Releases are Secrets of type helm.sh/release.v1; the synthetic kinds diff --git a/src/korvid/k8s/telemetry.py b/src/korvid/k8s/telemetry.py new file mode 100644 index 00000000..71a52426 --- /dev/null +++ b/src/korvid/k8s/telemetry.py @@ -0,0 +1,19 @@ +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass +from typing import Literal + +ReadOperation = Literal["list", "watch_open", "watch_event", "get", "error"] + + +@dataclass(frozen=True) +class ReadTelemetryEvent: + operation: ReadOperation + path: str + decoded_bytes: int = 0 + object_count: int = 0 + status: int | None = None + + +ReadTelemetry = Callable[[ReadTelemetryEvent], None] diff --git a/tests/k8s/test_client.py b/tests/k8s/test_client.py index d32ff4d1..dfd9885f 100644 --- a/tests/k8s/test_client.py +++ b/tests/k8s/test_client.py @@ -12,6 +12,7 @@ from korvid.k8s.discovery import ResourceMeta from korvid.k8s.errors import ApiStatusError from korvid.k8s.models import ReplicaSetSummary +from korvid.k8s.telemetry import ReadTelemetryEvent def _pod(name: str, ns: str = "default") -> dict[str, Any]: @@ -85,6 +86,25 @@ def stream(self, func: Any, *args: Any, **kwargs: Any) -> _FakeWatchStream: return _FakeWatchStream(self._events, self.captured_kwargs, self._raise_at, self._raise_exc) +async def test_list_namespaces_emits_list_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespace.return_value = { + "items": [{"metadata": {"name": "default"}}, {"metadata": {"name": "kube-system"}}] + } + + with patch.object(client, "_core_v1", fake_v1): + namespaces = await client.list_namespaces() + + assert namespaces == ["default", "kube-system"] + assert [event.operation for event in seen] == ["list"] + assert seen[0].path == "/api/v1/namespaces" + assert seen[0].object_count == 2 + assert seen[0].decoded_bytes > 0 + assert seen[0].status is None + + async def test_list_pods_parses_summaries() -> None: client = KubeClient() fake_v1 = AsyncMock() @@ -95,6 +115,23 @@ async def test_list_pods_parses_summaries() -> None: fake_v1.list_namespaced_pod.assert_awaited_once_with("default", _preload_content=False) +async def test_list_pods_emits_list_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespaced_pod.return_value = {"items": [_pod("a"), _pod("b")]} + + with patch.object(client, "_core_v1", fake_v1): + pods = await client.list_pods("default") + + assert [pod.name for pod in pods] == ["a", "b"] + assert [event.operation for event in seen] == ["list"] + assert seen[0].path == "/api/v1/namespaces/default/pods" + assert seen[0].object_count == 2 + assert seen[0].decoded_bytes > 0 + assert seen[0].status is None + + async def test_watch_pods_yields_list_items_first() -> None: """Pre-existing pods from the initial LIST appear as ADDED before watch events.""" client = KubeClient() @@ -119,6 +156,57 @@ async def test_watch_pods_yields_list_items_first() -> None: assert collected[2] == ("MODIFIED", "alpha") +async def test_pod_watch_emits_list_open_and_event_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespaced_pod.return_value = { + "metadata": {"resourceVersion": "100"}, + "items": [_pod("listed")], + } + fake_watch = _FakeWatch([{"type": "MODIFIED", "raw_object": _pod("watched")}]) + + with ( + patch.object(client, "_core_v1", fake_v1), + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + ): + collected = [ + (event_type, pod.name) async for event_type, pod in client.watch_pods("default") + ] + + assert collected == [("ADDED", "listed"), ("MODIFIED", "watched")] + assert [event.operation for event in seen] == ["list", "watch_open", "watch_event"] + assert {event.path for event in seen} == {"/api/v1/namespaces/default/pods"} + assert seen[0].object_count == 1 + assert seen[0].decoded_bytes > 0 + assert seen[1].decoded_bytes == 0 + assert seen[2].object_count == 1 + assert seen[2].decoded_bytes > 0 + + +async def test_no_telemetry_preserves_existing_watch_behavior() -> None: + client = KubeClient() + fake_v1 = AsyncMock() + fake_v1.list_namespaced_pod.return_value = { + "metadata": {"resourceVersion": "100"}, + "items": [_pod("listed")], + } + fake_watch = _FakeWatch([]) + + with ( + patch.object(client, "_core_v1", fake_v1), + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + patch( + "korvid.k8s.client.json.dumps", side_effect=AssertionError("unexpected serialization") + ), + ): + collected = [ + (event_type, pod.name) async for event_type, pod in client.watch_pods("default") + ] + + assert collected == [("ADDED", "listed")] + + async def test_watch_pods_passes_resource_version_to_watch() -> None: """resource_version captured from the LIST is forwarded to Watch.stream.""" client = KubeClient() @@ -153,6 +241,49 @@ async def test_watch_pods_list_api_error_raises_api_status_error() -> None: assert exc_info.value.status == 403 +async def test_watch_pods_list_error_emits_error_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespaced_pod.side_effect = ApiException(status=403, reason="Forbidden") + + with ( + patch.object(client, "_core_v1", fake_v1), + pytest.raises(ApiStatusError, match="API 403: Forbidden"), + ): + async for _ in client.watch_pods("default"): + pass + + assert [event.operation for event in seen] == ["error"] + assert seen[0].path == "/api/v1/namespaces/default/pods" + assert seen[0].status == 403 + assert seen[0].decoded_bytes == 0 + assert seen[0].object_count == 0 + + +async def test_watch_pods_watch_error_emits_error_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespaced_pod.return_value = { + "metadata": {"resourceVersion": "100"}, + "items": [], + } + fake_watch = _FakeWatch([], raise_at=0, raise_exc=ApiException(status=410, reason="Gone")) + + with ( + patch.object(client, "_core_v1", fake_v1), + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + pytest.raises(ApiStatusError, match="API 410: Gone"), + ): + async for _ in client.watch_pods("default"): + pass + + assert [event.operation for event in seen] == ["list", "watch_open", "error"] + assert seen[-1].path == "/api/v1/namespaces/default/pods" + assert seen[-1].status == 410 + + async def test_watch_pods_all_namespaces_uses_cluster_path() -> None: """watch_pods(None) LISTs /api/v1/pods without a /namespaces/ segment.""" client = KubeClient() @@ -186,6 +317,25 @@ async def test_list_namespaces_api_error_raises_api_status_error() -> None: await client.list_namespaces() +async def test_list_namespaces_error_emits_error_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespace.side_effect = ApiException(status=403, reason="Forbidden") + + with ( + patch.object(client, "_core_v1", fake_v1), + pytest.raises(ApiStatusError, match="API 403: Forbidden"), + ): + await client.list_namespaces() + + assert [event.operation for event in seen] == ["error"] + assert seen[0].path == "/api/v1/namespaces" + assert seen[0].status == 403 + assert seen[0].decoded_bytes == 0 + assert seen[0].object_count == 0 + + async def test_list_pods_api_error_raises_api_status_error() -> None: """ApiException must not cross the k8s boundary from list_pods.""" client = KubeClient() @@ -237,6 +387,36 @@ async def test_watch_objects_yields_list_items_first() -> None: assert collected[2] == ("MODIFIED", "dep-a") +async def test_watch_objects_emits_list_open_and_event_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + list_resp = { + "metadata": {"resourceVersion": "200"}, + "items": [_generic("dep-a")], + } + watch_events = [{"type": "MODIFIED", "raw_object": _generic("dep-a")}] + fake_watch = _FakeWatch(watch_events) + + with ( + patch.object(client, "_api", MagicMock()), + patch.object(client, "_request_json", AsyncMock(return_value=list_resp)), + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + ): + collected = [ + (ev, summary.name) async for ev, summary in client.watch_objects(meta, "default") + ] + + assert collected == [("ADDED", "dep-a"), ("MODIFIED", "dep-a")] + assert [event.operation for event in seen] == ["list", "watch_open", "watch_event"] + assert {event.path for event in seen} == {"/apis/apps/v1/namespaces/default/deployments"} + assert seen[0].object_count == 1 + assert seen[0].decoded_bytes > 0 + assert seen[1].decoded_bytes == 0 + assert seen[2].object_count == 1 + assert seen[2].decoded_bytes > 0 + + async def test_watch_objects_replicaset_yields_rich_summary() -> None: """ReplicaSet kinds get ReplicaSetSummary (revision/desired/ready) via summary_for.""" client = KubeClient() @@ -455,6 +635,30 @@ async def test_watch_non_405_api_status_error_still_raises() -> None: assert excinfo.value.body == '{"kind":"Status"}' +async def test_watch_objects_list_error_emits_error_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + + with ( + patch.object(client, "_api", MagicMock()), + patch.object( + client, + "_request_json", + AsyncMock(side_effect=ApiStatusError(401, "Unauthorized")), + ), + pytest.raises(ApiStatusError, match="API 401: Unauthorized"), + ): + async for _ in client.watch_objects(meta, "default"): + pass + + assert [event.operation for event in seen] == ["error"] + assert seen[0].path == "/apis/apps/v1/namespaces/default/deployments" + assert seen[0].status == 401 + assert seen[0].decoded_bytes == 0 + assert seen[0].object_count == 0 + + async def test_watch_non_405_api_exception_still_raises() -> None: """Only the deterministic 405 falls back to polling: other watch errors keep propagating so the WatchManager's retry/report loop stays in charge.""" @@ -473,11 +677,49 @@ async def test_watch_non_405_api_exception_still_raises() -> None: pass +async def test_watch_objects_watch_error_emits_error_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + list_resp: dict[str, Any] = {"metadata": {"resourceVersion": "9"}, "items": []} + fake_watch = _FakeWatch([], raise_at=0, raise_exc=ApiException(status=500, reason="boom")) + + with ( + patch.object(client, "_api", MagicMock()), + patch.object(client, "_request_json", AsyncMock(return_value=list_resp)), + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + pytest.raises(ApiStatusError, match="boom"), + ): + async for _ in client.watch_objects(meta, "default"): + pass + + assert [event.operation for event in seen] == ["list", "watch_open", "error"] + assert seen[-1].path == "/apis/apps/v1/namespaces/default/deployments" + assert seen[-1].status == 500 + + # --------------------------------------------------------------------------- # get_object # --------------------------------------------------------------------------- +async def test_get_object_emits_get_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + request_json_mock = AsyncMock(return_value=_generic("my-dep")) + + with patch.object(client, "_request_json", request_json_mock): + obj = await client.get_object(meta, "default", "my-dep") + + assert obj["metadata"]["name"] == "my-dep" + assert [event.operation for event in seen] == ["get"] + assert seen[0].path == "/apis/apps/v1/namespaces/default/deployments/my-dep" + assert seen[0].object_count == 1 + assert seen[0].decoded_bytes > 0 + assert seen[0].status is None + + async def test_get_object_raises_api_status_error() -> None: """ApiException from the raw GET is wrapped as ApiStatusError.""" client = KubeClient() @@ -731,6 +973,26 @@ async def test_list_objects_returns_generic_summaries() -> None: assert "/namespaces/default/deployments" in called_path +async def test_list_objects_emits_list_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + request_json_mock = AsyncMock(return_value={"items": [_generic("dep-a"), _generic("dep-b")]}) + + with ( + patch.object(client, "_api", MagicMock()), + patch.object(client, "_request_json", request_json_mock), + ): + summaries = await client.list_objects(meta, "default") + + assert [summary.name for summary in summaries] == ["dep-a", "dep-b"] + assert [event.operation for event in seen] == ["list"] + assert seen[0].path == "/apis/apps/v1/namespaces/default/deployments" + assert seen[0].object_count == 2 + assert seen[0].decoded_bytes > 0 + assert seen[0].status is None + + async def test_list_objects_all_namespaces_uses_cluster_path() -> None: """namespace=None produces a cluster-scoped path without /namespaces/.""" client = KubeClient() From e0d7659b9d75ee76d5f590e8406283f6cb9153cc Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 01:37:30 +0900 Subject: [PATCH 06/38] fix: reuse watch snapshot summaries Avoid duplicate generic watch summary projection, clean up reused item lists, and document the verified exception flow from review follow-up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/korvid/k8s/client.py | 19 +++++++++++-------- tests/k8s/test_client.py | 24 ++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 8 deletions(-) diff --git a/src/korvid/k8s/client.py b/src/korvid/k8s/client.py index b4fc9461..6f5715fd 100644 --- a/src/korvid/k8s/client.py +++ b/src/korvid/k8s/client.py @@ -320,7 +320,7 @@ async def list_namespaces(self) -> list[str]: raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc items = data.get("items", []) self._observe_read("list", path, payload=data, object_count=len(items)) - return [item["metadata"]["name"] for item in data.get("items", [])] + return [item["metadata"]["name"] for item in items] async def detect_cloud_provider(self) -> ProviderInfo: """Detect the cluster's cloud provider from a few nodes (issue #30). @@ -522,7 +522,7 @@ def _list_path(self, meta: ResourceMeta, namespace: str | None) -> str: async def _initial_object_snapshot( self, meta: ResourceMeta, namespace: str | None - ) -> tuple[str, dict[str, Any], dict[str, GenericSummary]]: + ) -> tuple[str, str | None, list[GenericSummary], dict[str, GenericSummary]]: list_path = self._list_path(meta, namespace) try: data = await self._request_json(list_path) @@ -532,11 +532,14 @@ async def _initial_object_snapshot( items = data.get("items", []) self._observe_read("list", list_path, payload=data, object_count=len(items)) + resource_version = (data.get("metadata") or {}).get("resourceVersion") + summaries: list[GenericSummary] = [] known: dict[str, GenericSummary] = {} for item in items: summary = self._object_summary(meta, item) + summaries.append(summary) known[f"{summary.namespace}/{summary.name}"] = summary - return list_path, data, known + return list_path, resource_version, summaries, known def _watch_objects_requires_poll_fallback( self, @@ -572,10 +575,10 @@ async def watch_objects( raise RuntimeError("connect() first") # LIST phase -------------------------------------------------------- - list_path, data, known = await self._initial_object_snapshot(meta, namespace) - resource_version: str | None = (data.get("metadata") or {}).get("resourceVersion") - for item in data.get("items", []): - summary = self._object_summary(meta, item) + list_path, resource_version, initial_summaries, known = await self._initial_object_snapshot( + meta, namespace + ) + for summary in initial_summaries: yield ("ADDED", summary) if not meta.watchable: @@ -659,7 +662,7 @@ async def list_objects(self, meta: ResourceMeta, namespace: str | None) -> list[ raise items = data.get("items", []) self._observe_read("list", path, payload=data, object_count=len(items)) - return [self._object_summary(meta, item) for item in data.get("items", [])] + return [self._object_summary(meta, item) for item in items] async def get_object( self, meta: ResourceMeta, namespace: str | None, name: str diff --git a/tests/k8s/test_client.py b/tests/k8s/test_client.py index dfd9885f..a55afc6e 100644 --- a/tests/k8s/test_client.py +++ b/tests/k8s/test_client.py @@ -417,6 +417,30 @@ async def test_watch_objects_emits_list_open_and_event_telemetry() -> None: assert seen[2].decoded_bytes > 0 +async def test_watch_objects_initial_snapshot_reuses_computed_summaries() -> None: + client = KubeClient() + meta = _deploy_meta() + list_resp = { + "metadata": {"resourceVersion": "200"}, + "items": [_generic("dep-a")], + } + fake_watch = _FakeWatch([]) + original_summary = client._object_summary + + with ( + patch.object(client, "_api", MagicMock()), + patch.object(client, "_request_json", AsyncMock(return_value=list_resp)), + patch.object(client, "_object_summary", side_effect=original_summary) as summary_mock, + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + ): + collected = [ + (ev, summary.name) async for ev, summary in client.watch_objects(meta, "default") + ] + + assert collected == [("ADDED", "dep-a")] + assert summary_mock.call_count == 1 + + async def test_watch_objects_replicaset_yields_rich_summary() -> None: """ReplicaSet kinds get ReplicaSetSummary (revision/desired/ready) via summary_for.""" client = KubeClient() From 808e00880bec70f3e4ea0467721af1d3029801e8 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 01:51:22 +0900 Subject: [PATCH 07/38] test: report scale benchmark latency and resource use Add stable JSON and Markdown measurements for issue #186 comparisons. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/metrics.py | 387 ++++++++++++++++++++++++++++++ tests/performance/test_metrics.py | 253 +++++++++++++++++++ 2 files changed, 640 insertions(+) create mode 100644 tests/performance/metrics.py create mode 100644 tests/performance/test_metrics.py diff --git a/tests/performance/metrics.py b/tests/performance/metrics.py new file mode 100644 index 00000000..6cd4a66b --- /dev/null +++ b/tests/performance/metrics.py @@ -0,0 +1,387 @@ +from __future__ import annotations + +import asyncio +import math +import tracemalloc +from collections import Counter +from collections.abc import Callable, Sequence +from contextlib import suppress +from dataclasses import dataclass +from time import monotonic + +import psutil # type: ignore[import-untyped] # dependency ships without inline stubs + +from korvid.k8s.telemetry import ReadTelemetryEvent + +_MIB = 1024 * 1024 + + +def _nearest_rank(samples: Sequence[float], percentile: float) -> float: + index = math.ceil(percentile * len(samples)) - 1 + return samples[index] + + +def _max_or_none(samples: Sequence[float]) -> float | None: + if not samples: + return None + return max(samples) + + +def _least_squares_slope(samples: Sequence[ProcessSample]) -> float | None: + if len(samples) < 2: + return None + xs = [sample.elapsed_seconds for sample in samples] + ys = [float(sample.rss_bytes) for sample in samples] + x_mean = sum(xs) / len(xs) + y_mean = sum(ys) / len(ys) + numerator = sum((x - x_mean) * (y - y_mean) for x, y in zip(xs, ys, strict=True)) + denominator = sum((x - x_mean) ** 2 for x in xs) + if denominator == 0: + return None + return (numerator / denominator) * 60 / _MIB + + +@dataclass(frozen=True) +class LatencySummary: + count: int + p50_seconds: float | None + p95_seconds: float | None + p99_seconds: float | None + maximum_seconds: float | None + + @classmethod + def from_samples(cls, samples: Sequence[float]) -> LatencySummary: + ordered = sorted(samples) + if not ordered: + return cls( + count=0, + p50_seconds=None, + p95_seconds=None, + p99_seconds=None, + maximum_seconds=None, + ) + return cls( + count=len(ordered), + p50_seconds=_nearest_rank(ordered, 0.50), + p95_seconds=_nearest_rank(ordered, 0.95), + p99_seconds=_nearest_rank(ordered, 0.99), + maximum_seconds=ordered[-1], + ) + + +@dataclass(frozen=True) +class ProcessSample: + elapsed_seconds: float + cpu_percent: float + rss_bytes: int + python_bytes: int + + +@dataclass(frozen=True) +class RunManifest: + profile_id: str + profile_hash: str + korvid_sha: str + python: str + textual: str + os: str + cpu_count: int + memory_bytes: int + + +@dataclass(frozen=True) +class ProcessSummary: + sample_count: int + cpu_percent_max: float | None + rss_bytes_max: int | None + python_bytes_max: int | None + rss_slope_mib_per_minute: float | None + + @classmethod + def from_samples(cls, samples: Sequence[ProcessSample]) -> ProcessSummary: + if not samples: + return cls( + sample_count=0, + cpu_percent_max=None, + rss_bytes_max=None, + python_bytes_max=None, + rss_slope_mib_per_minute=None, + ) + return cls( + sample_count=len(samples), + cpu_percent_max=max(sample.cpu_percent for sample in samples), + rss_bytes_max=max(sample.rss_bytes for sample in samples), + python_bytes_max=max(sample.python_bytes for sample in samples), + rss_slope_mib_per_minute=_least_squares_slope(samples), + ) + + +@dataclass(frozen=True) +class ApiSummary: + operations: dict[str, int] + paths: dict[str, dict[str, int]] + decoded_bytes: int + object_count: int + watch_events: int + reconnects: int + relists: int + throttles: int + authorization_failures: int + + @classmethod + def from_events(cls, events: Sequence[ReadTelemetryEvent]) -> ApiSummary: + operations = Counter[str]() + paths: dict[str, Counter[str]] = {} + decoded_bytes = 0 + object_count = 0 + watch_events = 0 + throttles = 0 + authorization_failures = 0 + for event in events: + operations[event.operation] += 1 + paths.setdefault(event.path, Counter())[event.operation] += 1 + decoded_bytes += event.decoded_bytes + object_count += event.object_count + if event.operation == "watch_event": + watch_events += event.object_count + if event.status == 429: + throttles += 1 + if event.status in {401, 403}: + authorization_failures += 1 + reconnects = sum(max(counts.get("watch_open", 0) - 1, 0) for counts in paths.values()) + relists = sum(max(counts.get("list", 0) - 1, 0) for counts in paths.values()) + return cls( + operations=dict(sorted(operations.items())), + paths={ + path: dict(sorted(counts.items())) + for path, counts in sorted(paths.items(), key=lambda item: item[0]) + }, + decoded_bytes=decoded_bytes, + object_count=object_count, + watch_events=watch_events, + reconnects=reconnects, + relists=relists, + throttles=throttles, + authorization_failures=authorization_failures, + ) + + +@dataclass(frozen=True) +class BenchmarkReport: + manifest: RunManifest + event_to_render: LatencySummary + input_latency: LatencySummary + process: ProcessSummary + api: ApiSummary + rendered_updates: int + render_passes: int + coalesced_updates: int + dropped_updates: int + final_digest: str + + +class ProcessSampler: + def __init__(self, interval_seconds: float, clock: Callable[[], float] = monotonic) -> None: + self._interval_seconds = interval_seconds + self._clock = clock + self._process = psutil.Process() + self._start_time: float | None = None + self._samples: list[ProcessSample] = [] + self._task: asyncio.Task[None] | None = None + + def start(self) -> None: + self._samples.clear() + self._start_time = self._clock() + self._process.cpu_percent() + self._task = asyncio.create_task(self._run()) + + async def stop(self) -> tuple[ProcessSample, ...]: + if self._task is None: + return tuple(self._samples) + task = self._task + self._task = None + task.cancel() + with suppress(asyncio.CancelledError): + await task + return tuple(self._samples) + + async def _run(self) -> None: + assert self._start_time is not None + while True: + self._samples.append( + ProcessSample( + elapsed_seconds=self._clock() - self._start_time, + cpu_percent=self._process.cpu_percent(), + rss_bytes=self._process.memory_info().rss, + python_bytes=tracemalloc.get_traced_memory()[0], + ) + ) + await asyncio.sleep(self._interval_seconds) + + +class BenchmarkRecorder: + def __init__(self) -> None: + self._pending_events: list[tuple[int, float]] = [] + self._event_to_render: list[float] = [] + self._input_latency: list[float] = [] + self._api_events: list[ReadTelemetryEvent] = [] + self._rendered_updates = 0 + self._render_passes = 0 + self._coalesced_updates = 0 + + def record_event(self, sequence: int, received_at: float) -> None: + self._pending_events.append((sequence, received_at)) + + def record_render(self, rendered_at: float) -> None: + if not self._pending_events: + return + pending = tuple(self._pending_events) + self._pending_events.clear() + self._render_passes += 1 + self._rendered_updates += len(pending) + self._coalesced_updates += max(len(pending) - 1, 0) + for _, received_at in pending: + self._event_to_render.append(rendered_at - received_at) + + def record_input(self, latency_seconds: float) -> None: + self._input_latency.append(latency_seconds) + + def record_api(self, event: ReadTelemetryEvent) -> None: + self._api_events.append(event) + + def report( + self, + manifest: RunManifest, + process_samples: Sequence[ProcessSample], + *, + final_digest: str, + ) -> BenchmarkReport: + return BenchmarkReport( + manifest=manifest, + event_to_render=LatencySummary.from_samples(self._event_to_render), + input_latency=LatencySummary.from_samples(self._input_latency), + process=ProcessSummary.from_samples(process_samples), + api=ApiSummary.from_events(self._api_events), + rendered_updates=self._rendered_updates, + render_passes=self._render_passes, + coalesced_updates=self._coalesced_updates, + dropped_updates=len(self._pending_events), + final_digest=final_digest, + ) + + +def report_payload(report: BenchmarkReport) -> dict[str, object]: + return { + "manifest": { + "profile_id": report.manifest.profile_id, + "profile_hash": report.manifest.profile_hash, + "korvid_sha": report.manifest.korvid_sha, + "python": report.manifest.python, + "textual": report.manifest.textual, + "os": report.manifest.os, + "cpu_count": report.manifest.cpu_count, + "memory_bytes": report.manifest.memory_bytes, + }, + "latency": { + "event_to_render": { + "count": report.event_to_render.count, + "p50_seconds": report.event_to_render.p50_seconds, + "p95_seconds": report.event_to_render.p95_seconds, + "p99_seconds": report.event_to_render.p99_seconds, + "maximum_seconds": report.event_to_render.maximum_seconds, + }, + "input": { + "count": report.input_latency.count, + "p50_seconds": report.input_latency.p50_seconds, + "p95_seconds": report.input_latency.p95_seconds, + "p99_seconds": report.input_latency.p99_seconds, + "maximum_seconds": report.input_latency.maximum_seconds, + }, + }, + "process": { + "sample_count": report.process.sample_count, + "cpu_percent_max": report.process.cpu_percent_max, + "rss_bytes_max": report.process.rss_bytes_max, + "python_bytes_max": report.process.python_bytes_max, + "rss_slope_mib_per_minute": report.process.rss_slope_mib_per_minute, + }, + "api": { + "operations": report.api.operations, + "paths": report.api.paths, + "decoded_bytes": report.api.decoded_bytes, + "object_count": report.api.object_count, + "watch_events": report.api.watch_events, + "reconnects": report.api.reconnects, + "relists": report.api.relists, + "throttles": report.api.throttles, + "authorization_failures": report.api.authorization_failures, + }, + "updates": { + "rendered_updates": report.rendered_updates, + "render_passes": report.render_passes, + "coalesced_updates": report.coalesced_updates, + "dropped_updates": report.dropped_updates, + }, + "digests": {"final": report.final_digest}, + } + + +def render_markdown(report: BenchmarkReport) -> str: + operation_lines = [ + f"- {operation}: `{count}`" for operation, count in report.api.operations.items() + ] + lines = [ + "# Large-cluster benchmark report", + "", + "## Run manifest", + f"- Profile ID: `{report.manifest.profile_id}`", + f"- Profile hash: `{report.manifest.profile_hash}`", + f"- Korvid SHA: `{report.manifest.korvid_sha}`", + "", + "## Latency", + f"- Event to render p95: `{_format_seconds(report.event_to_render.p95_seconds)}`", + f"- Event to render p99: `{_format_seconds(report.event_to_render.p99_seconds)}`", + f"- Event to render max: `{_format_seconds(report.event_to_render.maximum_seconds)}`", + f"- Input latency p95: `{_format_seconds(report.input_latency.p95_seconds)}`", + "", + "## Process", + f"- CPU max: `{_format_float(report.process.cpu_percent_max)}`", + f"- RSS max: `{_format_int(report.process.rss_bytes_max)}`", + f"- RSS slope: `{_format_slope(report.process.rss_slope_mib_per_minute)}`", + "", + "## Updates", + f"- Rendered updates: `{report.rendered_updates}`", + f"- Coalesced updates: `{report.coalesced_updates}`", + f"- Dropped updates: `{report.dropped_updates}`", + "", + "## API operations", + *operation_lines, + "", + "## Digests", + f"- Final digest: `{report.final_digest}`", + ] + return "\n".join(lines) + "\n" + + +def _format_seconds(value: float | None) -> str: + if value is None: + return "n/a" + return f"{value:.3f}s" + + +def _format_float(value: float | None) -> str: + if value is None: + return "n/a" + return f"{value:.1f}" + + +def _format_int(value: int | None) -> str: + if value is None: + return "n/a" + return str(value) + + +def _format_slope(value: float | None) -> str: + if value is None: + return "n/a" + return f"{value:.2f} MiB/min" diff --git a/tests/performance/test_metrics.py b/tests/performance/test_metrics.py new file mode 100644 index 00000000..3c00e870 --- /dev/null +++ b/tests/performance/test_metrics.py @@ -0,0 +1,253 @@ +from __future__ import annotations + +import asyncio +import importlib +import json +from typing import Any, cast + +import pytest + +from korvid.k8s.telemetry import ReadTelemetryEvent +from tests.performance.metrics import ( + BenchmarkRecorder, + LatencySummary, + ProcessSample, + ProcessSampler, + RunManifest, + render_markdown, + report_payload, +) + + +def run_manifest() -> RunManifest: + return RunManifest( + profile_id="smoke-1k", + profile_hash="profile-hash", + korvid_sha="3cbe600996043cd6cc9df114a71d10c938545789", + python="3.12.10", + textual="3.2.0", + os="Darwin", + cpu_count=8, + memory_bytes=16 * 1024 * 1024 * 1024, + ) + + +def process_samples() -> tuple[ProcessSample, ...]: + mib = 1024 * 1024 + return ( + ProcessSample( + elapsed_seconds=0.0, cpu_percent=10.0, rss_bytes=100 * mib, python_bytes=5 * mib + ), + ProcessSample( + elapsed_seconds=60.0, cpu_percent=20.0, rss_bytes=101 * mib, python_bytes=6 * mib + ), + ProcessSample( + elapsed_seconds=120.0, cpu_percent=30.0, rss_bytes=102 * mib, python_bytes=7 * mib + ), + ) + + +def test_latency_summary_uses_nearest_rank_percentiles() -> None: + summary = LatencySummary.from_samples([0.50, 0.10, 0.20, 0.40, 0.30]) + + assert summary.count == 5 + assert summary.p50_seconds == pytest.approx(0.30) + assert summary.p95_seconds == pytest.approx(0.50) + assert summary.p99_seconds == pytest.approx(0.50) + assert summary.maximum_seconds == pytest.approx(0.50) + + +def test_latency_summary_is_empty_without_samples() -> None: + summary = LatencySummary.from_samples([]) + + assert summary.count == 0 + assert summary.p50_seconds is None + assert summary.p95_seconds is None + assert summary.p99_seconds is None + assert summary.maximum_seconds is None + + +def test_render_flushes_all_pending_events_as_coalesced() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 1.0) + recorder.record_event(2, 1.1) + recorder.record_render(1.2) + + report = recorder.report(run_manifest(), process_samples(), final_digest="abc") + + assert report.rendered_updates == 2 + assert report.render_passes == 1 + assert report.coalesced_updates == 1 + assert report.dropped_updates == 0 + assert report.event_to_render.count == 2 + assert report.event_to_render.p50_seconds == pytest.approx(0.1) + assert report.event_to_render.maximum_seconds == pytest.approx(0.2) + + +def test_report_marks_pending_events_as_dropped() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 1.0) + recorder.record_event(2, 1.1) + + report = recorder.report(run_manifest(), (), final_digest="abc") + + assert report.rendered_updates == 0 + assert report.render_passes == 0 + assert report.coalesced_updates == 0 + assert report.dropped_updates == 2 + assert report.event_to_render.count == 0 + + +def test_report_counts_api_operations_without_path_loss() -> None: + recorder = BenchmarkRecorder() + recorder.record_api( + ReadTelemetryEvent("list", "/api/v1/pods", object_count=1000, decoded_bytes=2048) + ) + recorder.record_api(ReadTelemetryEvent("watch_open", "/api/v1/pods")) + + payload = report_payload(recorder.report(run_manifest(), [], final_digest="abc")) + api = cast(dict[str, object], payload["api"]) + operations = cast(dict[str, int], api["operations"]) + paths = cast(dict[str, object], api["paths"]) + pod_path = cast(dict[str, int], paths["/api/v1/pods"]) + + assert operations == {"list": 1, "watch_open": 1} + assert pod_path["list"] == 1 + assert pod_path["watch_open"] == 1 + assert api["decoded_bytes"] == 2048 + assert api["object_count"] == 1000 + + +def test_report_payload_is_json_serializable_and_stable() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 1.0) + recorder.record_render(1.2) + recorder.record_input(0.05) + recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=429)) + + payload = report_payload( + recorder.report(run_manifest(), process_samples(), final_digest="digest-123") + ) + encoded = json.dumps(payload, sort_keys=True) + + assert payload == { + "manifest": { + "profile_id": "smoke-1k", + "profile_hash": "profile-hash", + "korvid_sha": "3cbe600996043cd6cc9df114a71d10c938545789", + "python": "3.12.10", + "textual": "3.2.0", + "os": "Darwin", + "cpu_count": 8, + "memory_bytes": 17179869184, + }, + "latency": { + "event_to_render": { + "count": 1, + "p50_seconds": 0.19999999999999996, + "p95_seconds": 0.19999999999999996, + "p99_seconds": 0.19999999999999996, + "maximum_seconds": 0.19999999999999996, + }, + "input": { + "count": 1, + "p50_seconds": 0.05, + "p95_seconds": 0.05, + "p99_seconds": 0.05, + "maximum_seconds": 0.05, + }, + }, + "process": { + "sample_count": 3, + "cpu_percent_max": 30.0, + "rss_bytes_max": 106954752, + "python_bytes_max": 7340032, + "rss_slope_mib_per_minute": 1.0, + }, + "api": { + "operations": {"error": 1}, + "paths": {"/api/v1/pods": {"error": 1}}, + "decoded_bytes": 0, + "object_count": 0, + "watch_events": 0, + "reconnects": 0, + "relists": 0, + "throttles": 1, + "authorization_failures": 0, + }, + "updates": { + "rendered_updates": 1, + "render_passes": 1, + "coalesced_updates": 0, + "dropped_updates": 0, + }, + "digests": {"final": "digest-123"}, + } + assert '"rss_slope_mib_per_minute": 1.0' in encoded + + +def test_render_markdown_uses_stable_labels() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 1.0) + recorder.record_render(1.1) + recorder.record_input(0.02) + recorder.record_api(ReadTelemetryEvent("watch_open", "/api/v1/pods")) + + text = render_markdown(recorder.report(run_manifest(), process_samples(), final_digest="abc")) + + assert "# Large-cluster benchmark report" in text + assert "- Profile ID: `smoke-1k`" in text + assert "- Event to render p95: `0.100s`" in text + assert "- Input latency p95: `0.020s`" in text + assert "- RSS slope: `1.00 MiB/min`" in text + assert "- Rendered updates: `1`" in text + assert "- watch_open: `1`" in text + assert "- Final digest: `abc`" in text + + +@pytest.mark.asyncio +async def test_process_sampler_skips_warmup_sample( + monkeypatch: pytest.MonkeyPatch, +) -> None: + metrics = cast(Any, importlib.import_module("tests.performance.metrics")) + + class _MemoryInfo: + def __init__(self, rss: int) -> None: + self.rss = rss + + class _FakeProcess: + def __init__(self) -> None: + self.cpu_values = iter((0.0, 12.5, 15.0)) + self.rss_values = iter((100, 110)) + + def cpu_percent(self, interval: float | None = None) -> float: + return next(self.cpu_values) + + def memory_info(self) -> _MemoryInfo: + return _MemoryInfo(next(self.rss_values)) + + times = iter((10.0, 11.0)) + python_bytes = iter((1000, 1200)) + blocker = asyncio.Event() + original_sleep = asyncio.sleep + + async def _fake_sleep(_: float) -> None: + await blocker.wait() + + monkeypatch.setattr(metrics.psutil, "Process", _FakeProcess) + monkeypatch.setattr(metrics.tracemalloc, "get_traced_memory", lambda: (next(python_bytes), 0)) + monkeypatch.setattr(metrics.asyncio, "sleep", _fake_sleep) + + sampler = ProcessSampler(interval_seconds=0.01, clock=lambda: next(times)) + sampler.start() + await original_sleep(0) + samples = await sampler.stop() + + assert samples == ( + ProcessSample( + elapsed_seconds=1.0, + cpu_percent=12.5, + rss_bytes=100, + python_bytes=1000, + ), + ) From 152808f804960ec353d1b1affb290eb44da6a2be Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 02:02:15 +0900 Subject: [PATCH 08/38] test: harden scale benchmark metric publishing Freeze published API aggregates, harden ProcessSampler lifecycle, and count relists only after 410 recovery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/metrics.py | 44 ++++-- tests/performance/test_metrics.py | 226 +++++++++++++++++++++++++++--- 2 files changed, 237 insertions(+), 33 deletions(-) diff --git a/tests/performance/metrics.py b/tests/performance/metrics.py index 6cd4a66b..eaf2cac5 100644 --- a/tests/performance/metrics.py +++ b/tests/performance/metrics.py @@ -4,10 +4,11 @@ import math import tracemalloc from collections import Counter -from collections.abc import Callable, Sequence +from collections.abc import Callable, Mapping, Sequence from contextlib import suppress from dataclasses import dataclass from time import monotonic +from types import MappingProxyType import psutil # type: ignore[import-untyped] # dependency ships without inline stubs @@ -118,8 +119,8 @@ def from_samples(cls, samples: Sequence[ProcessSample]) -> ProcessSummary: @dataclass(frozen=True) class ApiSummary: - operations: dict[str, int] - paths: dict[str, dict[str, int]] + operations: Mapping[str, int] + paths: Mapping[str, Mapping[str, int]] decoded_bytes: int object_count: int watch_events: int @@ -137,6 +138,8 @@ def from_events(cls, events: Sequence[ReadTelemetryEvent]) -> ApiSummary: watch_events = 0 throttles = 0 authorization_failures = 0 + relists = 0 + relist_candidates: set[str] = set() for event in events: operations[event.operation] += 1 paths.setdefault(event.path, Counter())[event.operation] += 1 @@ -148,14 +151,20 @@ def from_events(cls, events: Sequence[ReadTelemetryEvent]) -> ApiSummary: throttles += 1 if event.status in {401, 403}: authorization_failures += 1 + if event.operation == "list" and event.path in relist_candidates: + relists += 1 + relist_candidates.remove(event.path) + if event.status == 410: + relist_candidates.add(event.path) reconnects = sum(max(counts.get("watch_open", 0) - 1, 0) for counts in paths.values()) - relists = sum(max(counts.get("list", 0) - 1, 0) for counts in paths.values()) return cls( - operations=dict(sorted(operations.items())), - paths={ - path: dict(sorted(counts.items())) - for path, counts in sorted(paths.items(), key=lambda item: item[0]) - }, + operations=MappingProxyType(dict(sorted(operations.items()))), + paths=MappingProxyType( + { + path: MappingProxyType(dict(sorted(counts.items()))) + for path, counts in sorted(paths.items(), key=lambda item: item[0]) + } + ), decoded_bytes=decoded_bytes, object_count=object_count, watch_events=watch_events, @@ -188,10 +197,18 @@ def __init__(self, interval_seconds: float, clock: Callable[[], float] = monoton self._start_time: float | None = None self._samples: list[ProcessSample] = [] self._task: asyncio.Task[None] | None = None + self._owns_tracemalloc = False def start(self) -> None: + if self._task is not None: + raise RuntimeError( + "ProcessSampler.start() cannot run while sampling is already running" + ) self._samples.clear() self._start_time = self._clock() + self._owns_tracemalloc = not tracemalloc.is_tracing() + if self._owns_tracemalloc: + tracemalloc.start() self._process.cpu_percent() self._task = asyncio.create_task(self._run()) @@ -203,6 +220,9 @@ async def stop(self) -> tuple[ProcessSample, ...]: task.cancel() with suppress(asyncio.CancelledError): await task + if self._owns_tracemalloc and tracemalloc.is_tracing(): + tracemalloc.stop() + self._owns_tracemalloc = False return tuple(self._samples) async def _run(self) -> None: @@ -271,6 +291,8 @@ def report( def report_payload(report: BenchmarkReport) -> dict[str, object]: + api_operations = dict(report.api.operations) + api_paths = {path: dict(counts) for path, counts in report.api.paths.items()} return { "manifest": { "profile_id": report.manifest.profile_id, @@ -306,8 +328,8 @@ def report_payload(report: BenchmarkReport) -> dict[str, object]: "rss_slope_mib_per_minute": report.process.rss_slope_mib_per_minute, }, "api": { - "operations": report.api.operations, - "paths": report.api.paths, + "operations": api_operations, + "paths": api_paths, "decoded_bytes": report.api.decoded_bytes, "object_count": report.api.object_count, "watch_events": report.api.watch_events, diff --git a/tests/performance/test_metrics.py b/tests/performance/test_metrics.py index 3c00e870..12b1e90a 100644 --- a/tests/performance/test_metrics.py +++ b/tests/performance/test_metrics.py @@ -19,6 +19,68 @@ ) +class _MemoryInfo: + def __init__(self, rss: int) -> None: + self.rss = rss + + +class _FakeProcess: + def __init__(self, cpu_values: tuple[float, ...], rss_values: tuple[int, ...]) -> None: + self.cpu_values = iter(cpu_values) + self.rss_values = iter(rss_values) + + def cpu_percent(self, interval: float | None = None) -> float: + return next(self.cpu_values) + + def memory_info(self) -> _MemoryInfo: + return _MemoryInfo(next(self.rss_values)) + + +def _metrics_module() -> Any: + return cast(Any, importlib.import_module("tests.performance.metrics")) + + +def _patch_sampler_runtime( + monkeypatch: pytest.MonkeyPatch, + *, + cpu_values: tuple[float, ...], + rss_values: tuple[int, ...], + tracing: bool, + python_bytes: tuple[int, ...], + strict_tracing: bool = False, +) -> tuple[dict[str, bool], list[str], Any]: + metrics = _metrics_module() + state = {"tracing": tracing} + lifecycle: list[str] = [] + blocker = asyncio.Event() + original_sleep = asyncio.sleep + traced_sizes = iter(python_bytes) + + async def _fake_sleep(_: float) -> None: + await blocker.wait() + + def _start() -> None: + lifecycle.append("start") + state["tracing"] = True + + def _stop() -> None: + lifecycle.append("stop") + state["tracing"] = False + + def _get_traced_memory() -> tuple[int, int]: + if strict_tracing and not state["tracing"]: + raise RuntimeError("tracemalloc not tracing") + return (next(traced_sizes), 0) + + monkeypatch.setattr(metrics.psutil, "Process", lambda: _FakeProcess(cpu_values, rss_values)) + monkeypatch.setattr(metrics.tracemalloc, "is_tracing", lambda: state["tracing"]) + monkeypatch.setattr(metrics.tracemalloc, "start", _start) + monkeypatch.setattr(metrics.tracemalloc, "stop", _stop) + monkeypatch.setattr(metrics.tracemalloc, "get_traced_memory", _get_traced_memory) + monkeypatch.setattr(metrics.asyncio, "sleep", _fake_sleep) + return state, lifecycle, original_sleep + + def run_manifest() -> RunManifest: return RunManifest( profile_id="smoke-1k", @@ -118,6 +180,65 @@ def test_report_counts_api_operations_without_path_loss() -> None: assert api["object_count"] == 1000 +def test_api_summary_does_not_treat_repeated_lists_as_relists() -> None: + recorder = BenchmarkRecorder() + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + + report = recorder.report(run_manifest(), (), final_digest="abc") + + assert report.api.relists == 0 + assert report.api.operations == {"list": 2} + assert report.api.paths == {"/api/v1/pods": {"list": 2}} + + +def test_api_summary_counts_relist_only_after_410_then_list() -> None: + recorder = BenchmarkRecorder() + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=410)) + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + + report = recorder.report(run_manifest(), (), final_digest="abc") + + assert report.api.relists == 1 + assert report.api.operations == {"error": 1, "list": 2} + assert report.api.paths == {"/api/v1/pods": {"error": 1, "list": 2}} + + +def test_api_summary_mappings_are_immutable_and_payloads_are_copied() -> None: + recorder = BenchmarkRecorder() + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + recorder.record_api(ReadTelemetryEvent("watch_open", "/api/v1/pods")) + + report = recorder.report(run_manifest(), (), final_digest="abc") + operations = cast(Any, report.api.operations) + paths = cast(Any, report.api.paths) + + with pytest.raises(TypeError, match="does not support item assignment"): + operations["list"] = 99 + with pytest.raises(TypeError, match="does not support item assignment"): + paths["/api/v1/pods"]["watch_open"] = 99 + + first_payload = report_payload(report) + first_api = cast(dict[str, object], first_payload["api"]) + first_operations = cast(dict[str, int], first_api["operations"]) + first_paths = cast(dict[str, object], first_api["paths"]) + first_pod_path = cast(dict[str, int], first_paths["/api/v1/pods"]) + first_operations["list"] = 41 + first_pod_path["watch_open"] = 42 + + second_payload = report_payload(report) + second_api = cast(dict[str, object], second_payload["api"]) + second_operations = cast(dict[str, int], second_api["operations"]) + second_paths = cast(dict[str, object], second_api["paths"]) + second_pod_path = cast(dict[str, int], second_paths["/api/v1/pods"]) + + assert report.api.operations == {"list": 1, "watch_open": 1} + assert report.api.paths == {"/api/v1/pods": {"list": 1, "watch_open": 1}} + assert second_operations == {"list": 1, "watch_open": 1} + assert second_pod_path == {"list": 1, "watch_open": 1} + + def test_report_payload_is_json_serializable_and_stable() -> None: recorder = BenchmarkRecorder() recorder.record_event(1, 1.0) @@ -206,37 +327,98 @@ def test_render_markdown_uses_stable_labels() -> None: @pytest.mark.asyncio -async def test_process_sampler_skips_warmup_sample( +async def test_process_sampler_rejects_double_start( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _, _, original_sleep = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0, 0.0), + rss_values=(100,), + tracing=True, + python_bytes=(1000,), + ) + + sampler = ProcessSampler(interval_seconds=0.01, clock=lambda: 10.0) + sampler.start() + await original_sleep(0) + try: + with pytest.raises(RuntimeError, match="already running"): + sampler.start() + finally: + await sampler.stop() + + +@pytest.mark.asyncio +async def test_process_sampler_starts_and_stops_owned_tracemalloc( monkeypatch: pytest.MonkeyPatch, ) -> None: - metrics = cast(Any, importlib.import_module("tests.performance.metrics")) + state, lifecycle, original_sleep = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0, 12.5), + rss_values=(100,), + tracing=False, + python_bytes=(1000,), + strict_tracing=True, + ) - class _MemoryInfo: - def __init__(self, rss: int) -> None: - self.rss = rss + sampler = ProcessSampler(interval_seconds=0.01, clock=lambda: 11.0) + sampler.start() + await original_sleep(0) + samples = await sampler.stop() + + assert samples == ( + ProcessSample( + elapsed_seconds=0.0, + cpu_percent=12.5, + rss_bytes=100, + python_bytes=1000, + ), + ) + assert lifecycle == ["start", "stop"] + assert state["tracing"] is False - class _FakeProcess: - def __init__(self) -> None: - self.cpu_values = iter((0.0, 12.5, 15.0)) - self.rss_values = iter((100, 110)) - def cpu_percent(self, interval: float | None = None) -> float: - return next(self.cpu_values) +@pytest.mark.asyncio +async def test_process_sampler_preserves_preexisting_tracemalloc( + monkeypatch: pytest.MonkeyPatch, +) -> None: + state, lifecycle, original_sleep = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0, 8.0), + rss_values=(120,), + tracing=True, + python_bytes=(2000,), + ) - def memory_info(self) -> _MemoryInfo: - return _MemoryInfo(next(self.rss_values)) + sampler = ProcessSampler(interval_seconds=0.01, clock=lambda: 5.0) + sampler.start() + await original_sleep(0) + samples = await sampler.stop() - times = iter((10.0, 11.0)) - python_bytes = iter((1000, 1200)) - blocker = asyncio.Event() - original_sleep = asyncio.sleep + assert samples == ( + ProcessSample( + elapsed_seconds=0.0, + cpu_percent=8.0, + rss_bytes=120, + python_bytes=2000, + ), + ) + assert lifecycle == [] + assert state["tracing"] is True - async def _fake_sleep(_: float) -> None: - await blocker.wait() - monkeypatch.setattr(metrics.psutil, "Process", _FakeProcess) - monkeypatch.setattr(metrics.tracemalloc, "get_traced_memory", lambda: (next(python_bytes), 0)) - monkeypatch.setattr(metrics.asyncio, "sleep", _fake_sleep) +@pytest.mark.asyncio +async def test_process_sampler_skips_warmup_sample( + monkeypatch: pytest.MonkeyPatch, +) -> None: + times = iter((10.0, 11.0)) + _, _, original_sleep = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0, 12.5, 15.0), + rss_values=(100, 110), + tracing=True, + python_bytes=(1000, 1200), + ) sampler = ProcessSampler(interval_seconds=0.01, clock=lambda: next(times)) sampler.start() From 9cf6565935da59113d43f3db3b3d6448eb4114af Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 02:13:44 +0900 Subject: [PATCH 09/38] fix: preserve tracemalloc across overlapping samplers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .superpowers/sdd/task-4-report.md | 325 ++++++++++++++++++++++++++++++ tests/performance/metrics.py | 30 ++- tests/performance/test_metrics.py | 32 +++ 3 files changed, 380 insertions(+), 7 deletions(-) create mode 100644 .superpowers/sdd/task-4-report.md diff --git a/.superpowers/sdd/task-4-report.md b/.superpowers/sdd/task-4-report.md new file mode 100644 index 00000000..5025475b --- /dev/null +++ b/.superpowers/sdd/task-4-report.md @@ -0,0 +1,325 @@ +# Task 4 Report + +## Status +DONE + +## Commit SHA(s) +- `94b2a1e925de8925eecd6c0b33d41e7d6ebe93c7` +- `afb8d4c5d98d297e048824bde7896aa1fc09b83d` + +## Files Changed +- `tests/performance/metrics.py` +- `tests/performance/test_metrics.py` +- `.superpowers/sdd/task-4-report.md` + +## RED + +Command: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q +``` + +Output: +```text +==================================== ERRORS ==================================== +______________ ERROR collecting tests/performance/test_metrics.py ______________ +ImportError while importing test module '/Users/hwang-inhwan/workspace/kube.worktrees/large-cluster-qualification-issue-186/tests/performance/test_metrics.py'. +Hint: make sure your test modules/packages have valid Python names. +Traceback: +../../../.local/share/uv/python/cpython-3.12.13-macos-aarch64-none/lib/python3.12/importlib/__init__.py:90: in import_module + return _bootstrap._gcd_import(name[level:], package, level) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +tests/performance/test_metrics.py:9: in + from tests.performance.metrics import ( +E ModuleNotFoundError: No module named 'tests.performance.metrics' +=========================== short test summary info ============================ +ERROR tests/performance/test_metrics.py +!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! +1 error in 0.29s +``` + +## GREEN / Validation + +Command: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q +uv run ruff check --fix tests/performance/metrics.py tests/performance/test_metrics.py +uv run ruff format tests/performance/metrics.py tests/performance/test_metrics.py +uv run mypy tests/performance/metrics.py tests/performance/test_metrics.py +``` + +Output: +```text +........ [100%] +8 passed in 0.07s +All checks passed! +2 files left unchanged +Success: no issues found in 2 source files +``` + +Additional targeted validation: + +Command: +```bash +uv run pytest -p no:tach tests/performance/test_profile.py tests/performance/test_workload.py -q +``` + +Output: +```text +.......... [100%] +10 passed in 1.16s +``` + +Command: +```bash +git diff --check +``` + +Output: +```text +``` + +## Self-Review +- Reused `ReadTelemetryEvent` exactly as requested for API accounting. +- Kept the change isolated to the new Task 4 metrics module and its tests. +- Verified nearest-rank percentile semantics, coalescing/dropped update accounting, API path preservation, least-squares RSS slope, stable JSON shape, Markdown labels, and `ProcessSampler` warm-up behavior. +- Used frozen dataclasses for published values and kept mutable collection state inside `BenchmarkRecorder`. + +## Concerns +- `psutil` does not ship typing stubs in this environment, so `tests/performance/metrics.py` uses an explicit `# type: ignore[import-untyped]` with a reason to satisfy strict mypy while preserving the required dependency. + +## Fix Review Findings + +### Finding 1: Immutable published API mappings and copied JSON payloads + +RED command: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q -k immutable_and_payloads_are_copied +``` + +RED output: +```text +F [100%] +=================================== FAILURES =================================== +_______ test_api_summary_mappings_are_immutable_and_payloads_are_copied ________ +tests/performance/test_metrics.py:130: in test_api_summary_mappings_are_immutable_and_payloads_are_copied + with pytest.raises(TypeError, match="does not support item assignment"): + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E Failed: DID NOT RAISE TypeError +=========================== short test summary info ============================ +FAILED tests/performance/test_metrics.py::test_api_summary_mappings_are_immutable_and_payloads_are_copied +1 failed, 8 deselected in 0.06s +``` + +GREEN commands: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q -k immutable_and_payloads_are_copied +uv run pytest -p no:tach tests/performance/test_metrics.py -q +``` + +GREEN output: +```text +. [100%] +1 passed, 8 deselected in 0.06s + +......... [100%] +9 passed in 0.07s +``` + +### Finding 2: Reject ProcessSampler double-start + +RED command: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q -k rejects_double_start +``` + +RED output: +```text +F [100%] +=================================== FAILURES =================================== +__________________ test_process_sampler_rejects_double_start ___________________ +tests/performance/test_metrics.py:274: in test_process_sampler_rejects_double_start + with pytest.raises(RuntimeError, match="already running"): + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E Failed: DID NOT RAISE RuntimeError +=========================== short test summary info ============================ +FAILED tests/performance/test_metrics.py::test_process_sampler_rejects_double_start +1 failed, 9 deselected in 0.07s +``` + +GREEN commands: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q -k rejects_double_start +uv run pytest -p no:tach tests/performance/test_metrics.py -q +``` + +GREEN output: +```text +. [100%] +1 passed, 9 deselected in 0.05s + +.......... [100%] +10 passed in 0.07s +``` + +### Finding 3: Start/own tracemalloc only when needed + +RED command: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q -k tracemalloc +``` + +RED output: +```text +.F [100%] +=================================== FAILURES =================================== +___________ test_process_sampler_starts_and_stops_owned_tracemalloc ____________ +tests/performance/test_metrics.py:335: in test_process_sampler_starts_and_stops_owned_tracemalloc + samples = await sampler.stop() + ^^^^^^^^^^^^^^^^^^^^ +tests/performance/metrics.py:210: in stop + await task +tests/performance/metrics.py:221: in _run + python_bytes=tracemalloc.get_traced_memory()[0], + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +tests/performance/test_metrics.py:322: in _get_traced_memory + raise RuntimeError("tracemalloc not tracing") +E RuntimeError: tracemalloc not tracing +=========================== short test summary info ============================ +FAILED tests/performance/test_metrics.py::test_process_sampler_starts_and_stops_owned_tracemalloc +1 failed, 1 passed, 10 deselected in 0.08s +``` + +GREEN commands: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q -k tracemalloc +uv run pytest -p no:tach tests/performance/test_metrics.py -q +``` + +GREEN output: +```text +.. [100%] +2 passed, 10 deselected in 0.05s + +............ [100%] +12 passed in 0.07s +``` + +### Finding 4: Count relists only after a 410 recovery + +RED command: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q -k relist +``` + +RED output: +```text +.F [100%] +=================================== FAILURES =================================== +__________ test_api_summary_does_not_treat_repeated_lists_as_relists ___________ +tests/performance/test_metrics.py:128: in test_api_summary_does_not_treat_repeated_lists_as_relists + assert report.api.relists == 0 +E AssertionError: assert 1 == 0 +E + where 1 = ApiSummary(operations=mappingproxy({'list': 2}), paths=mappingproxy({'/api/v1/pods': mappingproxy({'list': 2})}), decoded_bytes=0, object_count=0, watch_events=0, reconnects=0, relists=1, throttles=0, authorization_failures=0).relists +E + where ApiSummary(operations=mappingproxy({'list': 2}), paths=mappingproxy({'/api/v1/pods': mappingproxy({'list': 2})}), decoded_bytes=0, object_count=0, watch_events=0, reconnects=0, relists=1, throttles=0, authorization_failures=0) = BenchmarkReport(manifest=RunManifest(profile_id='smoke-1k', profile_hash='profile-hash', korvid_sha='3cbe600996043cd6c...orization_failures=0), rendered_updates=0, render_passes=0, coalesced_updates=0, dropped_updates=0, final_digest='abc').api +=========================== short test summary info ============================ +FAILED tests/performance/test_metrics.py::test_api_summary_does_not_treat_repeated_lists_as_relists +1 failed, 1 passed, 12 deselected in 0.07s +``` + +GREEN commands: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q -k relist +uv run pytest -p no:tach tests/performance/test_metrics.py -q +``` + +GREEN output: +```text +.. [100%] +2 passed, 12 deselected in 0.05s + +.............. [100%] +14 passed in 0.07s +``` + +### Final validation after review fixes + +Commands: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q +uv run ruff check --fix tests/performance/metrics.py tests/performance/test_metrics.py +uv run ruff format tests/performance/metrics.py tests/performance/test_metrics.py +uv run mypy tests/performance/metrics.py tests/performance/test_metrics.py +``` + +Output: +```text +.............. [100%] +14 passed in 0.12s +All checks passed! +2 files left unchanged +Success: no issues found in 2 source files +``` + +### Self-Review for Review Fixes +- Published `ApiSummary` collections now expose immutable mappings, while `report_payload()` deep-copies them into fresh JSON-ready dicts. +- `ProcessSampler.start()` now fails deterministically on double-start and tracks tracemalloc ownership so Task 6 snapshots remain intact when tracing was already active. +- Relist counting now follows the design requirement: only a later `list` after a same-path status-410 event increments `relists`. +- The fix stayed isolated to the Task 4 metrics module, Task 4 tests, and this durable report. + +### Finding 5: Keep process-global tracemalloc active across overlapping samplers + +RED command: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q -k keeps_owned_tracemalloc_until_last_overlapping_sampler_stops +``` + +RED output: +```text +F [100%] +=================================== FAILURES =================================== +_ test_process_sampler_keeps_owned_tracemalloc_until_last_overlapping_sampler_stops _ +tests/performance/test_metrics.py:374: in test_process_sampler_keeps_owned_tracemalloc_until_last_overlapping_sampler_stops + assert lifecycle == ["start"] +E AssertionError: assert ['start', 'stop'] == ['start'] +E +E Left contains one more item: 'stop' +E Use -v to get more diff +=========================== short test summary info ============================ +FAILED tests/performance/test_metrics.py::test_process_sampler_keeps_owned_tracemalloc_until_last_overlapping_sampler_stops +1 failed, 14 deselected in 0.07s +``` + +GREEN command: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q -k keeps_owned_tracemalloc_until_last_overlapping_sampler_stops +``` + +GREEN output: +```text +. [100%] +1 passed, 14 deselected in 0.05s +``` + +Final validation commands: +```bash +uv run pytest -p no:tach tests/performance/test_metrics.py -q +uv run ruff check --fix tests/performance/metrics.py tests/performance/test_metrics.py +uv run ruff format tests/performance/metrics.py tests/performance/test_metrics.py +uv run mypy tests/performance/metrics.py tests/performance/test_metrics.py +git diff --check +``` + +Final validation output: +```text +............... [100%] +15 passed in 0.04s +All checks passed! +2 files left unchanged +Success: no issues found in 2 source files +``` + +Self-review: +- Added one focused async regression test that proves overlapping samplers must not stop process-global tracing until the last internally-managed sampler exits. +- Replaced per-instance tracemalloc ownership with a single class-level managed-user count so internally-started tracing stays alive across overlap, while externally-started tracing still remains untouched. +- Kept double-start rejection unchanged and limited production edits to `ProcessSampler`. diff --git a/tests/performance/metrics.py b/tests/performance/metrics.py index eaf2cac5..9a1305b2 100644 --- a/tests/performance/metrics.py +++ b/tests/performance/metrics.py @@ -190,6 +190,8 @@ class BenchmarkReport: class ProcessSampler: + _managed_tracemalloc_users = 0 + def __init__(self, interval_seconds: float, clock: Callable[[], float] = monotonic) -> None: self._interval_seconds = interval_seconds self._clock = clock @@ -197,7 +199,7 @@ def __init__(self, interval_seconds: float, clock: Callable[[], float] = monoton self._start_time: float | None = None self._samples: list[ProcessSample] = [] self._task: asyncio.Task[None] | None = None - self._owns_tracemalloc = False + self._uses_managed_tracing = False def start(self) -> None: if self._task is not None: @@ -206,9 +208,7 @@ def start(self) -> None: ) self._samples.clear() self._start_time = self._clock() - self._owns_tracemalloc = not tracemalloc.is_tracing() - if self._owns_tracemalloc: - tracemalloc.start() + self._uses_managed_tracing = self._acquire_tracemalloc() self._process.cpu_percent() self._task = asyncio.create_task(self._run()) @@ -220,11 +220,27 @@ async def stop(self) -> tuple[ProcessSample, ...]: task.cancel() with suppress(asyncio.CancelledError): await task - if self._owns_tracemalloc and tracemalloc.is_tracing(): - tracemalloc.stop() - self._owns_tracemalloc = False + if self._uses_managed_tracing: + self._release_tracemalloc() + self._uses_managed_tracing = False return tuple(self._samples) + @classmethod + def _acquire_tracemalloc(cls) -> bool: + if tracemalloc.is_tracing(): + if cls._managed_tracemalloc_users == 0: + return False + else: + tracemalloc.start() + cls._managed_tracemalloc_users += 1 + return True + + @classmethod + def _release_tracemalloc(cls) -> None: + cls._managed_tracemalloc_users -= 1 + if cls._managed_tracemalloc_users == 0 and tracemalloc.is_tracing(): + tracemalloc.stop() + async def _run(self) -> None: assert self._start_time is not None while True: diff --git a/tests/performance/test_metrics.py b/tests/performance/test_metrics.py index 12b1e90a..995a04b3 100644 --- a/tests/performance/test_metrics.py +++ b/tests/performance/test_metrics.py @@ -348,6 +348,38 @@ async def test_process_sampler_rejects_double_start( await sampler.stop() +@pytest.mark.asyncio +async def test_process_sampler_keeps_owned_tracemalloc_until_last_overlapping_sampler_stops( + monkeypatch: pytest.MonkeyPatch, +) -> None: + state, lifecycle, original_sleep = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0, 12.5), + rss_values=(100,), + tracing=False, + python_bytes=(1000, 2000), + strict_tracing=True, + ) + + first = ProcessSampler(interval_seconds=0.01, clock=lambda: 11.0) + second = ProcessSampler(interval_seconds=0.01, clock=lambda: 12.0) + + first.start() + await original_sleep(0) + second.start() + await original_sleep(0) + + await first.stop() + + assert lifecycle == ["start"] + assert state["tracing"] is True + + await second.stop() + + assert lifecycle == ["start", "stop"] + assert state["tracing"] is False + + @pytest.mark.asyncio async def test_process_sampler_starts_and_stops_owned_tracemalloc( monkeypatch: pytest.MonkeyPatch, From fbf6a3aa294dd4c08befa9c1f10ae77a78d647fb Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 02:22:13 +0900 Subject: [PATCH 10/38] fix: rollback process sampler startup failures Restore tracemalloc ownership and close the unstarted coroutine if startup fails after ownership acquisition. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .superpowers/sdd/task-4-report.md | 334 ++---------------------------- tests/performance/metrics.py | 17 +- tests/performance/test_metrics.py | 26 +++ 3 files changed, 57 insertions(+), 320 deletions(-) diff --git a/.superpowers/sdd/task-4-report.md b/.superpowers/sdd/task-4-report.md index 5025475b..61190476 100644 --- a/.superpowers/sdd/task-4-report.md +++ b/.superpowers/sdd/task-4-report.md @@ -1,325 +1,25 @@ -# Task 4 Report - -## Status -DONE - -## Commit SHA(s) -- `94b2a1e925de8925eecd6c0b33d41e7d6ebe93c7` -- `afb8d4c5d98d297e048824bde7896aa1fc09b83d` - -## Files Changed -- `tests/performance/metrics.py` -- `tests/performance/test_metrics.py` -- `.superpowers/sdd/task-4-report.md` - -## RED +# Task 4 report +## RED evidence Command: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -``` +`uv run pytest -p no:tach tests/performance/test_metrics.py -k rolls_back_tracemalloc_if_task_creation_fails -q` -Output: -```text -==================================== ERRORS ==================================== -______________ ERROR collecting tests/performance/test_metrics.py ______________ -ImportError while importing test module '/Users/hwang-inhwan/workspace/kube.worktrees/large-cluster-qualification-issue-186/tests/performance/test_metrics.py'. -Hint: make sure your test modules/packages have valid Python names. -Traceback: -../../../.local/share/uv/python/cpython-3.12.13-macos-aarch64-none/lib/python3.12/importlib/__init__.py:90: in import_module - return _bootstrap._gcd_import(name[level:], package, level) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -tests/performance/test_metrics.py:9: in - from tests.performance.metrics import ( -E ModuleNotFoundError: No module named 'tests.performance.metrics' -=========================== short test summary info ============================ -ERROR tests/performance/test_metrics.py -!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! -1 error in 0.29s -``` - -## GREEN / Validation +Observed failure: +- `AssertionError: assert ['start'] == ['start', 'stop']` +- `pytest.PytestUnraisableExceptionWarning: Exception ignored in: ` +## GREEN evidence Command: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -uv run ruff check --fix tests/performance/metrics.py tests/performance/test_metrics.py -uv run ruff format tests/performance/metrics.py tests/performance/test_metrics.py -uv run mypy tests/performance/metrics.py tests/performance/test_metrics.py -``` - -Output: -```text -........ [100%] -8 passed in 0.07s -All checks passed! -2 files left unchanged -Success: no issues found in 2 source files -``` - -Additional targeted validation: - -Command: -```bash -uv run pytest -p no:tach tests/performance/test_profile.py tests/performance/test_workload.py -q -``` - -Output: -```text -.......... [100%] -10 passed in 1.16s -``` - -Command: -```bash -git diff --check -``` - -Output: -```text -``` - -## Self-Review -- Reused `ReadTelemetryEvent` exactly as requested for API accounting. -- Kept the change isolated to the new Task 4 metrics module and its tests. -- Verified nearest-rank percentile semantics, coalescing/dropped update accounting, API path preservation, least-squares RSS slope, stable JSON shape, Markdown labels, and `ProcessSampler` warm-up behavior. -- Used frozen dataclasses for published values and kept mutable collection state inside `BenchmarkRecorder`. - -## Concerns -- `psutil` does not ship typing stubs in this environment, so `tests/performance/metrics.py` uses an explicit `# type: ignore[import-untyped]` with a reason to satisfy strict mypy while preserving the required dependency. - -## Fix Review Findings - -### Finding 1: Immutable published API mappings and copied JSON payloads - -RED command: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -k immutable_and_payloads_are_copied -``` - -RED output: -```text -F [100%] -=================================== FAILURES =================================== -_______ test_api_summary_mappings_are_immutable_and_payloads_are_copied ________ -tests/performance/test_metrics.py:130: in test_api_summary_mappings_are_immutable_and_payloads_are_copied - with pytest.raises(TypeError, match="does not support item assignment"): - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -E Failed: DID NOT RAISE TypeError -=========================== short test summary info ============================ -FAILED tests/performance/test_metrics.py::test_api_summary_mappings_are_immutable_and_payloads_are_copied -1 failed, 8 deselected in 0.06s -``` - -GREEN commands: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -k immutable_and_payloads_are_copied -uv run pytest -p no:tach tests/performance/test_metrics.py -q -``` - -GREEN output: -```text -. [100%] -1 passed, 8 deselected in 0.06s - -......... [100%] -9 passed in 0.07s -``` - -### Finding 2: Reject ProcessSampler double-start - -RED command: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -k rejects_double_start -``` - -RED output: -```text -F [100%] -=================================== FAILURES =================================== -__________________ test_process_sampler_rejects_double_start ___________________ -tests/performance/test_metrics.py:274: in test_process_sampler_rejects_double_start - with pytest.raises(RuntimeError, match="already running"): - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -E Failed: DID NOT RAISE RuntimeError -=========================== short test summary info ============================ -FAILED tests/performance/test_metrics.py::test_process_sampler_rejects_double_start -1 failed, 9 deselected in 0.07s -``` - -GREEN commands: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -k rejects_double_start -uv run pytest -p no:tach tests/performance/test_metrics.py -q -``` - -GREEN output: -```text -. [100%] -1 passed, 9 deselected in 0.05s - -.......... [100%] -10 passed in 0.07s -``` - -### Finding 3: Start/own tracemalloc only when needed - -RED command: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -k tracemalloc -``` - -RED output: -```text -.F [100%] -=================================== FAILURES =================================== -___________ test_process_sampler_starts_and_stops_owned_tracemalloc ____________ -tests/performance/test_metrics.py:335: in test_process_sampler_starts_and_stops_owned_tracemalloc - samples = await sampler.stop() - ^^^^^^^^^^^^^^^^^^^^ -tests/performance/metrics.py:210: in stop - await task -tests/performance/metrics.py:221: in _run - python_bytes=tracemalloc.get_traced_memory()[0], - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -tests/performance/test_metrics.py:322: in _get_traced_memory - raise RuntimeError("tracemalloc not tracing") -E RuntimeError: tracemalloc not tracing -=========================== short test summary info ============================ -FAILED tests/performance/test_metrics.py::test_process_sampler_starts_and_stops_owned_tracemalloc -1 failed, 1 passed, 10 deselected in 0.08s -``` - -GREEN commands: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -k tracemalloc -uv run pytest -p no:tach tests/performance/test_metrics.py -q -``` - -GREEN output: -```text -.. [100%] -2 passed, 10 deselected in 0.05s - -............ [100%] -12 passed in 0.07s -``` - -### Finding 4: Count relists only after a 410 recovery - -RED command: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -k relist -``` - -RED output: -```text -.F [100%] -=================================== FAILURES =================================== -__________ test_api_summary_does_not_treat_repeated_lists_as_relists ___________ -tests/performance/test_metrics.py:128: in test_api_summary_does_not_treat_repeated_lists_as_relists - assert report.api.relists == 0 -E AssertionError: assert 1 == 0 -E + where 1 = ApiSummary(operations=mappingproxy({'list': 2}), paths=mappingproxy({'/api/v1/pods': mappingproxy({'list': 2})}), decoded_bytes=0, object_count=0, watch_events=0, reconnects=0, relists=1, throttles=0, authorization_failures=0).relists -E + where ApiSummary(operations=mappingproxy({'list': 2}), paths=mappingproxy({'/api/v1/pods': mappingproxy({'list': 2})}), decoded_bytes=0, object_count=0, watch_events=0, reconnects=0, relists=1, throttles=0, authorization_failures=0) = BenchmarkReport(manifest=RunManifest(profile_id='smoke-1k', profile_hash='profile-hash', korvid_sha='3cbe600996043cd6c...orization_failures=0), rendered_updates=0, render_passes=0, coalesced_updates=0, dropped_updates=0, final_digest='abc').api -=========================== short test summary info ============================ -FAILED tests/performance/test_metrics.py::test_api_summary_does_not_treat_repeated_lists_as_relists -1 failed, 1 passed, 12 deselected in 0.07s -``` - -GREEN commands: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -k relist -uv run pytest -p no:tach tests/performance/test_metrics.py -q -``` - -GREEN output: -```text -.. [100%] -2 passed, 12 deselected in 0.05s - -.............. [100%] -14 passed in 0.07s -``` - -### Final validation after review fixes - -Commands: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -uv run ruff check --fix tests/performance/metrics.py tests/performance/test_metrics.py -uv run ruff format tests/performance/metrics.py tests/performance/test_metrics.py -uv run mypy tests/performance/metrics.py tests/performance/test_metrics.py -``` - -Output: -```text -.............. [100%] -14 passed in 0.12s -All checks passed! -2 files left unchanged -Success: no issues found in 2 source files -``` - -### Self-Review for Review Fixes -- Published `ApiSummary` collections now expose immutable mappings, while `report_payload()` deep-copies them into fresh JSON-ready dicts. -- `ProcessSampler.start()` now fails deterministically on double-start and tracks tracemalloc ownership so Task 6 snapshots remain intact when tracing was already active. -- Relist counting now follows the design requirement: only a later `list` after a same-path status-410 event increments `relists`. -- The fix stayed isolated to the Task 4 metrics module, Task 4 tests, and this durable report. - -### Finding 5: Keep process-global tracemalloc active across overlapping samplers - -RED command: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -k keeps_owned_tracemalloc_until_last_overlapping_sampler_stops -``` - -RED output: -```text -F [100%] -=================================== FAILURES =================================== -_ test_process_sampler_keeps_owned_tracemalloc_until_last_overlapping_sampler_stops _ -tests/performance/test_metrics.py:374: in test_process_sampler_keeps_owned_tracemalloc_until_last_overlapping_sampler_stops - assert lifecycle == ["start"] -E AssertionError: assert ['start', 'stop'] == ['start'] -E -E Left contains one more item: 'stop' -E Use -v to get more diff -=========================== short test summary info ============================ -FAILED tests/performance/test_metrics.py::test_process_sampler_keeps_owned_tracemalloc_until_last_overlapping_sampler_stops -1 failed, 14 deselected in 0.07s -``` - -GREEN command: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -k keeps_owned_tracemalloc_until_last_overlapping_sampler_stops -``` - -GREEN output: -```text -. [100%] -1 passed, 14 deselected in 0.05s -``` +`uv run pytest -p no:tach tests/performance/test_metrics.py -k rolls_back_tracemalloc_if_task_creation_fails -q` -Final validation commands: -```bash -uv run pytest -p no:tach tests/performance/test_metrics.py -q -uv run ruff check --fix tests/performance/metrics.py tests/performance/test_metrics.py -uv run ruff format tests/performance/metrics.py tests/performance/test_metrics.py -uv run mypy tests/performance/metrics.py tests/performance/test_metrics.py -git diff --check -``` +Observed success: +- `1 passed, 15 deselected in 0.05s` -Final validation output: -```text -............... [100%] -15 passed in 0.04s -All checks passed! -2 files left unchanged -Success: no issues found in 2 source files -``` +Full file verification: +- `16 passed in 0.04s` -Self-review: -- Added one focused async regression test that proves overlapping samplers must not stop process-global tracing until the last internally-managed sampler exits. -- Replaced per-instance tracemalloc ownership with a single class-level managed-user count so internally-started tracing stays alive across overlap, while externally-started tracing still remains untouched. -- Kept double-start rejection unchanged and limited production edits to `ProcessSampler`. +## Self-review +- Added the smallest regression test for start-up rollback when `asyncio.create_task()` fails after tracemalloc ownership is acquired. +- Fixed `ProcessSampler.start()` with scoped rollback that releases owned tracemalloc, resets sampler state, and closes the unstarted coroutine before re-raising. +- Preserved double-start rejection, overlapping sampler ownership, and externally-owned tracemalloc behavior. +- Verified with focused test, full `tests/performance/test_metrics.py`, ruff, mypy, and `git diff --check`. diff --git a/tests/performance/metrics.py b/tests/performance/metrics.py index 9a1305b2..459659a8 100644 --- a/tests/performance/metrics.py +++ b/tests/performance/metrics.py @@ -4,7 +4,7 @@ import math import tracemalloc from collections import Counter -from collections.abc import Callable, Mapping, Sequence +from collections.abc import Callable, Coroutine, Mapping, Sequence from contextlib import suppress from dataclasses import dataclass from time import monotonic @@ -209,8 +209,19 @@ def start(self) -> None: self._samples.clear() self._start_time = self._clock() self._uses_managed_tracing = self._acquire_tracemalloc() - self._process.cpu_percent() - self._task = asyncio.create_task(self._run()) + task: Coroutine[object, object, None] | None = None + try: + self._process.cpu_percent() + task = self._run() + self._task = asyncio.create_task(task) + except BaseException: + if task is not None: + task.close() + if self._uses_managed_tracing: + self._release_tracemalloc() + self._uses_managed_tracing = False + self._start_time = None + raise async def stop(self) -> tuple[ProcessSample, ...]: if self._task is None: diff --git a/tests/performance/test_metrics.py b/tests/performance/test_metrics.py index 995a04b3..0cdbf3c0 100644 --- a/tests/performance/test_metrics.py +++ b/tests/performance/test_metrics.py @@ -348,6 +348,32 @@ async def test_process_sampler_rejects_double_start( await sampler.stop() +@pytest.mark.asyncio +async def test_process_sampler_rolls_back_tracemalloc_if_task_creation_fails( + monkeypatch: pytest.MonkeyPatch, +) -> None: + state, lifecycle, _ = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0,), + rss_values=(100,), + tracing=False, + python_bytes=(1000,), + ) + + def _fail_create_task(_: object) -> object: + raise RuntimeError("task creation failed") + + monkeypatch.setattr(_metrics_module().asyncio, "create_task", _fail_create_task) + + sampler = ProcessSampler(interval_seconds=0.01, clock=lambda: 10.0) + + with pytest.raises(RuntimeError, match="task creation failed"): + sampler.start() + + assert lifecycle == ["start", "stop"] + assert state["tracing"] is False + + @pytest.mark.asyncio async def test_process_sampler_keeps_owned_tracemalloc_until_last_overlapping_sampler_stops( monkeypatch: pytest.MonkeyPatch, From 10816f51aa9a7a811592be2bcfa87be71819afde Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 02:49:39 +0900 Subject: [PATCH 11/38] test: replay scale traffic through the real Textual app Exercise the production watch, store, render, and table path for issue #186. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/replay.py | 334 +++++++++++++++++++++++++++++++ tests/performance/test_replay.py | 56 ++++++ 2 files changed, 390 insertions(+) create mode 100644 tests/performance/replay.py create mode 100644 tests/performance/test_replay.py diff --git a/tests/performance/replay.py b/tests/performance/replay.py new file mode 100644 index 00000000..0b019391 --- /dev/null +++ b/tests/performance/replay.py @@ -0,0 +1,334 @@ +"""Real-app Textual replay harness for large-cluster qualification (issue #186). + +Drives production KorvidApp, WatchManager, ResourceStore, and ResourceTable +through a recorded WorkloadProfile and captures replay metrics. This module +is test-only instrumentation; it must not modify any production source file. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import os +import platform +import sys +from collections.abc import AsyncIterator, Iterable +from dataclasses import asdict, dataclass +from time import monotonic +from typing import Any, cast + +import psutil # type: ignore[import-untyped] # no inline stubs shipped +from textual import __version__ as _textual_version + +from korvid.core.config import KorvidConfig +from korvid.core.store import ALL_NAMESPACES, ResourceStore, Summary +from korvid.core.watch import WatchManager +from korvid.k8s.errors import ApiStatusError +from korvid.k8s.models import PodSummary +from korvid.k8s.telemetry import ReadTelemetryEvent +from korvid.ui.app import KorvidApp, PaneState +from korvid.ui.widgets.resource_table import ResourceTable +from tests.performance.metrics import ( + ApiSummary, + BenchmarkRecorder, + LatencySummary, + ProcessSampler, + ProcessSummary, + RunManifest, +) +from tests.performance.profile import FailureInjection, WorkloadProfile +from tests.performance.workload import ( + ScheduledEvent, + initial_pods, + scheduled_events, + summary_digest, +) +from tests.ui.waits import until + + +@dataclass(frozen=True) +class ReplayOptions: + """Tuning knobs for a replay run. + + Args: + time_scale: Multiplier applied to every scheduled-event sleep. + 0 skips all sleeps (fastest); 1.0 replays at real time. + sample_interval: Seconds between process-memory samples. + """ + + time_scale: float = 1.0 + sample_interval: float = 1.0 + + +@dataclass(frozen=True) +class ReplayReport: + """Metrics collected by a complete real-app replay run. + + Extends the fields of `BenchmarkReport` with replay-specific metadata + (`object_count`, `expected_digest`) so tests can assert both performance + counters and digest correctness in one object. + """ + + object_count: int + expected_digest: str + final_digest: str + dropped_updates: int + rendered_updates: int + render_passes: int + coalesced_updates: int + event_to_render: LatencySummary + input_latency: LatencySummary + process: ProcessSummary + api: ApiSummary + manifest: RunManifest + + +class MeasuredKorvidApp(KorvidApp): + """KorvidApp subclass that hooks `_render_table` to record render timing.""" + + def __init__(self, *args: Any, recorder: BenchmarkRecorder, **kwargs: Any) -> None: + super().__init__(*args, **kwargs) + self._benchmark_recorder = recorder + + def _render_table(self, kind: str, *, only: PaneState | None = None) -> None: + super()._render_table(kind, only=only) + self._benchmark_recorder.record_render(monotonic()) + + +# Maps FailureInjection.kind to the HTTP status code it raises. +_HARD_FAILURE_STATUS: dict[str, int] = { + "gone": 410, + "throttled": 429, + "forbidden": 403, +} + + +class _ReplaySource: + """Stateful async watch source that replays a `WorkloadProfile`. + + Each call (each watch connection) advances through the pre-computed event + stream. On reconnect the source re-LISTs from its tracked current state + so the `WatchManager`'s store-clear + reseed cycle produces the correct + snapshot. + """ + + def __init__( + self, + profile: WorkloadProfile, + events: tuple[ScheduledEvent, ...], + options: ReplayOptions, + recorder: BenchmarkRecorder, + churn_ready: asyncio.Event, + churn_start: asyncio.Event, + churn_done: asyncio.Event, + failures: dict[int, FailureInjection], + ) -> None: + self._profile = profile + self._events = events + self._options = options + self._recorder = recorder + self._churn_ready = churn_ready + self._churn_start = churn_start + self._churn_done = churn_done + self._failures = failures + self._generation = 0 + self._next_event_index = 0 + self._current: dict[str, PodSummary] = { + f"{p.namespace}/{p.name}": p for p in initial_pods(profile) + } + + def current_digest(self) -> str: + """Digest of the source's tracked expected state.""" + return summary_digest(self._current.values()) + + async def _handle_failure_if_any(self, event: ScheduledEvent, index: int) -> None: + """Apply failure injection for *event*; raises `ApiStatusError` for hard faults. + + `slow` delays the tick without dropping the event. Hard faults (gone, + throttled, forbidden) record an error telemetry entry, advance the + next-event cursor, and raise so `WatchManager` triggers a reconnect. + """ + failure = self._failures.get(event.sequence) + if failure is None: + return + if failure.kind == "slow": + tick = 1.0 / max(self._profile.steady_events_per_second, 1) + if tick * self._options.time_scale > 0: + await asyncio.sleep(tick * self._options.time_scale) + return + status = _HARD_FAILURE_STATUS[failure.kind] + self._recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=status)) + self._next_event_index = index + 1 + raise ApiStatusError(status, failure.kind) + + async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summary]]: + gen = self._generation + self._generation += 1 + + # --- LIST phase --- + if gen == 0: + list_pods: list[PodSummary] = list(initial_pods(self._profile)) + else: + list_pods = sorted(self._current.values(), key=lambda p: (p.namespace, p.name)) + + self._recorder.record_api( + ReadTelemetryEvent( + "list", + "/api/v1/pods", + object_count=len(list_pods), + decoded_bytes=len(list_pods) * 200, + ) + ) + for pod in list_pods: + self._recorder.record_event(0, monotonic()) + yield ("ADDED", pod) + + self._recorder.record_api(ReadTelemetryEvent("watch_open", "/api/v1/pods")) + + if gen == 0: + # Pause here until run_replay confirms the table is populated. + self._churn_ready.set() + await self._churn_start.wait() + + # --- WATCH phase --- + for i in range(self._next_event_index, len(self._events)): + event = self._events[i] + delay = event.offset_seconds * self._options.time_scale + if delay > 0: + await asyncio.sleep(delay) + + await self._handle_failure_if_any(event, i) + + key = f"{event.summary.namespace}/{event.summary.name}" + if event.event_type == "DELETED": + self._current.pop(key, None) + else: + self._current[key] = event.summary + + self._recorder.record_event(event.sequence, monotonic()) + yield (event.event_type, event.summary) + self._next_event_index = i + 1 + + self._churn_done.set() + # Stay open like a real watch stream so WatchManager does not reconnect. + while True: + await asyncio.sleep(3600.0) + + +def _build_manifest(profile: WorkloadProfile) -> RunManifest: + profile_hash = hashlib.sha256( + json.dumps(asdict(profile), sort_keys=True, separators=(",", ":"), default=str).encode() + ).hexdigest() + return RunManifest( + profile_id=profile.id, + profile_hash=profile_hash, + korvid_sha="dev", + python=sys.version, + textual=_textual_version, + os=platform.platform(), + cpu_count=os.cpu_count() or 1, + memory_bytes=psutil.virtual_memory().total, + ) + + +async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> ReplayReport: + """Run the full production Textual app against *profile* and return metrics. + + The function: + 1. Creates `ResourceStore`, `BenchmarkRecorder`, and one `_ReplaySource`. + 2. Emits initial Pods as ``ADDED`` (one logical LIST + WATCH OPEN). + 3. Waits on `churn_ready` after initial population. + 4. Waits until the `ResourceTable` shows all objects. + 5. Records key-press input latency. + 6. Signals the source to start scheduled-event churn. + 7. Awaits churn completion and the final render pass. + 8. Stops the watch manager and process sampler in `finally`. + 9. Compares the table/store digest to the source's expected state. + """ + events = scheduled_events(profile) + failures: dict[int, FailureInjection] = {f.at_event: f for f in profile.failures} + + store = ResourceStore() + recorder = BenchmarkRecorder() + sampler = ProcessSampler(options.sample_interval) + + churn_ready = asyncio.Event() + churn_start = asyncio.Event() + churn_done = asyncio.Event() + + source = _ReplaySource( + profile, + events, + options, + recorder, + churn_ready, + churn_start, + churn_done, + failures, + ) + watch_manager = WatchManager(store, source, retry_delay=0.0) + manifest = _build_manifest(profile) + + app = MeasuredKorvidApp( + config=KorvidConfig(namespace=ALL_NAMESPACES), + store=store, + watch_manager=watch_manager, + recorder=recorder, + ) + + sampler.start() + try: + async with app.run_test() as pilot: + table = app.query_one(ResourceTable) + + # Wait for the initial LIST to populate the table. + await until( + pilot, + lambda: table.row_count == profile.object_count, + timeout=30.0, + label="initial pods rendered", + ) + + # Record cursor key latency before churn starts. + t0 = monotonic() + await pilot.press("down") + recorder.record_input(monotonic() - t0) + t0 = monotonic() + await pilot.press("up") + recorder.record_input(monotonic() - t0) + + # Release the source to emit scheduled events. + churn_start.set() + + # Wait for all events to be emitted and all renders to complete. + await until( + pilot, + lambda: churn_done.is_set() and not recorder._pending_events, + timeout=30.0, + label="churn complete and all events rendered", + ) + finally: + process_samples = await sampler.stop() + await watch_manager.stop_all() + + # Compute digests: source tracks expected state; store reflects actual state. + final_digest = summary_digest(cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES))) + expected_digest = source.current_digest() + + benchmark = recorder.report(manifest, process_samples, final_digest=final_digest) + + return ReplayReport( + object_count=profile.object_count, + expected_digest=expected_digest, + final_digest=final_digest, + dropped_updates=benchmark.dropped_updates, + rendered_updates=benchmark.rendered_updates, + render_passes=benchmark.render_passes, + coalesced_updates=benchmark.coalesced_updates, + event_to_render=benchmark.event_to_render, + input_latency=benchmark.input_latency, + process=benchmark.process, + api=benchmark.api, + manifest=benchmark.manifest, + ) diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py new file mode 100644 index 00000000..e1ae57c3 --- /dev/null +++ b/tests/performance/test_replay.py @@ -0,0 +1,56 @@ +"""Real-app replay harness tests (Task 5, issue #186). + +Drives the production KorvidApp/WatchManager/ResourceStore/ResourceTable +stack with a synthetic WorkloadProfile and asserts digest correctness, +update accounting, and API telemetry. +""" + +from __future__ import annotations + +from tests.performance.profile import FailureInjection, WorkloadProfile +from tests.performance.replay import ReplayOptions, run_replay + + +async def test_replay_uses_real_app_and_reaches_expected_digest() -> None: + profile = WorkloadProfile( + schema_version=1, + id="test", + seed=186, + object_count=100, + namespace_count=10, + steady_events_per_second=10, + duration_seconds=1, + bursts=(), + failures=(), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + assert report.object_count == 100 + assert report.final_digest == report.expected_digest + assert report.dropped_updates == 0 + assert report.rendered_updates == 110 + assert report.input_latency.count > 0 + assert report.api.operations["list"] == 1 + assert report.api.operations["watch_open"] == 1 + assert report.api.operations.get("get", 0) == 0 + + +async def test_replay_gone_reconnects_and_digest_matches() -> None: + """gone at event 5 triggers one reconnect/re-LIST; final digest drops stale rows.""" + profile = WorkloadProfile( + schema_version=1, + id="test-gone", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=3, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="gone", at_event=5),), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + assert report.final_digest == report.expected_digest + assert report.dropped_updates == 0 + assert report.api.operations["list"] == 2 + assert report.api.operations["watch_open"] == 2 + assert report.api.reconnects == 1 + assert report.api.relists == 1 From 199fa235adce4a9a74108753ceb430ec26fd6215 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 03:05:23 +0900 Subject: [PATCH 12/38] test: fix input-timing and oracle-pin defects in replay harness Two spec-compliance fixes for the Task 5 replay harness (issue #186): 1. Input timing: churn_start.set() was called after pilot.press(), so cursor input was measured while the replay source was blocked. Move churn_start.set() before the key presses and snapshot churn_started_before_input = churn_start.is_set() immediately before the first press. Both tests now assert report.churn_started_before_input. 2. Oracle independence: expected_digest was computed from _ReplaySource.current_digest() (internal parallel bookkeeping) rather than the spec-named apply_events() oracle. Switch run_replay to filter hard-failure sequences (gone/throttled/forbidden) from the event list and compute expected_digest = summary_digest(apply_events(...)). Both tests now assert report.expected_digest == summary_digest(apply_events(...)) as an independent oracle pin. For the gone-reconnect test the failure event (sequence 5) is correctly excluded from the oracle since it was never yielded as a watch event. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/replay.py | 26 ++++++++++++++++++++------ tests/performance/test_replay.py | 17 +++++++++++++++-- 2 files changed, 35 insertions(+), 8 deletions(-) diff --git a/tests/performance/replay.py b/tests/performance/replay.py index 0b019391..02e77193 100644 --- a/tests/performance/replay.py +++ b/tests/performance/replay.py @@ -40,6 +40,7 @@ from tests.performance.profile import FailureInjection, WorkloadProfile from tests.performance.workload import ( ScheduledEvent, + apply_events, initial_pods, scheduled_events, summary_digest, @@ -79,6 +80,7 @@ class ReplayReport: coalesced_updates: int event_to_render: LatencySummary input_latency: LatencySummary + churn_started_before_input: bool process: ProcessSummary api: ApiSummary manifest: RunManifest @@ -278,6 +280,7 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay ) sampler.start() + churn_started_before_input = False try: async with app.run_test() as pilot: table = app.query_one(ResourceTable) @@ -290,7 +293,11 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay label="initial pods rendered", ) - # Record cursor key latency before churn starts. + # Release the source to emit scheduled events, then drive cursor + # input while churn is active (not before the source is unblocked). + churn_start.set() + + churn_started_before_input = churn_start.is_set() t0 = monotonic() await pilot.press("down") recorder.record_input(monotonic() - t0) @@ -298,9 +305,6 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay await pilot.press("up") recorder.record_input(monotonic() - t0) - # Release the source to emit scheduled events. - churn_start.set() - # Wait for all events to be emitted and all renders to complete. await until( pilot, @@ -312,9 +316,18 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay process_samples = await sampler.stop() await watch_manager.stop_all() - # Compute digests: source tracks expected state; store reflects actual state. + # Compute expected digest using the independent apply_events oracle. + # Hard failures (gone, throttled, forbidden) raise before yielding the + # event, so the corresponding watch events are never applied; filter them + # from the oracle to match the actual replay outcome. + hard_failure_seqs: frozenset[int] = frozenset( + f.at_event for f in profile.failures if f.kind in _HARD_FAILURE_STATUS + ) + oracle_events = tuple(e for e in events if e.sequence not in hard_failure_seqs) + expected_digest = summary_digest(apply_events(initial_pods(profile), oracle_events)) + + # Compute final digest from the store (actual state). final_digest = summary_digest(cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES))) - expected_digest = source.current_digest() benchmark = recorder.report(manifest, process_samples, final_digest=final_digest) @@ -328,6 +341,7 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay coalesced_updates=benchmark.coalesced_updates, event_to_render=benchmark.event_to_render, input_latency=benchmark.input_latency, + churn_started_before_input=churn_started_before_input, process=benchmark.process, api=benchmark.api, manifest=benchmark.manifest, diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py index e1ae57c3..d1d4deba 100644 --- a/tests/performance/test_replay.py +++ b/tests/performance/test_replay.py @@ -9,6 +9,7 @@ from tests.performance.profile import FailureInjection, WorkloadProfile from tests.performance.replay import ReplayOptions, run_replay +from tests.performance.workload import apply_events, initial_pods, scheduled_events, summary_digest async def test_replay_uses_real_app_and_reaches_expected_digest() -> None: @@ -24,11 +25,15 @@ async def test_replay_uses_real_app_and_reaches_expected_digest() -> None: failures=(), ) report = await run_replay(profile, ReplayOptions(time_scale=0)) + events = scheduled_events(profile) + oracle = summary_digest(apply_events(initial_pods(profile), events)) assert report.object_count == 100 - assert report.final_digest == report.expected_digest + assert report.expected_digest == oracle + assert report.final_digest == oracle assert report.dropped_updates == 0 assert report.rendered_updates == 110 assert report.input_latency.count > 0 + assert report.churn_started_before_input assert report.api.operations["list"] == 1 assert report.api.operations["watch_open"] == 1 assert report.api.operations.get("get", 0) == 0 @@ -48,8 +53,16 @@ async def test_replay_gone_reconnects_and_digest_matches() -> None: failures=(FailureInjection(kind="gone", at_event=5),), ) report = await run_replay(profile, ReplayOptions(time_scale=0)) - assert report.final_digest == report.expected_digest + events = scheduled_events(profile) + # The gone failure event itself is not applied as a watch event; filter it + # from the apply_events oracle so it matches the actual replay outcome. + hard_failure_seqs = {f.at_event for f in profile.failures if f.kind != "slow"} + applied = tuple(e for e in events if e.sequence not in hard_failure_seqs) + oracle = summary_digest(apply_events(initial_pods(profile), applied)) + assert report.expected_digest == oracle + assert report.final_digest == oracle assert report.dropped_updates == 0 + assert report.churn_started_before_input assert report.api.operations["list"] == 2 assert report.api.operations["watch_open"] == 2 assert report.api.reconnects == 1 From bb353164f4f4716a5a387b3ffbee516ec65cb017 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 03:18:40 +0900 Subject: [PATCH 13/38] test: add reproducible scale benchmark CLI Ship 1k, 10k, and 50k replay profiles with machine-readable reports for #186. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/cli.py | 183 ++++++++++++++++++++ tests/performance/profiles/burst-50k.json | 15 ++ tests/performance/profiles/smoke-1k.json | 13 ++ tests/performance/profiles/steady-10k.json | 13 ++ tests/performance/test_cli.py | 189 +++++++++++++++++++++ 5 files changed, 413 insertions(+) create mode 100644 tests/performance/cli.py create mode 100644 tests/performance/profiles/burst-50k.json create mode 100644 tests/performance/profiles/smoke-1k.json create mode 100644 tests/performance/profiles/steady-10k.json create mode 100644 tests/performance/test_cli.py diff --git a/tests/performance/cli.py b/tests/performance/cli.py new file mode 100644 index 00000000..df0661e8 --- /dev/null +++ b/tests/performance/cli.py @@ -0,0 +1,183 @@ +"""Reproducible scale-benchmark CLI for large-cluster qualification (issue #186). + +Usage: + uv run python -m tests.performance.cli replay \\ + --profile PATH [--time-scale FLOAT] [--sample-interval FLOAT] \\ + [--json PATH] [--out PATH] [--cpu-profile PATH] [--allocation-snapshot PATH] +""" + +from __future__ import annotations + +import argparse +import asyncio +import cProfile +import json +import sys +import tracemalloc +from pathlib import Path +from typing import Any + +from tests.performance.metrics import BenchmarkReport, render_markdown, report_payload +from tests.performance.profile import WorkloadProfile, load_profile +from tests.performance.replay import ReplayOptions, ReplayReport, run_replay + + +def _to_benchmark_report(replay: ReplayReport) -> BenchmarkReport: + return BenchmarkReport( + manifest=replay.manifest, + event_to_render=replay.event_to_render, + input_latency=replay.input_latency, + process=replay.process, + api=replay.api, + rendered_updates=replay.rendered_updates, + render_passes=replay.render_passes, + coalesced_updates=replay.coalesced_updates, + dropped_updates=replay.dropped_updates, + final_digest=replay.final_digest, + ) + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="python -m tests.performance.cli", + description="korvid large-cluster benchmark CLI", + ) + subparsers = parser.add_subparsers(dest="command", required=True) + + rp = subparsers.add_parser("replay", help="Replay a workload profile and report metrics.") + rp.add_argument("--profile", required=True, metavar="PATH", help="Workload profile JSON.") + rp.add_argument( + "--time-scale", + type=float, + default=1.0, + metavar="FLOAT", + help="Sleep multiplier: 0 skips all sleeps, 1.0 replays at real time.", + ) + rp.add_argument( + "--sample-interval", + type=float, + default=1.0, + metavar="FLOAT", + help="Seconds between process-memory samples (positive).", + ) + rp.add_argument( + "--json", + dest="json_path", + default=None, + metavar="PATH", + help="Write machine-readable JSON report.", + ) + rp.add_argument( + "--out", + dest="out_path", + default=None, + metavar="PATH", + help="Write Markdown report to file (also printed to stdout).", + ) + rp.add_argument( + "--cpu-profile", + default=None, + metavar="PATH", + help="Write cProfile pstats file.", + ) + rp.add_argument( + "--allocation-snapshot", + default=None, + metavar="PATH", + help="Write top-100 tracemalloc source locations.", + ) + return parser + + +def _run_with_cpu_profile( + profile: WorkloadProfile, + options: ReplayOptions, + cpu_profile_path: str, +) -> ReplayReport: + """Run *run_replay* and dump a cProfile pstats file to *cpu_profile_path*.""" + pr = cProfile.Profile() + pr.enable() + try: + return asyncio.run(run_replay(profile, options)) + finally: + pr.disable() + pr.dump_stats(cpu_profile_path) + + +def _flush_allocation_snapshot(path: str) -> None: + """Take a tracemalloc snapshot and write the top 100 lines to *path*.""" + if not tracemalloc.is_tracing(): + return + snapshot = tracemalloc.take_snapshot() + stats = snapshot.statistics("lineno")[:100] + Path(path).write_text("\n".join(str(stat) for stat in stats)) + tracemalloc.stop() + + +def _write_outputs(args: argparse.Namespace, replay: ReplayReport) -> None: + """Print Markdown to stdout and write optional --out / --json outputs.""" + benchmark = _to_benchmark_report(replay) + markdown = render_markdown(benchmark) + sys.stdout.write(markdown) + if args.out_path: + Path(args.out_path).write_text(markdown) + if args.json_path: + payload: dict[str, Any] = {"schema_version": 1, **report_payload(benchmark)} + Path(args.json_path).write_text(json.dumps(payload, indent=2)) + + +def _cmd_replay(args: argparse.Namespace) -> int: + if args.time_scale < 0: + print("error: --time-scale must be non-negative", file=sys.stderr) + return 1 + if args.sample_interval <= 0: + print("error: --sample-interval must be positive", file=sys.stderr) + return 1 + + try: + profile = load_profile(Path(args.profile)) + except Exception as exc: + print(f"error loading profile: {exc}", file=sys.stderr) + return 1 + + options = ReplayOptions(time_scale=args.time_scale, sample_interval=args.sample_interval) + + if args.allocation_snapshot: + tracemalloc.start() + + try: + if args.cpu_profile: + replay = _run_with_cpu_profile(profile, options, args.cpu_profile) + else: + replay = asyncio.run(run_replay(profile, options)) + except Exception as exc: + print(f"error during replay: {exc}", file=sys.stderr) + return 1 + finally: + if args.allocation_snapshot: + _flush_allocation_snapshot(args.allocation_snapshot) + + _write_outputs(args, replay) + if replay.dropped_updates > 0 or replay.expected_digest != replay.final_digest: + return 1 + return 0 + + +def main(argv: list[str] | None = None) -> int: + """CLI entry point for the large-cluster benchmark. + + Args: + argv: Argument list; defaults to `sys.argv[1:]` when `None`. + + Returns: + 0 on success; 1 for runtime error, dropped updates, or digest mismatch. + """ + parser = _build_parser() + args = parser.parse_args(argv) + if args.command == "replay": + return _cmd_replay(args) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/performance/profiles/burst-50k.json b/tests/performance/profiles/burst-50k.json new file mode 100644 index 00000000..c71fda15 --- /dev/null +++ b/tests/performance/profiles/burst-50k.json @@ -0,0 +1,15 @@ +{ + "schema_version": 1, + "id": "burst-50k", + "seed": 186, + "object_count": 50000, + "namespace_count": 500, + "steady_events_per_second": 200, + "duration_seconds": 30, + "bursts": [ + {"start_second": 5, "duration_seconds": 1, "events_per_second": 1000}, + {"start_second": 15, "duration_seconds": 1, "events_per_second": 1000}, + {"start_second": 25, "duration_seconds": 1, "events_per_second": 1000} + ], + "failures": [] +} diff --git a/tests/performance/profiles/smoke-1k.json b/tests/performance/profiles/smoke-1k.json new file mode 100644 index 00000000..20ff4213 --- /dev/null +++ b/tests/performance/profiles/smoke-1k.json @@ -0,0 +1,13 @@ +{ + "schema_version": 1, + "id": "smoke-1k", + "seed": 186, + "object_count": 1000, + "namespace_count": 20, + "steady_events_per_second": 20, + "duration_seconds": 5, + "bursts": [ + {"start_second": 2, "duration_seconds": 1, "events_per_second": 100} + ], + "failures": [] +} diff --git a/tests/performance/profiles/steady-10k.json b/tests/performance/profiles/steady-10k.json new file mode 100644 index 00000000..77d17bf1 --- /dev/null +++ b/tests/performance/profiles/steady-10k.json @@ -0,0 +1,13 @@ +{ + "schema_version": 1, + "id": "steady-10k", + "seed": 186, + "object_count": 10000, + "namespace_count": 100, + "steady_events_per_second": 100, + "duration_seconds": 30, + "bursts": [ + {"start_second": 10, "duration_seconds": 4, "events_per_second": 500} + ], + "failures": [] +} diff --git a/tests/performance/test_cli.py b/tests/performance/test_cli.py new file mode 100644 index 00000000..b37cba27 --- /dev/null +++ b/tests/performance/test_cli.py @@ -0,0 +1,189 @@ +"""CLI tests for the large-cluster benchmark tool (issue #186).""" + +from __future__ import annotations + +import json +from pathlib import Path +from types import MappingProxyType + +import pytest + +from tests.performance import cli +from tests.performance.metrics import ( + ApiSummary, + LatencySummary, + ProcessSummary, + RunManifest, +) +from tests.performance.profile import WorkloadProfile +from tests.performance.replay import ReplayOptions, ReplayReport + +# --------------------------------------------------------------------------- +# Fixtures / helpers +# --------------------------------------------------------------------------- + + +def _make_manifest() -> RunManifest: + return RunManifest( + profile_id="test", + profile_hash="abc123", + korvid_sha="dev", + python="3.11", + textual="0.0.0", + os="test", + cpu_count=1, + memory_bytes=0, + ) + + +def _make_latency() -> LatencySummary: + return LatencySummary( + count=0, + p50_seconds=None, + p95_seconds=None, + p99_seconds=None, + maximum_seconds=None, + ) + + +def _make_process() -> ProcessSummary: + return ProcessSummary( + sample_count=0, + cpu_percent_max=None, + rss_bytes_max=None, + python_bytes_max=None, + rss_slope_mib_per_minute=None, + ) + + +def _make_api() -> ApiSummary: + return ApiSummary( + operations=MappingProxyType({}), + paths=MappingProxyType({}), + decoded_bytes=0, + object_count=0, + watch_events=0, + reconnects=0, + relists=0, + throttles=0, + authorization_failures=0, + ) + + +def _make_minimal_profile() -> WorkloadProfile: + return WorkloadProfile( + schema_version=1, + id="smoke-mini", + seed=0, + object_count=4, + namespace_count=2, + steady_events_per_second=0, + duration_seconds=1, + bursts=(), + failures=(), + ) + + +def profile_path(tmp_path: Path) -> Path: + """Write a minimal valid profile to *tmp_path* and return its path.""" + path = tmp_path / "smoke-mini.json" + path.write_text( + json.dumps( + { + "schema_version": 1, + "id": "smoke-mini", + "seed": 0, + "object_count": 4, + "namespace_count": 2, + "steady_events_per_second": 0, + "duration_seconds": 1, + "bursts": [], + "failures": [], + } + ) + ) + return path + + +# --------------------------------------------------------------------------- +# Fake coroutines for monkeypatching +# --------------------------------------------------------------------------- + + +async def fake_run_replay( + profile: WorkloadProfile, + options: ReplayOptions, +) -> ReplayReport: + """Successful replay — digests match, no dropped updates.""" + return ReplayReport( + object_count=profile.object_count, + expected_digest="ok", + final_digest="ok", + dropped_updates=0, + rendered_updates=0, + render_passes=0, + coalesced_updates=0, + event_to_render=_make_latency(), + input_latency=_make_latency(), + churn_started_before_input=True, + process=_make_process(), + api=_make_api(), + manifest=_make_manifest(), + ) + + +async def fake_failed_report( + profile: WorkloadProfile, + options: ReplayOptions, +) -> ReplayReport: + """Failed replay — digest mismatch simulates store corruption.""" + return ReplayReport( + object_count=profile.object_count, + expected_digest="expected-abc", + final_digest="actual-xyz", + dropped_updates=0, + rendered_updates=0, + render_passes=0, + coalesced_updates=0, + event_to_render=_make_latency(), + input_latency=_make_latency(), + churn_started_before_input=True, + process=_make_process(), + api=_make_api(), + manifest=_make_manifest(), + ) + + +# --------------------------------------------------------------------------- +# Tests +# --------------------------------------------------------------------------- + + +def test_cli_writes_json_and_markdown(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + json_path = tmp_path / "result.json" + markdown_path = tmp_path / "result.md" + monkeypatch.setattr(cli, "run_replay", fake_run_replay) + result = cli.main( + [ + "replay", + "--profile", + str(profile_path(tmp_path)), + "--time-scale", + "0", + "--json", + str(json_path), + "--out", + str(markdown_path), + ] + ) + assert result == 0 + assert json.loads(json_path.read_text())["schema_version"] == 1 + assert "# Large-cluster benchmark" in markdown_path.read_text() + + +def test_cli_returns_nonzero_for_digest_or_drop_failure( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(cli, "load_profile", lambda _path: _make_minimal_profile()) + monkeypatch.setattr(cli, "run_replay", fake_failed_report) + assert cli.main(["replay", "--profile", "profile.json"]) == 1 From e1f7f71cd312d06e961519dfdee0759e43c07a54 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 03:28:24 +0900 Subject: [PATCH 14/38] test: harden benchmark CLI failures Narrow expected CLI failures, preserve unexpected tracebacks, stabilize JSON ordering, and cover dropped updates and argument validation for #186. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/cli.py | 7 ++-- tests/performance/test_cli.py | 74 ++++++++++++++++++++++++++++++++++- 2 files changed, 77 insertions(+), 4 deletions(-) diff --git a/tests/performance/cli.py b/tests/performance/cli.py index df0661e8..8c849a35 100644 --- a/tests/performance/cli.py +++ b/tests/performance/cli.py @@ -17,6 +17,7 @@ from pathlib import Path from typing import Any +from korvid.k8s.errors import ApiStatusError from tests.performance.metrics import BenchmarkReport, render_markdown, report_payload from tests.performance.profile import WorkloadProfile, load_profile from tests.performance.replay import ReplayOptions, ReplayReport, run_replay @@ -123,7 +124,7 @@ def _write_outputs(args: argparse.Namespace, replay: ReplayReport) -> None: Path(args.out_path).write_text(markdown) if args.json_path: payload: dict[str, Any] = {"schema_version": 1, **report_payload(benchmark)} - Path(args.json_path).write_text(json.dumps(payload, indent=2)) + Path(args.json_path).write_text(json.dumps(payload, indent=2, sort_keys=True)) def _cmd_replay(args: argparse.Namespace) -> int: @@ -136,7 +137,7 @@ def _cmd_replay(args: argparse.Namespace) -> int: try: profile = load_profile(Path(args.profile)) - except Exception as exc: + except (OSError, UnicodeError, ValueError) as exc: print(f"error loading profile: {exc}", file=sys.stderr) return 1 @@ -150,7 +151,7 @@ def _cmd_replay(args: argparse.Namespace) -> int: replay = _run_with_cpu_profile(profile, options, args.cpu_profile) else: replay = asyncio.run(run_replay(profile, options)) - except Exception as exc: + except (ApiStatusError, AssertionError, OSError) as exc: print(f"error during replay: {exc}", file=sys.stderr) return 1 finally: diff --git a/tests/performance/test_cli.py b/tests/performance/test_cli.py index b37cba27..d912815c 100644 --- a/tests/performance/test_cli.py +++ b/tests/performance/test_cli.py @@ -3,11 +3,13 @@ from __future__ import annotations import json +from dataclasses import replace from pathlib import Path from types import MappingProxyType import pytest +from korvid.k8s.errors import ApiStatusError from tests.performance import cli from tests.performance.metrics import ( ApiSummary, @@ -154,6 +156,30 @@ async def fake_failed_report( ) +async def fake_dropped_report( + profile: WorkloadProfile, + options: ReplayOptions, +) -> ReplayReport: + """Failed replay — matching digests with one unrendered update.""" + return replace(await fake_run_replay(profile, options), dropped_updates=1) + + +async def fake_api_failure( + profile: WorkloadProfile, + options: ReplayOptions, +) -> ReplayReport: + """Expected replay failure surfaced by the Kubernetes boundary.""" + raise ApiStatusError(503, "unavailable") + + +async def fake_programmer_error( + profile: WorkloadProfile, + options: ReplayOptions, +) -> ReplayReport: + """Unexpected implementation error that must retain its traceback.""" + raise TypeError("unexpected replay defect") + + # --------------------------------------------------------------------------- # Tests # --------------------------------------------------------------------------- @@ -178,12 +204,58 @@ def test_cli_writes_json_and_markdown(tmp_path: Path, monkeypatch: pytest.Monkey ) assert result == 0 assert json.loads(json_path.read_text())["schema_version"] == 1 + assert json_path.read_text().index('"api"') < json_path.read_text().index('"schema_version"') assert "# Large-cluster benchmark" in markdown_path.read_text() +@pytest.mark.parametrize("failed_replay", [fake_failed_report, fake_dropped_report]) def test_cli_returns_nonzero_for_digest_or_drop_failure( monkeypatch: pytest.MonkeyPatch, + failed_replay: object, ) -> None: monkeypatch.setattr(cli, "load_profile", lambda _path: _make_minimal_profile()) - monkeypatch.setattr(cli, "run_replay", fake_failed_report) + monkeypatch.setattr(cli, "run_replay", failed_replay) assert cli.main(["replay", "--profile", "profile.json"]) == 1 + + +@pytest.mark.parametrize( + ("option", "value", "message"), + [ + ("--time-scale", "-1", "--time-scale must be non-negative"), + ("--sample-interval", "0", "--sample-interval must be positive"), + ], +) +def test_cli_rejects_invalid_timing_options( + option: str, + value: str, + message: str, + capsys: pytest.CaptureFixture[str], +) -> None: + assert cli.main(["replay", "--profile", "profile.json", option, value]) == 1 + assert message in capsys.readouterr().err + + +def test_cli_reports_expected_replay_errors( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + monkeypatch.setattr(cli, "load_profile", lambda _path: _make_minimal_profile()) + monkeypatch.setattr(cli, "run_replay", fake_api_failure) + assert cli.main(["replay", "--profile", "profile.json"]) == 1 + assert "error during replay: API 503: unavailable" in capsys.readouterr().err + + +def test_cli_does_not_hide_unexpected_profile_errors(monkeypatch: pytest.MonkeyPatch) -> None: + def fail_profile(_path: Path) -> WorkloadProfile: + raise TypeError("unexpected profile defect") + + monkeypatch.setattr(cli, "load_profile", fail_profile) + with pytest.raises(TypeError, match="unexpected profile defect"): + cli.main(["replay", "--profile", "profile.json"]) + + +def test_cli_does_not_hide_unexpected_replay_errors(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(cli, "load_profile", lambda _path: _make_minimal_profile()) + monkeypatch.setattr(cli, "run_replay", fake_programmer_error) + with pytest.raises(TypeError, match="unexpected replay defect"): + cli.main(["replay", "--profile", "profile.json"]) From 1ffbec5ee015ae87bf5fe91568c24ce1de5d1026 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 04:05:57 +0900 Subject: [PATCH 15/38] test: fix absolute-offset delay bug in _ReplaySource; add time_scale=1 regression MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The watch-phase loop computed sleep durations from each event's absolute offset_seconds rather than elapsed wall-clock time, causing total sleep to equal the sum of all offsets (O(N × profile_duration)) instead of the profile duration. With smoke-1k (5 s, 180 events) the bug produced ~250 s of sleeping, guaranteeing a 30 s until() timeout on every time_scale=1 run. Fix: record self._replay_start when churn begins and compute delay = event.offset_seconds * time_scale - elapsed so each event waits only the time remaining to its scheduled position. Regression test: test_replay_time_scale_1_uses_relative_inter_event_delays uses a 3 s profile where the broken sum-of-offsets (~90 s) would exceed the until() guard. The test completes in 3.8 s on the fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/replay.py | 8 +++++++- tests/performance/test_replay.py | 25 +++++++++++++++++++++++++ 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/tests/performance/replay.py b/tests/performance/replay.py index 02e77193..2f1da221 100644 --- a/tests/performance/replay.py +++ b/tests/performance/replay.py @@ -136,6 +136,7 @@ def __init__( self._failures = failures self._generation = 0 self._next_event_index = 0 + self._replay_start: float = 0.0 self._current: dict[str, PodSummary] = { f"{p.namespace}/{p.name}": p for p in initial_pods(profile) } @@ -192,11 +193,16 @@ async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summ # Pause here until run_replay confirms the table is populated. self._churn_ready.set() await self._churn_start.wait() + # Record the wall-clock instant when churn begins so that + # event.offset_seconds (absolute positions within the profile) + # can be converted to correct inter-event delays below. + self._replay_start = monotonic() # --- WATCH phase --- for i in range(self._next_event_index, len(self._events)): event = self._events[i] - delay = event.offset_seconds * self._options.time_scale + elapsed = monotonic() - self._replay_start + delay = event.offset_seconds * self._options.time_scale - elapsed if delay > 0: await asyncio.sleep(delay) diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py index d1d4deba..d216283f 100644 --- a/tests/performance/test_replay.py +++ b/tests/performance/test_replay.py @@ -39,6 +39,31 @@ async def test_replay_uses_real_app_and_reaches_expected_digest() -> None: assert report.api.operations.get("get", 0) == 0 +async def test_replay_time_scale_1_uses_relative_inter_event_delays() -> None: + """time_scale=1 must replay within approximately profile.duration_seconds. + + If the delay computation uses each event's absolute offset_seconds instead + of the elapsed time since churn started, total sleep = sum(all offsets) ≈ + many multiples of the profile duration, causing the until() guard to fire. + """ + profile = WorkloadProfile( + schema_version=1, + id="test-ts1", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=3, + bursts=(), + failures=(), + ) + # With the bug the sum of absolute offsets is ~90 s > the 30 s until() + # timeout, causing an AssertionError before any assert below is reached. + report = await run_replay(profile, ReplayOptions(time_scale=1)) + assert report.dropped_updates == 0 + assert report.object_count == 20 + + async def test_replay_gone_reconnects_and_digest_matches() -> None: """gone at event 5 triggers one reconnect/re-LIST; final digest drops stale rows.""" profile = WorkloadProfile( From 38569b408dd1a60472b14d45bfc498f71c1da68d Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 04:13:56 +0900 Subject: [PATCH 16/38] test: add digest-correctness assertion to time_scale=1 regression test Per code review: also assert report.expected_digest == report.final_digest so that a hypothetical event-reordering defect introduced alongside the timing fix would be caught even if dropped_updates stays zero. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/test_replay.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py index d216283f..c46cca44 100644 --- a/tests/performance/test_replay.py +++ b/tests/performance/test_replay.py @@ -62,6 +62,7 @@ async def test_replay_time_scale_1_uses_relative_inter_event_delays() -> None: report = await run_replay(profile, ReplayOptions(time_scale=1)) assert report.dropped_updates == 0 assert report.object_count == 20 + assert report.expected_digest == report.final_digest async def test_replay_gone_reconnects_and_digest_matches() -> None: From e2acda11b5a15e09c7fdf3d6f90976b16eb4722a Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 05:00:29 +0900 Subject: [PATCH 17/38] test: add time_scale=1 HTTP 410 reconnect regression; correct event-rate parameters MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add test_replay_gone_reconnects_with_time_scale_1 which proves the relay harness correctly handles a HTTP 410 reconnect when time_scale=1 is active: the post-reconnect watch generation must continue using the original _replay_start (set at churn_start, not reset on reconnect) so event offsets remain relative to the global churn origin rather than re-sleeping the full absolute offset from the reconnect timestamp. RED/GREEN sensitivity confirmed: a temporary mutation that applied absolute- offset sleep for gen>0 (instead of offset*scale - elapsed) caused the churn to exceed the 30s until() timeout in the full test suite: AssertionError: churn complete and all events rendered not met within 30.0s Correct two adjacent test event-rate parameters: - test_replay_time_scale_1_uses_relative_inter_event_delays: 20 eps → 3 eps (sum-of-absolute-offsets with 20 eps = 90s, reliably showing the bug; 3 eps gives ~10s offsets, still > 30s with the bug, passes in 3.8s clean) - test_replay_gone_reconnects_and_digest_matches: 3 eps → 20 eps (reconnect test needs high event density to stress re-list correctness) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/test_replay.py | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py index c46cca44..8fa32a94 100644 --- a/tests/performance/test_replay.py +++ b/tests/performance/test_replay.py @@ -52,7 +52,7 @@ async def test_replay_time_scale_1_uses_relative_inter_event_delays() -> None: seed=186, object_count=20, namespace_count=4, - steady_events_per_second=20, + steady_events_per_second=3, duration_seconds=3, bursts=(), failures=(), @@ -73,7 +73,7 @@ async def test_replay_gone_reconnects_and_digest_matches() -> None: seed=186, object_count=20, namespace_count=4, - steady_events_per_second=3, + steady_events_per_second=20, duration_seconds=2, bursts=(), failures=(FailureInjection(kind="gone", at_event=5),), @@ -93,3 +93,26 @@ async def test_replay_gone_reconnects_and_digest_matches() -> None: assert report.api.operations["watch_open"] == 2 assert report.api.reconnects == 1 assert report.api.relists == 1 + + +async def test_replay_gone_reconnects_with_time_scale_1() -> None: + profile = WorkloadProfile( + schema_version=1, + id="test-gone-ts1", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="gone", at_event=5),), + ) + + report = await run_replay(profile, ReplayOptions(time_scale=1)) + + assert report.expected_digest == report.final_digest + assert report.dropped_updates == 0 + assert report.api.operations["list"] == 2 + assert report.api.operations["watch_open"] == 2 + assert report.api.reconnects == 1 + assert report.api.relists == 1 From a8bfb8ccfa0caddadc00590ff6876fad159a3f40 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 05:26:31 +0900 Subject: [PATCH 18/38] fix(task-7-rereview): restore test sensitivity and eliminate timing races Finding 1 (Critical): test_replay_time_scale_1_uses_relative_inter_event_delays was insensitive to the absolute-offset bug after steady_events_per_second was changed 20->3 in commit 2082aaf. With 9 events the sum_of_offsets is only 15 s < 30 s until() guard, so the test PASSES with the bug (false-GREEN confirmed: PASSED in 12.93 s with mutation). Fix: restore steady_events_per_second=20 (60 events, sum_of_offsets=91.5 s >> 30 s). RED confirmed: FAILED in 39.66 s with mutation. Finding 2 (Important): test_replay_gone_reconnects_with_time_scale_1 with duration_seconds=2 had only ~10.25 s margin above the 30 s until() guard, leaving ~1 s true safety when pilot.press() overhead is subtracted. Fix: increase duration_seconds=2->5 (95 post-failure events, sum_of_offsets=251.75 s, margin=221.75 s) for deterministic RED even in isolation. RED confirmed in isolation: FAILED in 39.67 s with reconnect mutation. Both tests now also use a sleep_callback (new ReplayOptions field) to accumulate total scheduled sleep time and assert total_sleep < duration_seconds * 3, providing a scale-independent deterministic check independent of the until() wall-clock guard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/replay.py | 10 +++++-- tests/performance/test_replay.py | 48 ++++++++++++++++++++++++++------ 2 files changed, 47 insertions(+), 11 deletions(-) diff --git a/tests/performance/replay.py b/tests/performance/replay.py index 2f1da221..d9179584 100644 --- a/tests/performance/replay.py +++ b/tests/performance/replay.py @@ -13,8 +13,8 @@ import os import platform import sys -from collections.abc import AsyncIterator, Iterable -from dataclasses import asdict, dataclass +from collections.abc import AsyncIterator, Callable, Iterable +from dataclasses import asdict, dataclass, field from time import monotonic from typing import Any, cast @@ -56,10 +56,14 @@ class ReplayOptions: time_scale: Multiplier applied to every scheduled-event sleep. 0 skips all sleeps (fastest); 1.0 replays at real time. sample_interval: Seconds between process-memory samples. + sleep_callback: Optional callable invoked with each scheduled sleep + duration (seconds) before the sleep occurs. Used by tests to + accumulate total sleep without modifying the production path. """ time_scale: float = 1.0 sample_interval: float = 1.0 + sleep_callback: Callable[[float], None] | None = field(default=None, hash=False, compare=False) @dataclass(frozen=True) @@ -204,6 +208,8 @@ async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summ elapsed = monotonic() - self._replay_start delay = event.offset_seconds * self._options.time_scale - elapsed if delay > 0: + if self._options.sleep_callback is not None: + self._options.sleep_callback(delay) await asyncio.sleep(delay) await self._handle_failure_if_any(event, i) diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py index 8fa32a94..fc224f20 100644 --- a/tests/performance/test_replay.py +++ b/tests/performance/test_replay.py @@ -40,11 +40,16 @@ async def test_replay_uses_real_app_and_reaches_expected_digest() -> None: async def test_replay_time_scale_1_uses_relative_inter_event_delays() -> None: - """time_scale=1 must replay within approximately profile.duration_seconds. + """time_scale=1 must use inter-event delays, not absolute offsets. If the delay computation uses each event's absolute offset_seconds instead - of the elapsed time since churn started, total sleep = sum(all offsets) ≈ - many multiples of the profile duration, causing the until() guard to fire. + of the elapsed time since churn started, total sleep = sum(all offsets) for + 60 events at 20 eps over 3 s ≈ 91.5 s > the 30 s until() guard, causing an + AssertionError before any assert below is reached. + + The sleep_callback assertion is a scale-independent deterministic check: + with the fix, total_sleep ≈ profile.duration_seconds (inter-event delays); + with the bug, total_sleep ≈ 91.5 s (sum of absolute offsets). """ profile = WorkloadProfile( schema_version=1, @@ -52,14 +57,20 @@ async def test_replay_time_scale_1_uses_relative_inter_event_delays() -> None: seed=186, object_count=20, namespace_count=4, - steady_events_per_second=3, + steady_events_per_second=20, duration_seconds=3, bursts=(), failures=(), ) - # With the bug the sum of absolute offsets is ~90 s > the 30 s until() - # timeout, causing an AssertionError before any assert below is reached. - report = await run_replay(profile, ReplayOptions(time_scale=1)) + total_sleep: list[float] = [0.0] + + def record_sleep(delay: float) -> None: + total_sleep[0] += delay + + report = await run_replay(profile, ReplayOptions(time_scale=1, sleep_callback=record_sleep)) + # Scale-independent check: inter-event delays sum to ≈ duration_seconds, not sum_of_offsets. + # Bug: total_sleep ≈ 91.5 s; fix: total_sleep ≈ 3 s. + assert total_sleep[0] < profile.duration_seconds * 3 assert report.dropped_updates == 0 assert report.object_count == 20 assert report.expected_digest == report.final_digest @@ -96,6 +107,18 @@ async def test_replay_gone_reconnects_and_digest_matches() -> None: async def test_replay_gone_reconnects_with_time_scale_1() -> None: + """HTTP 410 reconnect with time_scale=1 must use elapsed-based delay, not absolute offsets. + + With the absolute-offset reconnect bug, gen=1 events (post-reconnect) sleep + their full absolute offset_seconds values. For 95 events at 20 eps over + 5 s (offsets 0.30-5.00 s), the total gen=1 sleep ~251.75 s >> the 30 s + until() guard, causing a deterministic failure even when run in isolation + (margin 221.75 s, eliminating the pilot-overhead timing race in duration_seconds=2). + + The sleep_callback assertion provides a scale-independent deterministic check: + with the fix, total_sleep ≈ profile.duration_seconds; with the bug, gen=1 alone + contributes ≈ 251.75 s. + """ profile = WorkloadProfile( schema_version=1, id="test-gone-ts1", @@ -103,13 +126,20 @@ async def test_replay_gone_reconnects_with_time_scale_1() -> None: object_count=20, namespace_count=4, steady_events_per_second=20, - duration_seconds=2, + duration_seconds=5, bursts=(), failures=(FailureInjection(kind="gone", at_event=5),), ) + total_sleep: list[float] = [0.0] + + def record_sleep(delay: float) -> None: + total_sleep[0] += delay - report = await run_replay(profile, ReplayOptions(time_scale=1)) + report = await run_replay(profile, ReplayOptions(time_scale=1, sleep_callback=record_sleep)) + # Scale-independent check: total inter-event delays ≈ profile.duration_seconds. + # With the absolute-offset bug: gen=1 alone contributes ≈ 251.75 s. + assert total_sleep[0] < profile.duration_seconds * 3 assert report.expected_digest == report.final_digest assert report.dropped_updates == 0 assert report.api.operations["list"] == 2 From ec26bad4f932fc5ef91e6ccc9d7e808217aafb47 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 05:41:48 +0900 Subject: [PATCH 19/38] fix(task-7-follow-up): replace sleep_callback with virtual-time seam MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the sleep_callback recording mechanism with an injected monotonic clock (monotonic_fn) and async sleeper (async_sleep) on ReplayOptions. _ReplaySource stores self._now and self._sleep, resolved at construction from the options; production runs use time.monotonic and _sleep_default (a thin asyncio.sleep wrapper). Tests supply virtual_monotonic / virtual_sleep closures that advance a shared virtual_time float and do asyncio.sleep(0) to yield without real wall time. Both time_scale=1 tests now complete in ~0.01 s instead of 3–5 s and their total_sleep assertions fire immediately (2.59 s total) under the historical absolute-offset mutation, with no dependence on the until() 30 s guard. RED evidence (mutation: delay = offset * time_scale, no elapsed subtraction): test_replay_time_scale_1_uses_relative_inter_event_delays: assert 88.5 < 9 FAIL test_replay_gone_reconnects_with_time_scale_1: assert 247.5 < 15 FAIL Both fail in 2.59 s wall time. GREEN: 4 passed in 3.97 s (all replay tests, correct implementation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/replay.py | 41 ++++++++++++++------ tests/performance/test_replay.py | 66 +++++++++++++++++++++----------- 2 files changed, 72 insertions(+), 35 deletions(-) diff --git a/tests/performance/replay.py b/tests/performance/replay.py index d9179584..27e43b73 100644 --- a/tests/performance/replay.py +++ b/tests/performance/replay.py @@ -13,7 +13,7 @@ import os import platform import sys -from collections.abc import AsyncIterator, Callable, Iterable +from collections.abc import AsyncIterator, Awaitable, Callable, Iterable from dataclasses import asdict, dataclass, field from time import monotonic from typing import Any, cast @@ -48,6 +48,11 @@ from tests.ui.waits import until +async def _sleep_default(delay: float) -> None: + """Thin wrapper around asyncio.sleep used as the default async sleeper.""" + await asyncio.sleep(delay) + + @dataclass(frozen=True) class ReplayOptions: """Tuning knobs for a replay run. @@ -56,14 +61,21 @@ class ReplayOptions: time_scale: Multiplier applied to every scheduled-event sleep. 0 skips all sleeps (fastest); 1.0 replays at real time. sample_interval: Seconds between process-memory samples. - sleep_callback: Optional callable invoked with each scheduled sleep - duration (seconds) before the sleep occurs. Used by tests to - accumulate total sleep without modifying the production path. + monotonic_fn: Monotonic clock callable injected for testing. + Production runs leave this `None` (uses `time.monotonic`). + async_sleep: Async sleep callable injected for testing. + Production runs leave this `None` (uses `_sleep_default`). + A virtual sleeper can advance a shared clock variable and do + `asyncio.sleep(0)` to yield without real wall time, making + timing-sensitive tests instant and mutation-deterministic. """ time_scale: float = 1.0 sample_interval: float = 1.0 - sleep_callback: Callable[[float], None] | None = field(default=None, hash=False, compare=False) + monotonic_fn: Callable[[], float] | None = field(default=None, hash=False, compare=False) + async_sleep: Callable[[float], Awaitable[None]] | None = field( + default=None, hash=False, compare=False + ) @dataclass(frozen=True) @@ -144,6 +156,13 @@ def __init__( self._current: dict[str, PodSummary] = { f"{p.namespace}/{p.name}": p for p in initial_pods(profile) } + # Virtual-time seam: injected in tests; production uses real clock/sleep. + self._now: Callable[[], float] = ( + options.monotonic_fn if options.monotonic_fn is not None else monotonic + ) + self._sleep: Callable[[float], Awaitable[None]] = ( + options.async_sleep if options.async_sleep is not None else _sleep_default + ) def current_digest(self) -> str: """Digest of the source's tracked expected state.""" @@ -162,7 +181,7 @@ async def _handle_failure_if_any(self, event: ScheduledEvent, index: int) -> Non if failure.kind == "slow": tick = 1.0 / max(self._profile.steady_events_per_second, 1) if tick * self._options.time_scale > 0: - await asyncio.sleep(tick * self._options.time_scale) + await self._sleep(tick * self._options.time_scale) return status = _HARD_FAILURE_STATUS[failure.kind] self._recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=status)) @@ -197,20 +216,18 @@ async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summ # Pause here until run_replay confirms the table is populated. self._churn_ready.set() await self._churn_start.wait() - # Record the wall-clock instant when churn begins so that + # Record the virtual-clock instant when churn begins so that # event.offset_seconds (absolute positions within the profile) # can be converted to correct inter-event delays below. - self._replay_start = monotonic() + self._replay_start = self._now() # --- WATCH phase --- for i in range(self._next_event_index, len(self._events)): event = self._events[i] - elapsed = monotonic() - self._replay_start + elapsed = self._now() - self._replay_start delay = event.offset_seconds * self._options.time_scale - elapsed if delay > 0: - if self._options.sleep_callback is not None: - self._options.sleep_callback(delay) - await asyncio.sleep(delay) + await self._sleep(delay) await self._handle_failure_if_any(event, i) diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py index fc224f20..3db0c143 100644 --- a/tests/performance/test_replay.py +++ b/tests/performance/test_replay.py @@ -7,6 +7,8 @@ from __future__ import annotations +import asyncio + from tests.performance.profile import FailureInjection, WorkloadProfile from tests.performance.replay import ReplayOptions, run_replay from tests.performance.workload import apply_events, initial_pods, scheduled_events, summary_digest @@ -42,14 +44,15 @@ async def test_replay_uses_real_app_and_reaches_expected_digest() -> None: async def test_replay_time_scale_1_uses_relative_inter_event_delays() -> None: """time_scale=1 must use inter-event delays, not absolute offsets. - If the delay computation uses each event's absolute offset_seconds instead - of the elapsed time since churn started, total sleep = sum(all offsets) for - 60 events at 20 eps over 3 s ≈ 91.5 s > the 30 s until() guard, causing an - AssertionError before any assert below is reached. + Virtual-time seam: `monotonic_fn` returns a shared virtual clock and + `async_sleep` advances that clock then yields via `asyncio.sleep(0)`, + so the test completes in ~0 s of wall time regardless of profile length. - The sleep_callback assertion is a scale-independent deterministic check: - with the fix, total_sleep ≈ profile.duration_seconds (inter-event delays); - with the bug, total_sleep ≈ 91.5 s (sum of absolute offsets). + Sensitivity: with 60 events at 20 eps over 3 s, sum_of_offsets ~= 91.5 s. + Under the absolute-offset bug the accumulated delay totals ~= 91.5 s >> 9 s + (= duration_seconds x 3), so the assertion fails immediately without any + wall-clock race or timeout dependency. + With the correct fix, inter-event delays sum ~= 3 s < 9 s -> GREEN. """ profile = WorkloadProfile( schema_version=1, @@ -62,14 +65,22 @@ async def test_replay_time_scale_1_uses_relative_inter_event_delays() -> None: bursts=(), failures=(), ) + virtual_time: list[float] = [0.0] total_sleep: list[float] = [0.0] - def record_sleep(delay: float) -> None: + def virtual_monotonic() -> float: + return virtual_time[0] + + async def virtual_sleep(delay: float) -> None: total_sleep[0] += delay + virtual_time[0] += delay + await asyncio.sleep(0) # yield to event loop without real wall time - report = await run_replay(profile, ReplayOptions(time_scale=1, sleep_callback=record_sleep)) - # Scale-independent check: inter-event delays sum to ≈ duration_seconds, not sum_of_offsets. - # Bug: total_sleep ≈ 91.5 s; fix: total_sleep ≈ 3 s. + report = await run_replay( + profile, + ReplayOptions(time_scale=1, monotonic_fn=virtual_monotonic, async_sleep=virtual_sleep), + ) + # Bug: total_sleep ≈ 91.5 s; fix: total_sleep ≈ 3 s. Threshold = 9 s. assert total_sleep[0] < profile.duration_seconds * 3 assert report.dropped_updates == 0 assert report.object_count == 20 @@ -109,15 +120,16 @@ async def test_replay_gone_reconnects_and_digest_matches() -> None: async def test_replay_gone_reconnects_with_time_scale_1() -> None: """HTTP 410 reconnect with time_scale=1 must use elapsed-based delay, not absolute offsets. - With the absolute-offset reconnect bug, gen=1 events (post-reconnect) sleep - their full absolute offset_seconds values. For 95 events at 20 eps over - 5 s (offsets 0.30-5.00 s), the total gen=1 sleep ~251.75 s >> the 30 s - until() guard, causing a deterministic failure even when run in isolation - (margin 221.75 s, eliminating the pilot-overhead timing race in duration_seconds=2). + Virtual-time seam: same `monotonic_fn` / `async_sleep` pattern as + `test_replay_time_scale_1_uses_relative_inter_event_delays`. The shared + virtual clock is never reset across reconnect generations, so gen=1 events + correctly see the accumulated elapsed time from gen=0. - The sleep_callback assertion provides a scale-independent deterministic check: - with the fix, total_sleep ≈ profile.duration_seconds; with the bug, gen=1 alone - contributes ≈ 251.75 s. + Sensitivity: 95 post-reconnect events (20 eps x 5 s profile minus 5 pre-gone + events) have sum_of_offsets ~= 251.75 s. Under the reconnect-reset bug, + gen=1 sees elapsed=0 and accumulates full absolute offsets -> total_sleep >> 15 s + (= duration_seconds x 3), failing deterministically without a wall-clock race. + With the correct fix, total_sleep ~= 5 s < 15 s -> GREEN. """ profile = WorkloadProfile( schema_version=1, @@ -130,15 +142,23 @@ async def test_replay_gone_reconnects_with_time_scale_1() -> None: bursts=(), failures=(FailureInjection(kind="gone", at_event=5),), ) + virtual_time: list[float] = [0.0] total_sleep: list[float] = [0.0] - def record_sleep(delay: float) -> None: + def virtual_monotonic() -> float: + return virtual_time[0] + + async def virtual_sleep(delay: float) -> None: total_sleep[0] += delay + virtual_time[0] += delay + await asyncio.sleep(0) # yield to event loop without real wall time - report = await run_replay(profile, ReplayOptions(time_scale=1, sleep_callback=record_sleep)) + report = await run_replay( + profile, + ReplayOptions(time_scale=1, monotonic_fn=virtual_monotonic, async_sleep=virtual_sleep), + ) - # Scale-independent check: total inter-event delays ≈ profile.duration_seconds. - # With the absolute-offset bug: gen=1 alone contributes ≈ 251.75 s. + # Bug: gen=1 alone contributes ≈ 251.75 s; fix: total ≈ 5 s. Threshold = 15 s. assert total_sleep[0] < profile.duration_seconds * 3 assert report.expected_digest == report.final_digest assert report.dropped_updates == 0 From 358cf6dd5195b9b09fd5cfde37a3d1e7f0d635fb Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 06:17:56 +0900 Subject: [PATCH 20/38] test(task-7-review): pin replay timing across reconnects Require virtual sleep totals to match the final scheduled offset and assert the first post-410 delay. This makes the issue #186 regression fail for reconnect-origin resets and omitted sleeping, not only the original absolute-offset mutation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/test_replay.py | 39 +++++++++++++++++--------------- 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py index 3db0c143..c9a439f8 100644 --- a/tests/performance/test_replay.py +++ b/tests/performance/test_replay.py @@ -9,6 +9,8 @@ import asyncio +import pytest + from tests.performance.profile import FailureInjection, WorkloadProfile from tests.performance.replay import ReplayOptions, run_replay from tests.performance.workload import apply_events, initial_pods, scheduled_events, summary_digest @@ -48,11 +50,9 @@ async def test_replay_time_scale_1_uses_relative_inter_event_delays() -> None: `async_sleep` advances that clock then yields via `asyncio.sleep(0)`, so the test completes in ~0 s of wall time regardless of profile length. - Sensitivity: with 60 events at 20 eps over 3 s, sum_of_offsets ~= 91.5 s. - Under the absolute-offset bug the accumulated delay totals ~= 91.5 s >> 9 s - (= duration_seconds x 3), so the assertion fails immediately without any - wall-clock race or timeout dependency. - With the correct fix, inter-event delays sum ~= 3 s < 9 s -> GREEN. + Sensitivity: with 60 events at 20 eps over 3 s, correct sleeps sum exactly + to the final 2.95 s offset. The historical absolute-offset bug sums every + offset instead, while omitted sleeps sum to zero; both fail deterministically. """ profile = WorkloadProfile( schema_version=1, @@ -66,13 +66,13 @@ async def test_replay_time_scale_1_uses_relative_inter_event_delays() -> None: failures=(), ) virtual_time: list[float] = [0.0] - total_sleep: list[float] = [0.0] + sleep_delays: list[float] = [] def virtual_monotonic() -> float: return virtual_time[0] async def virtual_sleep(delay: float) -> None: - total_sleep[0] += delay + sleep_delays.append(delay) virtual_time[0] += delay await asyncio.sleep(0) # yield to event loop without real wall time @@ -80,8 +80,7 @@ async def virtual_sleep(delay: float) -> None: profile, ReplayOptions(time_scale=1, monotonic_fn=virtual_monotonic, async_sleep=virtual_sleep), ) - # Bug: total_sleep ≈ 91.5 s; fix: total_sleep ≈ 3 s. Threshold = 9 s. - assert total_sleep[0] < profile.duration_seconds * 3 + assert sum(sleep_delays) == pytest.approx(scheduled_events(profile)[-1].offset_seconds) assert report.dropped_updates == 0 assert report.object_count == 20 assert report.expected_digest == report.final_digest @@ -125,11 +124,10 @@ async def test_replay_gone_reconnects_with_time_scale_1() -> None: virtual clock is never reset across reconnect generations, so gen=1 events correctly see the accumulated elapsed time from gen=0. - Sensitivity: 95 post-reconnect events (20 eps x 5 s profile minus 5 pre-gone - events) have sum_of_offsets ~= 251.75 s. Under the reconnect-reset bug, - gen=1 sees elapsed=0 and accumulates full absolute offsets -> total_sleep >> 15 s - (= duration_seconds x 3), failing deterministically without a wall-clock race. - With the correct fix, total_sleep ~= 5 s < 15 s -> GREEN. + Sensitivity: correct sleeps sum exactly to the final 4.95 s offset, and the + first post-410 sleep remains one 0.05 s tick. Resetting the replay origin on + reconnect produces a 0.25 s first reconnect sleep and 5.15 s total; omitted + sleeps produce zero. Both fail deterministically. """ profile = WorkloadProfile( schema_version=1, @@ -143,13 +141,13 @@ async def test_replay_gone_reconnects_with_time_scale_1() -> None: failures=(FailureInjection(kind="gone", at_event=5),), ) virtual_time: list[float] = [0.0] - total_sleep: list[float] = [0.0] + sleep_delays: list[float] = [] def virtual_monotonic() -> float: return virtual_time[0] async def virtual_sleep(delay: float) -> None: - total_sleep[0] += delay + sleep_delays.append(delay) virtual_time[0] += delay await asyncio.sleep(0) # yield to event loop without real wall time @@ -158,8 +156,13 @@ async def virtual_sleep(delay: float) -> None: ReplayOptions(time_scale=1, monotonic_fn=virtual_monotonic, async_sleep=virtual_sleep), ) - # Bug: gen=1 alone contributes ≈ 251.75 s; fix: total ≈ 5 s. Threshold = 15 s. - assert total_sleep[0] < profile.duration_seconds * 3 + events = scheduled_events(profile) + failure_sequence = profile.failures[0].at_event + first_post_reconnect_delay = ( + events[failure_sequence].offset_seconds - events[failure_sequence - 1].offset_seconds + ) + assert sum(sleep_delays) == pytest.approx(events[-1].offset_seconds) + assert sleep_delays[failure_sequence - 1] == pytest.approx(first_post_reconnect_delay) assert report.expected_digest == report.final_digest assert report.dropped_updates == 0 assert report.api.operations["list"] == 2 From 9d79c6127151a913eb82ab31ac9fd6080952791f Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 06:54:27 +0900 Subject: [PATCH 21/38] feat: add live seed manifest generator Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/cli.py | 61 +++++++++ tests/performance/manifests.py | 116 ++++++++++++++++ tests/performance/profiles/aks-1k.json | 15 +++ tests/performance/test_cli.py | 136 +++++++++++++++++++ tests/performance/test_manifests.py | 175 +++++++++++++++++++++++++ tests/performance/test_profile.py | 27 +++- 6 files changed, 529 insertions(+), 1 deletion(-) create mode 100644 tests/performance/manifests.py create mode 100644 tests/performance/profiles/aks-1k.json create mode 100644 tests/performance/test_manifests.py diff --git a/tests/performance/cli.py b/tests/performance/cli.py index 8c849a35..596aa8d0 100644 --- a/tests/performance/cli.py +++ b/tests/performance/cli.py @@ -4,6 +4,9 @@ uv run python -m tests.performance.cli replay \\ --profile PATH [--time-scale FLOAT] [--sample-interval FLOAT] \\ [--json PATH] [--out PATH] [--cpu-profile PATH] [--allocation-snapshot PATH] + uv run python -m tests.performance.cli seed-manifests \\ + --run-id TEXT --namespace-count INT --pods-per-namespace INT \\ + --node-selector KEY=VALUE --output PATH """ from __future__ import annotations @@ -17,7 +20,10 @@ from pathlib import Path from typing import Any +import yaml + from korvid.k8s.errors import ApiStatusError +from tests.performance.manifests import build_seed_manifests from tests.performance.metrics import BenchmarkReport, render_markdown, report_payload from tests.performance.profile import WorkloadProfile, load_profile from tests.performance.replay import ReplayOptions, ReplayReport, run_replay @@ -87,6 +93,38 @@ def _build_parser() -> argparse.ArgumentParser: metavar="PATH", help="Write top-100 tracemalloc source locations.", ) + + sp = subparsers.add_parser( + "seed-manifests", + help="Render deterministic Namespace and Pod manifests for live AKS seeding.", + ) + sp.add_argument("--run-id", required=True, metavar="TEXT", help="Unique run identifier.") + sp.add_argument( + "--namespace-count", + required=True, + type=int, + metavar="INT", + help="Number of namespaces to create.", + ) + sp.add_argument( + "--pods-per-namespace", + required=True, + type=int, + metavar="INT", + help="Number of Pods to create in each namespace.", + ) + sp.add_argument( + "--node-selector", + required=True, + metavar="KEY=VALUE", + help="Exactly one nodeSelector key=value pair.", + ) + sp.add_argument( + "--output", + required=True, + metavar="PATH", + help="Destination path for the multi-document YAML output.", + ) return parser @@ -127,6 +165,27 @@ def _write_outputs(args: argparse.Namespace, replay: ReplayReport) -> None: Path(args.json_path).write_text(json.dumps(payload, indent=2, sort_keys=True)) +def _cmd_seed_manifests(args: argparse.Namespace) -> int: + try: + manifests = build_seed_manifests( + run_id=args.run_id, + namespace_count=args.namespace_count, + pods_per_namespace=args.pods_per_namespace, + node_selector=args.node_selector, + ) + except ValueError as exc: + print(f"error building manifests: {exc}", file=sys.stderr) + return 1 + + try: + text = yaml.safe_dump_all(manifests, sort_keys=False, explicit_start=True) + Path(args.output).write_text(text, encoding="utf-8") + except OSError as exc: + print(f"error writing manifests: {exc}", file=sys.stderr) + return 1 + return 0 + + def _cmd_replay(args: argparse.Namespace) -> int: if args.time_scale < 0: print("error: --time-scale must be non-negative", file=sys.stderr) @@ -177,6 +236,8 @@ def main(argv: list[str] | None = None) -> int: args = parser.parse_args(argv) if args.command == "replay": return _cmd_replay(args) + if args.command == "seed-manifests": + return _cmd_seed_manifests(args) return 1 diff --git a/tests/performance/manifests.py b/tests/performance/manifests.py new file mode 100644 index 00000000..1e7a79bb --- /dev/null +++ b/tests/performance/manifests.py @@ -0,0 +1,116 @@ +from __future__ import annotations + +import re + +_RUN_ID_PATTERN = re.compile(r"^[a-z0-9](?:[-a-z0-9]{0,46}[a-z0-9])?$") +_SELECTOR_KEY_PATTERN = re.compile(r"^[a-z0-9](?:[-a-z0-9./]{0,251}[a-z0-9])?$") +_SELECTOR_VALUE_PATTERN = re.compile(r"^[a-z0-9](?:[-a-z0-9.]{0,61}[a-z0-9])?$") +_MANAGED_BY = "korvid-performance" +_BENCH_IMAGE = "registry.k8s.io/pause:3.10" + + +def _validate_positive(value: int, label: str) -> int: + if value < 1: + raise ValueError(f"{label} must be a positive integer") + return value + + +def _validate_run_id(run_id: str) -> str: + if _RUN_ID_PATTERN.fullmatch(run_id): + return run_id + raise ValueError("run_id must be 1-48 lowercase letters, digits, or hyphens") + + +def _parse_node_selector(node_selector: str) -> dict[str, str]: + if node_selector.count("=") != 1: + raise ValueError("node_selector must be exactly one non-empty key=value pair") + key, value = node_selector.split("=", 1) + if not key or not value: + raise ValueError("node_selector must be exactly one non-empty key=value pair") + if key.strip() != key or value.strip() != value: + raise ValueError("node_selector must be exactly one non-empty key=value pair") + if not _SELECTOR_KEY_PATTERN.fullmatch(key) or not _SELECTOR_VALUE_PATTERN.fullmatch(value): + raise ValueError("node_selector must be exactly one non-empty key=value pair") + return {key: value} + + +def _common_labels(run_id: str) -> dict[str, str]: + return { + "app.kubernetes.io/managed-by": _MANAGED_BY, + "korvid.dev/performance-run": run_id, + } + + +def _namespace_name(run_id: str, namespace_index: int) -> str: + name = f"korvid-perf-{run_id}-{namespace_index}" + if len(name) > 63: + raise ValueError("generated namespace name must be 63 characters or fewer") + return name + + +def build_seed_manifests( + *, + run_id: str, + namespace_count: int, + pods_per_namespace: int, + node_selector: str, +) -> tuple[dict[str, object], ...]: + run = _validate_run_id(run_id) + namespaces = _validate_positive(namespace_count, "namespace_count") + pods_each = _validate_positive(pods_per_namespace, "pods_per_namespace") + selector = _parse_node_selector(node_selector) + labels = _common_labels(run) + + manifests: list[dict[str, object]] = [] + namespace_names = tuple(_namespace_name(run, index) for index in range(namespaces)) + for name in namespace_names: + manifests.append( + { + "apiVersion": "v1", + "kind": "Namespace", + "metadata": { + "name": name, + "labels": dict(labels), + }, + } + ) + + for object_index in range(namespaces * pods_each): + namespace_index = object_index % namespaces + pod_name = f"bench-{object_index // namespaces}" + manifests.append( + { + "apiVersion": "v1", + "kind": "Pod", + "metadata": { + "name": pod_name, + "namespace": namespace_names[namespace_index], + "labels": dict(labels), + }, + "spec": { + "nodeSelector": dict(selector), + "tolerations": [ + { + "key": "purpose", + "operator": "Equal", + "value": "perftest", + "effect": "NoSchedule", + } + ], + "containers": [ + { + "name": "bench", + "image": _BENCH_IMAGE, + "resources": { + "requests": { + "cpu": "5m", + "memory": "16Mi", + } + }, + } + ], + "restartPolicy": "Always", + }, + } + ) + return tuple(manifests) diff --git a/tests/performance/profiles/aks-1k.json b/tests/performance/profiles/aks-1k.json new file mode 100644 index 00000000..ec02f82b --- /dev/null +++ b/tests/performance/profiles/aks-1k.json @@ -0,0 +1,15 @@ +{ + "schema_version": 1, + "id": "aks-1k", + "seed": 186, + "object_count": 1000, + "namespace_count": 20, + "steady_events_per_second": 200, + "duration_seconds": 30, + "bursts": [ + {"start_second": 5, "duration_seconds": 1, "events_per_second": 1000}, + {"start_second": 15, "duration_seconds": 1, "events_per_second": 1000}, + {"start_second": 25, "duration_seconds": 1, "events_per_second": 1000} + ], + "failures": [] +} diff --git a/tests/performance/test_cli.py b/tests/performance/test_cli.py index d912815c..b12cb475 100644 --- a/tests/performance/test_cli.py +++ b/tests/performance/test_cli.py @@ -8,6 +8,7 @@ from types import MappingProxyType import pytest +import yaml from korvid.k8s.errors import ApiStatusError from tests.performance import cli @@ -259,3 +260,138 @@ def test_cli_does_not_hide_unexpected_replay_errors(monkeypatch: pytest.MonkeyPa monkeypatch.setattr(cli, "run_replay", fake_programmer_error) with pytest.raises(TypeError, match="unexpected replay defect"): cli.main(["replay", "--profile", "profile.json"]) + + +def test_cli_writes_seed_manifests_yaml(tmp_path: Path) -> None: + output_path = tmp_path / "seed.yaml" + + result = cli.main( + [ + "seed-manifests", + "--run-id", + "aks186", + "--namespace-count", + "2", + "--pods-per-namespace", + "2", + "--node-selector", + "korvid.dev/pool=perftest", + "--output", + str(output_path), + ] + ) + + assert result == 0 + documents = list(yaml.safe_load_all(output_path.read_text())) + assert [document["kind"] for document in documents] == [ + "Namespace", + "Namespace", + "Pod", + "Pod", + "Pod", + "Pod", + ] + assert documents[0]["metadata"]["name"] == "korvid-perf-aks186-0" + assert documents[2]["spec"]["nodeSelector"] == {"korvid.dev/pool": "perftest"} + + +@pytest.mark.parametrize( + ("arguments", "message"), + [ + ( + [ + "seed-manifests", + "--run-id", + "Bad", + "--namespace-count", + "1", + "--pods-per-namespace", + "1", + "--node-selector", + "korvid.dev/pool=perftest", + "--output", + "seed.yaml", + ], + "error building manifests: run_id must be 1-48 lowercase letters, digits, or hyphens", + ), + ( + [ + "seed-manifests", + "--run-id", + "aks186", + "--namespace-count", + "1", + "--pods-per-namespace", + "1", + "--node-selector", + "pool", + "--output", + "seed.yaml", + ], + "error building manifests: node_selector must be exactly one non-empty key=value pair", + ), + ], +) +def test_cli_seed_manifests_reports_invalid_inputs( + arguments: list[str], + message: str, + capsys: pytest.CaptureFixture[str], +) -> None: + assert cli.main(arguments) == 1 + assert message in capsys.readouterr().err + + +def test_cli_seed_manifests_reports_file_write_errors( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + def fail_write(self: Path, _text: str, *_args: object, **_kwargs: object) -> int: + raise OSError("disk full") + + monkeypatch.setattr(Path, "write_text", fail_write) + + assert ( + cli.main( + [ + "seed-manifests", + "--run-id", + "aks186", + "--namespace-count", + "1", + "--pods-per-namespace", + "1", + "--node-selector", + "korvid.dev/pool=perftest", + "--output", + "seed.yaml", + ] + ) + == 1 + ) + assert "error writing manifests: disk full" in capsys.readouterr().err + + +def test_cli_seed_manifests_does_not_hide_unexpected_programmer_errors( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def fail_build(*_args: object, **_kwargs: object) -> object: + raise TypeError("unexpected manifest defect") + + monkeypatch.setattr(cli, "build_seed_manifests", fail_build, raising=False) + + with pytest.raises(TypeError, match="unexpected manifest defect"): + cli.main( + [ + "seed-manifests", + "--run-id", + "aks186", + "--namespace-count", + "1", + "--pods-per-namespace", + "1", + "--node-selector", + "korvid.dev/pool=perftest", + "--output", + "seed.yaml", + ] + ) diff --git a/tests/performance/test_manifests.py b/tests/performance/test_manifests.py new file mode 100644 index 00000000..128d8c55 --- /dev/null +++ b/tests/performance/test_manifests.py @@ -0,0 +1,175 @@ +from __future__ import annotations + +from collections import Counter +from typing import Any + +import pytest + +from tests.performance.manifests import build_seed_manifests + + +def _labels(manifest: dict[str, object]) -> dict[str, str]: + metadata = manifest["metadata"] + assert isinstance(metadata, dict) + labels = metadata["labels"] + assert isinstance(labels, dict) + assert all(isinstance(key, str) and isinstance(value, str) for key, value in labels.items()) + return labels + + +def _namespace_name(manifest: dict[str, object]) -> str: + metadata = manifest["metadata"] + assert isinstance(metadata, dict) + name = metadata["name"] + assert isinstance(name, str) + return name + + +def _pod_names(manifests: tuple[dict[str, object], ...]) -> list[tuple[str, str]]: + rows: list[tuple[str, str]] = [] + for manifest in manifests: + metadata = manifest["metadata"] + assert isinstance(metadata, dict) + namespace = metadata["namespace"] + name = metadata["name"] + assert isinstance(namespace, str) + assert isinstance(name, str) + rows.append((namespace, name)) + return rows + + +def _pod_spec(manifest: dict[str, object]) -> dict[str, Any]: + spec = manifest["spec"] + assert isinstance(spec, dict) + return spec + + +def test_build_seed_manifests_returns_namespaces_then_pods_in_stable_order() -> None: + manifests = build_seed_manifests( + run_id="aks186", + namespace_count=2, + pods_per_namespace=3, + node_selector="korvid.dev/pool=perftest", + ) + + namespaces = manifests[:2] + pods = manifests[2:] + + assert [manifest["kind"] for manifest in namespaces] == ["Namespace", "Namespace"] + assert [_namespace_name(manifest) for manifest in namespaces] == [ + "korvid-perf-aks186-0", + "korvid-perf-aks186-1", + ] + assert _pod_names(pods) == [ + ("korvid-perf-aks186-0", "bench-0"), + ("korvid-perf-aks186-1", "bench-0"), + ("korvid-perf-aks186-0", "bench-1"), + ("korvid-perf-aks186-1", "bench-1"), + ("korvid-perf-aks186-0", "bench-2"), + ("korvid-perf-aks186-1", "bench-2"), + ] + assert all( + _labels(manifest) + == { + "app.kubernetes.io/managed-by": "korvid-performance", + "korvid.dev/performance-run": "aks186", + } + for manifest in manifests + ) + assert _pod_spec(pods[0])["nodeSelector"] == {"korvid.dev/pool": "perftest"} + assert _pod_spec(pods[0])["tolerations"] == [ + { + "key": "purpose", + "operator": "Equal", + "value": "perftest", + "effect": "NoSchedule", + } + ] + containers = _pod_spec(pods[0])["containers"] + assert containers == [ + { + "name": "bench", + "image": "registry.k8s.io/pause:3.10", + "resources": { + "requests": {"cpu": "5m", "memory": "16Mi"}, + }, + } + ] + + +def test_build_seed_manifests_spreads_standard_live_profile_evenly() -> None: + manifests = build_seed_manifests( + run_id="aks186", + namespace_count=20, + pods_per_namespace=50, + node_selector="korvid.dev/pool=perftest", + ) + + pods = manifests[20:] + counts = Counter(namespace for namespace, _name in _pod_names(pods)) + + assert len(manifests) == 1020 + assert len(counts) == 20 + assert set(counts.values()) == {50} + assert _pod_names(pods[:4]) == [ + ("korvid-perf-aks186-0", "bench-0"), + ("korvid-perf-aks186-1", "bench-0"), + ("korvid-perf-aks186-2", "bench-0"), + ("korvid-perf-aks186-3", "bench-0"), + ] + assert _pod_names(pods[-4:]) == [ + ("korvid-perf-aks186-16", "bench-49"), + ("korvid-perf-aks186-17", "bench-49"), + ("korvid-perf-aks186-18", "bench-49"), + ("korvid-perf-aks186-19", "bench-49"), + ] + + +@pytest.mark.parametrize( + "run_id", + ["", "Aks186", "-aks186", "aks186-", "aks_186", "a" * 49], +) +def test_build_seed_manifests_rejects_invalid_run_ids(run_id: str) -> None: + with pytest.raises( + ValueError, match="run_id must be 1-48 lowercase letters, digits, or hyphens" + ): + build_seed_manifests( + run_id=run_id, + namespace_count=1, + pods_per_namespace=1, + node_selector="korvid.dev/pool=perftest", + ) + + +@pytest.mark.parametrize("selector", ["", "pool", "=perftest", "pool=", "a=b=c", "pool = perftest"]) +def test_build_seed_manifests_rejects_malformed_node_selectors(selector: str) -> None: + with pytest.raises( + ValueError, match="node_selector must be exactly one non-empty key=value pair" + ): + build_seed_manifests( + run_id="aks186", + namespace_count=1, + pods_per_namespace=1, + node_selector=selector, + ) + + +@pytest.mark.parametrize( + ("namespace_count", "pods_per_namespace", "message"), + [ + (0, 1, "namespace_count must be a positive integer"), + (1, 0, "pods_per_namespace must be a positive integer"), + ], +) +def test_build_seed_manifests_rejects_non_positive_counts( + namespace_count: int, + pods_per_namespace: int, + message: str, +) -> None: + with pytest.raises(ValueError, match=message): + build_seed_manifests( + run_id="aks186", + namespace_count=namespace_count, + pods_per_namespace=pods_per_namespace, + node_selector="korvid.dev/pool=perftest", + ) diff --git a/tests/performance/test_profile.py b/tests/performance/test_profile.py index b24fc3d8..d0f1fa9c 100644 --- a/tests/performance/test_profile.py +++ b/tests/performance/test_profile.py @@ -3,7 +3,7 @@ import pytest -from tests.performance.profile import load_profile, planned_event_count +from tests.performance.profile import Burst, load_profile, planned_event_count def _write(tmp_path: Path, **overrides: object) -> Path: @@ -53,3 +53,28 @@ def test_profile_rejects_invalid_values( ) -> None: with pytest.raises(ValueError, match=message): load_profile(_write(tmp_path, **{field: value})) + + +def test_aks_1k_profile_pins_live_topology_and_reuses_burst_schedule() -> None: + profiles_dir = Path(__file__).with_name("profiles") + + profile = load_profile(profiles_dir / "aks-1k.json") + burst = load_profile(profiles_dir / "burst-50k.json") + + assert profile.schema_version == 1 + assert profile.id == "aks-1k" + assert profile.seed == 186 + assert profile.object_count == 1000 + assert profile.namespace_count == 20 + assert profile.steady_events_per_second == 200 + assert profile.duration_seconds == 30 + assert profile.bursts == ( + Burst(start_second=5, duration_seconds=1, events_per_second=1000), + Burst(start_second=15, duration_seconds=1, events_per_second=1000), + Burst(start_second=25, duration_seconds=1, events_per_second=1000), + ) + assert profile.failures == () + assert profile.steady_events_per_second == burst.steady_events_per_second + assert profile.duration_seconds == burst.duration_seconds + assert profile.bursts == burst.bursts + assert profile.failures == burst.failures From 88526834c83c8424a6185a102dfbfebb4c946010 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 07:44:15 +0900 Subject: [PATCH 22/38] test(issue-186-task-8.2): add guarded real-AKS application-path replay Add tests/performance/live.py: run_live_replay drives the real production stack (KubeClient -> WatchManager -> ResourceStore -> MeasuredKorvidApp) against an already-seeded, uniquely-owned 20-namespace/1,000-pod AKS topology (built by manifests.build_seed_manifests). Fail-closed before any mutation: - Cluster identity gate: active kubeconfig context, resolved API-server hostname, and an independent `az aks show --ids` lookup must all agree. - Ownership gate: every expected namespace and pod must already carry both ownership labels; every mismatch is collected before raising. - Guarded churn: each mutation is a JSON-Patch that `test`s the target pod's UID and both ownership labels before `replace`-ing status.phase - a failed `test` op aborts the whole run, with no unguarded fallback. Reuses replay.py/metrics.py/workload.py/manifests.py/profile.py unchanged. Promoted namespace_name/pod_name/validate_run_id/MANAGED_BY_LABEL/ MANAGED_BY_VALUE/RUN_LABEL to public names in manifests.py so live.py shares the exact seeding contract instead of duplicating it. Extend cli.py with a `replay-live` subcommand (mandatory --profile/--context/ --expected-cluster-id/--run-id; optional --duration overriding only duration_seconds; no --time-scale - live churn always replays at real wall-clock time), reusing the same Markdown/JSON output and exit-code rules as `replay`. Add tests/performance/test_live.py (26 tests) and extend test_cli.py (+15 tests) covering: identity-gate rejections (wrong context/resource-id/ hostname/malformed JSON/nonzero exit/missing executable), topology mismatch, ownership-gate rejections, deterministic index-to-live-object mapping, guarded-patch construction, guarded-churn success/abort, namespace-filtered watch source, full happy-path digest parity, guard-failure and CancelledError propagation with client/watch teardown, and CLI argument/ error-handling coverage. Followed RED-GREEN-REFACTOR throughout; see the task report for RED evidence and exact verification results. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/cli.py | 148 ++++++ tests/performance/live.py | 571 ++++++++++++++++++++++ tests/performance/manifests.py | 33 +- tests/performance/test_cli.py | 326 +++++++++++++ tests/performance/test_live.py | 843 +++++++++++++++++++++++++++++++++ 5 files changed, 1917 insertions(+), 4 deletions(-) create mode 100644 tests/performance/live.py create mode 100644 tests/performance/test_live.py diff --git a/tests/performance/cli.py b/tests/performance/cli.py index 596aa8d0..a8542751 100644 --- a/tests/performance/cli.py +++ b/tests/performance/cli.py @@ -14,6 +14,7 @@ import argparse import asyncio import cProfile +import dataclasses import json import sys import tracemalloc @@ -23,6 +24,7 @@ import yaml from korvid.k8s.errors import ApiStatusError +from tests.performance.live import run_live_replay from tests.performance.manifests import build_seed_manifests from tests.performance.metrics import BenchmarkReport, render_markdown, report_payload from tests.performance.profile import WorkloadProfile, load_profile @@ -125,6 +127,66 @@ def _build_parser() -> argparse.ArgumentParser: metavar="PATH", help="Destination path for the multi-document YAML output.", ) + + lp = subparsers.add_parser( + "replay-live", + help="Replay churn against an already-seeded, owned real AKS cluster.", + ) + lp.add_argument("--profile", required=True, metavar="PATH", help="Workload profile JSON.") + lp.add_argument( + "--context", required=True, metavar="TEXT", help="Exact active kubeconfig context." + ) + lp.add_argument( + "--expected-cluster-id", + dest="expected_cluster_id", + required=True, + metavar="TEXT", + help="Exact AKS cluster ARM resource ID (`az aks show --ids ...`).", + ) + lp.add_argument( + "--run-id", dest="run_id", required=True, metavar="TEXT", help="Unique run identifier." + ) + lp.add_argument( + "--duration", + type=int, + default=None, + metavar="INT", + help="Override profile duration_seconds (positive); rate, bursts, " + "seed, topology, and failures are preserved.", + ) + lp.add_argument( + "--sample-interval", + type=float, + default=1.0, + metavar="FLOAT", + help="Seconds between process-memory samples (positive).", + ) + lp.add_argument( + "--json", + dest="json_path", + default=None, + metavar="PATH", + help="Write machine-readable JSON report.", + ) + lp.add_argument( + "--out", + dest="out_path", + default=None, + metavar="PATH", + help="Write Markdown report to file (also printed to stdout).", + ) + lp.add_argument( + "--cpu-profile", + default=None, + metavar="PATH", + help="Write cProfile pstats file.", + ) + lp.add_argument( + "--allocation-snapshot", + default=None, + metavar="PATH", + help="Write top-100 tracemalloc source locations.", + ) return parser @@ -143,6 +205,33 @@ def _run_with_cpu_profile( pr.dump_stats(cpu_profile_path) +def _run_live_with_cpu_profile( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, + cpu_profile_path: str, +) -> ReplayReport: + """Run *run_live_replay* and dump a cProfile pstats file to *cpu_profile_path*.""" + pr = cProfile.Profile() + pr.enable() + try: + return asyncio.run( + run_live_replay( + profile, + options, + context=context, + expected_cluster_id=expected_cluster_id, + run_id=run_id, + ) + ) + finally: + pr.disable() + pr.dump_stats(cpu_profile_path) + + def _flush_allocation_snapshot(path: str) -> None: """Take a tracemalloc snapshot and write the top 100 lines to *path*.""" if not tracemalloc.is_tracing(): @@ -223,6 +312,63 @@ def _cmd_replay(args: argparse.Namespace) -> int: return 0 +def _cmd_replay_live(args: argparse.Namespace) -> int: + if args.duration is not None and args.duration <= 0: + print("error: --duration must be positive", file=sys.stderr) + return 1 + if args.sample_interval <= 0: + print("error: --sample-interval must be positive", file=sys.stderr) + return 1 + + try: + profile = load_profile(Path(args.profile)) + except (OSError, UnicodeError, ValueError) as exc: + print(f"error loading profile: {exc}", file=sys.stderr) + return 1 + + if args.duration is not None: + profile = dataclasses.replace(profile, duration_seconds=args.duration) + + # No --time-scale option: live churn always replays at real wall-clock + # time (ReplayOptions.time_scale defaults to 1.0). + options = ReplayOptions(sample_interval=args.sample_interval) + + if args.allocation_snapshot: + tracemalloc.start() + + try: + if args.cpu_profile: + replay = _run_live_with_cpu_profile( + profile, + options, + context=args.context, + expected_cluster_id=args.expected_cluster_id, + run_id=args.run_id, + cpu_profile_path=args.cpu_profile, + ) + else: + replay = asyncio.run( + run_live_replay( + profile, + options, + context=args.context, + expected_cluster_id=args.expected_cluster_id, + run_id=args.run_id, + ) + ) + except (ValueError, ApiStatusError, AssertionError, OSError) as exc: + print(f"error during replay: {exc}", file=sys.stderr) + return 1 + finally: + if args.allocation_snapshot: + _flush_allocation_snapshot(args.allocation_snapshot) + + _write_outputs(args, replay) + if replay.dropped_updates > 0 or replay.expected_digest != replay.final_digest: + return 1 + return 0 + + def main(argv: list[str] | None = None) -> int: """CLI entry point for the large-cluster benchmark. @@ -238,6 +384,8 @@ def main(argv: list[str] | None = None) -> int: return _cmd_replay(args) if args.command == "seed-manifests": return _cmd_seed_manifests(args) + if args.command == "replay-live": + return _cmd_replay_live(args) return 1 diff --git a/tests/performance/live.py b/tests/performance/live.py new file mode 100644 index 00000000..1ebc0bf0 --- /dev/null +++ b/tests/performance/live.py @@ -0,0 +1,571 @@ +"""Guarded real-AKS application-path replay prerequisite (issue #186 task 8.2). + +`run_live_replay` drives the *real* production stack (`KubeClient` -> +`WatchManager` -> `ResourceStore` -> `MeasuredKorvidApp`) against an +already-seeded AKS cluster (20 namespaces / 1,000 Pods, created by +`tests.performance.manifests.build_seed_manifests` and the `seed-manifests` +CLI subcommand). This module never provisions, tears down, or discovers a +cluster on its own - it only replays churn against a topology a human has +already created and identified explicitly via `--context`, +`--expected-cluster-id`, and `--run-id`. + +Every mutation is fail-closed: + +1. **Cluster identity gate** (`_verify_cluster_identity`): the active + kubeconfig context, its resolved API server hostname, and an independent + `az aks show --ids ` lookup must all agree before any client connects. +2. **Ownership gate** (`_verify_ownership`): every expected namespace and + every expected Pod must already carry both ownership labels + (`manifests.MANAGED_BY_LABEL`/`manifests.RUN_LABEL`) before any churn is + attempted. +3. **Guarded churn** (`drive_live_churn`): each mutation is a JSON-Patch that + `test`s the target Pod's UID and both ownership labels before `replace`ing + `status.phase`. A failed `test` op aborts the *entire* run - there is no + unguarded fallback. + +Unit tests substitute every external boundary via `LiveDependencies`; none of +them may contact Azure, a real kubeconfig, or a real cluster. +""" + +from __future__ import annotations + +import asyncio +import json +from collections.abc import AsyncIterator, Awaitable, Callable, Iterable +from dataclasses import dataclass, replace +from time import monotonic +from typing import Any, Protocol, cast +from urllib.parse import urlparse + +from kubernetes_asyncio import client as k8s_client +from kubernetes_asyncio import config as k8s_config + +from korvid.core.config import KorvidConfig +from korvid.core.store import ALL_NAMESPACES, ResourceStore +from korvid.core.watch import WatchManager, WatchSource +from korvid.k8s.client import KubeClient, resolve_context_name +from korvid.k8s.discovery import ResourceMeta +from korvid.k8s.errors import ApiStatusError +from korvid.k8s.models import GenericSummary, PodSummary +from korvid.k8s.telemetry import ReadTelemetry +from korvid.ui.widgets.resource_table import ResourceTable +from tests.performance import manifests +from tests.performance.metrics import BenchmarkRecorder, ProcessSampler +from tests.performance.profile import WorkloadProfile +from tests.performance.replay import MeasuredKorvidApp, ReplayOptions, ReplayReport, _build_manifest +from tests.performance.workload import ScheduledEvent, scheduled_events, summary_digest +from tests.ui.waits import until + +#: Namespace-scoped read for the ownership gate; not exposed via `PODS_META` +#: because Namespaces are cluster-scoped (`namespaced=False`). +_NAMESPACES_META = ResourceMeta("Namespace", "namespaces", "", "v1", False) + +#: Live topology is pinned to the aks-1k profile shape (issue #186 Task 8): +#: exactly 20 namespaces of 50 Pods each, matching `seed-manifests`'s output. +_REQUIRED_OBJECT_COUNT = 1000 +_REQUIRED_NAMESPACE_COUNT = 20 + + +async def _sleep_default(delay: float) -> None: + await asyncio.sleep(delay) + + +@dataclass(frozen=True) +class CommandResult: + """Captured outcome of a subprocess invocation (e.g. `az aks show`).""" + + exit_code: int + stdout: str + stderr: str + + +#: Runs an argv list and captures its result; the identity gate's only +#: subprocess seam. Production uses `_default_command_runner`; tests inject +#: a fake that never touches a real `az` binary. +CommandRunner = Callable[[list[str]], Awaitable[CommandResult]] + + +class KubeReadClient(Protocol): + """Structural subset of `KubeClient` the live read path needs. + + Deliberately narrow (vs. requiring the full `KubeClient`) so unit tests + can implement it with a plain in-memory fake instead of subclassing the + real, network-talking client. + """ + + async def connect(self, context: str | None = None) -> None: ... + + async def close(self) -> None: ... + + async def list_objects( + self, meta: ResourceMeta, namespace: str | None + ) -> list[GenericSummary]: ... + + async def list_pods(self, namespace: str) -> list[PodSummary]: ... + + def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSummary]]: ... + + +class MutationClient(Protocol): + """Issues one guarded status mutation; production talks JSON-Patch to a + real API server, tests mutate an in-memory fake cluster the same way.""" + + async def patch_pod_status_guarded( + self, namespace: str, name: str, *, uid: str, phase: str + ) -> None: ... + + async def close(self) -> None: ... + + +@dataclass(frozen=True) +class LiveDependencies: + """Every external boundary `run_live_replay` crosses, as an injectable + seam. Production callers get real wiring from `_default_dependencies`; + tests always construct this explicitly with fakes, so no unit test can + reach Azure, a real kubeconfig, or a real cluster.""" + + command_runner: CommandRunner + active_context: Callable[[], str | None] + context_host: Callable[[str], Awaitable[str]] + kube_client_factory: Callable[[ReadTelemetry], KubeReadClient] + mutation_client_factory: Callable[[str], MutationClient] + + +async def _default_command_runner(args: list[str]) -> CommandResult: + """Run *args* as a subprocess and capture stdout/stderr/exit code. + + A missing executable (e.g. `az` not on PATH) surfaces as a `ValueError` + so it fails the identity gate cleanly instead of raising an uncaught + `FileNotFoundError` deep inside `run_live_replay`. + """ + try: + proc = await asyncio.create_subprocess_exec( + *args, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + except FileNotFoundError as exc: + raise ValueError(f"executable not found: {args[0]!r} ({exc})") from exc + stdout_bytes, stderr_bytes = await proc.communicate() + return CommandResult( + proc.returncode if proc.returncode is not None else 1, + stdout_bytes.decode("utf-8", errors="replace"), + stderr_bytes.decode("utf-8", errors="replace"), + ) + + +def _default_active_context() -> str | None: + return resolve_context_name() + + +async def _default_context_host(context: str) -> str: + """Resolve the API server hostname *context* would dial. + + Loads the kubeconfig into a private `Configuration` (never persisted, + never the global default) - mirroring `KubeClient.probe_context`'s + isolation pattern - so resolving identity never disturbs any live + connection. + """ + configuration = k8s_client.Configuration() + await k8s_config.load_kube_config( + context=context, client_configuration=configuration, persist_config=False + ) + hostname = urlparse(configuration.host or "").hostname + if not hostname: + raise ValueError(f"could not resolve API server hostname for context {context!r}") + return hostname + + +def _json_pointer_escape(segment: str) -> str: + """Escape one JSON-Pointer (RFC 6901) reference token.""" + return segment.replace("~", "~0").replace("/", "~1") + + +def build_guarded_status_patch(*, uid: str, run_id: str, phase: str) -> list[dict[str, Any]]: + """The exact JSON-Patch op list a guarded status mutation issues. + + `test`s the target Pod's UID and both ownership labels before + `replace`-ing `status.phase`, so a stale, foreign, or replaced Pod aborts + the whole patch server-side - there is no unguarded fallback. + """ + managed_by_path = f"/metadata/labels/{_json_pointer_escape(manifests.MANAGED_BY_LABEL)}" + run_path = f"/metadata/labels/{_json_pointer_escape(manifests.RUN_LABEL)}" + return [ + {"op": "test", "path": "/metadata/uid", "value": uid}, + {"op": "test", "path": managed_by_path, "value": manifests.MANAGED_BY_VALUE}, + {"op": "test", "path": run_path, "value": run_id}, + {"op": "replace", "path": "/status/phase", "value": phase}, + ] + + +class _KubeMutationClient: + """Production `MutationClient`: issues a guarded JSON-Patch against the + real `pods/status` subresource of *context*.""" + + def __init__(self, context: str, run_id: str) -> None: + self._context = context + self._run_id = run_id + self._api: k8s_client.ApiClient | None = None + self._core_v1: k8s_client.CoreV1Api | None = None + + async def _ensure_connected(self) -> k8s_client.CoreV1Api: + if self._core_v1 is None: + configuration = k8s_client.Configuration() + await k8s_config.load_kube_config( + context=self._context, client_configuration=configuration, persist_config=False + ) + self._api = k8s_client.ApiClient(configuration) + self._core_v1 = k8s_client.CoreV1Api(self._api) + return self._core_v1 + + async def patch_pod_status_guarded( + self, namespace: str, name: str, *, uid: str, phase: str + ) -> None: + core_v1 = await self._ensure_connected() + ops = build_guarded_status_patch(uid=uid, run_id=self._run_id, phase=phase) + try: + await core_v1.patch_namespaced_pod_status( + name, + namespace, + ops, + _content_type="application/json-patch+json", # type: ignore[call-arg] # kubernetes_asyncio's .pyi stub omits _content_type; accepted via **kwargs at runtime + ) + except k8s_client.exceptions.ApiException as exc: + raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc + + async def close(self) -> None: + if self._api is not None: + await self._api.close() + + +def _default_dependencies(context: str) -> LiveDependencies: + """Real production wiring: subprocess `az`, real kubeconfig, real `KubeClient`.""" + return LiveDependencies( + command_runner=_default_command_runner, + active_context=_default_active_context, + context_host=_default_context_host, + kube_client_factory=lambda read_telemetry: KubeClient(read_telemetry=read_telemetry), + mutation_client_factory=lambda run_id: _KubeMutationClient(context, run_id), + ) + + +def live_object_identity(run_id: str, namespace_count: int, index: int) -> tuple[str, str]: + """Map a synthetic churn-schedule Pod index onto the exact seeded + `(namespace, name)` `manifests.build_seed_manifests` created for it - + guaranteeing seeding and live churn always agree on identity.""" + namespace = manifests.namespace_name(run_id, index % namespace_count) + name = manifests.pod_name(namespace_count, index) + return namespace, name + + +def _owns(labels: Iterable[tuple[str, str]], run_id: str) -> bool: + mapping = dict(labels) + return ( + mapping.get(manifests.MANAGED_BY_LABEL) == manifests.MANAGED_BY_VALUE + and mapping.get(manifests.RUN_LABEL) == run_id + ) + + +def _validate_time_scale(options: ReplayOptions) -> None: + if options.time_scale != 1.0: + raise ValueError( + f"run_live_replay requires options.time_scale == 1.0 (real wall time), " + f"got {options.time_scale!r}" + ) + + +def _validate_topology(profile: WorkloadProfile) -> None: + if profile.object_count != _REQUIRED_OBJECT_COUNT: + raise ValueError( + f"live replay requires object_count == {_REQUIRED_OBJECT_COUNT}, " + f"got {profile.object_count}" + ) + if profile.namespace_count != _REQUIRED_NAMESPACE_COUNT: + raise ValueError( + f"live replay requires namespace_count == {_REQUIRED_NAMESPACE_COUNT}, " + f"got {profile.namespace_count}" + ) + + +async def _verify_cluster_identity( + *, context: str, expected_cluster_id: str, deps: LiveDependencies +) -> None: + """Fail-closed 5-step cluster identity gate; every failure raises + `ValueError` before any client is constructed or any mutation attempted. + """ + active = deps.active_context() + if active != context: + raise ValueError( + f"active kubeconfig context {active!r} does not match required context {context!r}" + ) + + hostname = await deps.context_host(context) + + result = await deps.command_runner( + ["az", "aks", "show", "--ids", expected_cluster_id, "-o", "json"] + ) + if result.exit_code != 0: + raise ValueError(f"az aks show failed (exit {result.exit_code}): {result.stderr.strip()}") + try: + payload = json.loads(result.stdout) + except json.JSONDecodeError as exc: + raise ValueError(f"az aks show returned malformed JSON: {exc}") from exc + if not isinstance(payload, dict): + raise ValueError("az aks show returned malformed JSON: expected a JSON object") + + resource_id = payload.get("id") + if resource_id != expected_cluster_id: + raise ValueError( + f"az aks show returned id {resource_id!r}, expected {expected_cluster_id!r}" + ) + + fqdn = payload.get("fqdn") or "" + private_fqdn = payload.get("privateFqdn") or "" + expected_hostname = fqdn or private_fqdn + if not expected_hostname: + raise ValueError("az aks show returned neither fqdn nor privateFqdn") + if hostname != expected_hostname: + raise ValueError( + f"context {context!r} API hostname {hostname!r} does not match " + f"cluster hostname {expected_hostname!r}" + ) + + +async def _verify_ownership( + kube: KubeReadClient, *, run_id: str, namespace_count: int, object_count: int +) -> None: + """Ownership gate: every expected namespace and every expected Pod must + already exist with both ownership labels, checked before any churn. + + Collects every mismatch across all namespaces/Pods before raising, so a + single failed run surfaces the full blast radius at once instead of + stopping at the first namespace. + """ + expected_namespaces = [manifests.namespace_name(run_id, i) for i in range(namespace_count)] + actual_namespaces = {obj.name: obj for obj in await kube.list_objects(_NAMESPACES_META, None)} + + missing_namespaces = [name for name in expected_namespaces if name not in actual_namespaces] + if missing_namespaces: + raise ValueError(f"missing expected namespaces: {', '.join(missing_namespaces)}") + + mismatched_namespaces = [ + name for name in expected_namespaces if not _owns(actual_namespaces[name].labels, run_id) + ] + if mismatched_namespaces: + raise ValueError( + f"namespaces missing/mismatched ownership labels: {', '.join(mismatched_namespaces)}" + ) + + pods_per_namespace = object_count // namespace_count + expected_pod_names = [ + manifests.pod_name(namespace_count, local_index * namespace_count) + for local_index in range(pods_per_namespace) + ] + missing_pods: list[str] = [] + mismatched_pods: list[str] = [] + for namespace in expected_namespaces: + pods_by_name = {pod.name: pod for pod in await kube.list_pods(namespace)} + for name in expected_pod_names: + pod = pods_by_name.get(name) + if pod is None: + missing_pods.append(f"{namespace}/{name}") + elif not _owns(pod.labels, run_id): + mismatched_pods.append(f"{namespace}/{name}") + if missing_pods: + raise ValueError(f"missing expected pods: {', '.join(missing_pods)}") + if mismatched_pods: + raise ValueError(f"pods with mismatched ownership labels: {', '.join(mismatched_pods)}") + + +def make_live_watch_source( + kube: KubeReadClient, expected_namespaces: frozenset[str] +) -> WatchSource: + """Filter the real cluster-wide Pod watch to exactly the expected, + seeded namespaces - unrelated cluster Pods (or namespaces sharing the + cluster with this run) must never enter the benchmark store.""" + + async def _source(kind: str, _scope: str) -> AsyncIterator[tuple[str, PodSummary]]: + if kind != "pods": + raise ValueError(f"run_live_replay only watches pods, got kind={kind!r}") + async for event_type, pod in kube.watch_pods(None): + if pod.namespace in expected_namespaces: + yield (event_type, pod) + + return _source + + +async def drive_live_churn( + events: Iterable[ScheduledEvent], + *, + run_id: str, + namespace_count: int, + live_state: dict[tuple[str, str], PodSummary], + mutation_client: MutationClient, + recorder: BenchmarkRecorder, + options: ReplayOptions, +) -> None: + """Drive guarded churn at wall-clock time (matching `_ReplaySource`'s + inter-event delay math). Any guard failure (`ApiStatusError`) propagates + immediately and unconditionally aborts the run - there is no unguarded + fallback and no attempt to continue past a failed `test` op. + """ + now = options.monotonic_fn if options.monotonic_fn is not None else monotonic + sleep = options.async_sleep if options.async_sleep is not None else _sleep_default + start = now() + for event in events: + elapsed = now() - start + delay = event.offset_seconds * options.time_scale - elapsed + if delay > 0: + await sleep(delay) + + index = int(event.summary.name.removeprefix("pod-")) + namespace, name = live_object_identity(run_id, namespace_count, index) + current = live_state.get((namespace, name)) + if current is None: + raise ValueError(f"live churn target {namespace}/{name} is not in the seeded state") + + await mutation_client.patch_pod_status_guarded( + namespace, name, uid=current.uid, phase=event.summary.phase + ) + live_state[(namespace, name)] = replace(current, phase=event.summary.phase) + recorder.record_event(event.sequence, now()) + + +async def run_live_replay( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, + deps: LiveDependencies | None = None, +) -> ReplayReport: + """Replay churn against an already-seeded real AKS cluster and return metrics. + + Fail-closed order: `time_scale`/`run_id`/topology validation, the cluster + identity gate, the ownership gate - all *before* any mutation - then the + real application-path wiring (`KubeClient` -> `WatchManager` -> + `ResourceStore` -> `MeasuredKorvidApp`), guarded churn, and digest parity + against an independent post-churn re-read of the cluster. + """ + _validate_time_scale(options) + manifests.validate_run_id(run_id) + _validate_topology(profile) + + active_deps = deps if deps is not None else _default_dependencies(context) + await _verify_cluster_identity( + context=context, expected_cluster_id=expected_cluster_id, deps=active_deps + ) + + store = ResourceStore() + recorder = BenchmarkRecorder() + sampler = ProcessSampler(options.sample_interval) + kube = active_deps.kube_client_factory(recorder.record_api) + + await kube.connect(context) + try: + await _verify_ownership( + kube, + run_id=run_id, + namespace_count=profile.namespace_count, + object_count=profile.object_count, + ) + + expected_namespaces = frozenset( + manifests.namespace_name(run_id, i) for i in range(profile.namespace_count) + ) + source = make_live_watch_source(kube, expected_namespaces) + watch_manager = WatchManager(store, source, retry_delay=0.0) + manifest = _build_manifest(profile) + mutation_client = active_deps.mutation_client_factory(run_id) + + app = MeasuredKorvidApp( + config=KorvidConfig(namespace=ALL_NAMESPACES), + store=store, + watch_manager=watch_manager, + recorder=recorder, + ) + + # Snapshot Pod uids once, before churn, per namespace: guarded + # patches test against the uid observed at ownership-gate time, so a + # concurrently replaced Pod fails its `test` op instead of silently + # patching a different object. + live_state: dict[tuple[str, str], PodSummary] = {} + for namespace in expected_namespaces: + for pod in await kube.list_pods(namespace): + live_state[(namespace, pod.name)] = pod + + sampler.start() + churn_started_before_input = False + try: + async with app.run_test() as pilot: + table = app.query_one(ResourceTable) + + await until( + pilot, + lambda: table.row_count == profile.object_count, + timeout=60.0, + label="initial owned pods rendered", + ) + + events = scheduled_events(profile) + churn_task = asyncio.create_task( + drive_live_churn( + events, + run_id=run_id, + namespace_count=profile.namespace_count, + live_state=live_state, + mutation_client=mutation_client, + recorder=recorder, + options=options, + ) + ) + churn_started_before_input = True + + t0 = monotonic() + await pilot.press("down") + recorder.record_input(monotonic() - t0) + t0 = monotonic() + await pilot.press("up") + recorder.record_input(monotonic() - t0) + + await churn_task + + await until( + pilot, + lambda: not recorder._pending_events, + timeout=60.0, + label="churn complete and all events rendered", + ) + finally: + process_samples = await sampler.stop() + await watch_manager.stop_all() + await mutation_client.close() + + # Independently re-read the cluster's actual Pods for ground-truth + # digest parity, rather than trusting the driver's own bookkeeping. + final_pods: list[PodSummary] = [] + for namespace in expected_namespaces: + final_pods.extend(await kube.list_pods(namespace)) + expected_digest = summary_digest(final_pods) + final_digest = summary_digest(cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES))) + finally: + await kube.close() + + benchmark = recorder.report(manifest, process_samples, final_digest=final_digest) + + return ReplayReport( + object_count=profile.object_count, + expected_digest=expected_digest, + final_digest=final_digest, + dropped_updates=benchmark.dropped_updates, + rendered_updates=benchmark.rendered_updates, + render_passes=benchmark.render_passes, + coalesced_updates=benchmark.coalesced_updates, + event_to_render=benchmark.event_to_render, + input_latency=benchmark.input_latency, + churn_started_before_input=churn_started_before_input, + process=benchmark.process, + api=benchmark.api, + manifest=benchmark.manifest, + ) diff --git a/tests/performance/manifests.py b/tests/performance/manifests.py index 1e7a79bb..5b26c02e 100644 --- a/tests/performance/manifests.py +++ b/tests/performance/manifests.py @@ -8,6 +8,12 @@ _MANAGED_BY = "korvid-performance" _BENCH_IMAGE = "registry.k8s.io/pause:3.10" +#: Public ownership-label contract shared with `live.py`'s ownership gate, so +#: both modules agree on exactly what "owned by this seed run" means. +MANAGED_BY_LABEL = "app.kubernetes.io/managed-by" +MANAGED_BY_VALUE = _MANAGED_BY +RUN_LABEL = "korvid.dev/performance-run" + def _validate_positive(value: int, label: str) -> int: if value < 1: @@ -21,6 +27,13 @@ def _validate_run_id(run_id: str) -> str: raise ValueError("run_id must be 1-48 lowercase letters, digits, or hyphens") +def validate_run_id(run_id: str) -> str: + """Public entry point so other benchmark modules (e.g. `live.py`) share + the exact `run_id` contract `seed-manifests` enforces, without importing + a private name across modules.""" + return _validate_run_id(run_id) + + def _parse_node_selector(node_selector: str) -> dict[str, str]: if node_selector.count("=") != 1: raise ValueError("node_selector must be exactly one non-empty key=value pair") @@ -36,8 +49,8 @@ def _parse_node_selector(node_selector: str) -> dict[str, str]: def _common_labels(run_id: str) -> dict[str, str]: return { - "app.kubernetes.io/managed-by": _MANAGED_BY, - "korvid.dev/performance-run": run_id, + MANAGED_BY_LABEL: _MANAGED_BY, + RUN_LABEL: run_id, } @@ -48,6 +61,19 @@ def _namespace_name(run_id: str, namespace_index: int) -> str: return name +def namespace_name(run_id: str, namespace_index: int) -> str: + """Public entry point for the exact namespace-naming formula + `seed-manifests` uses, so `live.py` maps object indices onto the same + namespaces without duplicating (and risking drift from) this formula.""" + return _namespace_name(run_id, namespace_index) + + +def pod_name(namespace_count: int, object_index: int) -> str: + """Public entry point for the exact Pod-naming formula `seed-manifests` + uses; see `namespace_name`.""" + return f"bench-{object_index // namespace_count}" + + def build_seed_manifests( *, run_id: str, @@ -77,13 +103,12 @@ def build_seed_manifests( for object_index in range(namespaces * pods_each): namespace_index = object_index % namespaces - pod_name = f"bench-{object_index // namespaces}" manifests.append( { "apiVersion": "v1", "kind": "Pod", "metadata": { - "name": pod_name, + "name": pod_name(namespaces, object_index), "namespace": namespace_names[namespace_index], "labels": dict(labels), }, diff --git a/tests/performance/test_cli.py b/tests/performance/test_cli.py index b12cb475..573a708c 100644 --- a/tests/performance/test_cli.py +++ b/tests/performance/test_cli.py @@ -3,6 +3,7 @@ from __future__ import annotations import json +from collections.abc import Awaitable, Callable from dataclasses import replace from pathlib import Path from types import MappingProxyType @@ -181,6 +182,76 @@ async def fake_programmer_error( raise TypeError("unexpected replay defect") +# --------------------------------------------------------------------------- +# Fake coroutines for `replay-live` (mirrors `run_replay`'s keyword-only +# identity arguments). +# --------------------------------------------------------------------------- + + +def _make_recording_live_replay( + calls: list[dict[str, object]], +) -> Callable[..., Awaitable[ReplayReport]]: + """Build a fake `run_live_replay` that records every call's arguments into + *calls* (a fresh list per test, to stay independent of execution order).""" + + async def fake( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, + ) -> ReplayReport: + calls.append( + { + "profile": profile, + "options": options, + "context": context, + "expected_cluster_id": expected_cluster_id, + "run_id": run_id, + } + ) + return await fake_run_replay(profile, options) + + return fake + + +async def fake_live_failed_report( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, +) -> ReplayReport: + """Failed live replay — digest mismatch simulates store corruption.""" + return await fake_failed_report(profile, options) + + +async def fake_live_api_failure( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, +) -> ReplayReport: + """Expected live replay failure surfaced by a fail-closed gate.""" + raise ValueError("wrong active context: expected aks-context, got other-context") + + +async def fake_live_programmer_error( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, +) -> ReplayReport: + """Unexpected implementation error that must retain its traceback.""" + raise TypeError("unexpected live replay defect") + + # --------------------------------------------------------------------------- # Tests # --------------------------------------------------------------------------- @@ -395,3 +466,258 @@ def fail_build(*_args: object, **_kwargs: object) -> object: "seed.yaml", ] ) + + +# --------------------------------------------------------------------------- +# `replay-live` +# --------------------------------------------------------------------------- + +_LIVE_IDENTITY_ARGS = [ + "--context", + "aks-context", + "--expected-cluster-id", + "/subscriptions/sub/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters/aks", + "--run-id", + "aks186", +] + + +def test_cli_replay_live_writes_json_and_markdown( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + json_path = tmp_path / "result.json" + markdown_path = tmp_path / "result.md" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + result = cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + "--json", + str(json_path), + "--out", + str(markdown_path), + ] + ) + assert result == 0 + assert json.loads(json_path.read_text())["schema_version"] == 1 + assert "# Large-cluster benchmark" in markdown_path.read_text() + assert len(calls) == 1 + assert calls[0]["context"] == "aks-context" + assert calls[0]["run_id"] == "aks186" + assert calls[0]["expected_cluster_id"] == _LIVE_IDENTITY_ARGS[3] + # No --time-scale option: production real-time replay always uses 1.0. + assert calls[0]["options"].time_scale == 1.0 # type: ignore[attr-defined] + + +@pytest.mark.parametrize( + "missing_arguments", + [ + [], + ["--context", "aks-context"], + ["--expected-cluster-id", "/subscriptions/sub/x"], + ["--run-id", "aks186"], + ], +) +def test_cli_replay_live_requires_identity_arguments( + tmp_path: Path, missing_arguments: list[str] +) -> None: + """Every one of --context/--expected-cluster-id/--run-id is mandatory; + dropping any one of them (while supplying the others is exercised by the + full identity-args fixture in other tests) must fail argument parsing.""" + with pytest.raises(SystemExit): + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *missing_arguments, + ] + ) + + +def test_cli_replay_live_rejects_time_scale_option(tmp_path: Path) -> None: + """`replay-live` must never accept `--time-scale`: live churn always + replays at real wall-clock time (`ReplayOptions.time_scale == 1.0`).""" + with pytest.raises(SystemExit): + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + "--time-scale", + "0", + ] + ) + + +def test_cli_replay_live_rejects_non_positive_duration( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + assert ( + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + "--duration", + "0", + ] + ) + == 1 + ) + assert "--duration must be positive" in capsys.readouterr().err + + +def test_cli_replay_live_duration_overrides_profile_duration_only( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """`--duration` must override only `duration_seconds`, preserving the + profile's rate, bursts, seed, topology, and failures untouched.""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + result = cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + "--duration", + "5", + ] + ) + assert result == 0 + used_profile = calls[0]["profile"] + assert isinstance(used_profile, WorkloadProfile) + original = _make_minimal_profile() + assert used_profile.duration_seconds == 5 + assert used_profile.steady_events_per_second == original.steady_events_per_second + assert used_profile.bursts == original.bursts + assert used_profile.seed == original.seed + assert used_profile.object_count == original.object_count + assert used_profile.namespace_count == original.namespace_count + assert used_profile.failures == original.failures + + +def test_cli_replay_live_rejects_non_positive_sample_interval(tmp_path: Path) -> None: + assert ( + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + "--sample-interval", + "0", + ] + ) + == 1 + ) + + +@pytest.mark.parametrize( + "failed_replay", + [fake_live_failed_report], +) +def test_cli_replay_live_returns_nonzero_for_digest_failure( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + failed_replay: object, +) -> None: + monkeypatch.setattr(cli, "run_live_replay", failed_replay) + assert ( + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + ] + ) + == 1 + ) + + +def test_cli_replay_live_reports_expected_operational_errors( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + monkeypatch.setattr(cli, "run_live_replay", fake_live_api_failure) + assert ( + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + ] + ) + == 1 + ) + assert "error during replay: wrong active context" in capsys.readouterr().err + + +def test_cli_replay_live_does_not_hide_unexpected_programmer_errors( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(cli, "run_live_replay", fake_live_programmer_error) + with pytest.raises(TypeError, match="unexpected live replay defect"): + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + ] + ) + + +def test_cli_replay_live_does_not_hide_unexpected_profile_errors( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def fail_profile(_path: Path) -> WorkloadProfile: + raise TypeError("unexpected profile defect") + + monkeypatch.setattr(cli, "load_profile", fail_profile) + with pytest.raises(TypeError, match="unexpected profile defect"): + cli.main( + [ + "replay-live", + "--profile", + "profile.json", + *_LIVE_IDENTITY_ARGS, + ] + ) + + +def test_replay_and_seed_manifests_commands_still_work( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Adding `replay-live` must not disturb the pre-existing subcommands.""" + monkeypatch.setattr(cli, "run_replay", fake_run_replay) + assert cli.main(["replay", "--profile", str(profile_path(tmp_path)), "--time-scale", "0"]) == 0 + + output_path = tmp_path / "seed.yaml" + assert ( + cli.main( + [ + "seed-manifests", + "--run-id", + "aks186", + "--namespace-count", + "1", + "--pods-per-namespace", + "1", + "--node-selector", + "korvid.dev/pool=perftest", + "--output", + str(output_path), + ] + ) + == 0 + ) diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py new file mode 100644 index 00000000..218ba12d --- /dev/null +++ b/tests/performance/test_live.py @@ -0,0 +1,843 @@ +"""Guarded real-AKS application-path replay prerequisite tests (issue #186 task 8.2). + +Every test substitutes the identity/kubeconfig/subprocess/KubeClient seams +(`LiveDependencies`) with fakes: no test here may contact Azure, a real +kubeconfig, or a real cluster (see `live.py`'s module docstring). +""" + +from __future__ import annotations + +import asyncio +import dataclasses +import json +import re +from collections.abc import AsyncGenerator, AsyncIterator, Awaitable, Callable +from typing import Any + +import pytest + +from korvid.core.store import Summary +from korvid.k8s.discovery import ResourceMeta +from korvid.k8s.errors import ApiStatusError +from korvid.k8s.models import GenericSummary, PodSummary +from korvid.k8s.telemetry import ReadTelemetry, ReadTelemetryEvent +from tests.performance import live, manifests +from tests.performance.live import ( + CommandResult, + LiveDependencies, + build_guarded_status_patch, + drive_live_churn, + live_object_identity, + make_live_watch_source, + run_live_replay, +) +from tests.performance.manifests import build_seed_manifests +from tests.performance.metrics import BenchmarkRecorder +from tests.performance.profile import WorkloadProfile +from tests.performance.replay import ReplayOptions +from tests.performance.workload import scheduled_events, summary_digest + +RUN_ID = "aks186" +CONTEXT = "aks-korvid-perf" +SUBSCRIPTION = "00000000-0000-0000-0000-000000000000" +CLUSTER_ID = ( + f"/subscriptions/{SUBSCRIPTION}/resourceGroups/rg" + "/providers/Microsoft.ContainerService/managedClusters/aks-korvid-perf" +) +FQDN = "aks-korvid-perf-dns-abc123.hcp.eastus.azmk8s.io" + +# --------------------------------------------------------------------------- +# Fixtures / fakes +# --------------------------------------------------------------------------- + + +def _labels(run_id: str) -> tuple[tuple[str, str], ...]: + return ( + (manifests.MANAGED_BY_LABEL, manifests.MANAGED_BY_VALUE), + (manifests.RUN_LABEL, run_id), + ) + + +def _build_fake_topology( + run_id: str, namespace_count: int, object_count: int +) -> tuple[dict[str, GenericSummary], dict[tuple[str, str], PodSummary]]: + namespaces = { + manifests.namespace_name(run_id, i): GenericSummary( + name=manifests.namespace_name(run_id, i), + namespace="", + kind="Namespace", + created="2024-01-01T00:00:00Z", + uid=f"ns-uid-{i}", + labels=_labels(run_id), + ) + for i in range(namespace_count) + } + pods: dict[tuple[str, str], PodSummary] = {} + for index in range(object_count): + namespace, name = live_object_identity(run_id, namespace_count, index) + pods[(namespace, name)] = PodSummary( + name=name, + namespace=namespace, + phase="Running", + ready="1/1", + restarts=0, + node="node-0", + uid=f"pod-uid-{index}", + labels=_labels(run_id), + ) + return namespaces, pods + + +class _FakeKubeClient: + """Fake `KubeReadClient`: an in-memory cluster the fake mutation client + mutates directly, so the watch stream really observes guarded patches - + exactly like a real API server + watch, with zero network I/O.""" + + def __init__( + self, + read_telemetry: ReadTelemetry | None, + namespaces: dict[str, GenericSummary], + pods: dict[tuple[str, str], PodSummary], + *, + distractor_pods: tuple[PodSummary, ...] = (), + ) -> None: + self.read_telemetry = read_telemetry + self.namespaces = namespaces + self.pods = pods + self.distractor_pods = distractor_pods + self.connect_context: str | None = "__not_connected__" + self.closed = False + self.events: asyncio.Queue[tuple[str, PodSummary]] = asyncio.Queue() + + async def connect(self, context: str | None = None) -> None: + self.connect_context = context + + async def close(self) -> None: + self.closed = True + + async def list_objects(self, meta: ResourceMeta, namespace: str | None) -> list[GenericSummary]: + assert meta.kind == "Namespace" + assert namespace is None + return list(self.namespaces.values()) + + async def list_pods(self, namespace: str) -> list[PodSummary]: + return [pod for (ns, _name), pod in self.pods.items() if ns == namespace] + + async def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSummary]]: + assert namespace is None + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("list", "/api/v1/pods")) + for pod in self.pods.values(): + yield ("ADDED", pod) + for pod in self.distractor_pods: + yield ("ADDED", pod) + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("watch_open", "/api/v1/pods")) + while True: + event = await self.events.get() + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("watch_event", "/api/v1/pods")) + yield event + + +class _FakeMutationClient: + """Fake `MutationClient`: applies the guard checks a real JSON-Patch + `test` op would enforce, then mutates the shared fake cluster and wakes + the fake watch - so a guard failure here is exactly as fatal as a real + 412/422 from the API server.""" + + def __init__(self, kube: _FakeKubeClient, run_id: str) -> None: + self._kube = kube + self._run_id = run_id + self.calls: list[tuple[str, str, str]] = [] + self.closed = False + + async def patch_pod_status_guarded( + self, namespace: str, name: str, *, uid: str, phase: str + ) -> None: + self.calls.append((namespace, name, phase)) + current = self._kube.pods.get((namespace, name)) + labels = dict(current.labels) if current is not None else {} + guard_ok = ( + current is not None + and current.uid == uid + and labels.get(manifests.MANAGED_BY_LABEL) == manifests.MANAGED_BY_VALUE + and labels.get(manifests.RUN_LABEL) == self._run_id + ) + if not guard_ok or current is None: + raise ApiStatusError(422, "test operation failed for guarded patch") + updated = dataclasses.replace(current, phase=phase) + self._kube.pods[(namespace, name)] = updated + self._kube.events.put_nowait(("MODIFIED", updated)) + + async def close(self) -> None: + self.closed = True + + +def _never_called(label: str) -> Callable[..., Any]: + def _fail(*_args: object, **_kwargs: object) -> Any: + raise AssertionError(f"{label} must not be called") + + return _fail + + +def _ok_command_runner( + *, cluster_id: str = CLUSTER_ID, fqdn: str = FQDN, private_fqdn: str = "" +) -> Callable[[Any], Awaitable[CommandResult]]: + async def _run(_args: Any) -> CommandResult: + payload = {"id": cluster_id, "fqdn": fqdn, "privateFqdn": private_fqdn} + return CommandResult(0, json.dumps(payload), "") + + return _run + + +def _happy_deps( + kube: _FakeKubeClient, + run_id: str, + *, + mutation_clients: list[_FakeMutationClient] | None = None, + mutation_client_factory: Callable[[str], Any] | None = None, +) -> LiveDependencies: + async def context_host(context: str) -> str: + assert context == CONTEXT + return FQDN + + def default_mutation_factory(run_id_arg: str) -> _FakeMutationClient: + client = _FakeMutationClient(kube, run_id_arg) + if mutation_clients is not None: + mutation_clients.append(client) + return client + + def kube_factory(read_telemetry: ReadTelemetry) -> _FakeKubeClient: + kube.read_telemetry = read_telemetry + return kube + + return LiveDependencies( + command_runner=_ok_command_runner(), + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=kube_factory, + mutation_client_factory=mutation_client_factory or default_mutation_factory, + ) + + +def _virtual_clock() -> tuple[Callable[[], float], Callable[[float], Awaitable[None]]]: + virtual_time = [0.0] + + def monotonic_fn() -> float: + return virtual_time[0] + + async def async_sleep(delay: float) -> None: + virtual_time[0] += delay + await asyncio.sleep(0) + + return monotonic_fn, async_sleep + + +def _tiny_live_profile(*, seed: int = 1) -> WorkloadProfile: + """A profile satisfying the mandatory 1,000/20 live topology with a + minimal churn schedule (2 events), kept fast via the virtual clock.""" + return WorkloadProfile( + schema_version=1, + id="live-test", + seed=seed, + object_count=1000, + namespace_count=20, + steady_events_per_second=2, + duration_seconds=1, + bursts=(), + failures=(), + ) + + +# --------------------------------------------------------------------------- +# Deterministic mapping +# --------------------------------------------------------------------------- + + +def test_live_object_identity_matches_seed_manifest_layout() -> None: + run_id = "run1" + namespace_count = 4 + pods_per_namespace = 3 + manifest_docs = build_seed_manifests( + run_id=run_id, + namespace_count=namespace_count, + pods_per_namespace=pods_per_namespace, + node_selector="korvid.dev/pool=perftest", + ) + pod_docs = [doc for doc in manifest_docs if doc["kind"] == "Pod"] + expected_by_index = {} + for index, doc in enumerate(pod_docs): + metadata = doc["metadata"] + assert isinstance(metadata, dict) + expected_by_index[index] = (metadata["namespace"], metadata["name"]) + + for index in range(namespace_count * pods_per_namespace): + assert live_object_identity(run_id, namespace_count, index) == expected_by_index[index] + + +def test_live_object_identity_examples() -> None: + assert live_object_identity("aks186", 20, 0) == ("korvid-perf-aks186-0", "bench-0") + assert live_object_identity("aks186", 20, 19) == ("korvid-perf-aks186-19", "bench-0") + assert live_object_identity("aks186", 20, 20) == ("korvid-perf-aks186-0", "bench-1") + assert live_object_identity("aks186", 20, 999) == ("korvid-perf-aks186-19", "bench-49") + + +# --------------------------------------------------------------------------- +# Guarded patch construction +# --------------------------------------------------------------------------- + + +def test_build_guarded_status_patch_tests_uid_and_both_ownership_labels() -> None: + ops = build_guarded_status_patch(uid="uid-1", run_id="run1", phase="Pending") + assert ops == [ + {"op": "test", "path": "/metadata/uid", "value": "uid-1"}, + { + "op": "test", + "path": "/metadata/labels/app.kubernetes.io~1managed-by", + "value": "korvid-performance", + }, + { + "op": "test", + "path": "/metadata/labels/korvid.dev~1performance-run", + "value": "run1", + }, + {"op": "replace", "path": "/status/phase", "value": "Pending"}, + ] + + +# --------------------------------------------------------------------------- +# Guarded churn +# --------------------------------------------------------------------------- + + +async def test_drive_live_churn_sends_guarded_patches_for_every_event() -> None: + run_id = "run1" + namespace_count = 2 + _, pods = _build_fake_topology(run_id, namespace_count, 4) + kube = _FakeKubeClient(None, {}, pods) + mutation_client = _FakeMutationClient(kube, run_id) + profile = WorkloadProfile( + schema_version=1, + id="churn-test", + seed=7, + object_count=4, + namespace_count=namespace_count, + steady_events_per_second=4, + duration_seconds=1, + bursts=(), + failures=(), + ) + events = scheduled_events(profile) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + recorder = BenchmarkRecorder() + live_state = dict(pods) + await drive_live_churn( + events, + run_id=run_id, + namespace_count=namespace_count, + live_state=live_state, + mutation_client=mutation_client, + recorder=recorder, + options=options, + ) + assert len(mutation_client.calls) == len(events) + for call, event in zip(mutation_client.calls, events, strict=True): + namespace, name, phase = call + expected_namespace, expected_name = live_object_identity( + run_id, namespace_count, int(event.summary.name.removeprefix("pod-")) + ) + assert (namespace, name) == (expected_namespace, expected_name) + assert phase == event.summary.phase + + +async def test_drive_live_churn_aborts_on_guard_failure_and_never_continues() -> None: + run_id = "run1" + namespace_count = 2 + _, pods = _build_fake_topology(run_id, namespace_count, 4) + kube = _FakeKubeClient(None, {}, pods) + mutation_client = _FakeMutationClient(kube, run_id) + profile = WorkloadProfile( + schema_version=1, + id="churn-abort-test", + seed=7, + object_count=4, + namespace_count=namespace_count, + steady_events_per_second=4, + duration_seconds=1, + bursts=(), + failures=(), + ) + events = scheduled_events(profile) + assert len(events) >= 2 + + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + recorder = BenchmarkRecorder() + # Snapshot the driver's cached state *before* the target pod is replaced + # (a fresh uid) - exactly like `run_live_replay` caching uids once during + # the ownership gate, then churning without re-reading them each time. + live_state = dict(pods) + + # Simulate the target pod being replaced right before the second + # scheduled mutation - a real API server would fail the JSON-Patch + # `test` op the same way. + second_index = int(events[1].summary.name.removeprefix("pod-")) + second_key = live_object_identity(run_id, namespace_count, second_index) + kube.pods[second_key] = dataclasses.replace(kube.pods[second_key], uid="replaced-uid") + + with pytest.raises(ApiStatusError, match="test operation failed"): + await drive_live_churn( + events, + run_id=run_id, + namespace_count=namespace_count, + live_state=live_state, + mutation_client=mutation_client, + recorder=recorder, + options=options, + ) + # Aborted at the 2nd call; a 3rd event must never have been attempted. + assert len(mutation_client.calls) == 2 + + +# --------------------------------------------------------------------------- +# Namespace filtering +# --------------------------------------------------------------------------- + + +async def test_make_live_watch_source_filters_to_expected_namespaces() -> None: + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + distractor = PodSummary( + name="stray", + namespace="default", + phase="Running", + ready="1/1", + restarts=0, + node="node-x", + ) + kube = _FakeKubeClient(None, {}, pods, distractor_pods=(distractor,)) + expected_namespaces = frozenset(namespace for namespace, _name in pods) + source = make_live_watch_source(kube, expected_namespaces) + + seen: list[tuple[str, Summary]] = [] + agen = source("pods", "*") + try: + for _ in range(len(pods)): + seen.append(await agen.__anext__()) + # The distractor pod, and only it, must never surface. + with pytest.raises(asyncio.TimeoutError): + await asyncio.wait_for(agen.__anext__(), timeout=0.05) + finally: + assert isinstance(agen, AsyncGenerator) + await agen.aclose() + + assert {pod.namespace for _event, pod in seen} == expected_namespaces + assert all(pod.name != "stray" for _event, pod in seen) + + +async def test_make_live_watch_source_rejects_non_pod_kind() -> None: + kube = _FakeKubeClient(None, {}, {}) + source = make_live_watch_source(kube, frozenset()) + with pytest.raises(ValueError, match="only watches pods"): + await source("deployments", "*").__anext__() + + +# --------------------------------------------------------------------------- +# Gates: time scale / run_id / topology / identity - all reject before +# any client is constructed. +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_rejects_time_scale_other_than_one() -> None: + deps = LiveDependencies( + command_runner=_never_called("command_runner"), + active_context=_never_called("active_context"), + context_host=_never_called("context_host"), + kube_client_factory=_never_called("kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match=re.escape("time_scale == 1.0")): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_invalid_run_id() -> None: + deps = LiveDependencies( + command_runner=_never_called("command_runner"), + active_context=_never_called("active_context"), + context_host=_never_called("context_host"), + kube_client_factory=_never_called("kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="run_id must be 1-48"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id="Bad Run", + deps=deps, + ) + + +@pytest.mark.parametrize( + ("object_count", "namespace_count"), + [(999, 20), (1000, 19), (1001, 20), (100, 7)], +) +async def test_run_live_replay_rejects_topology_mismatch_before_identity_gate( + object_count: int, namespace_count: int +) -> None: + profile = WorkloadProfile( + schema_version=1, + id="bad-topology", + seed=1, + object_count=object_count, + namespace_count=namespace_count, + steady_events_per_second=1, + duration_seconds=1, + bursts=(), + failures=(), + ) + deps = LiveDependencies( + command_runner=_never_called("command_runner"), + active_context=_never_called("active_context"), + context_host=_never_called("context_host"), + kube_client_factory=_never_called("kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises( + ValueError, match=re.escape("object_count") + "|" + re.escape("namespace_count") + ): + await run_live_replay( + profile, + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_wrong_active_context_before_mutation() -> None: + deps = LiveDependencies( + command_runner=_never_called("command_runner"), + active_context=lambda: "some-other-context", + context_host=_never_called("context_host"), + kube_client_factory=_never_called("kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="active kubeconfig context"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_wrong_aks_resource_id_before_mutation() -> None: + async def context_host(_context: str) -> str: + return FQDN + + deps = LiveDependencies( + command_runner=_ok_command_runner(cluster_id="/subscriptions/x/wrong-cluster"), + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=_never_called("kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="expected"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_wrong_api_hostname_before_mutation() -> None: + async def context_host(_context: str) -> str: + return "not-the-real-host.example.com" + + deps = LiveDependencies( + command_runner=_ok_command_runner(), + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=_never_called("kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="does not match"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_malformed_identity_command_output() -> None: + async def command_runner(_args: Any) -> CommandResult: + return CommandResult(0, "not-json{{{", "") + + async def context_host(_context: str) -> str: + return FQDN + + deps = LiveDependencies( + command_runner=command_runner, + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=_never_called("kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="malformed JSON"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_nonzero_identity_command_exit() -> None: + async def command_runner(_args: Any) -> CommandResult: + return CommandResult(1, "", "ERROR: az login required") + + async def context_host(_context: str) -> str: + return FQDN + + deps = LiveDependencies( + command_runner=command_runner, + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=_never_called("kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="az aks show failed"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_default_command_runner_surfaces_missing_executable() -> None: + with pytest.raises(ValueError, match="executable not found"): + await live._default_command_runner(["korvid-test-definitely-missing-binary-xyz"]) + + +# --------------------------------------------------------------------------- +# Ownership gate +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_rejects_missing_namespace_before_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + del namespaces[manifests.namespace_name(RUN_ID, 5)] + kube = _FakeKubeClient(None, namespaces, pods) + deps = _happy_deps( + kube, RUN_ID, mutation_client_factory=_never_called("mutation_client_factory") + ) + with pytest.raises(ValueError, match="missing expected namespaces"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_cross_run_namespace_label_before_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + bad_name = manifests.namespace_name(RUN_ID, 3) + namespaces[bad_name] = dataclasses.replace(namespaces[bad_name], labels=_labels("other-run")) + kube = _FakeKubeClient(None, namespaces, pods) + deps = _happy_deps( + kube, RUN_ID, mutation_client_factory=_never_called("mutation_client_factory") + ) + with pytest.raises(ValueError, match="namespaces missing/mismatched ownership labels"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_missing_pod_before_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + key = live_object_identity(RUN_ID, 20, 42) + del pods[key] + kube = _FakeKubeClient(None, namespaces, pods) + deps = _happy_deps( + kube, RUN_ID, mutation_client_factory=_never_called("mutation_client_factory") + ) + with pytest.raises(ValueError, match="missing"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_cross_run_pod_label_before_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + key = live_object_identity(RUN_ID, 20, 42) + pods[key] = dataclasses.replace(pods[key], labels=_labels("some-other-run-id")) + kube = _FakeKubeClient(None, namespaces, pods) + deps = _happy_deps( + kube, RUN_ID, mutation_client_factory=_never_called("mutation_client_factory") + ) + with pytest.raises(ValueError, match="mismatched ownership labels"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +# --------------------------------------------------------------------------- +# Full happy path: real app-path wiring, telemetry, digest parity +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_full_happy_path_matches_cluster_digest() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + kube = _FakeKubeClient(None, namespaces, pods) + deps = _happy_deps(kube, RUN_ID) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions( + time_scale=1.0, sample_interval=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.object_count == 1000 + assert report.dropped_updates == 0 + assert report.expected_digest == report.final_digest + assert report.expected_digest == summary_digest(kube.pods.values()) + assert kube.connect_context == CONTEXT + assert kube.closed + assert report.api.operations["watch_open"] == 1 + assert report.churn_started_before_input + + +async def test_run_live_replay_aborts_and_still_closes_clients_on_guard_failure() -> None: + """A guard failure mid-churn (a real API server's `test` op rejection) + must propagate as `ApiStatusError` *and* still close the kube/mutation + clients via `run_live_replay`'s `finally` teardown - the same guarantee + `run_replay` gives on any mid-run failure.""" + + class _AlwaysFailingMutationClient: + def __init__(self) -> None: + self.closed = False + + async def patch_pod_status_guarded( + self, namespace: str, name: str, *, uid: str, phase: str + ) -> None: + raise ApiStatusError(422, "test operation failed for guarded patch") + + async def close(self) -> None: + self.closed = True + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + kube = _FakeKubeClient(None, namespaces, pods) + failing_client = _AlwaysFailingMutationClient() + deps = _happy_deps(kube, RUN_ID, mutation_client_factory=lambda run_id_arg: failing_client) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + with pytest.raises(ApiStatusError, match="test operation failed"): + await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert kube.closed + assert failing_client.closed + + +async def test_run_live_replay_propagates_cancelled_error_and_still_closes_clients() -> None: + """`asyncio.CancelledError` raised mid-churn (as real task cancellation + delivers into whatever the run is awaiting) must propagate out of + `run_live_replay` unchanged, and the sampler/watch-manager/mutation/kube + clients must still be closed via the nested `finally` blocks: cancellation + must never skip teardown or leave the real cluster's watches/clients open. + + This raises `CancelledError` directly from the injected `async_sleep` seam + rather than calling `Task.cancel()` on a task that owns an in-flight + `MeasuredKorvidApp.run_test()` - hard-cancelling *that* task triggers an + unrelated Textual/pytest GC quirk (a dangling internal reactive-watcher + coroutine surfacing as `PytestUnraisableExceptionWarning` at session + teardown) that has nothing to do with `live.py`'s own cleanup correctness. + """ + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + kube = _FakeKubeClient(None, namespaces, pods) + created_clients: list[_FakeMutationClient] = [] + deps = _happy_deps(kube, RUN_ID, mutation_clients=created_clients) + + async def cancelling_sleep(_delay: float) -> None: + raise asyncio.CancelledError + + # duration_seconds=1, steady_events_per_second=2 schedules events at + # offsets 0.0 and 0.5: the first mutates without sleeping (delay <= 0), + # the second's positive delay drives the cancelling sleep. + options = ReplayOptions(time_scale=1.0, monotonic_fn=lambda: 0.0, async_sleep=cancelling_sleep) + + with pytest.raises(asyncio.CancelledError): + await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert kube.closed + assert created_clients + assert created_clients[0].closed + # Cancellation must abort before the second churn mutation - no broad + # cleanup, no continuing past the point of cancellation. + assert len(created_clients[0].calls) == 1 From f5b995ccb5d53357dfa2fc2073a6412815145bb4 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 08:13:28 +0900 Subject: [PATCH 23/38] fix(issue-186-task-8.2-review): isolate harness reads from app-path API telemetry Address both Task 8.2 review findings in tests/performance/live.py: MEDIUM: run_live_replay wired a single KubeClient with recorder.record_api to both the ownership gate, the pre-churn uid snapshot, the app watch source, and the post-churn independent read, so ReplayReport.api mixed ~60 harness-only LISTs with the real application LIST/WATCH telemetry. LiveDependencies now exposes a second seam, harness_kube_client_factory, that constructs a non-instrumented KubeReadClient used only for _verify_ownership and the final independent re-read. The telemetry-wired client (kube_client_factory) is now only ever handed to make_live_watch_source, so ReplayReport.api reports exactly the production application read path. Identity ordering is preserved and tightened: the app-path client is now constructed only after the harness ownership gate passes (previously it was constructed and connected up front), and mutation_client_factory is still only called after ownership verification. LOW: _verify_ownership now returns the validated (namespace, name) -> PodSummary snapshot it already collected while checking ownership, and run_live_replay reuses that snapshot as the pre-churn uid snapshot instead of immediately re-listing every namespace's Pods a second time. Tests (TDD): _FakeKubeClient now models per-call telemetry on list_objects/list_pods (mirroring the real KubeClient) and tracks list_pods/list_objects/watch_pods call counts; _happy_deps constructs separate harness/app-path fake client instances sharing the same underlying namespaces/pods dicts. New/updated coverage proves: report.api.operations["list"] == 1 (only the app watch's own LIST, none of the harness's ~60 reads); the harness client is list_pods-ed exactly twice per namespace (ownership gate + final read, no redundant third uid pass); the app-path client's list_pods is never called; the harness client's watch_pods is never called; the application-path client is never even constructed when the ownership gate rejects; and _verify_ownership's return value matches the seeded topology. All prior Task 8.2 gate/churn/CLI tests are preserved and updated only where the LiveDependencies/_FakeKubeClient signatures changed. RED: reverting live.py to its pre-fix state with only the updated tests/performance/test_live.py in place produced "20 failed, 8 passed in 1.32s" (uv run pytest -p no:tach tests/performance/test_live.py -q), all failing on the new harness_kube_client_factory field/behavior. GREEN: uv run pytest -p no:tach tests/performance/test_live.py tests/performance/test_cli.py -q -> 55 passed; uv run pytest -p no:tach tests/performance/ -q -> 102 passed; uv run ruff check/format, uv run mypy tests/performance/live.py tests/performance/cli.py, and uv run tach check all pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/live.py | 205 +++++++++++++++++------------ tests/performance/test_live.py | 228 ++++++++++++++++++++++++++++----- 2 files changed, 321 insertions(+), 112 deletions(-) diff --git a/tests/performance/live.py b/tests/performance/live.py index 1ebc0bf0..7bb115f6 100644 --- a/tests/performance/live.py +++ b/tests/performance/live.py @@ -23,6 +23,15 @@ `status.phase`. A failed `test` op aborts the *entire* run - there is no unguarded fallback. +Two separate `KubeReadClient` connections are used deliberately: a +non-instrumented *harness* client (`LiveDependencies.harness_kube_client_factory`) +performs the ownership gate, the post-churn independent re-read, and nothing +else, while a single telemetry-wired *application-path* client +(`LiveDependencies.kube_client_factory`) is only ever handed to +`make_live_watch_source`. This keeps `ReplayReport.api` reporting exactly the +production application read path (the real `KubeClient.watch_pods` LIST+WATCH +telemetry) instead of being diluted by the harness's own bookkeeping reads. + Unit tests substitute every external boundary via `LiveDependencies`; none of them may contact Azure, a real kubeconfig, or a real cluster. """ @@ -122,12 +131,24 @@ class LiveDependencies: """Every external boundary `run_live_replay` crosses, as an injectable seam. Production callers get real wiring from `_default_dependencies`; tests always construct this explicitly with fakes, so no unit test can - reach Azure, a real kubeconfig, or a real cluster.""" + reach Azure, a real kubeconfig, or a real cluster. + + `kube_client_factory` and `harness_kube_client_factory` are deliberately + two separate seams (constructing two separate `KubeReadClient` + connections at runtime): the former is wired with `recorder.record_api` + and used *only* for the real application read path + (`make_live_watch_source`'s `watch_pods`), so `ReplayReport.api` reports + exactly the production LIST+WATCH telemetry an operator would see. The + latter is never wired to telemetry and is used *only* for the harness's + own bookkeeping reads (the ownership gate and the post-churn independent + re-read) - those reads must never dilute the application-path signal. + """ command_runner: CommandRunner active_context: Callable[[], str | None] context_host: Callable[[str], Awaitable[str]] kube_client_factory: Callable[[ReadTelemetry], KubeReadClient] + harness_kube_client_factory: Callable[[], KubeReadClient] mutation_client_factory: Callable[[str], MutationClient] @@ -239,12 +260,20 @@ async def close(self) -> None: def _default_dependencies(context: str) -> LiveDependencies: - """Real production wiring: subprocess `az`, real kubeconfig, real `KubeClient`.""" + """Real production wiring: subprocess `az`, real kubeconfig, real `KubeClient`. + + Two independent `KubeClient` connections are constructed: the + telemetry-wired one `run_live_replay` hands to `make_live_watch_source`, + and a plain, non-instrumented one (`read_telemetry=None`, so + `KubeClient._observe_read` is a no-op) for the harness's own ownership + and post-churn reads. + """ return LiveDependencies( command_runner=_default_command_runner, active_context=_default_active_context, context_host=_default_context_host, kube_client_factory=lambda read_telemetry: KubeClient(read_telemetry=read_telemetry), + harness_kube_client_factory=lambda: KubeClient(), mutation_client_factory=lambda run_id: _KubeMutationClient(context, run_id), ) @@ -333,13 +362,17 @@ async def _verify_cluster_identity( async def _verify_ownership( kube: KubeReadClient, *, run_id: str, namespace_count: int, object_count: int -) -> None: +) -> dict[tuple[str, str], PodSummary]: """Ownership gate: every expected namespace and every expected Pod must already exist with both ownership labels, checked before any churn. Collects every mismatch across all namespaces/Pods before raising, so a single failed run surfaces the full blast radius at once instead of stopping at the first namespace. + + Returns the validated `(namespace, name) -> PodSummary` snapshot so + callers can reuse it (e.g. as the pre-churn uid snapshot) instead of + immediately re-listing the same Pods a second time. """ expected_namespaces = [manifests.namespace_name(run_id, i) for i in range(namespace_count)] actual_namespaces = {obj.name: obj for obj in await kube.list_objects(_NAMESPACES_META, None)} @@ -363,6 +396,7 @@ async def _verify_ownership( ] missing_pods: list[str] = [] mismatched_pods: list[str] = [] + validated_pods: dict[tuple[str, str], PodSummary] = {} for namespace in expected_namespaces: pods_by_name = {pod.name: pod for pod in await kube.list_pods(namespace)} for name in expected_pod_names: @@ -371,10 +405,13 @@ async def _verify_ownership( missing_pods.append(f"{namespace}/{name}") elif not _owns(pod.labels, run_id): mismatched_pods.append(f"{namespace}/{name}") + else: + validated_pods[(namespace, name)] = pod if missing_pods: raise ValueError(f"missing expected pods: {', '.join(missing_pods)}") if mismatched_pods: raise ValueError(f"pods with mismatched ownership labels: {', '.join(mismatched_pods)}") + return validated_pods def make_live_watch_source( @@ -447,6 +484,13 @@ async def run_live_replay( real application-path wiring (`KubeClient` -> `WatchManager` -> `ResourceStore` -> `MeasuredKorvidApp`), guarded churn, and digest parity against an independent post-churn re-read of the cluster. + + Two `KubeReadClient` connections are used: `harness_kube` (never wired to + telemetry) performs the ownership gate and the post-churn independent + re-read; `kube` (wired to `recorder.record_api`) is only ever handed to + `make_live_watch_source`, so `ReplayReport.api` reports exactly the + production application read path. The ownership gate's validated Pod + snapshot is reused as the pre-churn uid snapshot - it is never re-listed. """ _validate_time_scale(options) manifests.validate_run_id(run_id) @@ -460,12 +504,14 @@ async def run_live_replay( store = ResourceStore() recorder = BenchmarkRecorder() sampler = ProcessSampler(options.sample_interval) - kube = active_deps.kube_client_factory(recorder.record_api) - await kube.connect(context) + harness_kube = active_deps.harness_kube_client_factory() + await harness_kube.connect(context) try: - await _verify_ownership( - kube, + # Reused directly as the pre-churn uid snapshot below - no second, + # redundant listing pass over the same Pods. + live_state = await _verify_ownership( + harness_kube, run_id=run_id, namespace_count=profile.namespace_count, object_count=profile.object_count, @@ -474,83 +520,84 @@ async def run_live_replay( expected_namespaces = frozenset( manifests.namespace_name(run_id, i) for i in range(profile.namespace_count) ) - source = make_live_watch_source(kube, expected_namespaces) - watch_manager = WatchManager(store, source, retry_delay=0.0) - manifest = _build_manifest(profile) - mutation_client = active_deps.mutation_client_factory(run_id) - - app = MeasuredKorvidApp( - config=KorvidConfig(namespace=ALL_NAMESPACES), - store=store, - watch_manager=watch_manager, - recorder=recorder, - ) - # Snapshot Pod uids once, before churn, per namespace: guarded - # patches test against the uid observed at ownership-gate time, so a - # concurrently replaced Pod fails its `test` op instead of silently - # patching a different object. - live_state: dict[tuple[str, str], PodSummary] = {} - for namespace in expected_namespaces: - for pod in await kube.list_pods(namespace): - live_state[(namespace, pod.name)] = pod - - sampler.start() - churn_started_before_input = False + kube = active_deps.kube_client_factory(recorder.record_api) + await kube.connect(context) try: - async with app.run_test() as pilot: - table = app.query_one(ResourceTable) - - await until( - pilot, - lambda: table.row_count == profile.object_count, - timeout=60.0, - label="initial owned pods rendered", - ) - - events = scheduled_events(profile) - churn_task = asyncio.create_task( - drive_live_churn( - events, - run_id=run_id, - namespace_count=profile.namespace_count, - live_state=live_state, - mutation_client=mutation_client, - recorder=recorder, - options=options, + source = make_live_watch_source(kube, expected_namespaces) + watch_manager = WatchManager(store, source, retry_delay=0.0) + manifest = _build_manifest(profile) + mutation_client = active_deps.mutation_client_factory(run_id) + + app = MeasuredKorvidApp( + config=KorvidConfig(namespace=ALL_NAMESPACES), + store=store, + watch_manager=watch_manager, + recorder=recorder, + ) + + sampler.start() + churn_started_before_input = False + try: + async with app.run_test() as pilot: + table = app.query_one(ResourceTable) + + await until( + pilot, + lambda: table.row_count == profile.object_count, + timeout=60.0, + label="initial owned pods rendered", ) - ) - churn_started_before_input = True - - t0 = monotonic() - await pilot.press("down") - recorder.record_input(monotonic() - t0) - t0 = monotonic() - await pilot.press("up") - recorder.record_input(monotonic() - t0) - - await churn_task - - await until( - pilot, - lambda: not recorder._pending_events, - timeout=60.0, - label="churn complete and all events rendered", - ) + + events = scheduled_events(profile) + churn_task = asyncio.create_task( + drive_live_churn( + events, + run_id=run_id, + namespace_count=profile.namespace_count, + live_state=live_state, + mutation_client=mutation_client, + recorder=recorder, + options=options, + ) + ) + churn_started_before_input = True + + t0 = monotonic() + await pilot.press("down") + recorder.record_input(monotonic() - t0) + t0 = monotonic() + await pilot.press("up") + recorder.record_input(monotonic() - t0) + + await churn_task + + await until( + pilot, + lambda: not recorder._pending_events, + timeout=60.0, + label="churn complete and all events rendered", + ) + finally: + process_samples = await sampler.stop() + await watch_manager.stop_all() + await mutation_client.close() + + # Independently re-read the cluster's actual Pods for ground-truth + # digest parity via the non-instrumented harness client, rather + # than trusting the driver's own bookkeeping or polluting the + # application-path telemetry with this harness-only read. + final_pods: list[PodSummary] = [] + for namespace in expected_namespaces: + final_pods.extend(await harness_kube.list_pods(namespace)) + expected_digest = summary_digest(final_pods) + final_digest = summary_digest( + cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES)) + ) finally: - process_samples = await sampler.stop() - await watch_manager.stop_all() - await mutation_client.close() - - # Independently re-read the cluster's actual Pods for ground-truth - # digest parity, rather than trusting the driver's own bookkeeping. - final_pods: list[PodSummary] = [] - for namespace in expected_namespaces: - final_pods.extend(await kube.list_pods(namespace)) - expected_digest = summary_digest(final_pods) - final_digest = summary_digest(cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES))) + await kube.close() finally: - await kube.close() + await harness_kube.close() benchmark = recorder.report(manifest, process_samples, final_digest=final_digest) diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py index 218ba12d..90809c5c 100644 --- a/tests/performance/test_live.py +++ b/tests/performance/test_live.py @@ -11,6 +11,7 @@ import dataclasses import json import re +from collections import Counter from collections.abc import AsyncGenerator, AsyncIterator, Awaitable, Callable from typing import Any @@ -91,23 +92,38 @@ def _build_fake_topology( class _FakeKubeClient: """Fake `KubeReadClient`: an in-memory cluster the fake mutation client mutates directly, so the watch stream really observes guarded patches - - exactly like a real API server + watch, with zero network I/O.""" + exactly like a real API server + watch, with zero network I/O. + + Mirrors the real `KubeClient`'s telemetry behavior exactly: `list_objects` + and `list_pods` (used only by the harness's ownership/final reads) emit a + "list" event, and `watch_pods` (the real application read path) emits + "list" then "watch_open"/"watch_event" - all conditional on + `read_telemetry` being wired, exactly like `KubeClient._observe_read` + being a no-op when `read_telemetry is None`. Two `_FakeKubeClient` + instances constructed over the *same* `namespaces`/`pods` dict objects + model two independent connections to one shared cluster, exactly like + `run_live_replay`'s real harness/app-path `KubeClient` pair. + """ def __init__( self, - read_telemetry: ReadTelemetry | None, namespaces: dict[str, GenericSummary], pods: dict[tuple[str, str], PodSummary], *, distractor_pods: tuple[PodSummary, ...] = (), ) -> None: - self.read_telemetry = read_telemetry + self.read_telemetry: ReadTelemetry | None = None self.namespaces = namespaces self.pods = pods self.distractor_pods = distractor_pods self.connect_context: str | None = "__not_connected__" self.closed = False self.events: asyncio.Queue[tuple[str, PodSummary]] = asyncio.Queue() + #: Per-namespace `list_pods` call log; asserts the ownership gate's + #: validated snapshot is reused rather than immediately re-listed. + self.list_pods_calls: list[str] = [] + self.list_objects_calls = 0 + self.watch_pods_calls = 0 async def connect(self, context: str | None = None) -> None: self.connect_context = context @@ -118,13 +134,20 @@ async def close(self) -> None: async def list_objects(self, meta: ResourceMeta, namespace: str | None) -> list[GenericSummary]: assert meta.kind == "Namespace" assert namespace is None + self.list_objects_calls += 1 + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("list", "/api/v1/namespaces")) return list(self.namespaces.values()) async def list_pods(self, namespace: str) -> list[PodSummary]: + self.list_pods_calls.append(namespace) + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("list", f"/api/v1/namespaces/{namespace}/pods")) return [pod for (ns, _name), pod in self.pods.items() if ns == namespace] async def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSummary]]: assert namespace is None + self.watch_pods_calls += 1 if self.read_telemetry is not None: self.read_telemetry(ReadTelemetryEvent("list", "/api/v1/pods")) for pod in self.pods.values(): @@ -192,31 +215,60 @@ async def _run(_args: Any) -> CommandResult: def _happy_deps( - kube: _FakeKubeClient, + namespaces: dict[str, GenericSummary], + pods: dict[tuple[str, str], PodSummary], run_id: str, *, + distractor_pods: tuple[PodSummary, ...] = (), mutation_clients: list[_FakeMutationClient] | None = None, mutation_client_factory: Callable[[str], Any] | None = None, + harness_clients: list[_FakeKubeClient] | None = None, + app_clients: list[_FakeKubeClient] | None = None, ) -> LiveDependencies: + """Build a `LiveDependencies` whose harness and application-path + `KubeReadClient`s are two *separate* `_FakeKubeClient` instances sharing + the same underlying `namespaces`/`pods` dicts by reference (so a guarded + mutation issued against the app-path instance is immediately visible to + the harness's later independent re-read) - exactly mirroring + `run_live_replay`'s real harness/app-path `KubeClient` pair. Pass + `harness_clients`/`app_clients` to capture the constructed instances for + assertions (telemetry exclusion, call counts, connect/close).""" + async def context_host(context: str) -> str: assert context == CONTEXT return FQDN + # The most recently constructed app-path client: `default_mutation_factory` + # is only ever invoked (by `run_live_replay`) after `kube_client_factory`, + # so this always resolves to the live watch's own instance/queue. + app_holder: list[_FakeKubeClient] = [] + def default_mutation_factory(run_id_arg: str) -> _FakeMutationClient: - client = _FakeMutationClient(kube, run_id_arg) + client = _FakeMutationClient(app_holder[-1], run_id_arg) if mutation_clients is not None: mutation_clients.append(client) return client def kube_factory(read_telemetry: ReadTelemetry) -> _FakeKubeClient: - kube.read_telemetry = read_telemetry - return kube + client = _FakeKubeClient(namespaces, pods, distractor_pods=distractor_pods) + client.read_telemetry = read_telemetry + app_holder.append(client) + if app_clients is not None: + app_clients.append(client) + return client + + def harness_factory() -> _FakeKubeClient: + client = _FakeKubeClient(namespaces, pods, distractor_pods=distractor_pods) + if harness_clients is not None: + harness_clients.append(client) + return client return LiveDependencies( command_runner=_ok_command_runner(), active_context=lambda: CONTEXT, context_host=context_host, kube_client_factory=kube_factory, + harness_kube_client_factory=harness_factory, mutation_client_factory=mutation_client_factory or default_mutation_factory, ) @@ -315,7 +367,7 @@ async def test_drive_live_churn_sends_guarded_patches_for_every_event() -> None: run_id = "run1" namespace_count = 2 _, pods = _build_fake_topology(run_id, namespace_count, 4) - kube = _FakeKubeClient(None, {}, pods) + kube = _FakeKubeClient({}, pods) mutation_client = _FakeMutationClient(kube, run_id) profile = WorkloadProfile( schema_version=1, @@ -357,7 +409,7 @@ async def test_drive_live_churn_aborts_on_guard_failure_and_never_continues() -> run_id = "run1" namespace_count = 2 _, pods = _build_fake_topology(run_id, namespace_count, 4) - kube = _FakeKubeClient(None, {}, pods) + kube = _FakeKubeClient({}, pods) mutation_client = _FakeMutationClient(kube, run_id) profile = WorkloadProfile( schema_version=1, @@ -419,7 +471,7 @@ async def test_make_live_watch_source_filters_to_expected_namespaces() -> None: restarts=0, node="node-x", ) - kube = _FakeKubeClient(None, {}, pods, distractor_pods=(distractor,)) + kube = _FakeKubeClient({}, pods, distractor_pods=(distractor,)) expected_namespaces = frozenset(namespace for namespace, _name in pods) source = make_live_watch_source(kube, expected_namespaces) @@ -440,7 +492,7 @@ async def test_make_live_watch_source_filters_to_expected_namespaces() -> None: async def test_make_live_watch_source_rejects_non_pod_kind() -> None: - kube = _FakeKubeClient(None, {}, {}) + kube = _FakeKubeClient({}, {}) source = make_live_watch_source(kube, frozenset()) with pytest.raises(ValueError, match="only watches pods"): await source("deployments", "*").__anext__() @@ -458,6 +510,7 @@ async def test_run_live_replay_rejects_time_scale_other_than_one() -> None: active_context=_never_called("active_context"), context_host=_never_called("context_host"), kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises(ValueError, match=re.escape("time_scale == 1.0")): @@ -477,6 +530,7 @@ async def test_run_live_replay_rejects_invalid_run_id() -> None: active_context=_never_called("active_context"), context_host=_never_called("context_host"), kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises(ValueError, match="run_id must be 1-48"): @@ -513,6 +567,7 @@ async def test_run_live_replay_rejects_topology_mismatch_before_identity_gate( active_context=_never_called("active_context"), context_host=_never_called("context_host"), kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises( @@ -534,6 +589,7 @@ async def test_run_live_replay_rejects_wrong_active_context_before_mutation() -> active_context=lambda: "some-other-context", context_host=_never_called("context_host"), kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises(ValueError, match="active kubeconfig context"): @@ -556,6 +612,7 @@ async def context_host(_context: str) -> str: active_context=lambda: CONTEXT, context_host=context_host, kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises(ValueError, match="expected"): @@ -578,6 +635,7 @@ async def context_host(_context: str) -> str: active_context=lambda: CONTEXT, context_host=context_host, kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises(ValueError, match="does not match"): @@ -603,6 +661,7 @@ async def context_host(_context: str) -> str: active_context=lambda: CONTEXT, context_host=context_host, kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises(ValueError, match="malformed JSON"): @@ -628,6 +687,7 @@ async def context_host(_context: str) -> str: active_context=lambda: CONTEXT, context_host=context_host, kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises(ValueError, match="az aks show failed"): @@ -654,9 +714,11 @@ async def test_default_command_runner_surfaces_missing_executable() -> None: async def test_run_live_replay_rejects_missing_namespace_before_churn() -> None: namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) del namespaces[manifests.namespace_name(RUN_ID, 5)] - kube = _FakeKubeClient(None, namespaces, pods) deps = _happy_deps( - kube, RUN_ID, mutation_client_factory=_never_called("mutation_client_factory") + namespaces, + pods, + RUN_ID, + mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises(ValueError, match="missing expected namespaces"): await run_live_replay( @@ -673,9 +735,11 @@ async def test_run_live_replay_rejects_cross_run_namespace_label_before_churn() namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) bad_name = manifests.namespace_name(RUN_ID, 3) namespaces[bad_name] = dataclasses.replace(namespaces[bad_name], labels=_labels("other-run")) - kube = _FakeKubeClient(None, namespaces, pods) deps = _happy_deps( - kube, RUN_ID, mutation_client_factory=_never_called("mutation_client_factory") + namespaces, + pods, + RUN_ID, + mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises(ValueError, match="namespaces missing/mismatched ownership labels"): await run_live_replay( @@ -692,9 +756,11 @@ async def test_run_live_replay_rejects_missing_pod_before_churn() -> None: namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) key = live_object_identity(RUN_ID, 20, 42) del pods[key] - kube = _FakeKubeClient(None, namespaces, pods) deps = _happy_deps( - kube, RUN_ID, mutation_client_factory=_never_called("mutation_client_factory") + namespaces, + pods, + RUN_ID, + mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises(ValueError, match="missing"): await run_live_replay( @@ -711,9 +777,11 @@ async def test_run_live_replay_rejects_cross_run_pod_label_before_churn() -> Non namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) key = live_object_identity(RUN_ID, 20, 42) pods[key] = dataclasses.replace(pods[key], labels=_labels("some-other-run-id")) - kube = _FakeKubeClient(None, namespaces, pods) deps = _happy_deps( - kube, RUN_ID, mutation_client_factory=_never_called("mutation_client_factory") + namespaces, + pods, + RUN_ID, + mutation_client_factory=_never_called("mutation_client_factory"), ) with pytest.raises(ValueError, match="mismatched ownership labels"): await run_live_replay( @@ -726,6 +794,52 @@ async def test_run_live_replay_rejects_cross_run_pod_label_before_churn() -> Non ) +async def test_run_live_replay_never_constructs_app_client_when_ownership_fails() -> None: + """The MEDIUM finding's harness/app-path split must not weaken the + ownership-before-mutation ordering: when the ownership gate rejects, the + application-path (telemetry-wired) `KubeClient` must never even be + constructed - only the harness client is used for the (failing) + ownership check.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + del namespaces[manifests.namespace_name(RUN_ID, 5)] + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + mutation_client_factory=_never_called("mutation_client_factory"), + ) + deps = dataclasses.replace(deps, kube_client_factory=_never_called("kube_client_factory")) + with pytest.raises(ValueError, match="missing expected namespaces"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +# --------------------------------------------------------------------------- +# `_verify_ownership` returns a reusable validated snapshot (no redundant +# re-listing for uids) +# --------------------------------------------------------------------------- + + +async def test_verify_ownership_returns_validated_pod_snapshot() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + kube = _FakeKubeClient(namespaces, pods) + + validated = await live._verify_ownership( + kube, run_id=RUN_ID, namespace_count=20, object_count=1000 + ) + + assert validated == pods + # Exactly one `list_pods` call per namespace: the gate itself, no + # redundant second pass. + assert sorted(kube.list_pods_calls) == sorted(namespaces) + + # --------------------------------------------------------------------------- # Full happy path: real app-path wiring, telemetry, digest parity # --------------------------------------------------------------------------- @@ -733,8 +847,11 @@ async def test_run_live_replay_rejects_cross_run_pod_label_before_churn() -> Non async def test_run_live_replay_full_happy_path_matches_cluster_digest() -> None: namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) - kube = _FakeKubeClient(None, namespaces, pods) - deps = _happy_deps(kube, RUN_ID) + harness_clients: list[_FakeKubeClient] = [] + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps( + namespaces, pods, RUN_ID, harness_clients=harness_clients, app_clients=app_clients + ) monotonic_fn, async_sleep = _virtual_clock() options = ReplayOptions( time_scale=1.0, sample_interval=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep @@ -752,18 +869,45 @@ async def test_run_live_replay_full_happy_path_matches_cluster_digest() -> None: assert report.object_count == 1000 assert report.dropped_updates == 0 assert report.expected_digest == report.final_digest - assert report.expected_digest == summary_digest(kube.pods.values()) - assert kube.connect_context == CONTEXT - assert kube.closed + assert report.expected_digest == summary_digest(pods.values()) + + # Exactly one harness client and one app-path client are constructed; + # both connect to the required context and both get closed. + assert len(harness_clients) == 1 + assert len(app_clients) == 1 + harness_client, app_client = harness_clients[0], app_clients[0] + assert harness_client.connect_context == CONTEXT + assert harness_client.closed + assert app_client.connect_context == CONTEXT + assert app_client.closed + + # MEDIUM finding: the app-path client's telemetry is the *only* source + # of `report.api` - its single "list" comes from `watch_pods`'s own + # internal LIST-then-WATCH, not from any harness read. + assert app_client.watch_pods_calls == 1 assert report.api.operations["watch_open"] == 1 + assert report.api.operations.get("list", 0) == 1 + + # The harness client never watches - it is only ever used for the + # ownership gate and the final independent re-read. + assert harness_client.watch_pods_calls == 0 + + # LOW finding: the ownership gate's validated snapshot is reused as the + # pre-churn uid snapshot - each namespace is only `list_pods`-ed twice on + # the harness client (the ownership gate itself, then the post-churn + # independent re-read), never a redundant third time for uids. + per_namespace_list_calls = Counter(harness_client.list_pods_calls) + assert set(per_namespace_list_calls.values()) == {2} + assert app_client.list_pods_calls == [] + assert report.churn_started_before_input async def test_run_live_replay_aborts_and_still_closes_clients_on_guard_failure() -> None: """A guard failure mid-churn (a real API server's `test` op rejection) - must propagate as `ApiStatusError` *and* still close the kube/mutation - clients via `run_live_replay`'s `finally` teardown - the same guarantee - `run_replay` gives on any mid-run failure.""" + must propagate as `ApiStatusError` *and* still close the harness/app-path + kube clients and the mutation client via `run_live_replay`'s `finally` + teardown - the same guarantee `run_replay` gives on any mid-run failure.""" class _AlwaysFailingMutationClient: def __init__(self) -> None: @@ -778,9 +922,17 @@ async def close(self) -> None: self.closed = True namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) - kube = _FakeKubeClient(None, namespaces, pods) + harness_clients: list[_FakeKubeClient] = [] + app_clients: list[_FakeKubeClient] = [] failing_client = _AlwaysFailingMutationClient() - deps = _happy_deps(kube, RUN_ID, mutation_client_factory=lambda run_id_arg: failing_client) + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + mutation_client_factory=lambda run_id_arg: failing_client, + harness_clients=harness_clients, + app_clients=app_clients, + ) monotonic_fn, async_sleep = _virtual_clock() options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) @@ -794,7 +946,8 @@ async def close(self) -> None: deps=deps, ) - assert kube.closed + assert harness_clients[0].closed + assert app_clients[0].closed assert failing_client.closed @@ -813,9 +966,17 @@ async def test_run_live_replay_propagates_cancelled_error_and_still_closes_clien teardown) that has nothing to do with `live.py`'s own cleanup correctness. """ namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) - kube = _FakeKubeClient(None, namespaces, pods) + harness_clients: list[_FakeKubeClient] = [] + app_clients: list[_FakeKubeClient] = [] created_clients: list[_FakeMutationClient] = [] - deps = _happy_deps(kube, RUN_ID, mutation_clients=created_clients) + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + mutation_clients=created_clients, + harness_clients=harness_clients, + app_clients=app_clients, + ) async def cancelling_sleep(_delay: float) -> None: raise asyncio.CancelledError @@ -835,7 +996,8 @@ async def cancelling_sleep(_delay: float) -> None: deps=deps, ) - assert kube.closed + assert harness_clients[0].closed + assert app_clients[0].closed assert created_clients assert created_clients[0].closed # Cancellation must abort before the second churn mutation - no broad From b6e151252fd65986f1e250984b55fa2d8fe92c8b Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 09:31:09 +0900 Subject: [PATCH 24/38] fix(issue-186-whole-branch-review): make the live qualification harness trustworthy Independent whole-branch review of the #186 performance harness found three blocking defects, seven important gaps, and fifteen quality findings. This closes every credible one. Blockers - Record event-to-render at owned MODIFIED *watch receipt* instead of after the patch acknowledgement. The watch event and the patch response race over independent connections, so recording at ack could append a pending entry after its own render (reproduced: a 60s opaque wait timeout and false dropped updates) and silently folded the write round trip into a read-path latency metric that is compared against the 1k/10k/50k baselines. - Churn a dedicated non-ownership `korvid.dev/performance-tick` label on the Pod's own metadata instead of the kubelet-owned `status.phase`, keeping the atomic UID and both ownership-label JSON Patch `test` operations. Read the ground-truth cluster snapshot while the watch is still live, revalidate ownership, wait for the store digest to converge, and re-assert the exact row count before teardown. - Drive mutations with explicit bounded concurrency and a bounded per-attempt timeout, retry only HTTP 429 with a bounded policy re-issuing the identical guarded patch, and report requested events/rate next to observed events, churn wall time, achieved rate, and mutation throttles (counted separately from application read telemetry). Important - Add `aks-live-1k` (1,800s at 20 events/s with three 30s bursts at 100 events/s), matching the published live plan, and point the command help and design doc at it; `aks-1k` stays the deterministic comparison schedule. - Revalidate profile invariants after `--duration` in both the CLI and `run_live_replay`, with an explicit error before any identity/ownership work. - Cancel and drain the churn task (and every mutation task inside its task group) before any client is closed, proven by a real outer-task cancellation test. - Reject unexpected Pods in owned namespaces, and reject any expected Pod that lost its UID identity or either ownership label on the post-churn read. - Make an injected `forbidden` replay failure a named terminal abort instead of a 30-second render timeout; add direct throttled/forbidden/slow coverage. - Name the recorded API errors in wait/convergence timeouts. Quality - Remove dead `current_digest`; add `BenchmarkRecorder.pending_count()`/ `api_errors()`; replace the tautological churn/input ordering flag with a real emitted/dispatched-event signal; give `until` a dedicated `WaitTimeout`; add `get_object` error telemetry; reuse the bound LIST payload items; use the injected clock for live input latency; handle output-path `OSError`; untrack the `.superpowers/sdd/task-4-report.md` session artifact; promote `build_manifest`; connect the mutation client eagerly under a bounded timeout; carry `object_index` on `ScheduledEvent` instead of parsing Pod names; bound the overall churn wait; count only resource-update renders. - Keep exact decoded-byte accounting: measured at 8.8ms for a 1,000-Pod LIST (0.4% of the 2s budget) and 9us per watch event, while the suggested `len(str(payload))` saves 1.3ms and reports Python repr characters instead of bytes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .superpowers/sdd/task-4-report.md | 25 - ...luster-performance-qualification-design.md | 42 +- src/korvid/k8s/client.py | 22 +- tests/k8s/test_client.py | 36 + tests/performance/cli.py | 98 +- tests/performance/live.py | 776 +++++++++++--- tests/performance/manifests.py | 6 + tests/performance/metrics.py | 98 ++ tests/performance/profile.py | 49 +- tests/performance/profiles/aks-live-1k.json | 15 + tests/performance/replay.py | 134 ++- tests/performance/test_cli.py | 101 ++ tests/performance/test_live.py | 997 +++++++++++++++++- tests/performance/test_metrics.py | 83 ++ tests/performance/test_profile.py | 66 +- tests/performance/test_replay.py | 187 +++- tests/performance/test_workload.py | 14 + tests/performance/workload.py | 5 + tests/ui/test_waits.py | 18 +- tests/ui/waits.py | 15 +- 20 files changed, 2503 insertions(+), 284 deletions(-) delete mode 100644 .superpowers/sdd/task-4-report.md create mode 100644 tests/performance/profiles/aks-live-1k.json diff --git a/.superpowers/sdd/task-4-report.md b/.superpowers/sdd/task-4-report.md deleted file mode 100644 index 61190476..00000000 --- a/.superpowers/sdd/task-4-report.md +++ /dev/null @@ -1,25 +0,0 @@ -# Task 4 report - -## RED evidence -Command: -`uv run pytest -p no:tach tests/performance/test_metrics.py -k rolls_back_tracemalloc_if_task_creation_fails -q` - -Observed failure: -- `AssertionError: assert ['start'] == ['start', 'stop']` -- `pytest.PytestUnraisableExceptionWarning: Exception ignored in: ` - -## GREEN evidence -Command: -`uv run pytest -p no:tach tests/performance/test_metrics.py -k rolls_back_tracemalloc_if_task_creation_fails -q` - -Observed success: -- `1 passed, 15 deselected in 0.05s` - -Full file verification: -- `16 passed in 0.04s` - -## Self-review -- Added the smallest regression test for start-up rollback when `asyncio.create_task()` fails after tracemalloc ownership is acquired. -- Fixed `ProcessSampler.start()` with scoped rollback that releases owned tracemalloc, resets sampler state, and closes the unstarted coroutine before re-raising. -- Preserved double-start rejection, overlapping sampler ownership, and externally-owned tracemalloc behavior. -- Verified with focused test, full `tests/performance/test_metrics.py`, ruff, mypy, and `git diff --check`. diff --git a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md index d1b6d2f0..ab6d5075 100644 --- a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md +++ b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md @@ -69,7 +69,27 @@ The initial profiles are: | `smoke-1k` | 1,000 | Fast deterministic correctness and report smoke | Normal CI, no wall-clock assertion | | `steady-10k` | 10,000 | Store/render scaling and sustained churn | Manual or scheduled | | `burst-50k` | 50,000 | Upper-envelope and burst-backlog measurement | Manual or scheduled | -| `live-aks-1k` | 1,000 Pods | Real API, network, LIST/WATCH, and UI qualification | Protected manual run | +| `aks-1k` | 1,000 | Deterministic comparison schedule for the live topology | Manual or scheduled | +| `aks-live-1k` | 1,000 Pods | Real API, network, LIST/WATCH, and UI qualification | Protected manual run | + +`aks-live-1k` (`tests/performance/profiles/aks-live-1k.json`) encodes the live +sequence below exactly - 1,800 seconds at 20 events/s with three 30-second +bursts at 100 events/s - so the event-to-render, backlog-drain, and RSS-slope +budgets in this document are measurable by a single protected run: + +```bash +uv run python -m tests.performance.cli replay-live \ + --profile tests/performance/profiles/aks-live-1k.json \ + --context aks-korvid-contract-test \ + --expected-cluster-id --run-id \ + --json live.json --out live.md +``` + +`aks-1k` keeps the short (30-second) schedule shared with `burst-50k`, so a +live run can be compared against the deterministic 1k/10k/50k baselines on the +same event schedule. `--duration` shortens a live smoke run; every burst and +failure point is re-validated against the shortened duration before the +cluster identity and ownership gates run. The deterministic generator emits stable names, namespaces, UIDs, resource versions, and event order from the profile seed. Repeating a profile with the @@ -184,8 +204,24 @@ The live sequence is: path that runs even after benchmark failure or timeout. Metadata-only updates create real watch traffic without restarting containers -or changing the workload's resource demand. The generator rate and observed API -throttling are both recorded; requested rate is never reported as achieved rate. +or changing the workload's resource demand. Churn writes one dedicated, +non-ownership label (`korvid.dev/performance-tick`) on the Pod's own metadata +under a JSON Patch that first `test`s the Pod UID and both ownership labels. +The kubelet-owned `status` subresource is never written: an externally patched +`status.phase` is reconciled back on the next node sync, which would both +corrupt digest parity and violate the metadata-only rule above. + +The generator rate and observed API throttling are both recorded; requested +rate is never reported as achieved rate. Reports carry the requested event +count and rate next to the observed event count, churn wall time, achieved +rate, and the count of mutation-side 429 retries, which is accounted +separately from application read-path throttles. + +Live churn is driven with explicit bounded concurrency and a bounded per-patch +timeout: a serial driver is capped at one round trip per event and cannot +approach the scheduled rate, which would silently understate the load the +report claims to have applied. Only HTTP 429 is retried, with a bounded policy +that re-issues the identical guarded patch. ### Guardrails diff --git a/src/korvid/k8s/client.py b/src/korvid/k8s/client.py index 6f5715fd..6c7c1af9 100644 --- a/src/korvid/k8s/client.py +++ b/src/korvid/k8s/client.py @@ -191,6 +191,18 @@ def _observe_read( object_count: int = 0, status: int | None = None, ) -> None: + """Report one read to the optional telemetry seam (issue #186). + + `decoded_bytes` is an exact canonical-JSON byte count, not an estimate: + the benchmark's API-load accounting is compared across runs and + profiles, so an approximation (e.g. `len(str(payload))`, which counts + Python `repr` characters and misencodes non-ASCII, `True`, and `None`) + would silently change the reported number without removing the work. + The whole method is inert unless a caller opted into telemetry, and the + measured cost of the exact count is ~8.8 ms for a 1,000-Pod + cluster-wide LIST (0.4% of the 2 s LIST-to-render budget) and ~9 us per + watch event (0.004% of the 250 ms event-to-render budget). + """ if self._read_telemetry is None: return decoded_bytes = 0 @@ -410,7 +422,7 @@ async def list_pods(self, namespace: str) -> list[PodSummary]: raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc items = data.get("items", []) self._observe_read("list", path, payload=data, object_count=len(items)) - return [self._pod_summary(item) for item in data.get("items", [])] + return [self._pod_summary(item) for item in items] async def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSummary]]: """LIST then watch pods; namespace=None watches cluster-wide.""" @@ -669,7 +681,13 @@ async def get_object( ) -> dict[str, Any]: """Fetch the raw manifest for a single object. ApiException → ApiStatusError.""" path = self._object_path(meta, namespace, name) - result = await self._request_json(path) + try: + result = await self._request_json(path) + except ApiStatusError as exc: + # Every other read path records the failure before propagating; + # without this a denied or throttled GET is a hole in the telemetry. + self._observe_read_error(path, exc) + raise self._observe_read("get", path, payload=result, object_count=1) return result diff --git a/tests/k8s/test_client.py b/tests/k8s/test_client.py index a55afc6e..0dc590f1 100644 --- a/tests/k8s/test_client.py +++ b/tests/k8s/test_client.py @@ -744,6 +744,42 @@ async def test_get_object_emits_get_telemetry() -> None: assert seen[0].status is None +async def test_get_object_emits_error_telemetry_and_reraises() -> None: + """Every other read path records an `error` event before propagating; a + silent GET failure leaves a telemetry gap exactly where a benchmark or an + operator is trying to explain a stalled view.""" + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + request_json_mock = AsyncMock(side_effect=ApiStatusError(404, "Not Found")) + + with ( + patch.object(client, "_request_json", request_json_mock), + pytest.raises(ApiStatusError, match="API 404: Not Found"), + ): + await client.get_object(meta, "default", "my-dep") + + assert [event.operation for event in seen] == ["error"] + assert seen[0].path == "/apis/apps/v1/namespaces/default/deployments/my-dep" + assert seen[0].status == 404 + + +async def test_list_pods_reports_the_same_items_it_counted() -> None: + """The telemetry count and the returned summaries must come from one bound + payload read, not two independent `data.get("items", [])` lookups.""" + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + mock_api = MagicMock() + mock_api.list_namespaced_pod = AsyncMock(return_value={"items": [_pod("a"), _pod("b")]}) + + with patch.object(client, "_core_v1", mock_api): + pods = await client.list_pods("default") + + assert [pod.name for pod in pods] == ["a", "b"] + assert [event.operation for event in seen] == ["list"] + assert seen[0].object_count == len(pods) + + async def test_get_object_raises_api_status_error() -> None: """ApiException from the raw GET is wrapped as ApiStatusError.""" client = KubeClient() diff --git a/tests/performance/cli.py b/tests/performance/cli.py index a8542751..c1542c80 100644 --- a/tests/performance/cli.py +++ b/tests/performance/cli.py @@ -7,6 +7,19 @@ uv run python -m tests.performance.cli seed-manifests \\ --run-id TEXT --namespace-count INT --pods-per-namespace INT \\ --node-selector KEY=VALUE --output PATH + uv run python -m tests.performance.cli replay-live \\ + --profile tests/performance/profiles/aks-live-1k.json \\ + --context TEXT --expected-cluster-id TEXT --run-id TEXT \\ + [--duration INT] [--sample-interval FLOAT] \\ + [--json PATH] [--out PATH] [--cpu-profile PATH] [--allocation-snapshot PATH] + +`aks-live-1k` is the live qualification profile: it encodes the published live +plan (1,000 Pods across 20 namespaces, 30 minutes at 20 events/s with three +30-second bursts at 100 events/s), so the design doc's event-to-render, +backlog-drain, and RSS-slope budgets are measurable. `aks-1k` keeps the short +deterministic schedule used to compare a live run against the synthetic +1k/10k/50k baselines; use `--duration` to shorten a live smoke run (bursts and +failure points are re-validated against the shortened duration). """ from __future__ import annotations @@ -27,8 +40,9 @@ from tests.performance.live import run_live_replay from tests.performance.manifests import build_seed_manifests from tests.performance.metrics import BenchmarkReport, render_markdown, report_payload -from tests.performance.profile import WorkloadProfile, load_profile +from tests.performance.profile import WorkloadProfile, load_profile, validate_profile from tests.performance.replay import ReplayOptions, ReplayReport, run_replay +from tests.ui.waits import WaitTimeout def _to_benchmark_report(replay: ReplayReport) -> BenchmarkReport: @@ -43,6 +57,7 @@ def _to_benchmark_report(replay: ReplayReport) -> BenchmarkReport: coalesced_updates=replay.coalesced_updates, dropped_updates=replay.dropped_updates, final_digest=replay.final_digest, + churn=replay.churn, ) @@ -132,7 +147,14 @@ def _build_parser() -> argparse.ArgumentParser: "replay-live", help="Replay churn against an already-seeded, owned real AKS cluster.", ) - lp.add_argument("--profile", required=True, metavar="PATH", help="Workload profile JSON.") + lp.add_argument( + "--profile", + required=True, + metavar="PATH", + help="Workload profile JSON. Use tests/performance/profiles/aks-live-1k.json " + "for a qualification run (the published 30-minute live plan); aks-1k is the " + "short deterministic comparison schedule.", + ) lp.add_argument( "--context", required=True, metavar="TEXT", help="Exact active kubeconfig context." ) @@ -242,16 +264,27 @@ def _flush_allocation_snapshot(path: str) -> None: tracemalloc.stop() -def _write_outputs(args: argparse.Namespace, replay: ReplayReport) -> None: - """Print Markdown to stdout and write optional --out / --json outputs.""" +def _write_outputs(args: argparse.Namespace, replay: ReplayReport) -> int: + """Print Markdown to stdout and write optional --out / --json outputs. + + Returns: + 0 on success, or 1 when a destination path cannot be written - a bad + `--out`/`--json` path is an operational error (as it already is for + `seed-manifests --output`), not a traceback after a long run. + """ benchmark = _to_benchmark_report(replay) markdown = render_markdown(benchmark) sys.stdout.write(markdown) - if args.out_path: - Path(args.out_path).write_text(markdown) - if args.json_path: - payload: dict[str, Any] = {"schema_version": 1, **report_payload(benchmark)} - Path(args.json_path).write_text(json.dumps(payload, indent=2, sort_keys=True)) + try: + if args.out_path: + Path(args.out_path).write_text(markdown) + if args.json_path: + payload: dict[str, Any] = {"schema_version": 1, **report_payload(benchmark)} + Path(args.json_path).write_text(json.dumps(payload, indent=2, sort_keys=True)) + except OSError as exc: + print(f"error writing report: {exc}", file=sys.stderr) + return 1 + return 0 def _cmd_seed_manifests(args: argparse.Namespace) -> int: @@ -299,19 +332,48 @@ def _cmd_replay(args: argparse.Namespace) -> int: replay = _run_with_cpu_profile(profile, options, args.cpu_profile) else: replay = asyncio.run(run_replay(profile, options)) - except (ApiStatusError, AssertionError, OSError) as exc: + except (ApiStatusError, WaitTimeout, OSError) as exc: print(f"error during replay: {exc}", file=sys.stderr) return 1 finally: if args.allocation_snapshot: _flush_allocation_snapshot(args.allocation_snapshot) - _write_outputs(args, replay) + if _write_outputs(args, replay): + return 1 if replay.dropped_updates > 0 or replay.expected_digest != replay.final_digest: return 1 return 0 +def _load_live_profile(args: argparse.Namespace) -> WorkloadProfile | None: + """Load `--profile` and apply `--duration`, or report why it cannot apply. + + `dataclasses.replace` bypasses `load_profile`, so every duration-dependent + invariant (burst containment/overlap, failure-injection bounds) is + re-checked here - before any cluster identity, ownership, or mutation work + is attempted. + + Returns: + The profile to replay, or `None` when it cannot be used (the reason is + already printed to stderr). + """ + try: + profile = load_profile(Path(args.profile)) + except (OSError, UnicodeError, ValueError) as exc: + print(f"error loading profile: {exc}", file=sys.stderr) + return None + if args.duration is None: + return profile + overridden = dataclasses.replace(profile, duration_seconds=args.duration) + try: + validate_profile(overridden) + except ValueError as exc: + print(f"error: --duration {args.duration} invalidates the profile: {exc}", file=sys.stderr) + return None + return overridden + + def _cmd_replay_live(args: argparse.Namespace) -> int: if args.duration is not None and args.duration <= 0: print("error: --duration must be positive", file=sys.stderr) @@ -320,15 +382,10 @@ def _cmd_replay_live(args: argparse.Namespace) -> int: print("error: --sample-interval must be positive", file=sys.stderr) return 1 - try: - profile = load_profile(Path(args.profile)) - except (OSError, UnicodeError, ValueError) as exc: - print(f"error loading profile: {exc}", file=sys.stderr) + profile = _load_live_profile(args) + if profile is None: return 1 - if args.duration is not None: - profile = dataclasses.replace(profile, duration_seconds=args.duration) - # No --time-scale option: live churn always replays at real wall-clock # time (ReplayOptions.time_scale defaults to 1.0). options = ReplayOptions(sample_interval=args.sample_interval) @@ -356,14 +413,15 @@ def _cmd_replay_live(args: argparse.Namespace) -> int: run_id=args.run_id, ) ) - except (ValueError, ApiStatusError, AssertionError, OSError) as exc: + except (ValueError, ApiStatusError, WaitTimeout, OSError) as exc: print(f"error during replay: {exc}", file=sys.stderr) return 1 finally: if args.allocation_snapshot: _flush_allocation_snapshot(args.allocation_snapshot) - _write_outputs(args, replay) + if _write_outputs(args, replay): + return 1 if replay.dropped_updates > 0 or replay.expected_digest != replay.final_digest: return 1 return 0 diff --git a/tests/performance/live.py b/tests/performance/live.py index 7bb115f6..de2c012d 100644 --- a/tests/performance/live.py +++ b/tests/performance/live.py @@ -16,21 +16,44 @@ `az aks show --ids ` lookup must all agree before any client connects. 2. **Ownership gate** (`_verify_ownership`): every expected namespace and every expected Pod must already carry both ownership labels - (`manifests.MANAGED_BY_LABEL`/`manifests.RUN_LABEL`) before any churn is - attempted. + (`manifests.MANAGED_BY_LABEL`/`manifests.RUN_LABEL`), and no *unexpected* + Pod may exist in an owned namespace, before any churn is attempted. 3. **Guarded churn** (`drive_live_churn`): each mutation is a JSON-Patch that - `test`s the target Pod's UID and both ownership labels before `replace`ing - `status.phase`. A failed `test` op aborts the *entire* run - there is no - unguarded fallback. + `test`s the target Pod's UID and both ownership labels before writing a + dedicated, non-ownership `manifests.TICK_LABEL` on the Pod's *own* + metadata. A failed `test` op aborts the *entire* run - there is no + unguarded fallback, and no ownership label, `status`, or spec field is ever + written. +4. **Post-churn revalidation** (`read_and_validate_owned_pods`): the + ground-truth cluster read re-checks the exact identity set, every UID, and + both ownership labels before its digest is trusted. + +Churn is metadata-only by design. The seeded Pods are real +`registry.k8s.io/pause:3.10` Pods whose `status` subresource is owned by the +kubelet: patching `status.phase` would be reverted on the next node sync and +would contradict the design doc's "metadata-only updates create real watch +traffic without restarting containers or changing the workload's resource +demand". `TICK_LABEL` is user-owned, is part of `PodSummary.labels` (so it +really does change the store digest and produce a watch event), and no +controller reconciles it away. + +Event-to-render latency is recorded where the *application* first sees an +event - `make_live_watch_source` records at watch receipt, exactly like the +deterministic `_ReplaySource` - never at patch acknowledgement. The watch +event and the patch response race over independent connections, so recording +at ack both misreports the interval (it would include the write round-trip) +and can append an event *after* its own render. Two separate `KubeReadClient` connections are used deliberately: a non-instrumented *harness* client (`LiveDependencies.harness_kube_client_factory`) -performs the ownership gate, the post-churn independent re-read, and nothing -else, while a single telemetry-wired *application-path* client +performs the ownership gate, the ground-truth re-read, and nothing else, while +a single telemetry-wired *application-path* client (`LiveDependencies.kube_client_factory`) is only ever handed to `make_live_watch_source`. This keeps `ReplayReport.api` reporting exactly the production application read path (the real `KubeClient.watch_pods` LIST+WATCH telemetry) instead of being diluted by the harness's own bookkeeping reads. +Mutation traffic is likewise never reported as application read telemetry: its +throttles are counted separately in `ChurnSummary.mutation_throttles`. Unit tests substitute every external boundary via `LiveDependencies`; none of them may contact Azure, a real kubeconfig, or a real cluster. @@ -40,8 +63,8 @@ import asyncio import json -from collections.abc import AsyncIterator, Awaitable, Callable, Iterable -from dataclasses import dataclass, replace +from collections.abc import AsyncIterator, Awaitable, Callable, Iterable, Mapping +from dataclasses import dataclass, field from time import monotonic from typing import Any, Protocol, cast from urllib.parse import urlparse @@ -59,11 +82,16 @@ from korvid.k8s.telemetry import ReadTelemetry from korvid.ui.widgets.resource_table import ResourceTable from tests.performance import manifests -from tests.performance.metrics import BenchmarkRecorder, ProcessSampler -from tests.performance.profile import WorkloadProfile -from tests.performance.replay import MeasuredKorvidApp, ReplayOptions, ReplayReport, _build_manifest +from tests.performance.metrics import BenchmarkRecorder, ChurnSummary, ProcessSampler +from tests.performance.profile import WorkloadProfile, validate_profile +from tests.performance.replay import ( + MeasuredKorvidApp, + ReplayOptions, + ReplayReport, + build_manifest, + wait_for, +) from tests.performance.workload import ScheduledEvent, scheduled_events, summary_digest -from tests.ui.waits import until #: Namespace-scoped read for the ownership gate; not exposed via `PODS_META` #: because Namespaces are cluster-scoped (`namespaced=False`). @@ -74,11 +102,100 @@ _REQUIRED_OBJECT_COUNT = 1000 _REQUIRED_NAMESPACE_COUNT = 20 +#: HTTP status the mutation path may retry. 429 (API Priority and Fairness) +#: is the only one: it is a "come back later" answer to a well-formed, +#: fully guarded request, and the retry re-issues the *identical* guarded +#: patch. Every other status - including a failed `test` op - aborts the run. +_RETRYABLE_MUTATION_STATUS = 429 + +#: How many times teardown re-attempts draining the churn task when its own +#: await is interrupted by a cancellation aimed at the caller. +_CANCEL_DRAIN_ATTEMPTS = 3 + async def _sleep_default(delay: float) -> None: await asyncio.sleep(delay) +@dataclass(frozen=True) +class LiveLimits: + """Every explicit bound the live run enforces. + + Nothing in a live run may be unbounded: an in-flight patch, the number of + concurrent patches, the wait for churn to finish, the wait for the store to + converge, and even the initial kubeconfig/credential-plugin connection all + have a stated ceiling. Tests inject small values; production uses defaults + sized for the published 30-minute live profile. + + Args: + churn_concurrency: Maximum simultaneously in-flight guarded patches. + Serial patching cannot approach the scheduled rate (one round trip + per event), so concurrency is explicit rather than implicit. + mutation_timeout_seconds: Ceiling for one guarded patch attempt. + mutation_throttle_retries: Bounded retries of the *identical* guarded + patch after HTTP 429, and only after 429. + mutation_retry_base_delay_seconds: First backoff delay; doubles per + retry. + mutation_connect_timeout_seconds: Ceiling for connecting the mutation + client (a kubeconfig exec credential plugin can block). + initial_render_timeout_seconds: Ceiling for the initial 1,000-row + render. + churn_grace_seconds: Allowance added to the profile's own scheduled + duration when bounding the churn wait. + convergence_timeout_seconds: Ceiling for the store digest to converge + with the independently read cluster digest. + """ + + churn_concurrency: int = 32 + mutation_timeout_seconds: float = 30.0 + mutation_throttle_retries: int = 5 + mutation_retry_base_delay_seconds: float = 0.5 + mutation_connect_timeout_seconds: float = 60.0 + initial_render_timeout_seconds: float = 60.0 + churn_grace_seconds: float = 300.0 + convergence_timeout_seconds: float = 120.0 + + +@dataclass +class ChurnProgress: + """Live counters for the churn phase, observable while it runs. + + Kept separate from `ChurnSummary` (the frozen report value) because + `run_live_replay` reads `started` *during* the run to decide whether churn + was really under way when input latency was measured. + """ + + requested_events: int = 0 + started: int = 0 + completed: int = 0 + mutation_throttles: int = 0 + first_started_at: float | None = None + last_completed_at: float | None = None + + def record_started(self, at: float) -> None: + if self.first_started_at is None: + self.first_started_at = at + self.started += 1 + + def record_completed(self, at: float) -> None: + self.completed += 1 + self.last_completed_at = at + + def wall_seconds(self) -> float | None: + if self.first_started_at is None or self.last_completed_at is None: + return None + return self.last_completed_at - self.first_started_at + + def summary(self, *, requested_duration_seconds: int) -> ChurnSummary: + return ChurnSummary.from_observations( + requested_events=self.requested_events, + requested_duration_seconds=requested_duration_seconds, + observed_events=self.completed, + wall_seconds=self.wall_seconds(), + mutation_throttles=self.mutation_throttles, + ) + + @dataclass(frozen=True) class CommandResult: """Captured outcome of a subprocess invocation (e.g. `az aks show`).""" @@ -116,11 +233,17 @@ def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSumma class MutationClient(Protocol): - """Issues one guarded status mutation; production talks JSON-Patch to a - real API server, tests mutate an in-memory fake cluster the same way.""" + """Issues one guarded, metadata-only mutation; production talks JSON-Patch + to a real API server, tests mutate an in-memory fake cluster the same way. - async def patch_pod_status_guarded( - self, namespace: str, name: str, *, uid: str, phase: str + There is deliberately no method that can write `status`, `spec`, an + ownership label, or delete anything. + """ + + async def connect(self) -> None: ... + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str ) -> None: ... async def close(self) -> None: ... @@ -140,8 +263,8 @@ class LiveDependencies: (`make_live_watch_source`'s `watch_pods`), so `ReplayReport.api` reports exactly the production LIST+WATCH telemetry an operator would see. The latter is never wired to telemetry and is used *only* for the harness's - own bookkeeping reads (the ownership gate and the post-churn independent - re-read) - those reads must never dilute the application-path signal. + own bookkeeping reads (the ownership gate and the ground-truth re-read) - + those reads must never dilute the application-path signal. """ command_runner: CommandRunner @@ -202,26 +325,34 @@ def _json_pointer_escape(segment: str) -> str: return segment.replace("~", "~0").replace("/", "~1") -def build_guarded_status_patch(*, uid: str, run_id: str, phase: str) -> list[dict[str, Any]]: - """The exact JSON-Patch op list a guarded status mutation issues. +def build_guarded_label_patch(*, uid: str, run_id: str, tick: str) -> list[dict[str, Any]]: + """The exact JSON-Patch op list a guarded churn mutation issues. + + `test`s the target Pod's UID and *both* ownership labels, then `add`s the + dedicated non-ownership `manifests.TICK_LABEL`, so a stale, foreign, or + replaced Pod aborts the whole patch server-side - there is no unguarded + fallback. `add` (not `replace`) is used for the tick because RFC 6902 + `replace` requires the member to exist already, while `add` on an object + member creates or overwrites it; the seeded Pods start without it. - `test`s the target Pod's UID and both ownership labels before - `replace`-ing `status.phase`, so a stale, foreign, or replaced Pod aborts - the whole patch server-side - there is no unguarded fallback. + Nothing outside `metadata.labels` is written: no `status`, no `spec`, and + neither ownership label is ever a target of a write op. """ managed_by_path = f"/metadata/labels/{_json_pointer_escape(manifests.MANAGED_BY_LABEL)}" run_path = f"/metadata/labels/{_json_pointer_escape(manifests.RUN_LABEL)}" + tick_path = f"/metadata/labels/{_json_pointer_escape(manifests.TICK_LABEL)}" return [ {"op": "test", "path": "/metadata/uid", "value": uid}, {"op": "test", "path": managed_by_path, "value": manifests.MANAGED_BY_VALUE}, {"op": "test", "path": run_path, "value": run_id}, - {"op": "replace", "path": "/status/phase", "value": phase}, + {"op": "add", "path": tick_path, "value": tick}, ] class _KubeMutationClient: - """Production `MutationClient`: issues a guarded JSON-Patch against the - real `pods/status` subresource of *context*.""" + """Production `MutationClient`: issues a guarded, metadata-only JSON-Patch + against the Pod resource itself (never the `pods/status` subresource) of + *context*.""" def __init__(self, context: str, run_id: str) -> None: self._context = context @@ -229,23 +360,34 @@ def __init__(self, context: str, run_id: str) -> None: self._api: k8s_client.ApiClient | None = None self._core_v1: k8s_client.CoreV1Api | None = None - async def _ensure_connected(self) -> k8s_client.CoreV1Api: - if self._core_v1 is None: - configuration = k8s_client.Configuration() - await k8s_config.load_kube_config( - context=self._context, client_configuration=configuration, persist_config=False - ) - self._api = k8s_client.ApiClient(configuration) - self._core_v1 = k8s_client.CoreV1Api(self._api) - return self._core_v1 + async def connect(self) -> None: + """Load the kubeconfig and build the API client. + + Called eagerly (under `LiveLimits.mutation_connect_timeout_seconds`) + before the measured window opens: `load_kube_config` can invoke an + exec credential plugin, and that latency must not be charged to the + first churn mutation - `KubeClient.probe_context` bounds exactly the + same call for exactly the same reason. + """ + if self._core_v1 is not None: + return + configuration = k8s_client.Configuration() + await k8s_config.load_kube_config( + context=self._context, client_configuration=configuration, persist_config=False + ) + self._api = k8s_client.ApiClient(configuration) + self._core_v1 = k8s_client.CoreV1Api(self._api) - async def patch_pod_status_guarded( - self, namespace: str, name: str, *, uid: str, phase: str + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str ) -> None: - core_v1 = await self._ensure_connected() - ops = build_guarded_status_patch(uid=uid, run_id=self._run_id, phase=phase) + await self.connect() + core_v1 = self._core_v1 + if core_v1 is None: # pragma: no cover - connect() always sets it + raise RuntimeError("mutation client is not connected") + ops = build_guarded_label_patch(uid=uid, run_id=self._run_id, tick=tick) try: - await core_v1.patch_namespaced_pod_status( + await core_v1.patch_namespaced_pod( name, namespace, ops, @@ -266,7 +408,7 @@ def _default_dependencies(context: str) -> LiveDependencies: telemetry-wired one `run_live_replay` hands to `make_live_watch_source`, and a plain, non-instrumented one (`read_telemetry=None`, so `KubeClient._observe_read` is a no-op) for the harness's own ownership - and post-churn reads. + and ground-truth reads. """ return LiveDependencies( command_runner=_default_command_runner, @@ -360,11 +502,25 @@ async def _verify_cluster_identity( ) +def _expected_pod_names(namespace_count: int, object_count: int) -> tuple[str, ...]: + """The exact Pod names `seed-manifests` creates in *every* namespace.""" + pods_per_namespace = object_count // namespace_count + return tuple( + manifests.pod_name(namespace_count, local_index * namespace_count) + for local_index in range(pods_per_namespace) + ) + + async def _verify_ownership( kube: KubeReadClient, *, run_id: str, namespace_count: int, object_count: int ) -> dict[tuple[str, str], PodSummary]: - """Ownership gate: every expected namespace and every expected Pod must - already exist with both ownership labels, checked before any churn. + """Ownership gate: *exactly* the expected namespaces and Pods must exist, + each with both ownership labels, checked before any churn. + + An unexpected Pod inside an owned namespace is rejected too: the + application watch filters by namespace, so a foreign Pod would enter the + benchmark store and turn a precise ownership violation into a generic + "1,000 rows never rendered" timeout minutes later. Collects every mismatch across all namespaces/Pods before raising, so a single failed run surfaces the full blast radius at once instead of @@ -389,17 +545,17 @@ async def _verify_ownership( f"namespaces missing/mismatched ownership labels: {', '.join(mismatched_namespaces)}" ) - pods_per_namespace = object_count // namespace_count - expected_pod_names = [ - manifests.pod_name(namespace_count, local_index * namespace_count) - for local_index in range(pods_per_namespace) - ] + wanted = _expected_pod_names(namespace_count, object_count) missing_pods: list[str] = [] mismatched_pods: list[str] = [] + unexpected_pods: list[str] = [] validated_pods: dict[tuple[str, str], PodSummary] = {} for namespace in expected_namespaces: pods_by_name = {pod.name: pod for pod in await kube.list_pods(namespace)} - for name in expected_pod_names: + unexpected_pods.extend( + f"{namespace}/{name}" for name in sorted(set(pods_by_name) - set(wanted)) + ) + for name in wanted: pod = pods_by_name.get(name) if pod is None: missing_pods.append(f"{namespace}/{name}") @@ -411,61 +567,379 @@ async def _verify_ownership( raise ValueError(f"missing expected pods: {', '.join(missing_pods)}") if mismatched_pods: raise ValueError(f"pods with mismatched ownership labels: {', '.join(mismatched_pods)}") + if unexpected_pods: + raise ValueError(f"unexpected pods in owned namespaces: {', '.join(unexpected_pods)}") return validated_pods +async def read_and_validate_owned_pods( + kube: KubeReadClient, + *, + run_id: str, + expected: Mapping[tuple[str, str], PodSummary], +) -> list[PodSummary]: + """Independently re-read the owned Pods and revalidate them before use. + + The ground-truth digest may only be computed from Pods that still are what + the ownership gate validated: same `(namespace, name)` identity set, same + UID (a delete/recreate produces a new one), and both ownership labels + intact. A Pod that lost either label, was replaced, disappeared, or a Pod + that appeared unexpectedly, is named in the raised error instead of being + silently folded into the digest. + """ + namespaces = sorted({namespace for namespace, _name in expected}) + found: dict[tuple[str, str], PodSummary] = {} + seen: set[tuple[str, str]] = set() + problems: list[str] = [] + for namespace in namespaces: + for pod in await kube.list_pods(namespace): + key = (namespace, pod.name) + baseline = expected.get(key) + if baseline is None: + problems.append(f"{namespace}/{pod.name} (unexpected pod)") + continue + seen.add(key) + if pod.uid != baseline.uid: + problems.append( + f"{namespace}/{pod.name} (uid changed {baseline.uid!r} -> {pod.uid!r})" + ) + elif not _owns(pod.labels, run_id): + problems.append(f"{namespace}/{pod.name} (lost ownership labels)") + else: + found[key] = pod + problems.extend( + f"{namespace}/{name} (missing)" + for namespace, name in expected + if (namespace, name) not in seen + ) + if problems: + raise ValueError(f"post-churn ownership revalidation failed: {', '.join(sorted(problems))}") + return list(found.values()) + + def make_live_watch_source( - kube: KubeReadClient, expected_namespaces: frozenset[str] + kube: KubeReadClient, + expected_namespaces: frozenset[str], + *, + run_id: str, + recorder: BenchmarkRecorder, + now: Callable[[], float] = monotonic, ) -> WatchSource: - """Filter the real cluster-wide Pod watch to exactly the expected, - seeded namespaces - unrelated cluster Pods (or namespaces sharing the - cluster with this run) must never enter the benchmark store.""" + """Filter the real cluster-wide Pod watch to exactly the expected, seeded + namespaces - unrelated cluster Pods (or namespaces sharing the cluster with + this run) must never enter the benchmark store. + + Event-to-render measurement starts *here*, at watch receipt of an owned + `MODIFIED` event, exactly where the deterministic `_ReplaySource` starts + it. Recording at patch acknowledgement instead would race the watch event + it is meant to measure (the two arrive over independent connections) and + would silently fold the write round-trip into a read-path latency metric. + + The event counter is shared across reconnects, so a re-LIST after a dropped + watch continues the same sequence. + """ + sequence = 0 async def _source(kind: str, _scope: str) -> AsyncIterator[tuple[str, PodSummary]]: + nonlocal sequence if kind != "pods": raise ValueError(f"run_live_replay only watches pods, got kind={kind!r}") async for event_type, pod in kube.watch_pods(None): - if pod.namespace in expected_namespaces: - yield (event_type, pod) + if pod.namespace not in expected_namespaces: + continue + if event_type == "MODIFIED" and _owns(pod.labels, run_id): + sequence += 1 + recorder.record_event(sequence, now()) + yield (event_type, pod) return _source +def _first_error(group: BaseExceptionGroup[BaseException]) -> BaseException: + """The most informative leaf of a `TaskGroup` failure. + + A guard failure that aborts the run is reported to the caller as the + `ApiStatusError` it really is, not as an `ExceptionGroup` wrapper. Sibling + tasks cancelled *because* of that failure contribute `CancelledError` + leaves, which are only returned when there is nothing else. + """ + leaves: list[BaseException] = [] + pending: list[BaseException] = list(group.exceptions) + while pending: + exc = pending.pop(0) + if isinstance(exc, BaseExceptionGroup): + pending.extend(exc.exceptions) + else: + leaves.append(exc) + for exc in leaves: + if not isinstance(exc, asyncio.CancelledError): + return exc + return leaves[0] if leaves else group + + +async def _mutate_once( + mutation_client: MutationClient, + *, + namespace: str, + name: str, + uid: str, + tick: str, + progress: ChurnProgress, + limits: LiveLimits, + sleep: Callable[[float], Awaitable[None]], + now: Callable[[], float], +) -> None: + """One guarded patch, bounded in time and retried only on HTTP 429. + + The retry re-issues the *identical* guarded patch, so it is still atomic + and still fail-closed: a Pod that lost its identity or labels between + attempts fails the `test` ops exactly as it would on the first attempt. + Every other status - including a failed `test` (422) - propagates + immediately and aborts the whole run. + """ + progress.record_started(now()) + attempt = 0 + while True: + try: + async with asyncio.timeout(limits.mutation_timeout_seconds): + await mutation_client.patch_pod_labels_guarded(namespace, name, uid=uid, tick=tick) + except ApiStatusError as exc: + if ( + exc.status != _RETRYABLE_MUTATION_STATUS + or attempt >= limits.mutation_throttle_retries + ): + raise + progress.mutation_throttles += 1 + attempt += 1 + await sleep(limits.mutation_retry_base_delay_seconds * (2 ** (attempt - 1))) + continue + progress.record_completed(now()) + return + + async def drive_live_churn( events: Iterable[ScheduledEvent], *, run_id: str, namespace_count: int, - live_state: dict[tuple[str, str], PodSummary], + live_state: Mapping[tuple[str, str], PodSummary], mutation_client: MutationClient, - recorder: BenchmarkRecorder, options: ReplayOptions, + progress: ChurnProgress, + limits: LiveLimits, ) -> None: - """Drive guarded churn at wall-clock time (matching `_ReplaySource`'s - inter-event delay math). Any guard failure (`ApiStatusError`) propagates - immediately and unconditionally aborts the run - there is no unguarded - fallback and no attempt to continue past a failed `test` op. + """Drive guarded churn at wall-clock time with explicit bounded concurrency. + + The schedule is followed exactly as `_ReplaySource` follows it (absolute + offsets converted to inter-event delays), but each mutation is dispatched + to a task instead of being awaited inline: one round trip per event caps a + serial driver far below the profile's scheduled rate, which would silently + understate the load the report claims to have applied. + + Concurrency is bounded by `LiveLimits.churn_concurrency` and every single + attempt by `LiveLimits.mutation_timeout_seconds`; nothing here is + unbounded. Any guard failure (`ApiStatusError`) cancels every sibling task + and propagates unchanged - there is no unguarded fallback and no attempt to + continue past a failed `test` op. + + This function never records event timing: event-to-render measurement + starts at watch receipt (`make_live_watch_source`), not at patch ack. """ now = options.monotonic_fn if options.monotonic_fn is not None else monotonic sleep = options.async_sleep if options.async_sleep is not None else _sleep_default + semaphore = asyncio.Semaphore(limits.churn_concurrency) start = now() - for event in events: - elapsed = now() - start - delay = event.offset_seconds * options.time_scale - elapsed - if delay > 0: - await sleep(delay) - - index = int(event.summary.name.removeprefix("pod-")) - namespace, name = live_object_identity(run_id, namespace_count, index) - current = live_state.get((namespace, name)) - if current is None: - raise ValueError(f"live churn target {namespace}/{name} is not in the seeded state") - - await mutation_client.patch_pod_status_guarded( - namespace, name, uid=current.uid, phase=event.summary.phase + + async def _run(namespace: str, name: str, uid: str, tick: str) -> None: + try: + await _mutate_once( + mutation_client, + namespace=namespace, + name=name, + uid=uid, + tick=tick, + progress=progress, + limits=limits, + sleep=sleep, + now=now, + ) + finally: + semaphore.release() + + try: + async with asyncio.TaskGroup() as group: + for event in events: + elapsed = now() - start + delay = event.offset_seconds * options.time_scale - elapsed + if delay > 0: + await sleep(delay) + + namespace, name = live_object_identity(run_id, namespace_count, event.object_index) + current = live_state.get((namespace, name)) + if current is None: + raise ValueError( + f"live churn target {namespace}/{name} is not in the seeded state" + ) + # Bound in-flight work *before* creating the task, so a slow + # API server throttles the driver instead of accumulating an + # unbounded backlog of pending patches. + await semaphore.acquire() + group.create_task(_run(namespace, name, current.uid, str(event.sequence))) + except BaseExceptionGroup as group_error: + raise _first_error(group_error) from None + + +async def _cancel_and_drain(task: asyncio.Task[None]) -> None: + """Cancel *task* and wait for it to finish before any client is closed. + + Every in-flight mutation lives inside the churn task's `TaskGroup`, so + awaiting the churn task is exactly what guarantees no patch is still in + flight when `mutation_client.close()` runs - and that no mutation outlives + teardown. A cancellation delivered to *this* coroutine while it drains is + absorbed and retried a bounded number of times: the caller's own + `CancelledError` is already propagating out of the enclosing `try`, and + skipping the drain is precisely the failure mode this exists to prevent. + """ + task.cancel() + for _ in range(_CANCEL_DRAIN_ATTEMPTS): + if task.done(): + break + try: + await asyncio.wait({task}) + except asyncio.CancelledError: + continue + if task.done() and not task.cancelled(): + # Retrieve any exception so it is never reported as "never retrieved"; + # the error that triggered teardown is the one the caller sees. + task.exception() + + +def _store_digest(store: ResourceStore) -> str: + return summary_digest(cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES))) + + +def _check_row_count(row_count: int, expected: int) -> None: + """Re-assert the exact rendered row count before teardown. + + A late `WatchManager` reconnect clears and re-seeds the store, so a digest + that matched a moment ago can be recomputed over a partially re-listed + store. Checking the rendered row count at the same instant turns that into + an explicit, named failure instead of a plausible-looking report. + """ + if row_count != expected: + raise ValueError( + f"rendered row count regressed before teardown: expected {expected}, got {row_count}" + ) + + +@dataclass +class _LiveRunState: + """Values produced inside the measured window and consumed after it.""" + + expected_digest: str = "" + final_digest: str = "" + churn_started_before_input: bool = False + progress: ChurnProgress = field(default_factory=ChurnProgress) + + +async def _run_measured_window( + *, + app: MeasuredKorvidApp, + store: ResourceStore, + harness_kube: KubeReadClient, + mutation_client: MutationClient, + recorder: BenchmarkRecorder, + profile: WorkloadProfile, + options: ReplayOptions, + limits: LiveLimits, + run_id: str, + live_state: dict[tuple[str, str], PodSummary], + state: _LiveRunState, +) -> None: + """Drive the app: initial render, churn, ground-truth read, convergence. + + Everything that must observe a *live* watch happens inside this function, + including the independent cluster read and the digest convergence wait. + Reading ground truth after the watch had already been stopped would compare + a frozen store against a cluster that was still changing. + """ + now = options.monotonic_fn if options.monotonic_fn is not None else monotonic + async with app.run_test() as pilot: + table = app.query_one(ResourceTable) + + await wait_for( + pilot, + lambda: table.row_count == profile.object_count, + timeout=limits.initial_render_timeout_seconds, + label="initial owned pods rendered", + recorder=recorder, + ) + + events = scheduled_events(profile) + state.progress.requested_events = len(events) + churn_task = asyncio.create_task( + drive_live_churn( + events, + run_id=run_id, + namespace_count=profile.namespace_count, + live_state=live_state, + mutation_client=mutation_client, + options=options, + progress=state.progress, + limits=limits, + ) ) - live_state[(namespace, name)] = replace(current, phase=event.summary.phase) - recorder.record_event(event.sequence, now()) + try: + if events: + # Real ordering signal: input latency is only meaningful if + # churn was actually under way, so wait for the first mutation + # to be dispatched (or for churn to fail) before measuring. + await wait_for( + pilot, + lambda: state.progress.started > 0 or churn_task.done(), + timeout=limits.initial_render_timeout_seconds, + label="first churn mutation started", + recorder=recorder, + ) + state.churn_started_before_input = state.progress.started > 0 + + t0 = now() + await pilot.press("down") + recorder.record_input(now() - t0) + t0 = now() + await pilot.press("up") + recorder.record_input(now() - t0) + + churn_timeout = ( + profile.duration_seconds * options.time_scale + limits.churn_grace_seconds + ) + async with asyncio.timeout(churn_timeout): + await churn_task + + # Ground truth, read independently *while the watch is still live* + # via the non-instrumented harness client (never polluting the + # application-path telemetry), and revalidated for ownership. + final_pods = await read_and_validate_owned_pods( + harness_kube, run_id=run_id, expected=live_state + ) + state.expected_digest = summary_digest(final_pods) + + await wait_for( + pilot, + lambda: ( + recorder.pending_count() == 0 and _store_digest(store) == state.expected_digest + ), + timeout=limits.convergence_timeout_seconds, + label=( + "store digest converged with the independently read cluster digest " + f"({state.expected_digest})" + ), + recorder=recorder, + ) + _check_row_count(table.row_count, profile.object_count) + state.final_digest = _store_digest(store) + finally: + await _cancel_and_drain(churn_task) async def run_live_replay( @@ -476,18 +950,20 @@ async def run_live_replay( expected_cluster_id: str, run_id: str, deps: LiveDependencies | None = None, + limits: LiveLimits | None = None, ) -> ReplayReport: """Replay churn against an already-seeded real AKS cluster and return metrics. - Fail-closed order: `time_scale`/`run_id`/topology validation, the cluster - identity gate, the ownership gate - all *before* any mutation - then the - real application-path wiring (`KubeClient` -> `WatchManager` -> - `ResourceStore` -> `MeasuredKorvidApp`), guarded churn, and digest parity - against an independent post-churn re-read of the cluster. + Fail-closed order: `time_scale`/`run_id`/topology/profile validation, the + cluster identity gate, the ownership gate - all *before* any mutation + client is constructed - then the real application-path wiring + (`KubeClient` -> `WatchManager` -> `ResourceStore` -> `MeasuredKorvidApp`), + guarded metadata-only churn, and digest parity against an independent, + revalidated re-read of the cluster taken while the watch is still live. Two `KubeReadClient` connections are used: `harness_kube` (never wired to - telemetry) performs the ownership gate and the post-churn independent - re-read; `kube` (wired to `recorder.record_api`) is only ever handed to + telemetry) performs the ownership gate and the ground-truth re-read; + `kube` (wired to `recorder.record_api`) is only ever handed to `make_live_watch_source`, so `ReplayReport.api` reports exactly the production application read path. The ownership gate's validated Pod snapshot is reused as the pre-churn uid snapshot - it is never re-listed. @@ -495,7 +971,12 @@ async def run_live_replay( _validate_time_scale(options) manifests.validate_run_id(run_id) _validate_topology(profile) + # Re-check duration-dependent invariants: a caller may hand us a profile + # rewritten with `dataclasses.replace` (the CLI's `--duration`), which + # bypasses `load_profile` entirely. + validate_profile(profile) + active_limits = limits if limits is not None else LiveLimits() active_deps = deps if deps is not None else _default_dependencies(context) await _verify_cluster_identity( context=context, expected_cluster_id=expected_cluster_id, deps=active_deps @@ -504,6 +985,7 @@ async def run_live_replay( store = ResourceStore() recorder = BenchmarkRecorder() sampler = ProcessSampler(options.sample_interval) + state = _LiveRunState() harness_kube = active_deps.harness_kube_client_factory() await harness_kube.connect(context) @@ -521,98 +1003,76 @@ async def run_live_replay( manifests.namespace_name(run_id, i) for i in range(profile.namespace_count) ) - kube = active_deps.kube_client_factory(recorder.record_api) - await kube.connect(context) + # Constructed only after the ownership gate passes, then connected + # eagerly under an explicit bound so credential-plugin latency is paid + # before the measured window rather than inside the first mutation. + mutation_client = active_deps.mutation_client_factory(run_id) try: - source = make_live_watch_source(kube, expected_namespaces) - watch_manager = WatchManager(store, source, retry_delay=0.0) - manifest = _build_manifest(profile) - mutation_client = active_deps.mutation_client_factory(run_id) - - app = MeasuredKorvidApp( - config=KorvidConfig(namespace=ALL_NAMESPACES), - store=store, - watch_manager=watch_manager, - recorder=recorder, - ) + async with asyncio.timeout(active_limits.mutation_connect_timeout_seconds): + await mutation_client.connect() - sampler.start() - churn_started_before_input = False + kube = active_deps.kube_client_factory(recorder.record_api) + await kube.connect(context) try: - async with app.run_test() as pilot: - table = app.query_one(ResourceTable) - - await until( - pilot, - lambda: table.row_count == profile.object_count, - timeout=60.0, - label="initial owned pods rendered", - ) - - events = scheduled_events(profile) - churn_task = asyncio.create_task( - drive_live_churn( - events, - run_id=run_id, - namespace_count=profile.namespace_count, - live_state=live_state, - mutation_client=mutation_client, - recorder=recorder, - options=options, - ) - ) - churn_started_before_input = True - - t0 = monotonic() - await pilot.press("down") - recorder.record_input(monotonic() - t0) - t0 = monotonic() - await pilot.press("up") - recorder.record_input(monotonic() - t0) - - await churn_task - - await until( - pilot, - lambda: not recorder._pending_events, - timeout=60.0, - label="churn complete and all events rendered", + source = make_live_watch_source( + kube, + expected_namespaces, + run_id=run_id, + recorder=recorder, + ) + watch_manager = WatchManager(store, source, retry_delay=0.0) + manifest = build_manifest(profile) + + app = MeasuredKorvidApp( + config=KorvidConfig(namespace=ALL_NAMESPACES), + store=store, + watch_manager=watch_manager, + recorder=recorder, + ) + + sampler.start() + try: + await _run_measured_window( + app=app, + store=store, + harness_kube=harness_kube, + mutation_client=mutation_client, + recorder=recorder, + profile=profile, + options=options, + limits=active_limits, + run_id=run_id, + live_state=live_state, + state=state, ) + finally: + process_samples = await sampler.stop() + await watch_manager.stop_all() finally: - process_samples = await sampler.stop() - await watch_manager.stop_all() - await mutation_client.close() - - # Independently re-read the cluster's actual Pods for ground-truth - # digest parity via the non-instrumented harness client, rather - # than trusting the driver's own bookkeeping or polluting the - # application-path telemetry with this harness-only read. - final_pods: list[PodSummary] = [] - for namespace in expected_namespaces: - final_pods.extend(await harness_kube.list_pods(namespace)) - expected_digest = summary_digest(final_pods) - final_digest = summary_digest( - cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES)) - ) + await kube.close() finally: - await kube.close() + await mutation_client.close() finally: await harness_kube.close() - benchmark = recorder.report(manifest, process_samples, final_digest=final_digest) + churn = state.progress.summary(requested_duration_seconds=profile.duration_seconds) + benchmark = recorder.report( + manifest, process_samples, final_digest=state.final_digest, churn=churn + ) return ReplayReport( object_count=profile.object_count, - expected_digest=expected_digest, - final_digest=final_digest, + expected_digest=state.expected_digest, + final_digest=state.final_digest, dropped_updates=benchmark.dropped_updates, rendered_updates=benchmark.rendered_updates, render_passes=benchmark.render_passes, coalesced_updates=benchmark.coalesced_updates, event_to_render=benchmark.event_to_render, input_latency=benchmark.input_latency, - churn_started_before_input=churn_started_before_input, + churn_started_before_input=state.churn_started_before_input, process=benchmark.process, api=benchmark.api, manifest=benchmark.manifest, + churn=churn, ) diff --git a/tests/performance/manifests.py b/tests/performance/manifests.py index 5b26c02e..fb22353d 100644 --- a/tests/performance/manifests.py +++ b/tests/performance/manifests.py @@ -14,6 +14,12 @@ MANAGED_BY_VALUE = _MANAGED_BY RUN_LABEL = "korvid.dev/performance-run" +#: Dedicated, *non-ownership* label the live harness rewrites to generate watch +#: traffic. It is never part of the ownership contract (nothing gates on its +#: value), it is user-owned so no controller reconciles it back, and churning it +#: keeps live mutations metadata-only exactly as the design doc requires. +TICK_LABEL = "korvid.dev/performance-tick" + def _validate_positive(value: int, label: str) -> int: if value < 1: diff --git a/tests/performance/metrics.py b/tests/performance/metrics.py index 459659a8..f8018408 100644 --- a/tests/performance/metrics.py +++ b/tests/performance/metrics.py @@ -175,6 +175,55 @@ def from_events(cls, events: Sequence[ReadTelemetryEvent]) -> ApiSummary: ) +@dataclass(frozen=True) +class ChurnSummary: + """What the churn generator was *asked* to do versus what it observably did. + + The design doc is explicit that "the generator rate and observed API + throttling are both recorded; requested rate is never reported as achieved + rate", so the requested schedule and the measured outcome are separate, + separately labelled fields. `mutation_throttles` counts 429 responses to + the harness's own write traffic and is deliberately *not* merged into + `ApiSummary.throttles`, which reports only the application read path. + """ + + requested_events: int + requested_events_per_second: float | None + observed_events: int + wall_seconds: float | None + achieved_events_per_second: float | None + mutation_throttles: int + + @classmethod + def from_observations( + cls, + *, + requested_events: int, + requested_duration_seconds: int, + observed_events: int, + wall_seconds: float | None, + mutation_throttles: int, + ) -> ChurnSummary: + requested_rate = ( + requested_events / requested_duration_seconds + if requested_duration_seconds > 0 + else None + ) + achieved_rate = ( + observed_events / wall_seconds + if wall_seconds is not None and wall_seconds > 0 + else None + ) + return cls( + requested_events=requested_events, + requested_events_per_second=requested_rate, + observed_events=observed_events, + wall_seconds=wall_seconds, + achieved_events_per_second=achieved_rate, + mutation_throttles=mutation_throttles, + ) + + @dataclass(frozen=True) class BenchmarkReport: manifest: RunManifest @@ -187,6 +236,8 @@ class BenchmarkReport: coalesced_updates: int dropped_updates: int final_digest: str + #: Present only for runs that drive real mutations (live replay). + churn: ChurnSummary | None = None class ProcessSampler: @@ -279,6 +330,22 @@ def __init__(self) -> None: def record_event(self, sequence: int, received_at: float) -> None: self._pending_events.append((sequence, received_at)) + def pending_count(self) -> int: + """Number of recorded events not yet flushed by a render pass. + + Public so replay/live harnesses can wait for the backlog to drain + without reaching into the recorder's internals. + """ + return len(self._pending_events) + + def api_errors(self) -> tuple[ReadTelemetryEvent, ...]: + """Every recorded `error` read-telemetry event, in arrival order. + + Public so a harness can turn an opaque wait timeout into a message + naming the underlying API status (403/410/429) that caused it. + """ + return tuple(event for event in self._api_events if event.operation == "error") + def record_render(self, rendered_at: float) -> None: if not self._pending_events: return @@ -302,6 +369,7 @@ def report( process_samples: Sequence[ProcessSample], *, final_digest: str, + churn: ChurnSummary | None = None, ) -> BenchmarkReport: return BenchmarkReport( manifest=manifest, @@ -314,6 +382,7 @@ def report( coalesced_updates=self._coalesced_updates, dropped_updates=len(self._pending_events), final_digest=final_digest, + churn=churn, ) @@ -371,11 +440,26 @@ def report_payload(report: BenchmarkReport) -> dict[str, object]: "coalesced_updates": report.coalesced_updates, "dropped_updates": report.dropped_updates, }, + "churn": _churn_payload(report.churn), "digests": {"final": report.final_digest}, } +def _churn_payload(churn: ChurnSummary | None) -> dict[str, object] | None: + if churn is None: + return None + return { + "requested_events": churn.requested_events, + "requested_events_per_second": churn.requested_events_per_second, + "observed_events": churn.observed_events, + "wall_seconds": churn.wall_seconds, + "achieved_events_per_second": churn.achieved_events_per_second, + "mutation_throttles": churn.mutation_throttles, + } + + def render_markdown(report: BenchmarkReport) -> str: + churn = report.churn operation_lines = [ f"- {operation}: `{count}`" for operation, count in report.api.operations.items() ] @@ -398,6 +482,14 @@ def render_markdown(report: BenchmarkReport) -> str: f"- RSS max: `{_format_int(report.process.rss_bytes_max)}`", f"- RSS slope: `{_format_slope(report.process.rss_slope_mib_per_minute)}`", "", + "## Churn", + f"- Requested events: `{_format_int(churn.requested_events if churn else None)}`", + f"- Requested churn rate: `{_format_rate(churn.requested_events_per_second if churn else None)}`", + f"- Observed events: `{_format_int(churn.observed_events if churn else None)}`", + f"- Churn wall time: `{_format_seconds(churn.wall_seconds if churn else None)}`", + f"- Achieved churn rate: `{_format_rate(churn.achieved_events_per_second if churn else None)}`", + f"- Mutation throttles (429): `{_format_int(churn.mutation_throttles if churn else None)}`", + "", "## Updates", f"- Rendered updates: `{report.rendered_updates}`", f"- Coalesced updates: `{report.coalesced_updates}`", @@ -430,6 +522,12 @@ def _format_int(value: int | None) -> str: return str(value) +def _format_rate(value: float | None) -> str: + if value is None: + return "n/a" + return f"{value:.2f} events/s" + + def _format_slope(value: float | None) -> str: if value is None: return "n/a" diff --git a/tests/performance/profile.py b/tests/performance/profile.py index 520beb44..39a5482d 100644 --- a/tests/performance/profile.py +++ b/tests/performance/profile.py @@ -72,7 +72,7 @@ def _int(raw: dict[str, Any], key: str, *, positive: bool = False) -> int: return value -def _bursts(raw: Any, duration: int) -> tuple[Burst, ...]: +def _bursts(raw: Any) -> tuple[Burst, ...]: if not isinstance(raw, list): raise ValueError("bursts must be a list") result: list[Burst] = [] @@ -84,14 +84,8 @@ def _bursts(raw: Any, duration: int) -> tuple[Burst, ...]: duration_seconds=_int(item, "duration_seconds", positive=True), events_per_second=_int(item, "events_per_second", positive=True), ) - if burst.start_second < 0 or burst.start_second + burst.duration_seconds > duration: - raise ValueError(f"burst {index} falls outside duration_seconds") result.append(burst) - ordered = sorted(result, key=lambda burst: burst.start_second) - for previous, current in pairwise(ordered): - if previous.start_second + previous.duration_seconds > current.start_second: - raise ValueError("bursts must not overlap") - return tuple(ordered) + return tuple(sorted(result, key=lambda burst: burst.start_second)) def _failures(raw: Any) -> tuple[FailureInjection, ...]: @@ -138,13 +132,46 @@ def load_profile(path: Path) -> WorkloadProfile: namespace_count=namespace_count, steady_events_per_second=steady, duration_seconds=duration, - bursts=_bursts(raw.get("bursts"), duration), + bursts=_bursts(raw.get("bursts")), failures=_failures(raw.get("failures")), ) + validate_profile(profile) + return profile + + +def validate_profile(profile: WorkloadProfile) -> None: + """Re-check every duration-dependent invariant of an assembled profile. + + `load_profile` calls this on load, and any caller that rewrites a loaded + profile - notably the CLI's `--duration` override, which uses + `dataclasses.replace` and therefore skips the loader entirely - must call + it again. Without it a shortened duration can leave a burst hanging past + the end of the run or a failure injection past the last planned event, + which only surfaces much later as an opaque generator assertion. + + Raises: + ValueError: a burst falls outside `duration_seconds`, two bursts + overlap, or a failure injection is scheduled past the last + planned event. + """ + for burst in profile.bursts: + if burst.start_second < 0 or burst.start_second + burst.duration_seconds > ( + profile.duration_seconds + ): + raise ValueError( + f"burst at second {burst.start_second} lasting {burst.duration_seconds}s " + f"falls outside duration_seconds={profile.duration_seconds}" + ) + ordered = sorted(profile.bursts, key=lambda burst: burst.start_second) + for previous, current in pairwise(ordered): + if previous.start_second + previous.duration_seconds > current.start_second: + raise ValueError("bursts must not overlap") total = planned_event_count(profile) if any(failure.at_event > total for failure in profile.failures): - raise ValueError("failure at_event exceeds planned event count") - return profile + raise ValueError( + f"failure at_event exceeds planned event count ({total}) for " + f"duration_seconds={profile.duration_seconds}" + ) def planned_event_count(profile: WorkloadProfile) -> int: diff --git a/tests/performance/profiles/aks-live-1k.json b/tests/performance/profiles/aks-live-1k.json new file mode 100644 index 00000000..13bc9cea --- /dev/null +++ b/tests/performance/profiles/aks-live-1k.json @@ -0,0 +1,15 @@ +{ + "schema_version": 1, + "id": "aks-live-1k", + "seed": 186, + "object_count": 1000, + "namespace_count": 20, + "steady_events_per_second": 20, + "duration_seconds": 1800, + "bursts": [ + {"start_second": 300, "duration_seconds": 30, "events_per_second": 100}, + {"start_second": 900, "duration_seconds": 30, "events_per_second": 100}, + {"start_second": 1500, "duration_seconds": 30, "events_per_second": 100} + ], + "failures": [] +} diff --git a/tests/performance/replay.py b/tests/performance/replay.py index 27e43b73..34df56ce 100644 --- a/tests/performance/replay.py +++ b/tests/performance/replay.py @@ -27,11 +27,13 @@ from korvid.k8s.errors import ApiStatusError from korvid.k8s.models import PodSummary from korvid.k8s.telemetry import ReadTelemetryEvent -from korvid.ui.app import KorvidApp, PaneState +from korvid.ui.app import KorvidApp +from korvid.ui.messages import ResourcesUpdated from korvid.ui.widgets.resource_table import ResourceTable from tests.performance.metrics import ( ApiSummary, BenchmarkRecorder, + ChurnSummary, LatencySummary, ProcessSampler, ProcessSummary, @@ -45,7 +47,7 @@ scheduled_events, summary_digest, ) -from tests.ui.waits import until +from tests.ui.waits import WaitTimeout, until async def _sleep_default(delay: float) -> None: @@ -53,6 +55,18 @@ async def _sleep_default(delay: float) -> None: await asyncio.sleep(delay) +class ReplayAborted(Exception): + """A replay ended before its schedule completed and can never complete. + + Raised for failure kinds the production `WatchManager` deliberately does + not retry (403 Forbidden is an authorization boundary, not a transient + fault): the stream is gone, the store has been cleared, and no further + event or render can arrive. Surfacing that immediately is strictly better + than letting the caller wait out a wall-clock timeout on a backlog that + will never drain. + """ + + @dataclass(frozen=True) class ReplayOptions: """Tuning knobs for a replay run. @@ -96,21 +110,34 @@ class ReplayReport: coalesced_updates: int event_to_render: LatencySummary input_latency: LatencySummary + #: Whether at least one churn event had actually been emitted (replay) or + #: dispatched (live) when input latency was first measured. churn_started_before_input: bool process: ProcessSummary api: ApiSummary manifest: RunManifest + #: Requested-versus-achieved churn accounting; `None` for deterministic + #: replay, which drives its own source rather than a real API server. + churn: ChurnSummary | None = None class MeasuredKorvidApp(KorvidApp): - """KorvidApp subclass that hooks `_render_table` to record render timing.""" + """KorvidApp subclass that records the timing of *resource-update* renders. + + The hook is deliberately `on_resources_updated` rather than + `_render_table`: the latter is also the choke point for cursor, filter, + sort, namespace-switch, and split-pane repaints (~10 call sites in + `ui/app.py`). Counting those would inflate `render_passes` and - worse - + let an unrelated repaint flush the pending-event backlog, attributing a + watch event's latency to a keypress that happened to repaint first. + """ def __init__(self, *args: Any, recorder: BenchmarkRecorder, **kwargs: Any) -> None: super().__init__(*args, **kwargs) self._benchmark_recorder = recorder - def _render_table(self, kind: str, *, only: PaneState | None = None) -> None: - super()._render_table(kind, only=only) + def on_resources_updated(self, message: ResourcesUpdated) -> None: + super().on_resources_updated(message) self._benchmark_recorder.record_render(monotonic()) @@ -121,6 +148,11 @@ def _render_table(self, kind: str, *, only: PaneState | None = None) -> None: "forbidden": 403, } +#: Statuses `WatchManager._watch_loop` refuses to retry: the stream ends for +#: good and the store is cleared, so the replay is over the moment one is +#: injected (see `ReplayAborted`). +_TERMINAL_FAILURE_STATUS: frozenset[int] = frozenset({403}) + class _ReplaySource: """Stateful async watch source that replays a `WorkloadProfile`. @@ -152,6 +184,13 @@ def __init__( self._failures = failures self._generation = 0 self._next_event_index = 0 + #: Number of scheduled churn events actually yielded to the watch + #: manager so far; the real signal behind + #: `ReplayReport.churn_started_before_input`. + self.emitted_events = 0 + #: Set to the injected failure whose status the watch manager will not + #: retry, so `run_replay` can abort with a named cause. + self.terminal_failure: FailureInjection | None = None self._replay_start: float = 0.0 self._current: dict[str, PodSummary] = { f"{p.namespace}/{p.name}": p for p in initial_pods(profile) @@ -164,10 +203,6 @@ def __init__( options.async_sleep if options.async_sleep is not None else _sleep_default ) - def current_digest(self) -> str: - """Digest of the source's tracked expected state.""" - return summary_digest(self._current.values()) - async def _handle_failure_if_any(self, event: ScheduledEvent, index: int) -> None: """Apply failure injection for *event*; raises `ApiStatusError` for hard faults. @@ -186,6 +221,12 @@ async def _handle_failure_if_any(self, event: ScheduledEvent, index: int) -> Non status = _HARD_FAILURE_STATUS[failure.kind] self._recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=status)) self._next_event_index = index + 1 + if status in _TERMINAL_FAILURE_STATUS: + # The watch manager will not reconnect after this: record the cause + # now so `run_replay` aborts with it instead of waiting out a + # timeout on a backlog that can never drain. + self.terminal_failure = failure + self._churn_done.set() raise ApiStatusError(status, failure.kind) async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summary]]: @@ -239,6 +280,7 @@ async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summ self._recorder.record_event(event.sequence, monotonic()) yield (event.event_type, event.summary) + self.emitted_events += 1 self._next_event_index = i + 1 self._churn_done.set() @@ -247,7 +289,12 @@ async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summ await asyncio.sleep(3600.0) -def _build_manifest(profile: WorkloadProfile) -> RunManifest: +def build_manifest(profile: WorkloadProfile) -> RunManifest: + """Resolved run manifest for *profile* (profile hash plus environment). + + Public because both replay harnesses (`replay.py` and `live.py`) build the + identical manifest for their reports. + """ profile_hash = hashlib.sha256( json.dumps(asdict(profile), sort_keys=True, separators=(",", ":"), default=str).encode() ).hexdigest() @@ -263,6 +310,37 @@ def _build_manifest(profile: WorkloadProfile) -> RunManifest: ) +async def wait_for( + pilot: Any, + condition: Callable[[], object], + *, + timeout: float, + label: str, + recorder: BenchmarkRecorder, +) -> None: + """`until`, but a timeout names the API errors that likely caused it. + + `KorvidApp.on_mount` replaces `WatchManager.on_error` with its own TUI + notification, so a 403/410/429 that killed the watch is otherwise invisible + to the harness: the operator would see only "not met within 60.0s" after + paying for a full cluster setup (or a full replay). The read telemetry has + already recorded those statuses, so they are appended to the message. + + Shared by both harnesses so a deterministic replay and a live run explain a + stalled wait the same way. + """ + try: + await until(pilot, condition, timeout=timeout, label=label) + except WaitTimeout as exc: + errors = recorder.api_errors() + if not errors: + raise + detail = ", ".join( + f"{event.operation} {event.path} status={event.status}" for event in errors + ) + raise WaitTimeout(f"{exc}; application read path reported API errors: {detail}") from exc + + async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> ReplayReport: """Run the full production Textual app against *profile* and return metrics. @@ -299,7 +377,7 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay failures, ) watch_manager = WatchManager(store, source, retry_delay=0.0) - manifest = _build_manifest(profile) + manifest = build_manifest(profile) app = MeasuredKorvidApp( config=KorvidConfig(namespace=ALL_NAMESPACES), @@ -315,18 +393,31 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay table = app.query_one(ResourceTable) # Wait for the initial LIST to populate the table. - await until( + await wait_for( pilot, lambda: table.row_count == profile.object_count, timeout=30.0, label="initial pods rendered", + recorder=recorder, ) # Release the source to emit scheduled events, then drive cursor # input while churn is active (not before the source is unblocked). churn_start.set() - churn_started_before_input = churn_start.is_set() + # Real ordering signal: wait for the source to actually put at + # least one churn event on the wire before measuring input latency, + # so the reported flag reflects observed emission rather than the + # fact that `churn_start.set()` was called on the previous line. + if events: + await wait_for( + pilot, + lambda: source.emitted_events > 0 or source.terminal_failure is not None, + timeout=30.0, + label="first churn event emitted", + recorder=recorder, + ) + churn_started_before_input = source.emitted_events > 0 t0 = monotonic() await pilot.press("down") recorder.record_input(monotonic() - t0) @@ -335,16 +426,29 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay recorder.record_input(monotonic() - t0) # Wait for all events to be emitted and all renders to complete. - await until( + await wait_for( pilot, - lambda: churn_done.is_set() and not recorder._pending_events, + lambda: ( + source.terminal_failure is not None + or (churn_done.is_set() and recorder.pending_count() == 0) + ), timeout=30.0, label="churn complete and all events rendered", + recorder=recorder, ) finally: process_samples = await sampler.stop() await watch_manager.stop_all() + if source.terminal_failure is not None: + failure = source.terminal_failure + status = _HARD_FAILURE_STATUS[failure.kind] + raise ReplayAborted( + f"replay aborted: injected {failure.kind!r} failure at event " + f"{failure.at_event} returned HTTP {status}, which the watch " + f"manager never retries; the stream ended and the store was cleared" + ) + # Compute expected digest using the independent apply_events oracle. # Hard failures (gone, throttled, forbidden) raise before yielding the # event, so the corresponding watch events are never applied; filter them diff --git a/tests/performance/test_cli.py b/tests/performance/test_cli.py index 573a708c..35456b3a 100644 --- a/tests/performance/test_cli.py +++ b/tests/performance/test_cli.py @@ -721,3 +721,104 @@ def test_replay_and_seed_manifests_commands_still_work( ) == 0 ) + + +def test_cli_replay_live_rejects_duration_that_orphans_a_burst( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """`--duration` rewrites the profile with `dataclasses.replace`, skipping + `load_profile`'s burst containment check. A shortened duration that leaves a + burst hanging past the end of the run must be rejected with an explicit + operational message *before* any cluster identity/ownership work, instead of + tripping the generator's internal assertion mid-run (which printed an empty + "error during replay: ").""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + live_profile = Path("tests/performance/profiles/aks-1k.json") + + exit_code = cli.main( + [ + "replay-live", + "--profile", + str(live_profile), + *_LIVE_IDENTITY_ARGS, + "--duration", + "10", + ] + ) + + assert exit_code == 1 + assert "falls outside duration_seconds" in capsys.readouterr().err + assert calls == [] + + +def test_cli_replay_live_accepts_duration_that_still_contains_every_burst( + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + + exit_code = cli.main( + [ + "replay-live", + "--profile", + "tests/performance/profiles/aks-1k.json", + *_LIVE_IDENTITY_ARGS, + "--duration", + "26", + ] + ) + + assert exit_code == 0 + used_profile = calls[0]["profile"] + assert isinstance(used_profile, WorkloadProfile) + assert used_profile.duration_seconds == 26 + + +@pytest.mark.parametrize("output_option", ["--out", "--json"]) +def test_cli_reports_output_write_errors_instead_of_raising( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + output_option: str, +) -> None: + """A bad `--out`/`--json` destination is an operational error, exactly like + `seed-manifests`' `--output`; it must return 1 with a message rather than + dumping a traceback after a completed (possibly very expensive) run.""" + + def fail_write(self: Path, _text: str, *_args: object, **_kwargs: object) -> int: + raise OSError("disk full") + + written_profile = profile_path(tmp_path) + monkeypatch.setattr(cli, "run_replay", fake_run_replay) + monkeypatch.setattr(Path, "write_text", fail_write) + + exit_code = cli.main( + [ + "replay", + "--profile", + str(written_profile), + "--time-scale", + "0", + output_option, + str(tmp_path / "report.out"), + ] + ) + + assert exit_code == 1 + assert "error writing report: disk full" in capsys.readouterr().err + + +def test_cli_replay_live_help_points_at_the_live_qualification_profile( + capsys: pytest.CaptureFixture[str], +) -> None: + """The published live plan (30 minutes at 20 events/s with three 30-second + 100 events/s bursts) lives in `aks-live-1k.json`; the command an operator + reaches for must name it, so the deterministic comparison profile is not + used by accident for a qualification run.""" + with pytest.raises(SystemExit): + cli.main(["replay-live", "--help"]) + + help_text = capsys.readouterr().out + assert "tests/performance/profiles/aks-live-1k.json" in help_text + assert Path("tests/performance/profiles/aks-live-1k.json").exists() diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py index 90809c5c..4f93fee4 100644 --- a/tests/performance/test_live.py +++ b/tests/performance/test_live.py @@ -24,19 +24,23 @@ from korvid.k8s.telemetry import ReadTelemetry, ReadTelemetryEvent from tests.performance import live, manifests from tests.performance.live import ( + ChurnProgress, CommandResult, LiveDependencies, - build_guarded_status_patch, + LiveLimits, + build_guarded_label_patch, drive_live_churn, live_object_identity, make_live_watch_source, + read_and_validate_owned_pods, run_live_replay, ) from tests.performance.manifests import build_seed_manifests from tests.performance.metrics import BenchmarkRecorder -from tests.performance.profile import WorkloadProfile +from tests.performance.profile import Burst, WorkloadProfile from tests.performance.replay import ReplayOptions from tests.performance.workload import scheduled_events, summary_digest +from tests.ui.waits import WaitTimeout RUN_ID = "aks186" CONTEXT = "aks-korvid-perf" @@ -124,6 +128,15 @@ def __init__( self.list_pods_calls: list[str] = [] self.list_objects_calls = 0 self.watch_pods_calls = 0 + #: True once the watch generator has been closed (the harness stopped + #: the WatchManager); lets tests assert *when* a read happened + #: relative to the application watch still being live. + self.watch_finished = False + #: Optional per-call spy invoked with the namespace being listed. + self.on_list_pods: Callable[[str], None] | None = None + #: Optional watch-open failure (e.g. a 403 the real client would + #: record as read telemetry before raising). + self.watch_error: ApiStatusError | None = None async def connect(self, context: str | None = None) -> None: self.connect_context = context @@ -141,6 +154,8 @@ async def list_objects(self, meta: ResourceMeta, namespace: str | None) -> list[ async def list_pods(self, namespace: str) -> list[PodSummary]: self.list_pods_calls.append(namespace) + if self.on_list_pods is not None: + self.on_list_pods(namespace) if self.read_telemetry is not None: self.read_telemetry(ReadTelemetryEvent("list", f"/api/v1/namespaces/{namespace}/pods")) return [pod for (ns, _name), pod in self.pods.items() if ns == namespace] @@ -148,38 +163,71 @@ async def list_pods(self, namespace: str) -> list[PodSummary]: async def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSummary]]: assert namespace is None self.watch_pods_calls += 1 - if self.read_telemetry is not None: - self.read_telemetry(ReadTelemetryEvent("list", "/api/v1/pods")) - for pod in self.pods.values(): - yield ("ADDED", pod) - for pod in self.distractor_pods: - yield ("ADDED", pod) - if self.read_telemetry is not None: - self.read_telemetry(ReadTelemetryEvent("watch_open", "/api/v1/pods")) - while True: - event = await self.events.get() + try: + if self.watch_error is not None: + if self.read_telemetry is not None: + self.read_telemetry( + ReadTelemetryEvent("error", "/api/v1/pods", status=self.watch_error.status) + ) + raise self.watch_error if self.read_telemetry is not None: - self.read_telemetry(ReadTelemetryEvent("watch_event", "/api/v1/pods")) - yield event + self.read_telemetry(ReadTelemetryEvent("list", "/api/v1/pods")) + for pod in list(self.pods.values()): + yield ("ADDED", pod) + for pod in self.distractor_pods: + yield ("ADDED", pod) + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("watch_open", "/api/v1/pods")) + while True: + event = await self.events.get() + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("watch_event", "/api/v1/pods")) + yield event + finally: + self.watch_finished = True class _FakeMutationClient: """Fake `MutationClient`: applies the guard checks a real JSON-Patch - `test` op would enforce, then mutates the shared fake cluster and wakes - the fake watch - so a guard failure here is exactly as fatal as a real - 412/422 from the API server.""" + `test` op would enforce, then writes the dedicated tick label on the + shared fake cluster and wakes the fake watch - so a guard failure here is + exactly as fatal as a real 422 from the API server.""" - def __init__(self, kube: _FakeKubeClient, run_id: str) -> None: - self._kube = kube + def __init__( + self, + kube: _FakeKubeClient | Callable[[], _FakeKubeClient], + run_id: str, + ) -> None: + # `run_live_replay` constructs (and eagerly connects) the mutation + # client *before* the application-path watch client, so tests that + # need the watch client resolve it lazily via a callable. + self._resolve_kube: Callable[[], _FakeKubeClient] = ( + kube if callable(kube) else (lambda: kube) + ) self._run_id = run_id self.calls: list[tuple[str, str, str]] = [] self.closed = False + self.connect_calls = 0 + self.in_flight = 0 + self.max_in_flight = 0 - async def patch_pod_status_guarded( - self, namespace: str, name: str, *, uid: str, phase: str + async def connect(self) -> None: + self.connect_calls += 1 + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str ) -> None: - self.calls.append((namespace, name, phase)) - current = self._kube.pods.get((namespace, name)) + self.calls.append((namespace, name, tick)) + self.in_flight += 1 + self.max_in_flight = max(self.max_in_flight, self.in_flight) + try: + await self._apply(namespace, name, uid=uid, tick=tick) + finally: + self.in_flight -= 1 + + async def _apply(self, namespace: str, name: str, *, uid: str, tick: str) -> None: + kube = self._resolve_kube() + current = kube.pods.get((namespace, name)) labels = dict(current.labels) if current is not None else {} guard_ok = ( current is not None @@ -189,9 +237,10 @@ async def patch_pod_status_guarded( ) if not guard_ok or current is None: raise ApiStatusError(422, "test operation failed for guarded patch") - updated = dataclasses.replace(current, phase=phase) - self._kube.pods[(namespace, name)] = updated - self._kube.events.put_nowait(("MODIFIED", updated)) + labels[manifests.TICK_LABEL] = tick + updated = dataclasses.replace(current, labels=tuple(sorted(labels.items()))) + kube.pods[(namespace, name)] = updated + kube.events.put_nowait(("MODIFIED", updated)) async def close(self) -> None: self.closed = True @@ -244,7 +293,7 @@ async def context_host(context: str) -> str: app_holder: list[_FakeKubeClient] = [] def default_mutation_factory(run_id_arg: str) -> _FakeMutationClient: - client = _FakeMutationClient(app_holder[-1], run_id_arg) + client = _FakeMutationClient(lambda: app_holder[-1], run_id_arg) if mutation_clients is not None: mutation_clients.append(client) return client @@ -340,8 +389,8 @@ def test_live_object_identity_examples() -> None: # --------------------------------------------------------------------------- -def test_build_guarded_status_patch_tests_uid_and_both_ownership_labels() -> None: - ops = build_guarded_status_patch(uid="uid-1", run_id="run1", phase="Pending") +def test_build_guarded_label_patch_tests_uid_and_both_ownership_labels() -> None: + ops = build_guarded_label_patch(uid="uid-1", run_id="run1", tick="7") assert ops == [ {"op": "test", "path": "/metadata/uid", "value": "uid-1"}, { @@ -354,10 +403,30 @@ def test_build_guarded_status_patch_tests_uid_and_both_ownership_labels() -> Non "path": "/metadata/labels/korvid.dev~1performance-run", "value": "run1", }, - {"op": "replace", "path": "/status/phase", "value": "Pending"}, + {"op": "add", "path": "/metadata/labels/korvid.dev~1performance-tick", "value": "7"}, ] +def test_build_guarded_label_patch_never_writes_status_spec_or_ownership() -> None: + """Churn must stay metadata-only on a *non-ownership* label: `status` is + kubelet-owned (a patched `status.phase` is reverted on the next node sync, + breaking digest parity) and the design doc requires metadata-only updates. + The two ownership labels are only ever `test` operands, never written.""" + ops = build_guarded_label_patch(uid="uid-1", run_id="run1", tick="7") + writes = [op for op in ops if op["op"] != "test"] + + assert len(writes) == 1 + assert writes[0]["path"] == "/metadata/labels/korvid.dev~1performance-tick" + assert not any(op["path"].startswith("/status") for op in ops) + assert not any(op["path"].startswith("/spec") for op in ops) + ownership_paths = { + "/metadata/labels/app.kubernetes.io~1managed-by", + "/metadata/labels/korvid.dev~1performance-run", + } + assert all(op["op"] == "test" for op in ops if op["path"] in ownership_paths) + assert manifests.TICK_LABEL not in {manifests.MANAGED_BY_LABEL, manifests.RUN_LABEL} + + # --------------------------------------------------------------------------- # Guarded churn # --------------------------------------------------------------------------- @@ -385,6 +454,7 @@ async def test_drive_live_churn_sends_guarded_patches_for_every_event() -> None: options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) recorder = BenchmarkRecorder() + progress = ChurnProgress(requested_events=len(events)) live_state = dict(pods) await drive_live_churn( events, @@ -392,17 +462,21 @@ async def test_drive_live_churn_sends_guarded_patches_for_every_event() -> None: namespace_count=namespace_count, live_state=live_state, mutation_client=mutation_client, - recorder=recorder, options=options, + progress=progress, + limits=LiveLimits(churn_concurrency=1), ) assert len(mutation_client.calls) == len(events) for call, event in zip(mutation_client.calls, events, strict=True): - namespace, name, phase = call + namespace, name, tick = call expected_namespace, expected_name = live_object_identity( - run_id, namespace_count, int(event.summary.name.removeprefix("pod-")) + run_id, namespace_count, event.object_index ) assert (namespace, name) == (expected_namespace, expected_name) - assert phase == event.summary.phase + assert tick == str(event.sequence) + assert progress.completed == len(events) + # Event timing is recorded at watch receipt, never by the write path. + assert recorder.pending_count() == 0 async def test_drive_live_churn_aborts_on_guard_failure_and_never_continues() -> None: @@ -428,7 +502,6 @@ async def test_drive_live_churn_aborts_on_guard_failure_and_never_continues() -> monotonic_fn, async_sleep = _virtual_clock() options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) - recorder = BenchmarkRecorder() # Snapshot the driver's cached state *before* the target pod is replaced # (a fresh uid) - exactly like `run_live_replay` caching uids once during # the ownership gate, then churning without re-reading them each time. @@ -437,7 +510,7 @@ async def test_drive_live_churn_aborts_on_guard_failure_and_never_continues() -> # Simulate the target pod being replaced right before the second # scheduled mutation - a real API server would fail the JSON-Patch # `test` op the same way. - second_index = int(events[1].summary.name.removeprefix("pod-")) + second_index = events[1].object_index second_key = live_object_identity(run_id, namespace_count, second_index) kube.pods[second_key] = dataclasses.replace(kube.pods[second_key], uid="replaced-uid") @@ -448,8 +521,9 @@ async def test_drive_live_churn_aborts_on_guard_failure_and_never_continues() -> namespace_count=namespace_count, live_state=live_state, mutation_client=mutation_client, - recorder=recorder, options=options, + progress=ChurnProgress(requested_events=len(events)), + limits=LiveLimits(churn_concurrency=1), ) # Aborted at the 2nd call; a 3rd event must never have been attempted. assert len(mutation_client.calls) == 2 @@ -473,7 +547,9 @@ async def test_make_live_watch_source_filters_to_expected_namespaces() -> None: ) kube = _FakeKubeClient({}, pods, distractor_pods=(distractor,)) expected_namespaces = frozenset(namespace for namespace, _name in pods) - source = make_live_watch_source(kube, expected_namespaces) + source = make_live_watch_source( + kube, expected_namespaces, run_id=run_id, recorder=BenchmarkRecorder() + ) seen: list[tuple[str, Summary]] = [] agen = source("pods", "*") @@ -493,7 +569,7 @@ async def test_make_live_watch_source_filters_to_expected_namespaces() -> None: async def test_make_live_watch_source_rejects_non_pod_kind() -> None: kube = _FakeKubeClient({}, {}) - source = make_live_watch_source(kube, frozenset()) + source = make_live_watch_source(kube, frozenset(), run_id="run1", recorder=BenchmarkRecorder()) with pytest.raises(ValueError, match="only watches pods"): await source("deployments", "*").__anext__() @@ -885,6 +961,9 @@ async def test_run_live_replay_full_happy_path_matches_cluster_digest() -> None: # of `report.api` - its single "list" comes from `watch_pods`'s own # internal LIST-then-WATCH, not from any harness read. assert app_client.watch_pods_calls == 1 + # The watch really is stopped by teardown, so `watch_finished` is a + # meaningful signal for the ground-truth-read ordering assertion. + assert app_client.watch_finished assert report.api.operations["watch_open"] == 1 assert report.api.operations.get("list", 0) == 1 @@ -912,9 +991,13 @@ async def test_run_live_replay_aborts_and_still_closes_clients_on_guard_failure( class _AlwaysFailingMutationClient: def __init__(self) -> None: self.closed = False + self.connect_calls = 0 + + async def connect(self) -> None: + self.connect_calls += 1 - async def patch_pod_status_guarded( - self, namespace: str, name: str, *, uid: str, phase: str + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str ) -> None: raise ApiStatusError(422, "test operation failed for guarded patch") @@ -979,11 +1062,15 @@ async def test_run_live_replay_propagates_cancelled_error_and_still_closes_clien ) async def cancelling_sleep(_delay: float) -> None: + # Yield first so the already-dispatched first mutation task runs to + # completion, then cancel exactly like a real `Task.cancel()` landing + # in the scheduler's sleep. + await asyncio.sleep(0) raise asyncio.CancelledError # duration_seconds=1, steady_events_per_second=2 schedules events at - # offsets 0.0 and 0.5: the first mutates without sleeping (delay <= 0), - # the second's positive delay drives the cancelling sleep. + # offsets 0.0 and 0.5: the first is dispatched without sleeping + # (delay <= 0), the second's positive delay drives the cancelling sleep. options = ReplayOptions(time_scale=1.0, monotonic_fn=lambda: 0.0, async_sleep=cancelling_sleep) with pytest.raises(asyncio.CancelledError): @@ -1003,3 +1090,827 @@ async def cancelling_sleep(_delay: float) -> None: # Cancellation must abort before the second churn mutation - no broad # cleanup, no continuing past the point of cancellation. assert len(created_clients[0].calls) == 1 + + +# --------------------------------------------------------------------------- +# C1: event timing is recorded at watch receipt, never at patch ack +# --------------------------------------------------------------------------- + + +async def test_make_live_watch_source_records_owned_modified_events_at_receipt() -> None: + """Latency measurement must start where the *application* first sees the + event, exactly like the deterministic `_ReplaySource`. Initial `ADDED` + rows from the LIST phase and events from foreign namespaces are not churn + events and must not be recorded.""" + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + kube = _FakeKubeClient({}, pods) + recorder = BenchmarkRecorder() + expected_namespaces = frozenset(namespace for namespace, _name in pods) + source = make_live_watch_source(kube, expected_namespaces, run_id=run_id, recorder=recorder) + + agen = source("pods", "*") + try: + for _ in range(len(pods)): + await agen.__anext__() + # Four ADDED rows from the LIST phase are not churn events. + assert recorder.pending_count() == 0 + + owned = next(iter(pods.values())) + kube.events.put_nowait(("MODIFIED", owned)) + await agen.__anext__() + assert recorder.pending_count() == 1 + + foreign = PodSummary( + name="stray", + namespace="default", + phase="Running", + ready="1/1", + restarts=0, + node="node-x", + ) + unowned = dataclasses.replace(owned, labels=()) + kube.events.put_nowait(("MODIFIED", foreign)) + kube.events.put_nowait(("MODIFIED", unowned)) + await agen.__anext__() + assert recorder.pending_count() == 1 + finally: + assert isinstance(agen, AsyncGenerator) + await agen.aclose() + + +async def test_run_live_replay_records_at_receipt_when_the_patch_ack_lags() -> None: + """Reproduces the ack-race the review found: a real API server delivers the + watch event over a different connection than the patch response, so the + event can be rendered *before* the patch call returns. Recording at ack + then appended a pending entry after its own render, which the final wait + could never drain (a 60 s timeout) and which the CLI reported as a dropped + update.""" + + class _AckLagsMutationClient(_FakeMutationClient): + async def _apply(self, namespace: str, name: str, *, uid: str, tick: str) -> None: + await super()._apply(namespace, name, uid=uid, tick=tick) + # Let the watch deliver and render before the caller resumes. + for _ in range(20): + await asyncio.sleep(0.005) + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, app_clients=app_clients) + deps = dataclasses.replace( + deps, + mutation_client_factory=lambda run_id: _AckLagsMutationClient( + lambda: app_clients[-1], run_id + ), + ) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.dropped_updates == 0 + assert report.event_to_render.count == 2 + assert report.expected_digest == report.final_digest + + +# --------------------------------------------------------------------------- +# C2: metadata-only, ownership-preserving churn and digest convergence +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_churns_the_tick_label_and_preserves_everything_else() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + mutation_clients: list[_FakeMutationClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, mutation_clients=mutation_clients) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + mutated = {(namespace, name) for namespace, name, _tick in mutation_clients[0].calls} + assert mutated + for key, pod in pods.items(): + labels = dict(pod.labels) + assert labels[manifests.MANAGED_BY_LABEL] == manifests.MANAGED_BY_VALUE + assert labels[manifests.RUN_LABEL] == RUN_ID + # Kubelet-owned status is never touched by churn. + assert pod.phase == "Running" + assert (manifests.TICK_LABEL in labels) is (key in mutated) + assert report.expected_digest == report.final_digest + + +async def test_run_live_replay_reads_ground_truth_while_the_watch_is_live() -> None: + """The ground-truth read and the convergence wait must both happen inside + the measured window: reading after `watch_manager.stop_all()` compares a + frozen store against a cluster that was still changing.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + harness_clients: list[_FakeKubeClient] = [] + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps( + namespaces, pods, RUN_ID, harness_clients=harness_clients, app_clients=app_clients + ) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + watch_live_during_reads: list[bool] = [] + + def spy(_namespace: str) -> None: + if len(harness_clients[0].list_pods_calls) > 20: # the post-churn re-read + watch_live_during_reads.append(not app_clients[0].watch_finished) + + async def run() -> None: + await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + original_factory = deps.harness_kube_client_factory + + def harness_factory() -> _FakeKubeClient: + client = original_factory() + assert isinstance(client, _FakeKubeClient) + client.on_list_pods = spy + return client + + deps = dataclasses.replace(deps, harness_kube_client_factory=harness_factory) + await run() + + assert len(watch_live_during_reads) == 20 + assert all(watch_live_during_reads) + + +async def test_run_live_replay_waits_for_the_store_digest_to_converge() -> None: + """Watch propagation lags the patch acknowledgement. Without an explicit + convergence wait the store digest is read while the last events are still + in flight, producing a false digest mismatch (CLI exit 1).""" + + class _LaggingWatchMutationClient(_FakeMutationClient): + async def _apply(self, namespace: str, name: str, *, uid: str, tick: str) -> None: + kube = self._resolve_kube() + before = kube.events.qsize() + await super()._apply(namespace, name, uid=uid, tick=tick) + # Pull the just-queued event back out and re-deliver it later, so + # the cluster is already mutated while the watch has not caught up. + assert kube.events.qsize() == before + 1 + delayed = kube.events.get_nowait() + asyncio.get_running_loop().call_later(1.0, kube.events.put_nowait, delayed) + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, app_clients=app_clients) + deps = dataclasses.replace( + deps, + mutation_client_factory=lambda run_id: _LaggingWatchMutationClient( + lambda: app_clients[-1], run_id + ), + ) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.expected_digest == report.final_digest + assert report.expected_digest == summary_digest(pods.values()) + assert report.dropped_updates == 0 + + +def test_check_row_count_rejects_a_regressed_render() -> None: + """A late watch reconnect clears and re-seeds the store, so a digest that + matched a moment ago can be recomputed over a partial store; the rendered + row count is re-asserted at the same instant.""" + live._check_row_count(1000, 1000) + + with pytest.raises(ValueError, match="rendered row count regressed"): + live._check_row_count(999, 1000) + + +# --------------------------------------------------------------------------- +# C3: bounded concurrency, bounded operation time, guarded 429-only retry +# --------------------------------------------------------------------------- + + +def _churn_profile(*, events_per_second: int, duration_seconds: int = 1) -> WorkloadProfile: + return WorkloadProfile( + schema_version=1, + id="churn-test", + seed=7, + object_count=4, + namespace_count=2, + steady_events_per_second=events_per_second, + duration_seconds=duration_seconds, + bursts=(), + failures=(), + ) + + +async def test_drive_live_churn_bounds_in_flight_mutations() -> None: + """Serial patching cannot approach the scheduled rate (one round trip per + event), so churn dispatches concurrently - but never without a ceiling.""" + + class _YieldingMutationClient(_FakeMutationClient): + async def _apply(self, namespace: str, name: str, *, uid: str, tick: str) -> None: + for _ in range(3): + await asyncio.sleep(0) + await super()._apply(namespace, name, uid=uid, tick=tick) + + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + kube = _FakeKubeClient({}, pods) + mutation_client = _YieldingMutationClient(kube, run_id) + events = scheduled_events(_churn_profile(events_per_second=40)) + monotonic_fn, async_sleep = _virtual_clock() + progress = ChurnProgress(requested_events=len(events)) + + await drive_live_churn( + events, + run_id=run_id, + namespace_count=2, + live_state=dict(pods), + mutation_client=mutation_client, + options=ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep), + progress=progress, + limits=LiveLimits(churn_concurrency=4), + ) + + assert progress.completed == len(events) == 40 + assert mutation_client.max_in_flight > 1, "churn must not be effectively serial" + assert mutation_client.max_in_flight <= 4 + + +async def test_drive_live_churn_retries_only_429_with_the_identical_guarded_patch() -> None: + """A single API Priority and Fairness throttle must not kill a 30-minute + run, but the retry re-issues the *identical* guarded patch (same uid, same + ownership tests) - it is never a relaxed or unguarded retry.""" + + class _ThrottlingMutationClient(_FakeMutationClient): + def __init__(self, kube: _FakeKubeClient, run_id: str, throttles: int) -> None: + super().__init__(kube, run_id) + self.remaining_throttles = throttles + self.patch_arguments: list[tuple[str, str, str, str]] = [] + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.patch_arguments.append((namespace, name, uid, tick)) + if self.remaining_throttles > 0: + self.remaining_throttles -= 1 + raise ApiStatusError(429, "Too Many Requests") + await super().patch_pod_labels_guarded(namespace, name, uid=uid, tick=tick) + + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + kube = _FakeKubeClient({}, pods) + mutation_client = _ThrottlingMutationClient(kube, run_id, throttles=2) + events = scheduled_events(_churn_profile(events_per_second=1)) + monotonic_fn, async_sleep = _virtual_clock() + progress = ChurnProgress(requested_events=len(events)) + + await drive_live_churn( + events, + run_id=run_id, + namespace_count=2, + live_state=dict(pods), + mutation_client=mutation_client, + options=ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep), + progress=progress, + limits=LiveLimits(churn_concurrency=1, mutation_throttle_retries=5), + ) + + assert len(mutation_client.patch_arguments) == 3 + assert len(set(mutation_client.patch_arguments)) == 1 + assert progress.mutation_throttles == 2 + assert progress.completed == 1 + + +async def test_drive_live_churn_gives_up_after_the_bounded_throttle_retries() -> None: + class _AlwaysThrottling(_FakeMutationClient): + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.calls.append((namespace, name, tick)) + raise ApiStatusError(429, "Too Many Requests") + + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + mutation_client = _AlwaysThrottling(_FakeKubeClient({}, pods), run_id) + events = scheduled_events(_churn_profile(events_per_second=1)) + monotonic_fn, async_sleep = _virtual_clock() + progress = ChurnProgress(requested_events=len(events)) + + with pytest.raises(ApiStatusError, match="API 429"): + await drive_live_churn( + events, + run_id=run_id, + namespace_count=2, + live_state=dict(pods), + mutation_client=mutation_client, + options=ReplayOptions( + time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ), + progress=progress, + limits=LiveLimits(churn_concurrency=1, mutation_throttle_retries=2), + ) + + assert len(mutation_client.calls) == 3 + assert progress.mutation_throttles == 2 + + +async def test_drive_live_churn_never_retries_a_failed_ownership_guard() -> None: + """422 is a failed JSON-Patch `test` op: the target is not what the run + validated. Retrying it - guarded or not - is exactly the behaviour the + safety contract forbids.""" + + class _GuardFailingMutationClient(_FakeMutationClient): + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.calls.append((namespace, name, tick)) + raise ApiStatusError(422, "test operation failed for guarded patch") + + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + mutation_client = _GuardFailingMutationClient(_FakeKubeClient({}, pods), run_id) + events = scheduled_events(_churn_profile(events_per_second=1)) + monotonic_fn, async_sleep = _virtual_clock() + progress = ChurnProgress(requested_events=len(events)) + + with pytest.raises(ApiStatusError, match="test operation failed"): + await drive_live_churn( + events, + run_id=run_id, + namespace_count=2, + live_state=dict(pods), + mutation_client=mutation_client, + options=ReplayOptions( + time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ), + progress=progress, + limits=LiveLimits(churn_concurrency=1, mutation_throttle_retries=5), + ) + + assert len(mutation_client.calls) == 1 + assert progress.mutation_throttles == 0 + + +async def test_drive_live_churn_bounds_every_mutation_attempt() -> None: + """A stalled patch must not hang the run: every attempt is bounded.""" + + class _HangingMutationClient(_FakeMutationClient): + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.calls.append((namespace, name, tick)) + await asyncio.sleep(30) + + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + mutation_client = _HangingMutationClient(_FakeKubeClient({}, pods), run_id) + events = scheduled_events(_churn_profile(events_per_second=1)) + progress = ChurnProgress(requested_events=len(events)) + + with pytest.raises(TimeoutError): + await drive_live_churn( + events, + run_id=run_id, + namespace_count=2, + live_state=dict(pods), + mutation_client=mutation_client, + options=ReplayOptions(time_scale=1.0), + progress=progress, + limits=LiveLimits(churn_concurrency=1, mutation_timeout_seconds=0.05), + ) + + assert len(mutation_client.calls) == 1 + assert progress.completed == 0 + + +async def test_run_live_replay_reports_requested_and_achieved_churn_rate() -> None: + """The design doc forbids presenting a requested rate as an achieved one: + both must be reported, and they must be allowed to differ.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = _happy_deps(namespaces, pods, RUN_ID) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + profile = dataclasses.replace(_tiny_live_profile(), duration_seconds=2) + report = await run_live_replay( + profile, + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.churn is not None + # 4 events requested over 2 seconds (2 events/s requested); the driver + # observed all 4 within a 1.0 s window, i.e. a different achieved rate. + assert report.churn.requested_events == 4 + assert report.churn.requested_events_per_second == 2.0 + assert report.churn.observed_events == 4 + assert report.churn.wall_seconds == 1.0 + assert report.churn.achieved_events_per_second == 4.0 + assert report.churn.requested_events_per_second != report.churn.achieved_events_per_second + assert report.churn.mutation_throttles == 0 + + +async def test_run_live_replay_counts_mutation_throttles_outside_read_telemetry() -> None: + """Harness write traffic must never be reported as application read-path + API telemetry.""" + + class _ThrottleOnceMutationClient(_FakeMutationClient): + def __init__( + self, kube: Callable[[], _FakeKubeClient] | _FakeKubeClient, run_id: str + ) -> None: + super().__init__(kube, run_id) + self.throttled = False + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + if not self.throttled: + self.throttled = True + raise ApiStatusError(429, "Too Many Requests") + await super().patch_pod_labels_guarded(namespace, name, uid=uid, tick=tick) + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, app_clients=app_clients) + deps = dataclasses.replace( + deps, + mutation_client_factory=lambda run_id: _ThrottleOnceMutationClient( + lambda: app_clients[-1], run_id + ), + ) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.churn is not None + assert report.churn.mutation_throttles == 1 + assert report.api.throttles == 0 + assert report.api.operations.get("error", 0) == 0 + assert report.expected_digest == report.final_digest + + +# --------------------------------------------------------------------------- +# I2: cancellation cancels and drains every mutation task before teardown +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_stops_mutating_when_the_outer_task_is_cancelled() -> None: + """Real `Task.cancel()` on the whole run: the churn task (and every patch + inside its task group) must be cancelled and awaited *before* the clients + are closed, so no mutation outlives teardown.""" + + class _PacedMutationClient(_FakeMutationClient): + async def _apply(self, namespace: str, name: str, *, uid: str, tick: str) -> None: + await asyncio.sleep(0.01) + await super()._apply(namespace, name, uid=uid, tick=tick) + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + app_clients: list[_FakeKubeClient] = [] + harness_clients: list[_FakeKubeClient] = [] + created: list[_PacedMutationClient] = [] + + def mutation_factory(run_id: str) -> _PacedMutationClient: + client = _PacedMutationClient(lambda: app_clients[-1], run_id) + created.append(client) + return client + + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + app_clients=app_clients, + harness_clients=harness_clients, + mutation_client_factory=mutation_factory, + ) + # 300 events over 30 s of real wall time: the run is still churning when + # the cancellation lands. + profile = dataclasses.replace( + _tiny_live_profile(), steady_events_per_second=10, duration_seconds=30 + ) + + task = asyncio.create_task( + run_live_replay( + profile, + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + ) + for _ in range(2000): + if created and len(created[0].calls) >= 2: + break + await asyncio.sleep(0.005) + assert created, "the mutation client was never constructed" + assert len(created[0].calls) >= 2, "churn never started" + + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + calls_at_cancellation = len(created[0].calls) + for _ in range(20): + await asyncio.sleep(0.01) + assert len(created[0].calls) == calls_at_cancellation + assert created[0].closed + assert harness_clients[0].closed + assert app_clients[0].closed + + +# --------------------------------------------------------------------------- +# I3/I4: exact identity and ownership, before *and* after churn +# --------------------------------------------------------------------------- + + +async def test_verify_ownership_rejects_unexpected_pod_in_an_owned_namespace() -> None: + """The application watch filters by namespace, so a foreign Pod in an owned + namespace enters the benchmark store; the gate must name it instead of + letting the initial-render wait time out with a generic message.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + stray_namespace = manifests.namespace_name(RUN_ID, 3) + pods[(stray_namespace, "intruder")] = PodSummary( + name="intruder", + namespace=stray_namespace, + phase="Running", + ready="1/1", + restarts=0, + node="node-9", + labels=_labels(RUN_ID), + ) + kube = _FakeKubeClient(namespaces, pods) + + with pytest.raises( + ValueError, match=f"unexpected pods in owned namespaces: {stray_namespace}/intruder" + ): + await live._verify_ownership(kube, run_id=RUN_ID, namespace_count=20, object_count=1000) + + +async def test_read_and_validate_owned_pods_returns_the_validated_snapshot() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + kube = _FakeKubeClient(namespaces, pods) + + validated = await read_and_validate_owned_pods(kube, run_id=RUN_ID, expected=pods) + + assert sorted((pod.namespace, pod.name) for pod in validated) == sorted(pods) + + +@pytest.mark.parametrize( + ("corrupt", "message"), + [ + ("labels", "lost ownership labels"), + ("uid", "uid changed"), + ("missing", "missing"), + ("extra", "unexpected pod"), + ], +) +async def test_read_and_validate_owned_pods_rejects_identity_or_ownership_loss( + corrupt: str, message: str +) -> None: + """The ground-truth digest may only be computed from Pods that still are + what the ownership gate validated.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + expected = dict(pods) + key = live_object_identity(RUN_ID, 20, 42) + if corrupt == "labels": + pods[key] = dataclasses.replace(pods[key], labels=_labels("some-other-run")) + elif corrupt == "uid": + pods[key] = dataclasses.replace(pods[key], uid="recreated-uid") + elif corrupt == "missing": + del pods[key] + else: + namespace = key[0] + pods[(namespace, "intruder")] = dataclasses.replace( + pods[key], name="intruder", uid="intruder-uid" + ) + kube = _FakeKubeClient(namespaces, pods) + + with pytest.raises(ValueError, match=f"post-churn ownership revalidation failed.*{message}"): + await read_and_validate_owned_pods(kube, run_id=RUN_ID, expected=expected) + + +async def test_run_live_replay_rejects_a_pod_that_lost_ownership_during_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + harness_clients: list[_FakeKubeClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, harness_clients=harness_clients) + victim = live_object_identity(RUN_ID, 20, 7) + + def strip_ownership_before_the_final_read(_namespace: str) -> None: + if len(harness_clients[0].list_pods_calls) > 20: + pods[victim] = dataclasses.replace(pods[victim], labels=()) + + original_factory = deps.harness_kube_client_factory + + def harness_factory() -> _FakeKubeClient: + client = original_factory() + assert isinstance(client, _FakeKubeClient) + client.on_list_pods = strip_ownership_before_the_final_read + return client + + deps = dataclasses.replace(deps, harness_kube_client_factory=harness_factory) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + with pytest.raises(ValueError, match="post-churn ownership revalidation failed"): + await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +# --------------------------------------------------------------------------- +# I7 / M12 / M7 / profile revalidation +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_names_watch_api_errors_in_a_wait_timeout() -> None: + """`KorvidApp.on_mount` overwrites `WatchManager.on_error` with its own TUI + notification, so a 403 that killed the watch is otherwise invisible: the + operator would see only "not met within Ns" after paying for cluster setup.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, app_clients=app_clients) + original_factory = deps.kube_client_factory + + def failing_kube_factory(read_telemetry: ReadTelemetry) -> _FakeKubeClient: + client = original_factory(read_telemetry) + assert isinstance(client, _FakeKubeClient) + client.watch_error = ApiStatusError(403, "Forbidden") + return client + + deps = dataclasses.replace(deps, kube_client_factory=failing_kube_factory) + + with pytest.raises(WaitTimeout, match="status=403"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(initial_render_timeout_seconds=0.3), + ) + + +async def test_run_live_replay_connects_the_mutation_client_before_the_app_starts() -> None: + """`load_kube_config` can invoke an exec credential plugin; that latency is + paid once, up front, under an explicit bound - not inside the first + measured mutation.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + mutation_clients: list[_FakeMutationClient] = [] + app_clients: list[_FakeKubeClient] = [] + connect_order: list[str] = [] + + def mutation_factory(run_id: str) -> _FakeMutationClient: + client = _FakeMutationClient(lambda: app_clients[-1], run_id) + mutation_clients.append(client) + return client + + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + app_clients=app_clients, + mutation_client_factory=mutation_factory, + ) + original_kube_factory = deps.kube_client_factory + + def recording_kube_factory(read_telemetry: ReadTelemetry) -> _FakeKubeClient: + connect_order.append("app-client") + return original_kube_factory(read_telemetry) # type: ignore[return-value] + + deps = dataclasses.replace(deps, kube_client_factory=recording_kube_factory) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert mutation_clients[0].connect_calls == 1 + assert connect_order == ["app-client"] + + +async def test_run_live_replay_bounds_the_mutation_client_connect() -> None: + class _HangingConnectMutationClient(_FakeMutationClient): + async def connect(self) -> None: + await asyncio.sleep(30) + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + created: list[_HangingConnectMutationClient] = [] + + def mutation_factory(run_id: str) -> _HangingConnectMutationClient: + client = _HangingConnectMutationClient(_FakeKubeClient(namespaces, pods), run_id) + created.append(client) + return client + + deps = _happy_deps(namespaces, pods, RUN_ID, mutation_client_factory=mutation_factory) + + with pytest.raises(TimeoutError): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(mutation_connect_timeout_seconds=0.05), + ) + + assert created[0].closed + + +async def test_run_live_replay_measures_input_latency_with_the_injected_clock() -> None: + """Churn uses the injected clock; input latency must use the same one, or a + virtual-clock latency assertion measures nothing.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = _happy_deps(namespaces, pods, RUN_ID) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.input_latency.count == 2 + assert report.input_latency.maximum_seconds == 0.0 + + +async def test_run_live_replay_rejects_a_profile_whose_bursts_escape_its_duration() -> None: + """A profile rewritten with `dataclasses.replace` (the CLI's `--duration`) + never passes through `load_profile`; the invariants are re-checked here, + before any cluster identity or ownership work.""" + profile = dataclasses.replace( + _tiny_live_profile(), + duration_seconds=2, + bursts=(Burst(start_second=1, duration_seconds=30, events_per_second=100),), + ) + deps = LiveDependencies( + command_runner=_never_called("command_runner"), + active_context=_never_called("active_context"), + context_host=_never_called("context_host"), + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + with pytest.raises(ValueError, match="falls outside duration_seconds"): + await run_live_replay( + profile, + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) diff --git a/tests/performance/test_metrics.py b/tests/performance/test_metrics.py index 0cdbf3c0..05df4c32 100644 --- a/tests/performance/test_metrics.py +++ b/tests/performance/test_metrics.py @@ -10,6 +10,7 @@ from korvid.k8s.telemetry import ReadTelemetryEvent from tests.performance.metrics import ( BenchmarkRecorder, + ChurnSummary, LatencySummary, ProcessSample, ProcessSampler, @@ -302,6 +303,7 @@ def test_report_payload_is_json_serializable_and_stable() -> None: "coalesced_updates": 0, "dropped_updates": 0, }, + "churn": None, "digests": {"final": "digest-123"}, } assert '"rss_slope_mib_per_minute": 1.0' in encoded @@ -491,3 +493,84 @@ async def test_process_sampler_skips_warmup_sample( python_bytes=1000, ), ) + + +def test_recorder_exposes_public_pending_count_and_api_errors() -> None: + """Replay/live harnesses must observe the backlog and API errors through + a public accessor instead of reaching into `_pending_events`/`_api_events` + across module boundaries.""" + recorder = BenchmarkRecorder() + assert recorder.pending_count() == 0 + assert recorder.api_errors() == () + + recorder.record_event(1, 10.0) + recorder.record_event(2, 11.0) + assert recorder.pending_count() == 2 + + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=403)) + recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=410)) + + assert [event.status for event in recorder.api_errors()] == [403, 410] + + recorder.record_render(12.0) + assert recorder.pending_count() == 0 + + +def test_report_separates_requested_churn_rate_from_achieved_rate() -> None: + """The design doc forbids reporting a requested rate as an achieved rate. + Both must be present, distinctly labelled, alongside the observed event + count, wall time, and the mutation-side throttle count (which is *not* the + application read path's `api.throttles`).""" + recorder = BenchmarkRecorder() + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + churn = ChurnSummary.from_observations( + requested_events=8400, + requested_duration_seconds=30, + observed_events=900, + wall_seconds=30.0, + mutation_throttles=7, + ) + + report = recorder.report(run_manifest(), (), final_digest="abc", churn=churn) + payload = cast(dict[str, object], report_payload(report)["churn"]) + markdown = render_markdown(report) + + assert report.churn is churn + assert churn.requested_events_per_second == 280.0 + assert churn.achieved_events_per_second == 30.0 + assert payload == { + "requested_events": 8400, + "requested_events_per_second": 280.0, + "observed_events": 900, + "wall_seconds": 30.0, + "achieved_events_per_second": 30.0, + "mutation_throttles": 7, + } + assert "Requested churn rate: `280.00 events/s`" in markdown + assert "Achieved churn rate: `30.00 events/s`" in markdown + assert "Mutation throttles (429): `7`" in markdown + # The application read path's throttle counter stays independent. + assert report.api.throttles == 0 + + +def test_report_payload_keeps_a_stable_churn_key_when_no_churn_was_driven() -> None: + recorder = BenchmarkRecorder() + report = recorder.report(run_manifest(), (), final_digest="abc") + + assert report.churn is None + assert report_payload(report)["churn"] is None + assert "Achieved churn rate: `n/a`" in render_markdown(report) + + +def test_churn_summary_reports_no_achieved_rate_without_elapsed_time() -> None: + churn = ChurnSummary.from_observations( + requested_events=10, + requested_duration_seconds=0, + observed_events=0, + wall_seconds=None, + mutation_throttles=0, + ) + + assert churn.achieved_events_per_second is None + assert churn.requested_events_per_second is None diff --git a/tests/performance/test_profile.py b/tests/performance/test_profile.py index d0f1fa9c..470735f3 100644 --- a/tests/performance/test_profile.py +++ b/tests/performance/test_profile.py @@ -1,9 +1,15 @@ import json +from dataclasses import replace from pathlib import Path import pytest -from tests.performance.profile import Burst, load_profile, planned_event_count +from tests.performance.profile import ( + Burst, + load_profile, + planned_event_count, + validate_profile, +) def _write(tmp_path: Path, **overrides: object) -> Path: @@ -56,6 +62,11 @@ def test_profile_rejects_invalid_values( def test_aks_1k_profile_pins_live_topology_and_reuses_burst_schedule() -> None: + """`aks-1k` is the *deterministic comparison* profile: the live topology on + `burst-50k`'s event schedule, so a live run can be scored against the + synthetic 1k/10k/50k baselines. The qualification run itself uses + `aks-live-1k` (see `test_aks_live_1k_profile_matches_the_published_live_plan`). + """ profiles_dir = Path(__file__).with_name("profiles") profile = load_profile(profiles_dir / "aks-1k.json") @@ -78,3 +89,56 @@ def test_aks_1k_profile_pins_live_topology_and_reuses_burst_schedule() -> None: assert profile.duration_seconds == burst.duration_seconds assert profile.bursts == burst.bursts assert profile.failures == burst.failures + + +def test_validate_profile_rejects_burst_outside_shortened_duration(tmp_path: Path) -> None: + """`--duration` shortens a loaded profile with `dataclasses.replace`, which + bypasses `load_profile`'s burst containment check. `validate_profile` is the + shared re-check both paths call, so the shortened profile is rejected with a + clear operational message instead of failing later inside the generator.""" + profile = load_profile(_write(tmp_path)) + shortened = replace(profile, duration_seconds=2) + + with pytest.raises(ValueError, match="falls outside duration_seconds"): + validate_profile(shortened) + + +def test_validate_profile_rejects_failure_beyond_shortened_planned_events( + tmp_path: Path, +) -> None: + profile = load_profile(_write(tmp_path, bursts=[], failures=[{"kind": "gone", "at_event": 75}])) + shortened = replace(profile, duration_seconds=1) + + with pytest.raises(ValueError, match="failure at_event exceeds planned event count"): + validate_profile(shortened) + + +def test_validate_profile_accepts_a_duration_that_still_contains_every_burst( + tmp_path: Path, +) -> None: + profile = load_profile(_write(tmp_path, failures=[])) + + validate_profile(replace(profile, duration_seconds=3)) + + assert planned_event_count(replace(profile, duration_seconds=3)) == 140 + + +def test_aks_live_1k_profile_matches_the_published_live_plan() -> None: + """The design doc's live sequence is 30 minutes of churn at 20 events/s with + three 30-second bursts at 100 events/s; the live qualification profile must + encode exactly that, otherwise three published budgets are unmeasurable.""" + profile = load_profile(Path(__file__).with_name("profiles") / "aks-live-1k.json") + + assert profile.id == "aks-live-1k" + assert profile.seed == 186 + assert profile.object_count == 1000 + assert profile.namespace_count == 20 + assert profile.steady_events_per_second == 20 + assert profile.duration_seconds == 1800 + assert profile.bursts == ( + Burst(start_second=300, duration_seconds=30, events_per_second=100), + Burst(start_second=900, duration_seconds=30, events_per_second=100), + Burst(start_second=1500, duration_seconds=30, events_per_second=100), + ) + assert profile.failures == () + assert planned_event_count(profile) == 43200 diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py index c9a439f8..b497ec9d 100644 --- a/tests/performance/test_replay.py +++ b/tests/performance/test_replay.py @@ -8,14 +8,54 @@ from __future__ import annotations import asyncio +from collections.abc import AsyncIterator +from time import monotonic import pytest +from korvid.core.config import KorvidConfig +from korvid.core.store import ALL_NAMESPACES, ResourceStore, Summary +from korvid.core.watch import WatchManager +from korvid.ui.messages import ResourcesUpdated +from tests.performance.metrics import BenchmarkRecorder, RunManifest from tests.performance.profile import FailureInjection, WorkloadProfile -from tests.performance.replay import ReplayOptions, run_replay +from tests.performance.replay import ( + MeasuredKorvidApp, + ReplayAborted, + ReplayOptions, + build_manifest, + run_replay, +) from tests.performance.workload import apply_events, initial_pods, scheduled_events, summary_digest +async def _never_watch(_kind: str, _scope: str) -> AsyncIterator[tuple[str, Summary]]: + """A watch source that never yields: the render-accounting test drives the + app directly and must not race a background stream.""" + await asyncio.Event().wait() + # Unreachable; present so the function is an async *generator*, which is + # what `WatchSource` requires. + yield ("ADDED", initial_pods(_manifest_profile())[0]) + + +def _manifest_profile() -> WorkloadProfile: + return WorkloadProfile( + schema_version=1, + id="render-accounting", + seed=1, + object_count=1, + namespace_count=1, + steady_events_per_second=0, + duration_seconds=1, + bursts=(), + failures=(), + ) + + +def _manifest_for_test() -> RunManifest: + return build_manifest(_manifest_profile()) + + async def test_replay_uses_real_app_and_reaches_expected_digest() -> None: profile = WorkloadProfile( schema_version=1, @@ -169,3 +209,148 @@ async def virtual_sleep(delay: float) -> None: assert report.api.operations["watch_open"] == 2 assert report.api.reconnects == 1 assert report.api.relists == 1 + + +async def test_replay_throttled_reconnects_and_digest_matches() -> None: + """A 429 ends one watch connection; `WatchManager` retries, the source + re-LISTs from its tracked state, and the run still reaches the oracle + digest with zero drops. The throttled event itself is never delivered.""" + profile = WorkloadProfile( + schema_version=1, + id="test-throttled", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="throttled", at_event=5),), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + events = scheduled_events(profile) + applied = tuple(e for e in events if e.sequence != 5) + oracle = summary_digest(apply_events(initial_pods(profile), applied)) + + assert report.expected_digest == oracle + assert report.final_digest == oracle + assert report.dropped_updates == 0 + assert report.api.operations["list"] == 2 + assert report.api.operations["watch_open"] == 2 + assert report.api.reconnects == 1 + assert report.api.throttles == 1 + # A 429 is not a 410: it must not be counted as a re-LIST recovery. + assert report.api.relists == 0 + + +async def test_replay_slow_delays_without_dropping_or_reconnecting() -> None: + """`slow` delays one event by one steady-rate tick and still delivers it: + no reconnect, no drop, and the stall is real extra time. + + The stall is injected at the *last* scheduled event on purpose. Mid-run the + absolute-offset schedule silently absorbs a one-tick stall (the following + event's delay simply shrinks by the same tick), so only a stall with no + remaining schedule to catch up in is observable as extra virtual time: + correct behaviour totals `last offset + one tick`, while ignoring or + dropping the `slow` injection totals exactly `last offset`. + """ + profile = WorkloadProfile( + schema_version=1, + id="test-slow", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="slow", at_event=40),), + ) + virtual_time: list[float] = [0.0] + sleep_delays: list[float] = [] + + def virtual_monotonic() -> float: + return virtual_time[0] + + async def virtual_sleep(delay: float) -> None: + sleep_delays.append(delay) + virtual_time[0] += delay + await asyncio.sleep(0) + + report = await run_replay( + profile, + ReplayOptions(time_scale=1, monotonic_fn=virtual_monotonic, async_sleep=virtual_sleep), + ) + events = scheduled_events(profile) + oracle = summary_digest(apply_events(initial_pods(profile), events)) + + assert report.expected_digest == oracle + assert report.final_digest == oracle + assert report.dropped_updates == 0 + assert report.api.operations["watch_open"] == 1 + assert report.api.reconnects == 0 + # The injected 1/20s stall is an *extra* sleep on top of the schedule. + assert sum(sleep_delays) == pytest.approx(events[-1].offset_seconds + 1 / 20) + + +async def test_replay_forbidden_aborts_with_an_explicit_terminal_error() -> None: + """403 is an authorization boundary: `WatchManager` never reconnects and + clears the store, so the run can never complete. That must surface at once + as a named terminal failure instead of a 30-second `until` timeout on a + permanently empty backlog.""" + profile = WorkloadProfile( + schema_version=1, + id="test-forbidden", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="forbidden", at_event=5),), + ) + with pytest.raises(ReplayAborted, match="403"): + await run_replay(profile, ReplayOptions(time_scale=0)) + + +async def test_replay_churn_started_before_input_is_false_without_any_events() -> None: + """The flag must be a real emitted-event signal: a profile that schedules + no churn at all cannot claim churn was active during input measurement.""" + profile = WorkloadProfile( + schema_version=1, + id="test-no-churn", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=0, + duration_seconds=1, + bursts=(), + failures=(), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + + assert scheduled_events(profile) == () + assert not report.churn_started_before_input + assert report.input_latency.count > 0 + + +async def test_measured_app_counts_only_resource_update_renders() -> None: + """`_render_table` is also called by cursor/filter/sort/split-pane paths. + Counting those inflates `render_passes` and lets an unrelated repaint flush + the pending-event backlog, so only store-driven renders may be recorded.""" + recorder = BenchmarkRecorder() + app = MeasuredKorvidApp( + config=KorvidConfig(namespace=ALL_NAMESPACES), + store=ResourceStore(), + watch_manager=WatchManager(ResourceStore(), _never_watch, retry_delay=0.0), + recorder=recorder, + ) + async with app.run_test(): + recorder.record_event(1, monotonic()) + app._render_table("pods") + assert recorder.pending_count() == 1 + + app.on_resources_updated(ResourcesUpdated("pods")) + assert recorder.pending_count() == 0 + + report = recorder.report(_manifest_for_test(), (), final_digest="d") + assert report.render_passes == 1 + assert report.rendered_updates == 1 diff --git a/tests/performance/test_workload.py b/tests/performance/test_workload.py index d22789a4..8cbe5b3e 100644 --- a/tests/performance/test_workload.py +++ b/tests/performance/test_workload.py @@ -47,3 +47,17 @@ def test_events_are_stably_scheduled_and_change_final_digest() -> None: assert [event.sequence for event in events] == list(range(1, 21)) assert all(left.offset_seconds <= right.offset_seconds for left, right in pairwise(events)) assert summary_digest(apply_events(initial, events)) != summary_digest(initial) + + +def test_scheduled_events_carry_the_selected_object_index() -> None: + """The live harness maps each churn event onto a real cluster Pod. Carrying + the generator's own object index removes the need to parse it back out of + the synthetic Pod name (`int(name.removeprefix("pod-"))`), which would + become a mid-churn `ValueError` if the naming scheme ever changed.""" + profile = _profile() + events = scheduled_events(profile) + + assert events + for event in events: + assert 0 <= event.object_index < profile.object_count + assert event.summary.name == f"pod-{event.object_index:06d}" diff --git a/tests/performance/workload.py b/tests/performance/workload.py index 8df08306..c3a8a9dc 100644 --- a/tests/performance/workload.py +++ b/tests/performance/workload.py @@ -16,6 +16,10 @@ class ScheduledEvent: offset_seconds: float event_type: str summary: PodSummary + #: Index of the object the generator selected, in `initial_pods` order. + #: Carried explicitly so live replay can map the event onto its seeded + #: `(namespace, name)` identity without parsing the synthetic Pod name. + object_index: int def initial_pods(profile: WorkloadProfile) -> tuple[PodSummary, ...]: @@ -65,6 +69,7 @@ def scheduled_events(profile: WorkloadProfile) -> tuple[ScheduledEvent, ...]: offset_seconds=second + tick / rate, event_type="MODIFIED", summary=updated, + object_index=index, ) ) assert len(result) == planned_event_count(profile) diff --git a/tests/ui/test_waits.py b/tests/ui/test_waits.py index 7ed28d86..d8e71505 100644 --- a/tests/ui/test_waits.py +++ b/tests/ui/test_waits.py @@ -4,7 +4,7 @@ import pytest -from .waits import until +from .waits import WaitTimeout, until class _FakePilot: @@ -36,7 +36,7 @@ async def test_until_pauses_cover_the_full_requested_timeout() -> None: """`int(timeout / 0.05)` truncation must not shorten the wait: the pauses add up to the advertised timeout, with a shorter final step.""" pilot = _FakePilot() - with pytest.raises(AssertionError, match=r"condition not met within 0\.12s"): + with pytest.raises(WaitTimeout, match=r"condition not met within 0\.12s"): await until(pilot, lambda: False, timeout=0.12) assert sum(pilot.pauses) == pytest.approx(0.12) @@ -44,12 +44,22 @@ async def test_until_pauses_cover_the_full_requested_timeout() -> None: async def test_until_sub_interval_timeout_still_pauses_once() -> None: """A timeout below one 50ms tick still yields to the app once.""" pilot = _FakePilot() - with pytest.raises(AssertionError, match=r"condition not met within 0\.03s"): + with pytest.raises(WaitTimeout, match=r"condition not met within 0\.03s"): await until(pilot, lambda: False, timeout=0.03) assert pilot.pauses == [pytest.approx(0.03)] async def test_until_raises_with_label_on_timeout() -> None: pilot = _FakePilot() - with pytest.raises(AssertionError, match=r"dialog visible not met within 0\.1s"): + with pytest.raises(WaitTimeout, match=r"dialog visible not met within 0\.1s"): await until(pilot, lambda: False, timeout=0.1, label="dialog visible") + + +async def test_until_timeout_is_not_an_assertion_error() -> None: + """A wait timeout is an operational outcome, not a programmer `assert`: + callers (e.g. the benchmark CLI) must be able to catch it without also + swallowing genuine `AssertionError`s raised by production code.""" + pilot = _FakePilot() + with pytest.raises(WaitTimeout) as caught: + await until(pilot, lambda: False, timeout=0.05, label="never") + assert not isinstance(caught.value, AssertionError) diff --git a/tests/ui/waits.py b/tests/ui/waits.py index 9a53f251..cffad631 100644 --- a/tests/ui/waits.py +++ b/tests/ui/waits.py @@ -10,6 +10,16 @@ from typing import Any +class WaitTimeout(Exception): + """Raised when `until` gives up waiting for a condition. + + Deliberately *not* an `AssertionError`: a wait timeout is an operational + outcome a caller may catch and report (e.g. the benchmark CLI turns it + into exit 1), while a genuine `assert` inside production code must keep + propagating as the programmer error it is. + """ + + async def until( pilot: Any, cond: Callable[[], object], @@ -23,6 +33,9 @@ async def until( condition is re-checked once after the final pause so it cannot fail on an outcome that arrived during the last tick, and `label` names the awaited outcome in the failure message for easier CI diagnosis. + + Raises: + WaitTimeout: `cond()` was still falsy after `timeout` seconds. """ remaining = timeout while remaining > 0: @@ -33,4 +46,4 @@ async def until( remaining -= step if cond(): return - raise AssertionError(f"{label} not met within {timeout}s") + raise WaitTimeout(f"{label} not met within {timeout}s") From a0eeebc18210055d45fc88ec2df44da2ebe09e7f Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 09:58:42 +0900 Subject: [PATCH 25/38] fix(issue-186-review): honor mutation Retry-After hints Preserve Kubernetes 429 retry metadata, bound retry delays, and spread concurrent workers with deterministic target-specific jitter.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...luster-performance-qualification-design.md | 5 +- src/korvid/k8s/errors.py | 10 +- tests/performance/live.py | 84 +++++++++++- tests/performance/test_live.py | 127 ++++++++++++++++++ 4 files changed, 221 insertions(+), 5 deletions(-) diff --git a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md index ab6d5075..b8c68218 100644 --- a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md +++ b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md @@ -220,8 +220,9 @@ separately from application read-path throttles. Live churn is driven with explicit bounded concurrency and a bounded per-patch timeout: a serial driver is capped at one round trip per event and cannot approach the scheduled rate, which would silently understate the load the -report claims to have applied. Only HTTP 429 is retried, with a bounded policy -that re-issues the identical guarded patch. +report claims to have applied. Only HTTP 429 is retried, with bounded +target-specific jitter that honors the API server's `Retry-After` hint up to an +explicit delay ceiling and re-issues the identical guarded patch. ### Guardrails diff --git a/src/korvid/k8s/errors.py b/src/korvid/k8s/errors.py index 43d9e093..1772cd3d 100644 --- a/src/korvid/k8s/errors.py +++ b/src/korvid/k8s/errors.py @@ -10,7 +10,14 @@ class ApiStatusError(Exception): """Raised by the k8s layer when an API request returns an HTTP error status.""" - def __init__(self, status: int, reason: str, body: str = "") -> None: + def __init__( + self, + status: int, + reason: str, + body: str = "", + *, + retry_after_seconds: float | None = None, + ) -> None: super().__init__(f"API {status}: {reason}") self.status = status self.reason = reason @@ -19,3 +26,4 @@ def __init__(self, status: int, reason: str, body: str = "") -> None: #: eviction's PDB denial vs API Priority and Fairness throttling, #: both 429 - inspect it. self.body = body + self.retry_after_seconds: float | None = retry_after_seconds diff --git a/tests/performance/live.py b/tests/performance/live.py index de2c012d..103ad316 100644 --- a/tests/performance/live.py +++ b/tests/performance/live.py @@ -62,7 +62,9 @@ from __future__ import annotations import asyncio +import hashlib import json +import math from collections.abc import AsyncIterator, Awaitable, Callable, Iterable, Mapping from dataclasses import dataclass, field from time import monotonic @@ -136,6 +138,8 @@ class LiveLimits: patch after HTTP 429, and only after 429. mutation_retry_base_delay_seconds: First backoff delay; doubles per retry. + mutation_retry_max_delay_seconds: Hard ceiling for exponential backoff, + deterministic jitter, and a server-provided `Retry-After` hint. mutation_connect_timeout_seconds: Ceiling for connecting the mutation client (a kubeconfig exec credential plugin can block). initial_render_timeout_seconds: Ceiling for the initial 1,000-row @@ -150,6 +154,7 @@ class LiveLimits: mutation_timeout_seconds: float = 30.0 mutation_throttle_retries: int = 5 mutation_retry_base_delay_seconds: float = 0.5 + mutation_retry_max_delay_seconds: float = 30.0 mutation_connect_timeout_seconds: float = 60.0 initial_render_timeout_seconds: float = 60.0 churn_grace_seconds: float = 300.0 @@ -394,7 +399,18 @@ async def patch_pod_labels_guarded( _content_type="application/json-patch+json", # type: ignore[call-arg] # kubernetes_asyncio's .pyi stub omits _content_type; accepted via **kwargs at runtime ) except k8s_client.exceptions.ApiException as exc: - raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc + raw_body = getattr(exc, "body", "") or "" + body = ( + raw_body.decode("utf-8", errors="replace") + if isinstance(raw_body, bytes) + else str(raw_body) + ) + raise ApiStatusError( + int(exc.status or 0), + str(exc.reason or ""), + body=body, + retry_after_seconds=_api_retry_after_seconds(exc, body), + ) from exc async def close(self) -> None: if self._api is not None: @@ -677,6 +693,60 @@ def _first_error(group: BaseExceptionGroup[BaseException]) -> BaseException: return leaves[0] if leaves else group +def _parse_retry_seconds(value: object) -> float | None: + try: + seconds = float(str(value).strip()) + except (TypeError, ValueError): + return None + if not math.isfinite(seconds) or seconds < 0: + return None + return seconds + + +def _api_retry_after_seconds(exc: BaseException, body: str) -> float | None: + headers = getattr(exc, "headers", None) + if isinstance(headers, Mapping): + for key, value in headers.items(): + if str(key).lower() == "retry-after": + parsed = _parse_retry_seconds(value) + if parsed is not None: + return parsed + + try: + payload = json.loads(body) + except (json.JSONDecodeError, TypeError): + return None + if not isinstance(payload, dict): + return None + details = payload.get("details") + if not isinstance(details, dict): + return None + return _parse_retry_seconds(details.get("retryAfterSeconds")) + + +def _mutation_retry_delay_seconds( + exc: ApiStatusError, + *, + namespace: str, + name: str, + uid: str, + tick: str, + attempt: int, + limits: LiveLimits, +) -> float: + backoff = limits.mutation_retry_base_delay_seconds * (2 ** (attempt - 1)) + digest = hashlib.sha256(f"{namespace}\0{name}\0{uid}\0{tick}\0{attempt}".encode()).digest() + jitter_fraction = int.from_bytes(digest[:8], "big") / (1 << 64) + jittered_backoff = backoff * jitter_fraction + server_hint = exc.retry_after_seconds or 0.0 + return float( + min( + max(server_hint, jittered_backoff), + limits.mutation_retry_max_delay_seconds, + ) + ) + + async def _mutate_once( mutation_client: MutationClient, *, @@ -711,7 +781,17 @@ async def _mutate_once( raise progress.mutation_throttles += 1 attempt += 1 - await sleep(limits.mutation_retry_base_delay_seconds * (2 ** (attempt - 1))) + await sleep( + _mutation_retry_delay_seconds( + exc, + namespace=namespace, + name=name, + uid=uid, + tick=tick, + attempt=attempt, + limits=limits, + ) + ) continue progress.record_completed(now()) return diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py index 4f93fee4..8b04a4d2 100644 --- a/tests/performance/test_live.py +++ b/tests/performance/test_live.py @@ -16,6 +16,8 @@ from typing import Any import pytest +from kubernetes_asyncio import client as k8s_client +from multidict import CIMultiDict, CIMultiDictProxy from korvid.core.store import Summary from korvid.k8s.discovery import ResourceMeta @@ -1407,6 +1409,131 @@ async def patch_pod_labels_guarded( assert progress.completed == 1 +@pytest.mark.parametrize( + ("headers", "body", "expected"), + [ + ({"Retry-After": "7"}, '{"details": {"retryAfterSeconds": 5}}', 7.0), + ({}, '{"details": {"retryAfterSeconds": 5}}', 5.0), + ], +) +async def test_mutation_client_preserves_server_retry_after_hint( + headers: dict[str, str], body: str, expected: float +) -> None: + class _ThrottledCoreV1: + async def patch_namespaced_pod(self, *_args: object, **_kwargs: object) -> None: + exc = k8s_client.exceptions.ApiException(status=429, reason="Too Many Requests") + object.__setattr__(exc, "headers", CIMultiDictProxy(CIMultiDict(headers))) + object.__setattr__(exc, "body", body.encode()) + raise exc + + client = live._KubeMutationClient(CONTEXT, RUN_ID) + client._core_v1 = _ThrottledCoreV1() # type: ignore[assignment] # focused adapter fake + + with pytest.raises(ApiStatusError, match="API 429") as caught: + await client.patch_pod_labels_guarded( + "korvid-perf-aks186-0", + "bench-0", + uid="pod-uid-0", + tick="1", + ) + + assert caught.value.body == body + assert caught.value.retry_after_seconds == expected + + +async def test_mutation_retry_respects_server_hint_with_an_explicit_delay_bound() -> None: + class _ThrottleOnce: + def __init__(self) -> None: + self.calls = 0 + + async def connect(self) -> None: + pass + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.calls += 1 + if self.calls == 1: + raise ApiStatusError( + 429, + "Too Many Requests", + retry_after_seconds=10.0, + ) + + async def close(self) -> None: + pass + + client = _ThrottleOnce() + sleeps: list[float] = [] + + async def _sleep(delay: float) -> None: + sleeps.append(delay) + + await live._mutate_once( + client, + namespace="korvid-perf-run1-0", + name="bench-0", + uid="pod-uid-0", + tick="1", + progress=ChurnProgress(), + limits=LiveLimits( + mutation_throttle_retries=1, + mutation_retry_base_delay_seconds=0.5, + mutation_retry_max_delay_seconds=3.0, + ), + sleep=_sleep, + now=lambda: 0.0, + ) + + assert client.calls == 2 + assert sleeps == [3.0] + + +async def test_mutation_retry_jitter_avoids_lockstep_workers() -> None: + class _ThrottleOnce: + def __init__(self) -> None: + self.calls = 0 + + async def connect(self) -> None: + pass + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.calls += 1 + if self.calls == 1: + raise ApiStatusError(429, "Too Many Requests") + + async def close(self) -> None: + pass + + sleeps: list[list[float]] = [[], []] + for index, name in enumerate(("bench-0", "bench-1")): + + async def _sleep(delay: float, target: list[float] = sleeps[index]) -> None: + target.append(delay) + + await live._mutate_once( + _ThrottleOnce(), + namespace="korvid-perf-run1-0", + name=name, + uid=f"pod-uid-{index}", + tick="1", + progress=ChurnProgress(), + limits=LiveLimits( + mutation_throttle_retries=1, + mutation_retry_base_delay_seconds=2.0, + mutation_retry_max_delay_seconds=3.0, + ), + sleep=_sleep, + now=lambda: 0.0, + ) + + assert 0.0 <= sleeps[0][0] <= 2.0 + assert 0.0 <= sleeps[1][0] <= 2.0 + assert sleeps[0] != sleeps[1] + + async def test_drive_live_churn_gives_up_after_the_bounded_throttle_retries() -> None: class _AlwaysThrottling(_FakeMutationClient): async def patch_pod_labels_guarded( From d04d8a5cbb89a5997456c547c51ad1d7230ca997 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 10:20:23 +0900 Subject: [PATCH 26/38] fix(issue-186-review): desynchronize hinted retries Treat Retry-After as a server floor and add stable target-specific jitter before applying the configured ceiling, preventing workers from retrying in lockstep when the hint dominates exponential backoff.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/live.py | 4 +++- tests/performance/test_live.py | 14 +++++++++----- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/tests/performance/live.py b/tests/performance/live.py index 103ad316..9fdfb5fa 100644 --- a/tests/performance/live.py +++ b/tests/performance/live.py @@ -741,7 +741,7 @@ def _mutation_retry_delay_seconds( server_hint = exc.retry_after_seconds or 0.0 return float( min( - max(server_hint, jittered_backoff), + server_hint + jittered_backoff, limits.mutation_retry_max_delay_seconds, ) ) @@ -764,6 +764,8 @@ async def _mutate_once( The retry re-issues the *identical* guarded patch, so it is still atomic and still fail-closed: a Pod that lost its identity or labels between attempts fails the `test` ops exactly as it would on the first attempt. + A server-provided delay is treated as a floor, then target-specific jitter + is added without exceeding the configured retry-delay ceiling. Every other status - including a failed `test` (422) - propagates immediately and aborts the whole run. """ diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py index 8b04a4d2..669433b8 100644 --- a/tests/performance/test_live.py +++ b/tests/performance/test_live.py @@ -1489,7 +1489,7 @@ async def _sleep(delay: float) -> None: assert sleeps == [3.0] -async def test_mutation_retry_jitter_avoids_lockstep_workers() -> None: +async def test_mutation_retry_jitter_avoids_lockstep_workers_when_server_hint_dominates() -> None: class _ThrottleOnce: def __init__(self) -> None: self.calls = 0 @@ -1502,7 +1502,11 @@ async def patch_pod_labels_guarded( ) -> None: self.calls += 1 if self.calls == 1: - raise ApiStatusError(429, "Too Many Requests") + raise ApiStatusError( + 429, + "Too Many Requests", + retry_after_seconds=1.0, + ) async def close(self) -> None: pass @@ -1522,15 +1526,15 @@ async def _sleep(delay: float, target: list[float] = sleeps[index]) -> None: progress=ChurnProgress(), limits=LiveLimits( mutation_throttle_retries=1, - mutation_retry_base_delay_seconds=2.0, + mutation_retry_base_delay_seconds=0.5, mutation_retry_max_delay_seconds=3.0, ), sleep=_sleep, now=lambda: 0.0, ) - assert 0.0 <= sleeps[0][0] <= 2.0 - assert 0.0 <= sleeps[1][0] <= 2.0 + assert 1.0 < sleeps[0][0] <= 1.5 + assert 1.0 < sleeps[1][0] <= 1.5 assert sleeps[0] != sleeps[1] From 86346dd54237eabdeb4ca6f3bce89770dbff057a Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 10:51:58 +0900 Subject: [PATCH 27/38] fix(issue-186-review): report terminal replay aborts Handle expected terminal replay failures as clean CLI errors while preserving propagation of unexpected programmer defects. Add a valid forbidden-failure profile regression test that proves no traceback is emitted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/cli.py | 4 ++-- tests/performance/test_cli.py | 19 ++++++++++++++++++- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/tests/performance/cli.py b/tests/performance/cli.py index c1542c80..f226868b 100644 --- a/tests/performance/cli.py +++ b/tests/performance/cli.py @@ -41,7 +41,7 @@ from tests.performance.manifests import build_seed_manifests from tests.performance.metrics import BenchmarkReport, render_markdown, report_payload from tests.performance.profile import WorkloadProfile, load_profile, validate_profile -from tests.performance.replay import ReplayOptions, ReplayReport, run_replay +from tests.performance.replay import ReplayAborted, ReplayOptions, ReplayReport, run_replay from tests.ui.waits import WaitTimeout @@ -332,7 +332,7 @@ def _cmd_replay(args: argparse.Namespace) -> int: replay = _run_with_cpu_profile(profile, options, args.cpu_profile) else: replay = asyncio.run(run_replay(profile, options)) - except (ApiStatusError, WaitTimeout, OSError) as exc: + except (ReplayAborted, ApiStatusError, WaitTimeout, OSError) as exc: print(f"error during replay: {exc}", file=sys.stderr) return 1 finally: diff --git a/tests/performance/test_cli.py b/tests/performance/test_cli.py index 35456b3a..f0f3e0e4 100644 --- a/tests/performance/test_cli.py +++ b/tests/performance/test_cli.py @@ -19,7 +19,7 @@ ProcessSummary, RunManifest, ) -from tests.performance.profile import WorkloadProfile +from tests.performance.profile import FailureInjection, WorkloadProfile from tests.performance.replay import ReplayOptions, ReplayReport # --------------------------------------------------------------------------- @@ -317,6 +317,23 @@ def test_cli_reports_expected_replay_errors( assert "error during replay: API 503: unavailable" in capsys.readouterr().err +def test_cli_reports_terminal_replay_abort_without_traceback( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + profile = replace( + _make_minimal_profile(), + steady_events_per_second=2, + failures=(FailureInjection(kind="forbidden", at_event=1),), + ) + monkeypatch.setattr(cli, "load_profile", lambda _path: profile) + + assert cli.main(["replay", "--profile", "profile.json", "--time-scale", "0"]) == 1 + stderr = capsys.readouterr().err + assert "error during replay: replay aborted:" in stderr + assert "Traceback" not in stderr + + def test_cli_does_not_hide_unexpected_profile_errors(monkeypatch: pytest.MonkeyPatch) -> None: def fail_profile(_path: Path) -> WorkloadProfile: raise TypeError("unexpected profile defect") From ce4e4faca4234d82b2fcb9928defb04509df6341 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 11:53:25 +0900 Subject: [PATCH 28/38] fix(issue-186-review): complete PR #202 qualification review wave Address all inline review findings for the large-cluster qualification harness: - generated Pods tolerate the documented korvid.dev/performance taint - fail-closed AKS gate validates the immutable dedicated-test resource group, cluster name, and required test-only tags from az aks show before clients - ownership/preflight rejects non-Running/non-Ready owned Pods (exactly 1000) - explicit read/setup connection timeout on identity, harness, and app connects - resolve the real korvid SHA (GITHUB_SHA/git HEAD); live fails closed if none - live-specific manifest records context/ARM id, Kubernetes and node-pool version/count metadata, persisted to JSON and Markdown - separate LIST-to-populated-table/startup phases from watch event-to-render - machine-readable process-start-to-interactive, LIST-to-populated-table, max backlog depth, and post-burst drain summaries - drive filter, sort, namespace switch, split pane, describe, and multi-log UI-at-scale scenarios through the real pilot during churn, recording outcomes - extend failure vocabulary with metrics_unavailable and slow_logs plus report evidence - require exactly four run-labelled live artifacts for replay-live - persist expected/final digest and an explicit match flag - fit the RSS slope only over post-warm-up steady-state samples - correct the design namespace contract to match the generator The audit-injection finding is rejected: the audit invariant governs product agent write tools, not this dedicated performance harness; the stronger cluster/ownership/UID JSON-Patch guards are preserved and tested. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...luster-performance-qualification-design.md | 2 +- tests/performance/cli.py | 65 ++- tests/performance/live.py | 331 +++++++++++++-- tests/performance/manifests.py | 4 +- tests/performance/metrics.py | 274 ++++++++++++- tests/performance/profile.py | 17 +- tests/performance/replay.py | 155 ++++++- tests/performance/test_cli.py | 85 +++- tests/performance/test_live.py | 380 +++++++++++++++++- tests/performance/test_manifests.py | 4 +- tests/performance/test_metrics.py | 119 +++++- tests/performance/test_profile.py | 13 + tests/performance/test_replay.py | 150 ++++++- 13 files changed, 1522 insertions(+), 77 deletions(-) diff --git a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md index b8c68218..1cc8badf 100644 --- a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md +++ b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md @@ -179,7 +179,7 @@ of memory while remaining well inside the verified quota. ### Workload The workload generator creates 20 labelled namespaces named -`korvid-performance--00` through `korvid-performance--19`, with 50 +`korvid-perf--0` through `korvid-perf--19`, with 50 Pods in each. Pods use `registry.k8s.io/pause:3.10`, select the `perftest` pool, tolerate only the performance taint, and request 5 millicores and 16 MiB. Every namespace and Pod has: diff --git a/tests/performance/cli.py b/tests/performance/cli.py index f226868b..d6f1bd04 100644 --- a/tests/performance/cli.py +++ b/tests/performance/cli.py @@ -52,11 +52,16 @@ def _to_benchmark_report(replay: ReplayReport) -> BenchmarkReport: input_latency=replay.input_latency, process=replay.process, api=replay.api, + phases=replay.phases, rendered_updates=replay.rendered_updates, render_passes=replay.render_passes, coalesced_updates=replay.coalesced_updates, dropped_updates=replay.dropped_updates, final_digest=replay.final_digest, + expected_digest=replay.expected_digest, + digest_match=replay.expected_digest == replay.final_digest, + failures_injected=replay.failures_injected, + ui_scenarios=replay.ui_scenarios, churn=replay.churn, ) @@ -188,26 +193,29 @@ def _build_parser() -> argparse.ArgumentParser: dest="json_path", default=None, metavar="PATH", - help="Write machine-readable JSON report.", + help="Required live artifact: machine-readable JSON report (filename must " + "include the run id).", ) lp.add_argument( "--out", dest="out_path", default=None, metavar="PATH", - help="Write Markdown report to file (also printed to stdout).", + help="Required live artifact: Markdown report (also printed to stdout; " + "filename must include the run id).", ) lp.add_argument( "--cpu-profile", default=None, metavar="PATH", - help="Write cProfile pstats file.", + help="Required live artifact: cProfile pstats file (filename must include the run id).", ) lp.add_argument( "--allocation-snapshot", default=None, metavar="PATH", - help="Write top-100 tracemalloc source locations.", + help="Required live artifact: top-100 tracemalloc source locations " + "(filename must include the run id).", ) return parser @@ -374,6 +382,45 @@ def _load_live_profile(args: argparse.Namespace) -> WorkloadProfile | None: return overridden +#: The four externally-retained artifacts a *successful* live qualification +#: must produce (design "Results and documentation": raw JSON samples, process +#: traces, allocation snapshots, plus the human-readable summary). Unlike an +#: ordinary offline `replay`, every live run must write all four to a +#: run-labelled destination so the evidence is retained and traceable. +_LIVE_ARTIFACT_FLAGS: dict[str, str] = { + "json_path": "--json", + "out_path": "--out", + "cpu_profile": "--cpu-profile", + "allocation_snapshot": "--allocation-snapshot", +} + + +def _validate_live_artifacts(args: argparse.Namespace, *, run_id: str) -> str | None: + """Require exactly the four run-labelled live artifact destinations. + + Returns an error message when any of the four is missing, when two point at + the same destination, or when a destination's filename does not carry the + run id (so a retained artifact can always be traced back to its run). + Returns `None` when all four are present, distinct, and run-labelled. + """ + missing = [flag for attr, flag in _LIVE_ARTIFACT_FLAGS.items() if not getattr(args, attr)] + if missing: + return ( + "a successful live qualification must write all four artifacts " + f"({', '.join(_LIVE_ARTIFACT_FLAGS.values())}); missing: {', '.join(missing)}" + ) + paths = {attr: Path(getattr(args, attr)) for attr in _LIVE_ARTIFACT_FLAGS} + if len({str(path) for path in paths.values()}) != len(paths): + return "the four live artifacts must be four distinct destinations" + for attr, path in paths.items(): + if run_id not in path.name: + return ( + f"live artifact {_LIVE_ARTIFACT_FLAGS[attr]} filename {path.name!r} " + f"must include the run id {run_id!r}" + ) + return None + + def _cmd_replay_live(args: argparse.Namespace) -> int: if args.duration is not None and args.duration <= 0: print("error: --duration must be positive", file=sys.stderr) @@ -386,10 +433,20 @@ def _cmd_replay_live(args: argparse.Namespace) -> int: if profile is None: return 1 + artifact_error = _validate_live_artifacts(args, run_id=args.run_id) + if artifact_error: + print(f"error: {artifact_error}", file=sys.stderr) + return 1 + # No --time-scale option: live churn always replays at real wall-clock # time (ReplayOptions.time_scale defaults to 1.0). options = ReplayOptions(sample_interval=args.sample_interval) + return _execute_live_replay(args, profile, options) + +def _execute_live_replay( + args: argparse.Namespace, profile: WorkloadProfile, options: ReplayOptions +) -> int: if args.allocation_snapshot: tracemalloc.start() diff --git a/tests/performance/live.py b/tests/performance/live.py index 9fdfb5fa..a73098aa 100644 --- a/tests/performance/live.py +++ b/tests/performance/live.py @@ -68,6 +68,7 @@ from collections.abc import AsyncIterator, Awaitable, Callable, Iterable, Mapping from dataclasses import dataclass, field from time import monotonic +from types import MappingProxyType from typing import Any, Protocol, cast from urllib.parse import urlparse @@ -84,13 +85,19 @@ from korvid.k8s.telemetry import ReadTelemetry from korvid.ui.widgets.resource_table import ResourceTable from tests.performance import manifests -from tests.performance.metrics import BenchmarkRecorder, ChurnSummary, ProcessSampler +from tests.performance.metrics import ( + BenchmarkRecorder, + ChurnSummary, + NodePoolInfo, + ProcessSampler, +) from tests.performance.profile import WorkloadProfile, validate_profile from tests.performance.replay import ( MeasuredKorvidApp, ReplayOptions, ReplayReport, build_manifest, + resolve_korvid_sha, wait_for, ) from tests.performance.workload import ScheduledEvent, scheduled_events, summary_digest @@ -104,6 +111,17 @@ _REQUIRED_OBJECT_COUNT = 1000 _REQUIRED_NAMESPACE_COUNT = 20 +#: The immutable dedicated-test target contract (design doc "Fixed target and +#: capacity"). The identity gate fails closed unless `az aks show` reports +#: exactly this resource group, cluster name, and both required tags, so a +#: production cluster (or its ARM id) can never satisfy the gate even if the +#: operator supplies a matching, valid ARM id and kubeconfig context. +_REQUIRED_RESOURCE_GROUP = "rg-korvid-contract-test" +_REQUIRED_CLUSTER_NAME = "aks-korvid-contract-test" +_REQUIRED_TAGS: Mapping[str, str] = MappingProxyType( + {"purpose": "korvid-contract-testing", "production-use": "prohibited"} +) + #: HTTP status the mutation path may retry. 429 (API Priority and Fairness) #: is the only one: it is a "come back later" answer to a well-formed, #: fully guarded request, and the retry re-issues the *identical* guarded @@ -142,6 +160,10 @@ class LiveLimits: deterministic jitter, and a server-provided `Retry-After` hint. mutation_connect_timeout_seconds: Ceiling for connecting the mutation client (a kubeconfig exec credential plugin can block). + read_connect_timeout_seconds: Ceiling for external read-path setup that + a kubeconfig exec credential plugin can block indefinitely - the + identity/context-host lookup, the harness read client connect, and + the application read client connect. initial_render_timeout_seconds: Ceiling for the initial 1,000-row render. churn_grace_seconds: Allowance added to the profile's own scheduled @@ -156,6 +178,7 @@ class LiveLimits: mutation_retry_base_delay_seconds: float = 0.5 mutation_retry_max_delay_seconds: float = 30.0 mutation_connect_timeout_seconds: float = 60.0 + read_connect_timeout_seconds: float = 60.0 initial_render_timeout_seconds: float = 60.0 churn_grace_seconds: float = 300.0 convergence_timeout_seconds: float = 120.0 @@ -278,6 +301,23 @@ class LiveDependencies: kube_client_factory: Callable[[ReadTelemetry], KubeReadClient] harness_kube_client_factory: Callable[[], KubeReadClient] mutation_client_factory: Callable[[str], MutationClient] + #: Resolves the immutable korvid commit for the run manifest. A live + #: evidence run fails closed when this returns `None` rather than publish + #: an untraceable artifact. Injected in tests for determinism. + resolve_sha: Callable[[], str | None] = resolve_korvid_sha + + +@dataclass(frozen=True) +class LiveClusterFacts: + """Verified, immutable facts about the live target, extracted from the + identity gate's `az aks show` payload and recorded in the live manifest so + retained evidence establishes exactly which cluster matrix was qualified. + """ + + context: str + cluster_id: str + kubernetes_version: str | None + node_pools: tuple[NodePoolInfo, ...] async def _default_command_runner(args: list[str]) -> CommandResult: @@ -453,6 +493,25 @@ def _owns(labels: Iterable[tuple[str, str]], run_id: str) -> bool: ) +def _is_running_ready(pod: PodSummary) -> bool: + """Whether *pod* is phase `Running` with every container ready. + + The published live protocol requires exactly 1,000 Running, Ready Pods + before measuring. `ready` is the kubectl-style `/` string, so + a Pod is ready only when both counts are equal and non-zero (e.g. `1/1`). + """ + if pod.phase != "Running": + return False + parts = pod.ready.split("/") + if len(parts) != 2: + return False + try: + ready, total = int(parts[0]), int(parts[1]) + except ValueError: + return False + return total > 0 and ready == total + + def _validate_time_scale(options: ReplayOptions) -> None: if options.time_scale != 1.0: raise ValueError( @@ -475,10 +534,14 @@ def _validate_topology(profile: WorkloadProfile) -> None: async def _verify_cluster_identity( - *, context: str, expected_cluster_id: str, deps: LiveDependencies -) -> None: + *, context: str, expected_cluster_id: str, deps: LiveDependencies, limits: LiveLimits +) -> LiveClusterFacts: """Fail-closed 5-step cluster identity gate; every failure raises `ValueError` before any client is constructed or any mutation attempted. + + Returns the verified, immutable cluster facts (context, ARM id, Kubernetes + version, node-pool topology) so the caller can record them in the live + manifest without a second, unbounded metadata lookup. """ active = deps.active_context() if active != context: @@ -486,7 +549,8 @@ async def _verify_cluster_identity( f"active kubeconfig context {active!r} does not match required context {context!r}" ) - hostname = await deps.context_host(context) + async with asyncio.timeout(limits.read_connect_timeout_seconds): + hostname = await deps.context_host(context) result = await deps.command_runner( ["az", "aks", "show", "--ids", expected_cluster_id, "-o", "json"] @@ -506,6 +570,8 @@ async def _verify_cluster_identity( f"az aks show returned id {resource_id!r}, expected {expected_cluster_id!r}" ) + _verify_immutable_target(payload) + fqdn = payload.get("fqdn") or "" private_fqdn = payload.get("privateFqdn") or "" expected_hostname = fqdn or private_fqdn @@ -517,6 +583,82 @@ async def _verify_cluster_identity( f"cluster hostname {expected_hostname!r}" ) + return LiveClusterFacts( + context=context, + cluster_id=expected_cluster_id, + kubernetes_version=_kubernetes_version(payload), + node_pools=_node_pools(payload), + ) + + +def _kubernetes_version(payload: dict[str, Any]) -> str | None: + """The control-plane Kubernetes version AKS reports for the cluster.""" + for key in ("currentKubernetesVersion", "kubernetesVersion"): + value = payload.get(key) + if isinstance(value, str) and value: + return value + return None + + +def _node_pools(payload: dict[str, Any]) -> tuple[NodePoolInfo, ...]: + """Node-pool name/version/count topology from the `az aks show` payload.""" + profiles = payload.get("agentPoolProfiles") + if not isinstance(profiles, list): + return () + pools: list[NodePoolInfo] = [] + for profile in profiles: + if not isinstance(profile, dict): + continue + name = profile.get("name") + if not isinstance(name, str): + continue + version = ( + profile.get("currentOrchestratorVersion") or profile.get("orchestratorVersion") or "" + ) + count = profile.get("count") + pools.append( + NodePoolInfo( + name=name, + kubernetes_version=str(version), + node_count=int(count) if isinstance(count, int) else 0, + ) + ) + return tuple(pools) + + +def _verify_immutable_target(payload: dict[str, Any]) -> None: + """Fail closed unless *payload* describes the fixed dedicated-test target. + + Validating only that `az` and the kubeconfig agree on the operator-supplied + ARM id is not enough: a production cluster plus its own matching production + id would pass. The dedicated-test contract is immutable - a fixed resource + group, cluster name, and both required test-only tags - so those attributes + are checked directly from the `az aks show` payload before any client is + constructed or any mutation is attempted. + """ + resource_group = payload.get("resourceGroup") + if resource_group != _REQUIRED_RESOURCE_GROUP: + raise ValueError( + f"az aks show resource group {resource_group!r} is not the required " + f"dedicated-test resource group {_REQUIRED_RESOURCE_GROUP!r}" + ) + name = payload.get("name") + if name != _REQUIRED_CLUSTER_NAME: + raise ValueError( + f"az aks show cluster name {name!r} is not the required dedicated-test " + f"cluster name {_REQUIRED_CLUSTER_NAME!r}" + ) + tags = payload.get("tags") + if not isinstance(tags, dict): + raise ValueError("az aks show returned no tags; required test-only tags are missing") + for key, expected_value in _REQUIRED_TAGS.items(): + if tags.get(key) != expected_value: + raise ValueError( + f"az aks show is missing required tag {key}={expected_value!r} " + f"(got {tags.get(key)!r}); refusing to target a cluster not marked " + f"as korvid contract-testing" + ) + def _expected_pod_names(namespace_count: int, object_count: int) -> tuple[str, ...]: """The exact Pod names `seed-manifests` creates in *every* namespace.""" @@ -527,16 +669,59 @@ def _expected_pod_names(namespace_count: int, object_count: int) -> tuple[str, . ) +@dataclass +class _PodOwnershipProblems: + """Accumulated ownership-gate failures across all namespaces/Pods.""" + + missing: list[str] = field(default_factory=list) + mismatched: list[str] = field(default_factory=list) + unexpected: list[str] = field(default_factory=list) + not_ready: list[str] = field(default_factory=list) + + +async def _collect_owned_pods( + kube: KubeReadClient, + *, + run_id: str, + expected_namespaces: list[str], + wanted: tuple[str, ...], +) -> tuple[dict[tuple[str, str], PodSummary], _PodOwnershipProblems]: + """List every expected namespace and classify its Pods against the contract.""" + problems = _PodOwnershipProblems() + validated: dict[tuple[str, str], PodSummary] = {} + for namespace in expected_namespaces: + pods_by_name = {pod.name: pod for pod in await kube.list_pods(namespace)} + problems.unexpected.extend( + f"{namespace}/{name}" for name in sorted(set(pods_by_name) - set(wanted)) + ) + for name in wanted: + pod = pods_by_name.get(name) + if pod is None: + problems.missing.append(f"{namespace}/{name}") + elif not _owns(pod.labels, run_id): + problems.mismatched.append(f"{namespace}/{name}") + elif not _is_running_ready(pod): + problems.not_ready.append( + f"{namespace}/{name} (phase={pod.phase}, ready={pod.ready})" + ) + else: + validated[(namespace, name)] = pod + return validated, problems + + async def _verify_ownership( kube: KubeReadClient, *, run_id: str, namespace_count: int, object_count: int ) -> dict[tuple[str, str], PodSummary]: """Ownership gate: *exactly* the expected namespaces and Pods must exist, - each with both ownership labels, checked before any churn. + each with both ownership labels *and* phase Running/Ready, before any churn. An unexpected Pod inside an owned namespace is rejected too: the application watch filters by namespace, so a foreign Pod would enter the benchmark store and turn a precise ownership violation into a generic - "1,000 rows never rendered" timeout minutes later. + "1,000 rows never rendered" timeout minutes later. A labelled but + non-Running/non-Ready Pod is rejected as well, since the published protocol + requires exactly `object_count` Running, Ready owned Pods before measuring + and the later table-row check cannot distinguish a Pending Pod's row. Collects every mismatch across all namespaces/Pods before raising, so a single failed run surfaces the full blast radius at once instead of @@ -562,29 +747,20 @@ async def _verify_ownership( ) wanted = _expected_pod_names(namespace_count, object_count) - missing_pods: list[str] = [] - mismatched_pods: list[str] = [] - unexpected_pods: list[str] = [] - validated_pods: dict[tuple[str, str], PodSummary] = {} - for namespace in expected_namespaces: - pods_by_name = {pod.name: pod for pod in await kube.list_pods(namespace)} - unexpected_pods.extend( - f"{namespace}/{name}" for name in sorted(set(pods_by_name) - set(wanted)) + validated_pods, problems = await _collect_owned_pods( + kube, run_id=run_id, expected_namespaces=expected_namespaces, wanted=wanted + ) + if problems.missing: + raise ValueError(f"missing expected pods: {', '.join(problems.missing)}") + if problems.mismatched: + raise ValueError(f"pods with mismatched ownership labels: {', '.join(problems.mismatched)}") + if problems.unexpected: + raise ValueError(f"unexpected pods in owned namespaces: {', '.join(problems.unexpected)}") + if problems.not_ready: + raise ValueError( + f"owned pods not Running or not Ready: {', '.join(problems.not_ready)}; " + f"the live run requires exactly {object_count} Running, Ready owned pods" ) - for name in wanted: - pod = pods_by_name.get(name) - if pod is None: - missing_pods.append(f"{namespace}/{name}") - elif not _owns(pod.labels, run_id): - mismatched_pods.append(f"{namespace}/{name}") - else: - validated_pods[(namespace, name)] = pod - if missing_pods: - raise ValueError(f"missing expected pods: {', '.join(missing_pods)}") - if mismatched_pods: - raise ValueError(f"pods with mismatched ownership labels: {', '.join(mismatched_pods)}") - if unexpected_pods: - raise ValueError(f"unexpected pods in owned namespaces: {', '.join(unexpected_pods)}") return validated_pods @@ -900,6 +1076,56 @@ def _store_digest(store: ResourceStore) -> str: return summary_digest(cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES))) +#: The scoped UI-at-scale scenarios issue #186 requires, each a real Textual +#: pilot key sequence driven during active churn. Sequences are self-restoring: +#: they return the workspace to a single pane showing all owned rows so the +#: post-churn row-count and digest-convergence checks still see 1,000 rows. +_UI_SCENARIOS: tuple[tuple[str, tuple[str, ...]], ...] = ( + # Filter to a substring present in every seeded Pod name ("bench-*"), so the + # filter exercises the real path without dropping any rows. + ("filter", ("slash", "b", "e", "n", "c", "h", "enter")), + # Sort by age (a metrics-free column) and back is unnecessary; sorting keeps + # every row visible. + ("sort", ("A",)), + # Namespace switch: scope to the highlighted row's namespace, then back to + # all namespaces so the full 1,000-row topology is restored. + ("namespace_switch", ("0", "0")), + # Split the workspace into two panes, then close the new pane. + ("split_pane", ("ctrl+w", "v", "ctrl+w", "q")), + # Describe the highlighted resource, then dismiss. + ("describe", ("d", "escape")), + # Multi-log the highlighted resource, then dismiss. + ("multi_log", ("L", "escape")), +) + + +async def drive_ui_scenarios( + pilot: Any, + recorder: BenchmarkRecorder, + *, + now: Callable[[], float], +) -> None: + """Drive the scoped UI-at-scale scenarios through the real Textual pilot. + + Each scenario is a real key sequence issued against the running app during + active churn; its latency and completion outcome are recorded. Every + scenario is read-only navigation - no scenario writes, deletes, or drains - + so this never weakens live safety. A scenario that raises is recorded as + `ok=False` and never aborts the safety-critical run; the sequences restore + a single-pane, all-rows workspace so later convergence checks are intact. + """ + for name, keys in _UI_SCENARIOS: + started = now() + ok = True + try: + for key in keys: + await pilot.press(key) + await pilot.pause() + except Exception: + ok = False + recorder.record_scenario(name, now() - started, ok) + + def _check_row_count(row_count: int, expected: int) -> None: """Re-assert the exact rendered row count before teardown. @@ -956,6 +1182,9 @@ async def _run_measured_window( label="initial owned pods rendered", recorder=recorder, ) + # The table is fully populated: mark the interactive boundary that + # closes the startup and LIST-to-populated-table phases. + recorder.mark_interactive(monotonic()) events = scheduled_events(profile) state.progress.requested_events = len(events) @@ -992,6 +1221,11 @@ async def _run_measured_window( await pilot.press("up") recorder.record_input(now() - t0) + # UI-at-scale evidence: drive the scoped scenarios (filter, sort, + # namespace switch, split pane, describe, multi-log) through the + # real pilot while churn is still active. Read-only navigation only. + await drive_ui_scenarios(pilot, recorder, now=now) + churn_timeout = ( profile.duration_seconds * options.time_scale + limits.churn_grace_seconds ) @@ -1060,18 +1294,31 @@ async def run_live_replay( active_limits = limits if limits is not None else LiveLimits() active_deps = deps if deps is not None else _default_dependencies(context) - await _verify_cluster_identity( - context=context, expected_cluster_id=expected_cluster_id, deps=active_deps + facts = await _verify_cluster_identity( + context=context, + expected_cluster_id=expected_cluster_id, + deps=active_deps, + limits=active_limits, ) + # Fail closed on an untraceable evidence run: a live report must be tied to + # an immutable korvid commit, resolved *before* any client is constructed. + korvid_sha = active_deps.resolve_sha() + if korvid_sha is None: + raise ValueError( + "cannot resolve an immutable korvid SHA for the live evidence run; " + "set GITHUB_SHA in CI or run from a git checkout with a resolvable HEAD" + ) + store = ResourceStore() recorder = BenchmarkRecorder() sampler = ProcessSampler(options.sample_interval) state = _LiveRunState() harness_kube = active_deps.harness_kube_client_factory() - await harness_kube.connect(context) try: + async with asyncio.timeout(active_limits.read_connect_timeout_seconds): + await harness_kube.connect(context) # Reused directly as the pre-churn uid snapshot below - no second, # redundant listing pass over the same Pods. live_state = await _verify_ownership( @@ -1094,8 +1341,9 @@ async def run_live_replay( await mutation_client.connect() kube = active_deps.kube_client_factory(recorder.record_api) - await kube.connect(context) try: + async with asyncio.timeout(active_limits.read_connect_timeout_seconds): + await kube.connect(context) source = make_live_watch_source( kube, expected_namespaces, @@ -1103,7 +1351,14 @@ async def run_live_replay( recorder=recorder, ) watch_manager = WatchManager(store, source, retry_delay=0.0) - manifest = build_manifest(profile) + manifest = build_manifest( + profile, + korvid_sha=korvid_sha, + context=facts.context, + cluster_id=facts.cluster_id, + kubernetes_version=facts.kubernetes_version, + node_pools=facts.node_pools, + ) app = MeasuredKorvidApp( config=KorvidConfig(namespace=ALL_NAMESPACES), @@ -1113,6 +1368,7 @@ async def run_live_replay( ) sampler.start() + recorder.mark_process_start(monotonic()) try: await _run_measured_window( app=app, @@ -1139,7 +1395,11 @@ async def run_live_replay( churn = state.progress.summary(requested_duration_seconds=profile.duration_seconds) benchmark = recorder.report( - manifest, process_samples, final_digest=state.final_digest, churn=churn + manifest, + process_samples, + final_digest=state.final_digest, + expected_digest=state.expected_digest, + churn=churn, ) return ReplayReport( @@ -1155,6 +1415,9 @@ async def run_live_replay( churn_started_before_input=state.churn_started_before_input, process=benchmark.process, api=benchmark.api, + phases=benchmark.phases, manifest=benchmark.manifest, + failures_injected=benchmark.failures_injected, + ui_scenarios=benchmark.ui_scenarios, churn=churn, ) diff --git a/tests/performance/manifests.py b/tests/performance/manifests.py index fb22353d..cdcd482c 100644 --- a/tests/performance/manifests.py +++ b/tests/performance/manifests.py @@ -122,9 +122,9 @@ def build_seed_manifests( "nodeSelector": dict(selector), "tolerations": [ { - "key": "purpose", + "key": "korvid.dev/performance", "operator": "Equal", - "value": "perftest", + "value": "true", "effect": "NoSchedule", } ], diff --git a/tests/performance/metrics.py b/tests/performance/metrics.py index f8018408..8a05eb36 100644 --- a/tests/performance/metrics.py +++ b/tests/performance/metrics.py @@ -6,7 +6,7 @@ from collections import Counter from collections.abc import Callable, Coroutine, Mapping, Sequence from contextlib import suppress -from dataclasses import dataclass +from dataclasses import dataclass, field from time import monotonic from types import MappingProxyType @@ -78,6 +78,19 @@ class ProcessSample: python_bytes: int +@dataclass(frozen=True) +class NodePoolInfo: + """One AKS node pool's identity, Kubernetes version, and node count. + + Part of the live run manifest so retained evidence records exactly which + node-pool topology was qualified (issue #186 / design "Fixed target"). + """ + + name: str + kubernetes_version: str + node_count: int + + @dataclass(frozen=True) class RunManifest: profile_id: str @@ -88,6 +101,13 @@ class RunManifest: os: str cpu_count: int memory_bytes: int + #: Live-only cluster-matrix fields. Absent for deterministic replay, which + #: does not run against a real cluster; present for a live evidence run so + #: the retained report establishes which cluster was actually qualified. + context: str | None = None + cluster_id: str | None = None + kubernetes_version: str | None = None + node_pools: tuple[NodePoolInfo, ...] = () @dataclass(frozen=True) @@ -97,9 +117,21 @@ class ProcessSummary: rss_bytes_max: int | None python_bytes_max: int | None rss_slope_mib_per_minute: float | None + #: Elapsed-seconds boundary below which samples are treated as warm-up + #: (process start plus initial table population) and excluded from the + #: steady-state RSS slope fit. The published budget is explicitly a + #: *post-warm-up* slope, so startup allocation must not contaminate it. + rss_slope_warmup_boundary_seconds: float + #: Number of post-warm-up samples the slope was actually fitted over. + rss_slope_sample_count: int @classmethod - def from_samples(cls, samples: Sequence[ProcessSample]) -> ProcessSummary: + def from_samples( + cls, + samples: Sequence[ProcessSample], + *, + warmup_boundary_seconds: float = 0.0, + ) -> ProcessSummary: if not samples: return cls( sample_count=0, @@ -107,13 +139,20 @@ def from_samples(cls, samples: Sequence[ProcessSample]) -> ProcessSummary: rss_bytes_max=None, python_bytes_max=None, rss_slope_mib_per_minute=None, + rss_slope_warmup_boundary_seconds=warmup_boundary_seconds, + rss_slope_sample_count=0, ) + steady_state = [ + sample for sample in samples if sample.elapsed_seconds >= warmup_boundary_seconds + ] return cls( sample_count=len(samples), cpu_percent_max=max(sample.cpu_percent for sample in samples), rss_bytes_max=max(sample.rss_bytes for sample in samples), python_bytes_max=max(sample.python_bytes for sample in samples), - rss_slope_mib_per_minute=_least_squares_slope(samples), + rss_slope_mib_per_minute=_least_squares_slope(steady_state), + rss_slope_warmup_boundary_seconds=warmup_boundary_seconds, + rss_slope_sample_count=len(steady_state), ) @@ -224,6 +263,35 @@ def from_observations( ) +@dataclass(frozen=True) +class ScenarioResult: + """Outcome and latency of one scoped UI-at-scale scenario driven through + the real Textual pilot during live churn (issue #186: filter, sort, + namespace switch, split pane, describe, multi-log).""" + + name: str + latency_seconds: float + ok: bool + + +@dataclass(frozen=True) +class PhaseSummary: + """Explicit phase measurements the numeric budgets are stated against. + + These are recorded from named lifecycle marks the harness emits, rather + than inferred from aggregate update counts, so each budget in the design + doc (process start to interactive table, LIST completion to populated + table, backlog depth, and post-burst drain time) has a machine-readable + counterpart in every report. + """ + + process_start_to_interactive_seconds: float | None + list_to_populated_table_seconds: float | None + max_backlog_depth: int + post_burst_drain_seconds: tuple[float, ...] + max_post_burst_drain_seconds: float | None + + @dataclass(frozen=True) class BenchmarkReport: manifest: RunManifest @@ -231,11 +299,27 @@ class BenchmarkReport: input_latency: LatencySummary process: ProcessSummary api: ApiSummary + phases: PhaseSummary rendered_updates: int render_passes: int coalesced_updates: int dropped_updates: int final_digest: str + #: The workload digest the run is expected to converge to. Persisting it + #: (and `digest_match`) means a report can demonstrate digest correctness + #: after the process exits and name which side differed. `None` only for + #: partial reports built without an oracle digest. + expected_digest: str | None = None + #: `True` only when an expected digest was supplied and equals the final + #: digest; pass/fail depends on this, so it is serialized explicitly. + digest_match: bool = False + #: How many of each injected failure kind (gone/throttled/forbidden/slow/ + #: metrics_unavailable/slow_logs) were actually exercised, so a report is + #: self-describing evidence that a failure profile ran rather than a bare + #: schema literal. + failures_injected: Mapping[str, int] = field(default_factory=lambda: MappingProxyType({})) + #: Scoped UI-at-scale scenarios driven during live churn (empty offline). + ui_scenarios: tuple[ScenarioResult, ...] = () #: Present only for runs that drive real mutations (live replay). churn: ChurnSummary | None = None @@ -326,9 +410,50 @@ def __init__(self) -> None: self._rendered_updates = 0 self._render_passes = 0 self._coalesced_updates = 0 + self._max_backlog_depth = 0 + self._process_start_at: float | None = None + self._list_complete_at: float | None = None + self._interactive_at: float | None = None + self._burst_end_pending: list[float] = [] + self._post_burst_drains: list[float] = [] + self._failures_injected: Counter[str] = Counter() + self._ui_scenarios: list[ScenarioResult] = [] def record_event(self, sequence: int, received_at: float) -> None: self._pending_events.append((sequence, received_at)) + self._max_backlog_depth = max(self._max_backlog_depth, len(self._pending_events)) + + def mark_process_start(self, at: float) -> None: + """Record the instant the benchmarked process/app began starting up. + + Paired with `mark_interactive` to measure process-start-to-interactive + and to derive the steady-state RSS-slope warm-up boundary. First mark + wins so a later, redundant call cannot move the origin. + """ + if self._process_start_at is None: + self._process_start_at = at + + def mark_list_complete(self, at: float) -> None: + """Record when the initial LIST finished streaming its rows. + + First mark wins: reconnect re-LISTs must not overwrite the initial + LIST-to-populated-table measurement. + """ + if self._list_complete_at is None: + self._list_complete_at = at + + def mark_interactive(self, at: float) -> None: + """Record when the table first became fully populated/interactive.""" + if self._interactive_at is None: + self._interactive_at = at + + def mark_burst_end(self, at: float) -> None: + """Record the end of a churn burst so post-burst drain can be timed. + + The drain is resolved by `record_render` the next time the pending + backlog empties at or after this instant. + """ + self._burst_end_pending.append(at) def pending_count(self) -> int: """Number of recorded events not yet flushed by a render pass. @@ -356,12 +481,61 @@ def record_render(self, rendered_at: float) -> None: self._coalesced_updates += max(len(pending) - 1, 0) for _, received_at in pending: self._event_to_render.append(rendered_at - received_at) + # The backlog is empty again: resolve every burst whose end has already + # passed into a drain measurement (end -> backlog-clear interval). + if self._burst_end_pending: + resolved = [end for end in self._burst_end_pending if end <= rendered_at] + for end in resolved: + self._post_burst_drains.append(rendered_at - end) + self._burst_end_pending = [end for end in self._burst_end_pending if end > rendered_at] def record_input(self, latency_seconds: float) -> None: self._input_latency.append(latency_seconds) def record_api(self, event: ReadTelemetryEvent) -> None: self._api_events.append(event) + # The first WATCH open follows the initial LIST completing on both the + # replay and live paths, so it is a uniform, telemetry-driven signal + # for the LIST-to-populated-table boundary (first mark wins). + if event.operation == "watch_open": + self.mark_list_complete(monotonic()) + + def record_failure(self, kind: str) -> None: + """Record that one injected failure of *kind* was actually exercised. + + Report evidence that a failure profile ran - distinct from the profile + merely declaring it - covering every versioned kind, including + `metrics_unavailable` and `slow_logs` which do not raise on the + resource watch path. + """ + self._failures_injected[kind] += 1 + + def record_scenario(self, name: str, latency_seconds: float, ok: bool) -> None: + """Record one scoped UI-at-scale scenario outcome and latency.""" + self._ui_scenarios.append(ScenarioResult(name=name, latency_seconds=latency_seconds, ok=ok)) + + def phases(self) -> PhaseSummary: + """The explicit phase measurements derived from the lifecycle marks.""" + startup: float | None = None + if self._process_start_at is not None and self._interactive_at is not None: + startup = self._interactive_at - self._process_start_at + list_to_table: float | None = None + if self._list_complete_at is not None and self._interactive_at is not None: + list_to_table = self._interactive_at - self._list_complete_at + return PhaseSummary( + process_start_to_interactive_seconds=startup, + list_to_populated_table_seconds=list_to_table, + max_backlog_depth=self._max_backlog_depth, + post_burst_drain_seconds=tuple(self._post_burst_drains), + max_post_burst_drain_seconds=( + max(self._post_burst_drains) if self._post_burst_drains else None + ), + ) + + def _warmup_boundary_seconds(self) -> float: + if self._process_start_at is None or self._interactive_at is None: + return 0.0 + return max(self._interactive_at - self._process_start_at, 0.0) def report( self, @@ -369,19 +543,28 @@ def report( process_samples: Sequence[ProcessSample], *, final_digest: str, + expected_digest: str | None = None, churn: ChurnSummary | None = None, ) -> BenchmarkReport: return BenchmarkReport( manifest=manifest, event_to_render=LatencySummary.from_samples(self._event_to_render), input_latency=LatencySummary.from_samples(self._input_latency), - process=ProcessSummary.from_samples(process_samples), + process=ProcessSummary.from_samples( + process_samples, + warmup_boundary_seconds=self._warmup_boundary_seconds(), + ), api=ApiSummary.from_events(self._api_events), + phases=self.phases(), rendered_updates=self._rendered_updates, render_passes=self._render_passes, coalesced_updates=self._coalesced_updates, dropped_updates=len(self._pending_events), final_digest=final_digest, + expected_digest=expected_digest, + digest_match=expected_digest is not None and expected_digest == final_digest, + failures_injected=MappingProxyType(dict(sorted(self._failures_injected.items()))), + ui_scenarios=tuple(self._ui_scenarios), churn=churn, ) @@ -399,6 +582,17 @@ def report_payload(report: BenchmarkReport) -> dict[str, object]: "os": report.manifest.os, "cpu_count": report.manifest.cpu_count, "memory_bytes": report.manifest.memory_bytes, + "context": report.manifest.context, + "cluster_id": report.manifest.cluster_id, + "kubernetes_version": report.manifest.kubernetes_version, + "node_pools": [ + { + "name": pool.name, + "kubernetes_version": pool.kubernetes_version, + "node_count": pool.node_count, + } + for pool in report.manifest.node_pools + ], }, "latency": { "event_to_render": { @@ -422,6 +616,17 @@ def report_payload(report: BenchmarkReport) -> dict[str, object]: "rss_bytes_max": report.process.rss_bytes_max, "python_bytes_max": report.process.python_bytes_max, "rss_slope_mib_per_minute": report.process.rss_slope_mib_per_minute, + "rss_slope_warmup_boundary_seconds": (report.process.rss_slope_warmup_boundary_seconds), + "rss_slope_sample_count": report.process.rss_slope_sample_count, + }, + "phases": { + "process_start_to_interactive_seconds": ( + report.phases.process_start_to_interactive_seconds + ), + "list_to_populated_table_seconds": (report.phases.list_to_populated_table_seconds), + "max_backlog_depth": report.phases.max_backlog_depth, + "post_burst_drain_seconds": list(report.phases.post_burst_drain_seconds), + "max_post_burst_drain_seconds": report.phases.max_post_burst_drain_seconds, }, "api": { "operations": api_operations, @@ -441,7 +646,20 @@ def report_payload(report: BenchmarkReport) -> dict[str, object]: "dropped_updates": report.dropped_updates, }, "churn": _churn_payload(report.churn), - "digests": {"final": report.final_digest}, + "failures_injected": dict(report.failures_injected), + "ui_scenarios": [ + { + "name": scenario.name, + "latency_seconds": scenario.latency_seconds, + "ok": scenario.ok, + } + for scenario in report.ui_scenarios + ], + "digests": { + "expected": report.expected_digest, + "final": report.final_digest, + "match": report.digest_match, + }, } @@ -463,13 +681,34 @@ def render_markdown(report: BenchmarkReport) -> str: operation_lines = [ f"- {operation}: `{count}`" for operation, count in report.api.operations.items() ] + failure_lines = [ + f"- {kind}: `{count}`" for kind, count in report.failures_injected.items() + ] or ["- none: `0`"] + scenario_lines = [ + f"- {scenario.name}: `{_format_seconds(scenario.latency_seconds)}` " + f"(ok={str(scenario.ok).lower()})" + for scenario in report.ui_scenarios + ] or ["- none"] + manifest_lines = [ + f"- Profile ID: `{report.manifest.profile_id}`", + f"- Profile hash: `{report.manifest.profile_hash}`", + f"- Korvid SHA: `{report.manifest.korvid_sha}`", + ] + if report.manifest.context is not None: + manifest_lines.append(f"- Context: `{report.manifest.context}`") + if report.manifest.cluster_id is not None: + manifest_lines.append(f"- Cluster ARM id: `{report.manifest.cluster_id}`") + if report.manifest.kubernetes_version is not None: + manifest_lines.append(f"- Kubernetes version: `{report.manifest.kubernetes_version}`") + for pool in report.manifest.node_pools: + manifest_lines.append( + f"- Node pool `{pool.name}`: {pool.node_count} node(s) at `{pool.kubernetes_version}`" + ) lines = [ "# Large-cluster benchmark report", "", "## Run manifest", - f"- Profile ID: `{report.manifest.profile_id}`", - f"- Profile hash: `{report.manifest.profile_hash}`", - f"- Korvid SHA: `{report.manifest.korvid_sha}`", + *manifest_lines, "", "## Latency", f"- Event to render p95: `{_format_seconds(report.event_to_render.p95_seconds)}`", @@ -481,6 +720,17 @@ def render_markdown(report: BenchmarkReport) -> str: f"- CPU max: `{_format_float(report.process.cpu_percent_max)}`", f"- RSS max: `{_format_int(report.process.rss_bytes_max)}`", f"- RSS slope: `{_format_slope(report.process.rss_slope_mib_per_minute)}`", + f"- RSS slope warm-up boundary: " + f"`{_format_seconds(report.process.rss_slope_warmup_boundary_seconds)}`", + f"- RSS slope samples: `{report.process.rss_slope_sample_count}`", + "", + "## Phases", + f"- Process start to interactive: " + f"`{_format_seconds(report.phases.process_start_to_interactive_seconds)}`", + f"- LIST to populated table: " + f"`{_format_seconds(report.phases.list_to_populated_table_seconds)}`", + f"- Max backlog depth: `{report.phases.max_backlog_depth}`", + f"- Max post-burst drain: `{_format_seconds(report.phases.max_post_burst_drain_seconds)}`", "", "## Churn", f"- Requested events: `{_format_int(churn.requested_events if churn else None)}`", @@ -498,8 +748,16 @@ def render_markdown(report: BenchmarkReport) -> str: "## API operations", *operation_lines, "", + "## Failures injected", + *failure_lines, + "", + "## UI-at-scale scenarios", + *scenario_lines, + "", "## Digests", + f"- Expected digest: `{report.expected_digest or 'n/a'}`", f"- Final digest: `{report.final_digest}`", + f"- Digest match: `{str(report.digest_match).lower()}`", ] return "\n".join(lines) + "\n" diff --git a/tests/performance/profile.py b/tests/performance/profile.py index 39a5482d..95928915 100644 --- a/tests/performance/profile.py +++ b/tests/performance/profile.py @@ -6,8 +6,10 @@ from pathlib import Path from typing import Any, Literal, cast -FailureKind = Literal["gone", "throttled", "forbidden", "slow"] -_FAILURE_KINDS = frozenset({"gone", "throttled", "forbidden", "slow"}) +FailureKind = Literal["gone", "throttled", "forbidden", "slow", "metrics_unavailable", "slow_logs"] +_FAILURE_KINDS = frozenset( + {"gone", "throttled", "forbidden", "slow", "metrics_unavailable", "slow_logs"} +) _PROFILE_KEYS = frozenset( { "schema_version", @@ -180,3 +182,14 @@ def planned_event_count(profile: WorkloadProfile) -> int: total -= round(burst.duration_seconds * profile.steady_events_per_second) total += round(burst.duration_seconds * burst.events_per_second) return total + + +def burst_end_offsets(profile: WorkloadProfile) -> tuple[float, ...]: + """Absolute second offsets at which each burst's window closes. + + Ordered ascending so a driver can mark post-burst backlog drain the moment + the schedule leaves a burst, on the same time axis event offsets use. + """ + return tuple( + sorted(float(burst.start_second + burst.duration_seconds) for burst in profile.bursts) + ) diff --git a/tests/performance/replay.py b/tests/performance/replay.py index 34df56ce..259e0141 100644 --- a/tests/performance/replay.py +++ b/tests/performance/replay.py @@ -12,10 +12,14 @@ import json import os import platform +import re +import subprocess import sys -from collections.abc import AsyncIterator, Awaitable, Callable, Iterable +from collections.abc import AsyncIterator, Awaitable, Callable, Iterable, Mapping from dataclasses import asdict, dataclass, field +from pathlib import Path from time import monotonic +from types import MappingProxyType from typing import Any, cast import psutil # type: ignore[import-untyped] # no inline stubs shipped @@ -35,11 +39,14 @@ BenchmarkRecorder, ChurnSummary, LatencySummary, + NodePoolInfo, + PhaseSummary, ProcessSampler, ProcessSummary, RunManifest, + ScenarioResult, ) -from tests.performance.profile import FailureInjection, WorkloadProfile +from tests.performance.profile import FailureInjection, WorkloadProfile, burst_end_offsets from tests.performance.workload import ( ScheduledEvent, apply_events, @@ -49,6 +56,57 @@ ) from tests.ui.waits import WaitTimeout, until +#: A resolved korvid revision must be an immutable 40-character git object name. +#: Anything else (e.g. the historical literal ``dev``) is not traceable to a +#: commit and is rejected by `resolve_korvid_sha`. +_SHA_PATTERN = re.compile(r"^[0-9a-f]{40}$") + + +def _git_head() -> str | None: + """Resolve the current repository HEAD commit, or `None` if unavailable. + + Isolated behind a seam so `resolve_korvid_sha` stays deterministic under + test: production reads the real git tree, tests inject a fixed value. + """ + try: + result = subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=Path(__file__).resolve().parent, + capture_output=True, + text=True, + check=False, + ) + except OSError: + return None + if result.returncode != 0: + return None + return result.stdout.strip() + + +def resolve_korvid_sha( + *, + env: Mapping[str, str] | None = None, + git_head: Callable[[], str | None] | None = None, +) -> str | None: + """Resolve the immutable korvid commit under test from trusted sources. + + Prefers the CI-provided `GITHUB_SHA` (GitHub Actions sets it to the exact + commit being built), then falls back to the local repository HEAD. Returns + `None` when neither yields an immutable 40-hex object name, so callers can + decide how to react: offline reports mark it `unknown` (still traceable - + the report states the SHA could not be resolved), while a live evidence + run fails closed rather than publish an untraceable artifact. + """ + environ: Mapping[str, str] = os.environ if env is None else env + candidate = environ.get("GITHUB_SHA", "").strip().lower() + if _SHA_PATTERN.fullmatch(candidate): + return candidate + resolver = git_head if git_head is not None else _git_head + head = (resolver() or "").strip().lower() + if _SHA_PATTERN.fullmatch(head): + return head + return None + async def _sleep_default(delay: float) -> None: """Thin wrapper around asyncio.sleep used as the default async sleeper.""" @@ -115,10 +173,18 @@ class ReplayReport: churn_started_before_input: bool process: ProcessSummary api: ApiSummary + #: Explicit phase measurements (startup, LIST-to-populated table, backlog + #: depth, post-burst drain) the numeric budgets are stated against. + phases: PhaseSummary manifest: RunManifest #: Requested-versus-achieved churn accounting; `None` for deterministic #: replay, which drives its own source rather than a real API server. churn: ChurnSummary | None = None + #: Per-kind count of injected failures actually exercised (report evidence + #: that a failure profile ran, including `metrics_unavailable`/`slow_logs`). + failures_injected: Mapping[str, int] = field(default_factory=lambda: MappingProxyType({})) + #: Scoped UI-at-scale scenarios driven during live churn (empty offline). + ui_scenarios: tuple[ScenarioResult, ...] = () class MeasuredKorvidApp(KorvidApp): @@ -206,18 +272,38 @@ def __init__( async def _handle_failure_if_any(self, event: ScheduledEvent, index: int) -> None: """Apply failure injection for *event*; raises `ApiStatusError` for hard faults. - `slow` delays the tick without dropping the event. Hard faults (gone, - throttled, forbidden) record an error telemetry entry, advance the - next-event cursor, and raise so `WatchManager` triggers a reconnect. + `slow` delays the tick without dropping the event. `metrics_unavailable` + and `slow_logs` record read telemetry on the metrics / log paths but + never disturb the resource watch: their whole point is proving resource + navigation and render progress are independent of the metrics poller and + of log consumption, so the resource event is still delivered on time. + Hard faults (gone, throttled, forbidden) record an error telemetry + entry, advance the next-event cursor, and raise so `WatchManager` + triggers a reconnect. """ failure = self._failures.get(event.sequence) if failure is None: return + self._recorder.record_failure(failure.kind) if failure.kind == "slow": tick = 1.0 / max(self._profile.steady_events_per_second, 1) if tick * self._options.time_scale > 0: await self._sleep(tick * self._options.time_scale) return + if failure.kind == "metrics_unavailable": + # The metrics poller fails (503) while the resource watch continues: + # evidence lands on the metrics read path, not the pods path. + self._recorder.record_api( + ReadTelemetryEvent("error", "/apis/metrics.k8s.io/v1beta1/pods", status=503) + ) + return + if failure.kind == "slow_logs": + # A slow log stream: evidence lands on the log read path, and the + # resource event is delivered without any added delay. + self._recorder.record_api( + ReadTelemetryEvent("error", "/api/v1/namespaces/_/pods/_/log", status=504) + ) + return status = _HARD_FAILURE_STATUS[failure.kind] self._recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=status)) self._next_event_index = index + 1 @@ -247,10 +333,16 @@ async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summ decoded_bytes=len(list_pods) * 200, ) ) + # LIST rows populate the initial table; they are NOT event-to-render + # samples. Recording them here would let the initial LIST dominate the + # replay p95 and make it incomparable with the live path, which only + # records owned watch events. The LIST is timed separately below as the + # LIST-to-populated-table startup phase. for pod in list_pods: - self._recorder.record_event(0, monotonic()) yield ("ADDED", pod) + # The first WATCH open marks the LIST-to-populated boundary (via the + # recorder's telemetry hook), uniformly with the live path. self._recorder.record_api(ReadTelemetryEvent("watch_open", "/api/v1/pods")) if gen == 0: @@ -262,6 +354,12 @@ async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summ # can be converted to correct inter-event delays below. self._replay_start = self._now() + # Burst-end offsets (absolute seconds) mark the moment each burst's + # window closes, so post-burst backlog drain can be timed on the same + # real-clock axis the render pass records on. + burst_ends = burst_end_offsets(self._profile) + next_burst = 0 + # --- WATCH phase --- for i in range(self._next_event_index, len(self._events)): event = self._events[i] @@ -270,6 +368,10 @@ async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summ if delay > 0: await self._sleep(delay) + while next_burst < len(burst_ends) and event.offset_seconds >= burst_ends[next_burst]: + self._recorder.mark_burst_end(monotonic()) + next_burst += 1 + await self._handle_failure_if_any(event, i) key = f"{event.summary.namespace}/{event.summary.name}" @@ -289,24 +391,49 @@ async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summ await asyncio.sleep(3600.0) -def build_manifest(profile: WorkloadProfile) -> RunManifest: +def build_manifest( + profile: WorkloadProfile, + *, + korvid_sha: str | None = None, + context: str | None = None, + cluster_id: str | None = None, + kubernetes_version: str | None = None, + node_pools: tuple[NodePoolInfo, ...] = (), +) -> RunManifest: """Resolved run manifest for *profile* (profile hash plus environment). Public because both replay harnesses (`replay.py` and `live.py`) build the identical manifest for their reports. + + Args: + korvid_sha: The immutable commit to record. When `None`, it is resolved + from `resolve_korvid_sha`; an offline run that cannot resolve one + records `unknown` (still traceable - the report states it is + unresolved) rather than a fake literal. A live evidence run resolves + and fails closed *before* calling this, so it never records + `unknown`. + context: Live kube context that was verified (live runs only). + cluster_id: Verified AKS ARM resource id (live runs only). + kubernetes_version: Kubernetes server version (live runs only). + node_pools: Node-pool topology metadata (live runs only). """ profile_hash = hashlib.sha256( json.dumps(asdict(profile), sort_keys=True, separators=(",", ":"), default=str).encode() ).hexdigest() + resolved_sha = korvid_sha if korvid_sha is not None else (resolve_korvid_sha() or "unknown") return RunManifest( profile_id=profile.id, profile_hash=profile_hash, - korvid_sha="dev", + korvid_sha=resolved_sha, python=sys.version, textual=_textual_version, os=platform.platform(), cpu_count=os.cpu_count() or 1, memory_bytes=psutil.virtual_memory().total, + context=context, + cluster_id=cluster_id, + kubernetes_version=kubernetes_version, + node_pools=node_pools, ) @@ -386,6 +513,9 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay recorder=recorder, ) + # Process/app start reference for the process-start-to-interactive phase + # and the steady-state RSS-slope warm-up boundary. + recorder.mark_process_start(monotonic()) sampler.start() churn_started_before_input = False try: @@ -400,6 +530,9 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay label="initial pods rendered", recorder=recorder, ) + # The table is fully populated: mark the interactive boundary that + # closes both the startup and LIST-to-populated-table phases. + recorder.mark_interactive(monotonic()) # Release the source to emit scheduled events, then drive cursor # input while churn is active (not before the source is unblocked). @@ -462,7 +595,9 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay # Compute final digest from the store (actual state). final_digest = summary_digest(cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES))) - benchmark = recorder.report(manifest, process_samples, final_digest=final_digest) + benchmark = recorder.report( + manifest, process_samples, final_digest=final_digest, expected_digest=expected_digest + ) return ReplayReport( object_count=profile.object_count, @@ -477,5 +612,7 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay churn_started_before_input=churn_started_before_input, process=benchmark.process, api=benchmark.api, + phases=benchmark.phases, manifest=benchmark.manifest, + failures_injected=benchmark.failures_injected, ) diff --git a/tests/performance/test_cli.py b/tests/performance/test_cli.py index f0f3e0e4..69f14a4d 100644 --- a/tests/performance/test_cli.py +++ b/tests/performance/test_cli.py @@ -16,6 +16,7 @@ from tests.performance.metrics import ( ApiSummary, LatencySummary, + PhaseSummary, ProcessSummary, RunManifest, ) @@ -57,6 +58,18 @@ def _make_process() -> ProcessSummary: rss_bytes_max=None, python_bytes_max=None, rss_slope_mib_per_minute=None, + rss_slope_warmup_boundary_seconds=0.0, + rss_slope_sample_count=0, + ) + + +def _make_phases() -> PhaseSummary: + return PhaseSummary( + process_start_to_interactive_seconds=None, + list_to_populated_table_seconds=None, + max_backlog_depth=0, + post_burst_drain_seconds=(), + max_post_burst_drain_seconds=None, ) @@ -132,6 +145,7 @@ async def fake_run_replay( churn_started_before_input=True, process=_make_process(), api=_make_api(), + phases=_make_phases(), manifest=_make_manifest(), ) @@ -154,6 +168,7 @@ async def fake_failed_report( churn_started_before_input=True, process=_make_process(), api=_make_api(), + phases=_make_phases(), manifest=_make_manifest(), ) @@ -499,11 +514,66 @@ def fail_build(*_args: object, **_kwargs: object) -> object: ] +def _live_artifacts(tmp_path: Path, run_id: str = "aks186") -> list[str]: + """The four run-labelled live artifact destinations a successful live + qualification must write; every filename carries the run id.""" + return [ + "--json", + str(tmp_path / f"{run_id}-live.json"), + "--out", + str(tmp_path / f"{run_id}-live.md"), + "--cpu-profile", + str(tmp_path / f"{run_id}-live.pstats"), + "--allocation-snapshot", + str(tmp_path / f"{run_id}-live.alloc.txt"), + ] + + +@pytest.mark.parametrize("drop", ["--json", "--out", "--cpu-profile", "--allocation-snapshot"]) +def test_cli_replay_live_requires_all_four_artifacts( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], drop: str +) -> None: + """A successful live qualification must produce all four externally-retained + artifacts; dropping any one fails before the run is attempted.""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + artifacts = _live_artifacts(tmp_path) + index = artifacts.index(drop) + del artifacts[index : index + 2] + + exit_code = cli.main( + ["replay-live", "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, *artifacts] + ) + + assert exit_code == 1 + assert "all four artifacts" in capsys.readouterr().err + assert calls == [] + + +def test_cli_replay_live_rejects_artifact_not_named_for_the_run( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """Each live artifact filename must carry the run id, or it cannot be traced + back to the run that produced it.""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + artifacts = _live_artifacts(tmp_path) + artifacts[1] = str(tmp_path / "result.json") # no run id in the filename + + exit_code = cli.main( + ["replay-live", "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, *artifacts] + ) + + assert exit_code == 1 + assert "must include the run id" in capsys.readouterr().err + assert calls == [] + + def test_cli_replay_live_writes_json_and_markdown( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: - json_path = tmp_path / "result.json" - markdown_path = tmp_path / "result.md" + json_path = tmp_path / "aks186-live.json" + markdown_path = tmp_path / "aks186-live.md" calls: list[dict[str, object]] = [] monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) result = cli.main( @@ -512,10 +582,7 @@ def test_cli_replay_live_writes_json_and_markdown( "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, - "--json", - str(json_path), - "--out", - str(markdown_path), + *_live_artifacts(tmp_path), ] ) assert result == 0 @@ -605,6 +672,7 @@ def test_cli_replay_live_duration_overrides_profile_duration_only( *_LIVE_IDENTITY_ARGS, "--duration", "5", + *_live_artifacts(tmp_path), ] ) assert result == 0 @@ -653,6 +721,7 @@ def test_cli_replay_live_returns_nonzero_for_digest_failure( "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, + *_live_artifacts(tmp_path), ] ) == 1 @@ -672,6 +741,7 @@ def test_cli_replay_live_reports_expected_operational_errors( "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, + *_live_artifacts(tmp_path), ] ) == 1 @@ -690,6 +760,7 @@ def test_cli_replay_live_does_not_hide_unexpected_programmer_errors( "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, + *_live_artifacts(tmp_path), ] ) @@ -770,6 +841,7 @@ def test_cli_replay_live_rejects_duration_that_orphans_a_burst( def test_cli_replay_live_accepts_duration_that_still_contains_every_burst( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: calls: list[dict[str, object]] = [] @@ -783,6 +855,7 @@ def test_cli_replay_live_accepts_duration_that_still_contains_every_burst( *_LIVE_IDENTITY_ARGS, "--duration", "26", + *_live_artifacts(tmp_path), ] ) diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py index 669433b8..67f99ab7 100644 --- a/tests/performance/test_live.py +++ b/tests/performance/test_live.py @@ -45,13 +45,16 @@ from tests.ui.waits import WaitTimeout RUN_ID = "aks186" -CONTEXT = "aks-korvid-perf" +CONTEXT = "aks-korvid-contract-test" SUBSCRIPTION = "00000000-0000-0000-0000-000000000000" +RESOURCE_GROUP = "rg-korvid-contract-test" +CLUSTER_NAME = "aks-korvid-contract-test" CLUSTER_ID = ( - f"/subscriptions/{SUBSCRIPTION}/resourceGroups/rg" - "/providers/Microsoft.ContainerService/managedClusters/aks-korvid-perf" + f"/subscriptions/{SUBSCRIPTION}/resourceGroups/{RESOURCE_GROUP}" + f"/providers/Microsoft.ContainerService/managedClusters/{CLUSTER_NAME}" ) -FQDN = "aks-korvid-perf-dns-abc123.hcp.eastus.azmk8s.io" +FQDN = "aks-korvid-contract-test-dns-abc123.hcp.eastus.azmk8s.io" +REQUIRED_TAGS = {"purpose": "korvid-contract-testing", "production-use": "prohibited"} # --------------------------------------------------------------------------- # Fixtures / fakes @@ -248,6 +251,16 @@ async def close(self) -> None: self.closed = True +async def _context_host_ok(_context: str) -> str: + return FQDN + + +def _report_as_benchmark(report: Any) -> Any: + from tests.performance.cli import _to_benchmark_report + + return _to_benchmark_report(report) + + def _never_called(label: str) -> Callable[..., Any]: def _fail(*_args: object, **_kwargs: object) -> Any: raise AssertionError(f"{label} must not be called") @@ -256,15 +269,42 @@ def _fail(*_args: object, **_kwargs: object) -> Any: def _ok_command_runner( - *, cluster_id: str = CLUSTER_ID, fqdn: str = FQDN, private_fqdn: str = "" + *, + cluster_id: str = CLUSTER_ID, + fqdn: str = FQDN, + private_fqdn: str = "", + resource_group: str = RESOURCE_GROUP, + name: str = CLUSTER_NAME, + tags: dict[str, str] | None = None, + kubernetes_version: str = "1.30.4", + agent_pool_profiles: list[dict[str, Any]] | None = None, ) -> Callable[[Any], Awaitable[CommandResult]]: async def _run(_args: Any) -> CommandResult: - payload = {"id": cluster_id, "fqdn": fqdn, "privateFqdn": private_fqdn} + payload = { + "id": cluster_id, + "fqdn": fqdn, + "privateFqdn": private_fqdn, + "resourceGroup": resource_group, + "name": name, + "tags": REQUIRED_TAGS if tags is None else tags, + "currentKubernetesVersion": kubernetes_version, + "agentPoolProfiles": ( + [ + {"name": "perftest", "count": 5, "currentOrchestratorVersion": "1.30.4"}, + {"name": "system", "count": 1, "currentOrchestratorVersion": "1.30.4"}, + ] + if agent_pool_profiles is None + else agent_pool_profiles + ), + } return CommandResult(0, json.dumps(payload), "") return _run +_FIXED_SHA = "1234567890abcdef1234567890abcdef12345678" + + def _happy_deps( namespaces: dict[str, GenericSummary], pods: dict[tuple[str, str], PodSummary], @@ -321,6 +361,7 @@ def harness_factory() -> _FakeKubeClient: kube_client_factory=kube_factory, harness_kube_client_factory=harness_factory, mutation_client_factory=mutation_client_factory or default_mutation_factory, + resolve_sha=lambda: _FIXED_SHA, ) @@ -2045,3 +2086,330 @@ async def test_run_live_replay_rejects_a_profile_whose_bursts_escape_its_duratio run_id=RUN_ID, deps=deps, ) + + +async def test_run_live_replay_bounds_the_context_host_lookup() -> None: + """A kubeconfig exec plugin invoked while resolving the API hostname can + hang; the identity gate must bound that lookup, not block forever.""" + + async def hanging_context_host(_context: str) -> str: + await asyncio.sleep(30) + return FQDN + + deps = LiveDependencies( + command_runner=_ok_command_runner(), + active_context=lambda: CONTEXT, + context_host=hanging_context_host, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + with pytest.raises(TimeoutError): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(read_connect_timeout_seconds=0.05), + ) + + +async def test_run_live_replay_bounds_the_harness_client_connect() -> None: + """A stuck harness-client connect (exec credential plugin) must be bounded + before the ownership gate can run.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + + class _HangingConnectKube(_FakeKubeClient): + async def connect(self, context: str | None = None) -> None: + await asyncio.sleep(30) + + created: list[_HangingConnectKube] = [] + + def harness_factory() -> _HangingConnectKube: + client = _HangingConnectKube(namespaces, pods) + created.append(client) + return client + + deps = dataclasses.replace( + _happy_deps(namespaces, pods, RUN_ID), + harness_kube_client_factory=harness_factory, + ) + + with pytest.raises(TimeoutError): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(read_connect_timeout_seconds=0.05), + ) + + assert created[0].closed + + +async def test_run_live_replay_bounds_the_app_client_connect() -> None: + """A stuck application-path client connect must be bounded too.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + + class _HangingConnectKube(_FakeKubeClient): + async def connect(self, context: str | None = None) -> None: + await asyncio.sleep(30) + + created: list[_HangingConnectKube] = [] + + def app_factory(read_telemetry: ReadTelemetry) -> _HangingConnectKube: + client = _HangingConnectKube(namespaces, pods) + client.read_telemetry = read_telemetry + created.append(client) + return client + + deps = dataclasses.replace( + _happy_deps(namespaces, pods, RUN_ID), + kube_client_factory=app_factory, + ) + + with pytest.raises(TimeoutError): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(read_connect_timeout_seconds=0.05), + ) + + assert created[0].closed + + +def _identity_deps(command_runner: Callable[[Any], Awaitable[CommandResult]]) -> LiveDependencies: + async def context_host(_context: str) -> str: + return FQDN + + return LiveDependencies( + command_runner=command_runner, + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + +async def test_run_live_replay_rejects_wrong_resource_group_before_mutation() -> None: + deps = _identity_deps(_ok_command_runner(resource_group="rg-production")) + with pytest.raises(ValueError, match="resource group"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_wrong_cluster_name_before_mutation() -> None: + deps = _identity_deps(_ok_command_runner(name="aks-production")) + with pytest.raises(ValueError, match="cluster name"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_missing_required_tag_before_mutation() -> None: + deps = _identity_deps(_ok_command_runner(tags={"purpose": "korvid-contract-testing"})) + with pytest.raises(ValueError, match="required tag"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_wrong_required_tag_value_before_mutation() -> None: + deps = _identity_deps( + _ok_command_runner(tags={"purpose": "korvid-contract-testing", "production-use": "allowed"}) + ) + with pytest.raises(ValueError, match="required tag"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_non_running_pod_before_churn() -> None: + """The ownership/preflight gate requires exactly 1,000 Running, Ready owned + Pods: a labelled but non-Running Pod must be reported and reject the run + before any mutation, since the later check only counts table rows.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + victim = (manifests.namespace_name(RUN_ID, 0), manifests.pod_name(20, 0)) + pods[victim] = dataclasses.replace(pods[victim], phase="Pending") + + deps = LiveDependencies( + command_runner=_ok_command_runner(), + active_context=lambda: CONTEXT, + context_host=_context_host_ok, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=lambda: _FakeKubeClient(namespaces, pods), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + with pytest.raises(ValueError, match="not Running or not Ready"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_not_ready_pod_before_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + victim = (manifests.namespace_name(RUN_ID, 1), manifests.pod_name(20, 1)) + pods[victim] = dataclasses.replace(pods[victim], ready="0/1") + + deps = LiveDependencies( + command_runner=_ok_command_runner(), + active_context=lambda: CONTEXT, + context_host=_context_host_ok, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=lambda: _FakeKubeClient(namespaces, pods), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + with pytest.raises(ValueError, match="not Running or not Ready"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_fails_closed_when_sha_cannot_be_resolved() -> None: + """A live evidence run must not publish an untraceable artifact: if no + immutable korvid SHA can be resolved, the run fails closed before any + client is constructed.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = dataclasses.replace( + _happy_deps(namespaces, pods, RUN_ID), + resolve_sha=lambda: None, + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + kube_client_factory=_never_called("kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + with pytest.raises(ValueError, match="immutable korvid SHA"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_builds_a_live_specific_manifest() -> None: + """Retained live evidence must record which cluster matrix was qualified: + the verified context/ARM id plus Kubernetes server version and node-pool + metadata, resolved through bounded seams and persisted in the manifest.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = _happy_deps(namespaces, pods, RUN_ID) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions( + time_scale=1.0, sample_interval=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + manifest = report.manifest + assert manifest.korvid_sha == _FIXED_SHA + assert manifest.context == CONTEXT + assert manifest.cluster_id == CLUSTER_ID + assert manifest.kubernetes_version == "1.30.4" + pool_names = {pool.name for pool in manifest.node_pools} + assert "perftest" in pool_names + perftest = next(pool for pool in manifest.node_pools if pool.name == "perftest") + assert perftest.node_count == 5 + assert perftest.kubernetes_version == "1.30.4" + + # Persisted in both machine-readable and human-readable output. + from tests.performance.metrics import render_markdown, report_payload + + payload = report_payload(_report_as_benchmark(report)) + manifest_payload = payload["manifest"] + assert isinstance(manifest_payload, dict) + assert manifest_payload["context"] == CONTEXT + assert manifest_payload["cluster_id"] == CLUSTER_ID + assert manifest_payload["kubernetes_version"] == "1.30.4" + assert manifest_payload["node_pools"] == [ + {"name": "perftest", "kubernetes_version": "1.30.4", "node_count": 5}, + {"name": "system", "kubernetes_version": "1.30.4", "node_count": 1}, + ] + + markdown = render_markdown(_report_as_benchmark(report)) + assert f"- Context: `{CONTEXT}`" in markdown + assert "- Kubernetes version: `1.30.4`" in markdown + assert "perftest" in markdown + + +async def test_run_live_replay_exercises_ui_at_scale_scenarios_during_churn() -> None: + """A passing live run must provide UI-at-scale evidence: filter, sort, + namespace switch, split pane, describe, and multi-log are driven through the + real Textual pilot during active churn and their outcomes/latencies are + recorded, without weakening the digest/ownership safety guarantees.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = _happy_deps(namespaces, pods, RUN_ID) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions( + time_scale=1.0, sample_interval=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + # Safety invariants still hold. + assert report.expected_digest == report.final_digest + assert report.dropped_updates == 0 + + recorded = {scenario.name for scenario in report.ui_scenarios} + assert recorded == {"filter", "sort", "namespace_switch", "split_pane", "describe", "multi_log"} + for scenario in report.ui_scenarios: + assert scenario.ok, f"scenario {scenario.name} did not complete" + assert scenario.latency_seconds >= 0.0 diff --git a/tests/performance/test_manifests.py b/tests/performance/test_manifests.py index 128d8c55..482c4f1d 100644 --- a/tests/performance/test_manifests.py +++ b/tests/performance/test_manifests.py @@ -79,9 +79,9 @@ def test_build_seed_manifests_returns_namespaces_then_pods_in_stable_order() -> assert _pod_spec(pods[0])["nodeSelector"] == {"korvid.dev/pool": "perftest"} assert _pod_spec(pods[0])["tolerations"] == [ { - "key": "purpose", + "key": "korvid.dev/performance", "operator": "Equal", - "value": "perftest", + "value": "true", "effect": "NoSchedule", } ] diff --git a/tests/performance/test_metrics.py b/tests/performance/test_metrics.py index 05df4c32..73417c51 100644 --- a/tests/performance/test_metrics.py +++ b/tests/performance/test_metrics.py @@ -262,6 +262,10 @@ def test_report_payload_is_json_serializable_and_stable() -> None: "os": "Darwin", "cpu_count": 8, "memory_bytes": 17179869184, + "context": None, + "cluster_id": None, + "kubernetes_version": None, + "node_pools": [], }, "latency": { "event_to_render": { @@ -285,6 +289,15 @@ def test_report_payload_is_json_serializable_and_stable() -> None: "rss_bytes_max": 106954752, "python_bytes_max": 7340032, "rss_slope_mib_per_minute": 1.0, + "rss_slope_warmup_boundary_seconds": 0.0, + "rss_slope_sample_count": 3, + }, + "phases": { + "process_start_to_interactive_seconds": None, + "list_to_populated_table_seconds": None, + "max_backlog_depth": 1, + "post_burst_drain_seconds": [], + "max_post_burst_drain_seconds": None, }, "api": { "operations": {"error": 1}, @@ -304,7 +317,9 @@ def test_report_payload_is_json_serializable_and_stable() -> None: "dropped_updates": 0, }, "churn": None, - "digests": {"final": "digest-123"}, + "failures_injected": {}, + "ui_scenarios": [], + "digests": {"expected": None, "final": "digest-123", "match": False}, } assert '"rss_slope_mib_per_minute": 1.0' in encoded @@ -574,3 +589,105 @@ def test_churn_summary_reports_no_achieved_rate_without_elapsed_time() -> None: assert churn.achieved_events_per_second is None assert churn.requested_events_per_second is None + + +def test_process_summary_fits_slope_only_over_post_warmup_samples() -> None: + """A steep startup allocation ramp followed by a flat steady state must + report a ~0 slope: only samples at/after the warm-up boundary count.""" + mib = 1024 * 1024 + samples = ( + # Warm-up: 100 -> 400 MiB while the initial table populates. + ProcessSample(elapsed_seconds=0.0, cpu_percent=1.0, rss_bytes=100 * mib, python_bytes=0), + ProcessSample(elapsed_seconds=2.0, cpu_percent=1.0, rss_bytes=250 * mib, python_bytes=0), + ProcessSample(elapsed_seconds=4.0, cpu_percent=1.0, rss_bytes=400 * mib, python_bytes=0), + # Steady state: flat at 400 MiB. + ProcessSample(elapsed_seconds=64.0, cpu_percent=1.0, rss_bytes=400 * mib, python_bytes=0), + ProcessSample(elapsed_seconds=124.0, cpu_percent=1.0, rss_bytes=400 * mib, python_bytes=0), + ) + + contaminated = _metrics_module().ProcessSummary.from_samples(samples) + steady = _metrics_module().ProcessSummary.from_samples(samples, warmup_boundary_seconds=4.0) + + assert contaminated.rss_slope_mib_per_minute is not None + assert contaminated.rss_slope_mib_per_minute > 10.0 # startup contaminates the fit + assert steady.rss_slope_mib_per_minute == pytest.approx(0.0) + assert steady.rss_slope_warmup_boundary_seconds == 4.0 + assert steady.rss_slope_sample_count == 3 + # The max/peak counters still consider every sample, not just steady state. + assert steady.rss_bytes_max == 400 * mib + + +def test_recorder_derives_warmup_boundary_from_lifecycle_marks() -> None: + """The slope warm-up boundary equals process-start-to-interactive, so a + single set of lifecycle marks drives both the phase summary and the slope + exclusion consistently.""" + mib = 1024 * 1024 + recorder = BenchmarkRecorder() + recorder.mark_process_start(1000.0) + recorder.mark_list_complete(1002.0) + recorder.mark_interactive(1005.0) # interactive 5s after start + samples = ( + ProcessSample(elapsed_seconds=0.0, cpu_percent=1.0, rss_bytes=100 * mib, python_bytes=0), + ProcessSample(elapsed_seconds=5.0, cpu_percent=1.0, rss_bytes=200 * mib, python_bytes=0), + ProcessSample(elapsed_seconds=65.0, cpu_percent=1.0, rss_bytes=200 * mib, python_bytes=0), + ) + + report = recorder.report(run_manifest(), samples, final_digest="d") + + assert report.process.rss_slope_warmup_boundary_seconds == 5.0 + assert report.process.rss_slope_sample_count == 2 + assert report.phases.process_start_to_interactive_seconds == 5.0 + assert report.phases.list_to_populated_table_seconds == 3.0 + + +def test_recorder_tracks_max_backlog_depth_and_post_burst_drain() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 10.0) + recorder.record_event(2, 10.1) + recorder.record_event(3, 10.2) + assert recorder.pending_count() == 3 + # A burst ends at t=10.3 while three events are still pending. + recorder.mark_burst_end(10.3) + recorder.record_render(11.0) # backlog drains to 0 at t=11.0 + + report = recorder.report(run_manifest(), (), final_digest="d") + + assert report.phases.max_backlog_depth == 3 + assert len(report.phases.post_burst_drain_seconds) == 1 + assert report.phases.post_burst_drain_seconds[0] == pytest.approx(0.7) + assert report.phases.max_post_burst_drain_seconds == pytest.approx(0.7) + + +def test_report_persists_expected_and_final_digest_with_match_flag() -> None: + recorder = BenchmarkRecorder() + match_report = recorder.report(run_manifest(), (), final_digest="abc", expected_digest="abc") + mismatch_report = recorder.report(run_manifest(), (), final_digest="abc", expected_digest="xyz") + + assert match_report.digest_match is True + assert mismatch_report.digest_match is False + + payload = cast(dict[str, object], report_payload(mismatch_report)["digests"]) + assert payload == {"expected": "xyz", "final": "abc", "match": False} + + markdown = render_markdown(mismatch_report) + assert "- Expected digest: `xyz`" in markdown + assert "- Final digest: `abc`" in markdown + assert "- Digest match: `false`" in markdown + + +def test_render_markdown_reports_phase_measurements() -> None: + recorder = BenchmarkRecorder() + recorder.mark_process_start(100.0) + recorder.mark_list_complete(101.0) + recorder.mark_interactive(102.0) + recorder.record_event(1, 200.0) + recorder.mark_burst_end(200.1) + recorder.record_render(200.5) + + markdown = render_markdown(recorder.report(run_manifest(), (), final_digest="d")) + + assert "## Phases" in markdown + assert "- Process start to interactive: `2.000s`" in markdown + assert "- LIST to populated table: `1.000s`" in markdown + assert "- Max backlog depth: `1`" in markdown + assert "- Max post-burst drain: `0.400s`" in markdown diff --git a/tests/performance/test_profile.py b/tests/performance/test_profile.py index 470735f3..d024a7a4 100644 --- a/tests/performance/test_profile.py +++ b/tests/performance/test_profile.py @@ -142,3 +142,16 @@ def test_aks_live_1k_profile_matches_the_published_live_plan() -> None: ) assert profile.failures == () assert planned_event_count(profile) == 43200 + + +@pytest.mark.parametrize( + "kind", ["gone", "throttled", "forbidden", "slow", "metrics_unavailable", "slow_logs"] +) +def test_load_profile_accepts_every_versioned_failure_kind(tmp_path: Path, kind: str) -> None: + profile = load_profile(_write(tmp_path, failures=[{"kind": kind, "at_event": 10}])) + assert profile.failures[0].kind == kind + + +def test_load_profile_rejects_unknown_failure_kind(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="kind must be one of"): + load_profile(_write(tmp_path, failures=[{"kind": "meltdown", "at_event": 10}])) diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py index b497ec9d..0f263621 100644 --- a/tests/performance/test_replay.py +++ b/tests/performance/test_replay.py @@ -18,12 +18,13 @@ from korvid.core.watch import WatchManager from korvid.ui.messages import ResourcesUpdated from tests.performance.metrics import BenchmarkRecorder, RunManifest -from tests.performance.profile import FailureInjection, WorkloadProfile +from tests.performance.profile import Burst, FailureInjection, WorkloadProfile from tests.performance.replay import ( MeasuredKorvidApp, ReplayAborted, ReplayOptions, build_manifest, + resolve_korvid_sha, run_replay, ) from tests.performance.workload import apply_events, initial_pods, scheduled_events, summary_digest @@ -75,7 +76,7 @@ async def test_replay_uses_real_app_and_reaches_expected_digest() -> None: assert report.expected_digest == oracle assert report.final_digest == oracle assert report.dropped_updates == 0 - assert report.rendered_updates == 110 + assert report.rendered_updates == 10 assert report.input_latency.count > 0 assert report.churn_started_before_input assert report.api.operations["list"] == 1 @@ -354,3 +355,148 @@ async def test_measured_app_counts_only_resource_update_renders() -> None: report = recorder.report(_manifest_for_test(), (), final_digest="d") assert report.render_passes == 1 assert report.rendered_updates == 1 + + +async def test_replay_measures_list_phase_separately_from_watch_events() -> None: + """Initial LIST rows must not be counted as event-to-render samples; they + are timed as a separate LIST-to-populated-table startup phase, so replay + p95 is comparable with the live watch-only event-to-render metric.""" + profile = WorkloadProfile( + schema_version=1, + id="list-sep", + seed=186, + object_count=100, + namespace_count=10, + steady_events_per_second=10, + duration_seconds=1, + bursts=(), + failures=(), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + + # 10 scheduled watch events, and *only* those, are event-to-render samples. + assert len(scheduled_events(profile)) == 10 + assert report.event_to_render.count == 10 + assert report.rendered_updates == 10 + + # The LIST-to-populated-table and startup phases are measured explicitly. + assert report.phases.list_to_populated_table_seconds is not None + assert report.phases.list_to_populated_table_seconds >= 0.0 + assert report.phases.process_start_to_interactive_seconds is not None + assert report.phases.process_start_to_interactive_seconds >= 0.0 + + +async def test_replay_records_post_burst_drain_and_backlog_depth() -> None: + """A burst produces a measurable backlog and a post-burst drain sample.""" + profile = WorkloadProfile( + schema_version=1, + id="burst-drain", + seed=186, + object_count=50, + namespace_count=5, + steady_events_per_second=5, + duration_seconds=3, + bursts=(Burst(start_second=1, duration_seconds=1, events_per_second=40),), + failures=(), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + + assert report.phases.max_backlog_depth >= 1 + assert report.phases.post_burst_drain_seconds != () + assert report.phases.max_post_burst_drain_seconds is not None + + +def test_resolve_korvid_sha_prefers_github_sha_then_git_head() -> None: + sha = "a" * 40 + other = "b" * 40 + assert resolve_korvid_sha(env={"GITHUB_SHA": sha}, git_head=lambda: other) == sha + assert resolve_korvid_sha(env={}, git_head=lambda: other) == other + # A non-immutable / missing value resolves to None rather than a fake SHA. + assert resolve_korvid_sha(env={"GITHUB_SHA": "dev"}, git_head=lambda: None) is None + + +def test_build_manifest_records_resolved_sha() -> None: + sha = "c" * 40 + manifest = build_manifest(_manifest_profile(), korvid_sha=sha) + assert manifest.korvid_sha == sha + + +def test_build_manifest_marks_unresolved_offline_sha_as_unknown( + monkeypatch: pytest.MonkeyPatch, +) -> None: + import tests.performance.replay as replay_module + + monkeypatch.setattr(replay_module, "resolve_korvid_sha", lambda: None) + manifest = build_manifest(_manifest_profile()) + assert manifest.korvid_sha == "unknown" + + +async def test_replay_metrics_unavailable_keeps_resource_navigation_healthy() -> None: + """`metrics_unavailable` records evidence on the metrics read path while the + resource watch/render reaches the oracle digest with zero drops - proving + navigation is independent of the metrics poller.""" + profile = WorkloadProfile( + schema_version=1, + id="metrics-unavail", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="metrics_unavailable", at_event=5),), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + events = scheduled_events(profile) + oracle = summary_digest(apply_events(initial_pods(profile), events)) + + # The failing event is still delivered (not a hard fault), so no filtering. + assert report.expected_digest == oracle + assert report.final_digest == oracle + assert report.dropped_updates == 0 + assert report.api.reconnects == 0 + assert report.failures_injected["metrics_unavailable"] == 1 + # Evidence lands on the metrics read path, never the pods path. + assert "/apis/metrics.k8s.io/v1beta1/pods" in report.api.paths + + +async def test_replay_slow_logs_do_not_block_resource_progress() -> None: + """`slow_logs` records evidence on the log read path and adds no delay to + the resource schedule - resource watch/render progress is independent of log + consumption.""" + profile = WorkloadProfile( + schema_version=1, + id="slow-logs", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="slow_logs", at_event=40),), + ) + virtual_time: list[float] = [0.0] + sleep_delays: list[float] = [] + + def virtual_monotonic() -> float: + return virtual_time[0] + + async def virtual_sleep(delay: float) -> None: + sleep_delays.append(delay) + virtual_time[0] += delay + await asyncio.sleep(0) + + report = await run_replay( + profile, + ReplayOptions(time_scale=1, monotonic_fn=virtual_monotonic, async_sleep=virtual_sleep), + ) + events = scheduled_events(profile) + oracle = summary_digest(apply_events(initial_pods(profile), events)) + + assert report.expected_digest == oracle + assert report.final_digest == oracle + assert report.dropped_updates == 0 + assert report.api.reconnects == 0 + assert report.failures_injected["slow_logs"] == 1 + # A slow log stream adds no extra sleep to the resource schedule. + assert sum(sleep_delays) == pytest.approx(events[-1].offset_seconds) From 1534bebc1455b5a06a147dd7d20f7cbe02067cfb Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 12:05:56 +0900 Subject: [PATCH 29/38] fix(issue-186-review): bound az aks show call in live identity gate The fail-closed live cluster identity gate awaited the first external `az aks show` command without a timeout, so a stuck kubeconfig exec / credential plugin could hang the gate indefinitely before any client was constructed. Wrap the command_runner await in `asyncio.timeout(limits.read_connect_timeout_seconds)`, consistent with the adjacent context-host lookup and setup connects. TDD: added test_run_live_replay_bounds_the_az_aks_show_lookup mirroring the context-host bounding test (RED: hung ~31s and raised ValueError; GREEN: bounded, raises TimeoutError in <0.05s). Full tests/performance/test_live.py: 72 passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/live.py | 7 ++++--- tests/performance/test_live.py | 23 +++++++++++++++++++++++ 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/tests/performance/live.py b/tests/performance/live.py index a73098aa..ac041e17 100644 --- a/tests/performance/live.py +++ b/tests/performance/live.py @@ -552,9 +552,10 @@ async def _verify_cluster_identity( async with asyncio.timeout(limits.read_connect_timeout_seconds): hostname = await deps.context_host(context) - result = await deps.command_runner( - ["az", "aks", "show", "--ids", expected_cluster_id, "-o", "json"] - ) + async with asyncio.timeout(limits.read_connect_timeout_seconds): + result = await deps.command_runner( + ["az", "aks", "show", "--ids", expected_cluster_id, "-o", "json"] + ) if result.exit_code != 0: raise ValueError(f"az aks show failed (exit {result.exit_code}): {result.stderr.strip()}") try: diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py index 67f99ab7..a55fafe0 100644 --- a/tests/performance/test_live.py +++ b/tests/performance/test_live.py @@ -2201,6 +2201,29 @@ async def context_host(_context: str) -> str: ) +async def test_run_live_replay_bounds_the_az_aks_show_lookup() -> None: + """The first external `az aks show` call in the identity gate can hang on a + stuck credential/exec plugin; it must be bounded by the read/connect timeout, + not block the fail-closed gate forever.""" + + async def hanging_command_runner(_args: Any) -> CommandResult: + await asyncio.sleep(30) + return CommandResult(0, "{}", "") + + deps = _identity_deps(hanging_command_runner) + + with pytest.raises(TimeoutError): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(read_connect_timeout_seconds=0.05), + ) + + async def test_run_live_replay_rejects_wrong_resource_group_before_mutation() -> None: deps = _identity_deps(_ok_command_runner(resource_group="rg-production")) with pytest.raises(ValueError, match="resource group"): From 35fa7a51a8857c8f3d9fdca0938635bbc2e3f5db Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 13:32:01 +0900 Subject: [PATCH 30/38] fix: use supported az aks show args Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/cli.py | 2 +- tests/performance/live.py | 15 +++++++++++++-- tests/performance/test_live.py | 29 +++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+), 3 deletions(-) diff --git a/tests/performance/cli.py b/tests/performance/cli.py index d6f1bd04..0163aa51 100644 --- a/tests/performance/cli.py +++ b/tests/performance/cli.py @@ -168,7 +168,7 @@ def _build_parser() -> argparse.ArgumentParser: dest="expected_cluster_id", required=True, metavar="TEXT", - help="Exact AKS cluster ARM resource ID (`az aks show --ids ...`).", + help="Exact AKS cluster ARM resource ID (must exactly match the returned `az aks show` id).", ) lp.add_argument( "--run-id", dest="run_id", required=True, metavar="TEXT", help="Unique run identifier." diff --git a/tests/performance/live.py b/tests/performance/live.py index ac041e17..30532bd2 100644 --- a/tests/performance/live.py +++ b/tests/performance/live.py @@ -13,7 +13,8 @@ 1. **Cluster identity gate** (`_verify_cluster_identity`): the active kubeconfig context, its resolved API server hostname, and an independent - `az aks show --ids ` lookup must all agree before any client connects. + `az aks show` lookup for the fixed dedicated-test resource group/name must + all agree before any client connects. 2. **Ownership gate** (`_verify_ownership`): every expected namespace and every expected Pod must already carry both ownership labels (`manifests.MANAGED_BY_LABEL`/`manifests.RUN_LABEL`), and no *unexpected* @@ -554,7 +555,17 @@ async def _verify_cluster_identity( async with asyncio.timeout(limits.read_connect_timeout_seconds): result = await deps.command_runner( - ["az", "aks", "show", "--ids", expected_cluster_id, "-o", "json"] + [ + "az", + "aks", + "show", + "--resource-group", + _REQUIRED_RESOURCE_GROUP, + "--name", + _REQUIRED_CLUSTER_NAME, + "-o", + "json", + ] ) if result.exit_code != 0: raise ValueError(f"az aks show failed (exit {result.exit_code}): {result.stderr.strip()}") diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py index a55fafe0..cb68e538 100644 --- a/tests/performance/test_live.py +++ b/tests/performance/test_live.py @@ -2201,6 +2201,35 @@ async def context_host(_context: str) -> str: ) +async def test_verify_cluster_identity_uses_fixed_resource_group_and_cluster_name_lookup() -> None: + captured_args: list[list[str]] = [] + + async def recording_command_runner(args: list[str]) -> CommandResult: + captured_args.append(args) + return await _ok_command_runner()(args) + + await live._verify_cluster_identity( + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + deps=_identity_deps(recording_command_runner), + limits=LiveLimits(), + ) + + assert captured_args == [ + [ + "az", + "aks", + "show", + "--resource-group", + RESOURCE_GROUP, + "--name", + CLUSTER_NAME, + "-o", + "json", + ] + ] + + async def test_run_live_replay_bounds_the_az_aks_show_lookup() -> None: """The first external `az aks show` call in the identity gate can hang on a stuck credential/exec plugin; it must be bounded by the read/connect timeout, From 39c07d64ca987c954d1da83e23484091bff940b3 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 19:45:42 +0900 Subject: [PATCH 31/38] fix(k8s): refresh expiring exec credentials across configuration copies Long-running sessions against AKS died with HTTP 401 after ~22 minutes: kubelogin's exec credential expired and nothing re-ran the exec plugin. `load_refreshable_kube_config()` now installs a `refresh_api_key_hook` that re-runs the kubeconfig loader shortly before the exec credential expires. Because `Configuration.set_default()` and a default-constructed `ApiClient()` both deep-copy the configuration, a shared refresh closure alone is not enough - each copy owns its own `api_key`. The hook therefore tracks a shared refresh generation and re-applies the loader's cached credential to any copy that has fallen behind, so every clone converges without re-spawning the exec plugin. Refresh is serialized behind a lock, bounded by the existing probe timeout, and errors propagate instead of being swallowed. A failed refresh deliberately leaves the generation untouched so other copies still attempt a refresh rather than trusting a stale token. `connect()`, `switch_context()`, `open_pod_exec()` and the live performance mutation client now pass the active configuration explicitly to `ApiClient`/`WsApiClient` so refreshed credentials actually reach the transport. Tests: tests/k8s/test_client.py (refresh, serialization, stale-copy propagation, static-token no-op, generation-invariant on failure, connect/switch/exec wiring) and tests/performance/test_live.py::test_mutation_client_connect_uses_refreshable_kube_config Refs #186 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/korvid/k8s/client.py | 78 ++++++++++- tests/k8s/test_client.py | 232 ++++++++++++++++++++++++++++++++ tests/k8s/test_client_resize.py | 7 +- tests/performance/live.py | 8 +- tests/performance/test_live.py | 27 ++++ 5 files changed, 341 insertions(+), 11 deletions(-) diff --git a/src/korvid/k8s/client.py b/src/korvid/k8s/client.py index 6c7c1af9..21b92572 100644 --- a/src/korvid/k8s/client.py +++ b/src/korvid/k8s/client.py @@ -10,7 +10,7 @@ import re from collections.abc import AsyncIterator, Mapping, Sequence from contextlib import AbstractAsyncContextManager, asynccontextmanager -from datetime import datetime +from datetime import datetime, timedelta from typing import Any, cast from urllib.parse import quote, urlencode @@ -144,6 +144,54 @@ def list_context_names(config_file: str | None = None) -> tuple[list[str], str | #: Bound on the `:ctx` auth probe round trip (issue #36): a wedged target #: cluster must fail the switch quickly instead of hanging the flow. _PROBE_TIMEOUT = 10.0 +_EXEC_CREDENTIAL_REFRESH_SKEW = timedelta(minutes=5) +_EXEC_CREDENTIAL_GENERATION_ATTR = "_korvid_exec_credential_generation" + + +def _exec_credential_expires_soon(loader: object) -> bool: + expiry = getattr(loader, "exec_plugin_expiry", None) + if not isinstance(expiry, datetime): + return False + return expiry - _EXEC_CREDENTIAL_REFRESH_SKEW <= datetime.now(tz=expiry.tzinfo) + + +async def load_refreshable_kube_config( + *, + context: str | None, + client_configuration: k8s_client.Configuration, + persist_config: bool, +) -> None: + """Load kubeconfig and refresh expiring exec credentials before API calls.""" + loader = await k8s_config.load_kube_config( + context=context, + client_configuration=client_configuration, + persist_config=persist_config, + ) + if not isinstance(getattr(loader, "exec_plugin_expiry", None), datetime): + return + + refresh_lock = asyncio.Lock() + refresh_generation = 0 + setattr(client_configuration, _EXEC_CREDENTIAL_GENERATION_ATTR, refresh_generation) + + async def _refresh(configuration: k8s_client.Configuration) -> None: + nonlocal refresh_generation + configuration_generation = getattr(configuration, _EXEC_CREDENTIAL_GENERATION_ATTR, 0) + if configuration_generation == refresh_generation and not _exec_credential_expires_soon( + loader + ): + return + async with refresh_lock: + configuration_generation = getattr(configuration, _EXEC_CREDENTIAL_GENERATION_ATTR, 0) + if configuration_generation < refresh_generation: + await asyncio.wait_for(loader.load_and_set(configuration), _PROBE_TIMEOUT) + setattr(configuration, _EXEC_CREDENTIAL_GENERATION_ATTR, refresh_generation) + elif _exec_credential_expires_soon(loader): + await asyncio.wait_for(loader.load_and_set(configuration), _PROBE_TIMEOUT) + refresh_generation += 1 + setattr(configuration, _EXEC_CREDENTIAL_GENERATION_ATTR, refresh_generation) + + client_configuration.refresh_api_key_hook = _refresh class KubeClient(ReadOps, WriteOps): @@ -244,8 +292,14 @@ def _object_summary(self, meta: ResourceMeta, manifest: dict[str, Any]) -> Gener return dataclasses.replace(summary, custom=evaluate_all(columns, manifest)) async def connect(self, context: str | None = None) -> None: - await k8s_config.load_kube_config(context=context) - self._api = k8s_client.ApiClient() + configuration = k8s_client.Configuration() + await load_refreshable_kube_config( + context=context, + client_configuration=configuration, + persist_config=True, + ) + k8s_client.Configuration.set_default(configuration) + self._api = k8s_client.ApiClient(configuration) self._core_v1 = k8s_client.CoreV1Api(self._api) # A new connection may target a different cluster; discard any # capability discovered against the previous one. @@ -308,8 +362,17 @@ async def switch_context(self, context: str | None) -> None: already-torn-down session forever. """ old_api = self._api - await asyncio.wait_for(k8s_config.load_kube_config(context=context), _PROBE_TIMEOUT) - self._api = k8s_client.ApiClient() + configuration = k8s_client.Configuration() + await asyncio.wait_for( + load_refreshable_kube_config( + context=context, + client_configuration=configuration, + persist_config=True, + ), + _PROBE_TIMEOUT, + ) + k8s_client.Configuration.set_default(configuration) + self._api = k8s_client.ApiClient(configuration) self._core_v1 = k8s_client.CoreV1Api(self._api) # Per-connection caches describe the previous cluster. self._pod_resize_supported = None @@ -375,12 +438,13 @@ def open_pod_exec( dedicated ``WsApiClient`` is created per session — it shares the kubeconfig ``connect()`` loaded — and closed with the session. """ - if self._core_v1 is None: + if self._core_v1 is None or self._api is None: raise RuntimeError("connect() first") + configuration = self._api.configuration @asynccontextmanager async def _session() -> AsyncIterator[Any]: - ws_api = WsApiClient() + ws_api = WsApiClient(configuration) try: core = k8s_client.CoreV1Api(ws_api) kwargs: dict[str, Any] = { diff --git a/tests/k8s/test_client.py b/tests/k8s/test_client.py index 0dc590f1..b0e14019 100644 --- a/tests/k8s/test_client.py +++ b/tests/k8s/test_client.py @@ -1,3 +1,6 @@ +import asyncio +from copy import deepcopy +from datetime import UTC, datetime, timedelta from pathlib import Path from typing import Any from unittest.mock import AsyncMock, MagicMock, patch @@ -15,6 +18,195 @@ from korvid.k8s.telemetry import ReadTelemetryEvent +async def test_load_refreshable_kube_config_refreshes_expired_exec_token( + monkeypatch: pytest.MonkeyPatch, +) -> None: + configuration = k8s_client.Configuration() + configuration.api_key["BearerToken"] = "stale-token" + loader = MagicMock() + loader.exec_plugin_expiry = datetime.now(UTC) - timedelta(seconds=1) + + async def _refresh(target: k8s_client.Configuration) -> None: + target.api_key["BearerToken"] = "fresh-token" + loader.exec_plugin_expiry = datetime.now(UTC) + timedelta(hours=1) + + loader.load_and_set = AsyncMock(side_effect=_refresh) + load_kube_config = AsyncMock(return_value=loader) + monkeypatch.setattr(k8s_config, "load_kube_config", load_kube_config) + + await client_mod.load_refreshable_kube_config( + context="aks", + client_configuration=configuration, + persist_config=False, + ) + + assert await configuration.get_api_key_with_prefix("BearerToken") == "fresh-token" + loader.load_and_set.assert_awaited_once_with(configuration) + + +async def test_load_refreshable_kube_config_serializes_concurrent_refresh( + monkeypatch: pytest.MonkeyPatch, +) -> None: + configuration = k8s_client.Configuration() + configuration.api_key["BearerToken"] = "stale-token" + loader = MagicMock() + loader.exec_plugin_expiry = datetime.now(UTC) - timedelta(seconds=1) + in_flight = 0 + max_in_flight = 0 + + async def _refresh(target: k8s_client.Configuration) -> None: + nonlocal in_flight, max_in_flight + in_flight += 1 + max_in_flight = max(max_in_flight, in_flight) + try: + await asyncio.sleep(0) + target.api_key["BearerToken"] = "fresh-token" + loader.exec_plugin_expiry = datetime.now(UTC) + timedelta(hours=1) + finally: + in_flight -= 1 + + loader.load_and_set = AsyncMock(side_effect=_refresh) + monkeypatch.setattr( + k8s_config, + "load_kube_config", + AsyncMock(return_value=loader), + ) + await client_mod.load_refreshable_kube_config( + context="aks", + client_configuration=configuration, + persist_config=False, + ) + + tokens = await asyncio.gather( + configuration.get_api_key_with_prefix("BearerToken"), + configuration.get_api_key_with_prefix("BearerToken"), + ) + + assert len(tokens) == 2 + assert all(token == "fresh-token" for token in tokens) + assert max_in_flight == 1 + loader.load_and_set.assert_awaited_once_with(configuration) + + +async def test_load_refreshable_kube_config_updates_stale_configuration_copy( + monkeypatch: pytest.MonkeyPatch, +) -> None: + configuration = k8s_client.Configuration() + configuration.api_key["BearerToken"] = "stale-token" + loader = MagicMock() + loader.exec_plugin_expiry = datetime.now(UTC) - timedelta(seconds=1) + + async def _refresh(target: k8s_client.Configuration) -> None: + target.api_key["BearerToken"] = "fresh-token" + loader.exec_plugin_expiry = datetime.now(UTC) + timedelta(hours=1) + + loader.load_and_set = AsyncMock(side_effect=_refresh) + monkeypatch.setattr( + k8s_config, + "load_kube_config", + AsyncMock(return_value=loader), + ) + await client_mod.load_refreshable_kube_config( + context="aks", + client_configuration=configuration, + persist_config=False, + ) + first_copy = deepcopy(configuration) + waiting_copy = deepcopy(configuration) + + assert await first_copy.get_api_key_with_prefix("BearerToken") == "fresh-token" + assert await waiting_copy.get_api_key_with_prefix("BearerToken") == "fresh-token" + assert loader.load_and_set.await_count == 2 + + +async def test_load_refreshable_kube_config_leaves_static_tokens_alone( + monkeypatch: pytest.MonkeyPatch, +) -> None: + configuration = k8s_client.Configuration() + configuration.api_key["BearerToken"] = "static-token" + loader = MagicMock(spec=["load_and_set"]) + loader.load_and_set = AsyncMock() + monkeypatch.setattr( + k8s_config, + "load_kube_config", + AsyncMock(return_value=loader), + ) + + await client_mod.load_refreshable_kube_config( + context="static", + client_configuration=configuration, + persist_config=False, + ) + + assert configuration.refresh_api_key_hook is None + assert await configuration.get_api_key_with_prefix("BearerToken") == "static-token" + loader.load_and_set.assert_not_awaited() + + +async def test_load_refreshable_kube_config_keeps_generation_when_refresh_fails( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A failed refresh must not mark other configuration copies as current.""" + configuration = k8s_client.Configuration() + configuration.api_key["BearerToken"] = "stale-token" + loader = MagicMock() + loader.exec_plugin_expiry = datetime.now(UTC) - timedelta(seconds=1) + attempts: list[k8s_client.Configuration] = [] + + async def _refresh(target: k8s_client.Configuration) -> None: + attempts.append(target) + if len(attempts) == 1: + raise ConnectionError("kubelogin unavailable") + target.api_key["BearerToken"] = "fresh-token" + loader.exec_plugin_expiry = datetime.now(UTC) + timedelta(hours=1) + + loader.load_and_set = AsyncMock(side_effect=_refresh) + monkeypatch.setattr( + k8s_config, + "load_kube_config", + AsyncMock(return_value=loader), + ) + await client_mod.load_refreshable_kube_config( + context="aks", + client_configuration=configuration, + persist_config=False, + ) + failing_copy = deepcopy(configuration) + other_copy = deepcopy(configuration) + + with pytest.raises(ConnectionError, match="kubelogin unavailable"): + await failing_copy.get_api_key_with_prefix("BearerToken") + + assert await other_copy.get_api_key_with_prefix("BearerToken") == "fresh-token" + assert await failing_copy.get_api_key_with_prefix("BearerToken") == "fresh-token" + + +async def test_connect_uses_refreshable_kube_config(monkeypatch: pytest.MonkeyPatch) -> None: + load_refreshable = AsyncMock() + api = MagicMock() + api_factory = MagicMock(return_value=api) + core_v1_factory = MagicMock() + set_default = MagicMock() + monkeypatch.setattr(client_mod, "load_refreshable_kube_config", load_refreshable) + monkeypatch.setattr(k8s_config, "load_kube_config", AsyncMock()) + monkeypatch.setattr(k8s_client, "ApiClient", api_factory) + monkeypatch.setattr(k8s_client, "CoreV1Api", core_v1_factory) + monkeypatch.setattr(k8s_client.Configuration, "set_default", set_default) + + client = KubeClient() + await client.connect("aks") + + load_refreshable.assert_awaited_once() + call = load_refreshable.await_args + assert call is not None + assert call.kwargs["context"] == "aks" + assert call.kwargs["persist_config"] is True + configuration = call.kwargs["client_configuration"] + set_default.assert_called_once_with(configuration) + api_factory.assert_called_once_with(configuration) + core_v1_factory.assert_called_once_with(api) + + def _pod(name: str, ns: str = "default") -> dict[str, Any]: return { "metadata": {"name": name, "namespace": ns}, @@ -1499,10 +1691,14 @@ def test_requires_connect(self) -> None: async def test_opens_ws_with_exec_params(self, monkeypatch: pytest.MonkeyPatch) -> None: sentinel_ws = object() + sentinel_configuration = object() closed: list[bool] = [] captured: dict[str, object] = {} class FakeWsApi: + def __init__(self, configuration: object) -> None: + captured["configuration"] = configuration + async def close(self) -> None: closed.append(True) @@ -1528,12 +1724,15 @@ async def connect_get_namespaced_pod_exec( monkeypatch.setattr(client_mod, "WsApiClient", FakeWsApi) monkeypatch.setattr(k8s_client, "CoreV1Api", FakeCoreWs) kube = client_mod.KubeClient() + kube._api = MagicMock() + kube._api.configuration = sentinel_configuration kube._core_v1 = object() # type: ignore[assignment] # connected marker async with kube.open_pod_exec( "prod", "api-0", "app", ["tar", "cf", "-"], stdin=False ) as ws: assert ws is sentinel_ws + assert captured["configuration"] is sentinel_configuration assert captured["name"] == "api-0" assert captured["namespace"] == "prod" assert captured["command"] == ["tar", "cf", "-"] @@ -1549,6 +1748,9 @@ async def test_omits_container_when_none(self, monkeypatch: pytest.MonkeyPatch) captured: dict[str, object] = {} class FakeWsApi: + def __init__(self, configuration: object) -> None: + pass + async def close(self) -> None: return None @@ -1572,6 +1774,7 @@ async def connect_get_namespaced_pod_exec( monkeypatch.setattr(client_mod, "WsApiClient", FakeWsApi) monkeypatch.setattr(k8s_client, "CoreV1Api", FakeCoreWs) kube = client_mod.KubeClient() + kube._api = MagicMock() kube._core_v1 = object() # type: ignore[assignment] # connected marker async with kube.open_pod_exec("ns", "p", None, ["tar"], stdin=True): @@ -1583,6 +1786,9 @@ async def test_ws_api_closed_on_error(self, monkeypatch: pytest.MonkeyPatch) -> closed: list[bool] = [] class FakeWsApi: + def __init__(self, configuration: object) -> None: + pass + async def close(self) -> None: closed.append(True) @@ -1596,6 +1802,7 @@ async def connect_get_namespaced_pod_exec(self, *a: object, **k: object) -> obje monkeypatch.setattr(client_mod, "WsApiClient", FakeWsApi) monkeypatch.setattr(k8s_client, "CoreV1Api", FakeCoreWs) kube = client_mod.KubeClient() + kube._api = MagicMock() kube._core_v1 = object() # type: ignore[assignment] # connected marker with pytest.raises(OSError, match="boom"): @@ -1918,6 +2125,31 @@ async def fake_load(**kwargs: Any) -> None: class TestSwitchContext: + async def test_uses_refreshable_kube_config(self, monkeypatch: pytest.MonkeyPatch) -> None: + load_refreshable = AsyncMock() + api = MagicMock() + api_factory = MagicMock(return_value=api) + core_v1_factory = MagicMock() + set_default = MagicMock() + monkeypatch.setattr(client_mod, "load_refreshable_kube_config", load_refreshable) + monkeypatch.setattr(k8s_config, "load_kube_config", AsyncMock()) + monkeypatch.setattr(k8s_client, "ApiClient", api_factory) + monkeypatch.setattr(k8s_client, "CoreV1Api", core_v1_factory) + monkeypatch.setattr(k8s_client.Configuration, "set_default", set_default) + + kube = KubeClient() + await kube.switch_context("ctx-b") + + load_refreshable.assert_awaited_once() + call = load_refreshable.await_args + assert call is not None + assert call.kwargs["context"] == "ctx-b" + assert call.kwargs["persist_config"] is True + configuration = call.kwargs["client_configuration"] + set_default.assert_called_once_with(configuration) + api_factory.assert_called_once_with(configuration) + core_v1_factory.assert_called_once_with(api) + async def test_swaps_connection_and_closes_old(self, monkeypatch: pytest.MonkeyPatch) -> None: load_calls: list[dict[str, Any]] = [] diff --git a/tests/k8s/test_client_resize.py b/tests/k8s/test_client_resize.py index d27c2091..712218f9 100644 --- a/tests/k8s/test_client_resize.py +++ b/tests/k8s/test_client_resize.py @@ -5,6 +5,7 @@ from unittest.mock import AsyncMock, MagicMock, patch import pytest +from kubernetes_asyncio import client as k8s_client from korvid.k8s.client import KubeClient @@ -155,8 +156,12 @@ async def test_connect_resets_resize_discovery_cache( async def fake_load(*args: object, **kwargs: object) -> None: return None + def fake_api(configuration: k8s_client.Configuration) -> object: + assert isinstance(configuration, k8s_client.Configuration) + return object() + monkeypatch.setattr("korvid.k8s.client.k8s_config.load_kube_config", fake_load) - monkeypatch.setattr("korvid.k8s.client.k8s_client.ApiClient", lambda: object()) + monkeypatch.setattr("korvid.k8s.client.k8s_client.ApiClient", fake_api) monkeypatch.setattr("korvid.k8s.client.k8s_client.CoreV1Api", lambda api: object()) client = KubeClient() client._pod_resize_supported = True diff --git a/tests/performance/live.py b/tests/performance/live.py index 30532bd2..2a561c68 100644 --- a/tests/performance/live.py +++ b/tests/performance/live.py @@ -79,7 +79,7 @@ from korvid.core.config import KorvidConfig from korvid.core.store import ALL_NAMESPACES, ResourceStore from korvid.core.watch import WatchManager, WatchSource -from korvid.k8s.client import KubeClient, resolve_context_name +from korvid.k8s.client import KubeClient, load_refreshable_kube_config, resolve_context_name from korvid.k8s.discovery import ResourceMeta from korvid.k8s.errors import ApiStatusError from korvid.k8s.models import GenericSummary, PodSummary @@ -418,8 +418,10 @@ async def connect(self) -> None: if self._core_v1 is not None: return configuration = k8s_client.Configuration() - await k8s_config.load_kube_config( - context=self._context, client_configuration=configuration, persist_config=False + await load_refreshable_kube_config( + context=self._context, + client_configuration=configuration, + persist_config=False, ) self._api = k8s_client.ApiClient(configuration) self._core_v1 = k8s_client.CoreV1Api(self._api) diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py index cb68e538..b4ad9521 100644 --- a/tests/performance/test_live.py +++ b/tests/performance/test_live.py @@ -14,9 +14,11 @@ from collections import Counter from collections.abc import AsyncGenerator, AsyncIterator, Awaitable, Callable from typing import Any +from unittest.mock import AsyncMock, MagicMock import pytest from kubernetes_asyncio import client as k8s_client +from kubernetes_asyncio import config as k8s_config from multidict import CIMultiDict, CIMultiDictProxy from korvid.core.store import Summary @@ -1482,6 +1484,31 @@ async def patch_namespaced_pod(self, *_args: object, **_kwargs: object) -> None: assert caught.value.retry_after_seconds == expected +async def test_mutation_client_connect_uses_refreshable_kube_config( + monkeypatch: pytest.MonkeyPatch, +) -> None: + load_refreshable = AsyncMock() + api = MagicMock() + api_factory = MagicMock(return_value=api) + core_v1_factory = MagicMock() + monkeypatch.setattr(live, "load_refreshable_kube_config", load_refreshable, raising=False) + monkeypatch.setattr(k8s_config, "load_kube_config", AsyncMock()) + monkeypatch.setattr(k8s_client, "ApiClient", api_factory) + monkeypatch.setattr(k8s_client, "CoreV1Api", core_v1_factory) + + client = live._KubeMutationClient(CONTEXT, RUN_ID) + await client.connect() + + load_refreshable.assert_awaited_once() + call = load_refreshable.await_args + assert call is not None + assert call.kwargs["context"] == CONTEXT + assert call.kwargs["persist_config"] is False + configuration = call.kwargs["client_configuration"] + api_factory.assert_called_once_with(configuration) + core_v1_factory.assert_called_once_with(api) + + async def test_mutation_retry_respects_server_hint_with_an_explicit_delay_bound() -> None: class _ThrottleOnce: def __init__(self) -> None: From 48103f95b01acc5f5d6f99e65cf572cc47c99ab3 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 20:55:37 +0900 Subject: [PATCH 32/38] test(ui): wait for the node-shell audit outcome, not the notification `test_node_shell_cleanup_failure_warns_and_audits` waited on the failure notification and then read the audit log after the app context closed. The outcome entry is appended *after* that notification, on a separate `asyncio.to_thread` hop, so a loaded full-suite run could tear the app down first and find only the `intent` entry. It now waits for the audit record itself. This originally also carried a `_rendered_cells` diff cache for `ResourceTable`; #209 landed the same optimisation on main as `_emitted`, with a memo and pruning on top, so that part is dropped in favour of the upstream implementation. Refs #186 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/ui/test_node_shell.py | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/tests/ui/test_node_shell.py b/tests/ui/test_node_shell.py index 9b1fafaa..8dd68c32 100644 --- a/tests/ui/test_node_shell.py +++ b/tests/ui/test_node_shell.py @@ -283,10 +283,18 @@ async def test_node_shell_cleanup_failure_warns_and_audits(tmp_path: Path) -> No await until(pilot, lambda: isinstance(app.screen, ConfirmScreen), label="dialog") await pilot.press("y") - def _warned() -> bool: - return any(DBG_POD in n.message for n in app._notifications) - - await until(pilot, _warned, label="cleanup failure notification") + def _audited() -> bool: + # The outcome entry is appended after the failure notification + # (a separate `asyncio.to_thread` hop), so waiting on the + # notification alone races the audit write at teardown. + if not audit_path.exists(): + return False + records = [json.loads(ln) for ln in audit_path.read_text().splitlines()] + shells = [e for e in records if e["action"] == "node-shell"] + return bool(shells) and "cleanup failed for" in shells[-1]["outcome"] + + await until(pilot, _audited, label="cleanup failure audited") + assert any(DBG_POD in n.message for n in app._notifications) entries = [json.loads(ln) for ln in audit_path.read_text().splitlines()] last = [e for e in entries if e["action"] == "node-shell"][-1] assert f"cleanup failed for: {DBG_POD}" in last["outcome"] From 357ab4fd5af7ae3d28228d8cdbc0c16b6d209d9a Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 21:52:02 +0900 Subject: [PATCH 33/38] fix(perf-cli): fail live runs on failed UI scenarios and aliased artifacts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two review findings on the live qualification command: `drive_ui_scenarios` deliberately records a key sequence that never reached its target state as `ScenarioResult(ok=False)` instead of raising, so the scenario outcomes never reached the exit status. A live run whose filter, split-pane, describe, or multi-log evidence failed still reported success — exactly the case where the qualification has nothing to show. `replay-live` now folds failed scenarios into its exit status and names them on stderr. Artifact distinctness compared raw path strings, so aliases such as `sub/../run.json` and `run.json` passed the check while resolving to the same file, letting a later artifact write silently destroy an earlier one. Distinctness is now decided on resolved paths. The design doc records both: failed UI-at-scale scenarios join the hard budget table at 0. Tests: tests/performance/test_cli.py:: test_cli_replay_live_fails_when_a_ui_scenario_did_not_pass test_cli_replay_live_rejects_artifact_paths_that_alias_one_file Refs #186 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...luster-performance-qualification-design.md | 6 ++ tests/performance/cli.py | 15 ++++- tests/performance/test_cli.py | 64 +++++++++++++++++++ 3 files changed, 84 insertions(+), 1 deletion(-) diff --git a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md index 1cc8badf..32249081 100644 --- a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md +++ b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md @@ -299,6 +299,12 @@ These budgets define usable behavior for the live 1,000-Pod profile: | Backlog drain after a 100 events/s burst | <= 3 seconds | | Post-warm-up RSS slope over 30 minutes | <= 1 MiB/minute | | Peak RSS at 1,000 Pods | <= 512 MiB | +| Failed UI-at-scale scenarios | 0 | + +`drive_ui_scenarios` records a key sequence that never reached its target state +as `ScenarioResult(ok=False)` rather than raising, so `replay-live` folds those +outcomes into its exit status: a run that produced no UI-at-scale evidence fails +instead of reporting success. The final document records both observed values and budgets. A budget may be changed only with measured rationale in review; baseline updates never silently diff --git a/tests/performance/cli.py b/tests/performance/cli.py index 0163aa51..6edcdeac 100644 --- a/tests/performance/cli.py +++ b/tests/performance/cli.py @@ -410,7 +410,10 @@ def _validate_live_artifacts(args: argparse.Namespace, *, run_id: str) -> str | f"({', '.join(_LIVE_ARTIFACT_FLAGS.values())}); missing: {', '.join(missing)}" ) paths = {attr: Path(getattr(args, attr)) for attr in _LIVE_ARTIFACT_FLAGS} - if len({str(path) for path in paths.values()}) != len(paths): + # Distinctness is decided on resolved paths: `sub/../run.json` and + # `run.json` are different strings but the same file, and the second write + # would silently destroy the first artifact. + if len({path.resolve() for path in paths.values()}) != len(paths): return "the four live artifacts must be four distinct destinations" for attr, path in paths.items(): if run_id not in path.name: @@ -479,6 +482,16 @@ def _execute_live_replay( if _write_outputs(args, replay): return 1 + failed_scenarios = [scenario.name for scenario in replay.ui_scenarios if not scenario.ok] + if failed_scenarios: + # `drive_ui_scenarios` records a key sequence that never reached its + # target state as `ok=False` instead of raising, so without this the + # run would "pass" with no UI-at-scale evidence behind it. + print( + "error: UI-at-scale scenarios did not pass: " + ", ".join(failed_scenarios), + file=sys.stderr, + ) + return 1 if replay.dropped_updates > 0 or replay.expected_digest != replay.final_digest: return 1 return 0 diff --git a/tests/performance/test_cli.py b/tests/performance/test_cli.py index 69f14a4d..f6340c50 100644 --- a/tests/performance/test_cli.py +++ b/tests/performance/test_cli.py @@ -19,6 +19,7 @@ PhaseSummary, ProcessSummary, RunManifest, + ScenarioResult, ) from tests.performance.profile import FailureInjection, WorkloadProfile from tests.performance.replay import ReplayOptions, ReplayReport @@ -912,3 +913,66 @@ def test_cli_replay_live_help_points_at_the_live_qualification_profile( help_text = capsys.readouterr().out assert "tests/performance/profiles/aks-live-1k.json" in help_text assert Path("tests/performance/profiles/aks-live-1k.json").exists() + + +async def fake_live_failed_ui_scenario( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, +) -> ReplayReport: + """Live replay whose churn and digests are clean but whose UI-at-scale + evidence is not: the split-pane scenario did not reach its target state.""" + report = await fake_run_replay(profile, options) + return replace( + report, + ui_scenarios=( + ScenarioResult(name="filter", latency_seconds=0.4, ok=True), + ScenarioResult(name="split_pane", latency_seconds=9.9, ok=False), + ), + ) + + +def test_cli_replay_live_fails_when_a_ui_scenario_did_not_pass( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """`drive_ui_scenarios` records a failed key sequence as `ok=False` instead + of raising, so the live command must fold those outcomes into its exit + status — otherwise a qualification "passes" with no UI-at-scale evidence.""" + monkeypatch.setattr(cli, "run_live_replay", fake_live_failed_ui_scenario) + + exit_code = cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + *_live_artifacts(tmp_path), + ] + ) + + assert exit_code == 1 + assert "split_pane" in capsys.readouterr().err + + +def test_cli_replay_live_rejects_artifact_paths_that_alias_one_file( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """Distinctness must be decided on resolved paths: `sub/../aks186-live.json` + and `aks186-live.json` are different strings but the same file, so the + second artifact write would silently destroy the first.""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + (tmp_path / "sub").mkdir() + artifacts = _live_artifacts(tmp_path) + artifacts[3] = str(tmp_path / "sub" / ".." / "aks186-live.json") + + exit_code = cli.main( + ["replay-live", "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, *artifacts] + ) + + assert exit_code == 1 + assert "distinct destinations" in capsys.readouterr().err + assert calls == [] From 2c0d33c0a507ce6c0ef673e7b7bc8a76be2618ca Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 22:53:56 +0900 Subject: [PATCH 34/38] fix(perf-live): make UI-at-scale scenarios prove what they claim Review round on PR #202 found the UI-at-scale evidence was hollow: a scenario was marked `ok=True` whenever `pilot.press` returned, without checking any resulting state. In the live wiring that meant three of the six scenarios could not have done anything and still "passed": - `describe` and `multi_log` bail out with an "unavailable" warning because no manifest/log provider was wired; - the namespace toggle was a no-op, because the configured namespace was already `ALL_NAMESPACES`, so `0` navigated to the scope it was on. None of those raises. Combined with the previous commit folding scenario outcomes into the exit status, a run could have "passed" on no evidence. Each scenario is now an ordered list of steps that each declare the observable state they must reach (filter pattern, active sort, current scope, pane count, describe screen, log pane) and wait for it; a step that does not get there records `ok=False`. The remaining steps still run so the workspace is restored and the digest/row-count convergence checks stay intact. To make the scenarios reachable the live app now gets read-only providers on the harness connection - a pods-only `get_manifest` and the read client's `stream_logs` - plus a favorite namespace so `1`/`0` really scopes down to a seeded namespace and back. That scope change restarts the application watch, which the happy-path test now pins explicitly: three watches, all still cluster-wide, because `make_live_watch_source` keeps the read pinned to the owned namespace set regardless of UI scope. Two further findings from the same round: - The live churn driver never marked burst boundaries, so every live report left `post_burst_drain_seconds` empty and printed "n/a" for the published <=3-second burst-drain budget. It now marks each boundary on the same clock the deterministic driver uses. - `load_profile` accepted duplicate `at_event` positions, which `run_replay` silently collapses into one injection while the profile hash and report still claim both. Duplicates are now rejected at load. Also extends the earlier node-shell audit-race fix: four more tests waited on a notification and then read an audit outcome that is appended afterwards on a separate `asyncio.to_thread` hop. They now share one helper that waits for the outcome entry itself. Tests: tests/performance/test_live.py:: test_ui_scenarios_are_not_marked_ok_when_the_app_state_never_changes test_run_live_replay_times_the_post_burst_drain tests/performance/test_profile.py:: test_load_profile_rejects_duplicate_failure_event_positions Refs #186 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/live.py | 237 +++++++++++++++++++++++++----- tests/performance/profile.py | 8 + tests/performance/test_live.py | 178 ++++++++++++++++++++-- tests/performance/test_profile.py | 16 ++ tests/ui/test_node_shell.py | 53 ++++--- 5 files changed, 424 insertions(+), 68 deletions(-) diff --git a/tests/performance/live.py b/tests/performance/live.py index 2a561c68..0369d78e 100644 --- a/tests/performance/live.py +++ b/tests/performance/live.py @@ -84,6 +84,7 @@ from korvid.k8s.errors import ApiStatusError from korvid.k8s.models import GenericSummary, PodSummary from korvid.k8s.telemetry import ReadTelemetry +from korvid.ui.widgets.describe_screen import DescribeScreen from korvid.ui.widgets.resource_table import ResourceTable from tests.performance import manifests from tests.performance.metrics import ( @@ -92,7 +93,7 @@ NodePoolInfo, ProcessSampler, ) -from tests.performance.profile import WorkloadProfile, validate_profile +from tests.performance.profile import WorkloadProfile, burst_end_offsets, validate_profile from tests.performance.replay import ( MeasuredKorvidApp, ReplayOptions, @@ -107,6 +108,9 @@ #: because Namespaces are cluster-scoped (`namespaced=False`). _NAMESPACES_META = ResourceMeta("Namespace", "namespaces", "", "v1", False) +#: Pods, for the read-only `describe` provider the UI-at-scale scenarios need. +_PODS_META = ResourceMeta("Pod", "pods", "", "v1", True) + #: Live topology is pinned to the aks-1k profile shape (issue #186 Task 8): #: exactly 20 namespaces of 50 Pods each, matching `seed-manifests`'s output. _REQUIRED_OBJECT_COUNT = 1000 @@ -260,6 +264,12 @@ async def list_pods(self, namespace: str) -> list[PodSummary]: ... def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSummary]]: ... + async def get_object( + self, meta: ResourceMeta, namespace: str | None, name: str + ) -> dict[str, Any]: ... + + def stream_logs(self, *args: Any, **kwargs: Any) -> AsyncIterator[Any]: ... + class MutationClient(Protocol): """Issues one guarded, metadata-only mutation; production talks JSON-Patch @@ -999,6 +1009,8 @@ async def drive_live_churn( options: ReplayOptions, progress: ChurnProgress, limits: LiveLimits, + profile: WorkloadProfile, + recorder: BenchmarkRecorder, ) -> None: """Drive guarded churn at wall-clock time with explicit bounded concurrency. @@ -1040,12 +1052,26 @@ async def _run(namespace: str, name: str, uid: str, tick: str) -> None: try: async with asyncio.TaskGroup() as group: + # Burst-end offsets (absolute seconds) mark the moment each burst's + # window closes, so the post-burst backlog drain can be timed on the + # same real-clock axis the render pass records on — exactly as the + # deterministic replay driver does. Without this the live report + # leaves `post_burst_drain_seconds` empty and the published + # burst-drain budget cannot be evaluated. + burst_ends = burst_end_offsets(profile) + next_burst = 0 for event in events: elapsed = now() - start delay = event.offset_seconds * options.time_scale - elapsed if delay > 0: await sleep(delay) + while ( + next_burst < len(burst_ends) and event.offset_seconds >= burst_ends[next_burst] + ): + recorder.mark_burst_end(monotonic()) + next_burst += 1 + namespace, name = live_object_identity(run_id, namespace_count, event.object_index) current = live_state.get((namespace, name)) if current is None: @@ -1094,23 +1120,134 @@ def _store_digest(store: ResourceStore) -> str: #: pilot key sequence driven during active churn. Sequences are self-restoring: #: they return the workspace to a single pane showing all owned rows so the #: post-churn row-count and digest-convergence checks still see 1,000 rows. -_UI_SCENARIOS: tuple[tuple[str, tuple[str, ...]], ...] = ( - # Filter to a substring present in every seeded Pod name ("bench-*"), so the - # filter exercises the real path without dropping any rows. - ("filter", ("slash", "b", "e", "n", "c", "h", "enter")), - # Sort by age (a metrics-free column) and back is unnecessary; sorting keeps - # every row visible. - ("sort", ("A",)), - # Namespace switch: scope to the highlighted row's namespace, then back to - # all namespaces so the full 1,000-row topology is restored. - ("namespace_switch", ("0", "0")), - # Split the workspace into two panes, then close the new pane. - ("split_pane", ("ctrl+w", "v", "ctrl+w", "q")), - # Describe the highlighted resource, then dismiss. - ("describe", ("d", "escape")), - # Multi-log the highlighted resource, then dismiss. - ("multi_log", ("L", "escape")), -) +#: +#: Every step declares the observable state it must reach. `pilot.press` +#: returning is not evidence: `describe` and `multi_log` bail out with an +#: "unavailable" warning when no manifest/log provider is wired, and the +#: all-namespaces toggle is a no-op while the configured namespace already is +#: `ALL_NAMESPACES` — none of which raises. Without the state check a run would +#: report UI-at-scale evidence it never gathered. + + +@dataclass(frozen=True) +class _UIStep: + """One key sequence plus the observable app state it must reach.""" + + keys: tuple[str, ...] + label: str + reached: Callable[[Any], bool] + + +@dataclass(frozen=True) +class _UIScenario: + """One named UI-at-scale scenario as an ordered list of verified steps.""" + + name: str + steps: tuple[_UIStep, ...] + + +#: Bound on one step reaching its target state. Generous on purpose: the step +#: competes with full-rate churn on the same event loop. +_UI_STEP_TIMEOUT = 30.0 + + +def _ui_scenarios(scoped_namespace: str) -> tuple[_UIScenario, ...]: + """Build the scenario table; the namespace scenario needs a real seeded + namespace to scope down to (and back out of).""" + return ( + # Filter to a substring present in every seeded Pod name ("bench-*"), so + # the filter exercises the real path without dropping any rows, then + # clear it so the bar releases focus and the workspace is restored. + _UIScenario( + "filter", + ( + _UIStep( + ("slash", "b", "e", "n", "c", "h"), + "filter applied", + lambda app: app.filter_pattern == "bench", + ), + _UIStep( + ("escape",), + "filter cleared", + lambda app: app.filter_pattern == "", + ), + ), + ), + # Sorting by age (a metrics-free column) keeps every row visible. + _UIScenario( + "sort", + ( + _UIStep( + ("A",), + "age sort active", + lambda app: app._sorts.get("pods") is not None, + ), + ), + ), + # Scope to one seeded namespace (favorite key 1), then back to all + # namespaces so the full 1,000-row topology is restored. + _UIScenario( + "namespace_switch", + ( + _UIStep( + ("1",), + "scoped to one namespace", + lambda app: app.current_scope == scoped_namespace, + ), + _UIStep( + ("0",), + "restored to all namespaces", + lambda app: app.current_scope == ALL_NAMESPACES, + ), + ), + ), + _UIScenario( + "split_pane", + ( + _UIStep(("ctrl+w", "v"), "second pane open", lambda app: len(app._panes) == 2), + _UIStep(("ctrl+w", "q"), "back to one pane", lambda app: len(app._panes) == 1), + ), + ), + _UIScenario( + "describe", + ( + _UIStep( + ("d",), + "describe screen open", + lambda app: isinstance(app.screen, DescribeScreen), + ), + _UIStep( + ("escape",), + "describe dismissed", + lambda app: not isinstance(app.screen, DescribeScreen), + ), + ), + ), + _UIScenario( + "multi_log", + ( + _UIStep(("L",), "log pane open", lambda app: bool(app._log_pane.display)), + _UIStep(("escape",), "log pane closed", lambda app: not app._log_pane.display), + ), + ), + ) + + +def _describe_provider( + kube: KubeReadClient, +) -> Callable[[str, str | None, str], Awaitable[dict[str, Any]]]: + """Read-only manifest fetcher for the `describe` UI-at-scale scenario. + + Only Pods are reachable in the live workspace, so anything else is a + programmer error rather than a cluster read. + """ + + async def get_manifest(kind: str, namespace: str | None, name: str) -> dict[str, Any]: + if kind != _PODS_META.plural: + raise ValueError(f"live qualification only describes pods, not {kind!r}") + return await kube.get_object(_PODS_META, namespace, name) + + return get_manifest async def drive_ui_scenarios( @@ -1118,26 +1255,37 @@ async def drive_ui_scenarios( recorder: BenchmarkRecorder, *, now: Callable[[], float], + app: Any, + scoped_namespace: str, ) -> None: """Drive the scoped UI-at-scale scenarios through the real Textual pilot. Each scenario is a real key sequence issued against the running app during - active churn; its latency and completion outcome are recorded. Every - scenario is read-only navigation - no scenario writes, deletes, or drains - - so this never weakens live safety. A scenario that raises is recorded as - `ok=False` and never aborts the safety-critical run; the sequences restore - a single-pane, all-rows workspace so later convergence checks are intact. + active churn; after every step the app must reach the observable state the + step names, or the scenario is recorded `ok=False`. Every scenario is + read-only navigation - no scenario writes, deletes, or drains - so this + never weakens live safety. A scenario that raises or that fails to reach a + target state is recorded as `ok=False` and never aborts the safety-critical + run; the remaining steps still run so the sequences restore a single-pane, + all-rows workspace and later convergence checks stay intact. """ - for name, keys in _UI_SCENARIOS: + for scenario in _ui_scenarios(scoped_namespace): started = now() ok = True - try: - for key in keys: - await pilot.press(key) - await pilot.pause() - except Exception: - ok = False - recorder.record_scenario(name, now() - started, ok) + for step in scenario.steps: + try: + for key in step.keys: + await pilot.press(key) + await wait_for( + pilot, + lambda step=step: step.reached(app), # type: ignore[misc] # bind per step + timeout=_UI_STEP_TIMEOUT, + label=f"{scenario.name}: {step.label}", + recorder=recorder, + ) + except Exception: + ok = False + recorder.record_scenario(scenario.name, now() - started, ok) def _check_row_count(row_count: int, expected: int) -> None: @@ -1212,6 +1360,8 @@ async def _run_measured_window( options=options, progress=state.progress, limits=limits, + profile=profile, + recorder=recorder, ) ) try: @@ -1238,7 +1388,13 @@ async def _run_measured_window( # UI-at-scale evidence: drive the scoped scenarios (filter, sort, # namespace switch, split pane, describe, multi-log) through the # real pilot while churn is still active. Read-only navigation only. - await drive_ui_scenarios(pilot, recorder, now=now) + await drive_ui_scenarios( + pilot, + recorder, + now=now, + app=app, + scoped_namespace=manifests.namespace_name(run_id, 0), + ) churn_timeout = ( profile.duration_seconds * options.time_scale + limits.churn_grace_seconds @@ -1375,10 +1531,25 @@ async def run_live_replay( ) app = MeasuredKorvidApp( - config=KorvidConfig(namespace=ALL_NAMESPACES), + config=KorvidConfig( + namespace=ALL_NAMESPACES, + # Key 1 scopes to a real seeded namespace; key 0 returns + # to all namespaces. Without a favorite the toggle is a + # no-op (the configured namespace already is + # ALL_NAMESPACES) and the scenario proves nothing. + favorite_namespaces=(manifests.namespace_name(run_id, 0),), + ), store=store, watch_manager=watch_manager, recorder=recorder, + # Read-only providers: `describe` and `multi_log` bail out + # with an "unavailable" warning when these are None, so the + # scenarios would report success having done nothing. The + # harness connection is used so these reads stay out of the + # measured application read path. + aliases={"pods": _PODS_META}, + get_manifest=_describe_provider(harness_kube), + stream_logs=harness_kube.stream_logs, ) sampler.start() diff --git a/tests/performance/profile.py b/tests/performance/profile.py index 95928915..0658136d 100644 --- a/tests/performance/profile.py +++ b/tests/performance/profile.py @@ -106,6 +106,14 @@ def _failures(raw: Any) -> tuple[FailureInjection, ...]: at_event=_int(item, "at_event", positive=True), ) ) + # `run_replay` indexes failures by `at_event`, so a duplicate position + # would silently discard all but one while the profile hash and the report + # still claim every declared injection was exercised. + positions = [failure.at_event for failure in result] + duplicates = sorted({position for position in positions if positions.count(position) > 1}) + if duplicates: + repeated = ", ".join(str(position) for position in duplicates) + raise ValueError(f"failures must declare distinct at_event positions; repeated: {repeated}") return tuple(sorted(result, key=lambda failure: failure.at_event)) diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py index b4ad9521..ea5357a0 100644 --- a/tests/performance/test_live.py +++ b/tests/performance/test_live.py @@ -21,9 +21,10 @@ from kubernetes_asyncio import config as k8s_config from multidict import CIMultiDict, CIMultiDictProxy -from korvid.core.store import Summary +from korvid.core.store import ALL_NAMESPACES, Summary from korvid.k8s.discovery import ResourceMeta from korvid.k8s.errors import ApiStatusError +from korvid.k8s.logs import LogLine from korvid.k8s.models import GenericSummary, PodSummary from korvid.k8s.telemetry import ReadTelemetry, ReadTelemetryEvent from tests.performance import live, manifests @@ -135,6 +136,8 @@ def __init__( self.list_pods_calls: list[str] = [] self.list_objects_calls = 0 self.watch_pods_calls = 0 + #: Namespace argument of each watch, in order. + self.watch_namespaces: list[str | None] = [] #: True once the watch generator has been closed (the harness stopped #: the WatchManager); lets tests assert *when* a read happened #: relative to the application watch still being live. @@ -144,6 +147,9 @@ def __init__( #: Optional watch-open failure (e.g. a 403 the real client would #: record as read telemetry before raising). self.watch_error: ApiStatusError | None = None + #: Read-only provider call logs for the UI-at-scale scenarios. + self.get_object_calls: list[tuple[str | None, str]] = [] + self.stream_logs_calls = 0 async def connect(self, context: str | None = None) -> None: self.connect_context = context @@ -151,6 +157,28 @@ async def connect(self, context: str | None = None) -> None: async def close(self) -> None: self.closed = True + async def get_object( + self, meta: ResourceMeta, namespace: str | None, name: str + ) -> dict[str, Any]: + """Read-only manifest fetch backing the `describe` UI-at-scale scenario.""" + self.get_object_calls.append((namespace, name)) + return { + "apiVersion": "v1", + "kind": "Pod", + "metadata": {"name": name, "namespace": namespace}, + "status": {"phase": "Running"}, + } + + async def stream_logs(self, *args: Any, **kwargs: Any) -> AsyncIterator[Any]: + """Read-only log stream backing the `multi_log` UI-at-scale scenario. + + Yields one line and then stays open, exactly like a real follow stream + the scenario dismisses with `escape`. + """ + self.stream_logs_calls += 1 + yield LogLine(pod="bench", container="app", text="benchmark log line") + await asyncio.Event().wait() + async def list_objects(self, meta: ResourceMeta, namespace: str | None) -> list[GenericSummary]: assert meta.kind == "Namespace" assert namespace is None @@ -167,9 +195,24 @@ async def list_pods(self, namespace: str) -> list[PodSummary]: self.read_telemetry(ReadTelemetryEvent("list", f"/api/v1/namespaces/{namespace}/pods")) return [pod for (ns, _name), pod in self.pods.items() if ns == namespace] + def _initial_watch_pods(self, namespace: str | None) -> list[PodSummary]: + """Pods the initial LIST of a watch returns: everything for the + all-namespaces watch (plus the unowned distractors a real cluster-wide + LIST would also return), only the namespace's own Pods when scoped.""" + if namespace is not None: + return [pod for pod in self.pods.values() if pod.namespace == namespace] + return [*self.pods.values(), *self.distractor_pods] + async def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSummary]]: - assert namespace is None + # `namespace is None` is the all-namespaces watch the measured window + # runs on; a concrete namespace is the scoped watch the + # `namespace_switch` UI-at-scale scenario really triggers. self.watch_pods_calls += 1 + self.watch_namespaces.append(namespace) + # Tracks the *currently open* watch: a scope change closes one + # generator and opens another, so "some generator finished" would not + # mean the application watch has stopped. + self.watch_finished = False try: if self.watch_error is not None: if self.read_telemetry is not None: @@ -179,9 +222,7 @@ async def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, Po raise self.watch_error if self.read_telemetry is not None: self.read_telemetry(ReadTelemetryEvent("list", "/api/v1/pods")) - for pod in list(self.pods.values()): - yield ("ADDED", pod) - for pod in self.distractor_pods: + for pod in self._initial_watch_pods(namespace): yield ("ADDED", pod) if self.read_telemetry is not None: self.read_telemetry(ReadTelemetryEvent("watch_open", "/api/v1/pods")) @@ -510,6 +551,8 @@ async def test_drive_live_churn_sends_guarded_patches_for_every_event() -> None: options=options, progress=progress, limits=LiveLimits(churn_concurrency=1), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), ) assert len(mutation_client.calls) == len(events) for call, event in zip(mutation_client.calls, events, strict=True): @@ -569,6 +612,8 @@ async def test_drive_live_churn_aborts_on_guard_failure_and_never_continues() -> options=options, progress=ChurnProgress(requested_events=len(events)), limits=LiveLimits(churn_concurrency=1), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), ) # Aborted at the 2nd call; a 3rd event must never have been attempted. assert len(mutation_client.calls) == 2 @@ -1002,15 +1047,21 @@ async def test_run_live_replay_full_happy_path_matches_cluster_digest() -> None: assert app_client.connect_context == CONTEXT assert app_client.closed - # MEDIUM finding: the app-path client's telemetry is the *only* source - # of `report.api` - its single "list" comes from `watch_pods`'s own - # internal LIST-then-WATCH, not from any harness read. - assert app_client.watch_pods_calls == 1 + # MEDIUM finding: the app-path client's telemetry is the *only* source of + # `report.api` - every "list" comes from `watch_pods`'s own internal + # LIST-then-WATCH, not from any harness read. Three watches, not one: the + # `namespace_switch` UI-at-scale scenario really scopes down to a seeded + # namespace and back, and a scope change restarts the application watch. + assert app_client.watch_pods_calls == 3 + # Every watch stays cluster-wide: `make_live_watch_source` pins the read to + # the owned namespace set regardless of the UI scope, so scoping the table + # to one namespace never re-targets (or widens) the underlying watch. + assert app_client.watch_namespaces == [None, None, None] # The watch really is stopped by teardown, so `watch_finished` is a # meaningful signal for the ground-truth-read ordering assertion. assert app_client.watch_finished - assert report.api.operations["watch_open"] == 1 - assert report.api.operations.get("list", 0) == 1 + assert report.api.operations["watch_open"] == 3 + assert report.api.operations.get("list", 0) == 3 # The harness client never watches - it is only ever used for the # ownership gate and the final independent re-read. @@ -1400,6 +1451,8 @@ async def _apply(self, namespace: str, name: str, *, uid: str, tick: str) -> Non options=ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep), progress=progress, limits=LiveLimits(churn_concurrency=4), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), ) assert progress.completed == len(events) == 40 @@ -1444,6 +1497,8 @@ async def patch_pod_labels_guarded( options=ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep), progress=progress, limits=LiveLimits(churn_concurrency=1, mutation_throttle_retries=5), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), ) assert len(mutation_client.patch_arguments) == 3 @@ -1633,6 +1688,8 @@ async def patch_pod_labels_guarded( ), progress=progress, limits=LiveLimits(churn_concurrency=1, mutation_throttle_retries=2), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), ) assert len(mutation_client.calls) == 3 @@ -1670,6 +1727,8 @@ async def patch_pod_labels_guarded( ), progress=progress, limits=LiveLimits(churn_concurrency=1, mutation_throttle_retries=5), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), ) assert len(mutation_client.calls) == 1 @@ -1702,6 +1761,8 @@ async def patch_pod_labels_guarded( options=ReplayOptions(time_scale=1.0), progress=progress, limits=LiveLimits(churn_concurrency=1, mutation_timeout_seconds=0.05), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), ) assert len(mutation_client.calls) == 1 @@ -2492,3 +2553,98 @@ async def test_run_live_replay_exercises_ui_at_scale_scenarios_during_churn() -> for scenario in report.ui_scenarios: assert scenario.ok, f"scenario {scenario.name} did not complete" assert scenario.latency_seconds >= 0.0 + + +class _InertPilot: + """A pilot whose key presses reach an app that never changes state. + + `pilot.press` returning without raising says nothing about whether the + binding did anything: in the live wiring `describe` and `multi_log` bail + out with an "unavailable" warning when no manifest/log provider is wired, + and the namespace toggle is a no-op while `config.namespace` already is + `ALL_NAMESPACES`. None of those raise. + """ + + def __init__(self) -> None: + self.pressed: list[str] = [] + + async def press(self, key: str) -> None: + self.pressed.append(key) + + async def pause(self, delay: float | None = None) -> None: + return None + + +class _InertApp: + """Live app state frozen at its post-LIST values: nothing a scenario is + supposed to change ever changes.""" + + def __init__(self) -> None: + self.filter_pattern = "" + self._sorts: dict[str, object] = {} + self.current_scope = ALL_NAMESPACES + self.screen = object() + self._panes = [object()] + + class _Pane: + display = False + + self._log_pane = _Pane() + + +async def test_ui_scenarios_are_not_marked_ok_when_the_app_state_never_changes() -> None: + """A scenario must assert the observable state it claims to exercise. + + Without this the live report claims UI-at-scale evidence for scenarios + that silently did nothing, and (since the CLI folds scenario outcomes into + the exit status) a qualification would "pass" on that empty evidence. + """ + recorder = BenchmarkRecorder() + ticks = iter(float(i) for i in range(10_000)) + + await live.drive_ui_scenarios( + _InertPilot(), + recorder, + now=lambda: next(ticks), + app=_InertApp(), + scoped_namespace=f"korvid-perf-{RUN_ID}-0", + ) + + results = recorder._ui_scenarios + assert results, "scenarios must still be recorded, not skipped" + assert [scenario.name for scenario in results if scenario.ok] == [] + + +async def test_run_live_replay_times_the_post_burst_drain() -> None: + """The live churn driver must mark each burst boundary. + + Without it `post_burst_drain_seconds` stays empty and + `max_post_burst_drain_seconds` is `None`, so the published <=3-second live + burst-drain budget cannot be evaluated at all — the live reports simply + print "n/a" where the evidence should be. + """ + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = _happy_deps(namespaces, pods, RUN_ID) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions( + time_scale=1.0, sample_interval=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ) + profile = dataclasses.replace( + _tiny_live_profile(), + steady_events_per_second=2, + duration_seconds=4, + bursts=(Burst(start_second=1, duration_seconds=1, events_per_second=4),), + ) + + report = await run_live_replay( + profile, + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.phases.post_burst_drain_seconds + assert report.phases.max_post_burst_drain_seconds is not None + assert report.phases.max_post_burst_drain_seconds >= 0.0 diff --git a/tests/performance/test_profile.py b/tests/performance/test_profile.py index d024a7a4..e35161f0 100644 --- a/tests/performance/test_profile.py +++ b/tests/performance/test_profile.py @@ -155,3 +155,19 @@ def test_load_profile_accepts_every_versioned_failure_kind(tmp_path: Path, kind: def test_load_profile_rejects_unknown_failure_kind(tmp_path: Path) -> None: with pytest.raises(ValueError, match="kind must be one of"): load_profile(_write(tmp_path, failures=[{"kind": "meltdown", "at_event": 10}])) + + +def test_load_profile_rejects_duplicate_failure_event_positions(tmp_path: Path) -> None: + """`run_replay` indexes failures by `at_event`, so a duplicate position + silently drops every failure but one — while the profile hash and the + report still claim both were injected.""" + with pytest.raises(ValueError, match="at_event"): + load_profile( + _write( + tmp_path, + failures=[ + {"kind": "gone", "at_event": 10}, + {"kind": "throttled", "at_event": 10}, + ], + ) + ) diff --git a/tests/ui/test_node_shell.py b/tests/ui/test_node_shell.py index 8dd68c32..7593edfa 100644 --- a/tests/ui/test_node_shell.py +++ b/tests/ui/test_node_shell.py @@ -197,6 +197,25 @@ def fake_call(argv): # type: ignore[no-untyped-def] # test helper yield call_records +async def _await_node_shell_outcome(pilot: Any, audit_path: Path) -> None: + """Wait until the node-shell *outcome* entry has been appended. + + Every outcome is written after its user-facing notification, on a separate + `asyncio.to_thread` hop. Waiting on the notification alone races the audit + write against app teardown, leaving only the `intent` entry behind on a + loaded run. + """ + + def _written() -> bool: + if not audit_path.exists(): + return False + records = [json.loads(ln) for ln in audit_path.read_text().splitlines()] + shells = [e for e in records if e["action"] == "node-shell"] + return bool(shells) and shells[-1]["outcome"] != "intent" + + await until(pilot, _written, label="node-shell outcome audited") + + async def test_s_on_nodes_view_opens_privileged_approval_dialog(tmp_path: Path) -> None: rec = DeleteRecorder() app = make_app(rec, tmp_path / "audit.jsonl") @@ -283,17 +302,7 @@ async def test_node_shell_cleanup_failure_warns_and_audits(tmp_path: Path) -> No await until(pilot, lambda: isinstance(app.screen, ConfirmScreen), label="dialog") await pilot.press("y") - def _audited() -> bool: - # The outcome entry is appended after the failure notification - # (a separate `asyncio.to_thread` hop), so waiting on the - # notification alone races the audit write at teardown. - if not audit_path.exists(): - return False - records = [json.loads(ln) for ln in audit_path.read_text().splitlines()] - shells = [e for e in records if e["action"] == "node-shell"] - return bool(shells) and "cleanup failed for" in shells[-1]["outcome"] - - await until(pilot, _audited, label="cleanup failure audited") + await _await_node_shell_outcome(pilot, audit_path) assert any(DBG_POD in n.message for n in app._notifications) entries = [json.loads(ln) for ln in audit_path.read_text().splitlines()] last = [e for e in entries if e["action"] == "node-shell"][-1] @@ -378,10 +387,8 @@ async def test_node_shell_create_failure_warns_about_policy(tmp_path: Path) -> N await until(pilot, lambda: isinstance(app.screen, ConfirmScreen), label="dialog") await pilot.press("y") - def _warned() -> bool: - return any("PodSecurity" in n.message for n in app._notifications) - - await until(pilot, _warned, label="policy hint notification") + await _await_node_shell_outcome(pilot, audit_path) + assert any("PodSecurity" in n.message for n in app._notifications) assert call_records == [] assert rec.deletes == [] entries = [json.loads(ln) for ln in audit_path.read_text().splitlines()] @@ -407,10 +414,8 @@ async def test_node_shell_unidentifiable_create_output_aborts(tmp_path: Path) -> await until(pilot, lambda: isinstance(app.screen, ConfirmScreen), label="dialog") await pilot.press("y") - def _warned() -> bool: - return any("did not report" in n.message for n in app._notifications) - - await until(pilot, _warned, label="unidentifiable-pod warning") + await _await_node_shell_outcome(pilot, audit_path) + assert any("did not report" in n.message for n in app._notifications) assert call_records == [] assert rec.deletes == [] entries = [json.loads(ln) for ln in audit_path.read_text().splitlines()] @@ -575,6 +580,7 @@ async def test_node_shell_nonzero_attach_exit_has_no_policy_hint(tmp_path: Path) await until(pilot, lambda: isinstance(app.screen, ConfirmScreen), label="dialog") await pilot.press("y") await until(pilot, lambda: rec.deletes, label="debug pod cleanup") + await _await_node_shell_outcome(pilot, audit_path) assert call_records != [] assert not any("PodSecurity" in n.message for n in app._notifications) entries = [json.loads(ln) for ln in audit_path.read_text().splitlines()] @@ -625,7 +631,8 @@ async def test_node_shell_create_without_uid_aborts(tmp_path: Path) -> None: without it the cleanup delete would lose its uid precondition and could remove a same-name replacement pod.""" rec = DeleteRecorder() - app = make_app(rec, tmp_path / "audit.jsonl") + audit_path = tmp_path / "audit.jsonl" + app = make_app(rec, audit_path) run_fake, _ = _kubectl_run( get_result=SimpleNamespace( returncode=0, @@ -641,10 +648,8 @@ async def test_node_shell_create_without_uid_aborts(tmp_path: Path) -> None: await until(pilot, lambda: isinstance(app.screen, ConfirmScreen), label="dialog") await pilot.press("y") - def _warned() -> bool: - return any("uid" in n.message for n in app._notifications) - - await until(pilot, _warned, label="missing-uid warning") + await _await_node_shell_outcome(pilot, audit_path) + assert any("uid" in n.message for n in app._notifications) assert call_records == [] assert rec.deletes == [] From 4807a7c85fdc63fcf4bb1f0e2b486ce293117df1 Mon Sep 17 00:00:00 2001 From: hellices Date: Thu, 6 Aug 2026 23:31:55 +0900 Subject: [PATCH 35/38] fix(perf-live): verify the rendered table and stop miscounting reconnects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second review round on PR #202. Three of these are defects the previous round's changes created or exposed. **The digest criterion never checked the table.** `expected_digest` and `final_digest` are both store digests, so a table showing 1,000 stale rows satisfied the published store/table digest criterion — and the row count was the only thing asserted about the rendering. That is exactly the guarantee the new cached in-place diff needs, since it now diffs against its own record of what it last wrote instead of reading the table back. `check_rendered_rows` projects each owned Pod onto the strings its row must display and checks them against the cells the `DataTable` actually holds. It is written from the Pod summary rather than by calling the widget's row builder on purpose: reusing the builder would only prove the widget agrees with itself. **Reconnects counted deliberate restarts.** `reconnects` was inferred as `watch_open - 1` per path. The `namespace_switch` scenario added last commit legitimately re-opens `/api/v1/pods` twice, so a perfectly healthy run reported two reconnects. A reconnect is now counted from recovery: a `watch_open` on a path whose stream previously errored. **An empty backlog at a burst boundary produced a bogus drain.** `mark_burst_end` always queued a pending marker, so with nothing to drain the next unrelated steady-state render reported its own latency as the drain (or the sample vanished if no later render arrived). Nothing to drain now records `0.0` immediately. **`_flush_allocation_snapshot` could mask a run failure.** It runs from a `finally`, so an unwritable destination raised `OSError` past the command's handler: a traceback after a 30-minute run, hiding whatever actually failed. It now reports and returns a status, tracing is always stopped, and the failure surfaces as exit 1. Docs: the published live command was no longer executable (all four artifacts are mandatory and each filename must carry the run id), and the determinism claim listed uids and resource versions, which the generator does not assign — on a live run they are whatever the cluster issued. Tests: tests/performance/test_live.py::test_rendered_rows_check_rejects_a_stale_cell tests/performance/test_metrics.py:: test_reconnects_count_only_watch_reopens_after_an_error test_reconnects_count_a_watch_reopened_after_a_dropped_stream test_mark_burst_end_records_zero_when_the_backlog_is_already_empty tests/performance/test_cli.py:: test_cli_replay_live_reports_an_unwritable_allocation_snapshot Refs #186 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...luster-performance-qualification-design.md | 18 ++++- tests/performance/cli.py | 55 +++++++++---- tests/performance/live.py | 38 +++++++++ tests/performance/metrics.py | 25 +++++- tests/performance/test_cli.py | 20 +++++ tests/performance/test_live.py | 77 +++++++++++++++++++ tests/performance/test_metrics.py | 63 +++++++++++++++ 7 files changed, 275 insertions(+), 21 deletions(-) diff --git a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md index 32249081..600f0f8a 100644 --- a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md +++ b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md @@ -82,18 +82,28 @@ uv run python -m tests.performance.cli replay-live \ --profile tests/performance/profiles/aks-live-1k.json \ --context aks-korvid-contract-test \ --expected-cluster-id --run-id \ - --json live.json --out live.md + --json -live.json --out -live.md \ + --cpu-profile -live.pstats \ + --allocation-snapshot -live-alloc.txt ``` +All four artifacts are mandatory and every filename must contain the run id: +a live run whose evidence is missing or untraceable is rejected before the +cluster is touched, and two destinations that resolve to the same file are +rejected as well. + `aks-1k` keeps the short (30-second) schedule shared with `burst-50k`, so a live run can be compared against the deterministic 1k/10k/50k baselines on the same event schedule. `--duration` shortens a live smoke run; every burst and failure point is re-validated against the shortened duration before the cluster identity and ownership gates run. -The deterministic generator emits stable names, namespaces, UIDs, resource -versions, and event order from the profile seed. Repeating a profile with the -same seed must produce the same object and event hashes. +The deterministic generator emits stable names, namespaces, and event order +from the profile seed. Repeating a profile with the same seed must produce the +same object and event hashes. Uids and resource versions are deliberately *not* +part of the deterministic workload: the synthetic generator does not assign +them, and on a live run they are whatever the cluster issued (the live path +reads each seeded Pod's real uid and pins it as the guard precondition). ### Runtime path diff --git a/tests/performance/cli.py b/tests/performance/cli.py index 6edcdeac..7ae6ec20 100644 --- a/tests/performance/cli.py +++ b/tests/performance/cli.py @@ -262,14 +262,28 @@ def _run_live_with_cpu_profile( pr.dump_stats(cpu_profile_path) -def _flush_allocation_snapshot(path: str) -> None: - """Take a tracemalloc snapshot and write the top 100 lines to *path*.""" +def _flush_allocation_snapshot(path: str) -> int: + """Take a tracemalloc snapshot and write the top 100 lines to *path*. + + Runs from a `finally`, so a write failure must not replace whatever + exception is already propagating (or bury a long run's real failure under a + traceback). Tracing is always stopped. + + Returns: + 0 on success, 1 when the snapshot could not be written. + """ if not tracemalloc.is_tracing(): - return - snapshot = tracemalloc.take_snapshot() - stats = snapshot.statistics("lineno")[:100] - Path(path).write_text("\n".join(str(stat) for stat in stats)) - tracemalloc.stop() + return 0 + try: + snapshot = tracemalloc.take_snapshot() + stats = snapshot.statistics("lineno")[:100] + Path(path).write_text("\n".join(str(stat) for stat in stats)) + except OSError as exc: + print(f"error writing allocation snapshot: {exc}", file=sys.stderr) + return 1 + finally: + tracemalloc.stop() + return 0 def _write_outputs(args: argparse.Namespace, replay: ReplayReport) -> int: @@ -332,6 +346,7 @@ def _cmd_replay(args: argparse.Namespace) -> int: options = ReplayOptions(time_scale=args.time_scale, sample_interval=args.sample_interval) + snapshot_failed = 0 if args.allocation_snapshot: tracemalloc.start() @@ -345,8 +360,18 @@ def _cmd_replay(args: argparse.Namespace) -> int: return 1 finally: if args.allocation_snapshot: - _flush_allocation_snapshot(args.allocation_snapshot) + snapshot_failed = _flush_allocation_snapshot(args.allocation_snapshot) + return _replay_exit_status(args, replay, snapshot_failed=snapshot_failed) + + +def _replay_exit_status( + args: argparse.Namespace, replay: ReplayReport, *, snapshot_failed: int +) -> int: + """Exit status shared by both replay commands: artifacts first, then the + correctness criteria (no dropped updates, digest parity).""" + if snapshot_failed: + return 1 if _write_outputs(args, replay): return 1 if replay.dropped_updates > 0 or replay.expected_digest != replay.final_digest: @@ -450,6 +475,7 @@ def _cmd_replay_live(args: argparse.Namespace) -> int: def _execute_live_replay( args: argparse.Namespace, profile: WorkloadProfile, options: ReplayOptions ) -> int: + snapshot_failed = 0 if args.allocation_snapshot: tracemalloc.start() @@ -478,23 +504,24 @@ def _execute_live_replay( return 1 finally: if args.allocation_snapshot: - _flush_allocation_snapshot(args.allocation_snapshot) + snapshot_failed = _flush_allocation_snapshot(args.allocation_snapshot) - if _write_outputs(args, replay): + if snapshot_failed: return 1 failed_scenarios = [scenario.name for scenario in replay.ui_scenarios if not scenario.ok] if failed_scenarios: # `drive_ui_scenarios` records a key sequence that never reached its # target state as `ok=False` instead of raising, so without this the - # run would "pass" with no UI-at-scale evidence behind it. + # run would "pass" with no UI-at-scale evidence behind it. Reported + # before the artifacts are written so the reason is not buried under + # the Markdown report. print( "error: UI-at-scale scenarios did not pass: " + ", ".join(failed_scenarios), file=sys.stderr, ) + _write_outputs(args, replay) return 1 - if replay.dropped_updates > 0 or replay.expected_digest != replay.final_digest: - return 1 - return 0 + return _replay_exit_status(args, replay, snapshot_failed=snapshot_failed) def main(argv: list[str] | None = None) -> int: diff --git a/tests/performance/live.py b/tests/performance/live.py index 0369d78e..e6c126f2 100644 --- a/tests/performance/live.py +++ b/tests/performance/live.py @@ -1288,6 +1288,41 @@ async def drive_ui_scenarios( recorder.record_scenario(scenario.name, now() - started, ok) +def check_rendered_rows(table: Any, pods: Iterable[PodSummary]) -> None: + """Verify the rendered table against the store, independently of the widget. + + The published digest criterion compares a store digest with a store digest: + a table showing 1,000 stale rows satisfies it. This projects each owned Pod + onto the strings its row must display and checks them against the cells the + `DataTable` actually holds, so a cell the in-place diff skipped - it now + diffs against its own record of what it last wrote - is caught rather than + reported as a clean run. + + Deliberately column-order agnostic and written from the Pod summary rather + than by calling the widget's row builder: reusing the builder would only + prove the widget agrees with itself. + + Raises: + ValueError: a Pod is missing from the table, or a rendered row does not + carry every value the store says it must show. + """ + rendered = { + str(row.key.value): [str(cell) for cell in table.get_row(row.key)] + for row in table.ordered_rows + } + for pod in pods: + key = f"{pod.namespace}/{pod.name}" + cells = rendered.get(key) + if cells is None: + raise ValueError(f"rendered table is missing owned pod {key}") + expected = (pod.name, pod.ready, pod.phase, str(pod.restarts)) + missing = [value for value in expected if value not in cells] + if missing: + raise ValueError( + f"rendered row for {key} is stale: expected cells {missing} not among {cells}" + ) + + def _check_row_count(row_count: int, expected: int) -> None: """Re-assert the exact rendered row count before teardown. @@ -1423,6 +1458,9 @@ async def _run_measured_window( recorder=recorder, ) _check_row_count(table.row_count, profile.object_count) + # Independent of the digest above, which compares a store digest + # with a store digest and would accept a table full of stale cells. + check_rendered_rows(table, final_pods) state.final_digest = _store_digest(store) finally: await _cancel_and_drain(churn_task) diff --git a/tests/performance/metrics.py b/tests/performance/metrics.py index 8a05eb36..6d2d140b 100644 --- a/tests/performance/metrics.py +++ b/tests/performance/metrics.py @@ -179,6 +179,14 @@ def from_events(cls, events: Sequence[ReadTelemetryEvent]) -> ApiSummary: authorization_failures = 0 relists = 0 relist_candidates: set[str] = set() + reconnects = 0 + #: Paths whose stream errored and has not been re-opened yet. A + #: reconnect is recovery from a dropped watch, so it must be counted + #: from that recovery - not inferred from the number of `watch_open`s. + #: A deliberate stop/start (the live `namespace_switch` scenario scopes + #: the table down and back) re-opens the same path with no error in + #: between and is not a reconnect. + dropped_paths: set[str] = set() for event in events: operations[event.operation] += 1 paths.setdefault(event.path, Counter())[event.operation] += 1 @@ -193,9 +201,13 @@ def from_events(cls, events: Sequence[ReadTelemetryEvent]) -> ApiSummary: if event.operation == "list" and event.path in relist_candidates: relists += 1 relist_candidates.remove(event.path) + if event.operation == "watch_open" and event.path in dropped_paths: + reconnects += 1 + dropped_paths.remove(event.path) + if event.operation == "error": + dropped_paths.add(event.path) if event.status == 410: relist_candidates.add(event.path) - reconnects = sum(max(counts.get("watch_open", 0) - 1, 0) for counts in paths.values()) return cls( operations=MappingProxyType(dict(sorted(operations.items()))), paths=MappingProxyType( @@ -450,9 +462,16 @@ def mark_interactive(self, at: float) -> None: def mark_burst_end(self, at: float) -> None: """Record the end of a churn burst so post-burst drain can be timed. - The drain is resolved by `record_render` the next time the pending - backlog empties at or after this instant. + With events still pending the drain is resolved by `record_render` the + next time the backlog empties at or after this instant. With an already + empty backlog there is nothing to drain, so the sample is `0.0` right + away: leaving the marker pending would let the next unrelated + steady-state render report its own latency as a drain, or drop the + sample entirely if no later render arrives. """ + if not self._pending_events: + self._post_burst_drains.append(0.0) + return self._burst_end_pending.append(at) def pending_count(self) -> int: diff --git a/tests/performance/test_cli.py b/tests/performance/test_cli.py index f6340c50..22d00a36 100644 --- a/tests/performance/test_cli.py +++ b/tests/performance/test_cli.py @@ -976,3 +976,23 @@ def test_cli_replay_live_rejects_artifact_paths_that_alias_one_file( assert exit_code == 1 assert "distinct destinations" in capsys.readouterr().err assert calls == [] + + +def test_cli_replay_live_reports_an_unwritable_allocation_snapshot( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """The snapshot is flushed from a `finally`, so an unwritable destination + would raise straight out of the command - producing a traceback after a + 30-minute run and masking whatever actually failed inside the run.""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + artifacts = _live_artifacts(tmp_path) + unwritable = tmp_path / "missing-dir" / "aks186-live.alloc.txt" + artifacts[artifacts.index("--allocation-snapshot") + 1] = str(unwritable) + + exit_code = cli.main( + ["replay-live", "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, *artifacts] + ) + + assert exit_code == 1 + assert "allocation snapshot" in capsys.readouterr().err diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py index ea5357a0..0b065d0f 100644 --- a/tests/performance/test_live.py +++ b/tests/performance/test_live.py @@ -13,6 +13,7 @@ import re from collections import Counter from collections.abc import AsyncGenerator, AsyncIterator, Awaitable, Callable +from types import SimpleNamespace from typing import Any from unittest.mock import AsyncMock, MagicMock @@ -2648,3 +2649,79 @@ async def test_run_live_replay_times_the_post_burst_drain() -> None: assert report.phases.post_burst_drain_seconds assert report.phases.max_post_burst_drain_seconds is not None assert report.phases.max_post_burst_drain_seconds >= 0.0 + + +def _pod_summary(namespace: str, name: str, **kwargs: Any) -> PodSummary: + defaults: dict[str, Any] = { + "namespace": namespace, + "name": name, + "ready": "1/1", + "phase": "Running", + "restarts": 0, + "node": "node-a", + "created": "", + "uid": "uid-1", + "labels": {}, + } + defaults.update(kwargs) + return PodSummary(**{k: v for k, v in defaults.items() if k in _POD_SUMMARY_FIELDS}) + + +_POD_SUMMARY_FIELDS = {field.name for field in dataclasses.fields(PodSummary)} + + +class _StubRow: + def __init__(self, value: str) -> None: + self.key = SimpleNamespace(value=value) + + +class _StubTable: + """Minimal stand-in exposing exactly what the rendered-row check reads.""" + + def __init__(self, rows: dict[str, list[object]]) -> None: + self._rows = rows + self.row_count = len(rows) + self.ordered_rows = [_StubRow(key) for key in rows] + + def get_row(self, row_key: object) -> list[object]: + key = row_key.value if hasattr(row_key, "value") else row_key + return self._rows[str(key)] + + +def test_rendered_rows_check_rejects_a_stale_cell() -> None: + """The published digest criterion compares a store digest with a store + digest, so 1,000 stale cells satisfy it. The rendered table must be checked + against the store independently — especially now that the in-place diff + updates cells from its own cached record of what it last wrote. + """ + pods = [ + _pod_summary("ns-a", "bench-0", phase="Running"), + _pod_summary("ns-a", "bench-1", phase="CrashLoopBackOff"), + ] + table = _StubTable( + { + "ns-a/bench-0": ["ns-a", "bench-0", "1/1", "Running", "0", "node-a"], + # Stale: the store says CrashLoopBackOff, the table still shows Running. + "ns-a/bench-1": ["ns-a", "bench-1", "1/1", "Running", "0", "node-a"], + } + ) + + with pytest.raises(ValueError, match="ns-a/bench-1"): + live.check_rendered_rows(table, pods) + + +def test_rendered_rows_check_accepts_a_table_that_matches_the_store() -> None: + pods = [ + _pod_summary("ns-a", "bench-0", phase="Running"), + _pod_summary("ns-a", "bench-1", phase="CrashLoopBackOff", ready="0/1", restarts=7), + ] + table = _StubTable( + { + "ns-a/bench-0": ["ns-a", "bench-0", "1/1", "Running", "0", "node-a"], + "ns-a/bench-1": ["ns-a", "bench-1", "0/1", "CrashLoopBackOff", "7", "node-a"], + } + ) + + live.check_rendered_rows(table, pods) + + assert table.row_count == 2 diff --git a/tests/performance/test_metrics.py b/tests/performance/test_metrics.py index 73417c51..a2e22614 100644 --- a/tests/performance/test_metrics.py +++ b/tests/performance/test_metrics.py @@ -9,6 +9,7 @@ from korvid.k8s.telemetry import ReadTelemetryEvent from tests.performance.metrics import ( + ApiSummary, BenchmarkRecorder, ChurnSummary, LatencySummary, @@ -691,3 +692,65 @@ def test_render_markdown_reports_phase_measurements() -> None: assert "- LIST to populated table: `1.000s`" in markdown assert "- Max backlog depth: `1`" in markdown assert "- Max post-burst drain: `0.400s`" in markdown + + +def test_mark_burst_end_records_zero_when_the_backlog_is_already_empty() -> None: + """Nothing to drain is a 0.0 sample, not a pending marker. + + Left pending, the marker is resolved by the next unrelated steady-state + render and reported as that render's full latency, or lost entirely when no + later render arrives - either way the burst-drain budget is measured against + something that is not a drain. + """ + recorder = BenchmarkRecorder() + + recorder.mark_burst_end(10.0) + + report = recorder.report(run_manifest(), (), final_digest="d") + assert report.phases.post_burst_drain_seconds == (0.0,) + assert report.phases.max_post_burst_drain_seconds == 0.0 + + +def test_mark_burst_end_still_times_a_real_drain() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 5.0) + + recorder.mark_burst_end(10.0) + recorder.record_render(12.0) + + report = recorder.report(run_manifest(), (), final_digest="d") + assert report.phases.post_burst_drain_seconds == pytest.approx((2.0,)) + + +def test_reconnects_count_only_watch_reopens_after_an_error() -> None: + """A deliberate stop/start is not a reconnect. + + The live `namespace_switch` scenario scopes the table down and back, which + restarts the application watch twice on the same path; inferring reconnects + from `watch_open` counts alone reports a perfectly healthy run as having + two reconnects. + """ + summary = ApiSummary.from_events( + [ + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ] + ) + + assert summary.reconnects == 0 + assert summary.operations["watch_open"] == 3 + + +def test_reconnects_count_a_watch_reopened_after_a_dropped_stream() -> None: + summary = ApiSummary.from_events( + [ + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ReadTelemetryEvent("error", "/api/v1/pods", status=410), + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ReadTelemetryEvent("error", "/api/v1/pods", status=500), + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ] + ) + + assert summary.reconnects == 2 From 4e38fb8ba6871e9251a727d6631c25d3e17ef235 Mon Sep 17 00:00:00 2001 From: hellices Date: Fri, 7 Aug 2026 00:13:46 +0900 Subject: [PATCH 36/38] fix(perf): stop double-marking bursts and leaking tracing on sampler failure Third review round on PR #202 raised no blocking comments; these are the suppressed findings that turned out to be real correctness bugs, two of them in the burst-drain machinery the previous commits touched. **A reconnect re-marked every burst that had already ended.** `_ReplaySource` restarts at `_next_event_index` on a new watch generation, but the burst cursor was a local reset to zero each time. After a 410 drops the stream mid-schedule, the first resumed event re-marked every closed burst. Reproduced with a single-burst profile and a 410 at event 50: two drain samples for one burst, the second a spurious `0.0`. The cursor now lives on the source, alongside `_next_event_index`. **A failed sampler leaked managed tracing.** `ProcessSampler.stop()` awaits the sampler task and released tracemalloc afterwards, so a `psutil`/`tracemalloc` error inside the task raised past the release - and past the caller's own watch-manager teardown. The release now runs from a `finally`. **The offline replay never checked its rendering either.** Both digests are computed from the store, so a table full of stale cells passed - the same hole `28dec94` closed on the live path, and the one that matters for the cached in-place diff. `check_rendered_rows` moved from `live.py` to `replay.py` (`live.py` already imports from it, so this avoids an import cycle) and now runs on the offline path too, inside the app block where the table still exists, and only when the run was not aborted by an injected failure. Tests: tests/performance/test_replay.py:: test_replay_does_not_re_mark_bursts_after_a_watch_reconnect tests/performance/test_metrics.py:: test_sampler_stop_releases_tracing_even_when_the_task_failed Refs #186 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/performance/live.py | 36 +------------------ tests/performance/metrics.py | 15 +++++--- tests/performance/replay.py | 58 ++++++++++++++++++++++++++++--- tests/performance/test_live.py | 5 +-- tests/performance/test_metrics.py | 27 ++++++++++++++ tests/performance/test_replay.py | 28 +++++++++++++++ 6 files changed, 123 insertions(+), 46 deletions(-) diff --git a/tests/performance/live.py b/tests/performance/live.py index e6c126f2..59a793fa 100644 --- a/tests/performance/live.py +++ b/tests/performance/live.py @@ -99,6 +99,7 @@ ReplayOptions, ReplayReport, build_manifest, + check_rendered_rows, resolve_korvid_sha, wait_for, ) @@ -1288,41 +1289,6 @@ async def drive_ui_scenarios( recorder.record_scenario(scenario.name, now() - started, ok) -def check_rendered_rows(table: Any, pods: Iterable[PodSummary]) -> None: - """Verify the rendered table against the store, independently of the widget. - - The published digest criterion compares a store digest with a store digest: - a table showing 1,000 stale rows satisfies it. This projects each owned Pod - onto the strings its row must display and checks them against the cells the - `DataTable` actually holds, so a cell the in-place diff skipped - it now - diffs against its own record of what it last wrote - is caught rather than - reported as a clean run. - - Deliberately column-order agnostic and written from the Pod summary rather - than by calling the widget's row builder: reusing the builder would only - prove the widget agrees with itself. - - Raises: - ValueError: a Pod is missing from the table, or a rendered row does not - carry every value the store says it must show. - """ - rendered = { - str(row.key.value): [str(cell) for cell in table.get_row(row.key)] - for row in table.ordered_rows - } - for pod in pods: - key = f"{pod.namespace}/{pod.name}" - cells = rendered.get(key) - if cells is None: - raise ValueError(f"rendered table is missing owned pod {key}") - expected = (pod.name, pod.ready, pod.phase, str(pod.restarts)) - missing = [value for value in expected if value not in cells] - if missing: - raise ValueError( - f"rendered row for {key} is stale: expected cells {missing} not among {cells}" - ) - - def _check_row_count(row_count: int, expected: int) -> None: """Re-assert the exact rendered row count before teardown. diff --git a/tests/performance/metrics.py b/tests/performance/metrics.py index 6d2d140b..73b3b891 100644 --- a/tests/performance/metrics.py +++ b/tests/performance/metrics.py @@ -376,11 +376,16 @@ async def stop(self) -> tuple[ProcessSample, ...]: task = self._task self._task = None task.cancel() - with suppress(asyncio.CancelledError): - await task - if self._uses_managed_tracing: - self._release_tracemalloc() - self._uses_managed_tracing = False + try: + with suppress(asyncio.CancelledError): + await task + finally: + # A sampling failure must still release managed tracing: raising + # past this point would leak the tracemalloc lease *and* skip the + # caller's own teardown (watch manager, benchmark tasks). + if self._uses_managed_tracing: + self._release_tracemalloc() + self._uses_managed_tracing = False return tuple(self._samples) @classmethod diff --git a/tests/performance/replay.py b/tests/performance/replay.py index 259e0141..78649639 100644 --- a/tests/performance/replay.py +++ b/tests/performance/replay.py @@ -250,6 +250,9 @@ def __init__( self._failures = failures self._generation = 0 self._next_event_index = 0 + #: Index of the next burst whose end has not been marked yet. Persisted + #: across watch generations (see the WATCH phase below). + self._next_burst = 0 #: Number of scheduled churn events actually yielded to the watch #: manager so far; the real signal behind #: `ReplayReport.churn_started_before_input`. @@ -356,9 +359,11 @@ async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summ # Burst-end offsets (absolute seconds) mark the moment each burst's # window closes, so post-burst backlog drain can be timed on the same - # real-clock axis the render pass records on. + # real-clock axis the render pass records on. The cursor lives on the + # source, not this generation: a 410/429 reconnect resumes later in the + # schedule, and a per-generation cursor would re-mark every burst that + # already ended, producing duplicate and time-shifted drain samples. burst_ends = burst_end_offsets(self._profile) - next_burst = 0 # --- WATCH phase --- for i in range(self._next_event_index, len(self._events)): @@ -368,9 +373,12 @@ async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summ if delay > 0: await self._sleep(delay) - while next_burst < len(burst_ends) and event.offset_seconds >= burst_ends[next_burst]: + while ( + self._next_burst < len(burst_ends) + and event.offset_seconds >= burst_ends[self._next_burst] + ): self._recorder.mark_burst_end(monotonic()) - next_burst += 1 + self._next_burst += 1 await self._handle_failure_if_any(event, i) @@ -437,6 +445,41 @@ def build_manifest( ) +def check_rendered_rows(table: Any, pods: Iterable[PodSummary]) -> None: + """Verify the rendered table against the store, independently of the widget. + + The published digest criterion compares a store digest with a store digest: + a table showing 1,000 stale rows satisfies it. This projects each owned Pod + onto the strings its row must display and checks them against the cells the + `DataTable` actually holds, so a cell the in-place diff skipped - it now + diffs against its own record of what it last wrote - is caught rather than + reported as a clean run. + + Deliberately column-order agnostic and written from the Pod summary rather + than by calling the widget's row builder: reusing the builder would only + prove the widget agrees with itself. + + Raises: + ValueError: a Pod is missing from the table, or a rendered row does not + carry every value the store says it must show. + """ + rendered = { + str(row.key.value): [str(cell) for cell in table.get_row(row.key)] + for row in table.ordered_rows + } + for pod in pods: + key = f"{pod.namespace}/{pod.name}" + cells = rendered.get(key) + if cells is None: + raise ValueError(f"rendered table is missing owned pod {key}") + expected = (pod.name, pod.ready, pod.phase, str(pod.restarts)) + missing = [value for value in expected if value not in cells] + if missing: + raise ValueError( + f"rendered row for {key} is stale: expected cells {missing} not among {cells}" + ) + + async def wait_for( pilot: Any, condition: Callable[[], object], @@ -569,6 +612,13 @@ async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> Replay label="churn complete and all events rendered", recorder=recorder, ) + if source.terminal_failure is None: + # The digests below are both computed from data, never from the + # rendering: a table full of stale cells would satisfy them. + # Checked here because the table only exists inside this block. + check_rendered_rows( + table, cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES)) + ) finally: process_samples = await sampler.stop() await watch_manager.stop_all() diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py index 0b065d0f..7da49787 100644 --- a/tests/performance/test_live.py +++ b/tests/performance/test_live.py @@ -29,6 +29,7 @@ from korvid.k8s.models import GenericSummary, PodSummary from korvid.k8s.telemetry import ReadTelemetry, ReadTelemetryEvent from tests.performance import live, manifests +from tests.performance import replay as replay_mod from tests.performance.live import ( ChurnProgress, CommandResult, @@ -2707,7 +2708,7 @@ def test_rendered_rows_check_rejects_a_stale_cell() -> None: ) with pytest.raises(ValueError, match="ns-a/bench-1"): - live.check_rendered_rows(table, pods) + replay_mod.check_rendered_rows(table, pods) def test_rendered_rows_check_accepts_a_table_that_matches_the_store() -> None: @@ -2722,6 +2723,6 @@ def test_rendered_rows_check_accepts_a_table_that_matches_the_store() -> None: } ) - live.check_rendered_rows(table, pods) + replay_mod.check_rendered_rows(table, pods) assert table.row_count == 2 diff --git a/tests/performance/test_metrics.py b/tests/performance/test_metrics.py index a2e22614..62fc98e4 100644 --- a/tests/performance/test_metrics.py +++ b/tests/performance/test_metrics.py @@ -1,6 +1,7 @@ from __future__ import annotations import asyncio +import contextlib import importlib import json from typing import Any, cast @@ -754,3 +755,29 @@ def test_reconnects_count_a_watch_reopened_after_a_dropped_stream() -> None: ) assert summary.reconnects == 2 + + +async def test_sampler_stop_releases_tracing_even_when_the_task_failed() -> None: + """A sampling failure must not leak managed tracing state. + + `stop()` awaits the sampler task; if that raises, the release below never + runs and the caller's own cleanup (watch manager teardown) is skipped too. + """ + sampler = ProcessSampler(interval_seconds=0.001) + sampler.start() + + async def _boom() -> None: + raise RuntimeError("psutil exploded") + + running = sampler._task + assert running is not None + running.cancel() + with contextlib.suppress(asyncio.CancelledError, BaseException): + await running + sampler._task = asyncio.get_running_loop().create_task(_boom()) + await asyncio.sleep(0) + + with pytest.raises(RuntimeError, match="psutil exploded"): + await sampler.stop() + + assert sampler._uses_managed_tracing is False diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py index 0f263621..d35e397c 100644 --- a/tests/performance/test_replay.py +++ b/tests/performance/test_replay.py @@ -500,3 +500,31 @@ async def virtual_sleep(delay: float) -> None: assert report.failures_injected["slow_logs"] == 1 # A slow log stream adds no extra sleep to the resource schedule. assert sum(sleep_delays) == pytest.approx(events[-1].offset_seconds) + + +async def test_replay_does_not_re_mark_bursts_after_a_watch_reconnect() -> None: + """A reconnect must not replay burst boundaries that already passed. + + `_ReplaySource` restarts at `_next_event_index` on a new generation. With + the burst cursor reset to zero, the first event after a 410 re-marks every + burst that already ended, producing duplicate and time-shifted drain + samples for a run that had exactly one burst. + """ + profile = WorkloadProfile( + schema_version=1, + id="burst-reconnect", + seed=186, + object_count=50, + namespace_count=5, + steady_events_per_second=5, + duration_seconds=4, + bursts=(Burst(start_second=1, duration_seconds=1, events_per_second=40),), + # A 410 forces the watch manager to drop and re-list mid-schedule, + # after the burst window has already closed. + failures=(FailureInjection(kind="gone", at_event=50),), + ) + + report = await run_replay(profile, ReplayOptions(time_scale=0)) + + assert report.api.reconnects == 1 # the schedule really was interrupted + assert len(report.phases.post_burst_drain_seconds) == len(profile.bursts) From ade1fd06cce131afd4beb508ec52bf1aa3041fc1 Mon Sep 17 00:00:00 2001 From: hellices Date: Fri, 7 Aug 2026 01:20:36 +0900 Subject: [PATCH 37/38] docs: publish the measured 1,000-pod scale envelope The qualification design says main carries compact baseline and optimized summaries plus the supported scale envelope and known limits. It did not. This adds them from the recorded live runs rather than leaving the numbers only in the issue thread. Records both budgets that pass and the two that miss: event-to-render p95 (299ms against a 250ms budget, but measured at 24 ev/s against a budget written at 20) and cursor-input p95 (2.4s against 100ms, more than 20x over and untouched by the render work). Also flags what is not trustworthy yet: the UI-at-scale interaction timings predate the harness fix that makes those scenarios wait for the target state, so they are upper bounds taken under CPU saturation, not measurements. Refs #186 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- README.md | 4 ++ docs/performance.md | 114 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 118 insertions(+) create mode 100644 docs/performance.md diff --git a/README.md b/README.md index 3f5bb7e5..54ced954 100644 --- a/README.md +++ b/README.md @@ -112,6 +112,10 @@ Full key reference: [docs/keybindings.md](docs/keybindings.md). - **[Air-gapped operation](docs/airgap.md)** — internal LLM/Helm/OLM/image endpoints, corporate CA trust (`network.ca_bundle`, Helm `--ca-file`), responsibility boundaries, and a readiness checklist. +- **[Performance and scale](docs/performance.md)** — the measured envelope + (1,000 pods at 24 watch events/second for 31 minutes against a real + cluster), the budgets that pass and the two that miss, and the known + limits — including which size makes interaction feel sluggish. - **[Threat model](docs/threat-model.md)** — exactly what crosses the embedded-provider boundary, what is redacted, the MCP and plugin trust boundaries, and the residual risks that are not mitigated. See diff --git a/docs/performance.md b/docs/performance.md new file mode 100644 index 00000000..4fa51c8c --- /dev/null +++ b/docs/performance.md @@ -0,0 +1,114 @@ +# Performance and scale envelope + +What korvid has actually been measured doing, on what hardware, against which +cluster. Every number here comes from a recorded run whose manifest pins the +cluster, the workload profile, and the korvid commit. Nothing here is +extrapolated: a scale that is not listed has not been measured. + +Reproduce any of it with the benchmark command described in +[the qualification design](dev/specs/2026-08-06-large-cluster-performance-qualification-design.md). + +## Supported envelope + +| | Measured | +|---|---| +| Objects in one view | 1,000 pods across 20 namespaces | +| Sustained churn | 24 watch events/second for 31 minutes | +| Session length | 31 minutes unattended, no reconnect, no relist | +| Correctness | 43,200/43,200 events applied, final digest matches, 0 dropped updates | + +Replay-only profiles reach 10,000 and 50,000 objects. Those establish that the +store and diff paths do not fall over at that size; they are **not** a claim +that the live budgets below hold there, because they do not exercise the API +server, the watch decoder, or a real terminal. + +## Live 1,000-pod result + +Run `i186-20260806-195353` against a dedicated AKS cluster (Kubernetes 1.35.6, +5 × `Standard_D4s_v5`), driven from macOS arm64 / Python 3.12 / 10 cores. +Measured before and after the render-path work described below. + +| Metric | Budget | Baseline | Optimized | | +|---|---:|---:|---:|---| +| Dropped updates | 0 | 0 | 0 | pass | +| Final digest mismatch | 0 | 0 | 0 | pass | +| LIST to 1,000-row table | ≤ 2 s | 1.12 s | 1.19 s | pass | +| Process start to interactive | ≤ 10 s | 2.75 s | 2.88 s | pass | +| Peak RSS | ≤ 512 MiB | 276 MiB | 271 MiB | pass | +| Post-warm-up RSS slope | ≤ 1 MiB/min | 1.06 MiB/min | 0.53 MiB/min | fixed | +| Event-to-render p95 | ≤ 250 ms @ 20 ev/s | 527 ms @ 24 ev/s | 299 ms @ 24 ev/s | **miss** | +| Cursor-input p95 | ≤ 100 ms | 2,311 ms | 2,447 ms | **miss** | + +Supporting numbers: event-to-render p50 266 → 156 ms, p99 659 → 356 ms, max +1,628 → 714 ms; max backlog depth 42 → 39; achieved churn 23.18 → 23.996 ev/s +against a 24.0 target. + +Render passes went *up* 51% (3,640 → 5,493) while latency went *down* 43%. +Before the change each pass was expensive enough that the update coalescer was +swallowing work to keep up; afterwards each pass is cheap enough to run more +often, so events reach the screen sooner and the backlog stays shallower. That +is also why achieved churn only reaches its 24.0 ev/s target after the change — +the event driver was previously being back-pressured by the renderer. + +### What made it faster + +A 31-minute CPU profile of the baseline run (1,877 s of samples, 2.5 B calls) +showed the cost was per-row-per-frame work on 1,000 rows, in three places: + +- the in-place table diff read every surviving row back out of the `DataTable` + to decide whether it had changed — 7.35 M `get_row` calls and 102 M cell + comparisons per run, answering a question the writer already knew; +- `format_age` re-parsed the same ~1,000 timestamp strings every frame, pulling + `dateutil` into the hot loop for 96 s combined; +- `phase_style` recomputed a small closed set of styles 7.3 M times. + +All three are now memoised. The one thing deliberately *not* cached is the +`Text` object returned for a phase cell: `DataTable` takes ownership of it and +mutates it, so a shared instance would corrupt unrelated rows. + +## Known limits + +**Cursor input is the binding constraint, not rendering.** Input +acknowledgement p95 sits above 2 s at 1,000 objects under full churn, more than +20× its budget, and the render-path work above did not move it. The client is +CPU-saturated (peak ~99.7%) at this size, so the render win buys headroom +rather than removing the ceiling. Treat 1,000 objects under sustained churn as +the point where interaction becomes visibly sluggish. + +**Event-to-render p95 misses its budget, but the budget and the measurement do +not line up.** The budget is written at 20 events/s; the live profile runs at +24. The optimized 299 ms is a miss at the higher rate and has not been +re-measured at 20. + +**UI-at-scale interaction timings are not yet trustworthy.** Filter, split-pane +and multi-log key sequences took seconds, not milliseconds, in both runs — but +both runs predate the harness fix that makes those scenarios wait for the +target UI state instead of for the keystroke to return. The recorded values are +upper bounds taken while the app was CPU-saturated, not clean measurements, and +they need re-running on the fixed harness before they mean anything. + +**Burst drain is unmeasured live.** The 3-second post-burst drain budget is +exercised in replay only; the live profile contains no burst. + +**Memory is not a limit at this size.** End-of-run allocation snapshots show no +unbounded growth attributable to korvid — the retained set is dominated by +transient watch-decode buffers and rich text fragments. The 0.53 MiB/min slope +over 31 minutes against a 271 MiB peak is drift, not a leak. + +## Long sessions against AKS + +Exec-plugin credentials (`kubelogin`, and any other +`client.authentication.k8s.io` provider) expire mid-session. korvid refreshes +them just before expiry and propagates the new token to every client, including +the websocket clients used for exec, logs, and port-forward. Without this a +watch established at connect time dies with HTTP 401 partway through — first +observed at ~22 minutes into a 30-minute run. If you see a session drop with +401 after a long idle period, that is a bug worth reporting, not expected +behavior. + +## Raw artifacts + +Each run emits a summary, a metrics JSON, a `cProfile` dump, a `tracemalloc` +snapshot, and the seed manifest that reproduces the exact workload. Those are +kept out of the product source history; issue #186 carries the run summaries +and links the artifacts. From c298a9fa8dded9515252629b6e21fbbf8d1c967c Mon Sep 17 00:00:00 2001 From: hellices Date: Fri, 7 Aug 2026 08:32:22 +0900 Subject: [PATCH 38/38] chore: retrigger CI after Actions outage The CI run for ade1fd0 has been stuck in queued for ~12 hours since the GitHub Actions major outage on 2026-08-06; its jobs API returns zero jobs and it can neither be cancelled nor re-run. Pushing an empty commit to get a clean run now that hosted runner queues have drained. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>