diff --git a/README.md b/README.md index 3f5bb7e5..54ced954 100644 --- a/README.md +++ b/README.md @@ -112,6 +112,10 @@ Full key reference: [docs/keybindings.md](docs/keybindings.md). - **[Air-gapped operation](docs/airgap.md)** — internal LLM/Helm/OLM/image endpoints, corporate CA trust (`network.ca_bundle`, Helm `--ca-file`), responsibility boundaries, and a readiness checklist. +- **[Performance and scale](docs/performance.md)** — the measured envelope + (1,000 pods at 24 watch events/second for 31 minutes against a real + cluster), the budgets that pass and the two that miss, and the known + limits — including which size makes interaction feel sluggish. - **[Threat model](docs/threat-model.md)** — exactly what crosses the embedded-provider boundary, what is redacted, the MCP and plugin trust boundaries, and the residual risks that are not mitigated. See diff --git a/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md new file mode 100644 index 00000000..600f0f8a --- /dev/null +++ b/docs/dev/specs/2026-08-06-large-cluster-performance-qualification-design.md @@ -0,0 +1,381 @@ +# Large-Cluster Performance Qualification Design + +**Issue:** #186 + +**Status:** Approved for planning + +## Goal + +Establish a reproducible large-cluster benchmark for korvid, validate it against +real load in the dedicated test AKS cluster, and optimize only bottlenecks +demonstrated by the resulting traces. + +The work is complete when korvid has a published 1,000-Pod support envelope, +repeatable replay coverage at larger scales, and before/after evidence for every +runtime optimization made under this issue. + +## Decision + +Use a hybrid, measurement-first qualification program: + +1. deterministic replay exercises the real store, watch, app, and table paths at + 1,000, 10,000, and 50,000 objects; +2. a guarded live run creates 1,000 real Pods on the existing + `aks-korvid-contract-test` cluster; +3. profiling identifies the dominant bottleneck; +4. one bottleneck is optimized at a time with a correctness regression test and + a deterministic performance comparison; +5. the same live workload is rerun before the support envelope is published. + +This is preferred over a live-only benchmark, which is too variable and costly +for regression testing, and over replay-only testing, which cannot validate API +server, network, watch-recovery, or scheduling behavior. + +## Scope and sequencing + +#186 remains the only selected product issue until its qualification cycle is +finished. The work is split into reviewable deliverables rather than one large +change: + +1. **Benchmark foundation:** deterministic profiles, metrics, reports, and + guarded AKS workload lifecycle. +2. **Baseline qualification:** deterministic results plus the first live + 1,000-Pod run and profile. +3. **Evidence-gated optimization:** a focused TDD change for each material + bottleneck, beginning with the largest measured contributor. +4. **Requalification:** identical replay and AKS runs, final budgets, support + envelope, and documentation. + +The benchmark foundation may be merged independently. Runtime changes that +overlap PR #197, especially changes in `ui/app.py`, begin only after PR #197 is +merged and the issue worktree is synchronized with the resulting `main`. + +## Benchmark architecture + +### Workload profiles + +Profiles are versioned JSON documents with a schema version, seed, object count, +namespace count, churn rate, burst pattern, failure injections, and duration. +Schema v1, as reviewed in `tests/performance/profile.py`, fixes the initial +state to all Pods already Running and Ready for the chosen object count; varied +initial-state distributions require a future schema version with an explicit +field. A resolved run manifest records the profile plus the korvid SHA, Python, +Textual, OS, CPU, memory, Kubernetes, cluster, and node-pool versions. + +The initial profiles are: + +| Profile | Objects | Purpose | Gate | +|---|---:|---|---| +| `smoke-1k` | 1,000 | Fast deterministic correctness and report smoke | Normal CI, no wall-clock assertion | +| `steady-10k` | 10,000 | Store/render scaling and sustained churn | Manual or scheduled | +| `burst-50k` | 50,000 | Upper-envelope and burst-backlog measurement | Manual or scheduled | +| `aks-1k` | 1,000 | Deterministic comparison schedule for the live topology | Manual or scheduled | +| `aks-live-1k` | 1,000 Pods | Real API, network, LIST/WATCH, and UI qualification | Protected manual run | + +`aks-live-1k` (`tests/performance/profiles/aks-live-1k.json`) encodes the live +sequence below exactly - 1,800 seconds at 20 events/s with three 30-second +bursts at 100 events/s - so the event-to-render, backlog-drain, and RSS-slope +budgets in this document are measurable by a single protected run: + +```bash +uv run python -m tests.performance.cli replay-live \ + --profile tests/performance/profiles/aks-live-1k.json \ + --context aks-korvid-contract-test \ + --expected-cluster-id --run-id \ + --json -live.json --out -live.md \ + --cpu-profile -live.pstats \ + --allocation-snapshot -live-alloc.txt +``` + +All four artifacts are mandatory and every filename must contain the run id: +a live run whose evidence is missing or untraceable is rejected before the +cluster is touched, and two destinations that resolve to the same file are +rejected as well. + +`aks-1k` keeps the short (30-second) schedule shared with `burst-50k`, so a +live run can be compared against the deterministic 1k/10k/50k baselines on the +same event schedule. `--duration` shortens a live smoke run; every burst and +failure point is re-validated against the shortened duration before the +cluster identity and ownership gates run. + +The deterministic generator emits stable names, namespaces, and event order +from the profile seed. Repeating a profile with the same seed must produce the +same object and event hashes. Uids and resource versions are deliberately *not* +part of the deterministic workload: the synthetic generator does not assign +them, and on a live run they are whatever the cluster issued (the live path +reads each seeded Pod's real uid and pins it as the guard precondition). + +### Runtime path + +Replay and live runs exercise the same application path: + +```text +LIST/WATCH or replay source + -> WatchManager + -> ResourceStore + -> ResourcesUpdated coalescing + -> KorvidApp render + -> ResourceTable +``` + +The benchmark does not replace these components with a simplified table model. +A headless Textual pilot drives the real app and records cursor-response latency +while updates are active. + +Instrumentation is observational. Production defaults remain unchanged, and no +benchmark dependency is required at normal runtime. Cross-platform RSS and CPU +sampling uses `psutil` as a development dependency; benchmark reports also +include Python allocation samples so process growth can be separated from +Python heap growth. + +### Metrics + +Every run emits JSON and Markdown containing: + +- process start to interactive app; +- LIST completion to first fully populated table; +- event receipt to reflected table state, p50/p95/p99 and maximum; +- cursor-input acknowledgement, p50/p95/p99 and maximum; +- update backlog depth and time to drain after each burst; +- process CPU, RSS, peak RSS, and post-warm-up RSS slope; +- Python allocation growth by source location; +- logical LIST/WATCH/GET counts, decoded response bytes, watch events, + reconnects, `410 Gone` relists, throttles, and authorization failures; +- rendered, coalesced, and dropped update counts; +- final store/table digest, which must match the expected workload digest. + +Samples use monotonic clocks. Normal unit tests assert metric semantics and +operation counts, never machine-dependent wall-clock thresholds. + +### API-load accounting + +The benchmark records logical operations at the `KubeClient` boundary and event +payload bytes after decoding. It distinguishes initial LIST, long-lived WATCH, +documented reconnect/re-LIST, user-requested GET, and unexpected GET. + +A passive resource view must use one LIST followed by one WATCH for each active +`(kind, scope)` pair. Object count must not increase GET count. Reconnects and +re-LISTs are reported separately rather than hidden inside aggregate request +counts. + +## Live AKS qualification + +### Fixed target and capacity + +The live target is only: + +- resource group `rg-korvid-contract-test`; +- cluster `aks-korvid-contract-test`; +- tags `purpose=korvid-contract-testing` and + `production-use=prohibited`; +- Kubernetes context `aks-korvid-contract-test`. + +The cluster currently has a stopped `Standard_D2s_v5` system node and a +zero-node `Standard_D2s_v5` workload pool. Korea Central has 100 available +DSv5-family vCPUs and 116 regional vCPUs. + +Create or reuse a user pool named `perftest` with: + +- `Standard_D4s_v5`; +- five nodes during the run and zero while idle; +- `maxPods=250`; +- label `korvid.dev/pool=perftest`; +- taint `korvid.dev/performance=true:NoSchedule`. + +Five nodes provide room for 1,000 test Pods plus required DaemonSets without +running at the 250-Pod-per-node ceiling. They also provide 20 vCPUs and 80 GiB +of memory while remaining well inside the verified quota. + +### Workload + +The workload generator creates 20 labelled namespaces named +`korvid-perf--0` through `korvid-perf--19`, with 50 +Pods in each. Pods use `registry.k8s.io/pause:3.10`, select the `perftest` pool, +tolerate only the performance taint, and request 5 millicores and 16 MiB. Every +namespace and Pod has: + +- `app.kubernetes.io/managed-by=korvid-performance`; +- a unique `korvid.dev/performance-run` value; +- the workload profile and seed. + +The live sequence is: + +1. start the stopped cluster and scale `perftest` to five; +2. run the janitor for stale, labelled performance namespaces; +3. create all 1,000 Pods and require exactly 1,000 Running, Ready Pods before + measuring, matching schema v1's fixed initial state; +4. capture cold startup and initial LIST/render metrics; +5. run 30 minutes of metadata-only watch churn at 20 events per second; +6. inject three 30-second bursts at 100 events per second during that window; +7. drive filter, sort, namespace switch, split pane, describe, and cursor input; +8. collect final digests, profiles, API counts, and cluster diagnostics; +9. delete only the run-labelled namespaces and verify they are gone; +10. scale `perftest` to zero and stop the cluster in an independent cleanup + path that runs even after benchmark failure or timeout. + +Metadata-only updates create real watch traffic without restarting containers +or changing the workload's resource demand. Churn writes one dedicated, +non-ownership label (`korvid.dev/performance-tick`) on the Pod's own metadata +under a JSON Patch that first `test`s the Pod UID and both ownership labels. +The kubelet-owned `status` subresource is never written: an externally patched +`status.phase` is reconciled back on the next node sync, which would both +corrupt digest parity and violate the metadata-only rule above. + +The generator rate and observed API throttling are both recorded; requested +rate is never reported as achieved rate. Reports carry the requested event +count and rate next to the observed event count, churn wall time, achieved +rate, and the count of mutation-side 429 retries, which is accounted +separately from application read-path throttles. + +Live churn is driven with explicit bounded concurrency and a bounded per-patch +timeout: a serial driver is capped at one round trip per event and cannot +approach the scheduled rate, which would silently understate the load the +report claims to have applied. Only HTTP 429 is retried, with bounded +target-specific jitter that honors the API server's `Retry-After` hint up to an +explicit delay ceiling and re-issues the identical guarded patch. + +### Guardrails + +Every mutating command fails closed unless the resource group, cluster name, +test-only tags, context, pool labels, namespace prefix, and run labels all +match. Cleanup lists the exact owned objects before deletion and refuses +unlabelled resources. + +The system pool is never targeted. Existing contract-test namespaces are not +shared with the performance workload. The stop-cluster action is independent of +the benchmark job so a timeout cannot leave compute running. + +## Failure profiles + +Failure behavior is deterministic first and live only when safe: + +- `410 Gone` forces re-LIST and validates that deleted objects do not remain; +- throttling validates bounded backoff and request accounting; +- partial RBAC denial validates one reported denial without namespace fan-out; +- unavailable metrics validates responsive resource navigation without the + metrics poller; +- slow API responses validate bounded input latency and visible backlog; +- slow log streams validate that resource watch/render progress is independent + of log consumption. + +The first live qualification covers normal LIST/WATCH plus real throttling if it +occurs naturally. Synthetic failure injection stays in replay until the normal +live path is stable, avoiding unnecessary API-server disruption. + +## Evidence-gated optimization + +The baseline is profiled before runtime code changes. A candidate is material +when it either: + +- misses a hard budget; +- contributes at least 25% of sampled CPU time during the affected phase; +- causes unbounded growth with object count, churn rate, or duration; or +- creates unexpected per-object API requests. + +The largest material contributor is addressed first. Each optimization must: + +1. add a correctness regression test that fails on the original behavior; +2. add or extend a deterministic benchmark comparison; +3. preserve cursor, viewport, filtering, sorting, split-pane, and recovery + behavior; +4. improve its target metric by at least 20% over the median of three + deterministic runs; +5. introduce no greater than 10% regression in another published performance + metric; +6. pass the full repository gate; +7. improve or preserve the same target in the repeated live run. + +Likely hot paths such as full-store sorting, full-row table diffing, render +message cadence, and hierarchy refresh are hypotheses only. They are not +changed unless the profile identifies them. + +If the first baseline meets every hard budget and no material contributor is +found, no speculative optimization is made. The measured support envelope is +still published. + +## Initial hard budgets + +These budgets define usable behavior for the live 1,000-Pod profile: + +| Metric | Budget | +|---|---:| +| Dropped updates | 0 | +| Final store/table digest mismatch | 0 | +| Unexpected per-object GETs for passive view | 0 | +| LIST completion to 1,000-row table | <= 2 seconds | +| Process start to interactive 1,000-row table | <= 10 seconds | +| Event-to-render p95 at 20 events/s | <= 250 ms | +| Cursor-input p95 during steady churn | <= 100 ms | +| Backlog drain after a 100 events/s burst | <= 3 seconds | +| Post-warm-up RSS slope over 30 minutes | <= 1 MiB/minute | +| Peak RSS at 1,000 Pods | <= 512 MiB | +| Failed UI-at-scale scenarios | 0 | + +`drive_ui_scenarios` records a key sequence that never reached its target state +as `ScenarioResult(ok=False)` rather than raising, so `replay-live` folds those +outcomes into its exit status: a run that produced no UI-at-scale evidence fails +instead of reporting success. + +The final document records both observed values and budgets. A budget may be +changed only with measured rationale in review; baseline updates never silently +overwrite historical results. The 10,000- and 50,000-object replay profiles +define measured upper envelopes, not claims that the same live budgets apply. + +## Results and documentation + +Main contains: + +- workload schemas and profiles; +- benchmark command and tests; +- reproducibility methodology; +- compact baseline and optimized summaries; +- the supported scale envelope and known limits. + +Large raw JSON samples, process traces, allocation snapshots, and cluster +diagnostics are committed to a separate `benchmark-results` branch. Issue #186 +links the immutable artifact commit and summarizes the run. Raw artifacts are +not added to the product source history. + +Reports never imply validation beyond the exact profile, cluster, and commit +recorded in the run manifest. + +## Test strategy + +- Schema tests reject unknown versions, invalid rates, impossible namespace + distributions, and unsafe live targets. +- Generator tests prove deterministic hashes and exact event counts. +- Metric tests use a fake monotonic clock and fixed process samples. +- Replay tests exercise 1,000 objects without timing assertions and verify + final digest, zero drops, bounded API operations, and report shape. +- Textual tests drive cursor input during controlled churn and verify + acknowledgement accounting. +- AKS lifecycle tests prove guard failures and label-scoped cleanup with fakes. +- Protected live execution verifies the 1,000-Pod workload and records timing + rather than asserting cloud wall-clock behavior inside the normal PR suite. + +## Acceptance criteria + +- The same seed produces identical object and event hashes. +- Replay runs at 1,000, 10,000, and 50,000 objects emit complete JSON and + Markdown reports. +- The protected AKS run has 1,000 simultaneously visible Pods on five + performance nodes and completes the 30-minute churn profile. +- Passive viewing shows bounded LIST/WATCH behavior with no per-object GET + fan-out. +- Reports include latency distributions, CPU/RSS, memory slope, backlog, + reconnects, throttles, dropped updates, and final digests. +- Every material bottleneck is either optimized with before/after evidence or + documented with a reason it is not safe or valuable to change. +- The final live rerun satisfies the hard budgets or publishes the missed budget + as an explicit unsupported limit. +- All benchmark namespaces are removed, `perftest` is at zero nodes, and the + test cluster is stopped after each live run. + +## Non-goals + +- Maintaining a permanently running large AKS cluster. +- Claiming production-scale support beyond the tested profiles. +- Adding wall-clock assertions to normal unit tests. +- Optimizing suspected hot paths before profiling. +- Running destructive failure injection against the AKS control plane. +- Starting another product issue before #186 reaches its documented outcome. diff --git a/docs/performance.md b/docs/performance.md new file mode 100644 index 00000000..4fa51c8c --- /dev/null +++ b/docs/performance.md @@ -0,0 +1,114 @@ +# Performance and scale envelope + +What korvid has actually been measured doing, on what hardware, against which +cluster. Every number here comes from a recorded run whose manifest pins the +cluster, the workload profile, and the korvid commit. Nothing here is +extrapolated: a scale that is not listed has not been measured. + +Reproduce any of it with the benchmark command described in +[the qualification design](dev/specs/2026-08-06-large-cluster-performance-qualification-design.md). + +## Supported envelope + +| | Measured | +|---|---| +| Objects in one view | 1,000 pods across 20 namespaces | +| Sustained churn | 24 watch events/second for 31 minutes | +| Session length | 31 minutes unattended, no reconnect, no relist | +| Correctness | 43,200/43,200 events applied, final digest matches, 0 dropped updates | + +Replay-only profiles reach 10,000 and 50,000 objects. Those establish that the +store and diff paths do not fall over at that size; they are **not** a claim +that the live budgets below hold there, because they do not exercise the API +server, the watch decoder, or a real terminal. + +## Live 1,000-pod result + +Run `i186-20260806-195353` against a dedicated AKS cluster (Kubernetes 1.35.6, +5 × `Standard_D4s_v5`), driven from macOS arm64 / Python 3.12 / 10 cores. +Measured before and after the render-path work described below. + +| Metric | Budget | Baseline | Optimized | | +|---|---:|---:|---:|---| +| Dropped updates | 0 | 0 | 0 | pass | +| Final digest mismatch | 0 | 0 | 0 | pass | +| LIST to 1,000-row table | ≤ 2 s | 1.12 s | 1.19 s | pass | +| Process start to interactive | ≤ 10 s | 2.75 s | 2.88 s | pass | +| Peak RSS | ≤ 512 MiB | 276 MiB | 271 MiB | pass | +| Post-warm-up RSS slope | ≤ 1 MiB/min | 1.06 MiB/min | 0.53 MiB/min | fixed | +| Event-to-render p95 | ≤ 250 ms @ 20 ev/s | 527 ms @ 24 ev/s | 299 ms @ 24 ev/s | **miss** | +| Cursor-input p95 | ≤ 100 ms | 2,311 ms | 2,447 ms | **miss** | + +Supporting numbers: event-to-render p50 266 → 156 ms, p99 659 → 356 ms, max +1,628 → 714 ms; max backlog depth 42 → 39; achieved churn 23.18 → 23.996 ev/s +against a 24.0 target. + +Render passes went *up* 51% (3,640 → 5,493) while latency went *down* 43%. +Before the change each pass was expensive enough that the update coalescer was +swallowing work to keep up; afterwards each pass is cheap enough to run more +often, so events reach the screen sooner and the backlog stays shallower. That +is also why achieved churn only reaches its 24.0 ev/s target after the change — +the event driver was previously being back-pressured by the renderer. + +### What made it faster + +A 31-minute CPU profile of the baseline run (1,877 s of samples, 2.5 B calls) +showed the cost was per-row-per-frame work on 1,000 rows, in three places: + +- the in-place table diff read every surviving row back out of the `DataTable` + to decide whether it had changed — 7.35 M `get_row` calls and 102 M cell + comparisons per run, answering a question the writer already knew; +- `format_age` re-parsed the same ~1,000 timestamp strings every frame, pulling + `dateutil` into the hot loop for 96 s combined; +- `phase_style` recomputed a small closed set of styles 7.3 M times. + +All three are now memoised. The one thing deliberately *not* cached is the +`Text` object returned for a phase cell: `DataTable` takes ownership of it and +mutates it, so a shared instance would corrupt unrelated rows. + +## Known limits + +**Cursor input is the binding constraint, not rendering.** Input +acknowledgement p95 sits above 2 s at 1,000 objects under full churn, more than +20× its budget, and the render-path work above did not move it. The client is +CPU-saturated (peak ~99.7%) at this size, so the render win buys headroom +rather than removing the ceiling. Treat 1,000 objects under sustained churn as +the point where interaction becomes visibly sluggish. + +**Event-to-render p95 misses its budget, but the budget and the measurement do +not line up.** The budget is written at 20 events/s; the live profile runs at +24. The optimized 299 ms is a miss at the higher rate and has not been +re-measured at 20. + +**UI-at-scale interaction timings are not yet trustworthy.** Filter, split-pane +and multi-log key sequences took seconds, not milliseconds, in both runs — but +both runs predate the harness fix that makes those scenarios wait for the +target UI state instead of for the keystroke to return. The recorded values are +upper bounds taken while the app was CPU-saturated, not clean measurements, and +they need re-running on the fixed harness before they mean anything. + +**Burst drain is unmeasured live.** The 3-second post-burst drain budget is +exercised in replay only; the live profile contains no burst. + +**Memory is not a limit at this size.** End-of-run allocation snapshots show no +unbounded growth attributable to korvid — the retained set is dominated by +transient watch-decode buffers and rich text fragments. The 0.53 MiB/min slope +over 31 minutes against a 271 MiB peak is drift, not a leak. + +## Long sessions against AKS + +Exec-plugin credentials (`kubelogin`, and any other +`client.authentication.k8s.io` provider) expire mid-session. korvid refreshes +them just before expiry and propagates the new token to every client, including +the websocket clients used for exec, logs, and port-forward. Without this a +watch established at connect time dies with HTTP 401 partway through — first +observed at ~22 minutes into a 30-minute run. If you see a session drop with +401 after a long idle period, that is a bug worth reporting, not expected +behavior. + +## Raw artifacts + +Each run emits a summary, a metrics JSON, a `cProfile` dump, a `tracemalloc` +snapshot, and the seed manifest that reproduces the exact workload. Those are +kept out of the product source history; issue #186 carries the run summaries +and links the artifacts. diff --git a/pyproject.toml b/pyproject.toml index cc424512..e3636a5d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -64,6 +64,7 @@ dev = [ "types-PyYAML", "azure-identity>=1.25.3", "types-regex>=2026.7.19.20260720", + "psutil>=6.1", ] [tool.ruff] diff --git a/src/korvid/k8s/client.py b/src/korvid/k8s/client.py index 085bf38d..21b92572 100644 --- a/src/korvid/k8s/client.py +++ b/src/korvid/k8s/client.py @@ -10,7 +10,7 @@ import re from collections.abc import AsyncIterator, Mapping, Sequence from contextlib import AbstractAsyncContextManager, asynccontextmanager -from datetime import datetime +from datetime import datetime, timedelta from typing import Any, cast from urllib.parse import quote, urlencode @@ -40,6 +40,7 @@ from korvid.k8s.metrics import PodMetrics, parse_pod_metrics_list from korvid.k8s.models import GenericSummary, PodSummary, summary_for from korvid.k8s.reads import ReadOps +from korvid.k8s.telemetry import ReadOperation, ReadTelemetry, ReadTelemetryEvent from korvid.k8s.writes import WriteOps logger = logging.getLogger(__name__) @@ -143,13 +144,64 @@ def list_context_names(config_file: str | None = None) -> tuple[list[str], str | #: Bound on the `:ctx` auth probe round trip (issue #36): a wedged target #: cluster must fail the switch quickly instead of hanging the flow. _PROBE_TIMEOUT = 10.0 +_EXEC_CREDENTIAL_REFRESH_SKEW = timedelta(minutes=5) +_EXEC_CREDENTIAL_GENERATION_ATTR = "_korvid_exec_credential_generation" + + +def _exec_credential_expires_soon(loader: object) -> bool: + expiry = getattr(loader, "exec_plugin_expiry", None) + if not isinstance(expiry, datetime): + return False + return expiry - _EXEC_CREDENTIAL_REFRESH_SKEW <= datetime.now(tz=expiry.tzinfo) + + +async def load_refreshable_kube_config( + *, + context: str | None, + client_configuration: k8s_client.Configuration, + persist_config: bool, +) -> None: + """Load kubeconfig and refresh expiring exec credentials before API calls.""" + loader = await k8s_config.load_kube_config( + context=context, + client_configuration=client_configuration, + persist_config=persist_config, + ) + if not isinstance(getattr(loader, "exec_plugin_expiry", None), datetime): + return + + refresh_lock = asyncio.Lock() + refresh_generation = 0 + setattr(client_configuration, _EXEC_CREDENTIAL_GENERATION_ATTR, refresh_generation) + + async def _refresh(configuration: k8s_client.Configuration) -> None: + nonlocal refresh_generation + configuration_generation = getattr(configuration, _EXEC_CREDENTIAL_GENERATION_ATTR, 0) + if configuration_generation == refresh_generation and not _exec_credential_expires_soon( + loader + ): + return + async with refresh_lock: + configuration_generation = getattr(configuration, _EXEC_CREDENTIAL_GENERATION_ATTR, 0) + if configuration_generation < refresh_generation: + await asyncio.wait_for(loader.load_and_set(configuration), _PROBE_TIMEOUT) + setattr(configuration, _EXEC_CREDENTIAL_GENERATION_ATTR, refresh_generation) + elif _exec_credential_expires_soon(loader): + await asyncio.wait_for(loader.load_and_set(configuration), _PROBE_TIMEOUT) + refresh_generation += 1 + setattr(configuration, _EXEC_CREDENTIAL_GENERATION_ATTR, refresh_generation) + + client_configuration.refresh_api_key_hook = _refresh class KubeClient(ReadOps, WriteOps): """Thin wrapper over kubernetes_asyncio; returns typed summaries.""" def __init__( - self, custom_columns: Mapping[str, tuple[CustomColumn, ...]] | None = None + self, + custom_columns: Mapping[str, tuple[CustomColumn, ...]] | None = None, + *, + read_telemetry: ReadTelemetry | None = None, ) -> None: self._api: k8s_client.ApiClient | None = None self._core_v1: k8s_client.CoreV1Api | None = None @@ -166,6 +218,62 @@ def __init__( self._pod_resize_supported: bool | None = None #: cloud provider detection result; None until the first lookup. self._provider_info: ProviderInfo | None = None + self._read_telemetry = read_telemetry + + @staticmethod + def _namespaces_path() -> str: + return "/api/v1/namespaces" + + @staticmethod + def _pods_path(namespace: str | None) -> str: + if namespace is None: + return "/api/v1/pods" + return f"/api/v1/namespaces/{_path_segment(namespace)}/pods" + + def _observe_read( + self, + operation: ReadOperation, + path: str, + *, + payload: object | None = None, + object_count: int = 0, + status: int | None = None, + ) -> None: + """Report one read to the optional telemetry seam (issue #186). + + `decoded_bytes` is an exact canonical-JSON byte count, not an estimate: + the benchmark's API-load accounting is compared across runs and + profiles, so an approximation (e.g. `len(str(payload))`, which counts + Python `repr` characters and misencodes non-ASCII, `True`, and `None`) + would silently change the reported number without removing the work. + The whole method is inert unless a caller opted into telemetry, and the + measured cost of the exact count is ~8.8 ms for a 1,000-Pod + cluster-wide LIST (0.4% of the 2 s LIST-to-render budget) and ~9 us per + watch event (0.004% of the 250 ms event-to-render budget). + """ + if self._read_telemetry is None: + return + decoded_bytes = 0 + if payload is not None: + decoded_bytes = len( + json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str).encode() + ) + self._read_telemetry( + ReadTelemetryEvent( + operation=operation, + path=path, + decoded_bytes=decoded_bytes, + object_count=object_count, + status=status, + ) + ) + + def _observe_read_error( + self, + path: str, + exc: ApiStatusError | k8s_client.exceptions.ApiException, + ) -> None: + self._observe_read("error", path, status=int(getattr(exc, "status", 0) or 0)) def _pod_summary(self, manifest: dict[str, Any]) -> PodSummary: """PodSummary + configured custom column values (issue #45).""" @@ -184,8 +292,14 @@ def _object_summary(self, meta: ResourceMeta, manifest: dict[str, Any]) -> Gener return dataclasses.replace(summary, custom=evaluate_all(columns, manifest)) async def connect(self, context: str | None = None) -> None: - await k8s_config.load_kube_config(context=context) - self._api = k8s_client.ApiClient() + configuration = k8s_client.Configuration() + await load_refreshable_kube_config( + context=context, + client_configuration=configuration, + persist_config=True, + ) + k8s_client.Configuration.set_default(configuration) + self._api = k8s_client.ApiClient(configuration) self._core_v1 = k8s_client.CoreV1Api(self._api) # A new connection may target a different cluster; discard any # capability discovered against the previous one. @@ -248,8 +362,17 @@ async def switch_context(self, context: str | None) -> None: already-torn-down session forever. """ old_api = self._api - await asyncio.wait_for(k8s_config.load_kube_config(context=context), _PROBE_TIMEOUT) - self._api = k8s_client.ApiClient() + configuration = k8s_client.Configuration() + await asyncio.wait_for( + load_refreshable_kube_config( + context=context, + client_configuration=configuration, + persist_config=True, + ), + _PROBE_TIMEOUT, + ) + k8s_client.Configuration.set_default(configuration) + self._api = k8s_client.ApiClient(configuration) self._core_v1 = k8s_client.CoreV1Api(self._api) # Per-connection caches describe the previous cluster. self._pod_resize_supported = None @@ -260,12 +383,19 @@ async def switch_context(self, context: str | None) -> None: async def list_namespaces(self) -> list[str]: if self._core_v1 is None: raise RuntimeError("connect() first") + path = self._namespaces_path() try: resp = await self._core_v1.list_namespace(_preload_content=False) data = await _to_dict(resp) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise except k8s_client.exceptions.ApiException as exc: + self._observe_read_error(path, exc) raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc - return [item["metadata"]["name"] for item in data.get("items", [])] + items = data.get("items", []) + self._observe_read("list", path, payload=data, object_count=len(items)) + return [item["metadata"]["name"] for item in items] async def detect_cloud_provider(self) -> ProviderInfo: """Detect the cluster's cloud provider from a few nodes (issue #30). @@ -308,12 +438,13 @@ def open_pod_exec( dedicated ``WsApiClient`` is created per session — it shares the kubeconfig ``connect()`` loaded — and closed with the session. """ - if self._core_v1 is None: + if self._core_v1 is None or self._api is None: raise RuntimeError("connect() first") + configuration = self._api.configuration @asynccontextmanager async def _session() -> AsyncIterator[Any]: - ws_api = WsApiClient() + ws_api = WsApiClient(configuration) try: core = k8s_client.CoreV1Api(ws_api) kwargs: dict[str, Any] = { @@ -343,12 +474,19 @@ async def _session() -> AsyncIterator[Any]: async def list_pods(self, namespace: str) -> list[PodSummary]: if self._core_v1 is None: raise RuntimeError("connect() first") + path = self._pods_path(namespace) try: resp = await self._core_v1.list_namespaced_pod(namespace, _preload_content=False) data = await _to_dict(resp) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise except k8s_client.exceptions.ApiException as exc: + self._observe_read_error(path, exc) raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc - return [self._pod_summary(item) for item in data.get("items", [])] + items = data.get("items", []) + self._observe_read("list", path, payload=data, object_count=len(items)) + return [self._pod_summary(item) for item in items] async def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSummary]]: """LIST then watch pods; namespace=None watches cluster-wide.""" @@ -363,6 +501,7 @@ async def _watch_pods_namespaced(self, namespace: str) -> AsyncIterator[tuple[st """Per-namespace pod watch via CoreV1Api (LIST then stream).""" if self._core_v1 is None: raise RuntimeError("connect() first") + path = self._pods_path(namespace) # LIST first: yield pre-existing pods as ADDED and anchor the watch at # the snapshot's resourceVersion so no events are missed between LIST @@ -372,11 +511,17 @@ async def _watch_pods_namespaced(self, namespace: str) -> AsyncIterator[tuple[st try: resp = await self._core_v1.list_namespaced_pod(namespace, _preload_content=False) data = await _to_dict(resp) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise except k8s_client.exceptions.ApiException as exc: + self._observe_read_error(path, exc) raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc + items = data.get("items", []) + self._observe_read("list", path, payload=data, object_count=len(items)) resource_version: str | None = (data.get("metadata") or {}).get("resourceVersion") - for item in data.get("items", []): + for item in items: yield ("ADDED", self._pod_summary(item)) watch_kwargs: dict[str, Any] = {} @@ -384,16 +529,23 @@ async def _watch_pods_namespaced(self, namespace: str) -> AsyncIterator[tuple[st watch_kwargs["resource_version"] = resource_version w = k8s_watch.Watch() + self._observe_read("watch_open", path) try: async with w.stream( self._core_v1.list_namespaced_pod, namespace, **watch_kwargs ) as stream: async for event in stream: + raw_object = event["raw_object"] + self._observe_read("watch_event", path, payload=raw_object, object_count=1) yield ( str(event["type"]), - self._pod_summary(event["raw_object"]), + self._pod_summary(raw_object), ) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise except k8s_client.exceptions.ApiException as exc: + self._observe_read_error(path, exc) raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc async def _watch_pods_cluster(self) -> AsyncIterator[tuple[str, PodSummary]]: @@ -401,11 +553,17 @@ async def _watch_pods_cluster(self) -> AsyncIterator[tuple[str, PodSummary]]: if self._api is None: raise RuntimeError("connect() first") - path = "/api/v1/pods" - data = await self._request_json(path) + path = self._pods_path(None) + try: + data = await self._request_json(path) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise + items = data.get("items", []) + self._observe_read("list", path, payload=data, object_count=len(items)) resource_version: str | None = (data.get("metadata") or {}).get("resourceVersion") - for item in data.get("items", []): + for item in items: yield ("ADDED", self._pod_summary(item)) watch_kwargs: dict[str, Any] = {} @@ -414,14 +572,21 @@ async def _watch_pods_cluster(self) -> AsyncIterator[tuple[str, PodSummary]]: watch_func = self._make_raw_watch_callable(path) w = k8s_watch.Watch() + self._observe_read("watch_open", path) try: async with w.stream(watch_func, **watch_kwargs) as stream: async for event in stream: + raw_object = event["raw_object"] + self._observe_read("watch_event", path, payload=raw_object, object_count=1) yield ( str(event["type"]), - self._pod_summary(event["raw_object"]), + self._pod_summary(raw_object), ) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise except k8s_client.exceptions.ApiException as exc: + self._observe_read_error(path, exc) raise ApiStatusError(int(exc.status or 0), str(exc.reason or "")) from exc def _list_path(self, meta: ResourceMeta, namespace: str | None) -> str: @@ -431,6 +596,42 @@ def _list_path(self, meta: ResourceMeta, namespace: str | None) -> str: return f"{meta.api_base}/namespaces/{_path_segment(namespace)}/{meta.plural}" return f"{meta.api_base}/{meta.plural}" + async def _initial_object_snapshot( + self, meta: ResourceMeta, namespace: str | None + ) -> tuple[str, str | None, list[GenericSummary], dict[str, GenericSummary]]: + list_path = self._list_path(meta, namespace) + try: + data = await self._request_json(list_path) + except ApiStatusError as exc: + self._observe_read_error(list_path, exc) + raise + + items = data.get("items", []) + self._observe_read("list", list_path, payload=data, object_count=len(items)) + resource_version = (data.get("metadata") or {}).get("resourceVersion") + summaries: list[GenericSummary] = [] + known: dict[str, GenericSummary] = {} + for item in items: + summary = self._object_summary(meta, item) + summaries.append(summary) + known[f"{summary.namespace}/{summary.name}"] = summary + return list_path, resource_version, summaries, known + + def _watch_objects_requires_poll_fallback( + self, + list_path: str, + exc: ApiStatusError | k8s_client.exceptions.ApiException, + ) -> bool: + status = int(getattr(exc, "status", 0) or 0) + self._observe_read_error(list_path, exc) + if status == 405: + return True + raise ApiStatusError( + status, + str(getattr(exc, "reason", "") or ""), + str(getattr(exc, "body", "") or ""), + ) from exc + async def watch_objects( self, meta: ResourceMeta, namespace: str | None ) -> AsyncIterator[tuple[str, GenericSummary]]: @@ -450,14 +651,10 @@ async def watch_objects( raise RuntimeError("connect() first") # LIST phase -------------------------------------------------------- - list_path = self._list_path(meta, namespace) - data = await self._request_json(list_path) - - resource_version: str | None = (data.get("metadata") or {}).get("resourceVersion") - known: dict[str, GenericSummary] = {} - for item in data.get("items", []): - summary = self._object_summary(meta, item) - known[f"{summary.namespace}/{summary.name}"] = summary + list_path, resource_version, initial_summaries, known = await self._initial_object_snapshot( + meta, namespace + ) + for summary in initial_summaries: yield ("ADDED", summary) if not meta.watchable: @@ -473,33 +670,28 @@ async def watch_objects( watch_func = self._make_raw_watch_callable(list_path) w = k8s_watch.Watch() + self._observe_read("watch_open", list_path) try: async with w.stream(watch_func, **watch_kwargs) as stream: async for event in stream: + raw_object = event["raw_object"] + self._observe_read("watch_event", list_path, payload=raw_object, object_count=1) yield ( str(event["type"]), - self._object_summary(meta, event["raw_object"]), + self._object_summary(meta, raw_object), ) except (k8s_client.exceptions.ApiException, ApiStatusError) as exc: # The raw-watch adapter surfaces HTTP errors as ApiStatusError # (via _raise_for_status); the kubernetes client's own paths # raise ApiException - both carry .status/.reason, and the 405 # fallback must catch both. - status = int(getattr(exc, "status", 0) or 0) - if status != 405: - # Preserve .body: same-status disambiguation (PDB denial vs - # APF throttling) depends on it; ApiException carries one too. - raise ApiStatusError( - status, - str(getattr(exc, "reason", "") or ""), - str(getattr(exc, "body", "") or ""), - ) from exc - # Discovery advertised watch but the server refuses it: as - # deterministic as it gets - poll instead of letting the - # manager burn retries clearing and re-seeding the store. - logger.info("%s rejects watch (405); falling back to LIST polling", meta.plural) - async for event in self._poll_objects(meta, list_path, known): - yield event + if self._watch_objects_requires_poll_fallback(list_path, exc): + # Discovery advertised watch but the server refuses it: as + # deterministic as it gets - poll instead of letting the + # manager burn retries clearing and re-seeding the store. + logger.info("%s rejects watch (405); falling back to LIST polling", meta.plural) + async for event in self._poll_objects(meta, list_path, known): + yield event async def _poll_objects( self, meta: ResourceMeta, list_path: str, known: dict[str, GenericSummary] @@ -513,9 +705,15 @@ async def _poll_objects( """ while True: await asyncio.sleep(LIST_POLL_INTERVAL) - data = await self._request_json(list_path) + try: + data = await self._request_json(list_path) + except ApiStatusError as exc: + self._observe_read_error(list_path, exc) + raise + items = data.get("items", []) + self._observe_read("list", list_path, payload=data, object_count=len(items)) current: dict[str, GenericSummary] = {} - for item in data.get("items", []): + for item in items: summary = self._object_summary(meta, item) current[f"{summary.namespace}/{summary.name}"] = summary yield ("ADDED", summary) @@ -532,14 +730,30 @@ async def list_objects(self, meta: ResourceMeta, namespace: str | None) -> list[ """ if self._api is None: raise RuntimeError("connect() first") - data = await self._request_json(self._list_path(meta, namespace)) - return [self._object_summary(meta, item) for item in data.get("items", [])] + path = self._list_path(meta, namespace) + try: + data = await self._request_json(path) + except ApiStatusError as exc: + self._observe_read_error(path, exc) + raise + items = data.get("items", []) + self._observe_read("list", path, payload=data, object_count=len(items)) + return [self._object_summary(meta, item) for item in items] async def get_object( self, meta: ResourceMeta, namespace: str | None, name: str ) -> dict[str, Any]: """Fetch the raw manifest for a single object. ApiException → ApiStatusError.""" - return await self._request_json(self._object_path(meta, namespace, name)) + path = self._object_path(meta, namespace, name) + try: + result = await self._request_json(path) + except ApiStatusError as exc: + # Every other read path records the failure before propagating; + # without this a denied or throttled GET is a hole in the telemetry. + self._observe_read_error(path, exc) + raise + self._observe_read("get", path, payload=result, object_count=1) + return result # Helm release browsing (issue #28) ---------------------------------- # Releases are Secrets of type helm.sh/release.v1; the synthetic kinds diff --git a/src/korvid/k8s/errors.py b/src/korvid/k8s/errors.py index 43d9e093..1772cd3d 100644 --- a/src/korvid/k8s/errors.py +++ b/src/korvid/k8s/errors.py @@ -10,7 +10,14 @@ class ApiStatusError(Exception): """Raised by the k8s layer when an API request returns an HTTP error status.""" - def __init__(self, status: int, reason: str, body: str = "") -> None: + def __init__( + self, + status: int, + reason: str, + body: str = "", + *, + retry_after_seconds: float | None = None, + ) -> None: super().__init__(f"API {status}: {reason}") self.status = status self.reason = reason @@ -19,3 +26,4 @@ def __init__(self, status: int, reason: str, body: str = "") -> None: #: eviction's PDB denial vs API Priority and Fairness throttling, #: both 429 - inspect it. self.body = body + self.retry_after_seconds: float | None = retry_after_seconds diff --git a/src/korvid/k8s/telemetry.py b/src/korvid/k8s/telemetry.py new file mode 100644 index 00000000..71a52426 --- /dev/null +++ b/src/korvid/k8s/telemetry.py @@ -0,0 +1,19 @@ +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass +from typing import Literal + +ReadOperation = Literal["list", "watch_open", "watch_event", "get", "error"] + + +@dataclass(frozen=True) +class ReadTelemetryEvent: + operation: ReadOperation + path: str + decoded_bytes: int = 0 + object_count: int = 0 + status: int | None = None + + +ReadTelemetry = Callable[[ReadTelemetryEvent], None] diff --git a/tests/k8s/test_client.py b/tests/k8s/test_client.py index d32ff4d1..b0e14019 100644 --- a/tests/k8s/test_client.py +++ b/tests/k8s/test_client.py @@ -1,3 +1,6 @@ +import asyncio +from copy import deepcopy +from datetime import UTC, datetime, timedelta from pathlib import Path from typing import Any from unittest.mock import AsyncMock, MagicMock, patch @@ -12,6 +15,196 @@ from korvid.k8s.discovery import ResourceMeta from korvid.k8s.errors import ApiStatusError from korvid.k8s.models import ReplicaSetSummary +from korvid.k8s.telemetry import ReadTelemetryEvent + + +async def test_load_refreshable_kube_config_refreshes_expired_exec_token( + monkeypatch: pytest.MonkeyPatch, +) -> None: + configuration = k8s_client.Configuration() + configuration.api_key["BearerToken"] = "stale-token" + loader = MagicMock() + loader.exec_plugin_expiry = datetime.now(UTC) - timedelta(seconds=1) + + async def _refresh(target: k8s_client.Configuration) -> None: + target.api_key["BearerToken"] = "fresh-token" + loader.exec_plugin_expiry = datetime.now(UTC) + timedelta(hours=1) + + loader.load_and_set = AsyncMock(side_effect=_refresh) + load_kube_config = AsyncMock(return_value=loader) + monkeypatch.setattr(k8s_config, "load_kube_config", load_kube_config) + + await client_mod.load_refreshable_kube_config( + context="aks", + client_configuration=configuration, + persist_config=False, + ) + + assert await configuration.get_api_key_with_prefix("BearerToken") == "fresh-token" + loader.load_and_set.assert_awaited_once_with(configuration) + + +async def test_load_refreshable_kube_config_serializes_concurrent_refresh( + monkeypatch: pytest.MonkeyPatch, +) -> None: + configuration = k8s_client.Configuration() + configuration.api_key["BearerToken"] = "stale-token" + loader = MagicMock() + loader.exec_plugin_expiry = datetime.now(UTC) - timedelta(seconds=1) + in_flight = 0 + max_in_flight = 0 + + async def _refresh(target: k8s_client.Configuration) -> None: + nonlocal in_flight, max_in_flight + in_flight += 1 + max_in_flight = max(max_in_flight, in_flight) + try: + await asyncio.sleep(0) + target.api_key["BearerToken"] = "fresh-token" + loader.exec_plugin_expiry = datetime.now(UTC) + timedelta(hours=1) + finally: + in_flight -= 1 + + loader.load_and_set = AsyncMock(side_effect=_refresh) + monkeypatch.setattr( + k8s_config, + "load_kube_config", + AsyncMock(return_value=loader), + ) + await client_mod.load_refreshable_kube_config( + context="aks", + client_configuration=configuration, + persist_config=False, + ) + + tokens = await asyncio.gather( + configuration.get_api_key_with_prefix("BearerToken"), + configuration.get_api_key_with_prefix("BearerToken"), + ) + + assert len(tokens) == 2 + assert all(token == "fresh-token" for token in tokens) + assert max_in_flight == 1 + loader.load_and_set.assert_awaited_once_with(configuration) + + +async def test_load_refreshable_kube_config_updates_stale_configuration_copy( + monkeypatch: pytest.MonkeyPatch, +) -> None: + configuration = k8s_client.Configuration() + configuration.api_key["BearerToken"] = "stale-token" + loader = MagicMock() + loader.exec_plugin_expiry = datetime.now(UTC) - timedelta(seconds=1) + + async def _refresh(target: k8s_client.Configuration) -> None: + target.api_key["BearerToken"] = "fresh-token" + loader.exec_plugin_expiry = datetime.now(UTC) + timedelta(hours=1) + + loader.load_and_set = AsyncMock(side_effect=_refresh) + monkeypatch.setattr( + k8s_config, + "load_kube_config", + AsyncMock(return_value=loader), + ) + await client_mod.load_refreshable_kube_config( + context="aks", + client_configuration=configuration, + persist_config=False, + ) + first_copy = deepcopy(configuration) + waiting_copy = deepcopy(configuration) + + assert await first_copy.get_api_key_with_prefix("BearerToken") == "fresh-token" + assert await waiting_copy.get_api_key_with_prefix("BearerToken") == "fresh-token" + assert loader.load_and_set.await_count == 2 + + +async def test_load_refreshable_kube_config_leaves_static_tokens_alone( + monkeypatch: pytest.MonkeyPatch, +) -> None: + configuration = k8s_client.Configuration() + configuration.api_key["BearerToken"] = "static-token" + loader = MagicMock(spec=["load_and_set"]) + loader.load_and_set = AsyncMock() + monkeypatch.setattr( + k8s_config, + "load_kube_config", + AsyncMock(return_value=loader), + ) + + await client_mod.load_refreshable_kube_config( + context="static", + client_configuration=configuration, + persist_config=False, + ) + + assert configuration.refresh_api_key_hook is None + assert await configuration.get_api_key_with_prefix("BearerToken") == "static-token" + loader.load_and_set.assert_not_awaited() + + +async def test_load_refreshable_kube_config_keeps_generation_when_refresh_fails( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A failed refresh must not mark other configuration copies as current.""" + configuration = k8s_client.Configuration() + configuration.api_key["BearerToken"] = "stale-token" + loader = MagicMock() + loader.exec_plugin_expiry = datetime.now(UTC) - timedelta(seconds=1) + attempts: list[k8s_client.Configuration] = [] + + async def _refresh(target: k8s_client.Configuration) -> None: + attempts.append(target) + if len(attempts) == 1: + raise ConnectionError("kubelogin unavailable") + target.api_key["BearerToken"] = "fresh-token" + loader.exec_plugin_expiry = datetime.now(UTC) + timedelta(hours=1) + + loader.load_and_set = AsyncMock(side_effect=_refresh) + monkeypatch.setattr( + k8s_config, + "load_kube_config", + AsyncMock(return_value=loader), + ) + await client_mod.load_refreshable_kube_config( + context="aks", + client_configuration=configuration, + persist_config=False, + ) + failing_copy = deepcopy(configuration) + other_copy = deepcopy(configuration) + + with pytest.raises(ConnectionError, match="kubelogin unavailable"): + await failing_copy.get_api_key_with_prefix("BearerToken") + + assert await other_copy.get_api_key_with_prefix("BearerToken") == "fresh-token" + assert await failing_copy.get_api_key_with_prefix("BearerToken") == "fresh-token" + + +async def test_connect_uses_refreshable_kube_config(monkeypatch: pytest.MonkeyPatch) -> None: + load_refreshable = AsyncMock() + api = MagicMock() + api_factory = MagicMock(return_value=api) + core_v1_factory = MagicMock() + set_default = MagicMock() + monkeypatch.setattr(client_mod, "load_refreshable_kube_config", load_refreshable) + monkeypatch.setattr(k8s_config, "load_kube_config", AsyncMock()) + monkeypatch.setattr(k8s_client, "ApiClient", api_factory) + monkeypatch.setattr(k8s_client, "CoreV1Api", core_v1_factory) + monkeypatch.setattr(k8s_client.Configuration, "set_default", set_default) + + client = KubeClient() + await client.connect("aks") + + load_refreshable.assert_awaited_once() + call = load_refreshable.await_args + assert call is not None + assert call.kwargs["context"] == "aks" + assert call.kwargs["persist_config"] is True + configuration = call.kwargs["client_configuration"] + set_default.assert_called_once_with(configuration) + api_factory.assert_called_once_with(configuration) + core_v1_factory.assert_called_once_with(api) def _pod(name: str, ns: str = "default") -> dict[str, Any]: @@ -85,6 +278,25 @@ def stream(self, func: Any, *args: Any, **kwargs: Any) -> _FakeWatchStream: return _FakeWatchStream(self._events, self.captured_kwargs, self._raise_at, self._raise_exc) +async def test_list_namespaces_emits_list_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespace.return_value = { + "items": [{"metadata": {"name": "default"}}, {"metadata": {"name": "kube-system"}}] + } + + with patch.object(client, "_core_v1", fake_v1): + namespaces = await client.list_namespaces() + + assert namespaces == ["default", "kube-system"] + assert [event.operation for event in seen] == ["list"] + assert seen[0].path == "/api/v1/namespaces" + assert seen[0].object_count == 2 + assert seen[0].decoded_bytes > 0 + assert seen[0].status is None + + async def test_list_pods_parses_summaries() -> None: client = KubeClient() fake_v1 = AsyncMock() @@ -95,6 +307,23 @@ async def test_list_pods_parses_summaries() -> None: fake_v1.list_namespaced_pod.assert_awaited_once_with("default", _preload_content=False) +async def test_list_pods_emits_list_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespaced_pod.return_value = {"items": [_pod("a"), _pod("b")]} + + with patch.object(client, "_core_v1", fake_v1): + pods = await client.list_pods("default") + + assert [pod.name for pod in pods] == ["a", "b"] + assert [event.operation for event in seen] == ["list"] + assert seen[0].path == "/api/v1/namespaces/default/pods" + assert seen[0].object_count == 2 + assert seen[0].decoded_bytes > 0 + assert seen[0].status is None + + async def test_watch_pods_yields_list_items_first() -> None: """Pre-existing pods from the initial LIST appear as ADDED before watch events.""" client = KubeClient() @@ -119,6 +348,57 @@ async def test_watch_pods_yields_list_items_first() -> None: assert collected[2] == ("MODIFIED", "alpha") +async def test_pod_watch_emits_list_open_and_event_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespaced_pod.return_value = { + "metadata": {"resourceVersion": "100"}, + "items": [_pod("listed")], + } + fake_watch = _FakeWatch([{"type": "MODIFIED", "raw_object": _pod("watched")}]) + + with ( + patch.object(client, "_core_v1", fake_v1), + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + ): + collected = [ + (event_type, pod.name) async for event_type, pod in client.watch_pods("default") + ] + + assert collected == [("ADDED", "listed"), ("MODIFIED", "watched")] + assert [event.operation for event in seen] == ["list", "watch_open", "watch_event"] + assert {event.path for event in seen} == {"/api/v1/namespaces/default/pods"} + assert seen[0].object_count == 1 + assert seen[0].decoded_bytes > 0 + assert seen[1].decoded_bytes == 0 + assert seen[2].object_count == 1 + assert seen[2].decoded_bytes > 0 + + +async def test_no_telemetry_preserves_existing_watch_behavior() -> None: + client = KubeClient() + fake_v1 = AsyncMock() + fake_v1.list_namespaced_pod.return_value = { + "metadata": {"resourceVersion": "100"}, + "items": [_pod("listed")], + } + fake_watch = _FakeWatch([]) + + with ( + patch.object(client, "_core_v1", fake_v1), + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + patch( + "korvid.k8s.client.json.dumps", side_effect=AssertionError("unexpected serialization") + ), + ): + collected = [ + (event_type, pod.name) async for event_type, pod in client.watch_pods("default") + ] + + assert collected == [("ADDED", "listed")] + + async def test_watch_pods_passes_resource_version_to_watch() -> None: """resource_version captured from the LIST is forwarded to Watch.stream.""" client = KubeClient() @@ -153,6 +433,49 @@ async def test_watch_pods_list_api_error_raises_api_status_error() -> None: assert exc_info.value.status == 403 +async def test_watch_pods_list_error_emits_error_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespaced_pod.side_effect = ApiException(status=403, reason="Forbidden") + + with ( + patch.object(client, "_core_v1", fake_v1), + pytest.raises(ApiStatusError, match="API 403: Forbidden"), + ): + async for _ in client.watch_pods("default"): + pass + + assert [event.operation for event in seen] == ["error"] + assert seen[0].path == "/api/v1/namespaces/default/pods" + assert seen[0].status == 403 + assert seen[0].decoded_bytes == 0 + assert seen[0].object_count == 0 + + +async def test_watch_pods_watch_error_emits_error_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespaced_pod.return_value = { + "metadata": {"resourceVersion": "100"}, + "items": [], + } + fake_watch = _FakeWatch([], raise_at=0, raise_exc=ApiException(status=410, reason="Gone")) + + with ( + patch.object(client, "_core_v1", fake_v1), + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + pytest.raises(ApiStatusError, match="API 410: Gone"), + ): + async for _ in client.watch_pods("default"): + pass + + assert [event.operation for event in seen] == ["list", "watch_open", "error"] + assert seen[-1].path == "/api/v1/namespaces/default/pods" + assert seen[-1].status == 410 + + async def test_watch_pods_all_namespaces_uses_cluster_path() -> None: """watch_pods(None) LISTs /api/v1/pods without a /namespaces/ segment.""" client = KubeClient() @@ -186,6 +509,25 @@ async def test_list_namespaces_api_error_raises_api_status_error() -> None: await client.list_namespaces() +async def test_list_namespaces_error_emits_error_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + fake_v1 = AsyncMock() + fake_v1.list_namespace.side_effect = ApiException(status=403, reason="Forbidden") + + with ( + patch.object(client, "_core_v1", fake_v1), + pytest.raises(ApiStatusError, match="API 403: Forbidden"), + ): + await client.list_namespaces() + + assert [event.operation for event in seen] == ["error"] + assert seen[0].path == "/api/v1/namespaces" + assert seen[0].status == 403 + assert seen[0].decoded_bytes == 0 + assert seen[0].object_count == 0 + + async def test_list_pods_api_error_raises_api_status_error() -> None: """ApiException must not cross the k8s boundary from list_pods.""" client = KubeClient() @@ -237,6 +579,60 @@ async def test_watch_objects_yields_list_items_first() -> None: assert collected[2] == ("MODIFIED", "dep-a") +async def test_watch_objects_emits_list_open_and_event_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + list_resp = { + "metadata": {"resourceVersion": "200"}, + "items": [_generic("dep-a")], + } + watch_events = [{"type": "MODIFIED", "raw_object": _generic("dep-a")}] + fake_watch = _FakeWatch(watch_events) + + with ( + patch.object(client, "_api", MagicMock()), + patch.object(client, "_request_json", AsyncMock(return_value=list_resp)), + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + ): + collected = [ + (ev, summary.name) async for ev, summary in client.watch_objects(meta, "default") + ] + + assert collected == [("ADDED", "dep-a"), ("MODIFIED", "dep-a")] + assert [event.operation for event in seen] == ["list", "watch_open", "watch_event"] + assert {event.path for event in seen} == {"/apis/apps/v1/namespaces/default/deployments"} + assert seen[0].object_count == 1 + assert seen[0].decoded_bytes > 0 + assert seen[1].decoded_bytes == 0 + assert seen[2].object_count == 1 + assert seen[2].decoded_bytes > 0 + + +async def test_watch_objects_initial_snapshot_reuses_computed_summaries() -> None: + client = KubeClient() + meta = _deploy_meta() + list_resp = { + "metadata": {"resourceVersion": "200"}, + "items": [_generic("dep-a")], + } + fake_watch = _FakeWatch([]) + original_summary = client._object_summary + + with ( + patch.object(client, "_api", MagicMock()), + patch.object(client, "_request_json", AsyncMock(return_value=list_resp)), + patch.object(client, "_object_summary", side_effect=original_summary) as summary_mock, + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + ): + collected = [ + (ev, summary.name) async for ev, summary in client.watch_objects(meta, "default") + ] + + assert collected == [("ADDED", "dep-a")] + assert summary_mock.call_count == 1 + + async def test_watch_objects_replicaset_yields_rich_summary() -> None: """ReplicaSet kinds get ReplicaSetSummary (revision/desired/ready) via summary_for.""" client = KubeClient() @@ -455,6 +851,30 @@ async def test_watch_non_405_api_status_error_still_raises() -> None: assert excinfo.value.body == '{"kind":"Status"}' +async def test_watch_objects_list_error_emits_error_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + + with ( + patch.object(client, "_api", MagicMock()), + patch.object( + client, + "_request_json", + AsyncMock(side_effect=ApiStatusError(401, "Unauthorized")), + ), + pytest.raises(ApiStatusError, match="API 401: Unauthorized"), + ): + async for _ in client.watch_objects(meta, "default"): + pass + + assert [event.operation for event in seen] == ["error"] + assert seen[0].path == "/apis/apps/v1/namespaces/default/deployments" + assert seen[0].status == 401 + assert seen[0].decoded_bytes == 0 + assert seen[0].object_count == 0 + + async def test_watch_non_405_api_exception_still_raises() -> None: """Only the deterministic 405 falls back to polling: other watch errors keep propagating so the WatchManager's retry/report loop stays in charge.""" @@ -473,11 +893,85 @@ async def test_watch_non_405_api_exception_still_raises() -> None: pass +async def test_watch_objects_watch_error_emits_error_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + list_resp: dict[str, Any] = {"metadata": {"resourceVersion": "9"}, "items": []} + fake_watch = _FakeWatch([], raise_at=0, raise_exc=ApiException(status=500, reason="boom")) + + with ( + patch.object(client, "_api", MagicMock()), + patch.object(client, "_request_json", AsyncMock(return_value=list_resp)), + patch("korvid.k8s.client.k8s_watch.Watch", return_value=fake_watch), + pytest.raises(ApiStatusError, match="boom"), + ): + async for _ in client.watch_objects(meta, "default"): + pass + + assert [event.operation for event in seen] == ["list", "watch_open", "error"] + assert seen[-1].path == "/apis/apps/v1/namespaces/default/deployments" + assert seen[-1].status == 500 + + # --------------------------------------------------------------------------- # get_object # --------------------------------------------------------------------------- +async def test_get_object_emits_get_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + request_json_mock = AsyncMock(return_value=_generic("my-dep")) + + with patch.object(client, "_request_json", request_json_mock): + obj = await client.get_object(meta, "default", "my-dep") + + assert obj["metadata"]["name"] == "my-dep" + assert [event.operation for event in seen] == ["get"] + assert seen[0].path == "/apis/apps/v1/namespaces/default/deployments/my-dep" + assert seen[0].object_count == 1 + assert seen[0].decoded_bytes > 0 + assert seen[0].status is None + + +async def test_get_object_emits_error_telemetry_and_reraises() -> None: + """Every other read path records an `error` event before propagating; a + silent GET failure leaves a telemetry gap exactly where a benchmark or an + operator is trying to explain a stalled view.""" + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + request_json_mock = AsyncMock(side_effect=ApiStatusError(404, "Not Found")) + + with ( + patch.object(client, "_request_json", request_json_mock), + pytest.raises(ApiStatusError, match="API 404: Not Found"), + ): + await client.get_object(meta, "default", "my-dep") + + assert [event.operation for event in seen] == ["error"] + assert seen[0].path == "/apis/apps/v1/namespaces/default/deployments/my-dep" + assert seen[0].status == 404 + + +async def test_list_pods_reports_the_same_items_it_counted() -> None: + """The telemetry count and the returned summaries must come from one bound + payload read, not two independent `data.get("items", [])` lookups.""" + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + mock_api = MagicMock() + mock_api.list_namespaced_pod = AsyncMock(return_value={"items": [_pod("a"), _pod("b")]}) + + with patch.object(client, "_core_v1", mock_api): + pods = await client.list_pods("default") + + assert [pod.name for pod in pods] == ["a", "b"] + assert [event.operation for event in seen] == ["list"] + assert seen[0].object_count == len(pods) + + async def test_get_object_raises_api_status_error() -> None: """ApiException from the raw GET is wrapped as ApiStatusError.""" client = KubeClient() @@ -731,6 +1225,26 @@ async def test_list_objects_returns_generic_summaries() -> None: assert "/namespaces/default/deployments" in called_path +async def test_list_objects_emits_list_telemetry() -> None: + seen: list[ReadTelemetryEvent] = [] + client = KubeClient(read_telemetry=seen.append) + meta = _deploy_meta() + request_json_mock = AsyncMock(return_value={"items": [_generic("dep-a"), _generic("dep-b")]}) + + with ( + patch.object(client, "_api", MagicMock()), + patch.object(client, "_request_json", request_json_mock), + ): + summaries = await client.list_objects(meta, "default") + + assert [summary.name for summary in summaries] == ["dep-a", "dep-b"] + assert [event.operation for event in seen] == ["list"] + assert seen[0].path == "/apis/apps/v1/namespaces/default/deployments" + assert seen[0].object_count == 2 + assert seen[0].decoded_bytes > 0 + assert seen[0].status is None + + async def test_list_objects_all_namespaces_uses_cluster_path() -> None: """namespace=None produces a cluster-scoped path without /namespaces/.""" client = KubeClient() @@ -1177,10 +1691,14 @@ def test_requires_connect(self) -> None: async def test_opens_ws_with_exec_params(self, monkeypatch: pytest.MonkeyPatch) -> None: sentinel_ws = object() + sentinel_configuration = object() closed: list[bool] = [] captured: dict[str, object] = {} class FakeWsApi: + def __init__(self, configuration: object) -> None: + captured["configuration"] = configuration + async def close(self) -> None: closed.append(True) @@ -1206,12 +1724,15 @@ async def connect_get_namespaced_pod_exec( monkeypatch.setattr(client_mod, "WsApiClient", FakeWsApi) monkeypatch.setattr(k8s_client, "CoreV1Api", FakeCoreWs) kube = client_mod.KubeClient() + kube._api = MagicMock() + kube._api.configuration = sentinel_configuration kube._core_v1 = object() # type: ignore[assignment] # connected marker async with kube.open_pod_exec( "prod", "api-0", "app", ["tar", "cf", "-"], stdin=False ) as ws: assert ws is sentinel_ws + assert captured["configuration"] is sentinel_configuration assert captured["name"] == "api-0" assert captured["namespace"] == "prod" assert captured["command"] == ["tar", "cf", "-"] @@ -1227,6 +1748,9 @@ async def test_omits_container_when_none(self, monkeypatch: pytest.MonkeyPatch) captured: dict[str, object] = {} class FakeWsApi: + def __init__(self, configuration: object) -> None: + pass + async def close(self) -> None: return None @@ -1250,6 +1774,7 @@ async def connect_get_namespaced_pod_exec( monkeypatch.setattr(client_mod, "WsApiClient", FakeWsApi) monkeypatch.setattr(k8s_client, "CoreV1Api", FakeCoreWs) kube = client_mod.KubeClient() + kube._api = MagicMock() kube._core_v1 = object() # type: ignore[assignment] # connected marker async with kube.open_pod_exec("ns", "p", None, ["tar"], stdin=True): @@ -1261,6 +1786,9 @@ async def test_ws_api_closed_on_error(self, monkeypatch: pytest.MonkeyPatch) -> closed: list[bool] = [] class FakeWsApi: + def __init__(self, configuration: object) -> None: + pass + async def close(self) -> None: closed.append(True) @@ -1274,6 +1802,7 @@ async def connect_get_namespaced_pod_exec(self, *a: object, **k: object) -> obje monkeypatch.setattr(client_mod, "WsApiClient", FakeWsApi) monkeypatch.setattr(k8s_client, "CoreV1Api", FakeCoreWs) kube = client_mod.KubeClient() + kube._api = MagicMock() kube._core_v1 = object() # type: ignore[assignment] # connected marker with pytest.raises(OSError, match="boom"): @@ -1596,6 +2125,31 @@ async def fake_load(**kwargs: Any) -> None: class TestSwitchContext: + async def test_uses_refreshable_kube_config(self, monkeypatch: pytest.MonkeyPatch) -> None: + load_refreshable = AsyncMock() + api = MagicMock() + api_factory = MagicMock(return_value=api) + core_v1_factory = MagicMock() + set_default = MagicMock() + monkeypatch.setattr(client_mod, "load_refreshable_kube_config", load_refreshable) + monkeypatch.setattr(k8s_config, "load_kube_config", AsyncMock()) + monkeypatch.setattr(k8s_client, "ApiClient", api_factory) + monkeypatch.setattr(k8s_client, "CoreV1Api", core_v1_factory) + monkeypatch.setattr(k8s_client.Configuration, "set_default", set_default) + + kube = KubeClient() + await kube.switch_context("ctx-b") + + load_refreshable.assert_awaited_once() + call = load_refreshable.await_args + assert call is not None + assert call.kwargs["context"] == "ctx-b" + assert call.kwargs["persist_config"] is True + configuration = call.kwargs["client_configuration"] + set_default.assert_called_once_with(configuration) + api_factory.assert_called_once_with(configuration) + core_v1_factory.assert_called_once_with(api) + async def test_swaps_connection_and_closes_old(self, monkeypatch: pytest.MonkeyPatch) -> None: load_calls: list[dict[str, Any]] = [] diff --git a/tests/k8s/test_client_resize.py b/tests/k8s/test_client_resize.py index d27c2091..712218f9 100644 --- a/tests/k8s/test_client_resize.py +++ b/tests/k8s/test_client_resize.py @@ -5,6 +5,7 @@ from unittest.mock import AsyncMock, MagicMock, patch import pytest +from kubernetes_asyncio import client as k8s_client from korvid.k8s.client import KubeClient @@ -155,8 +156,12 @@ async def test_connect_resets_resize_discovery_cache( async def fake_load(*args: object, **kwargs: object) -> None: return None + def fake_api(configuration: k8s_client.Configuration) -> object: + assert isinstance(configuration, k8s_client.Configuration) + return object() + monkeypatch.setattr("korvid.k8s.client.k8s_config.load_kube_config", fake_load) - monkeypatch.setattr("korvid.k8s.client.k8s_client.ApiClient", lambda: object()) + monkeypatch.setattr("korvid.k8s.client.k8s_client.ApiClient", fake_api) monkeypatch.setattr("korvid.k8s.client.k8s_client.CoreV1Api", lambda api: object()) client = KubeClient() client._pod_resize_supported = True diff --git a/tests/performance/__init__.py b/tests/performance/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/performance/cli.py b/tests/performance/cli.py new file mode 100644 index 00000000..7ae6ec20 --- /dev/null +++ b/tests/performance/cli.py @@ -0,0 +1,548 @@ +"""Reproducible scale-benchmark CLI for large-cluster qualification (issue #186). + +Usage: + uv run python -m tests.performance.cli replay \\ + --profile PATH [--time-scale FLOAT] [--sample-interval FLOAT] \\ + [--json PATH] [--out PATH] [--cpu-profile PATH] [--allocation-snapshot PATH] + uv run python -m tests.performance.cli seed-manifests \\ + --run-id TEXT --namespace-count INT --pods-per-namespace INT \\ + --node-selector KEY=VALUE --output PATH + uv run python -m tests.performance.cli replay-live \\ + --profile tests/performance/profiles/aks-live-1k.json \\ + --context TEXT --expected-cluster-id TEXT --run-id TEXT \\ + [--duration INT] [--sample-interval FLOAT] \\ + [--json PATH] [--out PATH] [--cpu-profile PATH] [--allocation-snapshot PATH] + +`aks-live-1k` is the live qualification profile: it encodes the published live +plan (1,000 Pods across 20 namespaces, 30 minutes at 20 events/s with three +30-second bursts at 100 events/s), so the design doc's event-to-render, +backlog-drain, and RSS-slope budgets are measurable. `aks-1k` keeps the short +deterministic schedule used to compare a live run against the synthetic +1k/10k/50k baselines; use `--duration` to shorten a live smoke run (bursts and +failure points are re-validated against the shortened duration). +""" + +from __future__ import annotations + +import argparse +import asyncio +import cProfile +import dataclasses +import json +import sys +import tracemalloc +from pathlib import Path +from typing import Any + +import yaml + +from korvid.k8s.errors import ApiStatusError +from tests.performance.live import run_live_replay +from tests.performance.manifests import build_seed_manifests +from tests.performance.metrics import BenchmarkReport, render_markdown, report_payload +from tests.performance.profile import WorkloadProfile, load_profile, validate_profile +from tests.performance.replay import ReplayAborted, ReplayOptions, ReplayReport, run_replay +from tests.ui.waits import WaitTimeout + + +def _to_benchmark_report(replay: ReplayReport) -> BenchmarkReport: + return BenchmarkReport( + manifest=replay.manifest, + event_to_render=replay.event_to_render, + input_latency=replay.input_latency, + process=replay.process, + api=replay.api, + phases=replay.phases, + rendered_updates=replay.rendered_updates, + render_passes=replay.render_passes, + coalesced_updates=replay.coalesced_updates, + dropped_updates=replay.dropped_updates, + final_digest=replay.final_digest, + expected_digest=replay.expected_digest, + digest_match=replay.expected_digest == replay.final_digest, + failures_injected=replay.failures_injected, + ui_scenarios=replay.ui_scenarios, + churn=replay.churn, + ) + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="python -m tests.performance.cli", + description="korvid large-cluster benchmark CLI", + ) + subparsers = parser.add_subparsers(dest="command", required=True) + + rp = subparsers.add_parser("replay", help="Replay a workload profile and report metrics.") + rp.add_argument("--profile", required=True, metavar="PATH", help="Workload profile JSON.") + rp.add_argument( + "--time-scale", + type=float, + default=1.0, + metavar="FLOAT", + help="Sleep multiplier: 0 skips all sleeps, 1.0 replays at real time.", + ) + rp.add_argument( + "--sample-interval", + type=float, + default=1.0, + metavar="FLOAT", + help="Seconds between process-memory samples (positive).", + ) + rp.add_argument( + "--json", + dest="json_path", + default=None, + metavar="PATH", + help="Write machine-readable JSON report.", + ) + rp.add_argument( + "--out", + dest="out_path", + default=None, + metavar="PATH", + help="Write Markdown report to file (also printed to stdout).", + ) + rp.add_argument( + "--cpu-profile", + default=None, + metavar="PATH", + help="Write cProfile pstats file.", + ) + rp.add_argument( + "--allocation-snapshot", + default=None, + metavar="PATH", + help="Write top-100 tracemalloc source locations.", + ) + + sp = subparsers.add_parser( + "seed-manifests", + help="Render deterministic Namespace and Pod manifests for live AKS seeding.", + ) + sp.add_argument("--run-id", required=True, metavar="TEXT", help="Unique run identifier.") + sp.add_argument( + "--namespace-count", + required=True, + type=int, + metavar="INT", + help="Number of namespaces to create.", + ) + sp.add_argument( + "--pods-per-namespace", + required=True, + type=int, + metavar="INT", + help="Number of Pods to create in each namespace.", + ) + sp.add_argument( + "--node-selector", + required=True, + metavar="KEY=VALUE", + help="Exactly one nodeSelector key=value pair.", + ) + sp.add_argument( + "--output", + required=True, + metavar="PATH", + help="Destination path for the multi-document YAML output.", + ) + + lp = subparsers.add_parser( + "replay-live", + help="Replay churn against an already-seeded, owned real AKS cluster.", + ) + lp.add_argument( + "--profile", + required=True, + metavar="PATH", + help="Workload profile JSON. Use tests/performance/profiles/aks-live-1k.json " + "for a qualification run (the published 30-minute live plan); aks-1k is the " + "short deterministic comparison schedule.", + ) + lp.add_argument( + "--context", required=True, metavar="TEXT", help="Exact active kubeconfig context." + ) + lp.add_argument( + "--expected-cluster-id", + dest="expected_cluster_id", + required=True, + metavar="TEXT", + help="Exact AKS cluster ARM resource ID (must exactly match the returned `az aks show` id).", + ) + lp.add_argument( + "--run-id", dest="run_id", required=True, metavar="TEXT", help="Unique run identifier." + ) + lp.add_argument( + "--duration", + type=int, + default=None, + metavar="INT", + help="Override profile duration_seconds (positive); rate, bursts, " + "seed, topology, and failures are preserved.", + ) + lp.add_argument( + "--sample-interval", + type=float, + default=1.0, + metavar="FLOAT", + help="Seconds between process-memory samples (positive).", + ) + lp.add_argument( + "--json", + dest="json_path", + default=None, + metavar="PATH", + help="Required live artifact: machine-readable JSON report (filename must " + "include the run id).", + ) + lp.add_argument( + "--out", + dest="out_path", + default=None, + metavar="PATH", + help="Required live artifact: Markdown report (also printed to stdout; " + "filename must include the run id).", + ) + lp.add_argument( + "--cpu-profile", + default=None, + metavar="PATH", + help="Required live artifact: cProfile pstats file (filename must include the run id).", + ) + lp.add_argument( + "--allocation-snapshot", + default=None, + metavar="PATH", + help="Required live artifact: top-100 tracemalloc source locations " + "(filename must include the run id).", + ) + return parser + + +def _run_with_cpu_profile( + profile: WorkloadProfile, + options: ReplayOptions, + cpu_profile_path: str, +) -> ReplayReport: + """Run *run_replay* and dump a cProfile pstats file to *cpu_profile_path*.""" + pr = cProfile.Profile() + pr.enable() + try: + return asyncio.run(run_replay(profile, options)) + finally: + pr.disable() + pr.dump_stats(cpu_profile_path) + + +def _run_live_with_cpu_profile( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, + cpu_profile_path: str, +) -> ReplayReport: + """Run *run_live_replay* and dump a cProfile pstats file to *cpu_profile_path*.""" + pr = cProfile.Profile() + pr.enable() + try: + return asyncio.run( + run_live_replay( + profile, + options, + context=context, + expected_cluster_id=expected_cluster_id, + run_id=run_id, + ) + ) + finally: + pr.disable() + pr.dump_stats(cpu_profile_path) + + +def _flush_allocation_snapshot(path: str) -> int: + """Take a tracemalloc snapshot and write the top 100 lines to *path*. + + Runs from a `finally`, so a write failure must not replace whatever + exception is already propagating (or bury a long run's real failure under a + traceback). Tracing is always stopped. + + Returns: + 0 on success, 1 when the snapshot could not be written. + """ + if not tracemalloc.is_tracing(): + return 0 + try: + snapshot = tracemalloc.take_snapshot() + stats = snapshot.statistics("lineno")[:100] + Path(path).write_text("\n".join(str(stat) for stat in stats)) + except OSError as exc: + print(f"error writing allocation snapshot: {exc}", file=sys.stderr) + return 1 + finally: + tracemalloc.stop() + return 0 + + +def _write_outputs(args: argparse.Namespace, replay: ReplayReport) -> int: + """Print Markdown to stdout and write optional --out / --json outputs. + + Returns: + 0 on success, or 1 when a destination path cannot be written - a bad + `--out`/`--json` path is an operational error (as it already is for + `seed-manifests --output`), not a traceback after a long run. + """ + benchmark = _to_benchmark_report(replay) + markdown = render_markdown(benchmark) + sys.stdout.write(markdown) + try: + if args.out_path: + Path(args.out_path).write_text(markdown) + if args.json_path: + payload: dict[str, Any] = {"schema_version": 1, **report_payload(benchmark)} + Path(args.json_path).write_text(json.dumps(payload, indent=2, sort_keys=True)) + except OSError as exc: + print(f"error writing report: {exc}", file=sys.stderr) + return 1 + return 0 + + +def _cmd_seed_manifests(args: argparse.Namespace) -> int: + try: + manifests = build_seed_manifests( + run_id=args.run_id, + namespace_count=args.namespace_count, + pods_per_namespace=args.pods_per_namespace, + node_selector=args.node_selector, + ) + except ValueError as exc: + print(f"error building manifests: {exc}", file=sys.stderr) + return 1 + + try: + text = yaml.safe_dump_all(manifests, sort_keys=False, explicit_start=True) + Path(args.output).write_text(text, encoding="utf-8") + except OSError as exc: + print(f"error writing manifests: {exc}", file=sys.stderr) + return 1 + return 0 + + +def _cmd_replay(args: argparse.Namespace) -> int: + if args.time_scale < 0: + print("error: --time-scale must be non-negative", file=sys.stderr) + return 1 + if args.sample_interval <= 0: + print("error: --sample-interval must be positive", file=sys.stderr) + return 1 + + try: + profile = load_profile(Path(args.profile)) + except (OSError, UnicodeError, ValueError) as exc: + print(f"error loading profile: {exc}", file=sys.stderr) + return 1 + + options = ReplayOptions(time_scale=args.time_scale, sample_interval=args.sample_interval) + + snapshot_failed = 0 + if args.allocation_snapshot: + tracemalloc.start() + + try: + if args.cpu_profile: + replay = _run_with_cpu_profile(profile, options, args.cpu_profile) + else: + replay = asyncio.run(run_replay(profile, options)) + except (ReplayAborted, ApiStatusError, WaitTimeout, OSError) as exc: + print(f"error during replay: {exc}", file=sys.stderr) + return 1 + finally: + if args.allocation_snapshot: + snapshot_failed = _flush_allocation_snapshot(args.allocation_snapshot) + + return _replay_exit_status(args, replay, snapshot_failed=snapshot_failed) + + +def _replay_exit_status( + args: argparse.Namespace, replay: ReplayReport, *, snapshot_failed: int +) -> int: + """Exit status shared by both replay commands: artifacts first, then the + correctness criteria (no dropped updates, digest parity).""" + if snapshot_failed: + return 1 + if _write_outputs(args, replay): + return 1 + if replay.dropped_updates > 0 or replay.expected_digest != replay.final_digest: + return 1 + return 0 + + +def _load_live_profile(args: argparse.Namespace) -> WorkloadProfile | None: + """Load `--profile` and apply `--duration`, or report why it cannot apply. + + `dataclasses.replace` bypasses `load_profile`, so every duration-dependent + invariant (burst containment/overlap, failure-injection bounds) is + re-checked here - before any cluster identity, ownership, or mutation work + is attempted. + + Returns: + The profile to replay, or `None` when it cannot be used (the reason is + already printed to stderr). + """ + try: + profile = load_profile(Path(args.profile)) + except (OSError, UnicodeError, ValueError) as exc: + print(f"error loading profile: {exc}", file=sys.stderr) + return None + if args.duration is None: + return profile + overridden = dataclasses.replace(profile, duration_seconds=args.duration) + try: + validate_profile(overridden) + except ValueError as exc: + print(f"error: --duration {args.duration} invalidates the profile: {exc}", file=sys.stderr) + return None + return overridden + + +#: The four externally-retained artifacts a *successful* live qualification +#: must produce (design "Results and documentation": raw JSON samples, process +#: traces, allocation snapshots, plus the human-readable summary). Unlike an +#: ordinary offline `replay`, every live run must write all four to a +#: run-labelled destination so the evidence is retained and traceable. +_LIVE_ARTIFACT_FLAGS: dict[str, str] = { + "json_path": "--json", + "out_path": "--out", + "cpu_profile": "--cpu-profile", + "allocation_snapshot": "--allocation-snapshot", +} + + +def _validate_live_artifacts(args: argparse.Namespace, *, run_id: str) -> str | None: + """Require exactly the four run-labelled live artifact destinations. + + Returns an error message when any of the four is missing, when two point at + the same destination, or when a destination's filename does not carry the + run id (so a retained artifact can always be traced back to its run). + Returns `None` when all four are present, distinct, and run-labelled. + """ + missing = [flag for attr, flag in _LIVE_ARTIFACT_FLAGS.items() if not getattr(args, attr)] + if missing: + return ( + "a successful live qualification must write all four artifacts " + f"({', '.join(_LIVE_ARTIFACT_FLAGS.values())}); missing: {', '.join(missing)}" + ) + paths = {attr: Path(getattr(args, attr)) for attr in _LIVE_ARTIFACT_FLAGS} + # Distinctness is decided on resolved paths: `sub/../run.json` and + # `run.json` are different strings but the same file, and the second write + # would silently destroy the first artifact. + if len({path.resolve() for path in paths.values()}) != len(paths): + return "the four live artifacts must be four distinct destinations" + for attr, path in paths.items(): + if run_id not in path.name: + return ( + f"live artifact {_LIVE_ARTIFACT_FLAGS[attr]} filename {path.name!r} " + f"must include the run id {run_id!r}" + ) + return None + + +def _cmd_replay_live(args: argparse.Namespace) -> int: + if args.duration is not None and args.duration <= 0: + print("error: --duration must be positive", file=sys.stderr) + return 1 + if args.sample_interval <= 0: + print("error: --sample-interval must be positive", file=sys.stderr) + return 1 + + profile = _load_live_profile(args) + if profile is None: + return 1 + + artifact_error = _validate_live_artifacts(args, run_id=args.run_id) + if artifact_error: + print(f"error: {artifact_error}", file=sys.stderr) + return 1 + + # No --time-scale option: live churn always replays at real wall-clock + # time (ReplayOptions.time_scale defaults to 1.0). + options = ReplayOptions(sample_interval=args.sample_interval) + return _execute_live_replay(args, profile, options) + + +def _execute_live_replay( + args: argparse.Namespace, profile: WorkloadProfile, options: ReplayOptions +) -> int: + snapshot_failed = 0 + if args.allocation_snapshot: + tracemalloc.start() + + try: + if args.cpu_profile: + replay = _run_live_with_cpu_profile( + profile, + options, + context=args.context, + expected_cluster_id=args.expected_cluster_id, + run_id=args.run_id, + cpu_profile_path=args.cpu_profile, + ) + else: + replay = asyncio.run( + run_live_replay( + profile, + options, + context=args.context, + expected_cluster_id=args.expected_cluster_id, + run_id=args.run_id, + ) + ) + except (ValueError, ApiStatusError, WaitTimeout, OSError) as exc: + print(f"error during replay: {exc}", file=sys.stderr) + return 1 + finally: + if args.allocation_snapshot: + snapshot_failed = _flush_allocation_snapshot(args.allocation_snapshot) + + if snapshot_failed: + return 1 + failed_scenarios = [scenario.name for scenario in replay.ui_scenarios if not scenario.ok] + if failed_scenarios: + # `drive_ui_scenarios` records a key sequence that never reached its + # target state as `ok=False` instead of raising, so without this the + # run would "pass" with no UI-at-scale evidence behind it. Reported + # before the artifacts are written so the reason is not buried under + # the Markdown report. + print( + "error: UI-at-scale scenarios did not pass: " + ", ".join(failed_scenarios), + file=sys.stderr, + ) + _write_outputs(args, replay) + return 1 + return _replay_exit_status(args, replay, snapshot_failed=snapshot_failed) + + +def main(argv: list[str] | None = None) -> int: + """CLI entry point for the large-cluster benchmark. + + Args: + argv: Argument list; defaults to `sys.argv[1:]` when `None`. + + Returns: + 0 on success; 1 for runtime error, dropped updates, or digest mismatch. + """ + parser = _build_parser() + args = parser.parse_args(argv) + if args.command == "replay": + return _cmd_replay(args) + if args.command == "seed-manifests": + return _cmd_seed_manifests(args) + if args.command == "replay-live": + return _cmd_replay_live(args) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/performance/live.py b/tests/performance/live.py new file mode 100644 index 00000000..59a793fa --- /dev/null +++ b/tests/performance/live.py @@ -0,0 +1,1612 @@ +"""Guarded real-AKS application-path replay prerequisite (issue #186 task 8.2). + +`run_live_replay` drives the *real* production stack (`KubeClient` -> +`WatchManager` -> `ResourceStore` -> `MeasuredKorvidApp`) against an +already-seeded AKS cluster (20 namespaces / 1,000 Pods, created by +`tests.performance.manifests.build_seed_manifests` and the `seed-manifests` +CLI subcommand). This module never provisions, tears down, or discovers a +cluster on its own - it only replays churn against a topology a human has +already created and identified explicitly via `--context`, +`--expected-cluster-id`, and `--run-id`. + +Every mutation is fail-closed: + +1. **Cluster identity gate** (`_verify_cluster_identity`): the active + kubeconfig context, its resolved API server hostname, and an independent + `az aks show` lookup for the fixed dedicated-test resource group/name must + all agree before any client connects. +2. **Ownership gate** (`_verify_ownership`): every expected namespace and + every expected Pod must already carry both ownership labels + (`manifests.MANAGED_BY_LABEL`/`manifests.RUN_LABEL`), and no *unexpected* + Pod may exist in an owned namespace, before any churn is attempted. +3. **Guarded churn** (`drive_live_churn`): each mutation is a JSON-Patch that + `test`s the target Pod's UID and both ownership labels before writing a + dedicated, non-ownership `manifests.TICK_LABEL` on the Pod's *own* + metadata. A failed `test` op aborts the *entire* run - there is no + unguarded fallback, and no ownership label, `status`, or spec field is ever + written. +4. **Post-churn revalidation** (`read_and_validate_owned_pods`): the + ground-truth cluster read re-checks the exact identity set, every UID, and + both ownership labels before its digest is trusted. + +Churn is metadata-only by design. The seeded Pods are real +`registry.k8s.io/pause:3.10` Pods whose `status` subresource is owned by the +kubelet: patching `status.phase` would be reverted on the next node sync and +would contradict the design doc's "metadata-only updates create real watch +traffic without restarting containers or changing the workload's resource +demand". `TICK_LABEL` is user-owned, is part of `PodSummary.labels` (so it +really does change the store digest and produce a watch event), and no +controller reconciles it away. + +Event-to-render latency is recorded where the *application* first sees an +event - `make_live_watch_source` records at watch receipt, exactly like the +deterministic `_ReplaySource` - never at patch acknowledgement. The watch +event and the patch response race over independent connections, so recording +at ack both misreports the interval (it would include the write round-trip) +and can append an event *after* its own render. + +Two separate `KubeReadClient` connections are used deliberately: a +non-instrumented *harness* client (`LiveDependencies.harness_kube_client_factory`) +performs the ownership gate, the ground-truth re-read, and nothing else, while +a single telemetry-wired *application-path* client +(`LiveDependencies.kube_client_factory`) is only ever handed to +`make_live_watch_source`. This keeps `ReplayReport.api` reporting exactly the +production application read path (the real `KubeClient.watch_pods` LIST+WATCH +telemetry) instead of being diluted by the harness's own bookkeeping reads. +Mutation traffic is likewise never reported as application read telemetry: its +throttles are counted separately in `ChurnSummary.mutation_throttles`. + +Unit tests substitute every external boundary via `LiveDependencies`; none of +them may contact Azure, a real kubeconfig, or a real cluster. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import math +from collections.abc import AsyncIterator, Awaitable, Callable, Iterable, Mapping +from dataclasses import dataclass, field +from time import monotonic +from types import MappingProxyType +from typing import Any, Protocol, cast +from urllib.parse import urlparse + +from kubernetes_asyncio import client as k8s_client +from kubernetes_asyncio import config as k8s_config + +from korvid.core.config import KorvidConfig +from korvid.core.store import ALL_NAMESPACES, ResourceStore +from korvid.core.watch import WatchManager, WatchSource +from korvid.k8s.client import KubeClient, load_refreshable_kube_config, resolve_context_name +from korvid.k8s.discovery import ResourceMeta +from korvid.k8s.errors import ApiStatusError +from korvid.k8s.models import GenericSummary, PodSummary +from korvid.k8s.telemetry import ReadTelemetry +from korvid.ui.widgets.describe_screen import DescribeScreen +from korvid.ui.widgets.resource_table import ResourceTable +from tests.performance import manifests +from tests.performance.metrics import ( + BenchmarkRecorder, + ChurnSummary, + NodePoolInfo, + ProcessSampler, +) +from tests.performance.profile import WorkloadProfile, burst_end_offsets, validate_profile +from tests.performance.replay import ( + MeasuredKorvidApp, + ReplayOptions, + ReplayReport, + build_manifest, + check_rendered_rows, + resolve_korvid_sha, + wait_for, +) +from tests.performance.workload import ScheduledEvent, scheduled_events, summary_digest + +#: Namespace-scoped read for the ownership gate; not exposed via `PODS_META` +#: because Namespaces are cluster-scoped (`namespaced=False`). +_NAMESPACES_META = ResourceMeta("Namespace", "namespaces", "", "v1", False) + +#: Pods, for the read-only `describe` provider the UI-at-scale scenarios need. +_PODS_META = ResourceMeta("Pod", "pods", "", "v1", True) + +#: Live topology is pinned to the aks-1k profile shape (issue #186 Task 8): +#: exactly 20 namespaces of 50 Pods each, matching `seed-manifests`'s output. +_REQUIRED_OBJECT_COUNT = 1000 +_REQUIRED_NAMESPACE_COUNT = 20 + +#: The immutable dedicated-test target contract (design doc "Fixed target and +#: capacity"). The identity gate fails closed unless `az aks show` reports +#: exactly this resource group, cluster name, and both required tags, so a +#: production cluster (or its ARM id) can never satisfy the gate even if the +#: operator supplies a matching, valid ARM id and kubeconfig context. +_REQUIRED_RESOURCE_GROUP = "rg-korvid-contract-test" +_REQUIRED_CLUSTER_NAME = "aks-korvid-contract-test" +_REQUIRED_TAGS: Mapping[str, str] = MappingProxyType( + {"purpose": "korvid-contract-testing", "production-use": "prohibited"} +) + +#: HTTP status the mutation path may retry. 429 (API Priority and Fairness) +#: is the only one: it is a "come back later" answer to a well-formed, +#: fully guarded request, and the retry re-issues the *identical* guarded +#: patch. Every other status - including a failed `test` op - aborts the run. +_RETRYABLE_MUTATION_STATUS = 429 + +#: How many times teardown re-attempts draining the churn task when its own +#: await is interrupted by a cancellation aimed at the caller. +_CANCEL_DRAIN_ATTEMPTS = 3 + + +async def _sleep_default(delay: float) -> None: + await asyncio.sleep(delay) + + +@dataclass(frozen=True) +class LiveLimits: + """Every explicit bound the live run enforces. + + Nothing in a live run may be unbounded: an in-flight patch, the number of + concurrent patches, the wait for churn to finish, the wait for the store to + converge, and even the initial kubeconfig/credential-plugin connection all + have a stated ceiling. Tests inject small values; production uses defaults + sized for the published 30-minute live profile. + + Args: + churn_concurrency: Maximum simultaneously in-flight guarded patches. + Serial patching cannot approach the scheduled rate (one round trip + per event), so concurrency is explicit rather than implicit. + mutation_timeout_seconds: Ceiling for one guarded patch attempt. + mutation_throttle_retries: Bounded retries of the *identical* guarded + patch after HTTP 429, and only after 429. + mutation_retry_base_delay_seconds: First backoff delay; doubles per + retry. + mutation_retry_max_delay_seconds: Hard ceiling for exponential backoff, + deterministic jitter, and a server-provided `Retry-After` hint. + mutation_connect_timeout_seconds: Ceiling for connecting the mutation + client (a kubeconfig exec credential plugin can block). + read_connect_timeout_seconds: Ceiling for external read-path setup that + a kubeconfig exec credential plugin can block indefinitely - the + identity/context-host lookup, the harness read client connect, and + the application read client connect. + initial_render_timeout_seconds: Ceiling for the initial 1,000-row + render. + churn_grace_seconds: Allowance added to the profile's own scheduled + duration when bounding the churn wait. + convergence_timeout_seconds: Ceiling for the store digest to converge + with the independently read cluster digest. + """ + + churn_concurrency: int = 32 + mutation_timeout_seconds: float = 30.0 + mutation_throttle_retries: int = 5 + mutation_retry_base_delay_seconds: float = 0.5 + mutation_retry_max_delay_seconds: float = 30.0 + mutation_connect_timeout_seconds: float = 60.0 + read_connect_timeout_seconds: float = 60.0 + initial_render_timeout_seconds: float = 60.0 + churn_grace_seconds: float = 300.0 + convergence_timeout_seconds: float = 120.0 + + +@dataclass +class ChurnProgress: + """Live counters for the churn phase, observable while it runs. + + Kept separate from `ChurnSummary` (the frozen report value) because + `run_live_replay` reads `started` *during* the run to decide whether churn + was really under way when input latency was measured. + """ + + requested_events: int = 0 + started: int = 0 + completed: int = 0 + mutation_throttles: int = 0 + first_started_at: float | None = None + last_completed_at: float | None = None + + def record_started(self, at: float) -> None: + if self.first_started_at is None: + self.first_started_at = at + self.started += 1 + + def record_completed(self, at: float) -> None: + self.completed += 1 + self.last_completed_at = at + + def wall_seconds(self) -> float | None: + if self.first_started_at is None or self.last_completed_at is None: + return None + return self.last_completed_at - self.first_started_at + + def summary(self, *, requested_duration_seconds: int) -> ChurnSummary: + return ChurnSummary.from_observations( + requested_events=self.requested_events, + requested_duration_seconds=requested_duration_seconds, + observed_events=self.completed, + wall_seconds=self.wall_seconds(), + mutation_throttles=self.mutation_throttles, + ) + + +@dataclass(frozen=True) +class CommandResult: + """Captured outcome of a subprocess invocation (e.g. `az aks show`).""" + + exit_code: int + stdout: str + stderr: str + + +#: Runs an argv list and captures its result; the identity gate's only +#: subprocess seam. Production uses `_default_command_runner`; tests inject +#: a fake that never touches a real `az` binary. +CommandRunner = Callable[[list[str]], Awaitable[CommandResult]] + + +class KubeReadClient(Protocol): + """Structural subset of `KubeClient` the live read path needs. + + Deliberately narrow (vs. requiring the full `KubeClient`) so unit tests + can implement it with a plain in-memory fake instead of subclassing the + real, network-talking client. + """ + + async def connect(self, context: str | None = None) -> None: ... + + async def close(self) -> None: ... + + async def list_objects( + self, meta: ResourceMeta, namespace: str | None + ) -> list[GenericSummary]: ... + + async def list_pods(self, namespace: str) -> list[PodSummary]: ... + + def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSummary]]: ... + + async def get_object( + self, meta: ResourceMeta, namespace: str | None, name: str + ) -> dict[str, Any]: ... + + def stream_logs(self, *args: Any, **kwargs: Any) -> AsyncIterator[Any]: ... + + +class MutationClient(Protocol): + """Issues one guarded, metadata-only mutation; production talks JSON-Patch + to a real API server, tests mutate an in-memory fake cluster the same way. + + There is deliberately no method that can write `status`, `spec`, an + ownership label, or delete anything. + """ + + async def connect(self) -> None: ... + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: ... + + async def close(self) -> None: ... + + +@dataclass(frozen=True) +class LiveDependencies: + """Every external boundary `run_live_replay` crosses, as an injectable + seam. Production callers get real wiring from `_default_dependencies`; + tests always construct this explicitly with fakes, so no unit test can + reach Azure, a real kubeconfig, or a real cluster. + + `kube_client_factory` and `harness_kube_client_factory` are deliberately + two separate seams (constructing two separate `KubeReadClient` + connections at runtime): the former is wired with `recorder.record_api` + and used *only* for the real application read path + (`make_live_watch_source`'s `watch_pods`), so `ReplayReport.api` reports + exactly the production LIST+WATCH telemetry an operator would see. The + latter is never wired to telemetry and is used *only* for the harness's + own bookkeeping reads (the ownership gate and the ground-truth re-read) - + those reads must never dilute the application-path signal. + """ + + command_runner: CommandRunner + active_context: Callable[[], str | None] + context_host: Callable[[str], Awaitable[str]] + kube_client_factory: Callable[[ReadTelemetry], KubeReadClient] + harness_kube_client_factory: Callable[[], KubeReadClient] + mutation_client_factory: Callable[[str], MutationClient] + #: Resolves the immutable korvid commit for the run manifest. A live + #: evidence run fails closed when this returns `None` rather than publish + #: an untraceable artifact. Injected in tests for determinism. + resolve_sha: Callable[[], str | None] = resolve_korvid_sha + + +@dataclass(frozen=True) +class LiveClusterFacts: + """Verified, immutable facts about the live target, extracted from the + identity gate's `az aks show` payload and recorded in the live manifest so + retained evidence establishes exactly which cluster matrix was qualified. + """ + + context: str + cluster_id: str + kubernetes_version: str | None + node_pools: tuple[NodePoolInfo, ...] + + +async def _default_command_runner(args: list[str]) -> CommandResult: + """Run *args* as a subprocess and capture stdout/stderr/exit code. + + A missing executable (e.g. `az` not on PATH) surfaces as a `ValueError` + so it fails the identity gate cleanly instead of raising an uncaught + `FileNotFoundError` deep inside `run_live_replay`. + """ + try: + proc = await asyncio.create_subprocess_exec( + *args, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + except FileNotFoundError as exc: + raise ValueError(f"executable not found: {args[0]!r} ({exc})") from exc + stdout_bytes, stderr_bytes = await proc.communicate() + return CommandResult( + proc.returncode if proc.returncode is not None else 1, + stdout_bytes.decode("utf-8", errors="replace"), + stderr_bytes.decode("utf-8", errors="replace"), + ) + + +def _default_active_context() -> str | None: + return resolve_context_name() + + +async def _default_context_host(context: str) -> str: + """Resolve the API server hostname *context* would dial. + + Loads the kubeconfig into a private `Configuration` (never persisted, + never the global default) - mirroring `KubeClient.probe_context`'s + isolation pattern - so resolving identity never disturbs any live + connection. + """ + configuration = k8s_client.Configuration() + await k8s_config.load_kube_config( + context=context, client_configuration=configuration, persist_config=False + ) + hostname = urlparse(configuration.host or "").hostname + if not hostname: + raise ValueError(f"could not resolve API server hostname for context {context!r}") + return hostname + + +def _json_pointer_escape(segment: str) -> str: + """Escape one JSON-Pointer (RFC 6901) reference token.""" + return segment.replace("~", "~0").replace("/", "~1") + + +def build_guarded_label_patch(*, uid: str, run_id: str, tick: str) -> list[dict[str, Any]]: + """The exact JSON-Patch op list a guarded churn mutation issues. + + `test`s the target Pod's UID and *both* ownership labels, then `add`s the + dedicated non-ownership `manifests.TICK_LABEL`, so a stale, foreign, or + replaced Pod aborts the whole patch server-side - there is no unguarded + fallback. `add` (not `replace`) is used for the tick because RFC 6902 + `replace` requires the member to exist already, while `add` on an object + member creates or overwrites it; the seeded Pods start without it. + + Nothing outside `metadata.labels` is written: no `status`, no `spec`, and + neither ownership label is ever a target of a write op. + """ + managed_by_path = f"/metadata/labels/{_json_pointer_escape(manifests.MANAGED_BY_LABEL)}" + run_path = f"/metadata/labels/{_json_pointer_escape(manifests.RUN_LABEL)}" + tick_path = f"/metadata/labels/{_json_pointer_escape(manifests.TICK_LABEL)}" + return [ + {"op": "test", "path": "/metadata/uid", "value": uid}, + {"op": "test", "path": managed_by_path, "value": manifests.MANAGED_BY_VALUE}, + {"op": "test", "path": run_path, "value": run_id}, + {"op": "add", "path": tick_path, "value": tick}, + ] + + +class _KubeMutationClient: + """Production `MutationClient`: issues a guarded, metadata-only JSON-Patch + against the Pod resource itself (never the `pods/status` subresource) of + *context*.""" + + def __init__(self, context: str, run_id: str) -> None: + self._context = context + self._run_id = run_id + self._api: k8s_client.ApiClient | None = None + self._core_v1: k8s_client.CoreV1Api | None = None + + async def connect(self) -> None: + """Load the kubeconfig and build the API client. + + Called eagerly (under `LiveLimits.mutation_connect_timeout_seconds`) + before the measured window opens: `load_kube_config` can invoke an + exec credential plugin, and that latency must not be charged to the + first churn mutation - `KubeClient.probe_context` bounds exactly the + same call for exactly the same reason. + """ + if self._core_v1 is not None: + return + configuration = k8s_client.Configuration() + await load_refreshable_kube_config( + context=self._context, + client_configuration=configuration, + persist_config=False, + ) + self._api = k8s_client.ApiClient(configuration) + self._core_v1 = k8s_client.CoreV1Api(self._api) + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + await self.connect() + core_v1 = self._core_v1 + if core_v1 is None: # pragma: no cover - connect() always sets it + raise RuntimeError("mutation client is not connected") + ops = build_guarded_label_patch(uid=uid, run_id=self._run_id, tick=tick) + try: + await core_v1.patch_namespaced_pod( + name, + namespace, + ops, + _content_type="application/json-patch+json", # type: ignore[call-arg] # kubernetes_asyncio's .pyi stub omits _content_type; accepted via **kwargs at runtime + ) + except k8s_client.exceptions.ApiException as exc: + raw_body = getattr(exc, "body", "") or "" + body = ( + raw_body.decode("utf-8", errors="replace") + if isinstance(raw_body, bytes) + else str(raw_body) + ) + raise ApiStatusError( + int(exc.status or 0), + str(exc.reason or ""), + body=body, + retry_after_seconds=_api_retry_after_seconds(exc, body), + ) from exc + + async def close(self) -> None: + if self._api is not None: + await self._api.close() + + +def _default_dependencies(context: str) -> LiveDependencies: + """Real production wiring: subprocess `az`, real kubeconfig, real `KubeClient`. + + Two independent `KubeClient` connections are constructed: the + telemetry-wired one `run_live_replay` hands to `make_live_watch_source`, + and a plain, non-instrumented one (`read_telemetry=None`, so + `KubeClient._observe_read` is a no-op) for the harness's own ownership + and ground-truth reads. + """ + return LiveDependencies( + command_runner=_default_command_runner, + active_context=_default_active_context, + context_host=_default_context_host, + kube_client_factory=lambda read_telemetry: KubeClient(read_telemetry=read_telemetry), + harness_kube_client_factory=lambda: KubeClient(), + mutation_client_factory=lambda run_id: _KubeMutationClient(context, run_id), + ) + + +def live_object_identity(run_id: str, namespace_count: int, index: int) -> tuple[str, str]: + """Map a synthetic churn-schedule Pod index onto the exact seeded + `(namespace, name)` `manifests.build_seed_manifests` created for it - + guaranteeing seeding and live churn always agree on identity.""" + namespace = manifests.namespace_name(run_id, index % namespace_count) + name = manifests.pod_name(namespace_count, index) + return namespace, name + + +def _owns(labels: Iterable[tuple[str, str]], run_id: str) -> bool: + mapping = dict(labels) + return ( + mapping.get(manifests.MANAGED_BY_LABEL) == manifests.MANAGED_BY_VALUE + and mapping.get(manifests.RUN_LABEL) == run_id + ) + + +def _is_running_ready(pod: PodSummary) -> bool: + """Whether *pod* is phase `Running` with every container ready. + + The published live protocol requires exactly 1,000 Running, Ready Pods + before measuring. `ready` is the kubectl-style `/` string, so + a Pod is ready only when both counts are equal and non-zero (e.g. `1/1`). + """ + if pod.phase != "Running": + return False + parts = pod.ready.split("/") + if len(parts) != 2: + return False + try: + ready, total = int(parts[0]), int(parts[1]) + except ValueError: + return False + return total > 0 and ready == total + + +def _validate_time_scale(options: ReplayOptions) -> None: + if options.time_scale != 1.0: + raise ValueError( + f"run_live_replay requires options.time_scale == 1.0 (real wall time), " + f"got {options.time_scale!r}" + ) + + +def _validate_topology(profile: WorkloadProfile) -> None: + if profile.object_count != _REQUIRED_OBJECT_COUNT: + raise ValueError( + f"live replay requires object_count == {_REQUIRED_OBJECT_COUNT}, " + f"got {profile.object_count}" + ) + if profile.namespace_count != _REQUIRED_NAMESPACE_COUNT: + raise ValueError( + f"live replay requires namespace_count == {_REQUIRED_NAMESPACE_COUNT}, " + f"got {profile.namespace_count}" + ) + + +async def _verify_cluster_identity( + *, context: str, expected_cluster_id: str, deps: LiveDependencies, limits: LiveLimits +) -> LiveClusterFacts: + """Fail-closed 5-step cluster identity gate; every failure raises + `ValueError` before any client is constructed or any mutation attempted. + + Returns the verified, immutable cluster facts (context, ARM id, Kubernetes + version, node-pool topology) so the caller can record them in the live + manifest without a second, unbounded metadata lookup. + """ + active = deps.active_context() + if active != context: + raise ValueError( + f"active kubeconfig context {active!r} does not match required context {context!r}" + ) + + async with asyncio.timeout(limits.read_connect_timeout_seconds): + hostname = await deps.context_host(context) + + async with asyncio.timeout(limits.read_connect_timeout_seconds): + result = await deps.command_runner( + [ + "az", + "aks", + "show", + "--resource-group", + _REQUIRED_RESOURCE_GROUP, + "--name", + _REQUIRED_CLUSTER_NAME, + "-o", + "json", + ] + ) + if result.exit_code != 0: + raise ValueError(f"az aks show failed (exit {result.exit_code}): {result.stderr.strip()}") + try: + payload = json.loads(result.stdout) + except json.JSONDecodeError as exc: + raise ValueError(f"az aks show returned malformed JSON: {exc}") from exc + if not isinstance(payload, dict): + raise ValueError("az aks show returned malformed JSON: expected a JSON object") + + resource_id = payload.get("id") + if resource_id != expected_cluster_id: + raise ValueError( + f"az aks show returned id {resource_id!r}, expected {expected_cluster_id!r}" + ) + + _verify_immutable_target(payload) + + fqdn = payload.get("fqdn") or "" + private_fqdn = payload.get("privateFqdn") or "" + expected_hostname = fqdn or private_fqdn + if not expected_hostname: + raise ValueError("az aks show returned neither fqdn nor privateFqdn") + if hostname != expected_hostname: + raise ValueError( + f"context {context!r} API hostname {hostname!r} does not match " + f"cluster hostname {expected_hostname!r}" + ) + + return LiveClusterFacts( + context=context, + cluster_id=expected_cluster_id, + kubernetes_version=_kubernetes_version(payload), + node_pools=_node_pools(payload), + ) + + +def _kubernetes_version(payload: dict[str, Any]) -> str | None: + """The control-plane Kubernetes version AKS reports for the cluster.""" + for key in ("currentKubernetesVersion", "kubernetesVersion"): + value = payload.get(key) + if isinstance(value, str) and value: + return value + return None + + +def _node_pools(payload: dict[str, Any]) -> tuple[NodePoolInfo, ...]: + """Node-pool name/version/count topology from the `az aks show` payload.""" + profiles = payload.get("agentPoolProfiles") + if not isinstance(profiles, list): + return () + pools: list[NodePoolInfo] = [] + for profile in profiles: + if not isinstance(profile, dict): + continue + name = profile.get("name") + if not isinstance(name, str): + continue + version = ( + profile.get("currentOrchestratorVersion") or profile.get("orchestratorVersion") or "" + ) + count = profile.get("count") + pools.append( + NodePoolInfo( + name=name, + kubernetes_version=str(version), + node_count=int(count) if isinstance(count, int) else 0, + ) + ) + return tuple(pools) + + +def _verify_immutable_target(payload: dict[str, Any]) -> None: + """Fail closed unless *payload* describes the fixed dedicated-test target. + + Validating only that `az` and the kubeconfig agree on the operator-supplied + ARM id is not enough: a production cluster plus its own matching production + id would pass. The dedicated-test contract is immutable - a fixed resource + group, cluster name, and both required test-only tags - so those attributes + are checked directly from the `az aks show` payload before any client is + constructed or any mutation is attempted. + """ + resource_group = payload.get("resourceGroup") + if resource_group != _REQUIRED_RESOURCE_GROUP: + raise ValueError( + f"az aks show resource group {resource_group!r} is not the required " + f"dedicated-test resource group {_REQUIRED_RESOURCE_GROUP!r}" + ) + name = payload.get("name") + if name != _REQUIRED_CLUSTER_NAME: + raise ValueError( + f"az aks show cluster name {name!r} is not the required dedicated-test " + f"cluster name {_REQUIRED_CLUSTER_NAME!r}" + ) + tags = payload.get("tags") + if not isinstance(tags, dict): + raise ValueError("az aks show returned no tags; required test-only tags are missing") + for key, expected_value in _REQUIRED_TAGS.items(): + if tags.get(key) != expected_value: + raise ValueError( + f"az aks show is missing required tag {key}={expected_value!r} " + f"(got {tags.get(key)!r}); refusing to target a cluster not marked " + f"as korvid contract-testing" + ) + + +def _expected_pod_names(namespace_count: int, object_count: int) -> tuple[str, ...]: + """The exact Pod names `seed-manifests` creates in *every* namespace.""" + pods_per_namespace = object_count // namespace_count + return tuple( + manifests.pod_name(namespace_count, local_index * namespace_count) + for local_index in range(pods_per_namespace) + ) + + +@dataclass +class _PodOwnershipProblems: + """Accumulated ownership-gate failures across all namespaces/Pods.""" + + missing: list[str] = field(default_factory=list) + mismatched: list[str] = field(default_factory=list) + unexpected: list[str] = field(default_factory=list) + not_ready: list[str] = field(default_factory=list) + + +async def _collect_owned_pods( + kube: KubeReadClient, + *, + run_id: str, + expected_namespaces: list[str], + wanted: tuple[str, ...], +) -> tuple[dict[tuple[str, str], PodSummary], _PodOwnershipProblems]: + """List every expected namespace and classify its Pods against the contract.""" + problems = _PodOwnershipProblems() + validated: dict[tuple[str, str], PodSummary] = {} + for namespace in expected_namespaces: + pods_by_name = {pod.name: pod for pod in await kube.list_pods(namespace)} + problems.unexpected.extend( + f"{namespace}/{name}" for name in sorted(set(pods_by_name) - set(wanted)) + ) + for name in wanted: + pod = pods_by_name.get(name) + if pod is None: + problems.missing.append(f"{namespace}/{name}") + elif not _owns(pod.labels, run_id): + problems.mismatched.append(f"{namespace}/{name}") + elif not _is_running_ready(pod): + problems.not_ready.append( + f"{namespace}/{name} (phase={pod.phase}, ready={pod.ready})" + ) + else: + validated[(namespace, name)] = pod + return validated, problems + + +async def _verify_ownership( + kube: KubeReadClient, *, run_id: str, namespace_count: int, object_count: int +) -> dict[tuple[str, str], PodSummary]: + """Ownership gate: *exactly* the expected namespaces and Pods must exist, + each with both ownership labels *and* phase Running/Ready, before any churn. + + An unexpected Pod inside an owned namespace is rejected too: the + application watch filters by namespace, so a foreign Pod would enter the + benchmark store and turn a precise ownership violation into a generic + "1,000 rows never rendered" timeout minutes later. A labelled but + non-Running/non-Ready Pod is rejected as well, since the published protocol + requires exactly `object_count` Running, Ready owned Pods before measuring + and the later table-row check cannot distinguish a Pending Pod's row. + + Collects every mismatch across all namespaces/Pods before raising, so a + single failed run surfaces the full blast radius at once instead of + stopping at the first namespace. + + Returns the validated `(namespace, name) -> PodSummary` snapshot so + callers can reuse it (e.g. as the pre-churn uid snapshot) instead of + immediately re-listing the same Pods a second time. + """ + expected_namespaces = [manifests.namespace_name(run_id, i) for i in range(namespace_count)] + actual_namespaces = {obj.name: obj for obj in await kube.list_objects(_NAMESPACES_META, None)} + + missing_namespaces = [name for name in expected_namespaces if name not in actual_namespaces] + if missing_namespaces: + raise ValueError(f"missing expected namespaces: {', '.join(missing_namespaces)}") + + mismatched_namespaces = [ + name for name in expected_namespaces if not _owns(actual_namespaces[name].labels, run_id) + ] + if mismatched_namespaces: + raise ValueError( + f"namespaces missing/mismatched ownership labels: {', '.join(mismatched_namespaces)}" + ) + + wanted = _expected_pod_names(namespace_count, object_count) + validated_pods, problems = await _collect_owned_pods( + kube, run_id=run_id, expected_namespaces=expected_namespaces, wanted=wanted + ) + if problems.missing: + raise ValueError(f"missing expected pods: {', '.join(problems.missing)}") + if problems.mismatched: + raise ValueError(f"pods with mismatched ownership labels: {', '.join(problems.mismatched)}") + if problems.unexpected: + raise ValueError(f"unexpected pods in owned namespaces: {', '.join(problems.unexpected)}") + if problems.not_ready: + raise ValueError( + f"owned pods not Running or not Ready: {', '.join(problems.not_ready)}; " + f"the live run requires exactly {object_count} Running, Ready owned pods" + ) + return validated_pods + + +async def read_and_validate_owned_pods( + kube: KubeReadClient, + *, + run_id: str, + expected: Mapping[tuple[str, str], PodSummary], +) -> list[PodSummary]: + """Independently re-read the owned Pods and revalidate them before use. + + The ground-truth digest may only be computed from Pods that still are what + the ownership gate validated: same `(namespace, name)` identity set, same + UID (a delete/recreate produces a new one), and both ownership labels + intact. A Pod that lost either label, was replaced, disappeared, or a Pod + that appeared unexpectedly, is named in the raised error instead of being + silently folded into the digest. + """ + namespaces = sorted({namespace for namespace, _name in expected}) + found: dict[tuple[str, str], PodSummary] = {} + seen: set[tuple[str, str]] = set() + problems: list[str] = [] + for namespace in namespaces: + for pod in await kube.list_pods(namespace): + key = (namespace, pod.name) + baseline = expected.get(key) + if baseline is None: + problems.append(f"{namespace}/{pod.name} (unexpected pod)") + continue + seen.add(key) + if pod.uid != baseline.uid: + problems.append( + f"{namespace}/{pod.name} (uid changed {baseline.uid!r} -> {pod.uid!r})" + ) + elif not _owns(pod.labels, run_id): + problems.append(f"{namespace}/{pod.name} (lost ownership labels)") + else: + found[key] = pod + problems.extend( + f"{namespace}/{name} (missing)" + for namespace, name in expected + if (namespace, name) not in seen + ) + if problems: + raise ValueError(f"post-churn ownership revalidation failed: {', '.join(sorted(problems))}") + return list(found.values()) + + +def make_live_watch_source( + kube: KubeReadClient, + expected_namespaces: frozenset[str], + *, + run_id: str, + recorder: BenchmarkRecorder, + now: Callable[[], float] = monotonic, +) -> WatchSource: + """Filter the real cluster-wide Pod watch to exactly the expected, seeded + namespaces - unrelated cluster Pods (or namespaces sharing the cluster with + this run) must never enter the benchmark store. + + Event-to-render measurement starts *here*, at watch receipt of an owned + `MODIFIED` event, exactly where the deterministic `_ReplaySource` starts + it. Recording at patch acknowledgement instead would race the watch event + it is meant to measure (the two arrive over independent connections) and + would silently fold the write round-trip into a read-path latency metric. + + The event counter is shared across reconnects, so a re-LIST after a dropped + watch continues the same sequence. + """ + sequence = 0 + + async def _source(kind: str, _scope: str) -> AsyncIterator[tuple[str, PodSummary]]: + nonlocal sequence + if kind != "pods": + raise ValueError(f"run_live_replay only watches pods, got kind={kind!r}") + async for event_type, pod in kube.watch_pods(None): + if pod.namespace not in expected_namespaces: + continue + if event_type == "MODIFIED" and _owns(pod.labels, run_id): + sequence += 1 + recorder.record_event(sequence, now()) + yield (event_type, pod) + + return _source + + +def _first_error(group: BaseExceptionGroup[BaseException]) -> BaseException: + """The most informative leaf of a `TaskGroup` failure. + + A guard failure that aborts the run is reported to the caller as the + `ApiStatusError` it really is, not as an `ExceptionGroup` wrapper. Sibling + tasks cancelled *because* of that failure contribute `CancelledError` + leaves, which are only returned when there is nothing else. + """ + leaves: list[BaseException] = [] + pending: list[BaseException] = list(group.exceptions) + while pending: + exc = pending.pop(0) + if isinstance(exc, BaseExceptionGroup): + pending.extend(exc.exceptions) + else: + leaves.append(exc) + for exc in leaves: + if not isinstance(exc, asyncio.CancelledError): + return exc + return leaves[0] if leaves else group + + +def _parse_retry_seconds(value: object) -> float | None: + try: + seconds = float(str(value).strip()) + except (TypeError, ValueError): + return None + if not math.isfinite(seconds) or seconds < 0: + return None + return seconds + + +def _api_retry_after_seconds(exc: BaseException, body: str) -> float | None: + headers = getattr(exc, "headers", None) + if isinstance(headers, Mapping): + for key, value in headers.items(): + if str(key).lower() == "retry-after": + parsed = _parse_retry_seconds(value) + if parsed is not None: + return parsed + + try: + payload = json.loads(body) + except (json.JSONDecodeError, TypeError): + return None + if not isinstance(payload, dict): + return None + details = payload.get("details") + if not isinstance(details, dict): + return None + return _parse_retry_seconds(details.get("retryAfterSeconds")) + + +def _mutation_retry_delay_seconds( + exc: ApiStatusError, + *, + namespace: str, + name: str, + uid: str, + tick: str, + attempt: int, + limits: LiveLimits, +) -> float: + backoff = limits.mutation_retry_base_delay_seconds * (2 ** (attempt - 1)) + digest = hashlib.sha256(f"{namespace}\0{name}\0{uid}\0{tick}\0{attempt}".encode()).digest() + jitter_fraction = int.from_bytes(digest[:8], "big") / (1 << 64) + jittered_backoff = backoff * jitter_fraction + server_hint = exc.retry_after_seconds or 0.0 + return float( + min( + server_hint + jittered_backoff, + limits.mutation_retry_max_delay_seconds, + ) + ) + + +async def _mutate_once( + mutation_client: MutationClient, + *, + namespace: str, + name: str, + uid: str, + tick: str, + progress: ChurnProgress, + limits: LiveLimits, + sleep: Callable[[float], Awaitable[None]], + now: Callable[[], float], +) -> None: + """One guarded patch, bounded in time and retried only on HTTP 429. + + The retry re-issues the *identical* guarded patch, so it is still atomic + and still fail-closed: a Pod that lost its identity or labels between + attempts fails the `test` ops exactly as it would on the first attempt. + A server-provided delay is treated as a floor, then target-specific jitter + is added without exceeding the configured retry-delay ceiling. + Every other status - including a failed `test` (422) - propagates + immediately and aborts the whole run. + """ + progress.record_started(now()) + attempt = 0 + while True: + try: + async with asyncio.timeout(limits.mutation_timeout_seconds): + await mutation_client.patch_pod_labels_guarded(namespace, name, uid=uid, tick=tick) + except ApiStatusError as exc: + if ( + exc.status != _RETRYABLE_MUTATION_STATUS + or attempt >= limits.mutation_throttle_retries + ): + raise + progress.mutation_throttles += 1 + attempt += 1 + await sleep( + _mutation_retry_delay_seconds( + exc, + namespace=namespace, + name=name, + uid=uid, + tick=tick, + attempt=attempt, + limits=limits, + ) + ) + continue + progress.record_completed(now()) + return + + +async def drive_live_churn( + events: Iterable[ScheduledEvent], + *, + run_id: str, + namespace_count: int, + live_state: Mapping[tuple[str, str], PodSummary], + mutation_client: MutationClient, + options: ReplayOptions, + progress: ChurnProgress, + limits: LiveLimits, + profile: WorkloadProfile, + recorder: BenchmarkRecorder, +) -> None: + """Drive guarded churn at wall-clock time with explicit bounded concurrency. + + The schedule is followed exactly as `_ReplaySource` follows it (absolute + offsets converted to inter-event delays), but each mutation is dispatched + to a task instead of being awaited inline: one round trip per event caps a + serial driver far below the profile's scheduled rate, which would silently + understate the load the report claims to have applied. + + Concurrency is bounded by `LiveLimits.churn_concurrency` and every single + attempt by `LiveLimits.mutation_timeout_seconds`; nothing here is + unbounded. Any guard failure (`ApiStatusError`) cancels every sibling task + and propagates unchanged - there is no unguarded fallback and no attempt to + continue past a failed `test` op. + + This function never records event timing: event-to-render measurement + starts at watch receipt (`make_live_watch_source`), not at patch ack. + """ + now = options.monotonic_fn if options.monotonic_fn is not None else monotonic + sleep = options.async_sleep if options.async_sleep is not None else _sleep_default + semaphore = asyncio.Semaphore(limits.churn_concurrency) + start = now() + + async def _run(namespace: str, name: str, uid: str, tick: str) -> None: + try: + await _mutate_once( + mutation_client, + namespace=namespace, + name=name, + uid=uid, + tick=tick, + progress=progress, + limits=limits, + sleep=sleep, + now=now, + ) + finally: + semaphore.release() + + try: + async with asyncio.TaskGroup() as group: + # Burst-end offsets (absolute seconds) mark the moment each burst's + # window closes, so the post-burst backlog drain can be timed on the + # same real-clock axis the render pass records on — exactly as the + # deterministic replay driver does. Without this the live report + # leaves `post_burst_drain_seconds` empty and the published + # burst-drain budget cannot be evaluated. + burst_ends = burst_end_offsets(profile) + next_burst = 0 + for event in events: + elapsed = now() - start + delay = event.offset_seconds * options.time_scale - elapsed + if delay > 0: + await sleep(delay) + + while ( + next_burst < len(burst_ends) and event.offset_seconds >= burst_ends[next_burst] + ): + recorder.mark_burst_end(monotonic()) + next_burst += 1 + + namespace, name = live_object_identity(run_id, namespace_count, event.object_index) + current = live_state.get((namespace, name)) + if current is None: + raise ValueError( + f"live churn target {namespace}/{name} is not in the seeded state" + ) + # Bound in-flight work *before* creating the task, so a slow + # API server throttles the driver instead of accumulating an + # unbounded backlog of pending patches. + await semaphore.acquire() + group.create_task(_run(namespace, name, current.uid, str(event.sequence))) + except BaseExceptionGroup as group_error: + raise _first_error(group_error) from None + + +async def _cancel_and_drain(task: asyncio.Task[None]) -> None: + """Cancel *task* and wait for it to finish before any client is closed. + + Every in-flight mutation lives inside the churn task's `TaskGroup`, so + awaiting the churn task is exactly what guarantees no patch is still in + flight when `mutation_client.close()` runs - and that no mutation outlives + teardown. A cancellation delivered to *this* coroutine while it drains is + absorbed and retried a bounded number of times: the caller's own + `CancelledError` is already propagating out of the enclosing `try`, and + skipping the drain is precisely the failure mode this exists to prevent. + """ + task.cancel() + for _ in range(_CANCEL_DRAIN_ATTEMPTS): + if task.done(): + break + try: + await asyncio.wait({task}) + except asyncio.CancelledError: + continue + if task.done() and not task.cancelled(): + # Retrieve any exception so it is never reported as "never retrieved"; + # the error that triggered teardown is the one the caller sees. + task.exception() + + +def _store_digest(store: ResourceStore) -> str: + return summary_digest(cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES))) + + +#: The scoped UI-at-scale scenarios issue #186 requires, each a real Textual +#: pilot key sequence driven during active churn. Sequences are self-restoring: +#: they return the workspace to a single pane showing all owned rows so the +#: post-churn row-count and digest-convergence checks still see 1,000 rows. +#: +#: Every step declares the observable state it must reach. `pilot.press` +#: returning is not evidence: `describe` and `multi_log` bail out with an +#: "unavailable" warning when no manifest/log provider is wired, and the +#: all-namespaces toggle is a no-op while the configured namespace already is +#: `ALL_NAMESPACES` — none of which raises. Without the state check a run would +#: report UI-at-scale evidence it never gathered. + + +@dataclass(frozen=True) +class _UIStep: + """One key sequence plus the observable app state it must reach.""" + + keys: tuple[str, ...] + label: str + reached: Callable[[Any], bool] + + +@dataclass(frozen=True) +class _UIScenario: + """One named UI-at-scale scenario as an ordered list of verified steps.""" + + name: str + steps: tuple[_UIStep, ...] + + +#: Bound on one step reaching its target state. Generous on purpose: the step +#: competes with full-rate churn on the same event loop. +_UI_STEP_TIMEOUT = 30.0 + + +def _ui_scenarios(scoped_namespace: str) -> tuple[_UIScenario, ...]: + """Build the scenario table; the namespace scenario needs a real seeded + namespace to scope down to (and back out of).""" + return ( + # Filter to a substring present in every seeded Pod name ("bench-*"), so + # the filter exercises the real path without dropping any rows, then + # clear it so the bar releases focus and the workspace is restored. + _UIScenario( + "filter", + ( + _UIStep( + ("slash", "b", "e", "n", "c", "h"), + "filter applied", + lambda app: app.filter_pattern == "bench", + ), + _UIStep( + ("escape",), + "filter cleared", + lambda app: app.filter_pattern == "", + ), + ), + ), + # Sorting by age (a metrics-free column) keeps every row visible. + _UIScenario( + "sort", + ( + _UIStep( + ("A",), + "age sort active", + lambda app: app._sorts.get("pods") is not None, + ), + ), + ), + # Scope to one seeded namespace (favorite key 1), then back to all + # namespaces so the full 1,000-row topology is restored. + _UIScenario( + "namespace_switch", + ( + _UIStep( + ("1",), + "scoped to one namespace", + lambda app: app.current_scope == scoped_namespace, + ), + _UIStep( + ("0",), + "restored to all namespaces", + lambda app: app.current_scope == ALL_NAMESPACES, + ), + ), + ), + _UIScenario( + "split_pane", + ( + _UIStep(("ctrl+w", "v"), "second pane open", lambda app: len(app._panes) == 2), + _UIStep(("ctrl+w", "q"), "back to one pane", lambda app: len(app._panes) == 1), + ), + ), + _UIScenario( + "describe", + ( + _UIStep( + ("d",), + "describe screen open", + lambda app: isinstance(app.screen, DescribeScreen), + ), + _UIStep( + ("escape",), + "describe dismissed", + lambda app: not isinstance(app.screen, DescribeScreen), + ), + ), + ), + _UIScenario( + "multi_log", + ( + _UIStep(("L",), "log pane open", lambda app: bool(app._log_pane.display)), + _UIStep(("escape",), "log pane closed", lambda app: not app._log_pane.display), + ), + ), + ) + + +def _describe_provider( + kube: KubeReadClient, +) -> Callable[[str, str | None, str], Awaitable[dict[str, Any]]]: + """Read-only manifest fetcher for the `describe` UI-at-scale scenario. + + Only Pods are reachable in the live workspace, so anything else is a + programmer error rather than a cluster read. + """ + + async def get_manifest(kind: str, namespace: str | None, name: str) -> dict[str, Any]: + if kind != _PODS_META.plural: + raise ValueError(f"live qualification only describes pods, not {kind!r}") + return await kube.get_object(_PODS_META, namespace, name) + + return get_manifest + + +async def drive_ui_scenarios( + pilot: Any, + recorder: BenchmarkRecorder, + *, + now: Callable[[], float], + app: Any, + scoped_namespace: str, +) -> None: + """Drive the scoped UI-at-scale scenarios through the real Textual pilot. + + Each scenario is a real key sequence issued against the running app during + active churn; after every step the app must reach the observable state the + step names, or the scenario is recorded `ok=False`. Every scenario is + read-only navigation - no scenario writes, deletes, or drains - so this + never weakens live safety. A scenario that raises or that fails to reach a + target state is recorded as `ok=False` and never aborts the safety-critical + run; the remaining steps still run so the sequences restore a single-pane, + all-rows workspace and later convergence checks stay intact. + """ + for scenario in _ui_scenarios(scoped_namespace): + started = now() + ok = True + for step in scenario.steps: + try: + for key in step.keys: + await pilot.press(key) + await wait_for( + pilot, + lambda step=step: step.reached(app), # type: ignore[misc] # bind per step + timeout=_UI_STEP_TIMEOUT, + label=f"{scenario.name}: {step.label}", + recorder=recorder, + ) + except Exception: + ok = False + recorder.record_scenario(scenario.name, now() - started, ok) + + +def _check_row_count(row_count: int, expected: int) -> None: + """Re-assert the exact rendered row count before teardown. + + A late `WatchManager` reconnect clears and re-seeds the store, so a digest + that matched a moment ago can be recomputed over a partially re-listed + store. Checking the rendered row count at the same instant turns that into + an explicit, named failure instead of a plausible-looking report. + """ + if row_count != expected: + raise ValueError( + f"rendered row count regressed before teardown: expected {expected}, got {row_count}" + ) + + +@dataclass +class _LiveRunState: + """Values produced inside the measured window and consumed after it.""" + + expected_digest: str = "" + final_digest: str = "" + churn_started_before_input: bool = False + progress: ChurnProgress = field(default_factory=ChurnProgress) + + +async def _run_measured_window( + *, + app: MeasuredKorvidApp, + store: ResourceStore, + harness_kube: KubeReadClient, + mutation_client: MutationClient, + recorder: BenchmarkRecorder, + profile: WorkloadProfile, + options: ReplayOptions, + limits: LiveLimits, + run_id: str, + live_state: dict[tuple[str, str], PodSummary], + state: _LiveRunState, +) -> None: + """Drive the app: initial render, churn, ground-truth read, convergence. + + Everything that must observe a *live* watch happens inside this function, + including the independent cluster read and the digest convergence wait. + Reading ground truth after the watch had already been stopped would compare + a frozen store against a cluster that was still changing. + """ + now = options.monotonic_fn if options.monotonic_fn is not None else monotonic + async with app.run_test() as pilot: + table = app.query_one(ResourceTable) + + await wait_for( + pilot, + lambda: table.row_count == profile.object_count, + timeout=limits.initial_render_timeout_seconds, + label="initial owned pods rendered", + recorder=recorder, + ) + # The table is fully populated: mark the interactive boundary that + # closes the startup and LIST-to-populated-table phases. + recorder.mark_interactive(monotonic()) + + events = scheduled_events(profile) + state.progress.requested_events = len(events) + churn_task = asyncio.create_task( + drive_live_churn( + events, + run_id=run_id, + namespace_count=profile.namespace_count, + live_state=live_state, + mutation_client=mutation_client, + options=options, + progress=state.progress, + limits=limits, + profile=profile, + recorder=recorder, + ) + ) + try: + if events: + # Real ordering signal: input latency is only meaningful if + # churn was actually under way, so wait for the first mutation + # to be dispatched (or for churn to fail) before measuring. + await wait_for( + pilot, + lambda: state.progress.started > 0 or churn_task.done(), + timeout=limits.initial_render_timeout_seconds, + label="first churn mutation started", + recorder=recorder, + ) + state.churn_started_before_input = state.progress.started > 0 + + t0 = now() + await pilot.press("down") + recorder.record_input(now() - t0) + t0 = now() + await pilot.press("up") + recorder.record_input(now() - t0) + + # UI-at-scale evidence: drive the scoped scenarios (filter, sort, + # namespace switch, split pane, describe, multi-log) through the + # real pilot while churn is still active. Read-only navigation only. + await drive_ui_scenarios( + pilot, + recorder, + now=now, + app=app, + scoped_namespace=manifests.namespace_name(run_id, 0), + ) + + churn_timeout = ( + profile.duration_seconds * options.time_scale + limits.churn_grace_seconds + ) + async with asyncio.timeout(churn_timeout): + await churn_task + + # Ground truth, read independently *while the watch is still live* + # via the non-instrumented harness client (never polluting the + # application-path telemetry), and revalidated for ownership. + final_pods = await read_and_validate_owned_pods( + harness_kube, run_id=run_id, expected=live_state + ) + state.expected_digest = summary_digest(final_pods) + + await wait_for( + pilot, + lambda: ( + recorder.pending_count() == 0 and _store_digest(store) == state.expected_digest + ), + timeout=limits.convergence_timeout_seconds, + label=( + "store digest converged with the independently read cluster digest " + f"({state.expected_digest})" + ), + recorder=recorder, + ) + _check_row_count(table.row_count, profile.object_count) + # Independent of the digest above, which compares a store digest + # with a store digest and would accept a table full of stale cells. + check_rendered_rows(table, final_pods) + state.final_digest = _store_digest(store) + finally: + await _cancel_and_drain(churn_task) + + +async def run_live_replay( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, + deps: LiveDependencies | None = None, + limits: LiveLimits | None = None, +) -> ReplayReport: + """Replay churn against an already-seeded real AKS cluster and return metrics. + + Fail-closed order: `time_scale`/`run_id`/topology/profile validation, the + cluster identity gate, the ownership gate - all *before* any mutation + client is constructed - then the real application-path wiring + (`KubeClient` -> `WatchManager` -> `ResourceStore` -> `MeasuredKorvidApp`), + guarded metadata-only churn, and digest parity against an independent, + revalidated re-read of the cluster taken while the watch is still live. + + Two `KubeReadClient` connections are used: `harness_kube` (never wired to + telemetry) performs the ownership gate and the ground-truth re-read; + `kube` (wired to `recorder.record_api`) is only ever handed to + `make_live_watch_source`, so `ReplayReport.api` reports exactly the + production application read path. The ownership gate's validated Pod + snapshot is reused as the pre-churn uid snapshot - it is never re-listed. + """ + _validate_time_scale(options) + manifests.validate_run_id(run_id) + _validate_topology(profile) + # Re-check duration-dependent invariants: a caller may hand us a profile + # rewritten with `dataclasses.replace` (the CLI's `--duration`), which + # bypasses `load_profile` entirely. + validate_profile(profile) + + active_limits = limits if limits is not None else LiveLimits() + active_deps = deps if deps is not None else _default_dependencies(context) + facts = await _verify_cluster_identity( + context=context, + expected_cluster_id=expected_cluster_id, + deps=active_deps, + limits=active_limits, + ) + + # Fail closed on an untraceable evidence run: a live report must be tied to + # an immutable korvid commit, resolved *before* any client is constructed. + korvid_sha = active_deps.resolve_sha() + if korvid_sha is None: + raise ValueError( + "cannot resolve an immutable korvid SHA for the live evidence run; " + "set GITHUB_SHA in CI or run from a git checkout with a resolvable HEAD" + ) + + store = ResourceStore() + recorder = BenchmarkRecorder() + sampler = ProcessSampler(options.sample_interval) + state = _LiveRunState() + + harness_kube = active_deps.harness_kube_client_factory() + try: + async with asyncio.timeout(active_limits.read_connect_timeout_seconds): + await harness_kube.connect(context) + # Reused directly as the pre-churn uid snapshot below - no second, + # redundant listing pass over the same Pods. + live_state = await _verify_ownership( + harness_kube, + run_id=run_id, + namespace_count=profile.namespace_count, + object_count=profile.object_count, + ) + + expected_namespaces = frozenset( + manifests.namespace_name(run_id, i) for i in range(profile.namespace_count) + ) + + # Constructed only after the ownership gate passes, then connected + # eagerly under an explicit bound so credential-plugin latency is paid + # before the measured window rather than inside the first mutation. + mutation_client = active_deps.mutation_client_factory(run_id) + try: + async with asyncio.timeout(active_limits.mutation_connect_timeout_seconds): + await mutation_client.connect() + + kube = active_deps.kube_client_factory(recorder.record_api) + try: + async with asyncio.timeout(active_limits.read_connect_timeout_seconds): + await kube.connect(context) + source = make_live_watch_source( + kube, + expected_namespaces, + run_id=run_id, + recorder=recorder, + ) + watch_manager = WatchManager(store, source, retry_delay=0.0) + manifest = build_manifest( + profile, + korvid_sha=korvid_sha, + context=facts.context, + cluster_id=facts.cluster_id, + kubernetes_version=facts.kubernetes_version, + node_pools=facts.node_pools, + ) + + app = MeasuredKorvidApp( + config=KorvidConfig( + namespace=ALL_NAMESPACES, + # Key 1 scopes to a real seeded namespace; key 0 returns + # to all namespaces. Without a favorite the toggle is a + # no-op (the configured namespace already is + # ALL_NAMESPACES) and the scenario proves nothing. + favorite_namespaces=(manifests.namespace_name(run_id, 0),), + ), + store=store, + watch_manager=watch_manager, + recorder=recorder, + # Read-only providers: `describe` and `multi_log` bail out + # with an "unavailable" warning when these are None, so the + # scenarios would report success having done nothing. The + # harness connection is used so these reads stay out of the + # measured application read path. + aliases={"pods": _PODS_META}, + get_manifest=_describe_provider(harness_kube), + stream_logs=harness_kube.stream_logs, + ) + + sampler.start() + recorder.mark_process_start(monotonic()) + try: + await _run_measured_window( + app=app, + store=store, + harness_kube=harness_kube, + mutation_client=mutation_client, + recorder=recorder, + profile=profile, + options=options, + limits=active_limits, + run_id=run_id, + live_state=live_state, + state=state, + ) + finally: + process_samples = await sampler.stop() + await watch_manager.stop_all() + finally: + await kube.close() + finally: + await mutation_client.close() + finally: + await harness_kube.close() + + churn = state.progress.summary(requested_duration_seconds=profile.duration_seconds) + benchmark = recorder.report( + manifest, + process_samples, + final_digest=state.final_digest, + expected_digest=state.expected_digest, + churn=churn, + ) + + return ReplayReport( + object_count=profile.object_count, + expected_digest=state.expected_digest, + final_digest=state.final_digest, + dropped_updates=benchmark.dropped_updates, + rendered_updates=benchmark.rendered_updates, + render_passes=benchmark.render_passes, + coalesced_updates=benchmark.coalesced_updates, + event_to_render=benchmark.event_to_render, + input_latency=benchmark.input_latency, + churn_started_before_input=state.churn_started_before_input, + process=benchmark.process, + api=benchmark.api, + phases=benchmark.phases, + manifest=benchmark.manifest, + failures_injected=benchmark.failures_injected, + ui_scenarios=benchmark.ui_scenarios, + churn=churn, + ) diff --git a/tests/performance/manifests.py b/tests/performance/manifests.py new file mode 100644 index 00000000..cdcd482c --- /dev/null +++ b/tests/performance/manifests.py @@ -0,0 +1,147 @@ +from __future__ import annotations + +import re + +_RUN_ID_PATTERN = re.compile(r"^[a-z0-9](?:[-a-z0-9]{0,46}[a-z0-9])?$") +_SELECTOR_KEY_PATTERN = re.compile(r"^[a-z0-9](?:[-a-z0-9./]{0,251}[a-z0-9])?$") +_SELECTOR_VALUE_PATTERN = re.compile(r"^[a-z0-9](?:[-a-z0-9.]{0,61}[a-z0-9])?$") +_MANAGED_BY = "korvid-performance" +_BENCH_IMAGE = "registry.k8s.io/pause:3.10" + +#: Public ownership-label contract shared with `live.py`'s ownership gate, so +#: both modules agree on exactly what "owned by this seed run" means. +MANAGED_BY_LABEL = "app.kubernetes.io/managed-by" +MANAGED_BY_VALUE = _MANAGED_BY +RUN_LABEL = "korvid.dev/performance-run" + +#: Dedicated, *non-ownership* label the live harness rewrites to generate watch +#: traffic. It is never part of the ownership contract (nothing gates on its +#: value), it is user-owned so no controller reconciles it back, and churning it +#: keeps live mutations metadata-only exactly as the design doc requires. +TICK_LABEL = "korvid.dev/performance-tick" + + +def _validate_positive(value: int, label: str) -> int: + if value < 1: + raise ValueError(f"{label} must be a positive integer") + return value + + +def _validate_run_id(run_id: str) -> str: + if _RUN_ID_PATTERN.fullmatch(run_id): + return run_id + raise ValueError("run_id must be 1-48 lowercase letters, digits, or hyphens") + + +def validate_run_id(run_id: str) -> str: + """Public entry point so other benchmark modules (e.g. `live.py`) share + the exact `run_id` contract `seed-manifests` enforces, without importing + a private name across modules.""" + return _validate_run_id(run_id) + + +def _parse_node_selector(node_selector: str) -> dict[str, str]: + if node_selector.count("=") != 1: + raise ValueError("node_selector must be exactly one non-empty key=value pair") + key, value = node_selector.split("=", 1) + if not key or not value: + raise ValueError("node_selector must be exactly one non-empty key=value pair") + if key.strip() != key or value.strip() != value: + raise ValueError("node_selector must be exactly one non-empty key=value pair") + if not _SELECTOR_KEY_PATTERN.fullmatch(key) or not _SELECTOR_VALUE_PATTERN.fullmatch(value): + raise ValueError("node_selector must be exactly one non-empty key=value pair") + return {key: value} + + +def _common_labels(run_id: str) -> dict[str, str]: + return { + MANAGED_BY_LABEL: _MANAGED_BY, + RUN_LABEL: run_id, + } + + +def _namespace_name(run_id: str, namespace_index: int) -> str: + name = f"korvid-perf-{run_id}-{namespace_index}" + if len(name) > 63: + raise ValueError("generated namespace name must be 63 characters or fewer") + return name + + +def namespace_name(run_id: str, namespace_index: int) -> str: + """Public entry point for the exact namespace-naming formula + `seed-manifests` uses, so `live.py` maps object indices onto the same + namespaces without duplicating (and risking drift from) this formula.""" + return _namespace_name(run_id, namespace_index) + + +def pod_name(namespace_count: int, object_index: int) -> str: + """Public entry point for the exact Pod-naming formula `seed-manifests` + uses; see `namespace_name`.""" + return f"bench-{object_index // namespace_count}" + + +def build_seed_manifests( + *, + run_id: str, + namespace_count: int, + pods_per_namespace: int, + node_selector: str, +) -> tuple[dict[str, object], ...]: + run = _validate_run_id(run_id) + namespaces = _validate_positive(namespace_count, "namespace_count") + pods_each = _validate_positive(pods_per_namespace, "pods_per_namespace") + selector = _parse_node_selector(node_selector) + labels = _common_labels(run) + + manifests: list[dict[str, object]] = [] + namespace_names = tuple(_namespace_name(run, index) for index in range(namespaces)) + for name in namespace_names: + manifests.append( + { + "apiVersion": "v1", + "kind": "Namespace", + "metadata": { + "name": name, + "labels": dict(labels), + }, + } + ) + + for object_index in range(namespaces * pods_each): + namespace_index = object_index % namespaces + manifests.append( + { + "apiVersion": "v1", + "kind": "Pod", + "metadata": { + "name": pod_name(namespaces, object_index), + "namespace": namespace_names[namespace_index], + "labels": dict(labels), + }, + "spec": { + "nodeSelector": dict(selector), + "tolerations": [ + { + "key": "korvid.dev/performance", + "operator": "Equal", + "value": "true", + "effect": "NoSchedule", + } + ], + "containers": [ + { + "name": "bench", + "image": _BENCH_IMAGE, + "resources": { + "requests": { + "cpu": "5m", + "memory": "16Mi", + } + }, + } + ], + "restartPolicy": "Always", + }, + } + ) + return tuple(manifests) diff --git a/tests/performance/metrics.py b/tests/performance/metrics.py new file mode 100644 index 00000000..73b3b891 --- /dev/null +++ b/tests/performance/metrics.py @@ -0,0 +1,816 @@ +from __future__ import annotations + +import asyncio +import math +import tracemalloc +from collections import Counter +from collections.abc import Callable, Coroutine, Mapping, Sequence +from contextlib import suppress +from dataclasses import dataclass, field +from time import monotonic +from types import MappingProxyType + +import psutil # type: ignore[import-untyped] # dependency ships without inline stubs + +from korvid.k8s.telemetry import ReadTelemetryEvent + +_MIB = 1024 * 1024 + + +def _nearest_rank(samples: Sequence[float], percentile: float) -> float: + index = math.ceil(percentile * len(samples)) - 1 + return samples[index] + + +def _max_or_none(samples: Sequence[float]) -> float | None: + if not samples: + return None + return max(samples) + + +def _least_squares_slope(samples: Sequence[ProcessSample]) -> float | None: + if len(samples) < 2: + return None + xs = [sample.elapsed_seconds for sample in samples] + ys = [float(sample.rss_bytes) for sample in samples] + x_mean = sum(xs) / len(xs) + y_mean = sum(ys) / len(ys) + numerator = sum((x - x_mean) * (y - y_mean) for x, y in zip(xs, ys, strict=True)) + denominator = sum((x - x_mean) ** 2 for x in xs) + if denominator == 0: + return None + return (numerator / denominator) * 60 / _MIB + + +@dataclass(frozen=True) +class LatencySummary: + count: int + p50_seconds: float | None + p95_seconds: float | None + p99_seconds: float | None + maximum_seconds: float | None + + @classmethod + def from_samples(cls, samples: Sequence[float]) -> LatencySummary: + ordered = sorted(samples) + if not ordered: + return cls( + count=0, + p50_seconds=None, + p95_seconds=None, + p99_seconds=None, + maximum_seconds=None, + ) + return cls( + count=len(ordered), + p50_seconds=_nearest_rank(ordered, 0.50), + p95_seconds=_nearest_rank(ordered, 0.95), + p99_seconds=_nearest_rank(ordered, 0.99), + maximum_seconds=ordered[-1], + ) + + +@dataclass(frozen=True) +class ProcessSample: + elapsed_seconds: float + cpu_percent: float + rss_bytes: int + python_bytes: int + + +@dataclass(frozen=True) +class NodePoolInfo: + """One AKS node pool's identity, Kubernetes version, and node count. + + Part of the live run manifest so retained evidence records exactly which + node-pool topology was qualified (issue #186 / design "Fixed target"). + """ + + name: str + kubernetes_version: str + node_count: int + + +@dataclass(frozen=True) +class RunManifest: + profile_id: str + profile_hash: str + korvid_sha: str + python: str + textual: str + os: str + cpu_count: int + memory_bytes: int + #: Live-only cluster-matrix fields. Absent for deterministic replay, which + #: does not run against a real cluster; present for a live evidence run so + #: the retained report establishes which cluster was actually qualified. + context: str | None = None + cluster_id: str | None = None + kubernetes_version: str | None = None + node_pools: tuple[NodePoolInfo, ...] = () + + +@dataclass(frozen=True) +class ProcessSummary: + sample_count: int + cpu_percent_max: float | None + rss_bytes_max: int | None + python_bytes_max: int | None + rss_slope_mib_per_minute: float | None + #: Elapsed-seconds boundary below which samples are treated as warm-up + #: (process start plus initial table population) and excluded from the + #: steady-state RSS slope fit. The published budget is explicitly a + #: *post-warm-up* slope, so startup allocation must not contaminate it. + rss_slope_warmup_boundary_seconds: float + #: Number of post-warm-up samples the slope was actually fitted over. + rss_slope_sample_count: int + + @classmethod + def from_samples( + cls, + samples: Sequence[ProcessSample], + *, + warmup_boundary_seconds: float = 0.0, + ) -> ProcessSummary: + if not samples: + return cls( + sample_count=0, + cpu_percent_max=None, + rss_bytes_max=None, + python_bytes_max=None, + rss_slope_mib_per_minute=None, + rss_slope_warmup_boundary_seconds=warmup_boundary_seconds, + rss_slope_sample_count=0, + ) + steady_state = [ + sample for sample in samples if sample.elapsed_seconds >= warmup_boundary_seconds + ] + return cls( + sample_count=len(samples), + cpu_percent_max=max(sample.cpu_percent for sample in samples), + rss_bytes_max=max(sample.rss_bytes for sample in samples), + python_bytes_max=max(sample.python_bytes for sample in samples), + rss_slope_mib_per_minute=_least_squares_slope(steady_state), + rss_slope_warmup_boundary_seconds=warmup_boundary_seconds, + rss_slope_sample_count=len(steady_state), + ) + + +@dataclass(frozen=True) +class ApiSummary: + operations: Mapping[str, int] + paths: Mapping[str, Mapping[str, int]] + decoded_bytes: int + object_count: int + watch_events: int + reconnects: int + relists: int + throttles: int + authorization_failures: int + + @classmethod + def from_events(cls, events: Sequence[ReadTelemetryEvent]) -> ApiSummary: + operations = Counter[str]() + paths: dict[str, Counter[str]] = {} + decoded_bytes = 0 + object_count = 0 + watch_events = 0 + throttles = 0 + authorization_failures = 0 + relists = 0 + relist_candidates: set[str] = set() + reconnects = 0 + #: Paths whose stream errored and has not been re-opened yet. A + #: reconnect is recovery from a dropped watch, so it must be counted + #: from that recovery - not inferred from the number of `watch_open`s. + #: A deliberate stop/start (the live `namespace_switch` scenario scopes + #: the table down and back) re-opens the same path with no error in + #: between and is not a reconnect. + dropped_paths: set[str] = set() + for event in events: + operations[event.operation] += 1 + paths.setdefault(event.path, Counter())[event.operation] += 1 + decoded_bytes += event.decoded_bytes + object_count += event.object_count + if event.operation == "watch_event": + watch_events += event.object_count + if event.status == 429: + throttles += 1 + if event.status in {401, 403}: + authorization_failures += 1 + if event.operation == "list" and event.path in relist_candidates: + relists += 1 + relist_candidates.remove(event.path) + if event.operation == "watch_open" and event.path in dropped_paths: + reconnects += 1 + dropped_paths.remove(event.path) + if event.operation == "error": + dropped_paths.add(event.path) + if event.status == 410: + relist_candidates.add(event.path) + return cls( + operations=MappingProxyType(dict(sorted(operations.items()))), + paths=MappingProxyType( + { + path: MappingProxyType(dict(sorted(counts.items()))) + for path, counts in sorted(paths.items(), key=lambda item: item[0]) + } + ), + decoded_bytes=decoded_bytes, + object_count=object_count, + watch_events=watch_events, + reconnects=reconnects, + relists=relists, + throttles=throttles, + authorization_failures=authorization_failures, + ) + + +@dataclass(frozen=True) +class ChurnSummary: + """What the churn generator was *asked* to do versus what it observably did. + + The design doc is explicit that "the generator rate and observed API + throttling are both recorded; requested rate is never reported as achieved + rate", so the requested schedule and the measured outcome are separate, + separately labelled fields. `mutation_throttles` counts 429 responses to + the harness's own write traffic and is deliberately *not* merged into + `ApiSummary.throttles`, which reports only the application read path. + """ + + requested_events: int + requested_events_per_second: float | None + observed_events: int + wall_seconds: float | None + achieved_events_per_second: float | None + mutation_throttles: int + + @classmethod + def from_observations( + cls, + *, + requested_events: int, + requested_duration_seconds: int, + observed_events: int, + wall_seconds: float | None, + mutation_throttles: int, + ) -> ChurnSummary: + requested_rate = ( + requested_events / requested_duration_seconds + if requested_duration_seconds > 0 + else None + ) + achieved_rate = ( + observed_events / wall_seconds + if wall_seconds is not None and wall_seconds > 0 + else None + ) + return cls( + requested_events=requested_events, + requested_events_per_second=requested_rate, + observed_events=observed_events, + wall_seconds=wall_seconds, + achieved_events_per_second=achieved_rate, + mutation_throttles=mutation_throttles, + ) + + +@dataclass(frozen=True) +class ScenarioResult: + """Outcome and latency of one scoped UI-at-scale scenario driven through + the real Textual pilot during live churn (issue #186: filter, sort, + namespace switch, split pane, describe, multi-log).""" + + name: str + latency_seconds: float + ok: bool + + +@dataclass(frozen=True) +class PhaseSummary: + """Explicit phase measurements the numeric budgets are stated against. + + These are recorded from named lifecycle marks the harness emits, rather + than inferred from aggregate update counts, so each budget in the design + doc (process start to interactive table, LIST completion to populated + table, backlog depth, and post-burst drain time) has a machine-readable + counterpart in every report. + """ + + process_start_to_interactive_seconds: float | None + list_to_populated_table_seconds: float | None + max_backlog_depth: int + post_burst_drain_seconds: tuple[float, ...] + max_post_burst_drain_seconds: float | None + + +@dataclass(frozen=True) +class BenchmarkReport: + manifest: RunManifest + event_to_render: LatencySummary + input_latency: LatencySummary + process: ProcessSummary + api: ApiSummary + phases: PhaseSummary + rendered_updates: int + render_passes: int + coalesced_updates: int + dropped_updates: int + final_digest: str + #: The workload digest the run is expected to converge to. Persisting it + #: (and `digest_match`) means a report can demonstrate digest correctness + #: after the process exits and name which side differed. `None` only for + #: partial reports built without an oracle digest. + expected_digest: str | None = None + #: `True` only when an expected digest was supplied and equals the final + #: digest; pass/fail depends on this, so it is serialized explicitly. + digest_match: bool = False + #: How many of each injected failure kind (gone/throttled/forbidden/slow/ + #: metrics_unavailable/slow_logs) were actually exercised, so a report is + #: self-describing evidence that a failure profile ran rather than a bare + #: schema literal. + failures_injected: Mapping[str, int] = field(default_factory=lambda: MappingProxyType({})) + #: Scoped UI-at-scale scenarios driven during live churn (empty offline). + ui_scenarios: tuple[ScenarioResult, ...] = () + #: Present only for runs that drive real mutations (live replay). + churn: ChurnSummary | None = None + + +class ProcessSampler: + _managed_tracemalloc_users = 0 + + def __init__(self, interval_seconds: float, clock: Callable[[], float] = monotonic) -> None: + self._interval_seconds = interval_seconds + self._clock = clock + self._process = psutil.Process() + self._start_time: float | None = None + self._samples: list[ProcessSample] = [] + self._task: asyncio.Task[None] | None = None + self._uses_managed_tracing = False + + def start(self) -> None: + if self._task is not None: + raise RuntimeError( + "ProcessSampler.start() cannot run while sampling is already running" + ) + self._samples.clear() + self._start_time = self._clock() + self._uses_managed_tracing = self._acquire_tracemalloc() + task: Coroutine[object, object, None] | None = None + try: + self._process.cpu_percent() + task = self._run() + self._task = asyncio.create_task(task) + except BaseException: + if task is not None: + task.close() + if self._uses_managed_tracing: + self._release_tracemalloc() + self._uses_managed_tracing = False + self._start_time = None + raise + + async def stop(self) -> tuple[ProcessSample, ...]: + if self._task is None: + return tuple(self._samples) + task = self._task + self._task = None + task.cancel() + try: + with suppress(asyncio.CancelledError): + await task + finally: + # A sampling failure must still release managed tracing: raising + # past this point would leak the tracemalloc lease *and* skip the + # caller's own teardown (watch manager, benchmark tasks). + if self._uses_managed_tracing: + self._release_tracemalloc() + self._uses_managed_tracing = False + return tuple(self._samples) + + @classmethod + def _acquire_tracemalloc(cls) -> bool: + if tracemalloc.is_tracing(): + if cls._managed_tracemalloc_users == 0: + return False + else: + tracemalloc.start() + cls._managed_tracemalloc_users += 1 + return True + + @classmethod + def _release_tracemalloc(cls) -> None: + cls._managed_tracemalloc_users -= 1 + if cls._managed_tracemalloc_users == 0 and tracemalloc.is_tracing(): + tracemalloc.stop() + + async def _run(self) -> None: + assert self._start_time is not None + while True: + self._samples.append( + ProcessSample( + elapsed_seconds=self._clock() - self._start_time, + cpu_percent=self._process.cpu_percent(), + rss_bytes=self._process.memory_info().rss, + python_bytes=tracemalloc.get_traced_memory()[0], + ) + ) + await asyncio.sleep(self._interval_seconds) + + +class BenchmarkRecorder: + def __init__(self) -> None: + self._pending_events: list[tuple[int, float]] = [] + self._event_to_render: list[float] = [] + self._input_latency: list[float] = [] + self._api_events: list[ReadTelemetryEvent] = [] + self._rendered_updates = 0 + self._render_passes = 0 + self._coalesced_updates = 0 + self._max_backlog_depth = 0 + self._process_start_at: float | None = None + self._list_complete_at: float | None = None + self._interactive_at: float | None = None + self._burst_end_pending: list[float] = [] + self._post_burst_drains: list[float] = [] + self._failures_injected: Counter[str] = Counter() + self._ui_scenarios: list[ScenarioResult] = [] + + def record_event(self, sequence: int, received_at: float) -> None: + self._pending_events.append((sequence, received_at)) + self._max_backlog_depth = max(self._max_backlog_depth, len(self._pending_events)) + + def mark_process_start(self, at: float) -> None: + """Record the instant the benchmarked process/app began starting up. + + Paired with `mark_interactive` to measure process-start-to-interactive + and to derive the steady-state RSS-slope warm-up boundary. First mark + wins so a later, redundant call cannot move the origin. + """ + if self._process_start_at is None: + self._process_start_at = at + + def mark_list_complete(self, at: float) -> None: + """Record when the initial LIST finished streaming its rows. + + First mark wins: reconnect re-LISTs must not overwrite the initial + LIST-to-populated-table measurement. + """ + if self._list_complete_at is None: + self._list_complete_at = at + + def mark_interactive(self, at: float) -> None: + """Record when the table first became fully populated/interactive.""" + if self._interactive_at is None: + self._interactive_at = at + + def mark_burst_end(self, at: float) -> None: + """Record the end of a churn burst so post-burst drain can be timed. + + With events still pending the drain is resolved by `record_render` the + next time the backlog empties at or after this instant. With an already + empty backlog there is nothing to drain, so the sample is `0.0` right + away: leaving the marker pending would let the next unrelated + steady-state render report its own latency as a drain, or drop the + sample entirely if no later render arrives. + """ + if not self._pending_events: + self._post_burst_drains.append(0.0) + return + self._burst_end_pending.append(at) + + def pending_count(self) -> int: + """Number of recorded events not yet flushed by a render pass. + + Public so replay/live harnesses can wait for the backlog to drain + without reaching into the recorder's internals. + """ + return len(self._pending_events) + + def api_errors(self) -> tuple[ReadTelemetryEvent, ...]: + """Every recorded `error` read-telemetry event, in arrival order. + + Public so a harness can turn an opaque wait timeout into a message + naming the underlying API status (403/410/429) that caused it. + """ + return tuple(event for event in self._api_events if event.operation == "error") + + def record_render(self, rendered_at: float) -> None: + if not self._pending_events: + return + pending = tuple(self._pending_events) + self._pending_events.clear() + self._render_passes += 1 + self._rendered_updates += len(pending) + self._coalesced_updates += max(len(pending) - 1, 0) + for _, received_at in pending: + self._event_to_render.append(rendered_at - received_at) + # The backlog is empty again: resolve every burst whose end has already + # passed into a drain measurement (end -> backlog-clear interval). + if self._burst_end_pending: + resolved = [end for end in self._burst_end_pending if end <= rendered_at] + for end in resolved: + self._post_burst_drains.append(rendered_at - end) + self._burst_end_pending = [end for end in self._burst_end_pending if end > rendered_at] + + def record_input(self, latency_seconds: float) -> None: + self._input_latency.append(latency_seconds) + + def record_api(self, event: ReadTelemetryEvent) -> None: + self._api_events.append(event) + # The first WATCH open follows the initial LIST completing on both the + # replay and live paths, so it is a uniform, telemetry-driven signal + # for the LIST-to-populated-table boundary (first mark wins). + if event.operation == "watch_open": + self.mark_list_complete(monotonic()) + + def record_failure(self, kind: str) -> None: + """Record that one injected failure of *kind* was actually exercised. + + Report evidence that a failure profile ran - distinct from the profile + merely declaring it - covering every versioned kind, including + `metrics_unavailable` and `slow_logs` which do not raise on the + resource watch path. + """ + self._failures_injected[kind] += 1 + + def record_scenario(self, name: str, latency_seconds: float, ok: bool) -> None: + """Record one scoped UI-at-scale scenario outcome and latency.""" + self._ui_scenarios.append(ScenarioResult(name=name, latency_seconds=latency_seconds, ok=ok)) + + def phases(self) -> PhaseSummary: + """The explicit phase measurements derived from the lifecycle marks.""" + startup: float | None = None + if self._process_start_at is not None and self._interactive_at is not None: + startup = self._interactive_at - self._process_start_at + list_to_table: float | None = None + if self._list_complete_at is not None and self._interactive_at is not None: + list_to_table = self._interactive_at - self._list_complete_at + return PhaseSummary( + process_start_to_interactive_seconds=startup, + list_to_populated_table_seconds=list_to_table, + max_backlog_depth=self._max_backlog_depth, + post_burst_drain_seconds=tuple(self._post_burst_drains), + max_post_burst_drain_seconds=( + max(self._post_burst_drains) if self._post_burst_drains else None + ), + ) + + def _warmup_boundary_seconds(self) -> float: + if self._process_start_at is None or self._interactive_at is None: + return 0.0 + return max(self._interactive_at - self._process_start_at, 0.0) + + def report( + self, + manifest: RunManifest, + process_samples: Sequence[ProcessSample], + *, + final_digest: str, + expected_digest: str | None = None, + churn: ChurnSummary | None = None, + ) -> BenchmarkReport: + return BenchmarkReport( + manifest=manifest, + event_to_render=LatencySummary.from_samples(self._event_to_render), + input_latency=LatencySummary.from_samples(self._input_latency), + process=ProcessSummary.from_samples( + process_samples, + warmup_boundary_seconds=self._warmup_boundary_seconds(), + ), + api=ApiSummary.from_events(self._api_events), + phases=self.phases(), + rendered_updates=self._rendered_updates, + render_passes=self._render_passes, + coalesced_updates=self._coalesced_updates, + dropped_updates=len(self._pending_events), + final_digest=final_digest, + expected_digest=expected_digest, + digest_match=expected_digest is not None and expected_digest == final_digest, + failures_injected=MappingProxyType(dict(sorted(self._failures_injected.items()))), + ui_scenarios=tuple(self._ui_scenarios), + churn=churn, + ) + + +def report_payload(report: BenchmarkReport) -> dict[str, object]: + api_operations = dict(report.api.operations) + api_paths = {path: dict(counts) for path, counts in report.api.paths.items()} + return { + "manifest": { + "profile_id": report.manifest.profile_id, + "profile_hash": report.manifest.profile_hash, + "korvid_sha": report.manifest.korvid_sha, + "python": report.manifest.python, + "textual": report.manifest.textual, + "os": report.manifest.os, + "cpu_count": report.manifest.cpu_count, + "memory_bytes": report.manifest.memory_bytes, + "context": report.manifest.context, + "cluster_id": report.manifest.cluster_id, + "kubernetes_version": report.manifest.kubernetes_version, + "node_pools": [ + { + "name": pool.name, + "kubernetes_version": pool.kubernetes_version, + "node_count": pool.node_count, + } + for pool in report.manifest.node_pools + ], + }, + "latency": { + "event_to_render": { + "count": report.event_to_render.count, + "p50_seconds": report.event_to_render.p50_seconds, + "p95_seconds": report.event_to_render.p95_seconds, + "p99_seconds": report.event_to_render.p99_seconds, + "maximum_seconds": report.event_to_render.maximum_seconds, + }, + "input": { + "count": report.input_latency.count, + "p50_seconds": report.input_latency.p50_seconds, + "p95_seconds": report.input_latency.p95_seconds, + "p99_seconds": report.input_latency.p99_seconds, + "maximum_seconds": report.input_latency.maximum_seconds, + }, + }, + "process": { + "sample_count": report.process.sample_count, + "cpu_percent_max": report.process.cpu_percent_max, + "rss_bytes_max": report.process.rss_bytes_max, + "python_bytes_max": report.process.python_bytes_max, + "rss_slope_mib_per_minute": report.process.rss_slope_mib_per_minute, + "rss_slope_warmup_boundary_seconds": (report.process.rss_slope_warmup_boundary_seconds), + "rss_slope_sample_count": report.process.rss_slope_sample_count, + }, + "phases": { + "process_start_to_interactive_seconds": ( + report.phases.process_start_to_interactive_seconds + ), + "list_to_populated_table_seconds": (report.phases.list_to_populated_table_seconds), + "max_backlog_depth": report.phases.max_backlog_depth, + "post_burst_drain_seconds": list(report.phases.post_burst_drain_seconds), + "max_post_burst_drain_seconds": report.phases.max_post_burst_drain_seconds, + }, + "api": { + "operations": api_operations, + "paths": api_paths, + "decoded_bytes": report.api.decoded_bytes, + "object_count": report.api.object_count, + "watch_events": report.api.watch_events, + "reconnects": report.api.reconnects, + "relists": report.api.relists, + "throttles": report.api.throttles, + "authorization_failures": report.api.authorization_failures, + }, + "updates": { + "rendered_updates": report.rendered_updates, + "render_passes": report.render_passes, + "coalesced_updates": report.coalesced_updates, + "dropped_updates": report.dropped_updates, + }, + "churn": _churn_payload(report.churn), + "failures_injected": dict(report.failures_injected), + "ui_scenarios": [ + { + "name": scenario.name, + "latency_seconds": scenario.latency_seconds, + "ok": scenario.ok, + } + for scenario in report.ui_scenarios + ], + "digests": { + "expected": report.expected_digest, + "final": report.final_digest, + "match": report.digest_match, + }, + } + + +def _churn_payload(churn: ChurnSummary | None) -> dict[str, object] | None: + if churn is None: + return None + return { + "requested_events": churn.requested_events, + "requested_events_per_second": churn.requested_events_per_second, + "observed_events": churn.observed_events, + "wall_seconds": churn.wall_seconds, + "achieved_events_per_second": churn.achieved_events_per_second, + "mutation_throttles": churn.mutation_throttles, + } + + +def render_markdown(report: BenchmarkReport) -> str: + churn = report.churn + operation_lines = [ + f"- {operation}: `{count}`" for operation, count in report.api.operations.items() + ] + failure_lines = [ + f"- {kind}: `{count}`" for kind, count in report.failures_injected.items() + ] or ["- none: `0`"] + scenario_lines = [ + f"- {scenario.name}: `{_format_seconds(scenario.latency_seconds)}` " + f"(ok={str(scenario.ok).lower()})" + for scenario in report.ui_scenarios + ] or ["- none"] + manifest_lines = [ + f"- Profile ID: `{report.manifest.profile_id}`", + f"- Profile hash: `{report.manifest.profile_hash}`", + f"- Korvid SHA: `{report.manifest.korvid_sha}`", + ] + if report.manifest.context is not None: + manifest_lines.append(f"- Context: `{report.manifest.context}`") + if report.manifest.cluster_id is not None: + manifest_lines.append(f"- Cluster ARM id: `{report.manifest.cluster_id}`") + if report.manifest.kubernetes_version is not None: + manifest_lines.append(f"- Kubernetes version: `{report.manifest.kubernetes_version}`") + for pool in report.manifest.node_pools: + manifest_lines.append( + f"- Node pool `{pool.name}`: {pool.node_count} node(s) at `{pool.kubernetes_version}`" + ) + lines = [ + "# Large-cluster benchmark report", + "", + "## Run manifest", + *manifest_lines, + "", + "## Latency", + f"- Event to render p95: `{_format_seconds(report.event_to_render.p95_seconds)}`", + f"- Event to render p99: `{_format_seconds(report.event_to_render.p99_seconds)}`", + f"- Event to render max: `{_format_seconds(report.event_to_render.maximum_seconds)}`", + f"- Input latency p95: `{_format_seconds(report.input_latency.p95_seconds)}`", + "", + "## Process", + f"- CPU max: `{_format_float(report.process.cpu_percent_max)}`", + f"- RSS max: `{_format_int(report.process.rss_bytes_max)}`", + f"- RSS slope: `{_format_slope(report.process.rss_slope_mib_per_minute)}`", + f"- RSS slope warm-up boundary: " + f"`{_format_seconds(report.process.rss_slope_warmup_boundary_seconds)}`", + f"- RSS slope samples: `{report.process.rss_slope_sample_count}`", + "", + "## Phases", + f"- Process start to interactive: " + f"`{_format_seconds(report.phases.process_start_to_interactive_seconds)}`", + f"- LIST to populated table: " + f"`{_format_seconds(report.phases.list_to_populated_table_seconds)}`", + f"- Max backlog depth: `{report.phases.max_backlog_depth}`", + f"- Max post-burst drain: `{_format_seconds(report.phases.max_post_burst_drain_seconds)}`", + "", + "## Churn", + f"- Requested events: `{_format_int(churn.requested_events if churn else None)}`", + f"- Requested churn rate: `{_format_rate(churn.requested_events_per_second if churn else None)}`", + f"- Observed events: `{_format_int(churn.observed_events if churn else None)}`", + f"- Churn wall time: `{_format_seconds(churn.wall_seconds if churn else None)}`", + f"- Achieved churn rate: `{_format_rate(churn.achieved_events_per_second if churn else None)}`", + f"- Mutation throttles (429): `{_format_int(churn.mutation_throttles if churn else None)}`", + "", + "## Updates", + f"- Rendered updates: `{report.rendered_updates}`", + f"- Coalesced updates: `{report.coalesced_updates}`", + f"- Dropped updates: `{report.dropped_updates}`", + "", + "## API operations", + *operation_lines, + "", + "## Failures injected", + *failure_lines, + "", + "## UI-at-scale scenarios", + *scenario_lines, + "", + "## Digests", + f"- Expected digest: `{report.expected_digest or 'n/a'}`", + f"- Final digest: `{report.final_digest}`", + f"- Digest match: `{str(report.digest_match).lower()}`", + ] + return "\n".join(lines) + "\n" + + +def _format_seconds(value: float | None) -> str: + if value is None: + return "n/a" + return f"{value:.3f}s" + + +def _format_float(value: float | None) -> str: + if value is None: + return "n/a" + return f"{value:.1f}" + + +def _format_int(value: int | None) -> str: + if value is None: + return "n/a" + return str(value) + + +def _format_rate(value: float | None) -> str: + if value is None: + return "n/a" + return f"{value:.2f} events/s" + + +def _format_slope(value: float | None) -> str: + if value is None: + return "n/a" + return f"{value:.2f} MiB/min" diff --git a/tests/performance/profile.py b/tests/performance/profile.py new file mode 100644 index 00000000..0658136d --- /dev/null +++ b/tests/performance/profile.py @@ -0,0 +1,203 @@ +from __future__ import annotations + +import json +from dataclasses import dataclass +from itertools import pairwise +from pathlib import Path +from typing import Any, Literal, cast + +FailureKind = Literal["gone", "throttled", "forbidden", "slow", "metrics_unavailable", "slow_logs"] +_FAILURE_KINDS = frozenset( + {"gone", "throttled", "forbidden", "slow", "metrics_unavailable", "slow_logs"} +) +_PROFILE_KEYS = frozenset( + { + "schema_version", + "id", + "seed", + "object_count", + "namespace_count", + "steady_events_per_second", + "duration_seconds", + "bursts", + "failures", + } +) +_BURST_KEYS = frozenset({"start_second", "duration_seconds", "events_per_second"}) +_FAILURE_KEYS = frozenset({"kind", "at_event"}) + + +@dataclass(frozen=True) +class Burst: + start_second: int + duration_seconds: int + events_per_second: int + + +@dataclass(frozen=True) +class FailureInjection: + kind: FailureKind + at_event: int + + +@dataclass(frozen=True) +class WorkloadProfile: + schema_version: int + id: str + seed: int + object_count: int + namespace_count: int + steady_events_per_second: int + duration_seconds: int + bursts: tuple[Burst, ...] + failures: tuple[FailureInjection, ...] + + +def _mapping(value: Any, label: str) -> dict[str, Any]: + if not isinstance(value, dict): + raise ValueError(f"{label} must be a mapping") + return value + + +def _unknown(raw: dict[str, Any], allowed: frozenset[str], label: str) -> None: + unknown = sorted(set(raw) - allowed) + if unknown: + raise ValueError(f"{label} has unknown keys: {', '.join(unknown)}") + + +def _int(raw: dict[str, Any], key: str, *, positive: bool = False) -> int: + value = raw.get(key) + if isinstance(value, bool) or not isinstance(value, int): + raise ValueError(f"{key} must be an integer") + if positive and value < 1: + raise ValueError(f"{key} must be a positive integer") + return value + + +def _bursts(raw: Any) -> tuple[Burst, ...]: + if not isinstance(raw, list): + raise ValueError("bursts must be a list") + result: list[Burst] = [] + for index, value in enumerate(raw, 1): + item = _mapping(value, f"burst {index}") + _unknown(item, _BURST_KEYS, f"burst {index}") + burst = Burst( + start_second=_int(item, "start_second"), + duration_seconds=_int(item, "duration_seconds", positive=True), + events_per_second=_int(item, "events_per_second", positive=True), + ) + result.append(burst) + return tuple(sorted(result, key=lambda burst: burst.start_second)) + + +def _failures(raw: Any) -> tuple[FailureInjection, ...]: + if not isinstance(raw, list): + raise ValueError("failures must be a list") + result: list[FailureInjection] = [] + for index, value in enumerate(raw, 1): + item = _mapping(value, f"failure {index}") + _unknown(item, _FAILURE_KEYS, f"failure {index}") + kind = item.get("kind") + if kind not in _FAILURE_KINDS: + raise ValueError(f"failure {index} kind must be one of {sorted(_FAILURE_KINDS)}") + result.append( + FailureInjection( + kind=cast(FailureKind, kind), + at_event=_int(item, "at_event", positive=True), + ) + ) + # `run_replay` indexes failures by `at_event`, so a duplicate position + # would silently discard all but one while the profile hash and the report + # still claim every declared injection was exercised. + positions = [failure.at_event for failure in result] + duplicates = sorted({position for position in positions if positions.count(position) > 1}) + if duplicates: + repeated = ", ".join(str(position) for position in duplicates) + raise ValueError(f"failures must declare distinct at_event positions; repeated: {repeated}") + return tuple(sorted(result, key=lambda failure: failure.at_event)) + + +def load_profile(path: Path) -> WorkloadProfile: + raw = _mapping(json.loads(path.read_text()), path.name) + _unknown(raw, _PROFILE_KEYS, path.name) + schema_version = _int(raw, "schema_version", positive=True) + if schema_version != 1: + raise ValueError("schema_version must be 1") + profile_id = raw.get("id") + if not isinstance(profile_id, str) or not profile_id.strip(): + raise ValueError("id must be a non-empty string") + object_count = _int(raw, "object_count", positive=True) + namespace_count = _int(raw, "namespace_count", positive=True) + if object_count % namespace_count: + raise ValueError("object_count must be divisible by namespace_count") + duration = _int(raw, "duration_seconds", positive=True) + steady = _int(raw, "steady_events_per_second") + if steady < 0: + raise ValueError("steady_events_per_second must be non-negative") + profile = WorkloadProfile( + schema_version=schema_version, + id=profile_id, + seed=_int(raw, "seed"), + object_count=object_count, + namespace_count=namespace_count, + steady_events_per_second=steady, + duration_seconds=duration, + bursts=_bursts(raw.get("bursts")), + failures=_failures(raw.get("failures")), + ) + validate_profile(profile) + return profile + + +def validate_profile(profile: WorkloadProfile) -> None: + """Re-check every duration-dependent invariant of an assembled profile. + + `load_profile` calls this on load, and any caller that rewrites a loaded + profile - notably the CLI's `--duration` override, which uses + `dataclasses.replace` and therefore skips the loader entirely - must call + it again. Without it a shortened duration can leave a burst hanging past + the end of the run or a failure injection past the last planned event, + which only surfaces much later as an opaque generator assertion. + + Raises: + ValueError: a burst falls outside `duration_seconds`, two bursts + overlap, or a failure injection is scheduled past the last + planned event. + """ + for burst in profile.bursts: + if burst.start_second < 0 or burst.start_second + burst.duration_seconds > ( + profile.duration_seconds + ): + raise ValueError( + f"burst at second {burst.start_second} lasting {burst.duration_seconds}s " + f"falls outside duration_seconds={profile.duration_seconds}" + ) + ordered = sorted(profile.bursts, key=lambda burst: burst.start_second) + for previous, current in pairwise(ordered): + if previous.start_second + previous.duration_seconds > current.start_second: + raise ValueError("bursts must not overlap") + total = planned_event_count(profile) + if any(failure.at_event > total for failure in profile.failures): + raise ValueError( + f"failure at_event exceeds planned event count ({total}) for " + f"duration_seconds={profile.duration_seconds}" + ) + + +def planned_event_count(profile: WorkloadProfile) -> int: + total = profile.duration_seconds * profile.steady_events_per_second + for burst in profile.bursts: + total -= round(burst.duration_seconds * profile.steady_events_per_second) + total += round(burst.duration_seconds * burst.events_per_second) + return total + + +def burst_end_offsets(profile: WorkloadProfile) -> tuple[float, ...]: + """Absolute second offsets at which each burst's window closes. + + Ordered ascending so a driver can mark post-burst backlog drain the moment + the schedule leaves a burst, on the same time axis event offsets use. + """ + return tuple( + sorted(float(burst.start_second + burst.duration_seconds) for burst in profile.bursts) + ) diff --git a/tests/performance/profiles/aks-1k.json b/tests/performance/profiles/aks-1k.json new file mode 100644 index 00000000..ec02f82b --- /dev/null +++ b/tests/performance/profiles/aks-1k.json @@ -0,0 +1,15 @@ +{ + "schema_version": 1, + "id": "aks-1k", + "seed": 186, + "object_count": 1000, + "namespace_count": 20, + "steady_events_per_second": 200, + "duration_seconds": 30, + "bursts": [ + {"start_second": 5, "duration_seconds": 1, "events_per_second": 1000}, + {"start_second": 15, "duration_seconds": 1, "events_per_second": 1000}, + {"start_second": 25, "duration_seconds": 1, "events_per_second": 1000} + ], + "failures": [] +} diff --git a/tests/performance/profiles/aks-live-1k.json b/tests/performance/profiles/aks-live-1k.json new file mode 100644 index 00000000..13bc9cea --- /dev/null +++ b/tests/performance/profiles/aks-live-1k.json @@ -0,0 +1,15 @@ +{ + "schema_version": 1, + "id": "aks-live-1k", + "seed": 186, + "object_count": 1000, + "namespace_count": 20, + "steady_events_per_second": 20, + "duration_seconds": 1800, + "bursts": [ + {"start_second": 300, "duration_seconds": 30, "events_per_second": 100}, + {"start_second": 900, "duration_seconds": 30, "events_per_second": 100}, + {"start_second": 1500, "duration_seconds": 30, "events_per_second": 100} + ], + "failures": [] +} diff --git a/tests/performance/profiles/burst-50k.json b/tests/performance/profiles/burst-50k.json new file mode 100644 index 00000000..c71fda15 --- /dev/null +++ b/tests/performance/profiles/burst-50k.json @@ -0,0 +1,15 @@ +{ + "schema_version": 1, + "id": "burst-50k", + "seed": 186, + "object_count": 50000, + "namespace_count": 500, + "steady_events_per_second": 200, + "duration_seconds": 30, + "bursts": [ + {"start_second": 5, "duration_seconds": 1, "events_per_second": 1000}, + {"start_second": 15, "duration_seconds": 1, "events_per_second": 1000}, + {"start_second": 25, "duration_seconds": 1, "events_per_second": 1000} + ], + "failures": [] +} diff --git a/tests/performance/profiles/smoke-1k.json b/tests/performance/profiles/smoke-1k.json new file mode 100644 index 00000000..20ff4213 --- /dev/null +++ b/tests/performance/profiles/smoke-1k.json @@ -0,0 +1,13 @@ +{ + "schema_version": 1, + "id": "smoke-1k", + "seed": 186, + "object_count": 1000, + "namespace_count": 20, + "steady_events_per_second": 20, + "duration_seconds": 5, + "bursts": [ + {"start_second": 2, "duration_seconds": 1, "events_per_second": 100} + ], + "failures": [] +} diff --git a/tests/performance/profiles/steady-10k.json b/tests/performance/profiles/steady-10k.json new file mode 100644 index 00000000..77d17bf1 --- /dev/null +++ b/tests/performance/profiles/steady-10k.json @@ -0,0 +1,13 @@ +{ + "schema_version": 1, + "id": "steady-10k", + "seed": 186, + "object_count": 10000, + "namespace_count": 100, + "steady_events_per_second": 100, + "duration_seconds": 30, + "bursts": [ + {"start_second": 10, "duration_seconds": 4, "events_per_second": 500} + ], + "failures": [] +} diff --git a/tests/performance/replay.py b/tests/performance/replay.py new file mode 100644 index 00000000..78649639 --- /dev/null +++ b/tests/performance/replay.py @@ -0,0 +1,668 @@ +"""Real-app Textual replay harness for large-cluster qualification (issue #186). + +Drives production KorvidApp, WatchManager, ResourceStore, and ResourceTable +through a recorded WorkloadProfile and captures replay metrics. This module +is test-only instrumentation; it must not modify any production source file. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import os +import platform +import re +import subprocess +import sys +from collections.abc import AsyncIterator, Awaitable, Callable, Iterable, Mapping +from dataclasses import asdict, dataclass, field +from pathlib import Path +from time import monotonic +from types import MappingProxyType +from typing import Any, cast + +import psutil # type: ignore[import-untyped] # no inline stubs shipped +from textual import __version__ as _textual_version + +from korvid.core.config import KorvidConfig +from korvid.core.store import ALL_NAMESPACES, ResourceStore, Summary +from korvid.core.watch import WatchManager +from korvid.k8s.errors import ApiStatusError +from korvid.k8s.models import PodSummary +from korvid.k8s.telemetry import ReadTelemetryEvent +from korvid.ui.app import KorvidApp +from korvid.ui.messages import ResourcesUpdated +from korvid.ui.widgets.resource_table import ResourceTable +from tests.performance.metrics import ( + ApiSummary, + BenchmarkRecorder, + ChurnSummary, + LatencySummary, + NodePoolInfo, + PhaseSummary, + ProcessSampler, + ProcessSummary, + RunManifest, + ScenarioResult, +) +from tests.performance.profile import FailureInjection, WorkloadProfile, burst_end_offsets +from tests.performance.workload import ( + ScheduledEvent, + apply_events, + initial_pods, + scheduled_events, + summary_digest, +) +from tests.ui.waits import WaitTimeout, until + +#: A resolved korvid revision must be an immutable 40-character git object name. +#: Anything else (e.g. the historical literal ``dev``) is not traceable to a +#: commit and is rejected by `resolve_korvid_sha`. +_SHA_PATTERN = re.compile(r"^[0-9a-f]{40}$") + + +def _git_head() -> str | None: + """Resolve the current repository HEAD commit, or `None` if unavailable. + + Isolated behind a seam so `resolve_korvid_sha` stays deterministic under + test: production reads the real git tree, tests inject a fixed value. + """ + try: + result = subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=Path(__file__).resolve().parent, + capture_output=True, + text=True, + check=False, + ) + except OSError: + return None + if result.returncode != 0: + return None + return result.stdout.strip() + + +def resolve_korvid_sha( + *, + env: Mapping[str, str] | None = None, + git_head: Callable[[], str | None] | None = None, +) -> str | None: + """Resolve the immutable korvid commit under test from trusted sources. + + Prefers the CI-provided `GITHUB_SHA` (GitHub Actions sets it to the exact + commit being built), then falls back to the local repository HEAD. Returns + `None` when neither yields an immutable 40-hex object name, so callers can + decide how to react: offline reports mark it `unknown` (still traceable - + the report states the SHA could not be resolved), while a live evidence + run fails closed rather than publish an untraceable artifact. + """ + environ: Mapping[str, str] = os.environ if env is None else env + candidate = environ.get("GITHUB_SHA", "").strip().lower() + if _SHA_PATTERN.fullmatch(candidate): + return candidate + resolver = git_head if git_head is not None else _git_head + head = (resolver() or "").strip().lower() + if _SHA_PATTERN.fullmatch(head): + return head + return None + + +async def _sleep_default(delay: float) -> None: + """Thin wrapper around asyncio.sleep used as the default async sleeper.""" + await asyncio.sleep(delay) + + +class ReplayAborted(Exception): + """A replay ended before its schedule completed and can never complete. + + Raised for failure kinds the production `WatchManager` deliberately does + not retry (403 Forbidden is an authorization boundary, not a transient + fault): the stream is gone, the store has been cleared, and no further + event or render can arrive. Surfacing that immediately is strictly better + than letting the caller wait out a wall-clock timeout on a backlog that + will never drain. + """ + + +@dataclass(frozen=True) +class ReplayOptions: + """Tuning knobs for a replay run. + + Args: + time_scale: Multiplier applied to every scheduled-event sleep. + 0 skips all sleeps (fastest); 1.0 replays at real time. + sample_interval: Seconds between process-memory samples. + monotonic_fn: Monotonic clock callable injected for testing. + Production runs leave this `None` (uses `time.monotonic`). + async_sleep: Async sleep callable injected for testing. + Production runs leave this `None` (uses `_sleep_default`). + A virtual sleeper can advance a shared clock variable and do + `asyncio.sleep(0)` to yield without real wall time, making + timing-sensitive tests instant and mutation-deterministic. + """ + + time_scale: float = 1.0 + sample_interval: float = 1.0 + monotonic_fn: Callable[[], float] | None = field(default=None, hash=False, compare=False) + async_sleep: Callable[[float], Awaitable[None]] | None = field( + default=None, hash=False, compare=False + ) + + +@dataclass(frozen=True) +class ReplayReport: + """Metrics collected by a complete real-app replay run. + + Extends the fields of `BenchmarkReport` with replay-specific metadata + (`object_count`, `expected_digest`) so tests can assert both performance + counters and digest correctness in one object. + """ + + object_count: int + expected_digest: str + final_digest: str + dropped_updates: int + rendered_updates: int + render_passes: int + coalesced_updates: int + event_to_render: LatencySummary + input_latency: LatencySummary + #: Whether at least one churn event had actually been emitted (replay) or + #: dispatched (live) when input latency was first measured. + churn_started_before_input: bool + process: ProcessSummary + api: ApiSummary + #: Explicit phase measurements (startup, LIST-to-populated table, backlog + #: depth, post-burst drain) the numeric budgets are stated against. + phases: PhaseSummary + manifest: RunManifest + #: Requested-versus-achieved churn accounting; `None` for deterministic + #: replay, which drives its own source rather than a real API server. + churn: ChurnSummary | None = None + #: Per-kind count of injected failures actually exercised (report evidence + #: that a failure profile ran, including `metrics_unavailable`/`slow_logs`). + failures_injected: Mapping[str, int] = field(default_factory=lambda: MappingProxyType({})) + #: Scoped UI-at-scale scenarios driven during live churn (empty offline). + ui_scenarios: tuple[ScenarioResult, ...] = () + + +class MeasuredKorvidApp(KorvidApp): + """KorvidApp subclass that records the timing of *resource-update* renders. + + The hook is deliberately `on_resources_updated` rather than + `_render_table`: the latter is also the choke point for cursor, filter, + sort, namespace-switch, and split-pane repaints (~10 call sites in + `ui/app.py`). Counting those would inflate `render_passes` and - worse - + let an unrelated repaint flush the pending-event backlog, attributing a + watch event's latency to a keypress that happened to repaint first. + """ + + def __init__(self, *args: Any, recorder: BenchmarkRecorder, **kwargs: Any) -> None: + super().__init__(*args, **kwargs) + self._benchmark_recorder = recorder + + def on_resources_updated(self, message: ResourcesUpdated) -> None: + super().on_resources_updated(message) + self._benchmark_recorder.record_render(monotonic()) + + +# Maps FailureInjection.kind to the HTTP status code it raises. +_HARD_FAILURE_STATUS: dict[str, int] = { + "gone": 410, + "throttled": 429, + "forbidden": 403, +} + +#: Statuses `WatchManager._watch_loop` refuses to retry: the stream ends for +#: good and the store is cleared, so the replay is over the moment one is +#: injected (see `ReplayAborted`). +_TERMINAL_FAILURE_STATUS: frozenset[int] = frozenset({403}) + + +class _ReplaySource: + """Stateful async watch source that replays a `WorkloadProfile`. + + Each call (each watch connection) advances through the pre-computed event + stream. On reconnect the source re-LISTs from its tracked current state + so the `WatchManager`'s store-clear + reseed cycle produces the correct + snapshot. + """ + + def __init__( + self, + profile: WorkloadProfile, + events: tuple[ScheduledEvent, ...], + options: ReplayOptions, + recorder: BenchmarkRecorder, + churn_ready: asyncio.Event, + churn_start: asyncio.Event, + churn_done: asyncio.Event, + failures: dict[int, FailureInjection], + ) -> None: + self._profile = profile + self._events = events + self._options = options + self._recorder = recorder + self._churn_ready = churn_ready + self._churn_start = churn_start + self._churn_done = churn_done + self._failures = failures + self._generation = 0 + self._next_event_index = 0 + #: Index of the next burst whose end has not been marked yet. Persisted + #: across watch generations (see the WATCH phase below). + self._next_burst = 0 + #: Number of scheduled churn events actually yielded to the watch + #: manager so far; the real signal behind + #: `ReplayReport.churn_started_before_input`. + self.emitted_events = 0 + #: Set to the injected failure whose status the watch manager will not + #: retry, so `run_replay` can abort with a named cause. + self.terminal_failure: FailureInjection | None = None + self._replay_start: float = 0.0 + self._current: dict[str, PodSummary] = { + f"{p.namespace}/{p.name}": p for p in initial_pods(profile) + } + # Virtual-time seam: injected in tests; production uses real clock/sleep. + self._now: Callable[[], float] = ( + options.monotonic_fn if options.monotonic_fn is not None else monotonic + ) + self._sleep: Callable[[float], Awaitable[None]] = ( + options.async_sleep if options.async_sleep is not None else _sleep_default + ) + + async def _handle_failure_if_any(self, event: ScheduledEvent, index: int) -> None: + """Apply failure injection for *event*; raises `ApiStatusError` for hard faults. + + `slow` delays the tick without dropping the event. `metrics_unavailable` + and `slow_logs` record read telemetry on the metrics / log paths but + never disturb the resource watch: their whole point is proving resource + navigation and render progress are independent of the metrics poller and + of log consumption, so the resource event is still delivered on time. + Hard faults (gone, throttled, forbidden) record an error telemetry + entry, advance the next-event cursor, and raise so `WatchManager` + triggers a reconnect. + """ + failure = self._failures.get(event.sequence) + if failure is None: + return + self._recorder.record_failure(failure.kind) + if failure.kind == "slow": + tick = 1.0 / max(self._profile.steady_events_per_second, 1) + if tick * self._options.time_scale > 0: + await self._sleep(tick * self._options.time_scale) + return + if failure.kind == "metrics_unavailable": + # The metrics poller fails (503) while the resource watch continues: + # evidence lands on the metrics read path, not the pods path. + self._recorder.record_api( + ReadTelemetryEvent("error", "/apis/metrics.k8s.io/v1beta1/pods", status=503) + ) + return + if failure.kind == "slow_logs": + # A slow log stream: evidence lands on the log read path, and the + # resource event is delivered without any added delay. + self._recorder.record_api( + ReadTelemetryEvent("error", "/api/v1/namespaces/_/pods/_/log", status=504) + ) + return + status = _HARD_FAILURE_STATUS[failure.kind] + self._recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=status)) + self._next_event_index = index + 1 + if status in _TERMINAL_FAILURE_STATUS: + # The watch manager will not reconnect after this: record the cause + # now so `run_replay` aborts with it instead of waiting out a + # timeout on a backlog that can never drain. + self.terminal_failure = failure + self._churn_done.set() + raise ApiStatusError(status, failure.kind) + + async def __call__(self, kind: str, scope: str) -> AsyncIterator[tuple[str, Summary]]: + gen = self._generation + self._generation += 1 + + # --- LIST phase --- + if gen == 0: + list_pods: list[PodSummary] = list(initial_pods(self._profile)) + else: + list_pods = sorted(self._current.values(), key=lambda p: (p.namespace, p.name)) + + self._recorder.record_api( + ReadTelemetryEvent( + "list", + "/api/v1/pods", + object_count=len(list_pods), + decoded_bytes=len(list_pods) * 200, + ) + ) + # LIST rows populate the initial table; they are NOT event-to-render + # samples. Recording them here would let the initial LIST dominate the + # replay p95 and make it incomparable with the live path, which only + # records owned watch events. The LIST is timed separately below as the + # LIST-to-populated-table startup phase. + for pod in list_pods: + yield ("ADDED", pod) + + # The first WATCH open marks the LIST-to-populated boundary (via the + # recorder's telemetry hook), uniformly with the live path. + self._recorder.record_api(ReadTelemetryEvent("watch_open", "/api/v1/pods")) + + if gen == 0: + # Pause here until run_replay confirms the table is populated. + self._churn_ready.set() + await self._churn_start.wait() + # Record the virtual-clock instant when churn begins so that + # event.offset_seconds (absolute positions within the profile) + # can be converted to correct inter-event delays below. + self._replay_start = self._now() + + # Burst-end offsets (absolute seconds) mark the moment each burst's + # window closes, so post-burst backlog drain can be timed on the same + # real-clock axis the render pass records on. The cursor lives on the + # source, not this generation: a 410/429 reconnect resumes later in the + # schedule, and a per-generation cursor would re-mark every burst that + # already ended, producing duplicate and time-shifted drain samples. + burst_ends = burst_end_offsets(self._profile) + + # --- WATCH phase --- + for i in range(self._next_event_index, len(self._events)): + event = self._events[i] + elapsed = self._now() - self._replay_start + delay = event.offset_seconds * self._options.time_scale - elapsed + if delay > 0: + await self._sleep(delay) + + while ( + self._next_burst < len(burst_ends) + and event.offset_seconds >= burst_ends[self._next_burst] + ): + self._recorder.mark_burst_end(monotonic()) + self._next_burst += 1 + + await self._handle_failure_if_any(event, i) + + key = f"{event.summary.namespace}/{event.summary.name}" + if event.event_type == "DELETED": + self._current.pop(key, None) + else: + self._current[key] = event.summary + + self._recorder.record_event(event.sequence, monotonic()) + yield (event.event_type, event.summary) + self.emitted_events += 1 + self._next_event_index = i + 1 + + self._churn_done.set() + # Stay open like a real watch stream so WatchManager does not reconnect. + while True: + await asyncio.sleep(3600.0) + + +def build_manifest( + profile: WorkloadProfile, + *, + korvid_sha: str | None = None, + context: str | None = None, + cluster_id: str | None = None, + kubernetes_version: str | None = None, + node_pools: tuple[NodePoolInfo, ...] = (), +) -> RunManifest: + """Resolved run manifest for *profile* (profile hash plus environment). + + Public because both replay harnesses (`replay.py` and `live.py`) build the + identical manifest for their reports. + + Args: + korvid_sha: The immutable commit to record. When `None`, it is resolved + from `resolve_korvid_sha`; an offline run that cannot resolve one + records `unknown` (still traceable - the report states it is + unresolved) rather than a fake literal. A live evidence run resolves + and fails closed *before* calling this, so it never records + `unknown`. + context: Live kube context that was verified (live runs only). + cluster_id: Verified AKS ARM resource id (live runs only). + kubernetes_version: Kubernetes server version (live runs only). + node_pools: Node-pool topology metadata (live runs only). + """ + profile_hash = hashlib.sha256( + json.dumps(asdict(profile), sort_keys=True, separators=(",", ":"), default=str).encode() + ).hexdigest() + resolved_sha = korvid_sha if korvid_sha is not None else (resolve_korvid_sha() or "unknown") + return RunManifest( + profile_id=profile.id, + profile_hash=profile_hash, + korvid_sha=resolved_sha, + python=sys.version, + textual=_textual_version, + os=platform.platform(), + cpu_count=os.cpu_count() or 1, + memory_bytes=psutil.virtual_memory().total, + context=context, + cluster_id=cluster_id, + kubernetes_version=kubernetes_version, + node_pools=node_pools, + ) + + +def check_rendered_rows(table: Any, pods: Iterable[PodSummary]) -> None: + """Verify the rendered table against the store, independently of the widget. + + The published digest criterion compares a store digest with a store digest: + a table showing 1,000 stale rows satisfies it. This projects each owned Pod + onto the strings its row must display and checks them against the cells the + `DataTable` actually holds, so a cell the in-place diff skipped - it now + diffs against its own record of what it last wrote - is caught rather than + reported as a clean run. + + Deliberately column-order agnostic and written from the Pod summary rather + than by calling the widget's row builder: reusing the builder would only + prove the widget agrees with itself. + + Raises: + ValueError: a Pod is missing from the table, or a rendered row does not + carry every value the store says it must show. + """ + rendered = { + str(row.key.value): [str(cell) for cell in table.get_row(row.key)] + for row in table.ordered_rows + } + for pod in pods: + key = f"{pod.namespace}/{pod.name}" + cells = rendered.get(key) + if cells is None: + raise ValueError(f"rendered table is missing owned pod {key}") + expected = (pod.name, pod.ready, pod.phase, str(pod.restarts)) + missing = [value for value in expected if value not in cells] + if missing: + raise ValueError( + f"rendered row for {key} is stale: expected cells {missing} not among {cells}" + ) + + +async def wait_for( + pilot: Any, + condition: Callable[[], object], + *, + timeout: float, + label: str, + recorder: BenchmarkRecorder, +) -> None: + """`until`, but a timeout names the API errors that likely caused it. + + `KorvidApp.on_mount` replaces `WatchManager.on_error` with its own TUI + notification, so a 403/410/429 that killed the watch is otherwise invisible + to the harness: the operator would see only "not met within 60.0s" after + paying for a full cluster setup (or a full replay). The read telemetry has + already recorded those statuses, so they are appended to the message. + + Shared by both harnesses so a deterministic replay and a live run explain a + stalled wait the same way. + """ + try: + await until(pilot, condition, timeout=timeout, label=label) + except WaitTimeout as exc: + errors = recorder.api_errors() + if not errors: + raise + detail = ", ".join( + f"{event.operation} {event.path} status={event.status}" for event in errors + ) + raise WaitTimeout(f"{exc}; application read path reported API errors: {detail}") from exc + + +async def run_replay(profile: WorkloadProfile, options: ReplayOptions) -> ReplayReport: + """Run the full production Textual app against *profile* and return metrics. + + The function: + 1. Creates `ResourceStore`, `BenchmarkRecorder`, and one `_ReplaySource`. + 2. Emits initial Pods as ``ADDED`` (one logical LIST + WATCH OPEN). + 3. Waits on `churn_ready` after initial population. + 4. Waits until the `ResourceTable` shows all objects. + 5. Records key-press input latency. + 6. Signals the source to start scheduled-event churn. + 7. Awaits churn completion and the final render pass. + 8. Stops the watch manager and process sampler in `finally`. + 9. Compares the table/store digest to the source's expected state. + """ + events = scheduled_events(profile) + failures: dict[int, FailureInjection] = {f.at_event: f for f in profile.failures} + + store = ResourceStore() + recorder = BenchmarkRecorder() + sampler = ProcessSampler(options.sample_interval) + + churn_ready = asyncio.Event() + churn_start = asyncio.Event() + churn_done = asyncio.Event() + + source = _ReplaySource( + profile, + events, + options, + recorder, + churn_ready, + churn_start, + churn_done, + failures, + ) + watch_manager = WatchManager(store, source, retry_delay=0.0) + manifest = build_manifest(profile) + + app = MeasuredKorvidApp( + config=KorvidConfig(namespace=ALL_NAMESPACES), + store=store, + watch_manager=watch_manager, + recorder=recorder, + ) + + # Process/app start reference for the process-start-to-interactive phase + # and the steady-state RSS-slope warm-up boundary. + recorder.mark_process_start(monotonic()) + sampler.start() + churn_started_before_input = False + try: + async with app.run_test() as pilot: + table = app.query_one(ResourceTable) + + # Wait for the initial LIST to populate the table. + await wait_for( + pilot, + lambda: table.row_count == profile.object_count, + timeout=30.0, + label="initial pods rendered", + recorder=recorder, + ) + # The table is fully populated: mark the interactive boundary that + # closes both the startup and LIST-to-populated-table phases. + recorder.mark_interactive(monotonic()) + + # Release the source to emit scheduled events, then drive cursor + # input while churn is active (not before the source is unblocked). + churn_start.set() + + # Real ordering signal: wait for the source to actually put at + # least one churn event on the wire before measuring input latency, + # so the reported flag reflects observed emission rather than the + # fact that `churn_start.set()` was called on the previous line. + if events: + await wait_for( + pilot, + lambda: source.emitted_events > 0 or source.terminal_failure is not None, + timeout=30.0, + label="first churn event emitted", + recorder=recorder, + ) + churn_started_before_input = source.emitted_events > 0 + t0 = monotonic() + await pilot.press("down") + recorder.record_input(monotonic() - t0) + t0 = monotonic() + await pilot.press("up") + recorder.record_input(monotonic() - t0) + + # Wait for all events to be emitted and all renders to complete. + await wait_for( + pilot, + lambda: ( + source.terminal_failure is not None + or (churn_done.is_set() and recorder.pending_count() == 0) + ), + timeout=30.0, + label="churn complete and all events rendered", + recorder=recorder, + ) + if source.terminal_failure is None: + # The digests below are both computed from data, never from the + # rendering: a table full of stale cells would satisfy them. + # Checked here because the table only exists inside this block. + check_rendered_rows( + table, cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES)) + ) + finally: + process_samples = await sampler.stop() + await watch_manager.stop_all() + + if source.terminal_failure is not None: + failure = source.terminal_failure + status = _HARD_FAILURE_STATUS[failure.kind] + raise ReplayAborted( + f"replay aborted: injected {failure.kind!r} failure at event " + f"{failure.at_event} returned HTTP {status}, which the watch " + f"manager never retries; the stream ended and the store was cleared" + ) + + # Compute expected digest using the independent apply_events oracle. + # Hard failures (gone, throttled, forbidden) raise before yielding the + # event, so the corresponding watch events are never applied; filter them + # from the oracle to match the actual replay outcome. + hard_failure_seqs: frozenset[int] = frozenset( + f.at_event for f in profile.failures if f.kind in _HARD_FAILURE_STATUS + ) + oracle_events = tuple(e for e in events if e.sequence not in hard_failure_seqs) + expected_digest = summary_digest(apply_events(initial_pods(profile), oracle_events)) + + # Compute final digest from the store (actual state). + final_digest = summary_digest(cast(Iterable[PodSummary], store.get("pods", ALL_NAMESPACES))) + + benchmark = recorder.report( + manifest, process_samples, final_digest=final_digest, expected_digest=expected_digest + ) + + return ReplayReport( + object_count=profile.object_count, + expected_digest=expected_digest, + final_digest=final_digest, + dropped_updates=benchmark.dropped_updates, + rendered_updates=benchmark.rendered_updates, + render_passes=benchmark.render_passes, + coalesced_updates=benchmark.coalesced_updates, + event_to_render=benchmark.event_to_render, + input_latency=benchmark.input_latency, + churn_started_before_input=churn_started_before_input, + process=benchmark.process, + api=benchmark.api, + phases=benchmark.phases, + manifest=benchmark.manifest, + failures_injected=benchmark.failures_injected, + ) diff --git a/tests/performance/test_cli.py b/tests/performance/test_cli.py new file mode 100644 index 00000000..22d00a36 --- /dev/null +++ b/tests/performance/test_cli.py @@ -0,0 +1,998 @@ +"""CLI tests for the large-cluster benchmark tool (issue #186).""" + +from __future__ import annotations + +import json +from collections.abc import Awaitable, Callable +from dataclasses import replace +from pathlib import Path +from types import MappingProxyType + +import pytest +import yaml + +from korvid.k8s.errors import ApiStatusError +from tests.performance import cli +from tests.performance.metrics import ( + ApiSummary, + LatencySummary, + PhaseSummary, + ProcessSummary, + RunManifest, + ScenarioResult, +) +from tests.performance.profile import FailureInjection, WorkloadProfile +from tests.performance.replay import ReplayOptions, ReplayReport + +# --------------------------------------------------------------------------- +# Fixtures / helpers +# --------------------------------------------------------------------------- + + +def _make_manifest() -> RunManifest: + return RunManifest( + profile_id="test", + profile_hash="abc123", + korvid_sha="dev", + python="3.11", + textual="0.0.0", + os="test", + cpu_count=1, + memory_bytes=0, + ) + + +def _make_latency() -> LatencySummary: + return LatencySummary( + count=0, + p50_seconds=None, + p95_seconds=None, + p99_seconds=None, + maximum_seconds=None, + ) + + +def _make_process() -> ProcessSummary: + return ProcessSummary( + sample_count=0, + cpu_percent_max=None, + rss_bytes_max=None, + python_bytes_max=None, + rss_slope_mib_per_minute=None, + rss_slope_warmup_boundary_seconds=0.0, + rss_slope_sample_count=0, + ) + + +def _make_phases() -> PhaseSummary: + return PhaseSummary( + process_start_to_interactive_seconds=None, + list_to_populated_table_seconds=None, + max_backlog_depth=0, + post_burst_drain_seconds=(), + max_post_burst_drain_seconds=None, + ) + + +def _make_api() -> ApiSummary: + return ApiSummary( + operations=MappingProxyType({}), + paths=MappingProxyType({}), + decoded_bytes=0, + object_count=0, + watch_events=0, + reconnects=0, + relists=0, + throttles=0, + authorization_failures=0, + ) + + +def _make_minimal_profile() -> WorkloadProfile: + return WorkloadProfile( + schema_version=1, + id="smoke-mini", + seed=0, + object_count=4, + namespace_count=2, + steady_events_per_second=0, + duration_seconds=1, + bursts=(), + failures=(), + ) + + +def profile_path(tmp_path: Path) -> Path: + """Write a minimal valid profile to *tmp_path* and return its path.""" + path = tmp_path / "smoke-mini.json" + path.write_text( + json.dumps( + { + "schema_version": 1, + "id": "smoke-mini", + "seed": 0, + "object_count": 4, + "namespace_count": 2, + "steady_events_per_second": 0, + "duration_seconds": 1, + "bursts": [], + "failures": [], + } + ) + ) + return path + + +# --------------------------------------------------------------------------- +# Fake coroutines for monkeypatching +# --------------------------------------------------------------------------- + + +async def fake_run_replay( + profile: WorkloadProfile, + options: ReplayOptions, +) -> ReplayReport: + """Successful replay — digests match, no dropped updates.""" + return ReplayReport( + object_count=profile.object_count, + expected_digest="ok", + final_digest="ok", + dropped_updates=0, + rendered_updates=0, + render_passes=0, + coalesced_updates=0, + event_to_render=_make_latency(), + input_latency=_make_latency(), + churn_started_before_input=True, + process=_make_process(), + api=_make_api(), + phases=_make_phases(), + manifest=_make_manifest(), + ) + + +async def fake_failed_report( + profile: WorkloadProfile, + options: ReplayOptions, +) -> ReplayReport: + """Failed replay — digest mismatch simulates store corruption.""" + return ReplayReport( + object_count=profile.object_count, + expected_digest="expected-abc", + final_digest="actual-xyz", + dropped_updates=0, + rendered_updates=0, + render_passes=0, + coalesced_updates=0, + event_to_render=_make_latency(), + input_latency=_make_latency(), + churn_started_before_input=True, + process=_make_process(), + api=_make_api(), + phases=_make_phases(), + manifest=_make_manifest(), + ) + + +async def fake_dropped_report( + profile: WorkloadProfile, + options: ReplayOptions, +) -> ReplayReport: + """Failed replay — matching digests with one unrendered update.""" + return replace(await fake_run_replay(profile, options), dropped_updates=1) + + +async def fake_api_failure( + profile: WorkloadProfile, + options: ReplayOptions, +) -> ReplayReport: + """Expected replay failure surfaced by the Kubernetes boundary.""" + raise ApiStatusError(503, "unavailable") + + +async def fake_programmer_error( + profile: WorkloadProfile, + options: ReplayOptions, +) -> ReplayReport: + """Unexpected implementation error that must retain its traceback.""" + raise TypeError("unexpected replay defect") + + +# --------------------------------------------------------------------------- +# Fake coroutines for `replay-live` (mirrors `run_replay`'s keyword-only +# identity arguments). +# --------------------------------------------------------------------------- + + +def _make_recording_live_replay( + calls: list[dict[str, object]], +) -> Callable[..., Awaitable[ReplayReport]]: + """Build a fake `run_live_replay` that records every call's arguments into + *calls* (a fresh list per test, to stay independent of execution order).""" + + async def fake( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, + ) -> ReplayReport: + calls.append( + { + "profile": profile, + "options": options, + "context": context, + "expected_cluster_id": expected_cluster_id, + "run_id": run_id, + } + ) + return await fake_run_replay(profile, options) + + return fake + + +async def fake_live_failed_report( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, +) -> ReplayReport: + """Failed live replay — digest mismatch simulates store corruption.""" + return await fake_failed_report(profile, options) + + +async def fake_live_api_failure( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, +) -> ReplayReport: + """Expected live replay failure surfaced by a fail-closed gate.""" + raise ValueError("wrong active context: expected aks-context, got other-context") + + +async def fake_live_programmer_error( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, +) -> ReplayReport: + """Unexpected implementation error that must retain its traceback.""" + raise TypeError("unexpected live replay defect") + + +# --------------------------------------------------------------------------- +# Tests +# --------------------------------------------------------------------------- + + +def test_cli_writes_json_and_markdown(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + json_path = tmp_path / "result.json" + markdown_path = tmp_path / "result.md" + monkeypatch.setattr(cli, "run_replay", fake_run_replay) + result = cli.main( + [ + "replay", + "--profile", + str(profile_path(tmp_path)), + "--time-scale", + "0", + "--json", + str(json_path), + "--out", + str(markdown_path), + ] + ) + assert result == 0 + assert json.loads(json_path.read_text())["schema_version"] == 1 + assert json_path.read_text().index('"api"') < json_path.read_text().index('"schema_version"') + assert "# Large-cluster benchmark" in markdown_path.read_text() + + +@pytest.mark.parametrize("failed_replay", [fake_failed_report, fake_dropped_report]) +def test_cli_returns_nonzero_for_digest_or_drop_failure( + monkeypatch: pytest.MonkeyPatch, + failed_replay: object, +) -> None: + monkeypatch.setattr(cli, "load_profile", lambda _path: _make_minimal_profile()) + monkeypatch.setattr(cli, "run_replay", failed_replay) + assert cli.main(["replay", "--profile", "profile.json"]) == 1 + + +@pytest.mark.parametrize( + ("option", "value", "message"), + [ + ("--time-scale", "-1", "--time-scale must be non-negative"), + ("--sample-interval", "0", "--sample-interval must be positive"), + ], +) +def test_cli_rejects_invalid_timing_options( + option: str, + value: str, + message: str, + capsys: pytest.CaptureFixture[str], +) -> None: + assert cli.main(["replay", "--profile", "profile.json", option, value]) == 1 + assert message in capsys.readouterr().err + + +def test_cli_reports_expected_replay_errors( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + monkeypatch.setattr(cli, "load_profile", lambda _path: _make_minimal_profile()) + monkeypatch.setattr(cli, "run_replay", fake_api_failure) + assert cli.main(["replay", "--profile", "profile.json"]) == 1 + assert "error during replay: API 503: unavailable" in capsys.readouterr().err + + +def test_cli_reports_terminal_replay_abort_without_traceback( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + profile = replace( + _make_minimal_profile(), + steady_events_per_second=2, + failures=(FailureInjection(kind="forbidden", at_event=1),), + ) + monkeypatch.setattr(cli, "load_profile", lambda _path: profile) + + assert cli.main(["replay", "--profile", "profile.json", "--time-scale", "0"]) == 1 + stderr = capsys.readouterr().err + assert "error during replay: replay aborted:" in stderr + assert "Traceback" not in stderr + + +def test_cli_does_not_hide_unexpected_profile_errors(monkeypatch: pytest.MonkeyPatch) -> None: + def fail_profile(_path: Path) -> WorkloadProfile: + raise TypeError("unexpected profile defect") + + monkeypatch.setattr(cli, "load_profile", fail_profile) + with pytest.raises(TypeError, match="unexpected profile defect"): + cli.main(["replay", "--profile", "profile.json"]) + + +def test_cli_does_not_hide_unexpected_replay_errors(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(cli, "load_profile", lambda _path: _make_minimal_profile()) + monkeypatch.setattr(cli, "run_replay", fake_programmer_error) + with pytest.raises(TypeError, match="unexpected replay defect"): + cli.main(["replay", "--profile", "profile.json"]) + + +def test_cli_writes_seed_manifests_yaml(tmp_path: Path) -> None: + output_path = tmp_path / "seed.yaml" + + result = cli.main( + [ + "seed-manifests", + "--run-id", + "aks186", + "--namespace-count", + "2", + "--pods-per-namespace", + "2", + "--node-selector", + "korvid.dev/pool=perftest", + "--output", + str(output_path), + ] + ) + + assert result == 0 + documents = list(yaml.safe_load_all(output_path.read_text())) + assert [document["kind"] for document in documents] == [ + "Namespace", + "Namespace", + "Pod", + "Pod", + "Pod", + "Pod", + ] + assert documents[0]["metadata"]["name"] == "korvid-perf-aks186-0" + assert documents[2]["spec"]["nodeSelector"] == {"korvid.dev/pool": "perftest"} + + +@pytest.mark.parametrize( + ("arguments", "message"), + [ + ( + [ + "seed-manifests", + "--run-id", + "Bad", + "--namespace-count", + "1", + "--pods-per-namespace", + "1", + "--node-selector", + "korvid.dev/pool=perftest", + "--output", + "seed.yaml", + ], + "error building manifests: run_id must be 1-48 lowercase letters, digits, or hyphens", + ), + ( + [ + "seed-manifests", + "--run-id", + "aks186", + "--namespace-count", + "1", + "--pods-per-namespace", + "1", + "--node-selector", + "pool", + "--output", + "seed.yaml", + ], + "error building manifests: node_selector must be exactly one non-empty key=value pair", + ), + ], +) +def test_cli_seed_manifests_reports_invalid_inputs( + arguments: list[str], + message: str, + capsys: pytest.CaptureFixture[str], +) -> None: + assert cli.main(arguments) == 1 + assert message in capsys.readouterr().err + + +def test_cli_seed_manifests_reports_file_write_errors( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + def fail_write(self: Path, _text: str, *_args: object, **_kwargs: object) -> int: + raise OSError("disk full") + + monkeypatch.setattr(Path, "write_text", fail_write) + + assert ( + cli.main( + [ + "seed-manifests", + "--run-id", + "aks186", + "--namespace-count", + "1", + "--pods-per-namespace", + "1", + "--node-selector", + "korvid.dev/pool=perftest", + "--output", + "seed.yaml", + ] + ) + == 1 + ) + assert "error writing manifests: disk full" in capsys.readouterr().err + + +def test_cli_seed_manifests_does_not_hide_unexpected_programmer_errors( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def fail_build(*_args: object, **_kwargs: object) -> object: + raise TypeError("unexpected manifest defect") + + monkeypatch.setattr(cli, "build_seed_manifests", fail_build, raising=False) + + with pytest.raises(TypeError, match="unexpected manifest defect"): + cli.main( + [ + "seed-manifests", + "--run-id", + "aks186", + "--namespace-count", + "1", + "--pods-per-namespace", + "1", + "--node-selector", + "korvid.dev/pool=perftest", + "--output", + "seed.yaml", + ] + ) + + +# --------------------------------------------------------------------------- +# `replay-live` +# --------------------------------------------------------------------------- + +_LIVE_IDENTITY_ARGS = [ + "--context", + "aks-context", + "--expected-cluster-id", + "/subscriptions/sub/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters/aks", + "--run-id", + "aks186", +] + + +def _live_artifacts(tmp_path: Path, run_id: str = "aks186") -> list[str]: + """The four run-labelled live artifact destinations a successful live + qualification must write; every filename carries the run id.""" + return [ + "--json", + str(tmp_path / f"{run_id}-live.json"), + "--out", + str(tmp_path / f"{run_id}-live.md"), + "--cpu-profile", + str(tmp_path / f"{run_id}-live.pstats"), + "--allocation-snapshot", + str(tmp_path / f"{run_id}-live.alloc.txt"), + ] + + +@pytest.mark.parametrize("drop", ["--json", "--out", "--cpu-profile", "--allocation-snapshot"]) +def test_cli_replay_live_requires_all_four_artifacts( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], drop: str +) -> None: + """A successful live qualification must produce all four externally-retained + artifacts; dropping any one fails before the run is attempted.""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + artifacts = _live_artifacts(tmp_path) + index = artifacts.index(drop) + del artifacts[index : index + 2] + + exit_code = cli.main( + ["replay-live", "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, *artifacts] + ) + + assert exit_code == 1 + assert "all four artifacts" in capsys.readouterr().err + assert calls == [] + + +def test_cli_replay_live_rejects_artifact_not_named_for_the_run( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """Each live artifact filename must carry the run id, or it cannot be traced + back to the run that produced it.""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + artifacts = _live_artifacts(tmp_path) + artifacts[1] = str(tmp_path / "result.json") # no run id in the filename + + exit_code = cli.main( + ["replay-live", "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, *artifacts] + ) + + assert exit_code == 1 + assert "must include the run id" in capsys.readouterr().err + assert calls == [] + + +def test_cli_replay_live_writes_json_and_markdown( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + json_path = tmp_path / "aks186-live.json" + markdown_path = tmp_path / "aks186-live.md" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + result = cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + *_live_artifacts(tmp_path), + ] + ) + assert result == 0 + assert json.loads(json_path.read_text())["schema_version"] == 1 + assert "# Large-cluster benchmark" in markdown_path.read_text() + assert len(calls) == 1 + assert calls[0]["context"] == "aks-context" + assert calls[0]["run_id"] == "aks186" + assert calls[0]["expected_cluster_id"] == _LIVE_IDENTITY_ARGS[3] + # No --time-scale option: production real-time replay always uses 1.0. + assert calls[0]["options"].time_scale == 1.0 # type: ignore[attr-defined] + + +@pytest.mark.parametrize( + "missing_arguments", + [ + [], + ["--context", "aks-context"], + ["--expected-cluster-id", "/subscriptions/sub/x"], + ["--run-id", "aks186"], + ], +) +def test_cli_replay_live_requires_identity_arguments( + tmp_path: Path, missing_arguments: list[str] +) -> None: + """Every one of --context/--expected-cluster-id/--run-id is mandatory; + dropping any one of them (while supplying the others is exercised by the + full identity-args fixture in other tests) must fail argument parsing.""" + with pytest.raises(SystemExit): + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *missing_arguments, + ] + ) + + +def test_cli_replay_live_rejects_time_scale_option(tmp_path: Path) -> None: + """`replay-live` must never accept `--time-scale`: live churn always + replays at real wall-clock time (`ReplayOptions.time_scale == 1.0`).""" + with pytest.raises(SystemExit): + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + "--time-scale", + "0", + ] + ) + + +def test_cli_replay_live_rejects_non_positive_duration( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + assert ( + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + "--duration", + "0", + ] + ) + == 1 + ) + assert "--duration must be positive" in capsys.readouterr().err + + +def test_cli_replay_live_duration_overrides_profile_duration_only( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """`--duration` must override only `duration_seconds`, preserving the + profile's rate, bursts, seed, topology, and failures untouched.""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + result = cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + "--duration", + "5", + *_live_artifacts(tmp_path), + ] + ) + assert result == 0 + used_profile = calls[0]["profile"] + assert isinstance(used_profile, WorkloadProfile) + original = _make_minimal_profile() + assert used_profile.duration_seconds == 5 + assert used_profile.steady_events_per_second == original.steady_events_per_second + assert used_profile.bursts == original.bursts + assert used_profile.seed == original.seed + assert used_profile.object_count == original.object_count + assert used_profile.namespace_count == original.namespace_count + assert used_profile.failures == original.failures + + +def test_cli_replay_live_rejects_non_positive_sample_interval(tmp_path: Path) -> None: + assert ( + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + "--sample-interval", + "0", + ] + ) + == 1 + ) + + +@pytest.mark.parametrize( + "failed_replay", + [fake_live_failed_report], +) +def test_cli_replay_live_returns_nonzero_for_digest_failure( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + failed_replay: object, +) -> None: + monkeypatch.setattr(cli, "run_live_replay", failed_replay) + assert ( + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + *_live_artifacts(tmp_path), + ] + ) + == 1 + ) + + +def test_cli_replay_live_reports_expected_operational_errors( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + monkeypatch.setattr(cli, "run_live_replay", fake_live_api_failure) + assert ( + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + *_live_artifacts(tmp_path), + ] + ) + == 1 + ) + assert "error during replay: wrong active context" in capsys.readouterr().err + + +def test_cli_replay_live_does_not_hide_unexpected_programmer_errors( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(cli, "run_live_replay", fake_live_programmer_error) + with pytest.raises(TypeError, match="unexpected live replay defect"): + cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + *_live_artifacts(tmp_path), + ] + ) + + +def test_cli_replay_live_does_not_hide_unexpected_profile_errors( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def fail_profile(_path: Path) -> WorkloadProfile: + raise TypeError("unexpected profile defect") + + monkeypatch.setattr(cli, "load_profile", fail_profile) + with pytest.raises(TypeError, match="unexpected profile defect"): + cli.main( + [ + "replay-live", + "--profile", + "profile.json", + *_LIVE_IDENTITY_ARGS, + ] + ) + + +def test_replay_and_seed_manifests_commands_still_work( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Adding `replay-live` must not disturb the pre-existing subcommands.""" + monkeypatch.setattr(cli, "run_replay", fake_run_replay) + assert cli.main(["replay", "--profile", str(profile_path(tmp_path)), "--time-scale", "0"]) == 0 + + output_path = tmp_path / "seed.yaml" + assert ( + cli.main( + [ + "seed-manifests", + "--run-id", + "aks186", + "--namespace-count", + "1", + "--pods-per-namespace", + "1", + "--node-selector", + "korvid.dev/pool=perftest", + "--output", + str(output_path), + ] + ) + == 0 + ) + + +def test_cli_replay_live_rejects_duration_that_orphans_a_burst( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """`--duration` rewrites the profile with `dataclasses.replace`, skipping + `load_profile`'s burst containment check. A shortened duration that leaves a + burst hanging past the end of the run must be rejected with an explicit + operational message *before* any cluster identity/ownership work, instead of + tripping the generator's internal assertion mid-run (which printed an empty + "error during replay: ").""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + live_profile = Path("tests/performance/profiles/aks-1k.json") + + exit_code = cli.main( + [ + "replay-live", + "--profile", + str(live_profile), + *_LIVE_IDENTITY_ARGS, + "--duration", + "10", + ] + ) + + assert exit_code == 1 + assert "falls outside duration_seconds" in capsys.readouterr().err + assert calls == [] + + +def test_cli_replay_live_accepts_duration_that_still_contains_every_burst( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + + exit_code = cli.main( + [ + "replay-live", + "--profile", + "tests/performance/profiles/aks-1k.json", + *_LIVE_IDENTITY_ARGS, + "--duration", + "26", + *_live_artifacts(tmp_path), + ] + ) + + assert exit_code == 0 + used_profile = calls[0]["profile"] + assert isinstance(used_profile, WorkloadProfile) + assert used_profile.duration_seconds == 26 + + +@pytest.mark.parametrize("output_option", ["--out", "--json"]) +def test_cli_reports_output_write_errors_instead_of_raising( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + output_option: str, +) -> None: + """A bad `--out`/`--json` destination is an operational error, exactly like + `seed-manifests`' `--output`; it must return 1 with a message rather than + dumping a traceback after a completed (possibly very expensive) run.""" + + def fail_write(self: Path, _text: str, *_args: object, **_kwargs: object) -> int: + raise OSError("disk full") + + written_profile = profile_path(tmp_path) + monkeypatch.setattr(cli, "run_replay", fake_run_replay) + monkeypatch.setattr(Path, "write_text", fail_write) + + exit_code = cli.main( + [ + "replay", + "--profile", + str(written_profile), + "--time-scale", + "0", + output_option, + str(tmp_path / "report.out"), + ] + ) + + assert exit_code == 1 + assert "error writing report: disk full" in capsys.readouterr().err + + +def test_cli_replay_live_help_points_at_the_live_qualification_profile( + capsys: pytest.CaptureFixture[str], +) -> None: + """The published live plan (30 minutes at 20 events/s with three 30-second + 100 events/s bursts) lives in `aks-live-1k.json`; the command an operator + reaches for must name it, so the deterministic comparison profile is not + used by accident for a qualification run.""" + with pytest.raises(SystemExit): + cli.main(["replay-live", "--help"]) + + help_text = capsys.readouterr().out + assert "tests/performance/profiles/aks-live-1k.json" in help_text + assert Path("tests/performance/profiles/aks-live-1k.json").exists() + + +async def fake_live_failed_ui_scenario( + profile: WorkloadProfile, + options: ReplayOptions, + *, + context: str, + expected_cluster_id: str, + run_id: str, +) -> ReplayReport: + """Live replay whose churn and digests are clean but whose UI-at-scale + evidence is not: the split-pane scenario did not reach its target state.""" + report = await fake_run_replay(profile, options) + return replace( + report, + ui_scenarios=( + ScenarioResult(name="filter", latency_seconds=0.4, ok=True), + ScenarioResult(name="split_pane", latency_seconds=9.9, ok=False), + ), + ) + + +def test_cli_replay_live_fails_when_a_ui_scenario_did_not_pass( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """`drive_ui_scenarios` records a failed key sequence as `ok=False` instead + of raising, so the live command must fold those outcomes into its exit + status — otherwise a qualification "passes" with no UI-at-scale evidence.""" + monkeypatch.setattr(cli, "run_live_replay", fake_live_failed_ui_scenario) + + exit_code = cli.main( + [ + "replay-live", + "--profile", + str(profile_path(tmp_path)), + *_LIVE_IDENTITY_ARGS, + *_live_artifacts(tmp_path), + ] + ) + + assert exit_code == 1 + assert "split_pane" in capsys.readouterr().err + + +def test_cli_replay_live_rejects_artifact_paths_that_alias_one_file( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """Distinctness must be decided on resolved paths: `sub/../aks186-live.json` + and `aks186-live.json` are different strings but the same file, so the + second artifact write would silently destroy the first.""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + (tmp_path / "sub").mkdir() + artifacts = _live_artifacts(tmp_path) + artifacts[3] = str(tmp_path / "sub" / ".." / "aks186-live.json") + + exit_code = cli.main( + ["replay-live", "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, *artifacts] + ) + + assert exit_code == 1 + assert "distinct destinations" in capsys.readouterr().err + assert calls == [] + + +def test_cli_replay_live_reports_an_unwritable_allocation_snapshot( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """The snapshot is flushed from a `finally`, so an unwritable destination + would raise straight out of the command - producing a traceback after a + 30-minute run and masking whatever actually failed inside the run.""" + calls: list[dict[str, object]] = [] + monkeypatch.setattr(cli, "run_live_replay", _make_recording_live_replay(calls)) + artifacts = _live_artifacts(tmp_path) + unwritable = tmp_path / "missing-dir" / "aks186-live.alloc.txt" + artifacts[artifacts.index("--allocation-snapshot") + 1] = str(unwritable) + + exit_code = cli.main( + ["replay-live", "--profile", str(profile_path(tmp_path)), *_LIVE_IDENTITY_ARGS, *artifacts] + ) + + assert exit_code == 1 + assert "allocation snapshot" in capsys.readouterr().err diff --git a/tests/performance/test_live.py b/tests/performance/test_live.py new file mode 100644 index 00000000..7da49787 --- /dev/null +++ b/tests/performance/test_live.py @@ -0,0 +1,2728 @@ +"""Guarded real-AKS application-path replay prerequisite tests (issue #186 task 8.2). + +Every test substitutes the identity/kubeconfig/subprocess/KubeClient seams +(`LiveDependencies`) with fakes: no test here may contact Azure, a real +kubeconfig, or a real cluster (see `live.py`'s module docstring). +""" + +from __future__ import annotations + +import asyncio +import dataclasses +import json +import re +from collections import Counter +from collections.abc import AsyncGenerator, AsyncIterator, Awaitable, Callable +from types import SimpleNamespace +from typing import Any +from unittest.mock import AsyncMock, MagicMock + +import pytest +from kubernetes_asyncio import client as k8s_client +from kubernetes_asyncio import config as k8s_config +from multidict import CIMultiDict, CIMultiDictProxy + +from korvid.core.store import ALL_NAMESPACES, Summary +from korvid.k8s.discovery import ResourceMeta +from korvid.k8s.errors import ApiStatusError +from korvid.k8s.logs import LogLine +from korvid.k8s.models import GenericSummary, PodSummary +from korvid.k8s.telemetry import ReadTelemetry, ReadTelemetryEvent +from tests.performance import live, manifests +from tests.performance import replay as replay_mod +from tests.performance.live import ( + ChurnProgress, + CommandResult, + LiveDependencies, + LiveLimits, + build_guarded_label_patch, + drive_live_churn, + live_object_identity, + make_live_watch_source, + read_and_validate_owned_pods, + run_live_replay, +) +from tests.performance.manifests import build_seed_manifests +from tests.performance.metrics import BenchmarkRecorder +from tests.performance.profile import Burst, WorkloadProfile +from tests.performance.replay import ReplayOptions +from tests.performance.workload import scheduled_events, summary_digest +from tests.ui.waits import WaitTimeout + +RUN_ID = "aks186" +CONTEXT = "aks-korvid-contract-test" +SUBSCRIPTION = "00000000-0000-0000-0000-000000000000" +RESOURCE_GROUP = "rg-korvid-contract-test" +CLUSTER_NAME = "aks-korvid-contract-test" +CLUSTER_ID = ( + f"/subscriptions/{SUBSCRIPTION}/resourceGroups/{RESOURCE_GROUP}" + f"/providers/Microsoft.ContainerService/managedClusters/{CLUSTER_NAME}" +) +FQDN = "aks-korvid-contract-test-dns-abc123.hcp.eastus.azmk8s.io" +REQUIRED_TAGS = {"purpose": "korvid-contract-testing", "production-use": "prohibited"} + +# --------------------------------------------------------------------------- +# Fixtures / fakes +# --------------------------------------------------------------------------- + + +def _labels(run_id: str) -> tuple[tuple[str, str], ...]: + return ( + (manifests.MANAGED_BY_LABEL, manifests.MANAGED_BY_VALUE), + (manifests.RUN_LABEL, run_id), + ) + + +def _build_fake_topology( + run_id: str, namespace_count: int, object_count: int +) -> tuple[dict[str, GenericSummary], dict[tuple[str, str], PodSummary]]: + namespaces = { + manifests.namespace_name(run_id, i): GenericSummary( + name=manifests.namespace_name(run_id, i), + namespace="", + kind="Namespace", + created="2024-01-01T00:00:00Z", + uid=f"ns-uid-{i}", + labels=_labels(run_id), + ) + for i in range(namespace_count) + } + pods: dict[tuple[str, str], PodSummary] = {} + for index in range(object_count): + namespace, name = live_object_identity(run_id, namespace_count, index) + pods[(namespace, name)] = PodSummary( + name=name, + namespace=namespace, + phase="Running", + ready="1/1", + restarts=0, + node="node-0", + uid=f"pod-uid-{index}", + labels=_labels(run_id), + ) + return namespaces, pods + + +class _FakeKubeClient: + """Fake `KubeReadClient`: an in-memory cluster the fake mutation client + mutates directly, so the watch stream really observes guarded patches - + exactly like a real API server + watch, with zero network I/O. + + Mirrors the real `KubeClient`'s telemetry behavior exactly: `list_objects` + and `list_pods` (used only by the harness's ownership/final reads) emit a + "list" event, and `watch_pods` (the real application read path) emits + "list" then "watch_open"/"watch_event" - all conditional on + `read_telemetry` being wired, exactly like `KubeClient._observe_read` + being a no-op when `read_telemetry is None`. Two `_FakeKubeClient` + instances constructed over the *same* `namespaces`/`pods` dict objects + model two independent connections to one shared cluster, exactly like + `run_live_replay`'s real harness/app-path `KubeClient` pair. + """ + + def __init__( + self, + namespaces: dict[str, GenericSummary], + pods: dict[tuple[str, str], PodSummary], + *, + distractor_pods: tuple[PodSummary, ...] = (), + ) -> None: + self.read_telemetry: ReadTelemetry | None = None + self.namespaces = namespaces + self.pods = pods + self.distractor_pods = distractor_pods + self.connect_context: str | None = "__not_connected__" + self.closed = False + self.events: asyncio.Queue[tuple[str, PodSummary]] = asyncio.Queue() + #: Per-namespace `list_pods` call log; asserts the ownership gate's + #: validated snapshot is reused rather than immediately re-listed. + self.list_pods_calls: list[str] = [] + self.list_objects_calls = 0 + self.watch_pods_calls = 0 + #: Namespace argument of each watch, in order. + self.watch_namespaces: list[str | None] = [] + #: True once the watch generator has been closed (the harness stopped + #: the WatchManager); lets tests assert *when* a read happened + #: relative to the application watch still being live. + self.watch_finished = False + #: Optional per-call spy invoked with the namespace being listed. + self.on_list_pods: Callable[[str], None] | None = None + #: Optional watch-open failure (e.g. a 403 the real client would + #: record as read telemetry before raising). + self.watch_error: ApiStatusError | None = None + #: Read-only provider call logs for the UI-at-scale scenarios. + self.get_object_calls: list[tuple[str | None, str]] = [] + self.stream_logs_calls = 0 + + async def connect(self, context: str | None = None) -> None: + self.connect_context = context + + async def close(self) -> None: + self.closed = True + + async def get_object( + self, meta: ResourceMeta, namespace: str | None, name: str + ) -> dict[str, Any]: + """Read-only manifest fetch backing the `describe` UI-at-scale scenario.""" + self.get_object_calls.append((namespace, name)) + return { + "apiVersion": "v1", + "kind": "Pod", + "metadata": {"name": name, "namespace": namespace}, + "status": {"phase": "Running"}, + } + + async def stream_logs(self, *args: Any, **kwargs: Any) -> AsyncIterator[Any]: + """Read-only log stream backing the `multi_log` UI-at-scale scenario. + + Yields one line and then stays open, exactly like a real follow stream + the scenario dismisses with `escape`. + """ + self.stream_logs_calls += 1 + yield LogLine(pod="bench", container="app", text="benchmark log line") + await asyncio.Event().wait() + + async def list_objects(self, meta: ResourceMeta, namespace: str | None) -> list[GenericSummary]: + assert meta.kind == "Namespace" + assert namespace is None + self.list_objects_calls += 1 + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("list", "/api/v1/namespaces")) + return list(self.namespaces.values()) + + async def list_pods(self, namespace: str) -> list[PodSummary]: + self.list_pods_calls.append(namespace) + if self.on_list_pods is not None: + self.on_list_pods(namespace) + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("list", f"/api/v1/namespaces/{namespace}/pods")) + return [pod for (ns, _name), pod in self.pods.items() if ns == namespace] + + def _initial_watch_pods(self, namespace: str | None) -> list[PodSummary]: + """Pods the initial LIST of a watch returns: everything for the + all-namespaces watch (plus the unowned distractors a real cluster-wide + LIST would also return), only the namespace's own Pods when scoped.""" + if namespace is not None: + return [pod for pod in self.pods.values() if pod.namespace == namespace] + return [*self.pods.values(), *self.distractor_pods] + + async def watch_pods(self, namespace: str | None) -> AsyncIterator[tuple[str, PodSummary]]: + # `namespace is None` is the all-namespaces watch the measured window + # runs on; a concrete namespace is the scoped watch the + # `namespace_switch` UI-at-scale scenario really triggers. + self.watch_pods_calls += 1 + self.watch_namespaces.append(namespace) + # Tracks the *currently open* watch: a scope change closes one + # generator and opens another, so "some generator finished" would not + # mean the application watch has stopped. + self.watch_finished = False + try: + if self.watch_error is not None: + if self.read_telemetry is not None: + self.read_telemetry( + ReadTelemetryEvent("error", "/api/v1/pods", status=self.watch_error.status) + ) + raise self.watch_error + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("list", "/api/v1/pods")) + for pod in self._initial_watch_pods(namespace): + yield ("ADDED", pod) + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("watch_open", "/api/v1/pods")) + while True: + event = await self.events.get() + if self.read_telemetry is not None: + self.read_telemetry(ReadTelemetryEvent("watch_event", "/api/v1/pods")) + yield event + finally: + self.watch_finished = True + + +class _FakeMutationClient: + """Fake `MutationClient`: applies the guard checks a real JSON-Patch + `test` op would enforce, then writes the dedicated tick label on the + shared fake cluster and wakes the fake watch - so a guard failure here is + exactly as fatal as a real 422 from the API server.""" + + def __init__( + self, + kube: _FakeKubeClient | Callable[[], _FakeKubeClient], + run_id: str, + ) -> None: + # `run_live_replay` constructs (and eagerly connects) the mutation + # client *before* the application-path watch client, so tests that + # need the watch client resolve it lazily via a callable. + self._resolve_kube: Callable[[], _FakeKubeClient] = ( + kube if callable(kube) else (lambda: kube) + ) + self._run_id = run_id + self.calls: list[tuple[str, str, str]] = [] + self.closed = False + self.connect_calls = 0 + self.in_flight = 0 + self.max_in_flight = 0 + + async def connect(self) -> None: + self.connect_calls += 1 + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.calls.append((namespace, name, tick)) + self.in_flight += 1 + self.max_in_flight = max(self.max_in_flight, self.in_flight) + try: + await self._apply(namespace, name, uid=uid, tick=tick) + finally: + self.in_flight -= 1 + + async def _apply(self, namespace: str, name: str, *, uid: str, tick: str) -> None: + kube = self._resolve_kube() + current = kube.pods.get((namespace, name)) + labels = dict(current.labels) if current is not None else {} + guard_ok = ( + current is not None + and current.uid == uid + and labels.get(manifests.MANAGED_BY_LABEL) == manifests.MANAGED_BY_VALUE + and labels.get(manifests.RUN_LABEL) == self._run_id + ) + if not guard_ok or current is None: + raise ApiStatusError(422, "test operation failed for guarded patch") + labels[manifests.TICK_LABEL] = tick + updated = dataclasses.replace(current, labels=tuple(sorted(labels.items()))) + kube.pods[(namespace, name)] = updated + kube.events.put_nowait(("MODIFIED", updated)) + + async def close(self) -> None: + self.closed = True + + +async def _context_host_ok(_context: str) -> str: + return FQDN + + +def _report_as_benchmark(report: Any) -> Any: + from tests.performance.cli import _to_benchmark_report + + return _to_benchmark_report(report) + + +def _never_called(label: str) -> Callable[..., Any]: + def _fail(*_args: object, **_kwargs: object) -> Any: + raise AssertionError(f"{label} must not be called") + + return _fail + + +def _ok_command_runner( + *, + cluster_id: str = CLUSTER_ID, + fqdn: str = FQDN, + private_fqdn: str = "", + resource_group: str = RESOURCE_GROUP, + name: str = CLUSTER_NAME, + tags: dict[str, str] | None = None, + kubernetes_version: str = "1.30.4", + agent_pool_profiles: list[dict[str, Any]] | None = None, +) -> Callable[[Any], Awaitable[CommandResult]]: + async def _run(_args: Any) -> CommandResult: + payload = { + "id": cluster_id, + "fqdn": fqdn, + "privateFqdn": private_fqdn, + "resourceGroup": resource_group, + "name": name, + "tags": REQUIRED_TAGS if tags is None else tags, + "currentKubernetesVersion": kubernetes_version, + "agentPoolProfiles": ( + [ + {"name": "perftest", "count": 5, "currentOrchestratorVersion": "1.30.4"}, + {"name": "system", "count": 1, "currentOrchestratorVersion": "1.30.4"}, + ] + if agent_pool_profiles is None + else agent_pool_profiles + ), + } + return CommandResult(0, json.dumps(payload), "") + + return _run + + +_FIXED_SHA = "1234567890abcdef1234567890abcdef12345678" + + +def _happy_deps( + namespaces: dict[str, GenericSummary], + pods: dict[tuple[str, str], PodSummary], + run_id: str, + *, + distractor_pods: tuple[PodSummary, ...] = (), + mutation_clients: list[_FakeMutationClient] | None = None, + mutation_client_factory: Callable[[str], Any] | None = None, + harness_clients: list[_FakeKubeClient] | None = None, + app_clients: list[_FakeKubeClient] | None = None, +) -> LiveDependencies: + """Build a `LiveDependencies` whose harness and application-path + `KubeReadClient`s are two *separate* `_FakeKubeClient` instances sharing + the same underlying `namespaces`/`pods` dicts by reference (so a guarded + mutation issued against the app-path instance is immediately visible to + the harness's later independent re-read) - exactly mirroring + `run_live_replay`'s real harness/app-path `KubeClient` pair. Pass + `harness_clients`/`app_clients` to capture the constructed instances for + assertions (telemetry exclusion, call counts, connect/close).""" + + async def context_host(context: str) -> str: + assert context == CONTEXT + return FQDN + + # The most recently constructed app-path client: `default_mutation_factory` + # is only ever invoked (by `run_live_replay`) after `kube_client_factory`, + # so this always resolves to the live watch's own instance/queue. + app_holder: list[_FakeKubeClient] = [] + + def default_mutation_factory(run_id_arg: str) -> _FakeMutationClient: + client = _FakeMutationClient(lambda: app_holder[-1], run_id_arg) + if mutation_clients is not None: + mutation_clients.append(client) + return client + + def kube_factory(read_telemetry: ReadTelemetry) -> _FakeKubeClient: + client = _FakeKubeClient(namespaces, pods, distractor_pods=distractor_pods) + client.read_telemetry = read_telemetry + app_holder.append(client) + if app_clients is not None: + app_clients.append(client) + return client + + def harness_factory() -> _FakeKubeClient: + client = _FakeKubeClient(namespaces, pods, distractor_pods=distractor_pods) + if harness_clients is not None: + harness_clients.append(client) + return client + + return LiveDependencies( + command_runner=_ok_command_runner(), + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=kube_factory, + harness_kube_client_factory=harness_factory, + mutation_client_factory=mutation_client_factory or default_mutation_factory, + resolve_sha=lambda: _FIXED_SHA, + ) + + +def _virtual_clock() -> tuple[Callable[[], float], Callable[[float], Awaitable[None]]]: + virtual_time = [0.0] + + def monotonic_fn() -> float: + return virtual_time[0] + + async def async_sleep(delay: float) -> None: + virtual_time[0] += delay + await asyncio.sleep(0) + + return monotonic_fn, async_sleep + + +def _tiny_live_profile(*, seed: int = 1) -> WorkloadProfile: + """A profile satisfying the mandatory 1,000/20 live topology with a + minimal churn schedule (2 events), kept fast via the virtual clock.""" + return WorkloadProfile( + schema_version=1, + id="live-test", + seed=seed, + object_count=1000, + namespace_count=20, + steady_events_per_second=2, + duration_seconds=1, + bursts=(), + failures=(), + ) + + +# --------------------------------------------------------------------------- +# Deterministic mapping +# --------------------------------------------------------------------------- + + +def test_live_object_identity_matches_seed_manifest_layout() -> None: + run_id = "run1" + namespace_count = 4 + pods_per_namespace = 3 + manifest_docs = build_seed_manifests( + run_id=run_id, + namespace_count=namespace_count, + pods_per_namespace=pods_per_namespace, + node_selector="korvid.dev/pool=perftest", + ) + pod_docs = [doc for doc in manifest_docs if doc["kind"] == "Pod"] + expected_by_index = {} + for index, doc in enumerate(pod_docs): + metadata = doc["metadata"] + assert isinstance(metadata, dict) + expected_by_index[index] = (metadata["namespace"], metadata["name"]) + + for index in range(namespace_count * pods_per_namespace): + assert live_object_identity(run_id, namespace_count, index) == expected_by_index[index] + + +def test_live_object_identity_examples() -> None: + assert live_object_identity("aks186", 20, 0) == ("korvid-perf-aks186-0", "bench-0") + assert live_object_identity("aks186", 20, 19) == ("korvid-perf-aks186-19", "bench-0") + assert live_object_identity("aks186", 20, 20) == ("korvid-perf-aks186-0", "bench-1") + assert live_object_identity("aks186", 20, 999) == ("korvid-perf-aks186-19", "bench-49") + + +# --------------------------------------------------------------------------- +# Guarded patch construction +# --------------------------------------------------------------------------- + + +def test_build_guarded_label_patch_tests_uid_and_both_ownership_labels() -> None: + ops = build_guarded_label_patch(uid="uid-1", run_id="run1", tick="7") + assert ops == [ + {"op": "test", "path": "/metadata/uid", "value": "uid-1"}, + { + "op": "test", + "path": "/metadata/labels/app.kubernetes.io~1managed-by", + "value": "korvid-performance", + }, + { + "op": "test", + "path": "/metadata/labels/korvid.dev~1performance-run", + "value": "run1", + }, + {"op": "add", "path": "/metadata/labels/korvid.dev~1performance-tick", "value": "7"}, + ] + + +def test_build_guarded_label_patch_never_writes_status_spec_or_ownership() -> None: + """Churn must stay metadata-only on a *non-ownership* label: `status` is + kubelet-owned (a patched `status.phase` is reverted on the next node sync, + breaking digest parity) and the design doc requires metadata-only updates. + The two ownership labels are only ever `test` operands, never written.""" + ops = build_guarded_label_patch(uid="uid-1", run_id="run1", tick="7") + writes = [op for op in ops if op["op"] != "test"] + + assert len(writes) == 1 + assert writes[0]["path"] == "/metadata/labels/korvid.dev~1performance-tick" + assert not any(op["path"].startswith("/status") for op in ops) + assert not any(op["path"].startswith("/spec") for op in ops) + ownership_paths = { + "/metadata/labels/app.kubernetes.io~1managed-by", + "/metadata/labels/korvid.dev~1performance-run", + } + assert all(op["op"] == "test" for op in ops if op["path"] in ownership_paths) + assert manifests.TICK_LABEL not in {manifests.MANAGED_BY_LABEL, manifests.RUN_LABEL} + + +# --------------------------------------------------------------------------- +# Guarded churn +# --------------------------------------------------------------------------- + + +async def test_drive_live_churn_sends_guarded_patches_for_every_event() -> None: + run_id = "run1" + namespace_count = 2 + _, pods = _build_fake_topology(run_id, namespace_count, 4) + kube = _FakeKubeClient({}, pods) + mutation_client = _FakeMutationClient(kube, run_id) + profile = WorkloadProfile( + schema_version=1, + id="churn-test", + seed=7, + object_count=4, + namespace_count=namespace_count, + steady_events_per_second=4, + duration_seconds=1, + bursts=(), + failures=(), + ) + events = scheduled_events(profile) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + recorder = BenchmarkRecorder() + progress = ChurnProgress(requested_events=len(events)) + live_state = dict(pods) + await drive_live_churn( + events, + run_id=run_id, + namespace_count=namespace_count, + live_state=live_state, + mutation_client=mutation_client, + options=options, + progress=progress, + limits=LiveLimits(churn_concurrency=1), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), + ) + assert len(mutation_client.calls) == len(events) + for call, event in zip(mutation_client.calls, events, strict=True): + namespace, name, tick = call + expected_namespace, expected_name = live_object_identity( + run_id, namespace_count, event.object_index + ) + assert (namespace, name) == (expected_namespace, expected_name) + assert tick == str(event.sequence) + assert progress.completed == len(events) + # Event timing is recorded at watch receipt, never by the write path. + assert recorder.pending_count() == 0 + + +async def test_drive_live_churn_aborts_on_guard_failure_and_never_continues() -> None: + run_id = "run1" + namespace_count = 2 + _, pods = _build_fake_topology(run_id, namespace_count, 4) + kube = _FakeKubeClient({}, pods) + mutation_client = _FakeMutationClient(kube, run_id) + profile = WorkloadProfile( + schema_version=1, + id="churn-abort-test", + seed=7, + object_count=4, + namespace_count=namespace_count, + steady_events_per_second=4, + duration_seconds=1, + bursts=(), + failures=(), + ) + events = scheduled_events(profile) + assert len(events) >= 2 + + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + # Snapshot the driver's cached state *before* the target pod is replaced + # (a fresh uid) - exactly like `run_live_replay` caching uids once during + # the ownership gate, then churning without re-reading them each time. + live_state = dict(pods) + + # Simulate the target pod being replaced right before the second + # scheduled mutation - a real API server would fail the JSON-Patch + # `test` op the same way. + second_index = events[1].object_index + second_key = live_object_identity(run_id, namespace_count, second_index) + kube.pods[second_key] = dataclasses.replace(kube.pods[second_key], uid="replaced-uid") + + with pytest.raises(ApiStatusError, match="test operation failed"): + await drive_live_churn( + events, + run_id=run_id, + namespace_count=namespace_count, + live_state=live_state, + mutation_client=mutation_client, + options=options, + progress=ChurnProgress(requested_events=len(events)), + limits=LiveLimits(churn_concurrency=1), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), + ) + # Aborted at the 2nd call; a 3rd event must never have been attempted. + assert len(mutation_client.calls) == 2 + + +# --------------------------------------------------------------------------- +# Namespace filtering +# --------------------------------------------------------------------------- + + +async def test_make_live_watch_source_filters_to_expected_namespaces() -> None: + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + distractor = PodSummary( + name="stray", + namespace="default", + phase="Running", + ready="1/1", + restarts=0, + node="node-x", + ) + kube = _FakeKubeClient({}, pods, distractor_pods=(distractor,)) + expected_namespaces = frozenset(namespace for namespace, _name in pods) + source = make_live_watch_source( + kube, expected_namespaces, run_id=run_id, recorder=BenchmarkRecorder() + ) + + seen: list[tuple[str, Summary]] = [] + agen = source("pods", "*") + try: + for _ in range(len(pods)): + seen.append(await agen.__anext__()) + # The distractor pod, and only it, must never surface. + with pytest.raises(asyncio.TimeoutError): + await asyncio.wait_for(agen.__anext__(), timeout=0.05) + finally: + assert isinstance(agen, AsyncGenerator) + await agen.aclose() + + assert {pod.namespace for _event, pod in seen} == expected_namespaces + assert all(pod.name != "stray" for _event, pod in seen) + + +async def test_make_live_watch_source_rejects_non_pod_kind() -> None: + kube = _FakeKubeClient({}, {}) + source = make_live_watch_source(kube, frozenset(), run_id="run1", recorder=BenchmarkRecorder()) + with pytest.raises(ValueError, match="only watches pods"): + await source("deployments", "*").__anext__() + + +# --------------------------------------------------------------------------- +# Gates: time scale / run_id / topology / identity - all reject before +# any client is constructed. +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_rejects_time_scale_other_than_one() -> None: + deps = LiveDependencies( + command_runner=_never_called("command_runner"), + active_context=_never_called("active_context"), + context_host=_never_called("context_host"), + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match=re.escape("time_scale == 1.0")): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_invalid_run_id() -> None: + deps = LiveDependencies( + command_runner=_never_called("command_runner"), + active_context=_never_called("active_context"), + context_host=_never_called("context_host"), + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="run_id must be 1-48"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id="Bad Run", + deps=deps, + ) + + +@pytest.mark.parametrize( + ("object_count", "namespace_count"), + [(999, 20), (1000, 19), (1001, 20), (100, 7)], +) +async def test_run_live_replay_rejects_topology_mismatch_before_identity_gate( + object_count: int, namespace_count: int +) -> None: + profile = WorkloadProfile( + schema_version=1, + id="bad-topology", + seed=1, + object_count=object_count, + namespace_count=namespace_count, + steady_events_per_second=1, + duration_seconds=1, + bursts=(), + failures=(), + ) + deps = LiveDependencies( + command_runner=_never_called("command_runner"), + active_context=_never_called("active_context"), + context_host=_never_called("context_host"), + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises( + ValueError, match=re.escape("object_count") + "|" + re.escape("namespace_count") + ): + await run_live_replay( + profile, + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_wrong_active_context_before_mutation() -> None: + deps = LiveDependencies( + command_runner=_never_called("command_runner"), + active_context=lambda: "some-other-context", + context_host=_never_called("context_host"), + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="active kubeconfig context"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_wrong_aks_resource_id_before_mutation() -> None: + async def context_host(_context: str) -> str: + return FQDN + + deps = LiveDependencies( + command_runner=_ok_command_runner(cluster_id="/subscriptions/x/wrong-cluster"), + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="expected"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_wrong_api_hostname_before_mutation() -> None: + async def context_host(_context: str) -> str: + return "not-the-real-host.example.com" + + deps = LiveDependencies( + command_runner=_ok_command_runner(), + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="does not match"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_malformed_identity_command_output() -> None: + async def command_runner(_args: Any) -> CommandResult: + return CommandResult(0, "not-json{{{", "") + + async def context_host(_context: str) -> str: + return FQDN + + deps = LiveDependencies( + command_runner=command_runner, + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="malformed JSON"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_nonzero_identity_command_exit() -> None: + async def command_runner(_args: Any) -> CommandResult: + return CommandResult(1, "", "ERROR: az login required") + + async def context_host(_context: str) -> str: + return FQDN + + deps = LiveDependencies( + command_runner=command_runner, + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="az aks show failed"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_default_command_runner_surfaces_missing_executable() -> None: + with pytest.raises(ValueError, match="executable not found"): + await live._default_command_runner(["korvid-test-definitely-missing-binary-xyz"]) + + +# --------------------------------------------------------------------------- +# Ownership gate +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_rejects_missing_namespace_before_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + del namespaces[manifests.namespace_name(RUN_ID, 5)] + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="missing expected namespaces"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_cross_run_namespace_label_before_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + bad_name = manifests.namespace_name(RUN_ID, 3) + namespaces[bad_name] = dataclasses.replace(namespaces[bad_name], labels=_labels("other-run")) + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="namespaces missing/mismatched ownership labels"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_missing_pod_before_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + key = live_object_identity(RUN_ID, 20, 42) + del pods[key] + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="missing"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_cross_run_pod_label_before_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + key = live_object_identity(RUN_ID, 20, 42) + pods[key] = dataclasses.replace(pods[key], labels=_labels("some-other-run-id")) + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + mutation_client_factory=_never_called("mutation_client_factory"), + ) + with pytest.raises(ValueError, match="mismatched ownership labels"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_never_constructs_app_client_when_ownership_fails() -> None: + """The MEDIUM finding's harness/app-path split must not weaken the + ownership-before-mutation ordering: when the ownership gate rejects, the + application-path (telemetry-wired) `KubeClient` must never even be + constructed - only the harness client is used for the (failing) + ownership check.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + del namespaces[manifests.namespace_name(RUN_ID, 5)] + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + mutation_client_factory=_never_called("mutation_client_factory"), + ) + deps = dataclasses.replace(deps, kube_client_factory=_never_called("kube_client_factory")) + with pytest.raises(ValueError, match="missing expected namespaces"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +# --------------------------------------------------------------------------- +# `_verify_ownership` returns a reusable validated snapshot (no redundant +# re-listing for uids) +# --------------------------------------------------------------------------- + + +async def test_verify_ownership_returns_validated_pod_snapshot() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + kube = _FakeKubeClient(namespaces, pods) + + validated = await live._verify_ownership( + kube, run_id=RUN_ID, namespace_count=20, object_count=1000 + ) + + assert validated == pods + # Exactly one `list_pods` call per namespace: the gate itself, no + # redundant second pass. + assert sorted(kube.list_pods_calls) == sorted(namespaces) + + +# --------------------------------------------------------------------------- +# Full happy path: real app-path wiring, telemetry, digest parity +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_full_happy_path_matches_cluster_digest() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + harness_clients: list[_FakeKubeClient] = [] + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps( + namespaces, pods, RUN_ID, harness_clients=harness_clients, app_clients=app_clients + ) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions( + time_scale=1.0, sample_interval=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.object_count == 1000 + assert report.dropped_updates == 0 + assert report.expected_digest == report.final_digest + assert report.expected_digest == summary_digest(pods.values()) + + # Exactly one harness client and one app-path client are constructed; + # both connect to the required context and both get closed. + assert len(harness_clients) == 1 + assert len(app_clients) == 1 + harness_client, app_client = harness_clients[0], app_clients[0] + assert harness_client.connect_context == CONTEXT + assert harness_client.closed + assert app_client.connect_context == CONTEXT + assert app_client.closed + + # MEDIUM finding: the app-path client's telemetry is the *only* source of + # `report.api` - every "list" comes from `watch_pods`'s own internal + # LIST-then-WATCH, not from any harness read. Three watches, not one: the + # `namespace_switch` UI-at-scale scenario really scopes down to a seeded + # namespace and back, and a scope change restarts the application watch. + assert app_client.watch_pods_calls == 3 + # Every watch stays cluster-wide: `make_live_watch_source` pins the read to + # the owned namespace set regardless of the UI scope, so scoping the table + # to one namespace never re-targets (or widens) the underlying watch. + assert app_client.watch_namespaces == [None, None, None] + # The watch really is stopped by teardown, so `watch_finished` is a + # meaningful signal for the ground-truth-read ordering assertion. + assert app_client.watch_finished + assert report.api.operations["watch_open"] == 3 + assert report.api.operations.get("list", 0) == 3 + + # The harness client never watches - it is only ever used for the + # ownership gate and the final independent re-read. + assert harness_client.watch_pods_calls == 0 + + # LOW finding: the ownership gate's validated snapshot is reused as the + # pre-churn uid snapshot - each namespace is only `list_pods`-ed twice on + # the harness client (the ownership gate itself, then the post-churn + # independent re-read), never a redundant third time for uids. + per_namespace_list_calls = Counter(harness_client.list_pods_calls) + assert set(per_namespace_list_calls.values()) == {2} + assert app_client.list_pods_calls == [] + + assert report.churn_started_before_input + + +async def test_run_live_replay_aborts_and_still_closes_clients_on_guard_failure() -> None: + """A guard failure mid-churn (a real API server's `test` op rejection) + must propagate as `ApiStatusError` *and* still close the harness/app-path + kube clients and the mutation client via `run_live_replay`'s `finally` + teardown - the same guarantee `run_replay` gives on any mid-run failure.""" + + class _AlwaysFailingMutationClient: + def __init__(self) -> None: + self.closed = False + self.connect_calls = 0 + + async def connect(self) -> None: + self.connect_calls += 1 + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + raise ApiStatusError(422, "test operation failed for guarded patch") + + async def close(self) -> None: + self.closed = True + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + harness_clients: list[_FakeKubeClient] = [] + app_clients: list[_FakeKubeClient] = [] + failing_client = _AlwaysFailingMutationClient() + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + mutation_client_factory=lambda run_id_arg: failing_client, + harness_clients=harness_clients, + app_clients=app_clients, + ) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + with pytest.raises(ApiStatusError, match="test operation failed"): + await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert harness_clients[0].closed + assert app_clients[0].closed + assert failing_client.closed + + +async def test_run_live_replay_propagates_cancelled_error_and_still_closes_clients() -> None: + """`asyncio.CancelledError` raised mid-churn (as real task cancellation + delivers into whatever the run is awaiting) must propagate out of + `run_live_replay` unchanged, and the sampler/watch-manager/mutation/kube + clients must still be closed via the nested `finally` blocks: cancellation + must never skip teardown or leave the real cluster's watches/clients open. + + This raises `CancelledError` directly from the injected `async_sleep` seam + rather than calling `Task.cancel()` on a task that owns an in-flight + `MeasuredKorvidApp.run_test()` - hard-cancelling *that* task triggers an + unrelated Textual/pytest GC quirk (a dangling internal reactive-watcher + coroutine surfacing as `PytestUnraisableExceptionWarning` at session + teardown) that has nothing to do with `live.py`'s own cleanup correctness. + """ + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + harness_clients: list[_FakeKubeClient] = [] + app_clients: list[_FakeKubeClient] = [] + created_clients: list[_FakeMutationClient] = [] + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + mutation_clients=created_clients, + harness_clients=harness_clients, + app_clients=app_clients, + ) + + async def cancelling_sleep(_delay: float) -> None: + # Yield first so the already-dispatched first mutation task runs to + # completion, then cancel exactly like a real `Task.cancel()` landing + # in the scheduler's sleep. + await asyncio.sleep(0) + raise asyncio.CancelledError + + # duration_seconds=1, steady_events_per_second=2 schedules events at + # offsets 0.0 and 0.5: the first is dispatched without sleeping + # (delay <= 0), the second's positive delay drives the cancelling sleep. + options = ReplayOptions(time_scale=1.0, monotonic_fn=lambda: 0.0, async_sleep=cancelling_sleep) + + with pytest.raises(asyncio.CancelledError): + await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert harness_clients[0].closed + assert app_clients[0].closed + assert created_clients + assert created_clients[0].closed + # Cancellation must abort before the second churn mutation - no broad + # cleanup, no continuing past the point of cancellation. + assert len(created_clients[0].calls) == 1 + + +# --------------------------------------------------------------------------- +# C1: event timing is recorded at watch receipt, never at patch ack +# --------------------------------------------------------------------------- + + +async def test_make_live_watch_source_records_owned_modified_events_at_receipt() -> None: + """Latency measurement must start where the *application* first sees the + event, exactly like the deterministic `_ReplaySource`. Initial `ADDED` + rows from the LIST phase and events from foreign namespaces are not churn + events and must not be recorded.""" + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + kube = _FakeKubeClient({}, pods) + recorder = BenchmarkRecorder() + expected_namespaces = frozenset(namespace for namespace, _name in pods) + source = make_live_watch_source(kube, expected_namespaces, run_id=run_id, recorder=recorder) + + agen = source("pods", "*") + try: + for _ in range(len(pods)): + await agen.__anext__() + # Four ADDED rows from the LIST phase are not churn events. + assert recorder.pending_count() == 0 + + owned = next(iter(pods.values())) + kube.events.put_nowait(("MODIFIED", owned)) + await agen.__anext__() + assert recorder.pending_count() == 1 + + foreign = PodSummary( + name="stray", + namespace="default", + phase="Running", + ready="1/1", + restarts=0, + node="node-x", + ) + unowned = dataclasses.replace(owned, labels=()) + kube.events.put_nowait(("MODIFIED", foreign)) + kube.events.put_nowait(("MODIFIED", unowned)) + await agen.__anext__() + assert recorder.pending_count() == 1 + finally: + assert isinstance(agen, AsyncGenerator) + await agen.aclose() + + +async def test_run_live_replay_records_at_receipt_when_the_patch_ack_lags() -> None: + """Reproduces the ack-race the review found: a real API server delivers the + watch event over a different connection than the patch response, so the + event can be rendered *before* the patch call returns. Recording at ack + then appended a pending entry after its own render, which the final wait + could never drain (a 60 s timeout) and which the CLI reported as a dropped + update.""" + + class _AckLagsMutationClient(_FakeMutationClient): + async def _apply(self, namespace: str, name: str, *, uid: str, tick: str) -> None: + await super()._apply(namespace, name, uid=uid, tick=tick) + # Let the watch deliver and render before the caller resumes. + for _ in range(20): + await asyncio.sleep(0.005) + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, app_clients=app_clients) + deps = dataclasses.replace( + deps, + mutation_client_factory=lambda run_id: _AckLagsMutationClient( + lambda: app_clients[-1], run_id + ), + ) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.dropped_updates == 0 + assert report.event_to_render.count == 2 + assert report.expected_digest == report.final_digest + + +# --------------------------------------------------------------------------- +# C2: metadata-only, ownership-preserving churn and digest convergence +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_churns_the_tick_label_and_preserves_everything_else() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + mutation_clients: list[_FakeMutationClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, mutation_clients=mutation_clients) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + mutated = {(namespace, name) for namespace, name, _tick in mutation_clients[0].calls} + assert mutated + for key, pod in pods.items(): + labels = dict(pod.labels) + assert labels[manifests.MANAGED_BY_LABEL] == manifests.MANAGED_BY_VALUE + assert labels[manifests.RUN_LABEL] == RUN_ID + # Kubelet-owned status is never touched by churn. + assert pod.phase == "Running" + assert (manifests.TICK_LABEL in labels) is (key in mutated) + assert report.expected_digest == report.final_digest + + +async def test_run_live_replay_reads_ground_truth_while_the_watch_is_live() -> None: + """The ground-truth read and the convergence wait must both happen inside + the measured window: reading after `watch_manager.stop_all()` compares a + frozen store against a cluster that was still changing.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + harness_clients: list[_FakeKubeClient] = [] + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps( + namespaces, pods, RUN_ID, harness_clients=harness_clients, app_clients=app_clients + ) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + watch_live_during_reads: list[bool] = [] + + def spy(_namespace: str) -> None: + if len(harness_clients[0].list_pods_calls) > 20: # the post-churn re-read + watch_live_during_reads.append(not app_clients[0].watch_finished) + + async def run() -> None: + await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + original_factory = deps.harness_kube_client_factory + + def harness_factory() -> _FakeKubeClient: + client = original_factory() + assert isinstance(client, _FakeKubeClient) + client.on_list_pods = spy + return client + + deps = dataclasses.replace(deps, harness_kube_client_factory=harness_factory) + await run() + + assert len(watch_live_during_reads) == 20 + assert all(watch_live_during_reads) + + +async def test_run_live_replay_waits_for_the_store_digest_to_converge() -> None: + """Watch propagation lags the patch acknowledgement. Without an explicit + convergence wait the store digest is read while the last events are still + in flight, producing a false digest mismatch (CLI exit 1).""" + + class _LaggingWatchMutationClient(_FakeMutationClient): + async def _apply(self, namespace: str, name: str, *, uid: str, tick: str) -> None: + kube = self._resolve_kube() + before = kube.events.qsize() + await super()._apply(namespace, name, uid=uid, tick=tick) + # Pull the just-queued event back out and re-deliver it later, so + # the cluster is already mutated while the watch has not caught up. + assert kube.events.qsize() == before + 1 + delayed = kube.events.get_nowait() + asyncio.get_running_loop().call_later(1.0, kube.events.put_nowait, delayed) + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, app_clients=app_clients) + deps = dataclasses.replace( + deps, + mutation_client_factory=lambda run_id: _LaggingWatchMutationClient( + lambda: app_clients[-1], run_id + ), + ) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.expected_digest == report.final_digest + assert report.expected_digest == summary_digest(pods.values()) + assert report.dropped_updates == 0 + + +def test_check_row_count_rejects_a_regressed_render() -> None: + """A late watch reconnect clears and re-seeds the store, so a digest that + matched a moment ago can be recomputed over a partial store; the rendered + row count is re-asserted at the same instant.""" + live._check_row_count(1000, 1000) + + with pytest.raises(ValueError, match="rendered row count regressed"): + live._check_row_count(999, 1000) + + +# --------------------------------------------------------------------------- +# C3: bounded concurrency, bounded operation time, guarded 429-only retry +# --------------------------------------------------------------------------- + + +def _churn_profile(*, events_per_second: int, duration_seconds: int = 1) -> WorkloadProfile: + return WorkloadProfile( + schema_version=1, + id="churn-test", + seed=7, + object_count=4, + namespace_count=2, + steady_events_per_second=events_per_second, + duration_seconds=duration_seconds, + bursts=(), + failures=(), + ) + + +async def test_drive_live_churn_bounds_in_flight_mutations() -> None: + """Serial patching cannot approach the scheduled rate (one round trip per + event), so churn dispatches concurrently - but never without a ceiling.""" + + class _YieldingMutationClient(_FakeMutationClient): + async def _apply(self, namespace: str, name: str, *, uid: str, tick: str) -> None: + for _ in range(3): + await asyncio.sleep(0) + await super()._apply(namespace, name, uid=uid, tick=tick) + + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + kube = _FakeKubeClient({}, pods) + mutation_client = _YieldingMutationClient(kube, run_id) + events = scheduled_events(_churn_profile(events_per_second=40)) + monotonic_fn, async_sleep = _virtual_clock() + progress = ChurnProgress(requested_events=len(events)) + + await drive_live_churn( + events, + run_id=run_id, + namespace_count=2, + live_state=dict(pods), + mutation_client=mutation_client, + options=ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep), + progress=progress, + limits=LiveLimits(churn_concurrency=4), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), + ) + + assert progress.completed == len(events) == 40 + assert mutation_client.max_in_flight > 1, "churn must not be effectively serial" + assert mutation_client.max_in_flight <= 4 + + +async def test_drive_live_churn_retries_only_429_with_the_identical_guarded_patch() -> None: + """A single API Priority and Fairness throttle must not kill a 30-minute + run, but the retry re-issues the *identical* guarded patch (same uid, same + ownership tests) - it is never a relaxed or unguarded retry.""" + + class _ThrottlingMutationClient(_FakeMutationClient): + def __init__(self, kube: _FakeKubeClient, run_id: str, throttles: int) -> None: + super().__init__(kube, run_id) + self.remaining_throttles = throttles + self.patch_arguments: list[tuple[str, str, str, str]] = [] + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.patch_arguments.append((namespace, name, uid, tick)) + if self.remaining_throttles > 0: + self.remaining_throttles -= 1 + raise ApiStatusError(429, "Too Many Requests") + await super().patch_pod_labels_guarded(namespace, name, uid=uid, tick=tick) + + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + kube = _FakeKubeClient({}, pods) + mutation_client = _ThrottlingMutationClient(kube, run_id, throttles=2) + events = scheduled_events(_churn_profile(events_per_second=1)) + monotonic_fn, async_sleep = _virtual_clock() + progress = ChurnProgress(requested_events=len(events)) + + await drive_live_churn( + events, + run_id=run_id, + namespace_count=2, + live_state=dict(pods), + mutation_client=mutation_client, + options=ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep), + progress=progress, + limits=LiveLimits(churn_concurrency=1, mutation_throttle_retries=5), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), + ) + + assert len(mutation_client.patch_arguments) == 3 + assert len(set(mutation_client.patch_arguments)) == 1 + assert progress.mutation_throttles == 2 + assert progress.completed == 1 + + +@pytest.mark.parametrize( + ("headers", "body", "expected"), + [ + ({"Retry-After": "7"}, '{"details": {"retryAfterSeconds": 5}}', 7.0), + ({}, '{"details": {"retryAfterSeconds": 5}}', 5.0), + ], +) +async def test_mutation_client_preserves_server_retry_after_hint( + headers: dict[str, str], body: str, expected: float +) -> None: + class _ThrottledCoreV1: + async def patch_namespaced_pod(self, *_args: object, **_kwargs: object) -> None: + exc = k8s_client.exceptions.ApiException(status=429, reason="Too Many Requests") + object.__setattr__(exc, "headers", CIMultiDictProxy(CIMultiDict(headers))) + object.__setattr__(exc, "body", body.encode()) + raise exc + + client = live._KubeMutationClient(CONTEXT, RUN_ID) + client._core_v1 = _ThrottledCoreV1() # type: ignore[assignment] # focused adapter fake + + with pytest.raises(ApiStatusError, match="API 429") as caught: + await client.patch_pod_labels_guarded( + "korvid-perf-aks186-0", + "bench-0", + uid="pod-uid-0", + tick="1", + ) + + assert caught.value.body == body + assert caught.value.retry_after_seconds == expected + + +async def test_mutation_client_connect_uses_refreshable_kube_config( + monkeypatch: pytest.MonkeyPatch, +) -> None: + load_refreshable = AsyncMock() + api = MagicMock() + api_factory = MagicMock(return_value=api) + core_v1_factory = MagicMock() + monkeypatch.setattr(live, "load_refreshable_kube_config", load_refreshable, raising=False) + monkeypatch.setattr(k8s_config, "load_kube_config", AsyncMock()) + monkeypatch.setattr(k8s_client, "ApiClient", api_factory) + monkeypatch.setattr(k8s_client, "CoreV1Api", core_v1_factory) + + client = live._KubeMutationClient(CONTEXT, RUN_ID) + await client.connect() + + load_refreshable.assert_awaited_once() + call = load_refreshable.await_args + assert call is not None + assert call.kwargs["context"] == CONTEXT + assert call.kwargs["persist_config"] is False + configuration = call.kwargs["client_configuration"] + api_factory.assert_called_once_with(configuration) + core_v1_factory.assert_called_once_with(api) + + +async def test_mutation_retry_respects_server_hint_with_an_explicit_delay_bound() -> None: + class _ThrottleOnce: + def __init__(self) -> None: + self.calls = 0 + + async def connect(self) -> None: + pass + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.calls += 1 + if self.calls == 1: + raise ApiStatusError( + 429, + "Too Many Requests", + retry_after_seconds=10.0, + ) + + async def close(self) -> None: + pass + + client = _ThrottleOnce() + sleeps: list[float] = [] + + async def _sleep(delay: float) -> None: + sleeps.append(delay) + + await live._mutate_once( + client, + namespace="korvid-perf-run1-0", + name="bench-0", + uid="pod-uid-0", + tick="1", + progress=ChurnProgress(), + limits=LiveLimits( + mutation_throttle_retries=1, + mutation_retry_base_delay_seconds=0.5, + mutation_retry_max_delay_seconds=3.0, + ), + sleep=_sleep, + now=lambda: 0.0, + ) + + assert client.calls == 2 + assert sleeps == [3.0] + + +async def test_mutation_retry_jitter_avoids_lockstep_workers_when_server_hint_dominates() -> None: + class _ThrottleOnce: + def __init__(self) -> None: + self.calls = 0 + + async def connect(self) -> None: + pass + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.calls += 1 + if self.calls == 1: + raise ApiStatusError( + 429, + "Too Many Requests", + retry_after_seconds=1.0, + ) + + async def close(self) -> None: + pass + + sleeps: list[list[float]] = [[], []] + for index, name in enumerate(("bench-0", "bench-1")): + + async def _sleep(delay: float, target: list[float] = sleeps[index]) -> None: + target.append(delay) + + await live._mutate_once( + _ThrottleOnce(), + namespace="korvid-perf-run1-0", + name=name, + uid=f"pod-uid-{index}", + tick="1", + progress=ChurnProgress(), + limits=LiveLimits( + mutation_throttle_retries=1, + mutation_retry_base_delay_seconds=0.5, + mutation_retry_max_delay_seconds=3.0, + ), + sleep=_sleep, + now=lambda: 0.0, + ) + + assert 1.0 < sleeps[0][0] <= 1.5 + assert 1.0 < sleeps[1][0] <= 1.5 + assert sleeps[0] != sleeps[1] + + +async def test_drive_live_churn_gives_up_after_the_bounded_throttle_retries() -> None: + class _AlwaysThrottling(_FakeMutationClient): + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.calls.append((namespace, name, tick)) + raise ApiStatusError(429, "Too Many Requests") + + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + mutation_client = _AlwaysThrottling(_FakeKubeClient({}, pods), run_id) + events = scheduled_events(_churn_profile(events_per_second=1)) + monotonic_fn, async_sleep = _virtual_clock() + progress = ChurnProgress(requested_events=len(events)) + + with pytest.raises(ApiStatusError, match="API 429"): + await drive_live_churn( + events, + run_id=run_id, + namespace_count=2, + live_state=dict(pods), + mutation_client=mutation_client, + options=ReplayOptions( + time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ), + progress=progress, + limits=LiveLimits(churn_concurrency=1, mutation_throttle_retries=2), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), + ) + + assert len(mutation_client.calls) == 3 + assert progress.mutation_throttles == 2 + + +async def test_drive_live_churn_never_retries_a_failed_ownership_guard() -> None: + """422 is a failed JSON-Patch `test` op: the target is not what the run + validated. Retrying it - guarded or not - is exactly the behaviour the + safety contract forbids.""" + + class _GuardFailingMutationClient(_FakeMutationClient): + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.calls.append((namespace, name, tick)) + raise ApiStatusError(422, "test operation failed for guarded patch") + + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + mutation_client = _GuardFailingMutationClient(_FakeKubeClient({}, pods), run_id) + events = scheduled_events(_churn_profile(events_per_second=1)) + monotonic_fn, async_sleep = _virtual_clock() + progress = ChurnProgress(requested_events=len(events)) + + with pytest.raises(ApiStatusError, match="test operation failed"): + await drive_live_churn( + events, + run_id=run_id, + namespace_count=2, + live_state=dict(pods), + mutation_client=mutation_client, + options=ReplayOptions( + time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ), + progress=progress, + limits=LiveLimits(churn_concurrency=1, mutation_throttle_retries=5), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), + ) + + assert len(mutation_client.calls) == 1 + assert progress.mutation_throttles == 0 + + +async def test_drive_live_churn_bounds_every_mutation_attempt() -> None: + """A stalled patch must not hang the run: every attempt is bounded.""" + + class _HangingMutationClient(_FakeMutationClient): + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + self.calls.append((namespace, name, tick)) + await asyncio.sleep(30) + + run_id = "run1" + _, pods = _build_fake_topology(run_id, 2, 4) + mutation_client = _HangingMutationClient(_FakeKubeClient({}, pods), run_id) + events = scheduled_events(_churn_profile(events_per_second=1)) + progress = ChurnProgress(requested_events=len(events)) + + with pytest.raises(TimeoutError): + await drive_live_churn( + events, + run_id=run_id, + namespace_count=2, + live_state=dict(pods), + mutation_client=mutation_client, + options=ReplayOptions(time_scale=1.0), + progress=progress, + limits=LiveLimits(churn_concurrency=1, mutation_timeout_seconds=0.05), + profile=_tiny_live_profile(), + recorder=BenchmarkRecorder(), + ) + + assert len(mutation_client.calls) == 1 + assert progress.completed == 0 + + +async def test_run_live_replay_reports_requested_and_achieved_churn_rate() -> None: + """The design doc forbids presenting a requested rate as an achieved one: + both must be reported, and they must be allowed to differ.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = _happy_deps(namespaces, pods, RUN_ID) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + profile = dataclasses.replace(_tiny_live_profile(), duration_seconds=2) + report = await run_live_replay( + profile, + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.churn is not None + # 4 events requested over 2 seconds (2 events/s requested); the driver + # observed all 4 within a 1.0 s window, i.e. a different achieved rate. + assert report.churn.requested_events == 4 + assert report.churn.requested_events_per_second == 2.0 + assert report.churn.observed_events == 4 + assert report.churn.wall_seconds == 1.0 + assert report.churn.achieved_events_per_second == 4.0 + assert report.churn.requested_events_per_second != report.churn.achieved_events_per_second + assert report.churn.mutation_throttles == 0 + + +async def test_run_live_replay_counts_mutation_throttles_outside_read_telemetry() -> None: + """Harness write traffic must never be reported as application read-path + API telemetry.""" + + class _ThrottleOnceMutationClient(_FakeMutationClient): + def __init__( + self, kube: Callable[[], _FakeKubeClient] | _FakeKubeClient, run_id: str + ) -> None: + super().__init__(kube, run_id) + self.throttled = False + + async def patch_pod_labels_guarded( + self, namespace: str, name: str, *, uid: str, tick: str + ) -> None: + if not self.throttled: + self.throttled = True + raise ApiStatusError(429, "Too Many Requests") + await super().patch_pod_labels_guarded(namespace, name, uid=uid, tick=tick) + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, app_clients=app_clients) + deps = dataclasses.replace( + deps, + mutation_client_factory=lambda run_id: _ThrottleOnceMutationClient( + lambda: app_clients[-1], run_id + ), + ) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.churn is not None + assert report.churn.mutation_throttles == 1 + assert report.api.throttles == 0 + assert report.api.operations.get("error", 0) == 0 + assert report.expected_digest == report.final_digest + + +# --------------------------------------------------------------------------- +# I2: cancellation cancels and drains every mutation task before teardown +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_stops_mutating_when_the_outer_task_is_cancelled() -> None: + """Real `Task.cancel()` on the whole run: the churn task (and every patch + inside its task group) must be cancelled and awaited *before* the clients + are closed, so no mutation outlives teardown.""" + + class _PacedMutationClient(_FakeMutationClient): + async def _apply(self, namespace: str, name: str, *, uid: str, tick: str) -> None: + await asyncio.sleep(0.01) + await super()._apply(namespace, name, uid=uid, tick=tick) + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + app_clients: list[_FakeKubeClient] = [] + harness_clients: list[_FakeKubeClient] = [] + created: list[_PacedMutationClient] = [] + + def mutation_factory(run_id: str) -> _PacedMutationClient: + client = _PacedMutationClient(lambda: app_clients[-1], run_id) + created.append(client) + return client + + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + app_clients=app_clients, + harness_clients=harness_clients, + mutation_client_factory=mutation_factory, + ) + # 300 events over 30 s of real wall time: the run is still churning when + # the cancellation lands. + profile = dataclasses.replace( + _tiny_live_profile(), steady_events_per_second=10, duration_seconds=30 + ) + + task = asyncio.create_task( + run_live_replay( + profile, + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + ) + for _ in range(2000): + if created and len(created[0].calls) >= 2: + break + await asyncio.sleep(0.005) + assert created, "the mutation client was never constructed" + assert len(created[0].calls) >= 2, "churn never started" + + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + calls_at_cancellation = len(created[0].calls) + for _ in range(20): + await asyncio.sleep(0.01) + assert len(created[0].calls) == calls_at_cancellation + assert created[0].closed + assert harness_clients[0].closed + assert app_clients[0].closed + + +# --------------------------------------------------------------------------- +# I3/I4: exact identity and ownership, before *and* after churn +# --------------------------------------------------------------------------- + + +async def test_verify_ownership_rejects_unexpected_pod_in_an_owned_namespace() -> None: + """The application watch filters by namespace, so a foreign Pod in an owned + namespace enters the benchmark store; the gate must name it instead of + letting the initial-render wait time out with a generic message.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + stray_namespace = manifests.namespace_name(RUN_ID, 3) + pods[(stray_namespace, "intruder")] = PodSummary( + name="intruder", + namespace=stray_namespace, + phase="Running", + ready="1/1", + restarts=0, + node="node-9", + labels=_labels(RUN_ID), + ) + kube = _FakeKubeClient(namespaces, pods) + + with pytest.raises( + ValueError, match=f"unexpected pods in owned namespaces: {stray_namespace}/intruder" + ): + await live._verify_ownership(kube, run_id=RUN_ID, namespace_count=20, object_count=1000) + + +async def test_read_and_validate_owned_pods_returns_the_validated_snapshot() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + kube = _FakeKubeClient(namespaces, pods) + + validated = await read_and_validate_owned_pods(kube, run_id=RUN_ID, expected=pods) + + assert sorted((pod.namespace, pod.name) for pod in validated) == sorted(pods) + + +@pytest.mark.parametrize( + ("corrupt", "message"), + [ + ("labels", "lost ownership labels"), + ("uid", "uid changed"), + ("missing", "missing"), + ("extra", "unexpected pod"), + ], +) +async def test_read_and_validate_owned_pods_rejects_identity_or_ownership_loss( + corrupt: str, message: str +) -> None: + """The ground-truth digest may only be computed from Pods that still are + what the ownership gate validated.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + expected = dict(pods) + key = live_object_identity(RUN_ID, 20, 42) + if corrupt == "labels": + pods[key] = dataclasses.replace(pods[key], labels=_labels("some-other-run")) + elif corrupt == "uid": + pods[key] = dataclasses.replace(pods[key], uid="recreated-uid") + elif corrupt == "missing": + del pods[key] + else: + namespace = key[0] + pods[(namespace, "intruder")] = dataclasses.replace( + pods[key], name="intruder", uid="intruder-uid" + ) + kube = _FakeKubeClient(namespaces, pods) + + with pytest.raises(ValueError, match=f"post-churn ownership revalidation failed.*{message}"): + await read_and_validate_owned_pods(kube, run_id=RUN_ID, expected=expected) + + +async def test_run_live_replay_rejects_a_pod_that_lost_ownership_during_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + harness_clients: list[_FakeKubeClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, harness_clients=harness_clients) + victim = live_object_identity(RUN_ID, 20, 7) + + def strip_ownership_before_the_final_read(_namespace: str) -> None: + if len(harness_clients[0].list_pods_calls) > 20: + pods[victim] = dataclasses.replace(pods[victim], labels=()) + + original_factory = deps.harness_kube_client_factory + + def harness_factory() -> _FakeKubeClient: + client = original_factory() + assert isinstance(client, _FakeKubeClient) + client.on_list_pods = strip_ownership_before_the_final_read + return client + + deps = dataclasses.replace(deps, harness_kube_client_factory=harness_factory) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + with pytest.raises(ValueError, match="post-churn ownership revalidation failed"): + await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +# --------------------------------------------------------------------------- +# I7 / M12 / M7 / profile revalidation +# --------------------------------------------------------------------------- + + +async def test_run_live_replay_names_watch_api_errors_in_a_wait_timeout() -> None: + """`KorvidApp.on_mount` overwrites `WatchManager.on_error` with its own TUI + notification, so a 403 that killed the watch is otherwise invisible: the + operator would see only "not met within Ns" after paying for cluster setup.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + app_clients: list[_FakeKubeClient] = [] + deps = _happy_deps(namespaces, pods, RUN_ID, app_clients=app_clients) + original_factory = deps.kube_client_factory + + def failing_kube_factory(read_telemetry: ReadTelemetry) -> _FakeKubeClient: + client = original_factory(read_telemetry) + assert isinstance(client, _FakeKubeClient) + client.watch_error = ApiStatusError(403, "Forbidden") + return client + + deps = dataclasses.replace(deps, kube_client_factory=failing_kube_factory) + + with pytest.raises(WaitTimeout, match="status=403"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(initial_render_timeout_seconds=0.3), + ) + + +async def test_run_live_replay_connects_the_mutation_client_before_the_app_starts() -> None: + """`load_kube_config` can invoke an exec credential plugin; that latency is + paid once, up front, under an explicit bound - not inside the first + measured mutation.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + mutation_clients: list[_FakeMutationClient] = [] + app_clients: list[_FakeKubeClient] = [] + connect_order: list[str] = [] + + def mutation_factory(run_id: str) -> _FakeMutationClient: + client = _FakeMutationClient(lambda: app_clients[-1], run_id) + mutation_clients.append(client) + return client + + deps = _happy_deps( + namespaces, + pods, + RUN_ID, + app_clients=app_clients, + mutation_client_factory=mutation_factory, + ) + original_kube_factory = deps.kube_client_factory + + def recording_kube_factory(read_telemetry: ReadTelemetry) -> _FakeKubeClient: + connect_order.append("app-client") + return original_kube_factory(read_telemetry) # type: ignore[return-value] + + deps = dataclasses.replace(deps, kube_client_factory=recording_kube_factory) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert mutation_clients[0].connect_calls == 1 + assert connect_order == ["app-client"] + + +async def test_run_live_replay_bounds_the_mutation_client_connect() -> None: + class _HangingConnectMutationClient(_FakeMutationClient): + async def connect(self) -> None: + await asyncio.sleep(30) + + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + created: list[_HangingConnectMutationClient] = [] + + def mutation_factory(run_id: str) -> _HangingConnectMutationClient: + client = _HangingConnectMutationClient(_FakeKubeClient(namespaces, pods), run_id) + created.append(client) + return client + + deps = _happy_deps(namespaces, pods, RUN_ID, mutation_client_factory=mutation_factory) + + with pytest.raises(TimeoutError): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(mutation_connect_timeout_seconds=0.05), + ) + + assert created[0].closed + + +async def test_run_live_replay_measures_input_latency_with_the_injected_clock() -> None: + """Churn uses the injected clock; input latency must use the same one, or a + virtual-clock latency assertion measures nothing.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = _happy_deps(namespaces, pods, RUN_ID) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions(time_scale=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.input_latency.count == 2 + assert report.input_latency.maximum_seconds == 0.0 + + +async def test_run_live_replay_rejects_a_profile_whose_bursts_escape_its_duration() -> None: + """A profile rewritten with `dataclasses.replace` (the CLI's `--duration`) + never passes through `load_profile`; the invariants are re-checked here, + before any cluster identity or ownership work.""" + profile = dataclasses.replace( + _tiny_live_profile(), + duration_seconds=2, + bursts=(Burst(start_second=1, duration_seconds=30, events_per_second=100),), + ) + deps = LiveDependencies( + command_runner=_never_called("command_runner"), + active_context=_never_called("active_context"), + context_host=_never_called("context_host"), + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + with pytest.raises(ValueError, match="falls outside duration_seconds"): + await run_live_replay( + profile, + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_bounds_the_context_host_lookup() -> None: + """A kubeconfig exec plugin invoked while resolving the API hostname can + hang; the identity gate must bound that lookup, not block forever.""" + + async def hanging_context_host(_context: str) -> str: + await asyncio.sleep(30) + return FQDN + + deps = LiveDependencies( + command_runner=_ok_command_runner(), + active_context=lambda: CONTEXT, + context_host=hanging_context_host, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + with pytest.raises(TimeoutError): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(read_connect_timeout_seconds=0.05), + ) + + +async def test_run_live_replay_bounds_the_harness_client_connect() -> None: + """A stuck harness-client connect (exec credential plugin) must be bounded + before the ownership gate can run.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + + class _HangingConnectKube(_FakeKubeClient): + async def connect(self, context: str | None = None) -> None: + await asyncio.sleep(30) + + created: list[_HangingConnectKube] = [] + + def harness_factory() -> _HangingConnectKube: + client = _HangingConnectKube(namespaces, pods) + created.append(client) + return client + + deps = dataclasses.replace( + _happy_deps(namespaces, pods, RUN_ID), + harness_kube_client_factory=harness_factory, + ) + + with pytest.raises(TimeoutError): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(read_connect_timeout_seconds=0.05), + ) + + assert created[0].closed + + +async def test_run_live_replay_bounds_the_app_client_connect() -> None: + """A stuck application-path client connect must be bounded too.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + + class _HangingConnectKube(_FakeKubeClient): + async def connect(self, context: str | None = None) -> None: + await asyncio.sleep(30) + + created: list[_HangingConnectKube] = [] + + def app_factory(read_telemetry: ReadTelemetry) -> _HangingConnectKube: + client = _HangingConnectKube(namespaces, pods) + client.read_telemetry = read_telemetry + created.append(client) + return client + + deps = dataclasses.replace( + _happy_deps(namespaces, pods, RUN_ID), + kube_client_factory=app_factory, + ) + + with pytest.raises(TimeoutError): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(read_connect_timeout_seconds=0.05), + ) + + assert created[0].closed + + +def _identity_deps(command_runner: Callable[[Any], Awaitable[CommandResult]]) -> LiveDependencies: + async def context_host(_context: str) -> str: + return FQDN + + return LiveDependencies( + command_runner=command_runner, + active_context=lambda: CONTEXT, + context_host=context_host, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + +async def test_verify_cluster_identity_uses_fixed_resource_group_and_cluster_name_lookup() -> None: + captured_args: list[list[str]] = [] + + async def recording_command_runner(args: list[str]) -> CommandResult: + captured_args.append(args) + return await _ok_command_runner()(args) + + await live._verify_cluster_identity( + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + deps=_identity_deps(recording_command_runner), + limits=LiveLimits(), + ) + + assert captured_args == [ + [ + "az", + "aks", + "show", + "--resource-group", + RESOURCE_GROUP, + "--name", + CLUSTER_NAME, + "-o", + "json", + ] + ] + + +async def test_run_live_replay_bounds_the_az_aks_show_lookup() -> None: + """The first external `az aks show` call in the identity gate can hang on a + stuck credential/exec plugin; it must be bounded by the read/connect timeout, + not block the fail-closed gate forever.""" + + async def hanging_command_runner(_args: Any) -> CommandResult: + await asyncio.sleep(30) + return CommandResult(0, "{}", "") + + deps = _identity_deps(hanging_command_runner) + + with pytest.raises(TimeoutError): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + limits=LiveLimits(read_connect_timeout_seconds=0.05), + ) + + +async def test_run_live_replay_rejects_wrong_resource_group_before_mutation() -> None: + deps = _identity_deps(_ok_command_runner(resource_group="rg-production")) + with pytest.raises(ValueError, match="resource group"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_wrong_cluster_name_before_mutation() -> None: + deps = _identity_deps(_ok_command_runner(name="aks-production")) + with pytest.raises(ValueError, match="cluster name"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_missing_required_tag_before_mutation() -> None: + deps = _identity_deps(_ok_command_runner(tags={"purpose": "korvid-contract-testing"})) + with pytest.raises(ValueError, match="required tag"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_wrong_required_tag_value_before_mutation() -> None: + deps = _identity_deps( + _ok_command_runner(tags={"purpose": "korvid-contract-testing", "production-use": "allowed"}) + ) + with pytest.raises(ValueError, match="required tag"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_non_running_pod_before_churn() -> None: + """The ownership/preflight gate requires exactly 1,000 Running, Ready owned + Pods: a labelled but non-Running Pod must be reported and reject the run + before any mutation, since the later check only counts table rows.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + victim = (manifests.namespace_name(RUN_ID, 0), manifests.pod_name(20, 0)) + pods[victim] = dataclasses.replace(pods[victim], phase="Pending") + + deps = LiveDependencies( + command_runner=_ok_command_runner(), + active_context=lambda: CONTEXT, + context_host=_context_host_ok, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=lambda: _FakeKubeClient(namespaces, pods), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + with pytest.raises(ValueError, match="not Running or not Ready"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_rejects_not_ready_pod_before_churn() -> None: + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + victim = (manifests.namespace_name(RUN_ID, 1), manifests.pod_name(20, 1)) + pods[victim] = dataclasses.replace(pods[victim], ready="0/1") + + deps = LiveDependencies( + command_runner=_ok_command_runner(), + active_context=lambda: CONTEXT, + context_host=_context_host_ok, + kube_client_factory=_never_called("kube_client_factory"), + harness_kube_client_factory=lambda: _FakeKubeClient(namespaces, pods), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + with pytest.raises(ValueError, match="not Running or not Ready"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_fails_closed_when_sha_cannot_be_resolved() -> None: + """A live evidence run must not publish an untraceable artifact: if no + immutable korvid SHA can be resolved, the run fails closed before any + client is constructed.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = dataclasses.replace( + _happy_deps(namespaces, pods, RUN_ID), + resolve_sha=lambda: None, + harness_kube_client_factory=_never_called("harness_kube_client_factory"), + kube_client_factory=_never_called("kube_client_factory"), + mutation_client_factory=_never_called("mutation_client_factory"), + ) + + with pytest.raises(ValueError, match="immutable korvid SHA"): + await run_live_replay( + _tiny_live_profile(), + ReplayOptions(time_scale=1.0), + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + +async def test_run_live_replay_builds_a_live_specific_manifest() -> None: + """Retained live evidence must record which cluster matrix was qualified: + the verified context/ARM id plus Kubernetes server version and node-pool + metadata, resolved through bounded seams and persisted in the manifest.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = _happy_deps(namespaces, pods, RUN_ID) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions( + time_scale=1.0, sample_interval=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + manifest = report.manifest + assert manifest.korvid_sha == _FIXED_SHA + assert manifest.context == CONTEXT + assert manifest.cluster_id == CLUSTER_ID + assert manifest.kubernetes_version == "1.30.4" + pool_names = {pool.name for pool in manifest.node_pools} + assert "perftest" in pool_names + perftest = next(pool for pool in manifest.node_pools if pool.name == "perftest") + assert perftest.node_count == 5 + assert perftest.kubernetes_version == "1.30.4" + + # Persisted in both machine-readable and human-readable output. + from tests.performance.metrics import render_markdown, report_payload + + payload = report_payload(_report_as_benchmark(report)) + manifest_payload = payload["manifest"] + assert isinstance(manifest_payload, dict) + assert manifest_payload["context"] == CONTEXT + assert manifest_payload["cluster_id"] == CLUSTER_ID + assert manifest_payload["kubernetes_version"] == "1.30.4" + assert manifest_payload["node_pools"] == [ + {"name": "perftest", "kubernetes_version": "1.30.4", "node_count": 5}, + {"name": "system", "kubernetes_version": "1.30.4", "node_count": 1}, + ] + + markdown = render_markdown(_report_as_benchmark(report)) + assert f"- Context: `{CONTEXT}`" in markdown + assert "- Kubernetes version: `1.30.4`" in markdown + assert "perftest" in markdown + + +async def test_run_live_replay_exercises_ui_at_scale_scenarios_during_churn() -> None: + """A passing live run must provide UI-at-scale evidence: filter, sort, + namespace switch, split pane, describe, and multi-log are driven through the + real Textual pilot during active churn and their outcomes/latencies are + recorded, without weakening the digest/ownership safety guarantees.""" + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = _happy_deps(namespaces, pods, RUN_ID) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions( + time_scale=1.0, sample_interval=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ) + + report = await run_live_replay( + _tiny_live_profile(), + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + # Safety invariants still hold. + assert report.expected_digest == report.final_digest + assert report.dropped_updates == 0 + + recorded = {scenario.name for scenario in report.ui_scenarios} + assert recorded == {"filter", "sort", "namespace_switch", "split_pane", "describe", "multi_log"} + for scenario in report.ui_scenarios: + assert scenario.ok, f"scenario {scenario.name} did not complete" + assert scenario.latency_seconds >= 0.0 + + +class _InertPilot: + """A pilot whose key presses reach an app that never changes state. + + `pilot.press` returning without raising says nothing about whether the + binding did anything: in the live wiring `describe` and `multi_log` bail + out with an "unavailable" warning when no manifest/log provider is wired, + and the namespace toggle is a no-op while `config.namespace` already is + `ALL_NAMESPACES`. None of those raise. + """ + + def __init__(self) -> None: + self.pressed: list[str] = [] + + async def press(self, key: str) -> None: + self.pressed.append(key) + + async def pause(self, delay: float | None = None) -> None: + return None + + +class _InertApp: + """Live app state frozen at its post-LIST values: nothing a scenario is + supposed to change ever changes.""" + + def __init__(self) -> None: + self.filter_pattern = "" + self._sorts: dict[str, object] = {} + self.current_scope = ALL_NAMESPACES + self.screen = object() + self._panes = [object()] + + class _Pane: + display = False + + self._log_pane = _Pane() + + +async def test_ui_scenarios_are_not_marked_ok_when_the_app_state_never_changes() -> None: + """A scenario must assert the observable state it claims to exercise. + + Without this the live report claims UI-at-scale evidence for scenarios + that silently did nothing, and (since the CLI folds scenario outcomes into + the exit status) a qualification would "pass" on that empty evidence. + """ + recorder = BenchmarkRecorder() + ticks = iter(float(i) for i in range(10_000)) + + await live.drive_ui_scenarios( + _InertPilot(), + recorder, + now=lambda: next(ticks), + app=_InertApp(), + scoped_namespace=f"korvid-perf-{RUN_ID}-0", + ) + + results = recorder._ui_scenarios + assert results, "scenarios must still be recorded, not skipped" + assert [scenario.name for scenario in results if scenario.ok] == [] + + +async def test_run_live_replay_times_the_post_burst_drain() -> None: + """The live churn driver must mark each burst boundary. + + Without it `post_burst_drain_seconds` stays empty and + `max_post_burst_drain_seconds` is `None`, so the published <=3-second live + burst-drain budget cannot be evaluated at all — the live reports simply + print "n/a" where the evidence should be. + """ + namespaces, pods = _build_fake_topology(RUN_ID, 20, 1000) + deps = _happy_deps(namespaces, pods, RUN_ID) + monotonic_fn, async_sleep = _virtual_clock() + options = ReplayOptions( + time_scale=1.0, sample_interval=1.0, monotonic_fn=monotonic_fn, async_sleep=async_sleep + ) + profile = dataclasses.replace( + _tiny_live_profile(), + steady_events_per_second=2, + duration_seconds=4, + bursts=(Burst(start_second=1, duration_seconds=1, events_per_second=4),), + ) + + report = await run_live_replay( + profile, + options, + context=CONTEXT, + expected_cluster_id=CLUSTER_ID, + run_id=RUN_ID, + deps=deps, + ) + + assert report.phases.post_burst_drain_seconds + assert report.phases.max_post_burst_drain_seconds is not None + assert report.phases.max_post_burst_drain_seconds >= 0.0 + + +def _pod_summary(namespace: str, name: str, **kwargs: Any) -> PodSummary: + defaults: dict[str, Any] = { + "namespace": namespace, + "name": name, + "ready": "1/1", + "phase": "Running", + "restarts": 0, + "node": "node-a", + "created": "", + "uid": "uid-1", + "labels": {}, + } + defaults.update(kwargs) + return PodSummary(**{k: v for k, v in defaults.items() if k in _POD_SUMMARY_FIELDS}) + + +_POD_SUMMARY_FIELDS = {field.name for field in dataclasses.fields(PodSummary)} + + +class _StubRow: + def __init__(self, value: str) -> None: + self.key = SimpleNamespace(value=value) + + +class _StubTable: + """Minimal stand-in exposing exactly what the rendered-row check reads.""" + + def __init__(self, rows: dict[str, list[object]]) -> None: + self._rows = rows + self.row_count = len(rows) + self.ordered_rows = [_StubRow(key) for key in rows] + + def get_row(self, row_key: object) -> list[object]: + key = row_key.value if hasattr(row_key, "value") else row_key + return self._rows[str(key)] + + +def test_rendered_rows_check_rejects_a_stale_cell() -> None: + """The published digest criterion compares a store digest with a store + digest, so 1,000 stale cells satisfy it. The rendered table must be checked + against the store independently — especially now that the in-place diff + updates cells from its own cached record of what it last wrote. + """ + pods = [ + _pod_summary("ns-a", "bench-0", phase="Running"), + _pod_summary("ns-a", "bench-1", phase="CrashLoopBackOff"), + ] + table = _StubTable( + { + "ns-a/bench-0": ["ns-a", "bench-0", "1/1", "Running", "0", "node-a"], + # Stale: the store says CrashLoopBackOff, the table still shows Running. + "ns-a/bench-1": ["ns-a", "bench-1", "1/1", "Running", "0", "node-a"], + } + ) + + with pytest.raises(ValueError, match="ns-a/bench-1"): + replay_mod.check_rendered_rows(table, pods) + + +def test_rendered_rows_check_accepts_a_table_that_matches_the_store() -> None: + pods = [ + _pod_summary("ns-a", "bench-0", phase="Running"), + _pod_summary("ns-a", "bench-1", phase="CrashLoopBackOff", ready="0/1", restarts=7), + ] + table = _StubTable( + { + "ns-a/bench-0": ["ns-a", "bench-0", "1/1", "Running", "0", "node-a"], + "ns-a/bench-1": ["ns-a", "bench-1", "0/1", "CrashLoopBackOff", "7", "node-a"], + } + ) + + replay_mod.check_rendered_rows(table, pods) + + assert table.row_count == 2 diff --git a/tests/performance/test_manifests.py b/tests/performance/test_manifests.py new file mode 100644 index 00000000..482c4f1d --- /dev/null +++ b/tests/performance/test_manifests.py @@ -0,0 +1,175 @@ +from __future__ import annotations + +from collections import Counter +from typing import Any + +import pytest + +from tests.performance.manifests import build_seed_manifests + + +def _labels(manifest: dict[str, object]) -> dict[str, str]: + metadata = manifest["metadata"] + assert isinstance(metadata, dict) + labels = metadata["labels"] + assert isinstance(labels, dict) + assert all(isinstance(key, str) and isinstance(value, str) for key, value in labels.items()) + return labels + + +def _namespace_name(manifest: dict[str, object]) -> str: + metadata = manifest["metadata"] + assert isinstance(metadata, dict) + name = metadata["name"] + assert isinstance(name, str) + return name + + +def _pod_names(manifests: tuple[dict[str, object], ...]) -> list[tuple[str, str]]: + rows: list[tuple[str, str]] = [] + for manifest in manifests: + metadata = manifest["metadata"] + assert isinstance(metadata, dict) + namespace = metadata["namespace"] + name = metadata["name"] + assert isinstance(namespace, str) + assert isinstance(name, str) + rows.append((namespace, name)) + return rows + + +def _pod_spec(manifest: dict[str, object]) -> dict[str, Any]: + spec = manifest["spec"] + assert isinstance(spec, dict) + return spec + + +def test_build_seed_manifests_returns_namespaces_then_pods_in_stable_order() -> None: + manifests = build_seed_manifests( + run_id="aks186", + namespace_count=2, + pods_per_namespace=3, + node_selector="korvid.dev/pool=perftest", + ) + + namespaces = manifests[:2] + pods = manifests[2:] + + assert [manifest["kind"] for manifest in namespaces] == ["Namespace", "Namespace"] + assert [_namespace_name(manifest) for manifest in namespaces] == [ + "korvid-perf-aks186-0", + "korvid-perf-aks186-1", + ] + assert _pod_names(pods) == [ + ("korvid-perf-aks186-0", "bench-0"), + ("korvid-perf-aks186-1", "bench-0"), + ("korvid-perf-aks186-0", "bench-1"), + ("korvid-perf-aks186-1", "bench-1"), + ("korvid-perf-aks186-0", "bench-2"), + ("korvid-perf-aks186-1", "bench-2"), + ] + assert all( + _labels(manifest) + == { + "app.kubernetes.io/managed-by": "korvid-performance", + "korvid.dev/performance-run": "aks186", + } + for manifest in manifests + ) + assert _pod_spec(pods[0])["nodeSelector"] == {"korvid.dev/pool": "perftest"} + assert _pod_spec(pods[0])["tolerations"] == [ + { + "key": "korvid.dev/performance", + "operator": "Equal", + "value": "true", + "effect": "NoSchedule", + } + ] + containers = _pod_spec(pods[0])["containers"] + assert containers == [ + { + "name": "bench", + "image": "registry.k8s.io/pause:3.10", + "resources": { + "requests": {"cpu": "5m", "memory": "16Mi"}, + }, + } + ] + + +def test_build_seed_manifests_spreads_standard_live_profile_evenly() -> None: + manifests = build_seed_manifests( + run_id="aks186", + namespace_count=20, + pods_per_namespace=50, + node_selector="korvid.dev/pool=perftest", + ) + + pods = manifests[20:] + counts = Counter(namespace for namespace, _name in _pod_names(pods)) + + assert len(manifests) == 1020 + assert len(counts) == 20 + assert set(counts.values()) == {50} + assert _pod_names(pods[:4]) == [ + ("korvid-perf-aks186-0", "bench-0"), + ("korvid-perf-aks186-1", "bench-0"), + ("korvid-perf-aks186-2", "bench-0"), + ("korvid-perf-aks186-3", "bench-0"), + ] + assert _pod_names(pods[-4:]) == [ + ("korvid-perf-aks186-16", "bench-49"), + ("korvid-perf-aks186-17", "bench-49"), + ("korvid-perf-aks186-18", "bench-49"), + ("korvid-perf-aks186-19", "bench-49"), + ] + + +@pytest.mark.parametrize( + "run_id", + ["", "Aks186", "-aks186", "aks186-", "aks_186", "a" * 49], +) +def test_build_seed_manifests_rejects_invalid_run_ids(run_id: str) -> None: + with pytest.raises( + ValueError, match="run_id must be 1-48 lowercase letters, digits, or hyphens" + ): + build_seed_manifests( + run_id=run_id, + namespace_count=1, + pods_per_namespace=1, + node_selector="korvid.dev/pool=perftest", + ) + + +@pytest.mark.parametrize("selector", ["", "pool", "=perftest", "pool=", "a=b=c", "pool = perftest"]) +def test_build_seed_manifests_rejects_malformed_node_selectors(selector: str) -> None: + with pytest.raises( + ValueError, match="node_selector must be exactly one non-empty key=value pair" + ): + build_seed_manifests( + run_id="aks186", + namespace_count=1, + pods_per_namespace=1, + node_selector=selector, + ) + + +@pytest.mark.parametrize( + ("namespace_count", "pods_per_namespace", "message"), + [ + (0, 1, "namespace_count must be a positive integer"), + (1, 0, "pods_per_namespace must be a positive integer"), + ], +) +def test_build_seed_manifests_rejects_non_positive_counts( + namespace_count: int, + pods_per_namespace: int, + message: str, +) -> None: + with pytest.raises(ValueError, match=message): + build_seed_manifests( + run_id="aks186", + namespace_count=namespace_count, + pods_per_namespace=pods_per_namespace, + node_selector="korvid.dev/pool=perftest", + ) diff --git a/tests/performance/test_metrics.py b/tests/performance/test_metrics.py new file mode 100644 index 00000000..62fc98e4 --- /dev/null +++ b/tests/performance/test_metrics.py @@ -0,0 +1,783 @@ +from __future__ import annotations + +import asyncio +import contextlib +import importlib +import json +from typing import Any, cast + +import pytest + +from korvid.k8s.telemetry import ReadTelemetryEvent +from tests.performance.metrics import ( + ApiSummary, + BenchmarkRecorder, + ChurnSummary, + LatencySummary, + ProcessSample, + ProcessSampler, + RunManifest, + render_markdown, + report_payload, +) + + +class _MemoryInfo: + def __init__(self, rss: int) -> None: + self.rss = rss + + +class _FakeProcess: + def __init__(self, cpu_values: tuple[float, ...], rss_values: tuple[int, ...]) -> None: + self.cpu_values = iter(cpu_values) + self.rss_values = iter(rss_values) + + def cpu_percent(self, interval: float | None = None) -> float: + return next(self.cpu_values) + + def memory_info(self) -> _MemoryInfo: + return _MemoryInfo(next(self.rss_values)) + + +def _metrics_module() -> Any: + return cast(Any, importlib.import_module("tests.performance.metrics")) + + +def _patch_sampler_runtime( + monkeypatch: pytest.MonkeyPatch, + *, + cpu_values: tuple[float, ...], + rss_values: tuple[int, ...], + tracing: bool, + python_bytes: tuple[int, ...], + strict_tracing: bool = False, +) -> tuple[dict[str, bool], list[str], Any]: + metrics = _metrics_module() + state = {"tracing": tracing} + lifecycle: list[str] = [] + blocker = asyncio.Event() + original_sleep = asyncio.sleep + traced_sizes = iter(python_bytes) + + async def _fake_sleep(_: float) -> None: + await blocker.wait() + + def _start() -> None: + lifecycle.append("start") + state["tracing"] = True + + def _stop() -> None: + lifecycle.append("stop") + state["tracing"] = False + + def _get_traced_memory() -> tuple[int, int]: + if strict_tracing and not state["tracing"]: + raise RuntimeError("tracemalloc not tracing") + return (next(traced_sizes), 0) + + monkeypatch.setattr(metrics.psutil, "Process", lambda: _FakeProcess(cpu_values, rss_values)) + monkeypatch.setattr(metrics.tracemalloc, "is_tracing", lambda: state["tracing"]) + monkeypatch.setattr(metrics.tracemalloc, "start", _start) + monkeypatch.setattr(metrics.tracemalloc, "stop", _stop) + monkeypatch.setattr(metrics.tracemalloc, "get_traced_memory", _get_traced_memory) + monkeypatch.setattr(metrics.asyncio, "sleep", _fake_sleep) + return state, lifecycle, original_sleep + + +def run_manifest() -> RunManifest: + return RunManifest( + profile_id="smoke-1k", + profile_hash="profile-hash", + korvid_sha="3cbe600996043cd6cc9df114a71d10c938545789", + python="3.12.10", + textual="3.2.0", + os="Darwin", + cpu_count=8, + memory_bytes=16 * 1024 * 1024 * 1024, + ) + + +def process_samples() -> tuple[ProcessSample, ...]: + mib = 1024 * 1024 + return ( + ProcessSample( + elapsed_seconds=0.0, cpu_percent=10.0, rss_bytes=100 * mib, python_bytes=5 * mib + ), + ProcessSample( + elapsed_seconds=60.0, cpu_percent=20.0, rss_bytes=101 * mib, python_bytes=6 * mib + ), + ProcessSample( + elapsed_seconds=120.0, cpu_percent=30.0, rss_bytes=102 * mib, python_bytes=7 * mib + ), + ) + + +def test_latency_summary_uses_nearest_rank_percentiles() -> None: + summary = LatencySummary.from_samples([0.50, 0.10, 0.20, 0.40, 0.30]) + + assert summary.count == 5 + assert summary.p50_seconds == pytest.approx(0.30) + assert summary.p95_seconds == pytest.approx(0.50) + assert summary.p99_seconds == pytest.approx(0.50) + assert summary.maximum_seconds == pytest.approx(0.50) + + +def test_latency_summary_is_empty_without_samples() -> None: + summary = LatencySummary.from_samples([]) + + assert summary.count == 0 + assert summary.p50_seconds is None + assert summary.p95_seconds is None + assert summary.p99_seconds is None + assert summary.maximum_seconds is None + + +def test_render_flushes_all_pending_events_as_coalesced() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 1.0) + recorder.record_event(2, 1.1) + recorder.record_render(1.2) + + report = recorder.report(run_manifest(), process_samples(), final_digest="abc") + + assert report.rendered_updates == 2 + assert report.render_passes == 1 + assert report.coalesced_updates == 1 + assert report.dropped_updates == 0 + assert report.event_to_render.count == 2 + assert report.event_to_render.p50_seconds == pytest.approx(0.1) + assert report.event_to_render.maximum_seconds == pytest.approx(0.2) + + +def test_report_marks_pending_events_as_dropped() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 1.0) + recorder.record_event(2, 1.1) + + report = recorder.report(run_manifest(), (), final_digest="abc") + + assert report.rendered_updates == 0 + assert report.render_passes == 0 + assert report.coalesced_updates == 0 + assert report.dropped_updates == 2 + assert report.event_to_render.count == 0 + + +def test_report_counts_api_operations_without_path_loss() -> None: + recorder = BenchmarkRecorder() + recorder.record_api( + ReadTelemetryEvent("list", "/api/v1/pods", object_count=1000, decoded_bytes=2048) + ) + recorder.record_api(ReadTelemetryEvent("watch_open", "/api/v1/pods")) + + payload = report_payload(recorder.report(run_manifest(), [], final_digest="abc")) + api = cast(dict[str, object], payload["api"]) + operations = cast(dict[str, int], api["operations"]) + paths = cast(dict[str, object], api["paths"]) + pod_path = cast(dict[str, int], paths["/api/v1/pods"]) + + assert operations == {"list": 1, "watch_open": 1} + assert pod_path["list"] == 1 + assert pod_path["watch_open"] == 1 + assert api["decoded_bytes"] == 2048 + assert api["object_count"] == 1000 + + +def test_api_summary_does_not_treat_repeated_lists_as_relists() -> None: + recorder = BenchmarkRecorder() + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + + report = recorder.report(run_manifest(), (), final_digest="abc") + + assert report.api.relists == 0 + assert report.api.operations == {"list": 2} + assert report.api.paths == {"/api/v1/pods": {"list": 2}} + + +def test_api_summary_counts_relist_only_after_410_then_list() -> None: + recorder = BenchmarkRecorder() + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=410)) + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + + report = recorder.report(run_manifest(), (), final_digest="abc") + + assert report.api.relists == 1 + assert report.api.operations == {"error": 1, "list": 2} + assert report.api.paths == {"/api/v1/pods": {"error": 1, "list": 2}} + + +def test_api_summary_mappings_are_immutable_and_payloads_are_copied() -> None: + recorder = BenchmarkRecorder() + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + recorder.record_api(ReadTelemetryEvent("watch_open", "/api/v1/pods")) + + report = recorder.report(run_manifest(), (), final_digest="abc") + operations = cast(Any, report.api.operations) + paths = cast(Any, report.api.paths) + + with pytest.raises(TypeError, match="does not support item assignment"): + operations["list"] = 99 + with pytest.raises(TypeError, match="does not support item assignment"): + paths["/api/v1/pods"]["watch_open"] = 99 + + first_payload = report_payload(report) + first_api = cast(dict[str, object], first_payload["api"]) + first_operations = cast(dict[str, int], first_api["operations"]) + first_paths = cast(dict[str, object], first_api["paths"]) + first_pod_path = cast(dict[str, int], first_paths["/api/v1/pods"]) + first_operations["list"] = 41 + first_pod_path["watch_open"] = 42 + + second_payload = report_payload(report) + second_api = cast(dict[str, object], second_payload["api"]) + second_operations = cast(dict[str, int], second_api["operations"]) + second_paths = cast(dict[str, object], second_api["paths"]) + second_pod_path = cast(dict[str, int], second_paths["/api/v1/pods"]) + + assert report.api.operations == {"list": 1, "watch_open": 1} + assert report.api.paths == {"/api/v1/pods": {"list": 1, "watch_open": 1}} + assert second_operations == {"list": 1, "watch_open": 1} + assert second_pod_path == {"list": 1, "watch_open": 1} + + +def test_report_payload_is_json_serializable_and_stable() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 1.0) + recorder.record_render(1.2) + recorder.record_input(0.05) + recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=429)) + + payload = report_payload( + recorder.report(run_manifest(), process_samples(), final_digest="digest-123") + ) + encoded = json.dumps(payload, sort_keys=True) + + assert payload == { + "manifest": { + "profile_id": "smoke-1k", + "profile_hash": "profile-hash", + "korvid_sha": "3cbe600996043cd6cc9df114a71d10c938545789", + "python": "3.12.10", + "textual": "3.2.0", + "os": "Darwin", + "cpu_count": 8, + "memory_bytes": 17179869184, + "context": None, + "cluster_id": None, + "kubernetes_version": None, + "node_pools": [], + }, + "latency": { + "event_to_render": { + "count": 1, + "p50_seconds": 0.19999999999999996, + "p95_seconds": 0.19999999999999996, + "p99_seconds": 0.19999999999999996, + "maximum_seconds": 0.19999999999999996, + }, + "input": { + "count": 1, + "p50_seconds": 0.05, + "p95_seconds": 0.05, + "p99_seconds": 0.05, + "maximum_seconds": 0.05, + }, + }, + "process": { + "sample_count": 3, + "cpu_percent_max": 30.0, + "rss_bytes_max": 106954752, + "python_bytes_max": 7340032, + "rss_slope_mib_per_minute": 1.0, + "rss_slope_warmup_boundary_seconds": 0.0, + "rss_slope_sample_count": 3, + }, + "phases": { + "process_start_to_interactive_seconds": None, + "list_to_populated_table_seconds": None, + "max_backlog_depth": 1, + "post_burst_drain_seconds": [], + "max_post_burst_drain_seconds": None, + }, + "api": { + "operations": {"error": 1}, + "paths": {"/api/v1/pods": {"error": 1}}, + "decoded_bytes": 0, + "object_count": 0, + "watch_events": 0, + "reconnects": 0, + "relists": 0, + "throttles": 1, + "authorization_failures": 0, + }, + "updates": { + "rendered_updates": 1, + "render_passes": 1, + "coalesced_updates": 0, + "dropped_updates": 0, + }, + "churn": None, + "failures_injected": {}, + "ui_scenarios": [], + "digests": {"expected": None, "final": "digest-123", "match": False}, + } + assert '"rss_slope_mib_per_minute": 1.0' in encoded + + +def test_render_markdown_uses_stable_labels() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 1.0) + recorder.record_render(1.1) + recorder.record_input(0.02) + recorder.record_api(ReadTelemetryEvent("watch_open", "/api/v1/pods")) + + text = render_markdown(recorder.report(run_manifest(), process_samples(), final_digest="abc")) + + assert "# Large-cluster benchmark report" in text + assert "- Profile ID: `smoke-1k`" in text + assert "- Event to render p95: `0.100s`" in text + assert "- Input latency p95: `0.020s`" in text + assert "- RSS slope: `1.00 MiB/min`" in text + assert "- Rendered updates: `1`" in text + assert "- watch_open: `1`" in text + assert "- Final digest: `abc`" in text + + +@pytest.mark.asyncio +async def test_process_sampler_rejects_double_start( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _, _, original_sleep = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0, 0.0), + rss_values=(100,), + tracing=True, + python_bytes=(1000,), + ) + + sampler = ProcessSampler(interval_seconds=0.01, clock=lambda: 10.0) + sampler.start() + await original_sleep(0) + try: + with pytest.raises(RuntimeError, match="already running"): + sampler.start() + finally: + await sampler.stop() + + +@pytest.mark.asyncio +async def test_process_sampler_rolls_back_tracemalloc_if_task_creation_fails( + monkeypatch: pytest.MonkeyPatch, +) -> None: + state, lifecycle, _ = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0,), + rss_values=(100,), + tracing=False, + python_bytes=(1000,), + ) + + def _fail_create_task(_: object) -> object: + raise RuntimeError("task creation failed") + + monkeypatch.setattr(_metrics_module().asyncio, "create_task", _fail_create_task) + + sampler = ProcessSampler(interval_seconds=0.01, clock=lambda: 10.0) + + with pytest.raises(RuntimeError, match="task creation failed"): + sampler.start() + + assert lifecycle == ["start", "stop"] + assert state["tracing"] is False + + +@pytest.mark.asyncio +async def test_process_sampler_keeps_owned_tracemalloc_until_last_overlapping_sampler_stops( + monkeypatch: pytest.MonkeyPatch, +) -> None: + state, lifecycle, original_sleep = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0, 12.5), + rss_values=(100,), + tracing=False, + python_bytes=(1000, 2000), + strict_tracing=True, + ) + + first = ProcessSampler(interval_seconds=0.01, clock=lambda: 11.0) + second = ProcessSampler(interval_seconds=0.01, clock=lambda: 12.0) + + first.start() + await original_sleep(0) + second.start() + await original_sleep(0) + + await first.stop() + + assert lifecycle == ["start"] + assert state["tracing"] is True + + await second.stop() + + assert lifecycle == ["start", "stop"] + assert state["tracing"] is False + + +@pytest.mark.asyncio +async def test_process_sampler_starts_and_stops_owned_tracemalloc( + monkeypatch: pytest.MonkeyPatch, +) -> None: + state, lifecycle, original_sleep = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0, 12.5), + rss_values=(100,), + tracing=False, + python_bytes=(1000,), + strict_tracing=True, + ) + + sampler = ProcessSampler(interval_seconds=0.01, clock=lambda: 11.0) + sampler.start() + await original_sleep(0) + samples = await sampler.stop() + + assert samples == ( + ProcessSample( + elapsed_seconds=0.0, + cpu_percent=12.5, + rss_bytes=100, + python_bytes=1000, + ), + ) + assert lifecycle == ["start", "stop"] + assert state["tracing"] is False + + +@pytest.mark.asyncio +async def test_process_sampler_preserves_preexisting_tracemalloc( + monkeypatch: pytest.MonkeyPatch, +) -> None: + state, lifecycle, original_sleep = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0, 8.0), + rss_values=(120,), + tracing=True, + python_bytes=(2000,), + ) + + sampler = ProcessSampler(interval_seconds=0.01, clock=lambda: 5.0) + sampler.start() + await original_sleep(0) + samples = await sampler.stop() + + assert samples == ( + ProcessSample( + elapsed_seconds=0.0, + cpu_percent=8.0, + rss_bytes=120, + python_bytes=2000, + ), + ) + assert lifecycle == [] + assert state["tracing"] is True + + +@pytest.mark.asyncio +async def test_process_sampler_skips_warmup_sample( + monkeypatch: pytest.MonkeyPatch, +) -> None: + times = iter((10.0, 11.0)) + _, _, original_sleep = _patch_sampler_runtime( + monkeypatch, + cpu_values=(0.0, 12.5, 15.0), + rss_values=(100, 110), + tracing=True, + python_bytes=(1000, 1200), + ) + + sampler = ProcessSampler(interval_seconds=0.01, clock=lambda: next(times)) + sampler.start() + await original_sleep(0) + samples = await sampler.stop() + + assert samples == ( + ProcessSample( + elapsed_seconds=1.0, + cpu_percent=12.5, + rss_bytes=100, + python_bytes=1000, + ), + ) + + +def test_recorder_exposes_public_pending_count_and_api_errors() -> None: + """Replay/live harnesses must observe the backlog and API errors through + a public accessor instead of reaching into `_pending_events`/`_api_events` + across module boundaries.""" + recorder = BenchmarkRecorder() + assert recorder.pending_count() == 0 + assert recorder.api_errors() == () + + recorder.record_event(1, 10.0) + recorder.record_event(2, 11.0) + assert recorder.pending_count() == 2 + + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=403)) + recorder.record_api(ReadTelemetryEvent("error", "/api/v1/pods", status=410)) + + assert [event.status for event in recorder.api_errors()] == [403, 410] + + recorder.record_render(12.0) + assert recorder.pending_count() == 0 + + +def test_report_separates_requested_churn_rate_from_achieved_rate() -> None: + """The design doc forbids reporting a requested rate as an achieved rate. + Both must be present, distinctly labelled, alongside the observed event + count, wall time, and the mutation-side throttle count (which is *not* the + application read path's `api.throttles`).""" + recorder = BenchmarkRecorder() + recorder.record_api(ReadTelemetryEvent("list", "/api/v1/pods")) + churn = ChurnSummary.from_observations( + requested_events=8400, + requested_duration_seconds=30, + observed_events=900, + wall_seconds=30.0, + mutation_throttles=7, + ) + + report = recorder.report(run_manifest(), (), final_digest="abc", churn=churn) + payload = cast(dict[str, object], report_payload(report)["churn"]) + markdown = render_markdown(report) + + assert report.churn is churn + assert churn.requested_events_per_second == 280.0 + assert churn.achieved_events_per_second == 30.0 + assert payload == { + "requested_events": 8400, + "requested_events_per_second": 280.0, + "observed_events": 900, + "wall_seconds": 30.0, + "achieved_events_per_second": 30.0, + "mutation_throttles": 7, + } + assert "Requested churn rate: `280.00 events/s`" in markdown + assert "Achieved churn rate: `30.00 events/s`" in markdown + assert "Mutation throttles (429): `7`" in markdown + # The application read path's throttle counter stays independent. + assert report.api.throttles == 0 + + +def test_report_payload_keeps_a_stable_churn_key_when_no_churn_was_driven() -> None: + recorder = BenchmarkRecorder() + report = recorder.report(run_manifest(), (), final_digest="abc") + + assert report.churn is None + assert report_payload(report)["churn"] is None + assert "Achieved churn rate: `n/a`" in render_markdown(report) + + +def test_churn_summary_reports_no_achieved_rate_without_elapsed_time() -> None: + churn = ChurnSummary.from_observations( + requested_events=10, + requested_duration_seconds=0, + observed_events=0, + wall_seconds=None, + mutation_throttles=0, + ) + + assert churn.achieved_events_per_second is None + assert churn.requested_events_per_second is None + + +def test_process_summary_fits_slope_only_over_post_warmup_samples() -> None: + """A steep startup allocation ramp followed by a flat steady state must + report a ~0 slope: only samples at/after the warm-up boundary count.""" + mib = 1024 * 1024 + samples = ( + # Warm-up: 100 -> 400 MiB while the initial table populates. + ProcessSample(elapsed_seconds=0.0, cpu_percent=1.0, rss_bytes=100 * mib, python_bytes=0), + ProcessSample(elapsed_seconds=2.0, cpu_percent=1.0, rss_bytes=250 * mib, python_bytes=0), + ProcessSample(elapsed_seconds=4.0, cpu_percent=1.0, rss_bytes=400 * mib, python_bytes=0), + # Steady state: flat at 400 MiB. + ProcessSample(elapsed_seconds=64.0, cpu_percent=1.0, rss_bytes=400 * mib, python_bytes=0), + ProcessSample(elapsed_seconds=124.0, cpu_percent=1.0, rss_bytes=400 * mib, python_bytes=0), + ) + + contaminated = _metrics_module().ProcessSummary.from_samples(samples) + steady = _metrics_module().ProcessSummary.from_samples(samples, warmup_boundary_seconds=4.0) + + assert contaminated.rss_slope_mib_per_minute is not None + assert contaminated.rss_slope_mib_per_minute > 10.0 # startup contaminates the fit + assert steady.rss_slope_mib_per_minute == pytest.approx(0.0) + assert steady.rss_slope_warmup_boundary_seconds == 4.0 + assert steady.rss_slope_sample_count == 3 + # The max/peak counters still consider every sample, not just steady state. + assert steady.rss_bytes_max == 400 * mib + + +def test_recorder_derives_warmup_boundary_from_lifecycle_marks() -> None: + """The slope warm-up boundary equals process-start-to-interactive, so a + single set of lifecycle marks drives both the phase summary and the slope + exclusion consistently.""" + mib = 1024 * 1024 + recorder = BenchmarkRecorder() + recorder.mark_process_start(1000.0) + recorder.mark_list_complete(1002.0) + recorder.mark_interactive(1005.0) # interactive 5s after start + samples = ( + ProcessSample(elapsed_seconds=0.0, cpu_percent=1.0, rss_bytes=100 * mib, python_bytes=0), + ProcessSample(elapsed_seconds=5.0, cpu_percent=1.0, rss_bytes=200 * mib, python_bytes=0), + ProcessSample(elapsed_seconds=65.0, cpu_percent=1.0, rss_bytes=200 * mib, python_bytes=0), + ) + + report = recorder.report(run_manifest(), samples, final_digest="d") + + assert report.process.rss_slope_warmup_boundary_seconds == 5.0 + assert report.process.rss_slope_sample_count == 2 + assert report.phases.process_start_to_interactive_seconds == 5.0 + assert report.phases.list_to_populated_table_seconds == 3.0 + + +def test_recorder_tracks_max_backlog_depth_and_post_burst_drain() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 10.0) + recorder.record_event(2, 10.1) + recorder.record_event(3, 10.2) + assert recorder.pending_count() == 3 + # A burst ends at t=10.3 while three events are still pending. + recorder.mark_burst_end(10.3) + recorder.record_render(11.0) # backlog drains to 0 at t=11.0 + + report = recorder.report(run_manifest(), (), final_digest="d") + + assert report.phases.max_backlog_depth == 3 + assert len(report.phases.post_burst_drain_seconds) == 1 + assert report.phases.post_burst_drain_seconds[0] == pytest.approx(0.7) + assert report.phases.max_post_burst_drain_seconds == pytest.approx(0.7) + + +def test_report_persists_expected_and_final_digest_with_match_flag() -> None: + recorder = BenchmarkRecorder() + match_report = recorder.report(run_manifest(), (), final_digest="abc", expected_digest="abc") + mismatch_report = recorder.report(run_manifest(), (), final_digest="abc", expected_digest="xyz") + + assert match_report.digest_match is True + assert mismatch_report.digest_match is False + + payload = cast(dict[str, object], report_payload(mismatch_report)["digests"]) + assert payload == {"expected": "xyz", "final": "abc", "match": False} + + markdown = render_markdown(mismatch_report) + assert "- Expected digest: `xyz`" in markdown + assert "- Final digest: `abc`" in markdown + assert "- Digest match: `false`" in markdown + + +def test_render_markdown_reports_phase_measurements() -> None: + recorder = BenchmarkRecorder() + recorder.mark_process_start(100.0) + recorder.mark_list_complete(101.0) + recorder.mark_interactive(102.0) + recorder.record_event(1, 200.0) + recorder.mark_burst_end(200.1) + recorder.record_render(200.5) + + markdown = render_markdown(recorder.report(run_manifest(), (), final_digest="d")) + + assert "## Phases" in markdown + assert "- Process start to interactive: `2.000s`" in markdown + assert "- LIST to populated table: `1.000s`" in markdown + assert "- Max backlog depth: `1`" in markdown + assert "- Max post-burst drain: `0.400s`" in markdown + + +def test_mark_burst_end_records_zero_when_the_backlog_is_already_empty() -> None: + """Nothing to drain is a 0.0 sample, not a pending marker. + + Left pending, the marker is resolved by the next unrelated steady-state + render and reported as that render's full latency, or lost entirely when no + later render arrives - either way the burst-drain budget is measured against + something that is not a drain. + """ + recorder = BenchmarkRecorder() + + recorder.mark_burst_end(10.0) + + report = recorder.report(run_manifest(), (), final_digest="d") + assert report.phases.post_burst_drain_seconds == (0.0,) + assert report.phases.max_post_burst_drain_seconds == 0.0 + + +def test_mark_burst_end_still_times_a_real_drain() -> None: + recorder = BenchmarkRecorder() + recorder.record_event(1, 5.0) + + recorder.mark_burst_end(10.0) + recorder.record_render(12.0) + + report = recorder.report(run_manifest(), (), final_digest="d") + assert report.phases.post_burst_drain_seconds == pytest.approx((2.0,)) + + +def test_reconnects_count_only_watch_reopens_after_an_error() -> None: + """A deliberate stop/start is not a reconnect. + + The live `namespace_switch` scenario scopes the table down and back, which + restarts the application watch twice on the same path; inferring reconnects + from `watch_open` counts alone reports a perfectly healthy run as having + two reconnects. + """ + summary = ApiSummary.from_events( + [ + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ] + ) + + assert summary.reconnects == 0 + assert summary.operations["watch_open"] == 3 + + +def test_reconnects_count_a_watch_reopened_after_a_dropped_stream() -> None: + summary = ApiSummary.from_events( + [ + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ReadTelemetryEvent("error", "/api/v1/pods", status=410), + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ReadTelemetryEvent("error", "/api/v1/pods", status=500), + ReadTelemetryEvent("watch_open", "/api/v1/pods"), + ] + ) + + assert summary.reconnects == 2 + + +async def test_sampler_stop_releases_tracing_even_when_the_task_failed() -> None: + """A sampling failure must not leak managed tracing state. + + `stop()` awaits the sampler task; if that raises, the release below never + runs and the caller's own cleanup (watch manager teardown) is skipped too. + """ + sampler = ProcessSampler(interval_seconds=0.001) + sampler.start() + + async def _boom() -> None: + raise RuntimeError("psutil exploded") + + running = sampler._task + assert running is not None + running.cancel() + with contextlib.suppress(asyncio.CancelledError, BaseException): + await running + sampler._task = asyncio.get_running_loop().create_task(_boom()) + await asyncio.sleep(0) + + with pytest.raises(RuntimeError, match="psutil exploded"): + await sampler.stop() + + assert sampler._uses_managed_tracing is False diff --git a/tests/performance/test_profile.py b/tests/performance/test_profile.py new file mode 100644 index 00000000..e35161f0 --- /dev/null +++ b/tests/performance/test_profile.py @@ -0,0 +1,173 @@ +import json +from dataclasses import replace +from pathlib import Path + +import pytest + +from tests.performance.profile import ( + Burst, + load_profile, + planned_event_count, + validate_profile, +) + + +def _write(tmp_path: Path, **overrides: object) -> Path: + payload: dict[str, object] = { + "schema_version": 1, + "id": "smoke", + "seed": 186, + "object_count": 1000, + "namespace_count": 20, + "steady_events_per_second": 20, + "duration_seconds": 5, + "bursts": [{"start_second": 2, "duration_seconds": 1, "events_per_second": 100}], + "failures": [{"kind": "gone", "at_event": 75}], + } + payload.update(overrides) + path = tmp_path / "profile.json" + path.write_text(json.dumps(payload)) + return path + + +def test_load_profile_is_strict_and_typed(tmp_path: Path) -> None: + profile = load_profile(_write(tmp_path)) + assert profile.id == "smoke" + assert profile.object_count == 1000 + assert profile.bursts[0].events_per_second == 100 + assert profile.failures[0].kind == "gone" + assert planned_event_count(profile) == 180 + + +def test_profile_rejects_unknown_keys(tmp_path: Path) -> None: + with pytest.raises(ValueError, match=r"unknown keys.*extra"): + load_profile(_write(tmp_path, extra=True)) + + +@pytest.mark.parametrize( + ("field", "value", "message"), + [ + ("schema_version", 2, "schema_version must be 1"), + ("object_count", 0, "object_count must be a positive integer"), + ("namespace_count", 21, "object_count must be divisible by namespace_count"), + ("steady_events_per_second", -1, "steady_events_per_second"), + ("duration_seconds", 0, "duration_seconds must be a positive integer"), + ], +) +def test_profile_rejects_invalid_values( + tmp_path: Path, field: str, value: object, message: str +) -> None: + with pytest.raises(ValueError, match=message): + load_profile(_write(tmp_path, **{field: value})) + + +def test_aks_1k_profile_pins_live_topology_and_reuses_burst_schedule() -> None: + """`aks-1k` is the *deterministic comparison* profile: the live topology on + `burst-50k`'s event schedule, so a live run can be scored against the + synthetic 1k/10k/50k baselines. The qualification run itself uses + `aks-live-1k` (see `test_aks_live_1k_profile_matches_the_published_live_plan`). + """ + profiles_dir = Path(__file__).with_name("profiles") + + profile = load_profile(profiles_dir / "aks-1k.json") + burst = load_profile(profiles_dir / "burst-50k.json") + + assert profile.schema_version == 1 + assert profile.id == "aks-1k" + assert profile.seed == 186 + assert profile.object_count == 1000 + assert profile.namespace_count == 20 + assert profile.steady_events_per_second == 200 + assert profile.duration_seconds == 30 + assert profile.bursts == ( + Burst(start_second=5, duration_seconds=1, events_per_second=1000), + Burst(start_second=15, duration_seconds=1, events_per_second=1000), + Burst(start_second=25, duration_seconds=1, events_per_second=1000), + ) + assert profile.failures == () + assert profile.steady_events_per_second == burst.steady_events_per_second + assert profile.duration_seconds == burst.duration_seconds + assert profile.bursts == burst.bursts + assert profile.failures == burst.failures + + +def test_validate_profile_rejects_burst_outside_shortened_duration(tmp_path: Path) -> None: + """`--duration` shortens a loaded profile with `dataclasses.replace`, which + bypasses `load_profile`'s burst containment check. `validate_profile` is the + shared re-check both paths call, so the shortened profile is rejected with a + clear operational message instead of failing later inside the generator.""" + profile = load_profile(_write(tmp_path)) + shortened = replace(profile, duration_seconds=2) + + with pytest.raises(ValueError, match="falls outside duration_seconds"): + validate_profile(shortened) + + +def test_validate_profile_rejects_failure_beyond_shortened_planned_events( + tmp_path: Path, +) -> None: + profile = load_profile(_write(tmp_path, bursts=[], failures=[{"kind": "gone", "at_event": 75}])) + shortened = replace(profile, duration_seconds=1) + + with pytest.raises(ValueError, match="failure at_event exceeds planned event count"): + validate_profile(shortened) + + +def test_validate_profile_accepts_a_duration_that_still_contains_every_burst( + tmp_path: Path, +) -> None: + profile = load_profile(_write(tmp_path, failures=[])) + + validate_profile(replace(profile, duration_seconds=3)) + + assert planned_event_count(replace(profile, duration_seconds=3)) == 140 + + +def test_aks_live_1k_profile_matches_the_published_live_plan() -> None: + """The design doc's live sequence is 30 minutes of churn at 20 events/s with + three 30-second bursts at 100 events/s; the live qualification profile must + encode exactly that, otherwise three published budgets are unmeasurable.""" + profile = load_profile(Path(__file__).with_name("profiles") / "aks-live-1k.json") + + assert profile.id == "aks-live-1k" + assert profile.seed == 186 + assert profile.object_count == 1000 + assert profile.namespace_count == 20 + assert profile.steady_events_per_second == 20 + assert profile.duration_seconds == 1800 + assert profile.bursts == ( + Burst(start_second=300, duration_seconds=30, events_per_second=100), + Burst(start_second=900, duration_seconds=30, events_per_second=100), + Burst(start_second=1500, duration_seconds=30, events_per_second=100), + ) + assert profile.failures == () + assert planned_event_count(profile) == 43200 + + +@pytest.mark.parametrize( + "kind", ["gone", "throttled", "forbidden", "slow", "metrics_unavailable", "slow_logs"] +) +def test_load_profile_accepts_every_versioned_failure_kind(tmp_path: Path, kind: str) -> None: + profile = load_profile(_write(tmp_path, failures=[{"kind": kind, "at_event": 10}])) + assert profile.failures[0].kind == kind + + +def test_load_profile_rejects_unknown_failure_kind(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="kind must be one of"): + load_profile(_write(tmp_path, failures=[{"kind": "meltdown", "at_event": 10}])) + + +def test_load_profile_rejects_duplicate_failure_event_positions(tmp_path: Path) -> None: + """`run_replay` indexes failures by `at_event`, so a duplicate position + silently drops every failure but one — while the profile hash and the + report still claim both were injected.""" + with pytest.raises(ValueError, match="at_event"): + load_profile( + _write( + tmp_path, + failures=[ + {"kind": "gone", "at_event": 10}, + {"kind": "throttled", "at_event": 10}, + ], + ) + ) diff --git a/tests/performance/test_replay.py b/tests/performance/test_replay.py new file mode 100644 index 00000000..d35e397c --- /dev/null +++ b/tests/performance/test_replay.py @@ -0,0 +1,530 @@ +"""Real-app replay harness tests (Task 5, issue #186). + +Drives the production KorvidApp/WatchManager/ResourceStore/ResourceTable +stack with a synthetic WorkloadProfile and asserts digest correctness, +update accounting, and API telemetry. +""" + +from __future__ import annotations + +import asyncio +from collections.abc import AsyncIterator +from time import monotonic + +import pytest + +from korvid.core.config import KorvidConfig +from korvid.core.store import ALL_NAMESPACES, ResourceStore, Summary +from korvid.core.watch import WatchManager +from korvid.ui.messages import ResourcesUpdated +from tests.performance.metrics import BenchmarkRecorder, RunManifest +from tests.performance.profile import Burst, FailureInjection, WorkloadProfile +from tests.performance.replay import ( + MeasuredKorvidApp, + ReplayAborted, + ReplayOptions, + build_manifest, + resolve_korvid_sha, + run_replay, +) +from tests.performance.workload import apply_events, initial_pods, scheduled_events, summary_digest + + +async def _never_watch(_kind: str, _scope: str) -> AsyncIterator[tuple[str, Summary]]: + """A watch source that never yields: the render-accounting test drives the + app directly and must not race a background stream.""" + await asyncio.Event().wait() + # Unreachable; present so the function is an async *generator*, which is + # what `WatchSource` requires. + yield ("ADDED", initial_pods(_manifest_profile())[0]) + + +def _manifest_profile() -> WorkloadProfile: + return WorkloadProfile( + schema_version=1, + id="render-accounting", + seed=1, + object_count=1, + namespace_count=1, + steady_events_per_second=0, + duration_seconds=1, + bursts=(), + failures=(), + ) + + +def _manifest_for_test() -> RunManifest: + return build_manifest(_manifest_profile()) + + +async def test_replay_uses_real_app_and_reaches_expected_digest() -> None: + profile = WorkloadProfile( + schema_version=1, + id="test", + seed=186, + object_count=100, + namespace_count=10, + steady_events_per_second=10, + duration_seconds=1, + bursts=(), + failures=(), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + events = scheduled_events(profile) + oracle = summary_digest(apply_events(initial_pods(profile), events)) + assert report.object_count == 100 + assert report.expected_digest == oracle + assert report.final_digest == oracle + assert report.dropped_updates == 0 + assert report.rendered_updates == 10 + assert report.input_latency.count > 0 + assert report.churn_started_before_input + assert report.api.operations["list"] == 1 + assert report.api.operations["watch_open"] == 1 + assert report.api.operations.get("get", 0) == 0 + + +async def test_replay_time_scale_1_uses_relative_inter_event_delays() -> None: + """time_scale=1 must use inter-event delays, not absolute offsets. + + Virtual-time seam: `monotonic_fn` returns a shared virtual clock and + `async_sleep` advances that clock then yields via `asyncio.sleep(0)`, + so the test completes in ~0 s of wall time regardless of profile length. + + Sensitivity: with 60 events at 20 eps over 3 s, correct sleeps sum exactly + to the final 2.95 s offset. The historical absolute-offset bug sums every + offset instead, while omitted sleeps sum to zero; both fail deterministically. + """ + profile = WorkloadProfile( + schema_version=1, + id="test-ts1", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=3, + bursts=(), + failures=(), + ) + virtual_time: list[float] = [0.0] + sleep_delays: list[float] = [] + + def virtual_monotonic() -> float: + return virtual_time[0] + + async def virtual_sleep(delay: float) -> None: + sleep_delays.append(delay) + virtual_time[0] += delay + await asyncio.sleep(0) # yield to event loop without real wall time + + report = await run_replay( + profile, + ReplayOptions(time_scale=1, monotonic_fn=virtual_monotonic, async_sleep=virtual_sleep), + ) + assert sum(sleep_delays) == pytest.approx(scheduled_events(profile)[-1].offset_seconds) + assert report.dropped_updates == 0 + assert report.object_count == 20 + assert report.expected_digest == report.final_digest + + +async def test_replay_gone_reconnects_and_digest_matches() -> None: + """gone at event 5 triggers one reconnect/re-LIST; final digest drops stale rows.""" + profile = WorkloadProfile( + schema_version=1, + id="test-gone", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="gone", at_event=5),), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + events = scheduled_events(profile) + # The gone failure event itself is not applied as a watch event; filter it + # from the apply_events oracle so it matches the actual replay outcome. + hard_failure_seqs = {f.at_event for f in profile.failures if f.kind != "slow"} + applied = tuple(e for e in events if e.sequence not in hard_failure_seqs) + oracle = summary_digest(apply_events(initial_pods(profile), applied)) + assert report.expected_digest == oracle + assert report.final_digest == oracle + assert report.dropped_updates == 0 + assert report.churn_started_before_input + assert report.api.operations["list"] == 2 + assert report.api.operations["watch_open"] == 2 + assert report.api.reconnects == 1 + assert report.api.relists == 1 + + +async def test_replay_gone_reconnects_with_time_scale_1() -> None: + """HTTP 410 reconnect with time_scale=1 must use elapsed-based delay, not absolute offsets. + + Virtual-time seam: same `monotonic_fn` / `async_sleep` pattern as + `test_replay_time_scale_1_uses_relative_inter_event_delays`. The shared + virtual clock is never reset across reconnect generations, so gen=1 events + correctly see the accumulated elapsed time from gen=0. + + Sensitivity: correct sleeps sum exactly to the final 4.95 s offset, and the + first post-410 sleep remains one 0.05 s tick. Resetting the replay origin on + reconnect produces a 0.25 s first reconnect sleep and 5.15 s total; omitted + sleeps produce zero. Both fail deterministically. + """ + profile = WorkloadProfile( + schema_version=1, + id="test-gone-ts1", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=5, + bursts=(), + failures=(FailureInjection(kind="gone", at_event=5),), + ) + virtual_time: list[float] = [0.0] + sleep_delays: list[float] = [] + + def virtual_monotonic() -> float: + return virtual_time[0] + + async def virtual_sleep(delay: float) -> None: + sleep_delays.append(delay) + virtual_time[0] += delay + await asyncio.sleep(0) # yield to event loop without real wall time + + report = await run_replay( + profile, + ReplayOptions(time_scale=1, monotonic_fn=virtual_monotonic, async_sleep=virtual_sleep), + ) + + events = scheduled_events(profile) + failure_sequence = profile.failures[0].at_event + first_post_reconnect_delay = ( + events[failure_sequence].offset_seconds - events[failure_sequence - 1].offset_seconds + ) + assert sum(sleep_delays) == pytest.approx(events[-1].offset_seconds) + assert sleep_delays[failure_sequence - 1] == pytest.approx(first_post_reconnect_delay) + assert report.expected_digest == report.final_digest + assert report.dropped_updates == 0 + assert report.api.operations["list"] == 2 + assert report.api.operations["watch_open"] == 2 + assert report.api.reconnects == 1 + assert report.api.relists == 1 + + +async def test_replay_throttled_reconnects_and_digest_matches() -> None: + """A 429 ends one watch connection; `WatchManager` retries, the source + re-LISTs from its tracked state, and the run still reaches the oracle + digest with zero drops. The throttled event itself is never delivered.""" + profile = WorkloadProfile( + schema_version=1, + id="test-throttled", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="throttled", at_event=5),), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + events = scheduled_events(profile) + applied = tuple(e for e in events if e.sequence != 5) + oracle = summary_digest(apply_events(initial_pods(profile), applied)) + + assert report.expected_digest == oracle + assert report.final_digest == oracle + assert report.dropped_updates == 0 + assert report.api.operations["list"] == 2 + assert report.api.operations["watch_open"] == 2 + assert report.api.reconnects == 1 + assert report.api.throttles == 1 + # A 429 is not a 410: it must not be counted as a re-LIST recovery. + assert report.api.relists == 0 + + +async def test_replay_slow_delays_without_dropping_or_reconnecting() -> None: + """`slow` delays one event by one steady-rate tick and still delivers it: + no reconnect, no drop, and the stall is real extra time. + + The stall is injected at the *last* scheduled event on purpose. Mid-run the + absolute-offset schedule silently absorbs a one-tick stall (the following + event's delay simply shrinks by the same tick), so only a stall with no + remaining schedule to catch up in is observable as extra virtual time: + correct behaviour totals `last offset + one tick`, while ignoring or + dropping the `slow` injection totals exactly `last offset`. + """ + profile = WorkloadProfile( + schema_version=1, + id="test-slow", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="slow", at_event=40),), + ) + virtual_time: list[float] = [0.0] + sleep_delays: list[float] = [] + + def virtual_monotonic() -> float: + return virtual_time[0] + + async def virtual_sleep(delay: float) -> None: + sleep_delays.append(delay) + virtual_time[0] += delay + await asyncio.sleep(0) + + report = await run_replay( + profile, + ReplayOptions(time_scale=1, monotonic_fn=virtual_monotonic, async_sleep=virtual_sleep), + ) + events = scheduled_events(profile) + oracle = summary_digest(apply_events(initial_pods(profile), events)) + + assert report.expected_digest == oracle + assert report.final_digest == oracle + assert report.dropped_updates == 0 + assert report.api.operations["watch_open"] == 1 + assert report.api.reconnects == 0 + # The injected 1/20s stall is an *extra* sleep on top of the schedule. + assert sum(sleep_delays) == pytest.approx(events[-1].offset_seconds + 1 / 20) + + +async def test_replay_forbidden_aborts_with_an_explicit_terminal_error() -> None: + """403 is an authorization boundary: `WatchManager` never reconnects and + clears the store, so the run can never complete. That must surface at once + as a named terminal failure instead of a 30-second `until` timeout on a + permanently empty backlog.""" + profile = WorkloadProfile( + schema_version=1, + id="test-forbidden", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="forbidden", at_event=5),), + ) + with pytest.raises(ReplayAborted, match="403"): + await run_replay(profile, ReplayOptions(time_scale=0)) + + +async def test_replay_churn_started_before_input_is_false_without_any_events() -> None: + """The flag must be a real emitted-event signal: a profile that schedules + no churn at all cannot claim churn was active during input measurement.""" + profile = WorkloadProfile( + schema_version=1, + id="test-no-churn", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=0, + duration_seconds=1, + bursts=(), + failures=(), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + + assert scheduled_events(profile) == () + assert not report.churn_started_before_input + assert report.input_latency.count > 0 + + +async def test_measured_app_counts_only_resource_update_renders() -> None: + """`_render_table` is also called by cursor/filter/sort/split-pane paths. + Counting those inflates `render_passes` and lets an unrelated repaint flush + the pending-event backlog, so only store-driven renders may be recorded.""" + recorder = BenchmarkRecorder() + app = MeasuredKorvidApp( + config=KorvidConfig(namespace=ALL_NAMESPACES), + store=ResourceStore(), + watch_manager=WatchManager(ResourceStore(), _never_watch, retry_delay=0.0), + recorder=recorder, + ) + async with app.run_test(): + recorder.record_event(1, monotonic()) + app._render_table("pods") + assert recorder.pending_count() == 1 + + app.on_resources_updated(ResourcesUpdated("pods")) + assert recorder.pending_count() == 0 + + report = recorder.report(_manifest_for_test(), (), final_digest="d") + assert report.render_passes == 1 + assert report.rendered_updates == 1 + + +async def test_replay_measures_list_phase_separately_from_watch_events() -> None: + """Initial LIST rows must not be counted as event-to-render samples; they + are timed as a separate LIST-to-populated-table startup phase, so replay + p95 is comparable with the live watch-only event-to-render metric.""" + profile = WorkloadProfile( + schema_version=1, + id="list-sep", + seed=186, + object_count=100, + namespace_count=10, + steady_events_per_second=10, + duration_seconds=1, + bursts=(), + failures=(), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + + # 10 scheduled watch events, and *only* those, are event-to-render samples. + assert len(scheduled_events(profile)) == 10 + assert report.event_to_render.count == 10 + assert report.rendered_updates == 10 + + # The LIST-to-populated-table and startup phases are measured explicitly. + assert report.phases.list_to_populated_table_seconds is not None + assert report.phases.list_to_populated_table_seconds >= 0.0 + assert report.phases.process_start_to_interactive_seconds is not None + assert report.phases.process_start_to_interactive_seconds >= 0.0 + + +async def test_replay_records_post_burst_drain_and_backlog_depth() -> None: + """A burst produces a measurable backlog and a post-burst drain sample.""" + profile = WorkloadProfile( + schema_version=1, + id="burst-drain", + seed=186, + object_count=50, + namespace_count=5, + steady_events_per_second=5, + duration_seconds=3, + bursts=(Burst(start_second=1, duration_seconds=1, events_per_second=40),), + failures=(), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + + assert report.phases.max_backlog_depth >= 1 + assert report.phases.post_burst_drain_seconds != () + assert report.phases.max_post_burst_drain_seconds is not None + + +def test_resolve_korvid_sha_prefers_github_sha_then_git_head() -> None: + sha = "a" * 40 + other = "b" * 40 + assert resolve_korvid_sha(env={"GITHUB_SHA": sha}, git_head=lambda: other) == sha + assert resolve_korvid_sha(env={}, git_head=lambda: other) == other + # A non-immutable / missing value resolves to None rather than a fake SHA. + assert resolve_korvid_sha(env={"GITHUB_SHA": "dev"}, git_head=lambda: None) is None + + +def test_build_manifest_records_resolved_sha() -> None: + sha = "c" * 40 + manifest = build_manifest(_manifest_profile(), korvid_sha=sha) + assert manifest.korvid_sha == sha + + +def test_build_manifest_marks_unresolved_offline_sha_as_unknown( + monkeypatch: pytest.MonkeyPatch, +) -> None: + import tests.performance.replay as replay_module + + monkeypatch.setattr(replay_module, "resolve_korvid_sha", lambda: None) + manifest = build_manifest(_manifest_profile()) + assert manifest.korvid_sha == "unknown" + + +async def test_replay_metrics_unavailable_keeps_resource_navigation_healthy() -> None: + """`metrics_unavailable` records evidence on the metrics read path while the + resource watch/render reaches the oracle digest with zero drops - proving + navigation is independent of the metrics poller.""" + profile = WorkloadProfile( + schema_version=1, + id="metrics-unavail", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="metrics_unavailable", at_event=5),), + ) + report = await run_replay(profile, ReplayOptions(time_scale=0)) + events = scheduled_events(profile) + oracle = summary_digest(apply_events(initial_pods(profile), events)) + + # The failing event is still delivered (not a hard fault), so no filtering. + assert report.expected_digest == oracle + assert report.final_digest == oracle + assert report.dropped_updates == 0 + assert report.api.reconnects == 0 + assert report.failures_injected["metrics_unavailable"] == 1 + # Evidence lands on the metrics read path, never the pods path. + assert "/apis/metrics.k8s.io/v1beta1/pods" in report.api.paths + + +async def test_replay_slow_logs_do_not_block_resource_progress() -> None: + """`slow_logs` records evidence on the log read path and adds no delay to + the resource schedule - resource watch/render progress is independent of log + consumption.""" + profile = WorkloadProfile( + schema_version=1, + id="slow-logs", + seed=186, + object_count=20, + namespace_count=4, + steady_events_per_second=20, + duration_seconds=2, + bursts=(), + failures=(FailureInjection(kind="slow_logs", at_event=40),), + ) + virtual_time: list[float] = [0.0] + sleep_delays: list[float] = [] + + def virtual_monotonic() -> float: + return virtual_time[0] + + async def virtual_sleep(delay: float) -> None: + sleep_delays.append(delay) + virtual_time[0] += delay + await asyncio.sleep(0) + + report = await run_replay( + profile, + ReplayOptions(time_scale=1, monotonic_fn=virtual_monotonic, async_sleep=virtual_sleep), + ) + events = scheduled_events(profile) + oracle = summary_digest(apply_events(initial_pods(profile), events)) + + assert report.expected_digest == oracle + assert report.final_digest == oracle + assert report.dropped_updates == 0 + assert report.api.reconnects == 0 + assert report.failures_injected["slow_logs"] == 1 + # A slow log stream adds no extra sleep to the resource schedule. + assert sum(sleep_delays) == pytest.approx(events[-1].offset_seconds) + + +async def test_replay_does_not_re_mark_bursts_after_a_watch_reconnect() -> None: + """A reconnect must not replay burst boundaries that already passed. + + `_ReplaySource` restarts at `_next_event_index` on a new generation. With + the burst cursor reset to zero, the first event after a 410 re-marks every + burst that already ended, producing duplicate and time-shifted drain + samples for a run that had exactly one burst. + """ + profile = WorkloadProfile( + schema_version=1, + id="burst-reconnect", + seed=186, + object_count=50, + namespace_count=5, + steady_events_per_second=5, + duration_seconds=4, + bursts=(Burst(start_second=1, duration_seconds=1, events_per_second=40),), + # A 410 forces the watch manager to drop and re-list mid-schedule, + # after the burst window has already closed. + failures=(FailureInjection(kind="gone", at_event=50),), + ) + + report = await run_replay(profile, ReplayOptions(time_scale=0)) + + assert report.api.reconnects == 1 # the schedule really was interrupted + assert len(report.phases.post_burst_drain_seconds) == len(profile.bursts) diff --git a/tests/performance/test_workload.py b/tests/performance/test_workload.py new file mode 100644 index 00000000..8cbe5b3e --- /dev/null +++ b/tests/performance/test_workload.py @@ -0,0 +1,63 @@ +from dataclasses import replace +from itertools import pairwise + +from tests.performance.profile import WorkloadProfile +from tests.performance.workload import ( + apply_events, + event_digest, + initial_pods, + scheduled_events, + summary_digest, +) + + +def _profile(seed: int = 186) -> WorkloadProfile: + return WorkloadProfile( + schema_version=1, + id="test", + seed=seed, + object_count=100, + namespace_count=10, + steady_events_per_second=10, + duration_seconds=2, + bursts=(), + failures=(), + ) + + +def test_same_seed_produces_identical_hashes() -> None: + first = _profile() + second = replace(first) + assert summary_digest(initial_pods(first)) == summary_digest(initial_pods(second)) + assert event_digest(scheduled_events(first)) == event_digest(scheduled_events(second)) + + +def test_different_seed_changes_event_hash_not_initial_hash() -> None: + first = _profile(186) + second = _profile(187) + assert summary_digest(initial_pods(first)) == summary_digest(initial_pods(second)) + assert event_digest(scheduled_events(first)) != event_digest(scheduled_events(second)) + + +def test_events_are_stably_scheduled_and_change_final_digest() -> None: + profile = _profile() + initial = initial_pods(profile) + events = scheduled_events(profile) + assert len(events) == 20 + assert [event.sequence for event in events] == list(range(1, 21)) + assert all(left.offset_seconds <= right.offset_seconds for left, right in pairwise(events)) + assert summary_digest(apply_events(initial, events)) != summary_digest(initial) + + +def test_scheduled_events_carry_the_selected_object_index() -> None: + """The live harness maps each churn event onto a real cluster Pod. Carrying + the generator's own object index removes the need to parse it back out of + the synthetic Pod name (`int(name.removeprefix("pod-"))`), which would + become a mid-churn `ValueError` if the naming scheme ever changed.""" + profile = _profile() + events = scheduled_events(profile) + + assert events + for event in events: + assert 0 <= event.object_index < profile.object_count + assert event.summary.name == f"pod-{event.object_index:06d}" diff --git a/tests/performance/workload.py b/tests/performance/workload.py new file mode 100644 index 00000000..c3a8a9dc --- /dev/null +++ b/tests/performance/workload.py @@ -0,0 +1,103 @@ +from __future__ import annotations + +import hashlib +import json +import random +from collections.abc import Iterable +from dataclasses import asdict, dataclass, replace + +from korvid.k8s.models import PodSummary +from tests.performance.profile import WorkloadProfile, planned_event_count + + +@dataclass(frozen=True) +class ScheduledEvent: + sequence: int + offset_seconds: float + event_type: str + summary: PodSummary + #: Index of the object the generator selected, in `initial_pods` order. + #: Carried explicitly so live replay can map the event onto its seeded + #: `(namespace, name)` identity without parsing the synthetic Pod name. + object_index: int + + +def initial_pods(profile: WorkloadProfile) -> tuple[PodSummary, ...]: + return tuple( + PodSummary( + name=f"pod-{index:06d}", + namespace=f"bench-{index % profile.namespace_count:04d}", + phase="Running", + ready="1/1", + restarts=0, + node=f"node-{index % 5:02d}", + ) + for index in range(profile.object_count) + ) + + +def _rate_at(profile: WorkloadProfile, second: float) -> int: + for burst in profile.bursts: + if burst.start_second <= second < burst.start_second + burst.duration_seconds: + return burst.events_per_second + return profile.steady_events_per_second + + +def scheduled_events(profile: WorkloadProfile) -> tuple[ScheduledEvent, ...]: + rng = random.Random(profile.seed) + current = list(initial_pods(profile)) + result: list[ScheduledEvent] = [] + sequence = 0 + for second in range(profile.duration_seconds): + rate = _rate_at(profile, float(second)) + if rate <= 0: + continue + for tick in range(rate): + sequence += 1 + index = rng.randrange(profile.object_count) + old = current[index] + updated = replace( + old, + phase="Pending" if old.phase == "Running" else "Running", + ready="0/1" if old.ready == "1/1" else "1/1", + restarts=old.restarts + 1, + ) + current[index] = updated + result.append( + ScheduledEvent( + sequence=sequence, + offset_seconds=second + tick / rate, + event_type="MODIFIED", + summary=updated, + object_index=index, + ) + ) + assert len(result) == planned_event_count(profile) + return tuple(result) + + +def _hash(payload: object) -> str: + encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str).encode() + return hashlib.sha256(encoded).hexdigest() + + +def summary_digest(summaries: Iterable[PodSummary]) -> str: + ordered = sorted(summaries, key=lambda pod: (pod.namespace, pod.name)) + return _hash([asdict(pod) for pod in ordered]) + + +def event_digest(events: Iterable[ScheduledEvent]) -> str: + return _hash([asdict(event) for event in events]) + + +def apply_events( + initial: Iterable[PodSummary], events: Iterable[ScheduledEvent] +) -> tuple[PodSummary, ...]: + current = {f"{pod.namespace}/{pod.name}": pod for pod in initial} + for event in events: + key = f"{event.summary.namespace}/{event.summary.name}" + if event.event_type == "DELETED": + current.pop(key, None) + else: + current[key] = event.summary + return tuple(sorted(current.values(), key=lambda pod: (pod.namespace, pod.name))) diff --git a/tests/ui/test_node_shell.py b/tests/ui/test_node_shell.py index 9b1fafaa..7593edfa 100644 --- a/tests/ui/test_node_shell.py +++ b/tests/ui/test_node_shell.py @@ -197,6 +197,25 @@ def fake_call(argv): # type: ignore[no-untyped-def] # test helper yield call_records +async def _await_node_shell_outcome(pilot: Any, audit_path: Path) -> None: + """Wait until the node-shell *outcome* entry has been appended. + + Every outcome is written after its user-facing notification, on a separate + `asyncio.to_thread` hop. Waiting on the notification alone races the audit + write against app teardown, leaving only the `intent` entry behind on a + loaded run. + """ + + def _written() -> bool: + if not audit_path.exists(): + return False + records = [json.loads(ln) for ln in audit_path.read_text().splitlines()] + shells = [e for e in records if e["action"] == "node-shell"] + return bool(shells) and shells[-1]["outcome"] != "intent" + + await until(pilot, _written, label="node-shell outcome audited") + + async def test_s_on_nodes_view_opens_privileged_approval_dialog(tmp_path: Path) -> None: rec = DeleteRecorder() app = make_app(rec, tmp_path / "audit.jsonl") @@ -283,10 +302,8 @@ async def test_node_shell_cleanup_failure_warns_and_audits(tmp_path: Path) -> No await until(pilot, lambda: isinstance(app.screen, ConfirmScreen), label="dialog") await pilot.press("y") - def _warned() -> bool: - return any(DBG_POD in n.message for n in app._notifications) - - await until(pilot, _warned, label="cleanup failure notification") + await _await_node_shell_outcome(pilot, audit_path) + assert any(DBG_POD in n.message for n in app._notifications) entries = [json.loads(ln) for ln in audit_path.read_text().splitlines()] last = [e for e in entries if e["action"] == "node-shell"][-1] assert f"cleanup failed for: {DBG_POD}" in last["outcome"] @@ -370,10 +387,8 @@ async def test_node_shell_create_failure_warns_about_policy(tmp_path: Path) -> N await until(pilot, lambda: isinstance(app.screen, ConfirmScreen), label="dialog") await pilot.press("y") - def _warned() -> bool: - return any("PodSecurity" in n.message for n in app._notifications) - - await until(pilot, _warned, label="policy hint notification") + await _await_node_shell_outcome(pilot, audit_path) + assert any("PodSecurity" in n.message for n in app._notifications) assert call_records == [] assert rec.deletes == [] entries = [json.loads(ln) for ln in audit_path.read_text().splitlines()] @@ -399,10 +414,8 @@ async def test_node_shell_unidentifiable_create_output_aborts(tmp_path: Path) -> await until(pilot, lambda: isinstance(app.screen, ConfirmScreen), label="dialog") await pilot.press("y") - def _warned() -> bool: - return any("did not report" in n.message for n in app._notifications) - - await until(pilot, _warned, label="unidentifiable-pod warning") + await _await_node_shell_outcome(pilot, audit_path) + assert any("did not report" in n.message for n in app._notifications) assert call_records == [] assert rec.deletes == [] entries = [json.loads(ln) for ln in audit_path.read_text().splitlines()] @@ -567,6 +580,7 @@ async def test_node_shell_nonzero_attach_exit_has_no_policy_hint(tmp_path: Path) await until(pilot, lambda: isinstance(app.screen, ConfirmScreen), label="dialog") await pilot.press("y") await until(pilot, lambda: rec.deletes, label="debug pod cleanup") + await _await_node_shell_outcome(pilot, audit_path) assert call_records != [] assert not any("PodSecurity" in n.message for n in app._notifications) entries = [json.loads(ln) for ln in audit_path.read_text().splitlines()] @@ -617,7 +631,8 @@ async def test_node_shell_create_without_uid_aborts(tmp_path: Path) -> None: without it the cleanup delete would lose its uid precondition and could remove a same-name replacement pod.""" rec = DeleteRecorder() - app = make_app(rec, tmp_path / "audit.jsonl") + audit_path = tmp_path / "audit.jsonl" + app = make_app(rec, audit_path) run_fake, _ = _kubectl_run( get_result=SimpleNamespace( returncode=0, @@ -633,10 +648,8 @@ async def test_node_shell_create_without_uid_aborts(tmp_path: Path) -> None: await until(pilot, lambda: isinstance(app.screen, ConfirmScreen), label="dialog") await pilot.press("y") - def _warned() -> bool: - return any("uid" in n.message for n in app._notifications) - - await until(pilot, _warned, label="missing-uid warning") + await _await_node_shell_outcome(pilot, audit_path) + assert any("uid" in n.message for n in app._notifications) assert call_records == [] assert rec.deletes == [] diff --git a/tests/ui/test_waits.py b/tests/ui/test_waits.py index 7ed28d86..d8e71505 100644 --- a/tests/ui/test_waits.py +++ b/tests/ui/test_waits.py @@ -4,7 +4,7 @@ import pytest -from .waits import until +from .waits import WaitTimeout, until class _FakePilot: @@ -36,7 +36,7 @@ async def test_until_pauses_cover_the_full_requested_timeout() -> None: """`int(timeout / 0.05)` truncation must not shorten the wait: the pauses add up to the advertised timeout, with a shorter final step.""" pilot = _FakePilot() - with pytest.raises(AssertionError, match=r"condition not met within 0\.12s"): + with pytest.raises(WaitTimeout, match=r"condition not met within 0\.12s"): await until(pilot, lambda: False, timeout=0.12) assert sum(pilot.pauses) == pytest.approx(0.12) @@ -44,12 +44,22 @@ async def test_until_pauses_cover_the_full_requested_timeout() -> None: async def test_until_sub_interval_timeout_still_pauses_once() -> None: """A timeout below one 50ms tick still yields to the app once.""" pilot = _FakePilot() - with pytest.raises(AssertionError, match=r"condition not met within 0\.03s"): + with pytest.raises(WaitTimeout, match=r"condition not met within 0\.03s"): await until(pilot, lambda: False, timeout=0.03) assert pilot.pauses == [pytest.approx(0.03)] async def test_until_raises_with_label_on_timeout() -> None: pilot = _FakePilot() - with pytest.raises(AssertionError, match=r"dialog visible not met within 0\.1s"): + with pytest.raises(WaitTimeout, match=r"dialog visible not met within 0\.1s"): await until(pilot, lambda: False, timeout=0.1, label="dialog visible") + + +async def test_until_timeout_is_not_an_assertion_error() -> None: + """A wait timeout is an operational outcome, not a programmer `assert`: + callers (e.g. the benchmark CLI) must be able to catch it without also + swallowing genuine `AssertionError`s raised by production code.""" + pilot = _FakePilot() + with pytest.raises(WaitTimeout) as caught: + await until(pilot, lambda: False, timeout=0.05, label="never") + assert not isinstance(caught.value, AssertionError) diff --git a/tests/ui/waits.py b/tests/ui/waits.py index 9a53f251..cffad631 100644 --- a/tests/ui/waits.py +++ b/tests/ui/waits.py @@ -10,6 +10,16 @@ from typing import Any +class WaitTimeout(Exception): + """Raised when `until` gives up waiting for a condition. + + Deliberately *not* an `AssertionError`: a wait timeout is an operational + outcome a caller may catch and report (e.g. the benchmark CLI turns it + into exit 1), while a genuine `assert` inside production code must keep + propagating as the programmer error it is. + """ + + async def until( pilot: Any, cond: Callable[[], object], @@ -23,6 +33,9 @@ async def until( condition is re-checked once after the final pause so it cannot fail on an outcome that arrived during the last tick, and `label` names the awaited outcome in the failure message for easier CI diagnosis. + + Raises: + WaitTimeout: `cond()` was still falsy after `timeout` seconds. """ remaining = timeout while remaining > 0: @@ -33,4 +46,4 @@ async def until( remaining -= step if cond(): return - raise AssertionError(f"{label} not met within {timeout}s") + raise WaitTimeout(f"{label} not met within {timeout}s") diff --git a/uv.lock b/uv.lock index b9e9e0b6..e2c63642 100644 --- a/uv.lock +++ b/uv.lock @@ -1017,6 +1017,7 @@ dev = [ { name = "deptry" }, { name = "mypy" }, { name = "pre-commit" }, + { name = "psutil" }, { name = "pytest" }, { name = "pytest-asyncio" }, { name = "pytest-cov" }, @@ -1052,6 +1053,7 @@ dev = [ { name = "deptry", specifier = ">=0.20" }, { name = "mypy", specifier = ">=1.14" }, { name = "pre-commit", specifier = ">=4" }, + { name = "psutil", specifier = ">=6.1" }, { name = "pytest", specifier = ">=8" }, { name = "pytest-asyncio", specifier = ">=0.24" }, { name = "pytest-cov", specifier = ">=5" }, @@ -1636,6 +1638,34 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3a/ed/1cdcab6ba3d6ab7feca11fc14f0eeea80755bb53ef4e892079f31b10a25f/propcache-0.5.2-py3-none-any.whl", hash = "sha256:be1ddfcbb376e3de5d2e2db1d58d6d67463e6b4f9f040c000de8e300295465fe", size = 14036, upload-time = "2026-05-08T21:02:10.673Z" }, ] +[[package]] +name = "psutil" +version = "7.2.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372", size = 493740, upload-time = "2026-01-28T18:14:54.428Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/08/510cbdb69c25a96f4ae523f733cdc963ae654904e8db864c07585ef99875/psutil-7.2.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:2edccc433cbfa046b980b0df0171cd25bcaeb3a68fe9022db0979e7aa74a826b", size = 130595, upload-time = "2026-01-28T18:14:57.293Z" }, + { url = "https://files.pythonhosted.org/packages/d6/f5/97baea3fe7a5a9af7436301f85490905379b1c6f2dd51fe3ecf24b4c5fbf/psutil-7.2.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e78c8603dcd9a04c7364f1a3e670cea95d51ee865e4efb3556a3a63adef958ea", size = 131082, upload-time = "2026-01-28T18:14:59.732Z" }, + { url = "https://files.pythonhosted.org/packages/37/d6/246513fbf9fa174af531f28412297dd05241d97a75911ac8febefa1a53c6/psutil-7.2.2-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1a571f2330c966c62aeda00dd24620425d4b0cc86881c89861fbc04549e5dc63", size = 181476, upload-time = "2026-01-28T18:15:01.884Z" }, + { url = "https://files.pythonhosted.org/packages/b8/b5/9182c9af3836cca61696dabe4fd1304e17bc56cb62f17439e1154f225dd3/psutil-7.2.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:917e891983ca3c1887b4ef36447b1e0873e70c933afc831c6b6da078ba474312", size = 184062, upload-time = "2026-01-28T18:15:04.436Z" }, + { url = "https://files.pythonhosted.org/packages/16/ba/0756dca669f5a9300d0cbcbfae9a4c30e446dfc7440ffe43ded5724bfd93/psutil-7.2.2-cp313-cp313t-win_amd64.whl", hash = "sha256:ab486563df44c17f5173621c7b198955bd6b613fb87c71c161f827d3fb149a9b", size = 139893, upload-time = "2026-01-28T18:15:06.378Z" }, + { url = "https://files.pythonhosted.org/packages/1c/61/8fa0e26f33623b49949346de05ec1ddaad02ed8ba64af45f40a147dbfa97/psutil-7.2.2-cp313-cp313t-win_arm64.whl", hash = "sha256:ae0aefdd8796a7737eccea863f80f81e468a1e4cf14d926bd9b6f5f2d5f90ca9", size = 135589, upload-time = "2026-01-28T18:15:08.03Z" }, + { url = "https://files.pythonhosted.org/packages/81/69/ef179ab5ca24f32acc1dac0c247fd6a13b501fd5534dbae0e05a1c48b66d/psutil-7.2.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:eed63d3b4d62449571547b60578c5b2c4bcccc5387148db46e0c2313dad0ee00", size = 130664, upload-time = "2026-01-28T18:15:09.469Z" }, + { url = "https://files.pythonhosted.org/packages/7b/64/665248b557a236d3fa9efc378d60d95ef56dd0a490c2cd37dafc7660d4a9/psutil-7.2.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7b6d09433a10592ce39b13d7be5a54fbac1d1228ed29abc880fb23df7cb694c9", size = 131087, upload-time = "2026-01-28T18:15:11.724Z" }, + { url = "https://files.pythonhosted.org/packages/d5/2e/e6782744700d6759ebce3043dcfa661fb61e2fb752b91cdeae9af12c2178/psutil-7.2.2-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fa4ecf83bcdf6e6c8f4449aff98eefb5d0604bf88cb883d7da3d8d2d909546a", size = 182383, upload-time = "2026-01-28T18:15:13.445Z" }, + { url = "https://files.pythonhosted.org/packages/57/49/0a41cefd10cb7505cdc04dab3eacf24c0c2cb158a998b8c7b1d27ee2c1f5/psutil-7.2.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e452c464a02e7dc7822a05d25db4cde564444a67e58539a00f929c51eddda0cf", size = 185210, upload-time = "2026-01-28T18:15:16.002Z" }, + { url = "https://files.pythonhosted.org/packages/dd/2c/ff9bfb544f283ba5f83ba725a3c5fec6d6b10b8f27ac1dc641c473dc390d/psutil-7.2.2-cp314-cp314t-win_amd64.whl", hash = "sha256:c7663d4e37f13e884d13994247449e9f8f574bc4655d509c3b95e9ec9e2b9dc1", size = 141228, upload-time = "2026-01-28T18:15:18.385Z" }, + { url = "https://files.pythonhosted.org/packages/f2/fc/f8d9c31db14fcec13748d373e668bc3bed94d9077dbc17fb0eebc073233c/psutil-7.2.2-cp314-cp314t-win_arm64.whl", hash = "sha256:11fe5a4f613759764e79c65cf11ebdf26e33d6dd34336f8a337aa2996d71c841", size = 136284, upload-time = "2026-01-28T18:15:19.912Z" }, + { url = "https://files.pythonhosted.org/packages/e7/36/5ee6e05c9bd427237b11b3937ad82bb8ad2752d72c6969314590dd0c2f6e/psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486", size = 129090, upload-time = "2026-01-28T18:15:22.168Z" }, + { url = "https://files.pythonhosted.org/packages/80/c4/f5af4c1ca8c1eeb2e92ccca14ce8effdeec651d5ab6053c589b074eda6e1/psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979", size = 129859, upload-time = "2026-01-28T18:15:23.795Z" }, + { url = "https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9", size = 155560, upload-time = "2026-01-28T18:15:25.976Z" }, + { url = "https://files.pythonhosted.org/packages/63/65/37648c0c158dc222aba51c089eb3bdfa238e621674dc42d48706e639204f/psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e", size = 156997, upload-time = "2026-01-28T18:15:27.794Z" }, + { url = "https://files.pythonhosted.org/packages/8e/13/125093eadae863ce03c6ffdbae9929430d116a246ef69866dad94da3bfbc/psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8", size = 148972, upload-time = "2026-01-28T18:15:29.342Z" }, + { url = "https://files.pythonhosted.org/packages/04/78/0acd37ca84ce3ddffaa92ef0f571e073faa6d8ff1f0559ab1272188ea2be/psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc", size = 148266, upload-time = "2026-01-28T18:15:31.597Z" }, + { url = "https://files.pythonhosted.org/packages/b4/90/e2159492b5426be0c1fef7acba807a03511f97c5f86b3caeda6ad92351a7/psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988", size = 137737, upload-time = "2026-01-28T18:15:33.849Z" }, + { url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617, upload-time = "2026-01-28T18:15:36.514Z" }, +] + [[package]] name = "pycparser" version = "3.0"